1、RBD块存储概述
什么是块存储：A block is a sequence of bytes (often 512). Block-based storage interfaces are a mature and common way to store data on media including HDDs, SSDs, CDs, floppy disks, and even tape
EBS
CBS
FC块 
Thin-provisioned
Images up to 16 exabytes
Configurable striping
In-memory caching
Snapshots
Copy-on-write cloning
Kernel driver support
KVM/libvirt support
Back-end for cloud solutions
Incremental backup
2、RBD与容器对接
三种⽅式
volume，卷的存储⽅式，⽀持多种驱动，FC，EBS，Ceph等
PV/PVC，Persistent Volume和Persistent Volume Claim
StorageClass，包含静态+动态两种 管理员定义好provioner 终端⽤户通过PVC关联
参考Ceph官⽅⽂档https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/，Ceph和kubernetes的对 接过程涉及到pool的创建，Ceph认证信息，配置⽂件，CSI驱动部署，StorageClass创建等⼀系列过 程。配置过程有⼀定的难度，如果对Ceph不熟悉的人，对接可能有⼀定难度，⽽Rook则将这些配置 过程简化，以云原⽣的⽅式实现对接，其默认已经继承好相关驱动，直接通过kubernetes创建 storageclass即可。
[root@master1 rbd]# kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-gwhh5 3/3 Running 0 18h
csi-cephfsplugin-jppz4 3/3 Running 0 18h
csi-cephfsplugin-provisioner-7dcc8ff54d-8jrfn 6/6 Running 0 18h
csi-cephfsplugin-provisioner-7dcc8ff54d-k68tk 6/6 Running 0 18h
csi-cephfsplugin-q7jzk 3/3 Running 0 18h
csi-rbdplugin-2f5lr 3/3 Running 0 18h
csi-rbdplugin-provisioner-d8bcc5fc4-5s825 6/6 Running 0 18h
csi-rbdplugin-provisioner-d8bcc5fc4-lqrkb 6/6 Running 0 18h
csi-rbdplugin-slnjr 3/3 Running 0 18h
csi-rbdplugin-sshhc 3/3 Running 0 18h
rook-ceph-crashcollector-master2-79959c4c4d-gd7vc 1/1 Running 0 18h
rook-ceph-crashcollector-node1-8876bb88c-mlz98 1/1 Running 0 18h
rook-ceph-crashcollector-node2-6f5c95d7f8-wkmrh 1/1 Running 0 18h
rook-ceph-mgr-a-78cd69f66d-tvf2p 1/1 Running 0 15h
rook-ceph-mon-a-5d76df486c-l42l2 1/1 Running 0 15h
rook-ceph-mon-b-5bdb94c779-bc5h8 1/1 Running 0 15h
rook-ceph-mon-c-74d55c5978-5xwr6 1/1 Running 0 15h
rook-ceph-operator-66c5bf9486-wcsfc 1/1 Running 0 18h
rook-ceph-osd-0-58658f4c7f-mcfqk 1/1 Running 0 15h
rook-ceph-osd-1-5b7f4764b9-4jf8j 1/1 Running 0 15h
rook-ceph-osd-2-7d579cbfcb-26nvq 1/1 Running 0 15h
rook-ceph-osd-prepare-master2-vsxvg 0/1 Completed 0 15h
rook-ceph-osd-prepare-node1-45gqg 0/1 Completed 0 15h
rook-ceph-osd-prepare-node2-p4q4m 0/1 Completed 0 15h
CSI驱动信息
包含rbd和cephfs的驱动，csi-cephfsplugin和csi-rbdplugin
驱动由provisioner和plugin组成
3、RBD块存储类
RBD相关的存储驱动和provisioner安装rook时候已经创建好，因此接下来只需要直接对接即可，Rook 提供了两种对接的⽅式：
1、lexVolume
2、CSI
其中Flex的⽅式⽐较⽼，默认驱动未安装，需要安装才可以对接，逐步淘汰，不建议使⽤，推荐使⽤CSI 的对接⽅式 。
[root@master1 rbd]# cat storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
—-
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: rook-ceph # namespace:cluster
pool: replicapool
imageFormat: “2”
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/fstype: ext4
allowVolumeExpansion: true
reclaimPolicy: Delete
校验pool安装情况
[root@rook-ceph-tools-54fc95f4f4-mg67d ceph]# ceph osd lspools
1 device_health_metrics
2 replicapool
校验StorageClass
[root@master1 rbd]# kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 73m
创建好storageclass之后，我们就可以通过PVC向storageclass申请容量空间了，PVC会⾃动和 storageclass完成存储容量的创建过程，包括⾃动创建PV，PV与后端存储⾃动完成RBD块存储的创建， 整个过程不需要我们关⼼，均通过storageclass和驱动⾃动完成，我们只需要关注使⽤即可，如下是⼀ 个wordpress博客应⽤连接MySQL数据库的⼀个云原⽣应⽤的范例
[root@master1 kubernetes]# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
labels:
app: wordpress
spec:
ports:
- port: 3306
selector:
app: wordpress
tier: mysql
clusterIP: None
—-
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
—-
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
labels:
app: wordpress
tier: mysql
spec:
selector:
matchLabels:
app: wordpress
tier: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: changeme
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
wordpress
[root@master1 kubernetes]# cat wordpress.yaml
apiVersion: v1
kind: Service
metadata:
name: wordpress
labels:
app: wordpress
spec:
ports:
- port: 80
selector:
app: wordpress
tier: frontend
type: LoadBalancer
—-
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
—-
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
tier: frontend
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
containers:
- image: wordpress:4.6.1-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
value: changeme
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim
4、PVC调⽤逻辑
PVC会完成⼀系列的出对接过程，包含有什么动作呢？PVC—>storageclass申请容量—>创建PV——>向 Ceph申请RBD块，完成和Ceph对接
[root@master1 kubernetes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-d0b0cf50-ecfc-41fe-8a10-faed6f466f63 20Gi RWO rook-ceph-block 3m58s
wp-pv-claim Bound pvc-221c1501-6be3-43ea-b9b3-e610f6f23dde 20Gi RWO rook-ceph-block 4m4s
PVC会⾃动创建PV
[root@master1 kubernetes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
pvc-221c1501-6be3-43ea-b9b3-e610f6f23dde 20Gi RWO Delete Bound default/wp-pv-claim rook-ceph-block 5m7s
pvc-d0b0cf50-ecfc-41fe-8a10-faed6f466f63 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 5m3s
5、Wordpress功能验证
wordpress和mysql实际存储空间存放在Ceph RBD块中，通过LoadBalancer的登录MySQL，验证 wordpress的安装情况，⾸先需要核对pods部署是否正常，确保均Running
6、 存储持久化模板
PVC使⽤模式适⽤于单个pods容器，如多个pods都需要有各⾃的存储如何实现呢？需要借助于 StatefulSet的volumeClaimTemplates功能，实现每个pods均有各⾃的存储。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx
serviceName: “nginx”
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- mountPath: /usr/share/nginx/html
name: www
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes:
- ReadWriteOnce
storageClassName: “rook-ceph-block”
resources:
requests:
storage: 10Gi
[root@master1 ~]# kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 104s
web-1 1/1 Running 0 62s
web-2 1/1 Running 0 27s
我们仔细观察整个过程出现了3个 Pod：web-0 和 web-1 and web-2。而且这3个 Pod 是按照顺序进行创建的，web-0 和web-1启动起来后 web-2 才开始创建。
StatefulSet 中的 Pod 拥有一个具有稳定的、独一无二的身份标志。这个标志基于 StatefulSet 控制器分配给每个 Pod 的唯一顺序索引。Pod 的名称的形式为
StatefulSet 中 Pod 副本的创建会按照序列号升序处理，副本的更新和删除会按照序列号降序处理。
若有收获，就点个赞吧
0 人点赞
