date: 2021-04-20title: Ceph与k8s完美集成 #标题
tags: 存储,k8s持久化 #标签
categories: k8s # 分类

本文目的是实现ceph和k8s完美集成，包括与volumes、pvc、sc等集成。

这里不讲ceph以及k8s集群如何安装，可以找我之前的博文或者使用其他方式将集群安装完成。

参考： 官方文档

ceph与volumes结合

参考：官方文档

k8s节点安装ceph工具

在k8s所有节点上安装。

  1. $ yum -y install ceph-common
  3. # 如果找不到软件包，则需要配置ceph仓库
  4. $ cat > /etc/yum.repos.d/ceph.repo << EOF
  5. [ceph-norch]
  6. name=ceph-norch
  7. baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/
  8. enabled=1
  9. gpgcheck=0
  10. [ceph-x86_64]
  11. name=ceph-x86_64 
  12. baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/
  13. enabled=1
  14. gpgcheck=0
  15. EOF

ceph集群中创建pool、块文件及认证用户

在任意一个ceph管理节点上执行。

  1. # 创建pool
  2. $ ceph osd pool create kubernetes 16 16
  4. # 初始化pool
  5. $ rbd pool init kubernetes
  7. # 创建块文件
  8. $ rbd create -p kubernetes --image-feature layering rbd.img --size 10G
  10. # 创建授权用户
  11. $ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
  12. [client.kubernetes]
  13.         key = AQCgD3xgde3HDRAA4iqESawxR8LgB3mAZ70fWQ==
  15. # 确认授权列表
  16. $ ceph auth get client.kubernetes    
  17. exported keyring for client.kubernetes
  18. [client.kubernetes]
  19.         key = AQCgD3xgde3HDRAA4iqESawxR8LgB3mAZ70fWQ==
  20.         caps mgr = "profile rbd pool=kubernetes"
  21.         caps mon = "profile rbd"
  22.         caps osd = "profile rbd pool=kubernetes"

创建secret资源对象

在k8s集群中进行如下操作。

  1. # 对ceph中创建的ceph授权key进行base64加密
  2. $ echo 'AQCgD3xgde3HDRAA4iqESawxR8LgB3mAZ70fWQ==' | base64
  3. QVFDZ0QzeGdkZTNIRFJBQTRpcUVTYXd4UjhMZ0IzbUFaNzBmV1E9PQo=
  5. # 编写secret资源对象
  6. cat > ceph_secret.yml << EOF
  7. apiVersion: v1
  8. kind: Secret
  9. metadata:
  10.   name: ceph-secret
  11. type: "kubernetes.io/rbd"
  12. data:
  13.   key: QVFDZ0QzeGdkZTNIRFJBQTRpcUVTYXd4UjhMZ0IzbUFaNzBmV1E9PQo=
  14. EOF
  16. # 上面的key指定的就是加密后的ceph授权秘钥
  18. # 创建secret资源对象
  19. $ kubectl apply -f ceph_secret.yml 
  20. secret/ceph-secret created
  22. # 确认已创建
  23. $ kubectl get secret/ceph-secret -o yaml        
  24. apiVersion: v1
  25. data:
  26.   key: QVFDZ0QzeGdkZTNIRFJBQTRpcUVTYXd4UjhMZ0IzbUFaNzBmV1E9PQo=
  27. kind: Secret
  28. metadata:
  29.        ............. # 忽略部分输出

k8s创建deployment资源对象使用ceph块设备

下面的yaml文件就考验各位对k8s的熟悉程度了，懒得解释。

  1. $ cat > nginx.yaml << EOF
  2. ---
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   name: web-nginx
  7.   labels:
  8.     k8s.cn/layer: web
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       k8s.cn/layer: web
  14.   template:
  15.     metadata:
  16.       labels:
  17.         k8s.cn/layer: web
  18.     spec:
  19.       containers:
  20.         - image: nginx
  21.           imagePullPolicy: IfNotPresent
  22.           name: nginx
  23.           ports:
  24.             - containerPort: 80
  25.               name: www
  26.               protocol: TCP
  27.           volumeMounts:
  28.             - mountPath: /data
  29.               name: ceph-demo
  30.       volumes:
  31.         - name: ceph-demo
  32.           rbd:
  33.             monitors:
  34.             - 192.168.20.10:6789
  35.             - 192.168.20.5:6789
  36.             - 192.168.20.6:6789
  37.             pool: kubernetes
  38.             image: rbd.img 
  39.             fsType: ext4 
  40.             user: kubernetes
  41.             secretRef:
  42.               name: ceph-secret
  43. EOF
  45. # 创建deployment
  46. $ kubectl apply -f nginx.yaml
  48. # 稍等片刻，确认容器运行成功
  49. $ kubectl get pods 
  50. NAME                                 READY   STATUS    RESTARTS   AGE
  51. web-nginx-6bf57f9cf7-dfvv4           1/1     Running   0          8s

进入容器进行验证

  1. # 查询pod
  2. $ kubectl get pods 
  3. NAME                                 READY   STATUS    RESTARTS   AGE
  4. web-nginx-6bf57f9cf7-dfvv4           1/1     Running   0          13s
  6. # 进入pod
  7. $ kubectl exec pod/web-nginx-6bf57f9cf7-dfvv4 -- /bin/bash
  9. # 查看/data 目录挂载情况
  10. root@web-nginx-6bf57f9cf7-dfvv4:/# df -hT data
  11. Filesystem     Type  Size  Used Avail Use% Mounted on
  12. /dev/rbd0      ext4  9.8G   37M  9.7G   1% /data

验证块设备挂载流程

经过上面的容器内验证，我们已经确认块设备成功挂载到了pod中，那么现在要看看这个块设备是如何挂载到pod中的。

  1. # 查看pod所在节点
  2. $ kubectl get pods -o wide
  3. NAME                                 READY   STATUS    RESTARTS   AGE    IP               NODE          NOMINATED NODE   READINESS GATES
  4. web-nginx-6bf57f9cf7-dfvv4           1/1     Running   0          11m    10.100.15.202    centos-20-4   <none>           <none>
  6. # centos-20-4 宿主机节点进行查看
  7. $ df -hT | grep rbd    # 可以看出来是先挂载到宿主机上，然后再映射到pod中的
  8. /dev/rbd0               ext4      9.8G   37M  9.7G    1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kubernetes-image-rbd.img

结论1：我自己删除pod后，使deployment控制器自动拉起pod，确定rbd.img中的数据不会丢失，说明如果你的rbd文件没有文件系统，那么会按照资源清单中指定的文件系统格式进行格式化，如果已存在文件系统，则直接挂载使用。

结论2：上面创建deployment的yaml文件不支持多副本，当扩容副本数量时，将会产生报错，一个rbd只能挂载到一个主机上。所以，如果想要扩展多个pod共享一个rbd块设备，那么只有指定一个固定节点运行pod了，下面是我创建的第二个deployment资源对象，通过nodeName字段指定其只能运行在centos-20-4 宿主机节点上，如下（除了资源对象名字以外，其他都一样）：

  1. $ cat nginx2.yaml 
  2. ---
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   name: web-nginx2
  7.   labels:
  8.     k8s.cn/layer: web2
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       k8s.cn/layer: web2
  14.   template:
  15.     metadata:
  16.       labels:
  17.         k8s.cn/layer: web2
  18.     spec:
  19.       nodeName: centos-20-4
  20.       containers:
  21.         - image: nginx
  22.           imagePullPolicy: IfNotPresent
  23.           name: nginx2
  24.           ports:
  25.             - containerPort: 80
  26.               name: www
  27.               protocol: TCP
  28.           volumeMounts:
  29.             - mountPath: /data
  30.               name: ceph-demo2
  31.       volumes:
  32.         - name: ceph-demo2
  33.           rbd:
  34.             monitors:
  35.             - 192.168.20.10:6789
  36.             - 192.168.20.5:6789
  37.             - 192.168.20.6:6789
  38.             pool: kubernetes
  39.             image: rbd.img 
  40.             fsType: ext4 
  41.             user: kubernetes
  42.             secretRef:
  43.               name: ceph-secret

Ceph与PV/PVC集成

准备工作

参考ceph与volumes结合小节，创建好pool、用户认证、secrets（如已创建，可以忽略，直接复用即可）。

  1. # ceph节点创建块文件
  2. $ rbd create -p kubernetes --image-feature layering demo-1.img --size 10G
  4. # 确认块设备文件已存在
  5. $ rbd -p kubernetes ls
  6. demo-1.img
  7. rbd.img

创建PV

接下来的所有yaml文件字段值，这里不展开解释，如果你不懂，请自行查阅其他资料。

  1. # 定义yaml文件
  2. $ cat > pv.yaml << EOF
  3. apiVersion: v1
  4. kind: PersistentVolume
  5. metadata:
  6.   name: rbd-demo
  7. spec:
  8.   accessModes:
  9.    - ReadWriteOnce
  10.   capacity:
  11.     storage: 10G
  12.   rbd:
  13.     monitors:
  14.      - 192.168.20.10:678
  15.      - 192.168.20.5:6789
  16.      - 192.168.20.6:6789
  17.     pool: kubernetes
  18.     image: demo-1.img 
  19.     fsType: ext4
  20.     user: kubernetes
  21.     secretRef:
  22.       name: ceph-secret
  23.   persistentVolumeReclaimPolicy: Retain
  24.   storageClassName: rbd
  25. EOF
  27. # 创建PV
  28. $ kubectl apply -f pv.yaml 
  29. persistentvolume/rbd-demo created
  31. # 确认PV已创建
  32. $ kubectl get pv           
  33. NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
  34. rbd-demo   10G        RWO            Retain           Available           rbd                     13s

定义PVC，引用PV

  1. # 定义pvc的yaml文件
  2. $ cat > pvc.yaml << EOF
  3. apiVersion: v1
  4. kind: PersistentVolumeClaim
  5. metadata:
  6.   name: pvc-demo
  7. spec:
  8.   accessModes:
  9.    - ReadWriteOnce        # 访问模式需要和PV一致
  10.   volumeName: rbd-demo     # 指定PV的名称
  11.   resources:
  12.     requests:
  13.       storage: 10G
  14.   storageClassName: rbd   # 这个字段需要和PV的yaml文件中的storageClassName字段值保持一致
  15. EOF
  18. # 创建pvc
  19. $ kubectl apply -f pvc.yaml 
  21. # 确认PVC已创建
  22. $ kubectl get pvc   
  23. NAME       STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  24. pvc-demo   Bound    rbd-demo   10G        RWO            rbd            100s

确认PVC和PV处于Bound状态

  1. $ kubectl get pv,pvc
  2. NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE
  3. persistentvolume/rbd-demo   10G        RWO            Retain           Bound    default/pvc-demo   rbd                     3m43s
  5. NAME                             STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  6. persistentvolumeclaim/pvc-demo   Bound    rbd-demo   10G        RWO            rbd            32s

创建deployment挂载PVC

  1. $ cat > nginx.yaml << EOF
  2. ---
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   name: web-nginx
  7.   labels:
  8.     k8s.cn/layer: web
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       k8s.cn/layer: web
  14.   template:
  15.     metadata:
  16.       labels:
  17.         k8s.cn/layer: web
  18.     spec:
  19.       containers:
  20.         - image: nginx
  21.           imagePullPolicy: IfNotPresent
  22.           name: nginx
  23.           ports:
  24.             - containerPort: 80
  25.               name: www
  26.               protocol: TCP
  27.           volumeMounts:
  28.             - mountPath: /data
  29.               name: rbd   # 此处需要和下面volumes.name字段的值一致
  30.       volumes:
  31.         - name: rbd 
  32.           persistentVolumeClaim:
  33.             claimName: pvc-demo    # 这里需要指定pvc的名字
  34. EOF
  36. # 创建deployment
  37. $ kubectl apply -f nginx.yaml 
  38. deployment.apps/web-nginx created
  40. # 确认deployment已创建
  41. $ kubectl get pods 
  42. NAME                         READY   STATUS    RESTARTS   AGE
  43. web-nginx-57b545d4d8-c9zk4   1/1     Running   0          4m

验证挂载情况

  1. $ kubectl exec -it web-nginx-57b545d4d8-c9zk4 -- /bin/bash
  2. root@web-nginx-57b545d4d8-c9zk4:/# df -hT /data
  3. Filesystem     Type  Size  Used Avail Use% Mounted on
  4. /dev/rbd0      ext4  9.8G   37M  9.7G   1% /data
  7. # 查看pod所在节点
  8. $ kubectl get pods -o wide
  9. NAME                         READY   STATUS    RESTARTS   AGE     IP              NODE          NOMINATED NODE   READINESS GATES
  10. web-nginx-57b545d4d8-c9zk4   1/1     Running   0          6m27s   10.100.15.205   centos-20-4   <none>           <none>
  12. # 查看宿主机挂载情况
  13. $ df -hT | grep rbd
  14. /dev/rbd0               ext4      9.8G   37M  9.7G    1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kubernetes-image-demo-1.img
  17. # 宿主机上的挂载目录和你pod中的data目录内容是完全一致的。

Ceph与SC集成架构

参考：官方文档。

在K8s v1.13及以后的版本中，可以通过Ceph -csi驱动使用Ceph块设备镜像，它动态地提供RBD映像来支持K8s存储卷，并将这些RBD镜像映射为运行引用RBD支持的卷的pods的工作节点上的块设备(可以将这些RBD镜像映射为镜像中包含的文件系统)。

下面是k8s使用ceph-csi驱动工作的一个架构示意图：

Ceph与k8s完美集成 - 图1

在上图中，k8s收到用户请求后，会通过ceph-csi去动态的创建PV、PVC以及ceph集群中的块文件，并且将其挂载到pod对应的节点上。

部署ceph-csi驱动程序

1、查询ceph集群信息（ceph控制节点上执行）

  1. $ ceph mon dump
  2. dumped monmap epoch 3
  3. epoch 3
  4. fsid d94fee92-ef1a-4f1f-80a5-1c7e1caf4a4a   # 我们需要这个id
  5. last_changed 2021-04-14 17:58:46.874896
  6. created 2021-04-14 17:54:27.836955
  7. min_mon_release 14 (nautilus)
  8. # 以及下面的monitor监听地址
  9. 0: [v2:192.168.20.10:3300/0,v1:192.168.20.10:6789/0] mon.centos-20-10
  10. 1: [v2:192.168.20.5:3300/0,v1:192.168.20.5:6789/0] mon.centos-20-5
  11. 2: [v2:192.168.20.6:3300/0,v1:192.168.20.6:6789/0] mon.centos-20-6

2、创建ceph-csi配置文件（k8s集群节点上执行）

  1. # 定义configmap文件
  2. $ cat <<EOF > csi-config-map.yaml
  3. ---
  4. apiVersion: v1
  5. kind: ConfigMap
  6. data:
  7.   config.json: |-
  8.     [
  9.       {
  10.         "clusterID": "d94fee92-ef1a-4f1f-80a5-1c7e1caf4a4a",  # 替换为你查出来的ceph集群id
  11.         "monitors": [         # 将下面替换为你ceph集群的monitor地址
  12.           "192.168.20.5:6789",
  13.           "192.168.20.6:6789",
  14.           "192.168.20.10:6789"
  15.         ]
  16.       }
  17.     ]
  18. metadata:
  19.   name: ceph-csi-config
  20. EOF
  22. # 创建configmap
  23. $ kubectl apply -f csi-config-map.yaml

3、定义认证信息文件

  1. # ceph集群中查询kubernetes用户对应的key
  2. # 如果后续创建pvc失败，可以尝试使用admin用户替换kubernetes用户
  3. # 正常来说kubernetes用户是没问题的
  4. $ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
  5. [client.kubernetes]
  6.         key = AQDMi35gIuRmIxAApx47Id2rsMPF33R5r4jrwQ==
  9. # k8s中编写secret文件
  10. $ cat <<EOF > csi-rbd-secret.yaml
  11. ---
  12. apiVersion: v1
  13. kind: Secret
  14. metadata:
  15.   name: csi-rbd-secret
  16.   namespace: default
  17. stringData:
  18.   userID: kubernetes
  19.   userKey: AQCadX5ggqz0GRAAuQuf/Ks3B7aJoK5L3SqXDQ==
  20. EOF
  22. # 创建secret
  23. $ kubectl apply -f csi-rbd-secret.yaml
  25. # ceph-csi还需要一个额外的ConfigMap对象来定义密钥管理服务(KMS)提供程序细节。
  26. # 如果没有设置KMS，则在csi-kms-config-map中放置一个空配置。
  27. # yaml文件或参考示例 https://github.com/ceph/ceph-csi/tree/master/examples/kms
  29. # 这里直接创建一个空的configmap即可，如下：
  30. $ cat <<EOF > csi-kms-config-map.yaml
  31. ---
  32. apiVersion: v1
  33. kind: ConfigMap
  34. data:
  35.   config.json: |-
  36.     {}
  37. metadata:
  38.   name: ceph-csi-encryption-kms-config
  39. EOF
  41.  # 创建configmap
  42.  $ kubectl apply -f csi-kms-config-map.yaml

4、创建RBAC授权认证

  1. $ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
  2. $ kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml

5、安装ceph-csi驱动插件

  1. # 如下是官方的yaml文件链接，注意：有问题
  2. $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
  3. $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

在官方提供的yaml文件中，已知有两个问题，具体如下：

  • 问题1：kubernetes/csi-rbdplugin-provisioner.yaml配置了节点亲和度，并且副本数为3，如果你的k8s集群中只有两个node节点，那么将会有一个pod调度失败，比如我这里是1个master，2个node节点，这样执行创建动作后，就会有一个副本调度失败（不要想着将副本数改为2，这个东西貌似涉及到leader选举，所以还是建议3副本），需要配置下污点容忍，使其可以被调度到master节点上，如果你的k8s集群中有三个及以上的node节点，那么可以忽略这个问题。
  • 问题2：官方的yaml文件中，定义的quay.io/cephcsi/cephcsi镜像标签都是canary版本，也就是开发测试版本，如果你是用于生产，还是建议去 官方镜像仓库 找到最新的稳定版，并替换为相应的版本号。

下面是我修改后的两个yaml文件（只是增加了一个污点容忍的配置并且修改了quay.io/cephcsi/cephcsi的镜像标签）：

  1. $ cat csi-rbdplugin-provisioner.yaml
  2. ---
  3. kind: Service
  4. apiVersion: v1
  5. metadata:
  6.   name: csi-rbdplugin-provisioner
  7.   labels:
  8.     app: csi-metrics
  9. spec:
  10.   selector:
  11.     app: csi-rbdplugin-provisioner
  12.   ports:
  13.     - name: http-metrics
  14.       port: 8080
  15.       protocol: TCP
  16.       targetPort: 8680
  18. ---
  19. kind: Deployment
  20. apiVersion: apps/v1
  21. metadata:
  22.   name: csi-rbdplugin-provisioner
  23. spec:
  24.   replicas: 3
  25.   selector:
  26.     matchLabels:
  27.       app: csi-rbdplugin-provisioner
  28.   template:
  29.     metadata:
  30.       labels:
  31.         app: csi-rbdplugin-provisioner
  32.     spec:
  33.       tolerations:      # 只是增加了这个字段
  34.         - effect: NoSchedule
  35.           key: node-role.kubernetes.io/master
  36.           operator: Exists
  37.       affinity:
  38.         podAntiAffinity:
  39.           requiredDuringSchedulingIgnoredDuringExecution:
  40.             - labelSelector:
  41.                 matchExpressions:
  42.                   - key: app
  43.                     operator: In
  44.                     values:
  45.                       - csi-rbdplugin-provisioner
  46.               topologyKey: "kubernetes.io/hostname"
  47.       serviceAccountName: rbd-csi-provisioner
  48.       priorityClassName: system-cluster-critical
  49.       containers:
  50.         - name: csi-provisioner
  51.           image: k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4
  52.           args:
  53.             - "--csi-address=$(ADDRESS)"
  54.             - "--v=5"
  55.             - "--timeout=150s"
  56.             - "--retry-interval-start=500ms"
  57.             - "--leader-election=true"
  58.             #  set it to true to use topology based provisioning
  59.             - "--feature-gates=Topology=false"
  60.             # if fstype is not specified in storageclass, ext4 is default
  61.             - "--default-fstype=ext4"
  62.             - "--extra-create-metadata=true"
  63.           env:
  64.             - name: ADDRESS
  65.               value: unix:///csi/csi-provisioner.sock
  66.           imagePullPolicy: "IfNotPresent"
  67.           volumeMounts:
  68.             - name: socket-dir
  69.               mountPath: /csi
  70.         - name: csi-snapshotter
  71.           image: k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
  72.           args:
  73.             - "--csi-address=$(ADDRESS)"
  74.             - "--v=5"
  75.             - "--timeout=150s"
  76.             - "--leader-election=true"
  77.           env:
  78.             - name: ADDRESS
  79.               value: unix:///csi/csi-provisioner.sock
  80.           imagePullPolicy: "IfNotPresent"
  81.           securityContext:
  82.             privileged: true
  83.           volumeMounts:
  84.             - name: socket-dir
  85.               mountPath: /csi
  86.         - name: csi-attacher
  87.           image: k8s.gcr.io/sig-storage/csi-attacher:v3.0.2
  88.           args:
  89.             - "--v=5"
  90.             - "--csi-address=$(ADDRESS)"
  91.             - "--leader-election=true"
  92.             - "--retry-interval-start=500ms"
  93.           env:
  94.             - name: ADDRESS
  95.               value: /csi/csi-provisioner.sock
  96.           imagePullPolicy: "IfNotPresent"
  97.           volumeMounts:
  98.             - name: socket-dir
  99.               mountPath: /csi
  100.         - name: csi-resizer
  101.           image: k8s.gcr.io/sig-storage/csi-resizer:v1.0.1
  102.           args:
  103.             - "--csi-address=$(ADDRESS)"
  104.             - "--v=5"
  105.             - "--timeout=150s"
  106.             - "--leader-election"
  107.             - "--retry-interval-start=500ms"
  108.             - "--handle-volume-inuse-error=false"
  109.           env:
  110.             - name: ADDRESS
  111.               value: unix:///csi/csi-provisioner.sock
  112.           imagePullPolicy: "IfNotPresent"
  113.           volumeMounts:
  114.             - name: socket-dir
  115.               mountPath: /csi
  116.         - name: csi-rbdplugin
  117.           securityContext:
  118.             privileged: true
  119.             capabilities:
  120.               add: ["SYS_ADMIN"]
  121.           # for stable functionality replace canary with latest release version
  122.           image: quay.io/cephcsi/cephcsi:v3.3.0
  123.           args:
  124.             - "--nodeid=$(NODE_ID)"
  125.             - "--type=rbd"
  126.             - "--controllerserver=true"
  127.             - "--endpoint=$(CSI_ENDPOINT)"
  128.             - "--v=5"
  129.             - "--drivername=rbd.csi.ceph.com"
  130.             - "--pidlimit=-1"
  131.             - "--rbdhardmaxclonedepth=8"
  132.             - "--rbdsoftmaxclonedepth=4"
  133.           env:
  134.             - name: POD_IP
  135.               valueFrom:
  136.                 fieldRef:
  137.                   fieldPath: status.podIP
  138.             - name: NODE_ID
  139.               valueFrom:
  140.                 fieldRef:
  141.                   fieldPath: spec.nodeName
  142.             # - name: POD_NAMESPACE
  143.             #   valueFrom:
  144.             #     fieldRef:
  145.             #       fieldPath: spec.namespace
  146.             # - name: KMS_CONFIGMAP_NAME
  147.             #   value: encryptionConfig
  148.             - name: CSI_ENDPOINT
  149.               value: unix:///csi/csi-provisioner.sock
  150.           imagePullPolicy: "IfNotPresent"
  151.           volumeMounts:
  152.             - name: socket-dir
  153.               mountPath: /csi
  154.             - mountPath: /dev
  155.               name: host-dev
  156.             - mountPath: /sys
  157.               name: host-sys
  158.             - mountPath: /lib/modules
  159.               name: lib-modules
  160.               readOnly: true
  161.             - name: ceph-csi-config
  162.               mountPath: /etc/ceph-csi-config/
  163.             - name: ceph-csi-encryption-kms-config
  164.               mountPath: /etc/ceph-csi-encryption-kms-config/
  165.             - name: keys-tmp-dir
  166.               mountPath: /tmp/csi/keys
  167.         - name: csi-rbdplugin-controller
  168.           securityContext:
  169.             privileged: true
  170.             capabilities:
  171.               add: ["SYS_ADMIN"]
  172.           # for stable functionality replace canary with latest release version
  173.           image: quay.io/cephcsi/cephcsi:v3.3.0
  174.           args:
  175.             - "--type=controller"
  176.             - "--v=5"
  177.             - "--drivername=rbd.csi.ceph.com"
  178.             - "--drivernamespace=$(DRIVER_NAMESPACE)"
  179.           env:
  180.             - name: DRIVER_NAMESPACE
  181.               valueFrom:
  182.                 fieldRef:
  183.                   fieldPath: metadata.namespace
  184.           imagePullPolicy: "IfNotPresent"
  185.           volumeMounts:
  186.             - name: ceph-csi-config
  187.               mountPath: /etc/ceph-csi-config/
  188.             - name: keys-tmp-dir
  189.               mountPath: /tmp/csi/keys
  190.         - name: liveness-prometheus
  191.           image: quay.io/cephcsi/cephcsi:v3.3.0
  192.           args:
  193.             - "--type=liveness"
  194.             - "--endpoint=$(CSI_ENDPOINT)"
  195.             - "--metricsport=8680"
  196.             - "--metricspath=/metrics"
  197.             - "--polltime=60s"
  198.             - "--timeout=3s"
  199.           env:
  200.             - name: CSI_ENDPOINT
  201.               value: unix:///csi/csi-provisioner.sock
  202.             - name: POD_IP
  203.               valueFrom:
  204.                 fieldRef:
  205.                   fieldPath: status.podIP
  206.           volumeMounts:
  207.             - name: socket-dir
  208.               mountPath: /csi
  209.           imagePullPolicy: "IfNotPresent"
  210.       volumes:
  211.         - name: host-dev
  212.           hostPath:
  213.             path: /dev
  214.         - name: host-sys
  215.           hostPath:
  216.             path: /sys
  217.         - name: lib-modules
  218.           hostPath:
  219.             path: /lib/modules
  220.         - name: socket-dir
  221.           emptyDir: {
  222.             medium: "Memory"
  223.           }
  224.         - name: ceph-csi-config
  225.           configMap:
  226.             name: ceph-csi-config
  227.         - name: ceph-csi-encryption-kms-config
  228.           configMap:
  229.             name: ceph-csi-encryption-kms-config
  230.         - name: keys-tmp-dir
  231.           emptyDir: {
  232.             medium: "Memory"
  233.           }
  246. $ cat csi-rbdplugin.yaml
  247. ---
  248. kind: DaemonSet
  249. apiVersion: apps/v1
  250. metadata:
  251.   name: csi-rbdplugin
  252. spec:
  253.   selector:
  254.     matchLabels:
  255.       app: csi-rbdplugin
  256.   template:
  257.     metadata:
  258.       labels:
  259.         app: csi-rbdplugin
  260.     spec:
  261.       serviceAccountName: rbd-csi-nodeplugin
  262.       hostNetwork: true
  263.       hostPID: true
  264.       priorityClassName: system-node-critical
  265.       # to use e.g. Rook orchestrated cluster, and mons' FQDN is
  266.       # resolved through k8s service, set dns policy to cluster first
  267.       dnsPolicy: ClusterFirstWithHostNet
  268.       containers:
  269.         - name: driver-registrar
  270.           # This is necessary only for systems with SELinux, where
  271.           # non-privileged sidecar containers cannot access unix domain socket
  272.           # created by privileged CSI driver container.
  273.           securityContext:
  274.             privileged: true
  275.           image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1
  276.           args:
  277.             - "--v=5"
  278.             - "--csi-address=/csi/csi.sock"
  279.             - "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
  280.           env:
  281.             - name: KUBE_NODE_NAME
  282.               valueFrom:
  283.                 fieldRef:
  284.                   fieldPath: spec.nodeName
  285.           volumeMounts:
  286.             - name: socket-dir
  287.               mountPath: /csi
  288.             - name: registration-dir
  289.               mountPath: /registration
  290.         - name: csi-rbdplugin
  291.           securityContext:
  292.             privileged: true
  293.             capabilities:
  294.               add: ["SYS_ADMIN"]
  295.             allowPrivilegeEscalation: true
  296.           # for stable functionality replace canary with latest release version
  297.           image: quay.io/cephcsi/cephcsi:v3.3.0
  298.           args:
  299.             - "--nodeid=$(NODE_ID)"
  300.             - "--type=rbd"
  301.             - "--nodeserver=true"
  302.             - "--endpoint=$(CSI_ENDPOINT)"
  303.             - "--v=5"
  304.             - "--drivername=rbd.csi.ceph.com"
  305.             # If topology based provisioning is desired, configure required
  306.             # node labels representing the nodes topology domain
  307.             # and pass the label names below, for CSI to consume and advertise
  308.             # its equivalent topology domain
  309.             # - "--domainlabels=failure-domain/region,failure-domain/zone"
  310.           env:
  311.             - name: POD_IP
  312.               valueFrom:
  313.                 fieldRef:
  314.                   fieldPath: status.podIP
  315.             - name: NODE_ID
  316.               valueFrom:
  317.                 fieldRef:
  318.                   fieldPath: spec.nodeName
  319.             # - name: POD_NAMESPACE
  320.             #   valueFrom:
  321.             #     fieldRef:
  322.             #       fieldPath: spec.namespace
  323.             # - name: KMS_CONFIGMAP_NAME
  324.             #   value: encryptionConfig
  325.             - name: CSI_ENDPOINT
  326.               value: unix:///csi/csi.sock
  327.           imagePullPolicy: "IfNotPresent"
  328.           volumeMounts:
  329.             - name: socket-dir
  330.               mountPath: /csi
  331.             - mountPath: /dev
  332.               name: host-dev
  333.             - mountPath: /sys
  334.               name: host-sys
  335.             - mountPath: /run/mount
  336.               name: host-mount
  337.             - mountPath: /lib/modules
  338.               name: lib-modules
  339.               readOnly: true
  340.             - name: ceph-csi-config
  341.               mountPath: /etc/ceph-csi-config/
  342.             - name: ceph-csi-encryption-kms-config
  343.               mountPath: /etc/ceph-csi-encryption-kms-config/
  344.             - name: plugin-dir
  345.               mountPath: /var/lib/kubelet/plugins
  346.               mountPropagation: "Bidirectional"
  347.             - name: mountpoint-dir
  348.               mountPath: /var/lib/kubelet/pods
  349.               mountPropagation: "Bidirectional"
  350.             - name: keys-tmp-dir
  351.               mountPath: /tmp/csi/keys
  352.         - name: liveness-prometheus
  353.           securityContext:
  354.             privileged: true
  355.           image: quay.io/cephcsi/cephcsi:v3.3.0
  356.           args:
  357.             - "--type=liveness"
  358.             - "--endpoint=$(CSI_ENDPOINT)"
  359.             - "--metricsport=8680"
  360.             - "--metricspath=/metrics"
  361.             - "--polltime=60s"
  362.             - "--timeout=3s"
  363.           env:
  364.             - name: CSI_ENDPOINT
  365.               value: unix:///csi/csi.sock
  366.             - name: POD_IP
  367.               valueFrom:
  368.                 fieldRef:
  369.                   fieldPath: status.podIP
  370.           volumeMounts:
  371.             - name: socket-dir
  372.               mountPath: /csi
  373.           imagePullPolicy: "IfNotPresent"
  374.       volumes:
  375.         - name: socket-dir
  376.           hostPath:
  377.             path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
  378.             type: DirectoryOrCreate
  379.         - name: plugin-dir
  380.           hostPath:
  381.             path: /var/lib/kubelet/plugins
  382.             type: Directory
  383.         - name: mountpoint-dir
  384.           hostPath:
  385.             path: /var/lib/kubelet/pods
  386.             type: DirectoryOrCreate
  387.         - name: registration-dir
  388.           hostPath:
  389.             path: /var/lib/kubelet/plugins_registry/
  390.             type: Directory
  391.         - name: host-dev
  392.           hostPath:
  393.             path: /dev
  394.         - name: host-sys
  395.           hostPath:
  396.             path: /sys
  397.         - name: host-mount
  398.           hostPath:
  399.             path: /run/mount
  400.         - name: lib-modules
  401.           hostPath:
  402.             path: /lib/modules
  403.         - name: ceph-csi-config
  404.           configMap:
  405.             name: ceph-csi-config
  406.         - name: ceph-csi-encryption-kms-config
  407.           configMap:
  408.             name: ceph-csi-encryption-kms-config
  409.         - name: keys-tmp-dir
  410.           emptyDir: {
  411.             medium: "Memory"
  412.           }
  413. ---
  414. # This is a service to expose the liveness metrics
  415. apiVersion: v1
  416. kind: Service
  417. metadata:
  418.   name: csi-metrics-rbdplugin
  419.   labels:
  420.     app: csi-metrics
  421. spec:
  422.   ports:
  423.     - name: http-metrics
  424.       port: 8080
  425.       protocol: TCP
  426.       targetPort: 8680
  427.   selector:
  428.     app: csi-rbdplugin

上面的两个yaml文件中需要使用几个k8s官方的镜像，所以我买了台云主机将镜像手动下载下来了，建议在安装插件之前，将我百度网盘中的镜像（提取码：1234）导入到你的k8s环境中，如果你有其他方式下载k8s官方的镜像，忽略这个建议即可。

当你导入镜像后，就可以执行下面两个指令创建相应的资源对象了：

  1. $ kubectl apply -f csi-rbdplugin-provisioner.yaml
  2. $ kubectl apply -f csi-rbdplugin.yaml

创建存储类

你可以定义多个存储类，比如你的ceph集群中有ssd和hdd两种类型的存储介质，那么你可以创建两种存储类，分别对应不同的存储介质。

  1. # 定义yaml文件
  2. $ cat <<EOF > csi-rbd-sc.yaml
  3. ---
  4. apiVersion: storage.k8s.io/v1
  5. kind: StorageClass
  6. metadata:
  7.    name: csi-rbd-sc
  8. provisioner: rbd.csi.ceph.com   # 指定所用驱动
  9. parameters:
  10.    clusterID: d94fee92-ef1a-4f1f-80a5-1c7e1caf4a4a    # 你的ceph集群id，可以通过ceph -s 指令查询
  11.    pool: kubernetes   # 指定你要使用ceph集群中的哪个pool
  12.    imageFeatures: layering
  13.    csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret   # 指定你访问ceph集群的用户及秘钥（我们将其写入到了secret资源对象中，所以指定这个secret资源对象即可）
  14.    csi.storage.k8s.io/provisioner-secret-namespace: default  # 上面指定的secret属于哪个名称空间
  15.    csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret  # 指定挂载时使用的用户资源（一般和上面保持一致）
  16.    csi.storage.k8s.io/node-stage-secret-namespace: default   # 同上
  17.    csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  18.    csi.storage.k8s.io/controller-expand-secret-namespace: default
  19. reclaimPolicy: Delete    # 指定回收策略
  20. allowVolumeExpansion: true
  21. mountOptions:
  22.    - discard
  23. EOF
  26. # 创建资源对象
  27. $ kubectl apply -f csi-rbd-sc.yaml
  29. # 确认sc已创建
  30. $ kubectl get sc
  31. NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
  32. csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                 2s

创建PVC动态申请空间

PVC支持块级别或文件系统级别去申请存储资源。一般推荐使用文件系统级别去申请，如果申请块设备，那么只能以块设备的方式映射到pod中，还需要在pod运行后对其进行磁盘格式化等操作，这个不太可能这么做，如果你想了解下如何申请块设备，可以参考官方文档（其实都差不多，只是将下面的Filesystem改成了Block，只是pod使用这个块时，就不能用volumeMounts这个关键字了，需要使用volumeDevices）：

  1. # 申请文件系统级别存储资源
  2. $ cat <<EOF > rbd-pvc.yaml
  3. ---
  4. apiVersion: v1
  5. kind: PersistentVolumeClaim
  6. metadata:
  7.   name: rbd-pvc
  8. spec:
  9.   accessModes:
  10.     - ReadWriteOnce
  11.   volumeMode: Filesystem
  12.   resources:
  13.     requests:
  14.       storage: 1Gi
  15.   storageClassName: csi-rbd-sc
  16. EOF
  18. # 创建PVC
  19. $ kubectl apply -f rbd-pvc.yaml

查看SC、PV、PVC资源对象

当我们PVC创建成功后，PVC会找SC去创建相应的PV，并将PV和PVC进行绑定。整个过程于用户而言，都是自动的。

  1. $ kubectl get pv,pvc,sc
  2. NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
  3. persistentvolume/pvc-74eb57c1-9cb5-442f-8dfe-71f2e869f4df   1Gi        RWO            Delete           Bound    default/rbd-pvc   csi-rbd-sc              18s
  5. NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  6. persistentvolumeclaim/rbd-pvc   Bound    pvc-74eb57c1-9cb5-442f-8dfe-71f2e869f4df   1Gi        RWO            csi-rbd-sc     18s
  8. NAME                                     PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
  9. storageclass.storage.k8s.io/csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   75m
  11. # 到ceph集群中查看，也有对应的一个块文件
  12. $ rbd -p kubernetes ls
  13. csi-vol-1310f455-a180-11eb-9aeb-6e7f74732123

至此，就实现了PV、PVC及SC整套流程的使用，也是生产中最为实用的一个存储解决方案。

创建pod挂载PVC

现在就定义一个deployment资源对象，来挂载使用下我们上面创建的PVC。

  1. $ cat > nginx.yaml << EOF
  2. ---
  3. apiVersion: apps/v1
  4. kind: Deployment
  5. metadata:
  6.   name: web-nginx
  7.   labels:
  8.     k8s.cn/layer: web
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       k8s.cn/layer: web
  14.   template:
  15.     metadata:
  16.       labels:
  17.         k8s.cn/layer: web
  18.     spec:
  19.       containers:
  20.         - image: nginx
  21.           imagePullPolicy: IfNotPresent
  22.           name: nginx
  23.           ports:
  24.             - containerPort: 80
  25.               name: www
  26.               protocol: TCP
  27.           volumeMounts:
  28.             - mountPath: /data
  29.               name: rbd    # 这里要和下面定义的volumes名字一致
  30.       volumes:
  31.         - name: rbd 
  32.           persistentVolumeClaim:
  33.             claimName: rbd-pvc   # 这里需要指定pvc的名字
  34. EOF
  38. # 创建deployment
  39. $ kubectl apply -f nginx.yaml

验证存储挂载情况

  1. # 自行查阅pod名称并进入pod
  2. $ kubectl exec -it web-nginx-7698cd7569-5bjf7 -- /bin/bash
  5. root@web-nginx-7698cd7569-5bjf7:/# df -hT /data      # 查看 /data 目录
  6. Filesystem     Type  Size  Used Avail Use% Mounted on
  7. /dev/rbd0      ext4  976M  2.6M  958M   1% /data

至此，已经实现了sc的功能，但由于我们使用deployment资源对象来验证的，并没有将其功能完全展示出来，我们还需要在使用前去创建PVC，但如果你使用StatefulSets来使用存储资源，那么将会更自动化些，下面来试试。

StorageClass最佳实践

一句话： StorageClass和StatefulSets结合使用，才可以展现其最大魅力。

下面的yaml文件参考官方文档。

  1. # 定义StatefulSets资源对象
  2. $ cat > nginx.yaml << EOF
  3. apiVersion: apps/v1
  4. kind: StatefulSet
  5. metadata:
  6.   name: web
  7. spec:
  8.   selector:
  9.     matchLabels:
  10.       app: nginx # has to match .spec.template.metadata.labels
  11.   serviceName: "nginx"
  12.   replicas: 3 # by default is 1
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: nginx # has to match .spec.selector.matchLabels
  17.     spec:
  18.       terminationGracePeriodSeconds: 10
  19.       containers:
  20.       - name: nginx
  21.         image: nginx
  22.         ports:
  23.         - containerPort: 80
  24.           name: web
  25.         volumeMounts:
  26.         - name: www
  27.           mountPath: /usr/share/nginx/html
  28.   volumeClaimTemplates:    # 主要是这一字段，通过定义卷模板，去创建相应的pvc，pv
  29.   - metadata:
  30.       name: www
  31.     spec:
  32.       accessModes: [ "ReadWriteOnce" ]
  33.       storageClassName: "csi-rbd-sc"
  34.       resources:
  35.         requests:
  36.           storage: 1Gi
  37. EOF
  40. # 创建statefulset资源对象
  41. $ kubectl apply -f nginx.yaml
  43. # 查看创建的资源对象（可以看到，自动创建了相应的pv，pvc并绑定）
  44. $ kubectl get sts,pvc,pv
  45. NAME                   READY   AGE
  46. statefulset.apps/web   3/3     20m
  48. NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  49. persistentvolumeclaim/www-web-0   Bound    pvc-aff8fcc7-41bb-467a-a3d7-e29bbdede904   1Gi        RWO            csi-rbd-sc     20m
  50. persistentvolumeclaim/www-web-1   Bound    pvc-cc929a1b-0401-48aa-bf2a-dc2dca28079a   1Gi        RWO            csi-rbd-sc     20m
  51. persistentvolumeclaim/www-web-2   Bound    pvc-b077642f-b802-4b1a-b02b-df02d55a8891   1Gi        RWO            csi-rbd-sc     19m
  53. NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
  54. persistentvolume/pvc-aff8fcc7-41bb-467a-a3d7-e29bbdede904   1Gi        RWO            Delete           Bound    default/www-web-0   csi-rbd-sc              20m
  55. persistentvolume/pvc-b077642f-b802-4b1a-b02b-df02d55a8891   1Gi        RWO            Delete           Bound    default/www-web-2   csi-rbd-sc              19m
  56. persistentvolume/pvc-cc929a1b-0401-48aa-bf2a-dc2dca28079a   1Gi        RWO            Delete           Bound    default/www-web-1   csi-rbd-sc              20m
  58. $ kubectl get pods | grep web
  59. web-0                                        1/1     Running   0          19m
  60. web-1                                        1/1     Running   0          19m
  61. web-2                                        1/1     Running   0          18m