k8s-v1.25.0持久化存储ceph-csi方式（ceph-v17.2.3、rbd块存储）

ceph-v17.2.3对接k8s-v1.25.0（动态存储）

1、cephadm部署ceph-v17.2.3集群

2、k8s-v1.25.0部署ceph-csi-v3.7.0

ceph集群+k8s集群（ubuntu22.04）

ceph-v17.2.3

k8s-v1.25.0（containerd-1.6.8）

ceph-csi-v3.7.0

动态pvc

ceph集群操作

  1. ceph osd pool create kubernetes 8 8
  3. ceph osd pool application enable kubernetes rbd
  5. rbd pool init -p kubernetes
  7. ceph osd pool ls
  1. ceph mon dump
  1. ceph auth get client.admin 2>&1 |grep "key = " |awk '{print  $3'}
  1. ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kubernetes'
  3. ceph auth get-key client.kube

k8s集群操作

  1. apt update
  3. apt install ceph-common=17.2.0-0ubuntu0.22.04.1 -y
  1. mkdir ~/{ceph-csi,000}
  3. git clone https://github.com/ceph/ceph-csi.git
  5. cd ~/000 && git clone https://hub.fastgit.xyz/ceph/ceph-csi.git
  1. kubectl create ns ceph-csi
  3. kubectl describe node master |grep Taint
  4. kubectl taint node master node-role.kubernetes.io/master-

1、csi-config-map.yaml

  1. # 1、csi-config-map.yaml
  2. cat > ~/ceph-csi/csi-config-map.yaml << 'EOF' 
  3. ---
  4. apiVersion: v1
  5. kind: ConfigMap
  6. data:
  7.   config.json: |-
  8.     [
  9.       {
  10.         "clusterID": "4c4d9c82-267d-11ed-888e-000c29db6f93",
  11.         "monitors": [
  12.           "192.168.1.204:6789",
  13.           "192.168.1.205:6789",
  14.           "192.168.1.206:6789",
  15.           "192.168.1.207:6789"
  16.         ]
  17.       }
  18.     ]
  19. metadata:
  20.   name: ceph-csi-config
  21.   namespace: ceph-csi
  22. EOF

clusterID使用ceph mon dump命令在ceph集群上查看

2、csi-nodeplugin-rbac.yaml

3、csi-provisioner-rbac.yaml

4、csidriver.yaml

5、csi-rbdplugin-provisioner.yaml

6、csi-rbdplugin.yaml

  1. cat > ceph.yml < 'eof'
  2. # 2、csi-nodeplugin-rbac.yaml
  3. cat > ~/ceph-csi/csi-nodeplugin-rbac.yaml << 'EOF'
  4. ---
  5. apiVersion: v1
  6. kind: ServiceAccount
  7. metadata:
  8.   name: rbd-csi-nodeplugin
  9.   namespace: ceph-csi
  10. ---
  11. kind: ClusterRole
  12. apiVersion: rbac.authorization.k8s.io/v1
  13. metadata:
  14.   name: rbd-csi-nodeplugin
  15. rules:
  16.   - apiGroups: [""]
  17.     resources: ["nodes"]
  18.     verbs: ["get"]
  19.   - apiGroups: [""]
  20.     resources: ["secrets"]
  21.     verbs: ["get"]
  22.   - apiGroups: [""]
  23.     resources: ["configmaps"]
  24.     verbs: ["get"]
  25.   - apiGroups: [""]
  26.     resources: ["serviceaccounts"]
  27.     verbs: ["get"]
  28.   - apiGroups: [""]
  29.     resources: ["persistentvolumes"]
  30.     verbs: ["get"]
  31.   - apiGroups: ["storage.k8s.io"]
  32.     resources: ["volumeattachments"]
  33.     verbs: ["list", "get"]
  34.   - apiGroups: [""]
  35.     resources: ["serviceaccounts/token"]
  36.     verbs: ["create"]
  37. ---
  38. kind: ClusterRoleBinding
  39. apiVersion: rbac.authorization.k8s.io/v1
  40. metadata:
  41.   name: rbd-csi-nodeplugin
  42. subjects:
  43.   - kind: ServiceAccount
  44.     name: rbd-csi-nodeplugin
  45.     namespace: ceph-csi
  46. roleRef:
  47.   kind: ClusterRole
  48.   name: rbd-csi-nodeplugin
  49.   apiGroup: rbac.authorization.k8s.io
  50. EOF
  52. # 3、csi-provisioner-rbac.yaml
  53. cat > ~/ceph-csi/csi-provisioner-rbac.yaml << 'EOF'
  54. ---
  55. apiVersion: v1
  56. kind: ServiceAccount
  57. metadata:
  58.   name: rbd-csi-provisioner
  59.   namespace: ceph-csi
  61. ---
  62. kind: ClusterRole
  63. apiVersion: rbac.authorization.k8s.io/v1
  64. metadata:
  65.   name: rbd-external-provisioner-runner
  66. rules:
  67.   - apiGroups: [""]
  68.     resources: ["nodes"]
  69.     verbs: ["get", "list", "watch"]
  70.   - apiGroups: [""]
  71.     resources: ["secrets"]
  72.     verbs: ["get", "list", "watch"]
  73.   - apiGroups: [""]
  74.     resources: ["events"]
  75.     verbs: ["list", "watch", "create", "update", "patch"]
  76.   - apiGroups: [""]
  77.     resources: ["persistentvolumes"]
  78.     verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
  79.   - apiGroups: [""]
  80.     resources: ["persistentvolumeclaims"]
  81.     verbs: ["get", "list", "watch", "update"]
  82.   - apiGroups: [""]
  83.     resources: ["persistentvolumeclaims/status"]
  84.     verbs: ["update", "patch"]
  85.   - apiGroups: ["storage.k8s.io"]
  86.     resources: ["storageclasses"]
  87.     verbs: ["get", "list", "watch"]
  88.   - apiGroups: ["snapshot.storage.k8s.io"]
  89.     resources: ["volumesnapshots"]
  90.     verbs: ["get", "list", "patch"]
  91.   - apiGroups: ["snapshot.storage.k8s.io"]
  92.     resources: ["volumesnapshots/status"]
  93.     verbs: ["get", "list", "patch"]
  94.   - apiGroups: ["snapshot.storage.k8s.io"]
  95.     resources: ["volumesnapshotcontents"]
  96.     verbs: ["create", "get", "list", "watch", "update", "delete", "patch"]
  97.   - apiGroups: ["snapshot.storage.k8s.io"]
  98.     resources: ["volumesnapshotclasses"]
  99.     verbs: ["get", "list", "watch"]
  100.   - apiGroups: ["storage.k8s.io"]
  101.     resources: ["volumeattachments"]
  102.     verbs: ["get", "list", "watch", "update", "patch"]
  103.   - apiGroups: ["storage.k8s.io"]
  104.     resources: ["volumeattachments/status"]
  105.     verbs: ["patch"]
  106.   - apiGroups: ["storage.k8s.io"]
  107.     resources: ["csinodes"]
  108.     verbs: ["get", "list", "watch"]
  109.   - apiGroups: ["snapshot.storage.k8s.io"]
  110.     resources: ["volumesnapshotcontents/status"]
  111.     verbs: ["update", "patch"]
  112.   - apiGroups: [""]
  113.     resources: ["configmaps"]
  114.     verbs: ["get"]
  115.   - apiGroups: [""]
  116.     resources: ["serviceaccounts"]
  117.     verbs: ["get"]
  118.   - apiGroups: [""]
  119.     resources: ["serviceaccounts/token"]
  120.     verbs: ["create"]
  121. ---
  122. kind: ClusterRoleBinding
  123. apiVersion: rbac.authorization.k8s.io/v1
  124. metadata:
  125.   name: rbd-csi-provisioner-role
  126. subjects:
  127.   - kind: ServiceAccount
  128.     name: rbd-csi-provisioner
  129.     namespace: ceph-csi
  130. roleRef:
  131.   kind: ClusterRole
  132.   name: rbd-external-provisioner-runner
  133.   apiGroup: rbac.authorization.k8s.io
  135. ---
  136. kind: Role
  137. apiVersion: rbac.authorization.k8s.io/v1
  138. metadata:
  139.   namespace: ceph-csi
  140.   name: rbd-external-provisioner-cfg
  141. rules:
  142.   - apiGroups: [""]
  143.     resources: ["configmaps"]
  144.     verbs: ["get", "list", "watch", "create", "update", "delete"]
  145.   - apiGroups: ["coordination.k8s.io"]
  146.     resources: ["leases"]
  147.     verbs: ["get", "watch", "list", "delete", "update", "create"]
  149. ---
  150. kind: RoleBinding
  151. apiVersion: rbac.authorization.k8s.io/v1
  152. metadata:
  153.   name: rbd-csi-provisioner-role-cfg
  154.   namespace: ceph-csi
  155. subjects:
  156.   - kind: ServiceAccount
  157.     name: rbd-csi-provisioner
  158.     namespace: ceph-csi
  159. roleRef:
  160.   kind: Role
  161.   name: rbd-external-provisioner-cfg
  162.   apiGroup: rbac.authorization.k8s.io
  163. EOF
  165. # 4、csidriver.yaml
  166. cat > ~/ceph-csi/csidriver.yaml << 'EOF'
  167. apiVersion: storage.k8s.io/v1
  168. kind: CSIDriver
  169. metadata:
  170.   name: "rbd.csi.ceph.com"
  171.   namespace: ceph-csi
  172. spec:
  173.   attachRequired: true
  174.   podInfoOnMount: false
  175. EOF
  177. # 5、csi-rbdplugin-provisioner.yaml
  178. cat > ~/ceph-csi/csi-rbdplugin-provisioner.yaml << 'EOF'
  179. ---
  180. kind: Service
  181. apiVersion: v1
  182. metadata:
  183.   name: csi-rbdplugin-provisioner
  184.   namespace: ceph-csi
  185.   labels:
  186.     app: csi-metrics
  187. spec:
  188.   selector:
  189.     app: csi-rbdplugin-provisioner
  190.   ports:
  191.     - name: http-metrics
  192.       port: 8080
  193.       protocol: TCP
  194.       targetPort: 8680
  196. ---
  197. kind: Deployment
  198. apiVersion: apps/v1
  199. metadata:
  200.   name: csi-rbdplugin-provisioner
  201.   namespace: ceph-csi
  202. spec:
  203.   replicas: 3
  204.   selector:
  205.     matchLabels:
  206.       app: csi-rbdplugin-provisioner
  207.   template:
  208.     metadata:
  209.       labels:
  210.         app: csi-rbdplugin-provisioner
  211.     spec:
  212.       affinity:
  213.         podAntiAffinity:
  214.           requiredDuringSchedulingIgnoredDuringExecution:
  215.             - labelSelector:
  216.                 matchExpressions:
  217.                   - key: app
  218.                     operator: In
  219.                     values:
  220.                       - csi-rbdplugin-provisioner
  221.               topologyKey: "kubernetes.io/hostname"
  222.       serviceAccountName: rbd-csi-provisioner
  223.       priorityClassName: system-cluster-critical
  224.       containers:
  225.         - name: csi-provisioner
  226.           image: dyrnq/csi-provisioner:v3.2.1
  227.           args:
  228.             - "--csi-address=$(ADDRESS)"
  229.             - "--v=1"
  230.             - "--timeout=150s"
  231.             - "--retry-interval-start=500ms"
  232.             - "--leader-election=true"
  233.             - "--feature-gates=Topology=false"
  234.             - "--feature-gates=HonorPVReclaimPolicy=true"
  235.             - "--prevent-volume-mode-conversion=true"
  236.             - "--default-fstype=ext4"
  237.             - "--extra-create-metadata=true"
  238.           env:
  239.             - name: ADDRESS
  240.               value: unix:///csi/csi-provisioner.sock
  241.           imagePullPolicy: "IfNotPresent"
  242.           volumeMounts:
  243.             - name: socket-dir
  244.               mountPath: /csi
  245.         - name: csi-snapshotter
  246.           image: dyrnq/csi-snapshotter:v6.0.1
  247.           args:
  248.             - "--csi-address=$(ADDRESS)"
  249.             - "--v=1"
  250.             - "--timeout=150s"
  251.             - "--leader-election=true"
  252.             - "--extra-create-metadata=true"
  253.           env:
  254.             - name: ADDRESS
  255.               value: unix:///csi/csi-provisioner.sock
  256.           imagePullPolicy: "IfNotPresent"
  257.           volumeMounts:
  258.             - name: socket-dir
  259.               mountPath: /csi
  260.         - name: csi-attacher
  261.           image: dyrnq/csi-attacher:v3.5.0
  262.           args:
  263.             - "--v=1"
  264.             - "--csi-address=$(ADDRESS)"
  265.             - "--leader-election=true"
  266.             - "--retry-interval-start=500ms"
  267.           env:
  268.             - name: ADDRESS
  269.               value: /csi/csi-provisioner.sock
  270.           imagePullPolicy: "IfNotPresent"
  271.           volumeMounts:
  272.             - name: socket-dir
  273.               mountPath: /csi
  274.         - name: csi-resizer
  275.           image: dyrnq/csi-resizer:v1.5.0
  276.           args:
  277.             - "--csi-address=$(ADDRESS)"
  278.             - "--v=1"
  279.             - "--timeout=150s"
  280.             - "--leader-election"
  281.             - "--retry-interval-start=500ms"
  282.             - "--handle-volume-inuse-error=false"
  283.             - "--feature-gates=RecoverVolumeExpansionFailure=true"
  284.           env:
  285.             - name: ADDRESS
  286.               value: unix:///csi/csi-provisioner.sock
  287.           imagePullPolicy: "IfNotPresent"
  288.           volumeMounts:
  289.             - name: socket-dir
  290.               mountPath: /csi
  291.         - name: csi-rbdplugin
  292.           image: quay.io/cephcsi/cephcsi:canary
  293.           args:
  294.             - "--nodeid=$(NODE_ID)"
  295.             - "--type=rbd"
  296.             - "--controllerserver=true"
  297.             - "--endpoint=$(CSI_ENDPOINT)"
  298.             - "--csi-addons-endpoint=$(CSI_ADDONS_ENDPOINT)"
  299.             - "--v=5"
  300.             - "--drivername=rbd.csi.ceph.com"
  301.             - "--pidlimit=-1"
  302.             - "--rbdhardmaxclonedepth=8"
  303.             - "--rbdsoftmaxclonedepth=4"
  304.             - "--enableprofiling=false"
  305.             - "--setmetadata=true"
  306.           env:
  307.             - name: POD_IP
  308.               valueFrom:
  309.                 fieldRef:
  310.                   fieldPath: status.podIP
  311.             - name: NODE_ID
  312.               valueFrom:
  313.                 fieldRef:
  314.                   fieldPath: spec.nodeName
  315.             - name: POD_NAMESPACE
  316.               valueFrom:
  317.                 fieldRef:
  318.                   fieldPath: metadata.namespace
  319.             - name: CSI_ENDPOINT
  320.               value: unix:///csi/csi-provisioner.sock
  321.             - name: CSI_ADDONS_ENDPOINT
  322.               value: unix:///csi/csi-addons.sock
  323.           imagePullPolicy: "IfNotPresent"
  324.           volumeMounts:
  325.             - name: socket-dir
  326.               mountPath: /csi
  327.             - mountPath: /dev
  328.               name: host-dev
  329.             - mountPath: /sys
  330.               name: host-sys
  331.             - mountPath: /lib/modules
  332.               name: lib-modules
  333.               readOnly: true
  334.             - name: ceph-csi-config
  335.               mountPath: /etc/ceph-csi-config/
  336.             - name: keys-tmp-dir
  337.               mountPath: /tmp/csi/keys
  338.             - name: oidc-token
  339.               mountPath: /run/secrets/tokens
  340.               readOnly: true
  341.         - name: csi-rbdplugin-controller
  342.           image: quay.io/cephcsi/cephcsi:canary
  343.           args:
  344.             - "--type=controller"
  345.             - "--v=5"
  346.             - "--drivername=rbd.csi.ceph.com"
  347.             - "--drivernamespace=$(DRIVER_NAMESPACE)"
  348.             - "--setmetadata=true"
  349.           env:
  350.             - name: DRIVER_NAMESPACE
  351.               valueFrom:
  352.                 fieldRef:
  353.                   fieldPath: metadata.namespace
  354.           imagePullPolicy: "IfNotPresent"
  355.           volumeMounts:
  356.             - name: ceph-csi-config
  357.               mountPath: /etc/ceph-csi-config/
  358.             - name: keys-tmp-dir
  359.               mountPath: /tmp/csi/keys
  360.         - name: liveness-prometheus
  361.           image: quay.io/cephcsi/cephcsi:canary
  362.           args:
  363.             - "--type=liveness"
  364.             - "--endpoint=$(CSI_ENDPOINT)"
  365.             - "--metricsport=8680"
  366.             - "--metricspath=/metrics"
  367.             - "--polltime=60s"
  368.             - "--timeout=3s"
  369.           env:
  370.             - name: CSI_ENDPOINT
  371.               value: unix:///csi/csi-provisioner.sock
  372.             - name: POD_IP
  373.               valueFrom:
  374.                 fieldRef:
  375.                   fieldPath: status.podIP
  376.           volumeMounts:
  377.             - name: socket-dir
  378.               mountPath: /csi
  379.           imagePullPolicy: "IfNotPresent"
  380.       volumes:
  381.         - name: host-dev
  382.           hostPath:
  383.             path: /dev
  384.         - name: host-sys
  385.           hostPath:
  386.             path: /sys
  387.         - name: lib-modules
  388.           hostPath:
  389.             path: /lib/modules
  390.         - name: socket-dir
  391.           emptyDir: {
  392.             medium: "Memory"
  393.           }
  394.         - name: ceph-csi-config
  395.           configMap:
  396.             name: ceph-csi-config
  397.         - name: keys-tmp-dir
  398.           emptyDir: {
  399.             medium: "Memory"
  400.           }
  401.         - name: oidc-token
  402.           projected:
  403.             sources:
  404.               - serviceAccountToken:
  405.                   path: oidc-token
  406.                   expirationSeconds: 3600
  407.                   audience: ceph-csi-kms
  408. EOF
  410. # 6、csi-rbdplugin.yaml
  411. cat > ~/ceph-csi/csi-rbdplugin.yaml << 'EOF'
  412. ---
  413. kind: DaemonSet
  414. apiVersion: apps/v1
  415. metadata:
  416.   name: csi-rbdplugin
  417.   namespace: ceph-csi
  418. spec:
  419.   selector:
  420.     matchLabels:
  421.       app: csi-rbdplugin
  422.   template:
  423.     metadata:
  424.       labels:
  425.         app: csi-rbdplugin
  426.     spec:
  427.       serviceAccountName: rbd-csi-nodeplugin
  428.       hostNetwork: true
  429.       hostPID: true
  430.       priorityClassName: system-node-critical
  431.       dnsPolicy: ClusterFirstWithHostNet
  432.       containers:
  433.         - name: driver-registrar
  434.           securityContext:
  435.             privileged: true
  436.             allowPrivilegeEscalation: true
  437.           image: dyrnq/csi-node-driver-registrar:v2.5.1
  438.           args:
  439.             - "--v=1"
  440.             - "--csi-address=/csi/csi.sock"
  441.             - "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
  442.           env:
  443.             - name: KUBE_NODE_NAME
  444.               valueFrom:
  445.                 fieldRef:
  446.                   fieldPath: spec.nodeName
  447.           volumeMounts:
  448.             - name: socket-dir
  449.               mountPath: /csi
  450.             - name: registration-dir
  451.               mountPath: /registration
  452.         - name: csi-rbdplugin
  453.           securityContext:
  454.             privileged: true
  455.             capabilities:
  456.               add: ["SYS_ADMIN"]
  457.             allowPrivilegeEscalation: true
  458.           image: quay.io/cephcsi/cephcsi:canary
  459.           args:
  460.             - "--nodeid=$(NODE_ID)"
  461.             - "--pluginpath=/var/lib/kubelet/plugins"
  462.             - "--stagingpath=/var/lib/kubelet/plugins/kubernetes.io/csi/"
  463.             - "--type=rbd"
  464.             - "--nodeserver=true"
  465.             - "--endpoint=$(CSI_ENDPOINT)"
  466.             - "--csi-addons-endpoint=$(CSI_ADDONS_ENDPOINT)"
  467.             - "--v=5"
  468.             - "--drivername=rbd.csi.ceph.com"
  469.             - "--enableprofiling=false"
  470.           env:
  471.             - name: POD_IP
  472.               valueFrom:
  473.                 fieldRef:
  474.                   fieldPath: status.podIP
  475.             - name: NODE_ID
  476.               valueFrom:
  477.                 fieldRef:
  478.                   fieldPath: spec.nodeName
  479.             - name: POD_NAMESPACE
  480.               valueFrom:
  481.                 fieldRef:
  482.                   fieldPath: metadata.namespace
  483.             - name: CSI_ENDPOINT
  484.               value: unix:///csi/csi.sock
  485.             - name: CSI_ADDONS_ENDPOINT
  486.               value: unix:///csi/csi-addons.sock
  487.           imagePullPolicy: "IfNotPresent"
  488.           volumeMounts:
  489.             - name: socket-dir
  490.               mountPath: /csi
  491.             - mountPath: /dev
  492.               name: host-dev
  493.             - mountPath: /sys
  494.               name: host-sys
  495.             - mountPath: /run/mount
  496.               name: host-mount
  497.             - mountPath: /etc/selinux
  498.               name: etc-selinux
  499.               readOnly: true
  500.             - mountPath: /lib/modules
  501.               name: lib-modules
  502.               readOnly: true
  503.             - name: ceph-csi-config
  504.               mountPath: /etc/ceph-csi-config/
  505.             - name: plugin-dir
  506.               mountPath: /var/lib/kubelet/plugins
  507.               mountPropagation: "Bidirectional"
  508.             - name: mountpoint-dir
  509.               mountPath: /var/lib/kubelet/pods
  510.               mountPropagation: "Bidirectional"
  511.             - name: keys-tmp-dir
  512.               mountPath: /tmp/csi/keys
  513.             - name: ceph-logdir
  514.               mountPath: /var/log/ceph
  515.             - name: oidc-token
  516.               mountPath: /run/secrets/tokens
  517.               readOnly: true
  518.         - name: liveness-prometheus
  519.           securityContext:
  520.             privileged: true
  521.             allowPrivilegeEscalation: true
  522.           image: quay.io/cephcsi/cephcsi:canary
  523.           args:
  524.             - "--type=liveness"
  525.             - "--endpoint=$(CSI_ENDPOINT)"
  526.             - "--metricsport=8680"
  527.             - "--metricspath=/metrics"
  528.             - "--polltime=60s"
  529.             - "--timeout=3s"
  530.           env:
  531.             - name: CSI_ENDPOINT
  532.               value: unix:///csi/csi.sock
  533.             - name: POD_IP
  534.               valueFrom:
  535.                 fieldRef:
  536.                   fieldPath: status.podIP
  537.           volumeMounts:
  538.             - name: socket-dir
  539.               mountPath: /csi
  540.           imagePullPolicy: "IfNotPresent"
  541.       volumes:
  542.         - name: socket-dir
  543.           hostPath:
  544.             path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
  545.             type: DirectoryOrCreate
  546.         - name: plugin-dir
  547.           hostPath:
  548.             path: /var/lib/kubelet/plugins
  549.             type: Directory
  550.         - name: mountpoint-dir
  551.           hostPath:
  552.             path: /var/lib/kubelet/pods
  553.             type: DirectoryOrCreate
  554.         - name: ceph-logdir
  555.           hostPath:
  556.             path: /var/log/ceph
  557.             type: DirectoryOrCreate
  558.         - name: registration-dir
  559.           hostPath:
  560.             path: /var/lib/kubelet/plugins_registry/
  561.             type: Directory
  562.         - name: host-dev
  563.           hostPath:
  564.             path: /dev
  565.         - name: host-sys
  566.           hostPath:
  567.             path: /sys
  568.         - name: etc-selinux
  569.           hostPath:
  570.             path: /etc/selinux
  571.         - name: host-mount
  572.           hostPath:
  573.             path: /run/mount
  574.         - name: lib-modules
  575.           hostPath:
  576.             path: /lib/modules
  577.         - name: ceph-csi-config
  578.           configMap:
  579.             name: ceph-csi-config
  580.         - name: keys-tmp-dir
  581.           emptyDir: {
  582.             medium: "Memory"
  583.           }
  584.         - name: oidc-token
  585.           projected:
  586.             sources:
  587.               - serviceAccountToken:
  588.                   path: oidc-token
  589.                   expirationSeconds: 3600
  590.                   audience: ceph-csi-kms
  591. ---
  592. apiVersion: v1
  593. kind: Service
  594. metadata:
  595.   name: csi-metrics-rbdplugin
  596.   namespace: ceph-csi
  597.   labels:
  598.     app: csi-metrics
  599. spec:
  600.   ports:
  601.     - name: http-metrics
  602.       port: 8080
  603.       protocol: TCP
  604.       targetPort: 8680
  605.   selector:
  606.     app: csi-rbdplugin
  607. EOF
  608. eof

7、secret.yaml

  1. # 7、secret.yaml
  2. cat > ~/ceph-csi/secret.yaml << 'EOF'
  3. ---
  4. apiVersion: v1
  5. kind: Secret
  6. metadata:
  7.   name: csi-rbd-secret
  8.   namespace: ceph-csi
  9. stringData:
  10.   userID: kube
  11.   userKey: AQD/4gpjDnt0KhAANfaKRnFlXIXVqPdf1IHOUA==
  12. EOF

userID就是ceph auth add client.kube mon ‘allow r’ osd ‘allow rwx pool=kubernetes’创建的kube

userKey使用ceph auth get-key client.kube命令在ceph集群上查看

8、storageclass.yaml

  1. # 8、storageclass.yaml
  2. cat > ~/ceph-csi/storageclass.yaml << 'EOF'
  3. ---
  4. apiVersion: storage.k8s.io/v1
  5. kind: StorageClass
  6. metadata:
  7.    name: csi-rbd-sc
  8. provisioner: rbd.csi.ceph.com
  9. parameters:
  10.    clusterID: 4c4d9c82-267d-11ed-888e-000c29db6f93
  11.    pool: kubernetes
  12.    imageFeatures: layering
  13.    csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  14.    csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi
  15.    csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  16.    csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi
  17.    csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  18.    csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi
  19.    csi.storage.k8s.io/fstype: ext4
  20. reclaimPolicy: Delete
  21. allowVolumeExpansion: true
  22. mountOptions:
  23.    - discard
  24. EOF

clusterID在ceph集群上用ceph mon dump命令查看

pool使用ceph osd pool create kubernetes 8 8命令在ceph集群上创建的kubernetes

下载镜像

  1. # 1、master上操作
  2. # 查看镜像
  3. find ./ -type f |xargs grep 'image: '|sort|uniq|awk '{print $3}'|grep ^[a-zA-Z]|grep -Evw 'error|kubeRbacProxy'|sort -rn|uniq |grep -n ".*"
  5. 1:quay.io/cephcsi/cephcsi:canary
  6. 2:dyrnq/csi-snapshotter:v6.0.1
  7. 3:dyrnq/csi-resizer:v1.5.0
  8. 4:dyrnq/csi-provisioner:v3.2.1
  9. 5:dyrnq/csi-node-driver-registrar:v2.5.1
  10. 6:dyrnq/csi-attacher:v3.5.0
  12. #在线预拉取镜像
  13. find ./ -type f |xargs grep 'image: '|sort|uniq|awk '{print $3}'|grep ^[a-zA-Z]|grep -Evw 'error|kubeRbacProxy'|sort -rn|uniq |xargs -i docker pull {}
  15. find ./ -type f |xargs grep 'image: '|sort|uniq|awk '{print $3}'|grep ^[a-zA-Z]|grep -Evw 'error|kubeRbacProxy'|sort -rn|uniq |xargs -i ctr -n k8s.io i pull {}
  18. #2、node上操作
  19. cat > pull.sh << 'EOF'
  20.           quay.io/cephcsi/cephcsi:canary
  21.           dyrnq/csi-snapshotter:v6.0.1
  22.           dyrnq/csi-resizer:v1.5.0
  23.           dyrnq/csi-provisioner:v3.2.1
  24.           dyrnq/csi-node-driver-registrar:v2.5.1
  25.           dyrnq/csi-attacher:v3.5.0
  26. EOF
  28. cat pull.sh |xargs -i docker pull {}
  29. cat pull.sh |xargs -i crictl  pull {}
  1. kubectl apply \
  2. -f csi-config-map.yaml \
  3. -f csi-nodeplugin-rbac.yaml \
  4. -f csi-provisioner-rbac.yaml \
  5. -f csidriver.yaml
  7. kubectl apply \
  8. -f csi-rbdplugin-provisioner.yaml \
  9. -f csi-rbdplugin.yaml
  11. kubectl apply \
  12. -f secret.yaml \
  13. -f storageclass.yaml
  1. kubectl delete \
  2. -f storageclass.yaml \
  3. -f secret.yaml
  5. kubectl delete \
  6. -f csi-rbdplugin.yaml \
  7. -f csi-rbdplugin-provisioner.yaml 
  9. kubectl delete \
  10. -f csidriver.yaml \
  11. -f csi-provisioner-rbac.yaml \
  12. -f csi-nodeplugin-rbac.yaml \
  13. -f csi-config-map.yaml

测试动态pvc

9、pvc.yaml

  1. cat > ~/000/pvc.yaml << 'EOF'
  2. ---
  3. apiVersion: v1
  4. kind: PersistentVolumeClaim
  5. metadata:
  6.   name: rbd-pvc
  7. spec:
  8.   accessModes:
  9.     - ReadWriteOnce
  10.   resources:
  11.     requests:
  12.       storage: 1Gi
  13.   storageClassName: csi-rbd-sc
  14. EOF

10、pod.yaml

  1. cat > ~/000/pod.yaml << 'EOF'
  2. ---
  3. apiVersion: v1
  4. kind: Pod
  5. metadata:
  6.   name: csi-rbd-demo-pod
  7. spec:
  8.   containers:
  9.     - name: web-server
  10.       image: nginx:alpine
  11.       imagePullPolicy: IfNotPresent
  12.       volumeMounts:
  13.         - name: mypvc
  14.           mountPath: /var/lib/www/html
  15.   volumes:
  16.     - name: mypvc
  17.       persistentVolumeClaim:
  18.         claimName: rbd-pvc
  19.         readOnly: false
  20. EOF