安装依赖lvm2

在可以联网的机器上下载lvm2安装包，下载好后打包上传到要安装lvm2的服务器中。

yum install -y lvm2 --downloadonly --downloaddir /home/lvm2
# 执行下面命令开始安装。
rpm -Uvh *.rpm --nodeps --force

Ceph 先决条件

需要在所有集群节点添加一块硬盘50G即可根据需要而定，不要分区和格式化。
为了配置 Ceph 存储集群，至少需要以下本地存储选项之一：

原始设备（无分区或格式化文件系统）
原始分区（无格式化文件系统）
block模式下存储类可用的 PV

您可以使用以下命令确认您的分区或设备是否使用文件系统进行格式化。

lsblk -f

NAME                  FSTYPE      LABEL UUID                                   MOUNTPOINT
vda
└─vda1                LVM2_member       >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB
 ├─ubuntu--vg-root   ext4              c2366f76-6e21-4f10-a8f3-6776212e2fe4   /
 └─ubuntu--vg-swap_1 swap              9492a3dc-ad75-47cd-9596-678e8cf17ff9   [SWAP]
vdb

如果该FSTYPE字段不为空，则在相应设备的顶部有一个文件系统。在这个例子中，你可以使用vdbfor Ceph 而不能使用vda它的分区。

安装Rook 集群

下载所需的镜像
在可以联网的机器上下载docker镜像

docker pull quay.io/cephcsi/cephcsi:v3.4.0
docker pull registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0
docker pull registry.aliyuncs.com/google_containers/csi-resizer:v1.3.0
docker pull registry.aliyuncs.com/google_containers/csi-provisioner:v3.0.0
docker pull registry.aliyuncs.com/google_containers/csi-snapshotter:v4.2.0
docker pull registry.aliyuncs.com/google_containers/csi-attacher:v3.3.0
docker pull ceph/ceph:v16
docker pull rook/ceph:v1.7.7

# 打tag上传到私有镜像仓库harbor中
docker tag quay.io/cephcsi/cephcsi:v3.4.0 192.168.28.150:8001/kubesphere-install/cephcsi/cephcsi:v3.4.0
docker tag registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0 192.168.28.150:8001/kubesphere-install/longhornio/csi-node-driver-registrar:v2.3.0
docker tag registry.aliyuncs.com/google_containers/csi-resizer:v1.2.0 192.168.28.150:8001/kubesphere-install/longhornio/csi-resizer:v1.2.0
docker tag registry.aliyuncs.com/google_containers/csi-provisioner:v2.1.2 192.168.28.150:8001/kubesphere-install/longhornio/csi-provisioner:v2.1.2
docker tag registry.aliyuncs.com/google_containers/csi-snapshotter:v3.0.3 192.168.28.150:8001/kubesphere-install/longhornio/csi-snapshotter:v3.0.3
docker tag registry.aliyuncs.com/google_containers/csi-attacher:v3.2.1 192.168.28.150:8001/kubesphere-install/longhornio/csi-attacher:v3.2.1
docker tag ceph/ceph:v16 192.168.28.150:8001/kubesphere-install/ceph/ceph:v16
docker tag rook/ceph:v1.7.7 192.168.28.150:8001/kubesphere-install/rook/ceph:v1.7.7

# 开始push到harbor中
docker push 192.168.28.150:8001/kubesphere-install/cephcsi/cephcsi:v3.4.0
docker push 192.168.28.150:8001/kubesphere-install/longhornio/csi-node-driver-registrar:v2.3.0
docker push 192.168.28.150:8001/kubesphere-install/longhornio/csi-resizer:v1.2.0
docker push 192.168.28.150:8001/kubesphere-install/longhornio/csi-provisioner:v2.1.2
docker push 192.168.28.150:8001/kubesphere-install/longhornio/csi-snapshotter:v3.0.3
docker push 192.168.28.150:8001/kubesphere-install/longhornio/csi-attacher:v3.2.1
docker push 192.168.28.150:8001/kubesphere-install/ceph/ceph:v16
docker push 192.168.28.150:8001/kubesphere-install/rook/ceph:v1.7.7

下载rook部署文件

# 可以在联网的机器中用git下载rook部署文件，下载好后打包上传到要安装的服务器中
$ git clone --single-branch --branch release-1.7 https://github.com/rook/rook.git

修改operator.yaml，大约在78行，添加如下配置

ROOK_CSI_CEPH_IMAGE: "192.168.28.150:8001/kubesphere-install/cephcsi/cephcsi:v3.4.0"
ROOK_CSI_REGISTRAR_IMAGE: "192.168.28.150:8001/kubesphere-install/longhornio/csi-node-driver-registrar:v2.3.0"
ROOK_CSI_RESIZER_IMAGE: "192.168.28.150:8001/kubesphere-install/longhornio/csi-resizer:v1.2.0"
ROOK_CSI_PROVISIONER_IMAGE: "192.168.28.150:8001/kubesphere-install/longhornio/csi-provisioner:v2.1.2"
ROOK_CSI_SNAPSHOTTER_IMAGE: "192.168.28.150:8001/kubesphere-install/longhornio/csi-snapshotter:v3.0.3"
ROOK_CSI_ATTACHER_IMAGE: "192.168.28.150:8001/kubesphere-install/longhornio/csi-attacher:v3.2.1"

大约372行添加如下配置

image: 192.168.28.150:8001/kubesphere-install/rook/ceph:v1.7.7

修改cluster.yaml配置文件，大约在24行，添加如下配置

image: 192.168.28.150:8001/kubesphere-install/ceph/ceph:v16

开始部署rook-ceph集群

cd rook/cluster/examples/kubernetes/ceph
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
kubectl create -f cluster.yaml

查看部署

kubectl -n rook-ceph get pod

部署Rook Toolbox

rook 工具箱可以作为 Kubernetes 集群中的部署运行，您可以在其中连接和运行任意 Ceph 命令。
将工具规范另存为toolbox.yaml：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-tools
  namespace: rook-ceph
  labels:
    app: rook-ceph-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rook-ceph-tools
  template:
    metadata:
      labels:
        app: rook-ceph-tools
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: rook-ceph-tools
        image: 192.168.28.150:8001/kubesphere-install/rook/ceph:v1.7.7
        command: ["/tini"]
        args: ["-g", "--", "/usr/local/bin/toolbox.sh"]
        imagePullPolicy: IfNotPresent
        env:
          - name: ROOK_CEPH_USERNAME
            valueFrom:
              secretKeyRef:
                name: rook-ceph-mon
                key: ceph-username
          - name: ROOK_CEPH_SECRET
            valueFrom:
              secretKeyRef:
                name: rook-ceph-mon
                key: ceph-secret
        volumeMounts:
          - mountPath: /etc/ceph
            name: ceph-config
          - name: mon-endpoint-volume
            mountPath: /etc/rook
      volumes:
        - name: mon-endpoint-volume
          configMap:
            name: rook-ceph-mon-endpoints
            items:
            - key: data
              path: mon-endpoints
        - name: ceph-config
          emptyDir: {}
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5

启动 rook-ceph-tools pod：

kubectl create -f toolbox.yaml

等待工具箱 pod 下载其容器并进入running状态：

kubectl -n rook-ceph rollout status deploy/rook-ceph-tools

rook-ceph-tools pod 运行后，您可以使用以下命令连接到它：

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

工具箱中的所有可用工具都可以满足您的故障排除需求。
示例：

ceph status
ceph osd status
ceph df
rados df

完成工具箱后，您可以删除部署：

kubectl -n rook-ceph delete deploy/rook-ceph-tools

Toolbox Job（工具箱作业）

如果您想将 Ceph 命令作为一次性操作运行并稍后从日志中收集结果，您可以将脚本作为 Kubernetes 作业运行。工具箱作业将运行嵌入在作业规范中的脚本。该脚本具有 bash 脚本的全部灵活性。

在本示例中，该ceph status命令在创建作业时执行。

apiVersion: batch/v1
kind: Job
metadata:
  name: rook-ceph-toolbox-job
  namespace: rook-ceph
  labels:
    app: ceph-toolbox-job
spec:
  template:
    spec:
      initContainers:
      - name: config-init
        image: 192.168.28.150:8001/kubesphere-install/rook/ceph:v1.7.7
        command: ["/usr/local/bin/toolbox.sh"]
        args: ["--skip-watch"]
        imagePullPolicy: IfNotPresent
        env:
        - name: ROOK_CEPH_USERNAME
          valueFrom:
            secretKeyRef:
              name: rook-ceph-mon
              key: ceph-username
        - name: ROOK_CEPH_SECRET
          valueFrom:
            secretKeyRef:
              name: rook-ceph-mon
              key: ceph-secret
        volumeMounts:
        - mountPath: /etc/ceph
          name: ceph-config
        - name: mon-endpoint-volume
          mountPath: /etc/rook
      containers:
      - name: script
        image: 192.168.28.150:8001/kubesphere-install/rook/ceph:v1.7.7
        volumeMounts:
        - mountPath: /etc/ceph
          name: ceph-config
          readOnly: true
        command:
        - "bash"
        - "-c"
        - |
          # Modify this script to run any ceph, rbd, radosgw-admin, or other commands that could
          # be run in the toolbox pod. The output of the commands can be seen by getting the pod log.
          #
          # example: print the ceph status
          ceph status
      volumes:
      - name: mon-endpoint-volume
        configMap:
          name: rook-ceph-mon-endpoints
          items:
          - key: data
            path: mon-endpoints
      - name: ceph-config
        emptyDir: {}
      restartPolicy: Never

创建工具箱作业：

kubectl create -f toolbox-job.yaml

作业完成后，查看脚本结果：

kubectl -n rook-ceph logs -l job-name=rook-ceph-toolbox-job

创建Shared Filesystem（共享文件系统）

可以使用来自多个 pod 的读/写权限挂载共享文件系统。这对于可以使用共享文件系统进行集群的应用程序可能很有用。

创建文件系统

通过为CephFilesystemCRD 中的元数据池、数据池和元数据服务器指定所需的设置来创建文件系统。

将此共享文件系统定义另存为filesystem.yaml：

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true

Rook 操作员将创建启动服务所需的所有池和其他资源。这可能需要一分钟才能完成。

# 创建文件系统
kubectl create -f filesystem.yaml
[...]

# 要确认文件系统已配置，请等待 mds pod 启动
kubectl -n rook-ceph get pod -l app=rook-ceph-mds

要查看文件系统的详细状态，请启动并连接到Rook 工具箱。将ceph status为该mds服务显示一个新行。在此示例中，有一个 MDS 的活动实例已启动，其中一个 MDS 实例处于standby-replay模式以备故障转移。

ceph status

供应存储

在 Rook 开始供应存储之前，需要根据文件系统创建一个 StorageClass。这是 Kubernetes 与 CSI 驱动程序互操作以创建持久卷所必需的。

将此存储类定义另存为storageclass.yaml：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  # clusterID is the namespace where the rook cluster is running
  # If you change this namespace, also change the namespace below where the secret namespaces are defined
  clusterID: rook-ceph

  # CephFS filesystem name into which the volume shall be created
  fsName: myfs

  # Ceph pool into which the volume shall be created
  # Required for provisionVolume: "true"
  pool: myfs-data0

  # The secrets contain Ceph admin credentials. These are generated automatically by the operator
  # in the same namespace as the cluster.
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph

reclaimPolicy: Delete

创建存储类

kubectl create -f cluster/examples/kubernetes/ceph/csi/cephfs/storageclass.yaml

设置默认StorageClass

在安装kubesphere之前需要有默认得StorageClass存储

kubectl patch storageclass rook-cephfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

验证

发现rook-cephfs (default) 已经是默认存储了

kubectl get sc

NAME                    PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-cephfs (default)   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   3m6s

Kubernetes文档

离线安装ROOK-Ceph分布式存储并作为默认存储