1、 对象存储概述

对象存储通常是指⽤于上传put和下载get之⽤的存储系统，通常⽤于存储静态⽂件，如图⽚，视频，⾳ 频等，对象存储⼀旦上传之后⽆法再线进⾏修改，如需修改则需要将其下载到本地，对象存储最早由 aws的S3提供，Ceph的对象存储提供两种接⼝⽀持：
S3⻛格接口
Swift⻛格接口
image.png
Ceph对象存储具有以下特点
RESTful Interface
S3- and Swift-compliant APIs
S3-style subdomains
Unified S3/Swift namespace
User management
Usage tracking
Striped objects
Cloud solution integration
Multi-site deployment
Multi-site replication

2、 部署RGW集群

通过rook部署RGW集群
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: my-store
namespace: rook-ceph # namespace:cluster
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: none
dataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: none
preservePoolsOnDelete: false
gateway:
port: 80
instances: 1
placement:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-rgw
topologyKey: kubernetes.io/hostname
# A key/value list of annotations
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - rgw-node
# topologySpreadConstraints:
# tolerations:
# - key: rgw-node
# operator: Exists
# podAffinity:
# podAntiAffinity:
# A key/value list of annotations
annotations:
labels:
resources:
healthCheck:
bucket:
disabled: false
interval: 60s
livenessProbe:
disabled: false
执⾏apply object.yaml安装⽂件之后会⾃动创建rgw对象存储的pods
rook-ceph-rgw-my-store-a-79f9c84bbb-vb66v 1/1 Running 0 69s
此时rgw已经部署完毕，默认部署了1个rgw实例，通过ceph -s可以查看到rgw已经部署到Ceph集群中
[root@rook-ceph-tools-54fc95f4f4-mg67d /]# ceph -s
cluster:
id: 2d792034-41f1-4ce2-bdc0-3951bc09cab0
health: HEALTH_WARN
clock skew detected on mon.e

services:
mon: 4 daemons, quorum a,b,c,e (age 17h)
mgr: a(active, since 7d)
mds: 2/2 daemons up, 2 hot standby
osd: 5 osds: 5 up (since 4d), 5 in (since 4d)
rgw: 1 daemon active (1 hosts, 1 zones)

data:
volumes: 1/1 healthy
pools: 11 pools, 177 pgs
objects: 739 objects, 1.3 GiB
usage: 5.8 GiB used, 1.7 TiB / 1.7 TiB avail
pgs: 177 active+clean

io:
client: 1.7 KiB/s rd, 3 op/s rd, 0 op/s wr
RGW默认会创建若⼲个pool来存储对象存储的数据，pool包含metadata pool和data pool
[root@rook-ceph-tools-54fc95f4f4-mg67d /]# ceph osd lspools
1 device_health_metrics
2 replicapool
3 myfs-metadata
4 myfs-data0
5 my-store.rgw.buckets.index
6 my-store.rgw.buckets.non-ec
7 my-store.rgw.meta
8 .rgw.root
9 my-store.rgw.control
10 my-store.rgw.log
11 my-store.rgw.buckets.data

3、RGW高可用集群

RGW是⼀个⽆状态化的http服务器，通过8080端⼝处理http的put/get请求，⽣产环境中需要部署多个 RGW以满⾜⾼可⽤的要求，在非容器化的场景下可通过haproxy+keepalived的⽅式来构建 RGW的⾼可⽤集群
Ceph集群中部署多个RGW实例
HAproxy提供负载均衡能⼒
keepalived提供VIP和保障haproxy的⾼可⽤性
通过rook构建RGW集群默认部署了⼀个RGW实例，⽆法实现⾼可⽤性的要求，需要部署多个instances 以满⾜⾼可⽤的诉求，如何部署呢 ？？？
[root@master1 ceph]# cat object.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: my-store
namespace: rook-ceph # namespace:cluster
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: none
dataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: none
preservePoolsOnDelete: false
gateway:
port: 80
instances: 2 #会部署两个rgw实例
placement:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-rgw
topologyKey: kubernetes.io/hostname
# A key/value list of annotations
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - rgw-node
# topologySpreadConstraints:
# tolerations:
# - key: rgw-node
# operator: Exists
# podAffinity:
# podAntiAffinity:
# A key/value list of annotations
annotations:
labels:
resources:
healthCheck:
bucket:
disabled: false
interval: 60s
livenessProbe:
disabled: false
执⾏完毕之后会创建两个rgw的pods
[root@master1 ceph]# kubectl get pods -n rook-ceph -o wide | grep rgw
rook-ceph-rgw-my-store-a-79f9c84bbb-gd6hg 1/1 Running 0 118s 10.244.3.139 node2
rook-ceph-rgw-my-store-a-79f9c84bbb-vb66v 1/1 Running 0 7m54s 10.244.4.28 node3
对外提供服务通过service的VIP实现，VIP通过80端⼝映射到后端的两个pods上
[root@master1 ceph]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-cephfsplugin-metrics ClusterIP 10.101.225.186 8080/TCP,8081/TCP 7d18h
csi-rbdplugin-metrics ClusterIP 10.105.109.150 8080/TCP,8081/TCP 7d18h
rook-ceph-mgr ClusterIP 10.105.216.252 9283/TCP 7d17h
rook-ceph-mgr-dashboard ClusterIP 10.109.241.108 8443/TCP 7d17h
rook-ceph-mon-a ClusterIP 10.107.198.193 6789/TCP,3300/TCP 7d18h
rook-ceph-mon-b ClusterIP 10.96.73.13 6789/TCP,3300/TCP 7d18h
rook-ceph-mon-c ClusterIP 10.104.20.43 6789/TCP,3300/TCP 7d18h
rook-ceph-mon-e ClusterIP 10.110.152.125 6789/TCP,3300/TCP 4d18h
rook-ceph-rgw-my-store ClusterIP 10.109.213.232 80/TCP 9m44s
[root@master1 ceph]# kubectl describe svc rook-ceph-rgw-my-store -n rook-ceph
Name: rook-ceph-rgw-my-store
Namespace: rook-ceph
Labels: app=rook-ceph-rgw
ceph_daemon_id=my-store
ceph_daemon_type=rgw
rgw=my-store
rook_cluster=rook-ceph
rook_object_store=my-store
Annotations:
Selector: app=rook-ceph-rgw,ceph_daemon_id=my-store,rgw=my-store,rook_cluster=rook-ceph,rook_object_store=my-store
Type: ClusterIP
IP Families:
IP: 10.109.213.232
IPs:
Port: http 80/TCP
TargetPort: 8080/TCP
Endpoints: 10.244.3.139:8080,10.244.4.28:8080
Session Affinity: None
Events:

4、 RGW⾼级调度

和前⾯mon，mds所⼀样，rgw⽀持⾼级的调度机制，通过nodeAffinity，podAntiAffinity， podAffinity，tolerations⾼级调度算法将RGW调度到特定的节点上的诉求。
[root@master1 ceph]# cat object.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: my-store
namespace: rook-ceph # namespace:cluster
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: none
dataPool:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: none
preservePoolsOnDelete: false
gateway:
port: 80
instances: 2
placement:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-rgw
topologyKey: kubernetes.io/hostname
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-rgw
operator: In
values:
- enabled
annotations:
labels:
resources:
healthCheck:
bucket:
disabled: false
interval: 60s
livenessProbe:
disabled: false
创建上面的资源清单文件，可以看到pod调度失败了，我们给node加上标签
image.png
[root@master1 ceph]# kubectl label node node1 ceph-rwg=enabled
node/node1 labeled
[root@master1 ceph]# kubectl label node node2 ceph-rwg=enabled
node/node2 labeled
[root@master1 ceph]# kubectl label node node3 ceph-rwg=enabled
node/node3 labeled

5、连接外部集群

Rook提供了管理外部RGW对象存储的能⼒，通过CephObjectStore⾃定义资源对象提供了管理外部 RGW对象存储的能⼒，其会通过创建⼀个service实现和外部对象存储管理，kubernetes集群内部访问 service的vip地址即可实现访问
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: external-store
namespace: rook-ceph
spec:
gateway:
port: 8080
externalRgwEndpoints:
- ip: 192.168.39.182
healthCheck:
bucket:
enabled: true
interval: 60s

6、 创建Bucket存储桶

Ceph RGW集群创建好之后就可以往RGW存储对象存储数据了，存储数据之前需要有创建bucket， bucket是存储桶，是对象存储中逻辑的存储空间，可以通过Ceph原⽣的接⼝管理存储桶⼊radowgwadmin，云原⽣环境下，推荐使⽤云原⽣的⽅式管理bucket，何为云原⽣模式？即资源对象抽象化为 kubernetes⻛格的⽅式来管理，尽可能减少使⽤原⽣命令，如创建资源池，创建bucket等。

6.1 ⾸先需要创建存储消费桶的StorageClass

[root@master1 ceph]# cat storageclass-bucket-delete.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-delete-bucket
provisioner: rook-ceph.ceph.rook.io/bucket # driver:namespace:cluster
reclaimPolicy: Delete
parameters:
objectStoreName: my-store
objectStoreNamespace: rook-ceph # namespace:cluster
region: us-east-1

[root@master1 ceph]# kubectl apply -f storageclass-bucket-delete.yaml
storageclass.storage.k8s.io/rook-ceph-delete-bucket created

6.2 通过ObjectBucketClaim向StorageClass申请存储桶，创建了⼀个名为ceph-bkt的bucket

[root@master1 ceph]# cat object-bucket-claim-delete.yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: ceph-delete-bucket
spec:
generateBucketName: ceph-bkt
storageClassName: rook-ceph-delete-bucket
additionalConfig:

[root@master1 ceph]# kubectl apply -f object-bucket-claim-delete.yaml
objectbucketclaim.objectbucket.io/ceph-delete-bucket created

7、 容器访问对象存储

创建ObjectBucketClaim之后，operator会⾃动创建bucket相关的访问信息，包括访问的路径，认证的 访问所需的key，这些信息分别存放在configmap和secrets对象中，可以通过如下的⽅式获取

7.1获取访问地址

获取ACCESS KEY和SECRET_ACCESS_KEY，Secrets通过base64加密，需要获取解密后的字符串
export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o jsonpath=‘{.data.BUCKET_HOST}’) export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o jsonpath=‘{.data.AWS_ACCESS_KEY_ID}’ | base64 —decode) export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o jsonpath=‘{.data.AWS_SECRET_ACCESS_KEY}’ | base64 —decode)

7.2 安装s3cmd客户端

[root@node-1 ceph]# kubectl exec -it rook-ceph-tools-77bf5b9b7d-7klln -n rookceph /bin/bash
yum install s3cmd -y

7.3 配置s3cmd

s3cmd —configure

7.4 上传object对象

[root@rook-ceph-tools-77bf5b9b7d-7klln /]# s3cmd put /etc/passwd* s3://cephbkt-1aad80c9-1bbd-45e4-bc25-a63684dbbf24
upload: ‘/etc/passwd’ -> ‘s3://ceph-bkt-1aad80c9-1bbd-45e4-bc25- a63684dbbf24/passwd’ [1 of 2] 1168 of 1168 100% in 0s 114.02 KB/s done upload: ‘/etc/passwd-‘ -> ‘s3://ceph-bkt-1aad80c9-1bbd-45e4-bc25- a63684dbbf24/passwd-‘ [2 of 2] 1112 of 1112 100% in 0s 114.13 KB/s done
s3cmd其他操作包括： put 上传 get 下载 ls 查看 rm 删除 info 查看信息