Ceph操作维护手册

一、更换磁盘

1、 确认需要更换的磁盘对应的盘符、OSD ID；

  1. root@Ceph-B**A**-04-S2-60:~# df -h |grep 17
  2. /dev/sdp3       1.1T  566G  551G  51% /var/lib/ceph/osd/ceph-17


*sdp为服务器上对应的盘符；
*ceph-17 17为OSD对应的ID；

2、 确认磁盘在服务器上对应的槽位；

  1. root@Ceph-B**A**-04-S2-60:~# ll /dev/disk/by-path/ |grep sdp
  2. lrwxrwxrwx 1 root root    9 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0 -> ../../sdp
  3. lrwxrwxrwx 1 root root   10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part1 -> ../../sdp1
  4. lrwxrwxrwx 1 root root   10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part2 -> ../../sdp2
  5. lrwxrwxrwx 1 root root   10 Aug 17 19:17 pci-0000:02:00.0-scsi-0:0:18:0-part3 -> ../../sdp3


*根据盘符找到服务器上对应的磁盘槽位，scsi-0:0:18:0标识第18号槽位；

3、 确认主机的SN；

  1. root@Ceph-B**A**-04-S2-60:~# dmidecode -s system-serial-number
  2. HVNHB**


4、 在Crushmap设置故障OSD OUT标志；
ceph osd out 17

*设置OUT标识，告诉集群不要再往这个OSD磁盘上读写数据；

5、 查看集群状态；
ceph –s

*确认OSD设置OUT标志后，集群有没有出现异常；
6、 关闭OSD对应的实例进程；
stop cep-osd id=17

7、 从Crushmap中移除OSD；
ceph osd crush remove osd.17

8、 删除集群中OSD认证信息；
ceph auth del osd.17

*清除旧的认证key信息，新的磁盘加入集群后会重新生产认证密钥；

9、 从集群中移除OSD；
ceph osd rm 17

10、 卸载OSD磁盘挂载点；
umount /var/lib/ceph/osd/ceph-17

11、 IDC维保服务商更换新的磁盘；

*根据确认的服务器SN和对应的磁盘槽位，将旧的磁盘取出，更换新的磁盘；
*登入IDRAC将磁盘调整为非RAID模式；

12、 初始化新的磁盘；
ceph-deploy disk zap Ceph-BXXAXX-04-S2-60.domain.tld:sdp

*更换新的磁盘后，有可能磁盘的盘符会变掉，初始化新的磁盘前，最好确认清楚（建议lsblk确认）；

13、 OSD进程初始化；
ceph-deploy osd create Ceph-BXXAXX-04-S2-60.domain.tld:sdp

*如果更换的是旧盘，这一步操作有可能失败报错，此时最好用dd（或者分区工具）摧毁磁盘上残留的Raid或分区表信息，然后重新初始化尝试；

14、 调整Crushmap；
ceph osd crush add osd.17 1.17 root=sata pod=sata-pod-01 chassis=sata-chassis-01 host=Ceph-BA-04-S2-60-sata

*新的OSD权重和位置信息，可以使用ceph osd tree确认服务器上其他OSD的位置和权重做参考；

15、 优化挂载参数；
mount -o remount,nobarrier /var/lib/ceph/osd/ceph-17

*根据需要做优化调整，主要是优化filestore做后端存储引擎时，文件系统层面的提升；
*ceph –s关注集群在数据回填（backfill）中，集群读写IO；

16、 完成。

二、关机、重启

1、 查看确认Ceph集群状态；
ceph –s

*主要是看Ceph集群当前的状态，有没有pg在做scrub，有没有OSD down、out，数据有没有在做大量的backfill，remap等；

2、 设置OSD NOOUT标志；
ceph osd set nooout

3、 关闭本机上所有的OSD进程服务；
stop ceph-osd-all

*等待所有的OSD进程关闭，使用ps aux |grep ceph确认；

4、 关机或重启；
sync；sync；shutdown –h now
or
sync;sync;reboot

*如果更换、扩容内存，关机后可以操作；
*如果迁移服务器，关机之后可以迁移（在服务器开启前，网络要保证和迁移前相同）；

5、 开机或重启后，启动OSD进程服务；
start ceph-osd-all-starter

6、 取消OSD NOOUT标志；
ceph osd unset noout

7、 查看集群状态；
ceph –s

*确认集群恢复到正常状态；
8、 完成。

三、PG异常状态修复

常见状态说明

状态（state） 说明(description)
Active Ceph可正常处理此pg请求
Clean PG内所有的对象都被正确的复制了对应的份数
Down 一个包含必备数据的副本离线，所以PG也离线了
Degraded PG中的一些对象还没有被复制到规定的份数
Inconsistent Ceph检测到PG中对象的一份或多份数据不一致
Peering PG正在互联过程中
Recovering Ceph正在迁移/同步对象和其副本
Incomplete Ceph 探测到某一PG可能丢失了写入信息，或者没有健康的副本
Stale PG状态未知，从PG mapping更新后Monitor一直没有收到更新


1、 查找Inconsistent pg并修复

  1. # ceph pg dump | grep insconsistent
  2.       or
  3. # ceph health detail
  5. # ceph pg repair {pgnum}


*如果修复不好，可以尝试ceph pg scrub {pgnum}或ceph pg deep-scrub {pgnum}

2、 查找stale pg 并修复

  1. # ceph pg dump |grep stale
  2. # ceph pg map {pgname}
  3. # ceph pg force_create_pg {pgnum}

四、对象存储帐户管理

1、 创建帐户；

  1. root@node1:~# radosgw-admin user create --display-name="Johnny wick" --uid=Johnny
  2. {
  3.     "user_id": "Johnny",
  4.     "display_name": "Johnny wick",
  5.     "email": "",
  6.     "suspended": 0,
  7.     "max_buckets": 1000,
  8.     "auid": 0,
  9.     "subusers": [],
  10.     "keys": [
  11.         {
  12.             "user": "Johnny",
  13.             "access_key": "S54GYGAB21NYEF765DSV",
  14.             "secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"
  15.         }
  16.     ],
  17.     "swift_keys": [],
  18.     "caps": [],
  19.     "op_mask": "read, write, delete",
  20.     "default_placement": "",
  21.     "placement_tags": [],
  22.     "bucket_quota": {
  23.         "enabled": true,
  24.         "max_size_kb": -1,
  25.         "max_objects": 100
  26.     },
  27.     "user_quota": {
  28.         "enabled": false,
  29.         "max_size_kb": -1,
  30.         "max_objects": -1
  31.     },
  32.     "temp_url_keys": []
  33. }

2、 停用帐户；

  1. root@node1:~# radosgw-admin user suspend --uid=Johnny
  2. {
  3.     "user_id": "Johnny",
  4.     "display_name": "Johnny wick",
  5.     "email": "",
  6.     "suspended": 1,
  7.     "max_buckets": 1000,
  8.     "auid": 0,
  9.     "subusers": [],
  10.     "keys": [
  11.         {
  12.             "user": "Johnny",
  13.             "access_key": "S54GYGAB21NYEF765DSV",
  14.             "secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"
  15.         }
  16.     ],
  17.     "swift_keys": [],
  18.     "caps": [],
  19.     "op_mask": "read, write, delete",
  20.     "default_placement": "",
  21.     "placement_tags": [],
  22.     "bucket_quota": {
  23.         "enabled": true,
  24.         "max_size_kb": -1,
  25.         "max_objects": 100
  26.     },
  27.     "user_quota": {
  28.         "enabled": false,
  29.         "max_size_kb": -1,
  30.         "max_objects": -1
  31.     },
  32.     "temp_url_keys": []
  33. }

3、 启用帐户；

  1. root@node1:~# radosgw-admin user enable --uid=Johnny
  2. {
  3.     "user_id": "Johnny",
  4.     "display_name": "Johnny wick",
  5.     "email": "",
  6.     "suspended": 0,
  7.     "max_buckets": 1000,
  8.     "auid": 0,
  9.     "subusers": [],
  10.     "keys": [
  11.         {
  12.             "user": "Johnny",
  13.             "access_key": "S54GYGAB21NYEF765DSV",
  14.             "secret_key": "i6AswOcSF5xaeLm4sahRDpYpClpVTYe6FURWuDrU"
  15.         }
  16.     ],
  17.     "swift_keys": [],
  18.     "caps": [],
  19.     "op_mask": "read, write, delete",
  20.     "default_placement": "",
  21.     "placement_tags": [],
  22.     "bucket_quota": {
  23.         "enabled": true,
  24.         "max_size_kb": -1,
  25.         "max_objects": 100
  26.     },
  27.     "user_quota": {
  28.         "enabled": false,
  29.         "max_size_kb": -1,
  30.         "max_objects": -1
  31.     },
  32.    "temp_url_keys": []
  33. }

4、 删除帐户；

  1. root@node1:~# radosgw-admin user rm --uid=Johnny
  2. root@node1:~# radosgw-admin user info --uid=Johnny
  3. could not fetch user info: no user info saved


5、 查看帐户；

  1. root@node1:~# radosgw-admin user info --uid=Johnny
  2. {
  3.     "user_id": "Johnny",
  4.     "display_name": "Johnny wick",
  5.     "email": "",
  6.     "suspended": 0,
  7.     "max_buckets": 1000,
  8.     "auid": 0,
  9.     "subusers": [],
  10.     "keys": [
  11.         {
  12.             "user": "Johnny",
  13.             "access_key": "B6WXEJ94UG9V68VFD3UK",
  14.             "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
  15.         }
  16.     ],
  17.     "swift_keys": [],
  18.     "caps": [],
  19.     "op_mask": "read, write, delete",
  20.     "default_placement": "",
  21.     "placement_tags": [],
  22.     "bucket_quota": {
  23.         "enabled": true,
  24.         "max_size_kb": -1,
  25.         "max_objects": 100
  26.     },
  27.     "user_quota": {
  28.         "enabled": false,
  29.         "max_size_kb": -1,
  30.         "max_objects": -1
  31.     },
  32.     "temp_url_keys": []
  33. }

6、 设置用户配额；

  1. root@node1:~# radosgw-admin quota set --quota-scope=user --uid=Johnny --max-objects=2048 --max-size=204800
  2. root@node1:~# radosgw-admin user info --uid=Johnny
  3. {
  4.     "user_id": "Johnny",
  5.     "display_name": "Johnny wick",
  6.     "email": "",
  7.     "suspended": 0,
  8.     "max_buckets": 1000,
  9.     "auid": 0,
  10.     "subusers": [],
  11.     "keys": [
  12.         {
  13.             "user": "Johnny",
  14.             "access_key": "B6WXEJ94UG9V68VFD3UK",
  15.             "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
  16.         }
  17.     ],
  18.     "swift_keys": [],
  19.     "caps": [],
  20.     "op_mask": "read, write, delete",
  21.     "default_placement": "",
  22.     "placement_tags": [],
  23.     "bucket_quota": {
  24.         "enabled": true,
  25.         "max_size_kb": -1,
  26.         "max_objects": 100
  27.     },
  28.     "user_quota": {
  29.        "enabled": false,
  30.        "max_size_kb": 200,
  31.        "max_objects": 2048
  32.    },
  33.     "temp_url_keys": []
  34. }
  36. root@node1:~# radosgw-admin quota enable --quota-scope=user --uid=Johnny
  37. root@node1:~# radosgw-admin user info --uid=Johnny
  38. {
  39.     "user_id": "Johnny",
  40.     "display_name": "Johnny wick",
  41.     "email": "",
  42.     "suspended": 0,
  43.     "max_buckets": 1000,
  44.     "auid": 0,
  45.     "subusers": [],
  46.     "keys": [
  47.         {
  48.             "user": "Johnny",
  49.             "access_key": "B6WXEJ94UG9V68VFD3UK",
  50.             "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
  51.         }
  52.     ],
  53.     "swift_keys": [],
  54.     "caps": [],
  55.     "op_mask": "read, write, delete",
  56.     "default_placement": "",
  57.     "placement_tags": [],
  58.     "bucket_quota": {
  59.         "enabled": true,
  60.         "max_size_kb": -1,
  61.         "max_objects": 100
  62.     },
  63.     "user_quota": {
  64.         "enabled": true,
  65.         "max_size_kb": 200,
  66.         "max_objects": 2048
  67.     },
  68.     "temp_url_keys": []
  69. }

7、 设置bucket配额；

  1. root@node1:~# radosgw-admin quota set quota-scope=bucket --uid=Johnny --bucket=bucket-test01 --max-objects=512 --max-size=51200
  2. root@node1:~# radosgw-admin quota set quota-scope=bucket --uid=Johnny --bucket=bucket-test02 --max-objects=1024 --max-size=102400
  3. root@node1:~# radosgw-admin quota enable --quota-scope=bucket --uid=Johnny --bucket=bucket-test01
  4. root@node1:~# radosgw-admin quota enable --quota-scope=bucket --uid=Johnny --bucket=bucket-test02
  5. root@node1:~# radosgw-admin bucket stats --bucket=bucket-test02
  6. {
  7.     "bucket": "bucket-test02",
  8.     "pool": ".rgw.buckets",
  9.     "index_pool": ".rgw.buckets.index",
  10.     "id": "default.124124.2",
  11.     "marker": "default.124124.2",
  12.     "owner": "Johnny",
  13.     "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",
  14.     "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",
  15.     "mtime": "2021-04-14 10:33:46.000000",
  16.     "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",
  17.     "usage": {},
  18.     "bucket_quota": {
  19.        "enabled": true,
  20.        "max_size_kb": 100,
  21.        "max_objects": 1024
  22.    }
  23. }
  24. root@node1:~# radosgw-admin bucket stats --bucket=bucket-test01
  25. {
  26.     "bucket": "bucket-test01",
  27.     "pool": ".rgw.buckets",
  28.     "index_pool": ".rgw.buckets.index",
  29.     "id": "default.124118.1",
  30.     "marker": "default.124118.1",
  31.     "owner": "Johnny",
  32.     "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",
  33.     "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",
  34.     "mtime": "2021-04-14 10:33:45.000000",
  35.     "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",
  36.     "usage": {},
  37.     "bucket_quota": {
  38.        "enabled": true,
  39.        "max_size_kb": 50,
  40.        "max_objects": 512
  41.    }
  42. }

8、 禁用配额

  1. root@node1:~# radosgw-admin quota disable --quota-scope=user --uid=Johnny
  2. root@node1:~# radosgw-admin user info --uid=Johnny
  3. {
  4.     "user_id": "Johnny",
  5.     "display_name": "Johnny wick",
  6.     "email": "",
  7.     "suspended": 0,
  8.     "max_buckets": 1000,
  9.     "auid": 0,
  10.     "subusers": [],
  11.     "keys": [
  12.         {
  13.             "user": "Johnny",
  14.             "access_key": "B6WXEJ94UG9V68VFD3UK",
  15.             "secret_key": "zaSwdDGSIWtWKYRp5dXQJymGXuzQrYa9pnaK4QxM"
  16.         }
  17.     ],
  18.     "swift_keys": [],
  19.     "caps": [],
  20.     "op_mask": "read, write, delete",
  21.     "default_placement": "",
  22.     "placement_tags": [],
  23.     "bucket_quota": {
  24.         "enabled": true,
  25.         "max_size_kb": -1,
  26.         "max_objects": 100
  27.     },
  28.     "user_quota": {
  29.         "enabled": false,
  30.         "max_size_kb": 200,
  31.         "max_objects": 2048
  32.     },
  33.     "temp_url_keys": []
  34. }
  37. root@node1:~# radosgw-admin quota disable --quota-scope=bucket --uid=Johnny --bucket=bucket-test02
  38. root@node1:~# radosgw-admin bucket stats --bucket=bucket-test02
  39. {
  40.     "bucket": "bucket-test02",
  41.     "pool": ".rgw.buckets",
  42.     "index_pool": ".rgw.buckets.index",
  43.     "id": "default.124124.2",
  44.     "marker": "default.124124.2",
  45.     "owner": "Johnny",
  46.     "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1",
  47.     "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0",
  48.     "mtime": "2021-04-14 10:42:23.000000",
  49.     "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#",
  50.     "usage": {},
  51.     "bucket_quota": {
  52.         "enabled": false,
  53.         "max_size_kb": 100,
  54.         "max_objects": 1024
  55.     }
  56. }

五、对象存储bucket管理

1、 创建bucket；

  1. root@node1:~# s3cmd ls
  2. root@node1:~# s3cmd mb s3://bkt-a01
  3. Bucket 's3://bkt-a01/' created
  4. root@node1:~# s3cmd ls
  5. 2021-04-14 03:01 s3://bkt-a01

2、 删除bucket；

  1. root@node1:~# s3cmd ls
  2. 2021-04-14 03:01  s3://bkt-a01
  3. root@node1:~# s3cmd rb s3://bkt-a01
  4. Bucket 's3://bkt-a01/' removed
  5. root@node1:~# s3cmd ls

3、 上传文件到bucket；

  1. root@node1:~# s3cmd put ceph_0.94.10-1trusty_amd64.deb s3://bkt-a02/
  2. upload: 'ceph_0.94.10-1trusty_amd64.deb' -> 's3://bkt-a02/ceph_0.94.10-1trusty_amd64.deb' [1 of 1]
  3.  10931302 of 10931302   100% in   0s    30.35 MB/s  done
  4. root@node1:~# s3cmd ls s3://bkt-a02/
  5. 2021-04-14 03:14 10.4249019623M d43b24ab71fb36c6626b2d243e5a03bd s3://bkt-a02/ceph_0.94.10-1trusty_amd64.deb

4、 下载bucket里面文件；

  1. root@node1:/tmp# s3cmd get s3://bkt-a02/ceph_0.94.10-1trusty_amd64.deb /tmp/
  2. download: 's3://bkt-a02/ceph_0.94.10-1trusty_amd64.deb' -> '/tmp/ceph_0.94.10-1trusty_amd64.deb'  [1 of 1]
  3.  10931302 of 10931302   100% in  14s   712.42 kB/s  done
  4. root@node1:/tmp# ls /tmp/
  5. ceph_0.94.10-1trusty_amd64.deb


5、 删除bucket里面文件；

  1. root@node1:/tmp# s3cmd rm s3://bkt-a02/ceph_0.94.10-1trusty_amd64.deb
  2. delete: 's3://bkt-a02/ceph_0.94.10-1trusty_amd64.deb'
  3. root@node1:/tmp# s3cmd ls s3://bkt-a02/


6、 查看bucket里面文件；

  1. root@node1:~/my-cluster# s3cmd info s3://bkt-a02/ceph-deploy_1.5.35_all.deb
  2. s3://bkt-a02/ceph-deploy_1.5.35_all.deb (object):
  3.    File size: 96446
  4.    Last mod: Wed, 14 Apr 2021 03:23:22 GMT
  5.    MIME type: application/x-debian-package
  6.    Storage:  STANDARD
  7.    MD5 sum:  05c4b861386a5b18ad008731ba7bf3d7
  8.    SSE:      none
  9.    Policy:   <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bkt-a02</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>ceph-deploy_1.5.35_all.deb</Key><LastModified>2021-04-14T03:23:22.000Z</LastModified><ETag>&quot;05c4b861386a5b18ad008731ba7bf3d7&quot;</ETag><Size>96446</Size><StorageClass>STANDARD</StorageClass><Owner><ID>Johnny</ID><DisplayName>Johnny wick</DisplayName></Owner></Contents></ListBucketResult>
  10.    CORS:     none
  11.    ACL:      Johnny wick: FULL_CONTROL
  12.    x-amz-meta-s3cmd-attrs: atime:1560760630/ctime:1560498948/gid:0/gname:root/md5:05c4b861386a5b18ad008731ba7bf3d7/mode:33188/mtime:1560498933/uid:0/uname:root

7、拷贝bucket里面文件；

  1. root@node1:~/my-cluster# s3cmd ls s3://bkt-a02
  2. 2021-04-14 03:23 94.185546875k 05c4b861386a5b18ad008731ba7bf3d7 s3://bkt-a02/ceph-deploy_1.5.35_all.deb
  3. root@node1:~/my-cluster# s3cmd ls s3://bkt-a03
  4. root@node1:~/my-cluster# s3cmd cp s3://bkt-a02/ceph-deploy_1.5.35_all.deb s3://bkt-a03
  5. remote copy: 's3://bkt-a02/ceph-deploy_1.5.35_all.deb' -> 's3://bkt-a03/ceph-deploy_1.5.35_all.deb'
  6. root@node1:~/my-cluster# s3cmd ls s3://bkt-a03
  7. 2021-04-14 05:28 94.185546875k 05c4b861386a5b18ad008731ba7bf3d7 s3://bkt-a03/ceph-deploy_1.5.35_all.deb


8、移动bucket里面文件；

  1. root@node1:~/my-cluster# s3cmd mb s3://bkt-a04
  2. Bucket 's3://bkt-a04/' created
  3. root@node1:~/my-cluster# s3cmd mv s3://bkt-a02/ceph-deploy_1.5.35_all.deb s3://bkt-a04/
  4. move: 's3://bkt-a02/ceph-deploy_1.5.35_all.deb' -> 's3://bkt-a04/ceph-deploy_1.5.35_all.deb'
  5. root@node1:~/my-cluster# s3cmd ls s3://bkt-a02/
  6. root@node1:~/my-cluster# s3cmd ls s3://bkt-a04
  7. 2021-04-14 05:34 94.185546875k 05c4b861386a5b18ad008731ba7bf3d7 s3://bkt-a04/ceph-deploy_1.5.35_all.deb
  8. root@node1:~/my-cluster#

9、 查看bucket使用容量；

  1. root@node1:~/my-cluster# s3cmd du s3://bkt-a04
  2. 94.185546875k 1 objects s3://bkt-a04/

10、 设置bucket文件权限

  1. root@node1:~/my-cluster# s3cmd modify s3://bkt-a04/ --acl-public --recursive
  2. modify: 's3://bkt-a04/ceph-deploy_1.5.35_all.deb'

六、结束