1- 角色扩容

1.1- mon扩容

一个Ceph存储集群至少需要一个Ceph Monitor和Ceph Manager才能运行。为了获得高可用性，Ceph存储群集通常运行多个Ceph Monitor，这样单个Ceph Monitor的故障不会使Ceph存储群集停机。Ceph使用Paxos算法，该算法需要大多数Monitor（即，大于N / 2，其中N是Monitor的数量）才能形成仲裁。Monitor的奇数往往会更好，尽管这不是必需的。

  • 扩容mon

    1. ceph-deploy mon add mon3
    2. ceph-deploy mon add mon4
    3. ceph-deploy mon add mon5

  • 验证查看

    1. ceph mon stat

  • 查看仲裁选举情况 ceph quorum_status —format json-pretty

添加了新的Ceph Monitor，Ceph将开始同步Monitor并形成仲裁。可以通过执行以下命令检查仲裁状态:

  1. [root@mon1 ceph]# ceph -s
  2.   cluster:
  3.     id:     4f79f6df-abf6-4461-a5be-9e1aa6adb9a9
  4.     health: HEALTH_WARN
  5.             clock skew detected on mon.mon2
  7.   services:
  8.     mon: 2 daemons, quorum mon1,mon2 (age 10m)
  9.     mgr: mon1(active, since 16s), standbys: mon2
  10.     osd: 0 osds: 0 up, 0 in
  12.   data:
  13.     pools:   0 pools, 0 pgs
  14.     objects: 0 objects, 0 B
  15.     usage:   0 B used, 0 B / 0 B avail
  16.     pgs:
  18. [root@mon1 ceph]# ceph quorum_status --format json-pretty
  20. {
  21.     "election_epoch": 4,
  22.     "quorum": [
  23.         0,
  24.         1
  25.     ],
  26.     "quorum_names": [
  27.         "mon1",
  28.         "mon2"
  29.     ],
  30.     "quorum_leader_name": "mon1",
  31.     "quorum_age": 980,
  32.     "monmap": {
  33.         "epoch": 1,
  34.         "fsid": "4f79f6df-abf6-4461-a5be-9e1aa6adb9a9",
  35.         "modified": "2020-12-28 07:53:10.709467",
  36.         "created": "2020-12-28 07:53:10.709467",
  37.         "min_mon_release": 14,
  38.         "min_mon_release_name": "nautilus",
  39.         "features": {
  40.             "persistent": [
  41.                 "kraken",
  42.                 "luminous",
  43.                 "mimic",
  44.                 "osdmap-prune",
  45.                 "nautilus"
  46.             ],
  47.             "optional": []
  48.         },
  49.         "mons": [
  50.             {
  51.                 "rank": 0,
  52.                 "name": "mon1",
  53.                 "public_addrs": {
  54.                     "addrvec": [
  55.                         {
  56.                             "type": "v2",
  57.                             "addr": "10.68.3.121:3300",
  58.                             "nonce": 0
  59.                         },
  60.                         {
  61.                             "type": "v1",
  62.                             "addr": "10.68.3.121:6789",
  63.                             "nonce": 0
  64.                         }
  65.                     ]
  66.                 },
  67.                 "addr": "10.68.3.121:6789/0",
  68.                 "public_addr": "10.68.3.121:6789/0"
  69.             },
  70.             {
  71.                 "rank": 1,
  72.                 "name": "mon2",
  73.                 "public_addrs": {
  74.                     "addrvec": [
  75.                         {
  76.                             "type": "v2",
  77.                             "addr": "10.68.3.122:3300",
  78.                             "nonce": 0
  79.                         },
  80.                         {
  81.                             "type": "v1",
  82.                             "addr": "10.68.3.122:6789",
  83.                             "nonce": 0
  84.                         }
  85.                     ]
  86.                 },
  87.                 "addr": "10.68.3.122:6789/0",
  88.                 "public_addr": "10.68.3.122:6789/0"
  89.             }
  90.         ]
  91.     }
  92. }
  • 查看mon状态 ceph mon stat / ceph mon dump
    1. [root@mon1 ceph]# ceph mon stat
    2. e1: 2 mons at {mon1=[v2:10.68.3.121:3300/0,v1:10.68.3.121:6789/0],mon2=[v2:10.68.3.122:3300/0,v1:10.68.3.122:6789/0]}, election epoch 4, leader 0 mon1, quorum 0,1 mon1,mon2
    1. [root@mon1 ceph]# ceph mon dump
    2. dumped monmap epoch 4
    3. epoch 4
    4. fsid 4f79f6df-abf6-4461-a5be-9e1aa6adb9a9
    5. last_changed 2020-12-28 16:36:15.204720
    6. created 2020-12-28 07:53:10.709467
    7. min_mon_release 14 (nautilus)
    8. 0: [v2:10.68.3.121:3300/0,v1:10.68.3.121:6789/0] mon.mon1
    9. 1: [v2:10.68.3.122:3300/0,v1:10.68.3.122:6789/0] mon.mon2
    10. 2: [v2:10.68.3.123:3300/0,v1:10.68.3.123:6789/0] mon.mon3
    11. 3: [v2:10.68.3.124:3300/0,v1:10.68.3.124:6789/0] mon.mon4
    12. 4: [v2:10.68.3.125:3300/0,v1:10.68.3.125:6789/0] mon.mon5

4.2- 扩容mgr节点

Ceph Manager守护程序以 active/standby模式运行。部署其他manager daemons可确保如果一个守护程序或主机发生故障，另一守护程序或主机可以接管而不会中断服务。

  • 添加mon3,mon4,mon5节点

    1. ceph-deploy mgr create mon3 mon4 mon5

  • 验证

    1. ceph mgr dump

    4.3- 扩容rgw

    1. ceph-deploy rgw mon3 mon4 mon

    4.3- 扩容OSD

  • 添加节点上剩余的osd磁盘 ```bash

    如果使用volume卷则卷地址为 vg/lv

    ceph-deploy osd create —bluestore mon3 —fs-type xfs —data /dev/sdd —journal journal/sdd

```