1 为什么使用cluster模式

官网地址：https://redis.io/topics/cluster-tutorial

在主从模式下解决了单点故障问题，并且主从分离以后redis master节点的读的压力。但是master故障后，不会自动转移，此时哨兵模式解决了该问题。在哨兵模式下，从节点无法写入数据，主节点写能力受网络和磁盘影响较大，cluster模式将数据分片，存在各个master节点上，能够有效的解决master节点的写压力；

cluster模式具有以下特点：

  • 无中心：各个节点都可以写数据
  • 至少6个节点，三主三从，可扩展；
  • 各个节点通信通过gossip协议；
  • 数据分布在各个节点上；

cluster模式的数据分片是通过crc16算法对存入数据的key进行计算来决定改key存放的节点；每个master节点都会给分配hash槽位（16384个），slave节点不会被分配。

2 基本配置解释

  1. ################################ REDIS CLUSTER  ###############################
  2. # 是否开启cluster
  3. cluster-enabled yes
  4. # 集群配置文件，不推荐手动修改，文件由redis自身修改
  5. cluster-config-file nodes-7001.conf
  6. # 节点超时时间
  7. cluster-node-timeout 15000
  8. ########################## CLUSTER DOCKER/NAT support  ########################
  9. # 声明自身的ip地址，端口号，以及总线端口（一般情况端口号加10000）
  10. cluster-announce-ip 192.168.1.1
  11. cluster-announce-port 7001
  12. cluster-announce-bus-port 17001

3 搭建环境

端口 redis01 redis02 redis03 redis04 redis05 redis06
port 7001 7002 7003 7004 7005 7006
总线port 17001 17002 17003 17004 17005 17006

redis.conf配置文件:

  1. ################################# GENERAL #####################################
  2. # 以前端形式运行，如果守护进程运行，容器没有主进程无法启动
  3. daemonize no   
  4. pidfile /var/run/redis_7001.pid
  5. loglevel notice
  6. logfile ""
  7. databases 16
  8. ################################## NETWORK #####################################
  9. bind 0.0.0.0
  10. protected-mode yes
  11. port 6381
  12. tcp-backlog 511
  13. timeout 0
  14. tcp-keepalive 300
  15. ################################ SNAPSHOTTING  ################################
  16. save 3600 1
  17. save 300 100
  18. save 60 10000
  19. stop-writes-on-bgsave-error yes
  20. rdbchecksum yes
  21. dbfilename dump.rdb
  22. rdb-del-sync-files no
  23. dir /data
  24. ################################## SECURITY ###################################
  25. requirepass 123456
  26. ################################# REPLICATION #################################
  27. masterauth 123456
  28. ################################ REDIS CLUSTER  ###############################
  29. cluster-enabled yes
  30. cluster-config-file nodes-7001.conf
  31. cluster-node-timeout 15000
  32. ########################## CLUSTER DOCKER/NAT support  ########################
  33. cluster-announce-ip 192.168.1.11
  34. cluster-announce-port 7001
  35. cluster-announce-bus-port 17001

启动脚本如下：

  1. [root@es01 cluster]# cat redis01/run-redis.sh 
  2. docker rm -f cluster-redis01
  4. current_dir=$(cd $(dirname $0);pwd)
  6. docker run -d --name cluster-redis01 \
  7.     -v $current_dir/redis.conf:/etc/redis/redis.conf \
  8.     --net host \
  9.     -v $current_dir/data/:/data \
  10.     -v /etc/localtime:/etc/localtime \
  11.     redis:6.2.6  /etc/redis/redis.conf

工作目录如下：

  1. [root@es01 cluster]# tree .
  2. .
  3. ├── redis01
  4. │   ├── data
  5. │   │   └── nodes-7001.conf
  6. │   ├── redis.conf
  7. │   └── run-redis.sh
  8. ├── redis02
  9. │   ├── data
  10. │   │   └── nodes-7002.conf
  11. │   ├── redis.conf
  12. │   └── run-redis.sh
  13. ├── redis03
  14. │   ├── data
  15. │   │   └── nodes-7003.conf
  16. │   ├── redis.conf
  17. │   └── run-redis.sh
  18. ├── redis04
  19. │   ├── data
  20. │   │   └── nodes-7004.conf
  21. │   ├── redis.conf
  22. │   └── run-redis.sh
  23. ├── redis05
  24. │   ├── data
  25. │   │   └── nodes-7005.conf
  26. │   ├── redis.conf
  27. │   └── run-redis.sh
  28. ├── redis06
  29. │   ├── data
  30. │   │   └── nodes-7006.conf
  31. │   ├── redis.conf
  32. │   └── run-redis.sh
  33. └── upClusterRedis.sh
  35. 12 directories, 19 files

创建集群，副本数为1

  1. redis-cli -a 123456 --cluster create 192.168.1.11:7001 192.168.1.11:7002 192.168.1.11:7003 192.168.1.11:7004 192.168.1.11:7005 192.168.1.11:7006 --cluster-replicas 1
  1. [root@es01 cluster]# redis-cli -a 123456 --cluster create 192.168.1.11:7001 192.168.1.11:7002 192.168.1.11:7003 192.168.1.11:7004 192.168.1.11:7005 192.168.1.11:7006 --cluster-replicas 1 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Performing hash slots allocation on 6 nodes...
  4. Master[0] -> Slots 0 - 5460
  5. Master[1] -> Slots 5461 - 10922
  6. Master[2] -> Slots 10923 - 16383
  7. Adding replica 192.168.1.11:7005 to 192.168.1.11:7001
  8. Adding replica 192.168.1.11:7006 to 192.168.1.11:7002
  9. Adding replica 192.168.1.11:7004 to 192.168.1.11:7003
  10. >>> Trying to optimize slaves allocation for anti-affinity
  11. [WARNING] Some slaves are in the same host as their master
  12. M: e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001
  13.    slots:[0-5460] (5461 slots) master
  14. M: 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002
  15.    slots:[5461-10922] (5462 slots) master
  16. M: 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003
  17.    slots:[10923-16383] (5461 slots) master
  18. S: 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004
  19.    replicates 9eee9fdef750a76681c9a434a5b38998d33765dc
  20. S: fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005
  21.    replicates e28d175a242a4eddeec24d2cf3862fb8e23467dc
  22. S: 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006
  23.    replicates 24c158d00fb92da6d0ba39ad5eef21db6f5fab00
  24. Can I set the above configuration? (type 'yes' to accept): yes
  25. >>> Nodes configuration updated
  26. >>> Assign a different config epoch to each node
  27. >>> Sending CLUSTER MEET messages to join the cluster
  28. Waiting for the cluster to join
  29. .
  30. >>> Performing Cluster Check (using node 192.168.1.11:7001)
  31. M: e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001
  32.    slots:[0-5460] (5461 slots) master
  33.    1 additional replica(s)
  34. M: 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003
  35.    slots:[10923-16383] (5461 slots) master
  36.    1 additional replica(s)
  37. S: fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005
  38.    slots: (0 slots) slave
  39.    replicates e28d175a242a4eddeec24d2cf3862fb8e23467dc
  40. M: 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002
  41.    slots:[5461-10922] (5462 slots) master
  42.    1 additional replica(s)
  43. S: 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004
  44.    slots: (0 slots) slave
  45.    replicates 9eee9fdef750a76681c9a434a5b38998d33765dc
  46. S: 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006
  47.    slots: (0 slots) slave
  48.    replicates 24c158d00fb92da6d0ba39ad5eef21db6f5fab00
  49. [OK] All nodes agree about slots configuration.
  50. >>> Check for open slots...
  51. >>> Check slots coverage...
  52. [OK] All 16384 slots covered.

检查cluster集群

  1. [root@es01 cluster]# redis-cli -a 123456 --cluster check 192.168.1.11:7001 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. 192.168.1.11:7001 (e28d175a...) -> 0 keys | 5461 slots | 1 slaves.
  4. 192.168.1.11:7003 (9eee9fde...) -> 0 keys | 5461 slots | 1 slaves.
  5. 192.168.1.11:7002 (24c158d0...) -> 0 keys | 5462 slots | 1 slaves.
  6. [OK] 0 keys in 3 masters.
  7. 0.00 keys per slot on average.
  8. >>> Performing Cluster Check (using node 192.168.1.11:7001)
  9. M: e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001
  10.    slots:[0-5460] (5461 slots) master
  11.    1 additional replica(s)
  12. M: 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003
  13.    slots:[10923-16383] (5461 slots) master
  14.    1 additional replica(s)
  15. S: fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005
  16.    slots: (0 slots) slave
  17.    replicates e28d175a242a4eddeec24d2cf3862fb8e23467dc
  18. M: 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002
  19.    slots:[5461-10922] (5462 slots) master
  20.    1 additional replica(s)
  21. S: 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004
  22.    slots: (0 slots) slave
  23.    replicates 9eee9fdef750a76681c9a434a5b38998d33765dc
  24. S: 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006
  25.    slots: (0 slots) slave
  26.    replicates 24c158d00fb92da6d0ba39ad5eef21db6f5fab00
  27. [OK] All nodes agree about slots configuration.
  28. >>> Check for open slots...
  29. >>> Check slots coverage...
  30. [OK] All 16384 slots covered.

查看集群信息

  1. [root@es01 cluster]# redis-cli -a 123456 -p 7001
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. 127.0.0.1:7001> CLUSTER INFO
  4. cluster_state:ok
  5. cluster_slots_assigned:16384
  6. cluster_slots_ok:16384
  7. cluster_slots_pfail:0
  8. cluster_slots_fail:0
  9. cluster_known_nodes:6
  10. cluster_size:3
  11. cluster_current_epoch:6
  12. cluster_my_epoch:1
  13. cluster_stats_messages_ping_sent:103
  14. cluster_stats_messages_pong_sent:113
  15. cluster_stats_messages_sent:216
  16. cluster_stats_messages_ping_received:108
  17. cluster_stats_messages_pong_received:103
  18. cluster_stats_messages_meet_received:5
  19. cluster_stats_messages_received:216
  20. 127.0.0.1:7001> 
  21. 127.0.0.1:7001> CLUSTER NODES
  22. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644637243462 3 connected 10923-16383
  23. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 slave e28d175a242a4eddeec24d2cf3862fb8e23467dc 0 1644637242445 1 connected
  24. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 myself,master - 0 1644637244000 1 connected 0-5460
  25. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644637242000 2 connected 5461-10922
  26. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644637241000 3 connected
  27. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644637244482 2 connected

整理一下当前的集群信息

master 192.168.1.11:7001 192.168.1.11:7002 192.168.1.11:7003
slave 192.168.1.11:7005 192.168.1.11:7006 192.168.1.11:7004

设置一个key，k1刚好经过crc16算法，存放在7003节点，当设置k2时可以看到经过计算后，k2存放在7001节点。

  1. [root@es01 redis01]# redis-cli -a 123456 -p 7003 -c 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. 127.0.0.1:7003> set k1 v1 
  4. OK
  5. 127.0.0.1:7003> set k2 v2
  6. -> Redirected to slot [449] located at 192.168.1.11:7001
  7. OK

此处需要注意的时，连接redis集群需要加上-c参数，告诉客户端以集群方式连接，如果不加会报错：

  1. [root@es01 cluster]# redis-cli -a 123456 -p 7003
  2. 127.0.0.1:7003> set k2 v2 
  3. (error) MOVED 449 192.168.1.11:7001

4 故障转移

模拟故障转移，将7001停掉

  1. docker stop cluster-redis01

查看7005日志，当7001宕机后，日志显示连接master节点进行同步失败，然后进行故障转移，7005 slave节点升级为master节点。

  1. # docker logs -f cluster-redis05
  2. 1:S 12 Feb 2022 14:10:45.296 # Connection with master lost.
  3. 1:S 12 Feb 2022 14:10:45.296 * Caching the disconnected master state.
  4. 1:S 12 Feb 2022 14:10:45.296 * Reconnecting to MASTER 192.168.1.11:7001
  5. 1:S 12 Feb 2022 14:10:45.296 * MASTER <-> REPLICA sync started
  6. 1:S 12 Feb 2022 14:10:45.296 # Error condition on socket for SYNC: Connection refused
  7. 1:S 12 Feb 2022 14:10:45.624 * Connecting to MASTER 192.168.1.11:7001
  8. 1:S 12 Feb 2022 14:10:45.624 * MASTER <-> REPLICA sync started
  9. 1:S 12 Feb 2022 14:10:45.624 # Error condition on socket for SYNC: Connection refused
  10. 1:S 12 Feb 2022 14:10:46.639 * Connecting to MASTER 192.168.1.11:7001
  11. 1:S 12 Feb 2022 14:10:46.640 * MASTER <-> REPLICA sync started
  12. 1:S 12 Feb 2022 14:10:46.640 # Error condition on socket for SYNC: Connection refused
  13. 1:S 12 Feb 2022 14:10:47.648 * Connecting to MASTER 192.168.1.11:7001
  14. 1:S 12 Feb 2022 14:10:47.648 * MASTER <-> REPLICA sync started
  15. 1:S 12 Feb 2022 14:10:47.648 # Error condition on socket for SYNC: Connection refused
  16. 1:S 12 Feb 2022 14:10:48.658 * Connecting to MASTER 192.168.1.11:7001
  17. 1:S 12 Feb 2022 14:10:48.658 * MASTER <-> REPLICA sync started
  18. 1:S 12 Feb 2022 14:10:48.658 # Error condition on socket for SYNC: Connection refused
  19. 1:S 12 Feb 2022 14:10:49.668 * Connecting to MASTER 192.168.1.11:7001
  20. 1:S 12 Feb 2022 14:10:49.668 * MASTER <-> REPLICA sync started
  21. 1:S 12 Feb 2022 14:10:49.668 # Error condition on socket for SYNC: Connection refused
  22. 1:S 12 Feb 2022 14:10:50.680 * Connecting to MASTER 192.168.1.11:7001
  23. 1:S 12 Feb 2022 14:10:50.680 * MASTER <-> REPLICA sync started
  24. 1:S 12 Feb 2022 14:10:50.680 # Error condition on socket for SYNC: Connection refused
  25. 1:S 12 Feb 2022 14:10:51.689 * Connecting to MASTER 192.168.1.11:7001
  26. 1:S 12 Feb 2022 14:10:51.689 * MASTER <-> REPLICA sync started
  27. 1:S 12 Feb 2022 14:10:51.689 # Error condition on socket for SYNC: Connection refused
  28. 1:S 12 Feb 2022 14:10:52.697 * Connecting to MASTER 192.168.1.11:7001
  29. 1:S 12 Feb 2022 14:10:52.697 * MASTER <-> REPLICA sync started
  30. 1:S 12 Feb 2022 14:10:52.697 # Error condition on socket for SYNC: Connection refused
  31. 1:S 12 Feb 2022 14:10:53.707 * Connecting to MASTER 192.168.1.11:7001
  32. 1:S 12 Feb 2022 14:10:53.707 * MASTER <-> REPLICA sync started
  33. 1:S 12 Feb 2022 14:10:53.707 # Error condition on socket for SYNC: Connection refused
  34. 1:S 12 Feb 2022 14:10:54.716 * Connecting to MASTER 192.168.1.11:7001
  35. 1:S 12 Feb 2022 14:10:54.716 * MASTER <-> REPLICA sync started
  36. 1:S 12 Feb 2022 14:10:54.716 # Error condition on socket for SYNC: Connection refused
  37. 1:S 12 Feb 2022 14:10:55.724 * Connecting to MASTER 192.168.1.11:7001
  38. 1:S 12 Feb 2022 14:10:55.724 * MASTER <-> REPLICA sync started
  39. 1:S 12 Feb 2022 14:10:55.724 # Error condition on socket for SYNC: Connection refused
  40. 1:S 12 Feb 2022 14:10:56.731 * Connecting to MASTER 192.168.1.11:7001
  41. 1:S 12 Feb 2022 14:10:56.731 * MASTER <-> REPLICA sync started
  42. 1:S 12 Feb 2022 14:10:56.731 # Error condition on socket for SYNC: Connection refused
  43. 1:S 12 Feb 2022 14:10:57.746 * Connecting to MASTER 192.168.1.11:7001
  44. 1:S 12 Feb 2022 14:10:57.746 * MASTER <-> REPLICA sync started
  45. 1:S 12 Feb 2022 14:10:57.746 # Error condition on socket for SYNC: Connection refused
  46. 1:S 12 Feb 2022 14:10:58.765 * Connecting to MASTER 192.168.1.11:7001
  47. 1:S 12 Feb 2022 14:10:58.765 * MASTER <-> REPLICA sync started
  48. 1:S 12 Feb 2022 14:10:58.765 # Error condition on socket for SYNC: Connection refused
  49. 1:S 12 Feb 2022 14:10:59.787 * Connecting to MASTER 192.168.1.11:7001
  50. 1:S 12 Feb 2022 14:10:59.787 * MASTER <-> REPLICA sync started
  51. 1:S 12 Feb 2022 14:10:59.787 # Error condition on socket for SYNC: Connection refused
  52. 1:S 12 Feb 2022 14:11:00.805 * Connecting to MASTER 192.168.1.11:7001
  53. 1:S 12 Feb 2022 14:11:00.806 * MASTER <-> REPLICA sync started
  54. 1:S 12 Feb 2022 14:11:00.806 # Error condition on socket for SYNC: Connection refused
  55. 1:S 12 Feb 2022 14:11:01.835 * Connecting to MASTER 192.168.1.11:7001
  56. 1:S 12 Feb 2022 14:11:01.835 * MASTER <-> REPLICA sync started
  57. 1:S 12 Feb 2022 14:11:01.835 # Error condition on socket for SYNC: Connection refused
  58. 1:S 12 Feb 2022 14:11:02.853 * Connecting to MASTER 192.168.1.11:7001
  59. 1:S 12 Feb 2022 14:11:02.853 * MASTER <-> REPLICA sync started
  60. 1:S 12 Feb 2022 14:11:02.853 # Error condition on socket for SYNC: Connection refused
  61. 1:S 12 Feb 2022 14:11:03.873 * Connecting to MASTER 192.168.1.11:7001
  62. 1:S 12 Feb 2022 14:11:03.873 * MASTER <-> REPLICA sync started
  63. 1:S 12 Feb 2022 14:11:03.873 # Error condition on socket for SYNC: Connection refused
  64. 1:S 12 Feb 2022 14:11:04.898 * Connecting to MASTER 192.168.1.11:7001
  65. 1:S 12 Feb 2022 14:11:04.898 * MASTER <-> REPLICA sync started
  66. 1:S 12 Feb 2022 14:11:04.898 # Error condition on socket for SYNC: Connection refused
  67. 1:S 12 Feb 2022 14:11:05.101 * FAIL message received from 4e3c5f793a98e274c105a946350f6b887bb1833d about e28d175a242a4eddeec24d2cf3862fb8e23467dc
  68. 1:S 12 Feb 2022 14:11:05.101 # Start of election delayed for 931 milliseconds (rank #0, offset 12582).
  69. 1:S 12 Feb 2022 14:11:05.101 # Cluster state changed: fail
  70. 1:S 12 Feb 2022 14:11:05.920 * Connecting to MASTER 192.168.1.11:7001
  71. 1:S 12 Feb 2022 14:11:05.920 * MASTER <-> REPLICA sync started
  72. 1:S 12 Feb 2022 14:11:05.920 # Error condition on socket for SYNC: Connection refused
  73. 1:S 12 Feb 2022 14:11:06.122 # Starting a failover election for epoch 7.
  74. 1:S 12 Feb 2022 14:11:06.124 # Failover election won: I'm the new master.
  75. 1:S 12 Feb 2022 14:11:06.124 # configEpoch set to 7 after successful failover
  76. 1:M 12 Feb 2022 14:11:06.124 * Discarding previously cached master state.
  77. 1:M 12 Feb 2022 14:11:06.124 # Setting secondary replication ID to 516ab19c01da7134706508b65a31eb3d8ad0bdc2, valid up to offset: 12583. New replication ID is 6134525316061eafaa676761e20fe052967766ec
  78. 1:M 12 Feb 2022 14:11:06.124 # Cluster state changed: ok

查看集群状态，7001状态为fail，7005节点已经切换成master节点。

  1. [root@es01 redis01]# redis-cli -c -a 123456 -p 7005
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. 127.0.0.1:7005> role
  4. 1) "master"
  5. 2) (integer) 12582
  6. 3) (empty array)
  7. 127.0.0.1:7005> 
  8. 127.0.0.1:7005> CLUSTER NODES
  9. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 myself,master - 0 1644646563000 7 connected 0-5460
  10. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644646565134 3 connected 10923-16383
  11. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644646564115 3 connected
  12. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644646562078 2 connected 5461-10922
  13. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 master,fail - 1644646251588 1644646244502 1 disconnected
  14. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644646562000 2 connected

重新启动7001节点

  1. docker  start cluster-redis01

查看7005节点日志

  1. 1:M 12 Feb 2022 14:17:24.713 * Clear FAIL state for node e28d175a242a4eddeec24d2cf3862fb8e23467dc: replica is reachable again.
  2. 1:M 12 Feb 2022 14:17:24.714 * Replica 192.168.1.11:7001 asks for synchronization
  3. 1:M 12 Feb 2022 14:17:24.714 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '302913ac0294db7856142a306df5ff4f8640db4c', my replication IDs are '6134525316061eafaa676761e20fe052967766ec' and '516ab19c01da7134706508b65a31eb3d8ad0bdc2')
  4. 1:M 12 Feb 2022 14:17:24.714 * Starting BGSAVE for SYNC with target: disk
  5. 1:M 12 Feb 2022 14:17:24.715 * Background saving started by pid 20
  6. 20:C 12 Feb 2022 14:17:24.718 * DB saved on disk
  7. 20:C 12 Feb 2022 14:17:24.718 * RDB: 2 MB of memory used by copy-on-write
  8. 1:M 12 Feb 2022 14:17:24.813 * Background saving terminated with success
  9. 1:M 12 Feb 2022 14:17:24.813 * Synchronization with replica 192.168.1.11:7001 succeeded

查看集群状态，7001节点重新加入集群成为7005节点的slave

  1. 127.0.0.1:7005> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 myself,master - 0 1644646692000 7 connected 0-5460
  3. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644646690000 3 connected 10923-16383
  4. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644646692623 3 connected
  5. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644646690580 2 connected 5461-10922
  6. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644646689000 7 connected
  7. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644646691601 2 connected

5 添加节点

5.1 添加主节点

端口号为7007（需要先启动一个容器），执行以下命令添加

  1. [root@es01 add01]# redis-cli -a 123456 --cluster add-node 192.168.1.11:7007 192.168.1.11:7001 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Adding node 192.168.1.11:7007 to cluster 192.168.1.11:7001
  4. >>> Performing Cluster Check (using node 192.168.1.11:7001)
  5. S: e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001
  6.    slots: (0 slots) slave
  7.    replicates fe4efdc11de7ac0d44636fdde10b6da41fea74f9
  8. M: fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005
  9.    slots:[0-5460] (5461 slots) master
  10.    1 additional replica(s)
  11. S: 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004
  12.    slots: (0 slots) slave
  13.    replicates 9eee9fdef750a76681c9a434a5b38998d33765dc
  14. M: 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003
  15.    slots:[10923-16383] (5461 slots) master
  16.    1 additional replica(s)
  17. M: 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002
  18.    slots:[5461-10922] (5462 slots) master
  19.    1 additional replica(s)
  20. S: 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006
  21.    slots: (0 slots) slave
  22.    replicates 24c158d00fb92da6d0ba39ad5eef21db6f5fab00
  23. [OK] All nodes agree about slots configuration.
  24. >>> Check for open slots...
  25. >>> Check slots coverage...
  26. [OK] All 16384 slots covered.
  27. >>> Send CLUSTER MEET to node 192.168.1.11:7007 to make it join the cluster.
  28. [OK] New node added correctly.

5.2 分配槽位

添加节点需要给新节点分配操作，没有槽位的节点时没有办法工作的，不能够存放数据。
新添加的节点槽位未分配

  1. 127.0.0.1:7001> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 master - 0 1644647700826 7 connected 0-5460
  3. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644647699000 3 connected
  4. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644647700000 3 connected 10923-16383
  5. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 myself,slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644647700000 7 connected
  6. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644647699809 2 connected 5461-10922
  7. 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007@17007 master - 0 1644647700000 0 connected
  8. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644647701850 2 connected

执行以下命令进行槽位分配

  1. # redis-cli -a 123456 --cluster reshard 192.168.1.11:7001 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Performing Cluster Check (using node 192.168.1.11:7001)
  4. S: f399aa5bbf0c81041b019e9c376707aec1b1c2f6 192.168.1.11:7001
  5.    slots: (0 slots) slave
  6.    replicates fe996744c3905edfcd5fb8ed6382fe20cee3125a
  7. M: 2e7416a90b21d9f7bfaa5f8fc3fd50a8c97e12e4 192.168.1.11:7002
  8.    slots:[6827-10922] (4096 slots) master
  9.    1 additional replica(s)
  10. M: fe996744c3905edfcd5fb8ed6382fe20cee3125a 192.168.1.11:7005
  11.    slots:[1365-5460] (4096 slots) master
  12.    1 additional replica(s)
  13. S: 20adb3b635d399f5d8abc15543d494fa51373f7f 192.168.1.11:7006
  14.    slots: (0 slots) slave
  15.    replicates 2e7416a90b21d9f7bfaa5f8fc3fd50a8c97e12e4
  16. M: ffc30b239ac989150c1eab5f11390e6c6b1ec36e 192.168.1.11:7007
  17.    slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
  18. M: 4cf440d6c66cfcb4a27805848f35c8f96578056f 192.168.1.11:7003
  19.    slots:[12288-16383] (4096 slots) master
  20.    1 additional replica(s)
  21. S: 437567fb43a22af468507ce80cf152792ed23afa 192.168.1.11:7004
  22.    slots: (0 slots) slave
  23.    replicates 4cf440d6c66cfcb4a27805848f35c8f96578056f
  24. [OK] All nodes agree about slots configuration.
  25. >>> Check for open slots...
  26. >>> Check slots coverage...
  27. [OK] All 16384 slots covered.
  28. How many slots do you want to move (from 1 to 16384)? 2048   # 分配的槽位个数
  29. What is the receiving node ID? ffc30b239ac989150c1eab5f11390e6c6b1ec36e
  30. Please enter all the source node IDs.
  31.   Type 'all' to use all the nodes as source nodes for the hash slots.
  32.   Type 'done' once you entered all the source nodes IDs.
  33. Source node #1: all # 输入all代表从现有的所有master进行分配

查看节点信息，可以看到7007节点从其他几个主节点获取到了2048个槽位

  1. 127.0.0.1:7001> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 master - 0 1644648073898 7 connected 682-5460
  3. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644648072871 3 connected
  4. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644648072000 3 connected 11605-16383
  5. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 myself,slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644648071000 7 connected
  6. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644648070824 2 connected 6144-10922
  7. 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007@17007 master - 0 1644648072000 8 connected 0-681 5461-6143 10923-11604
  8. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644648070000 2 connected

5.4 添加从节点

执行以下命令添加

  1. # redis-cli -a 123456 -c --cluster add-node --cluster-slave --cluster-master-id 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7008 192.168.1.11:7001 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Adding node 192.168.1.11:7008 to cluster 192.168.1.11:7001
  4. >>> Performing Cluster Check (using node 192.168.1.11:7001)
  5. S: e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001
  6.    slots: (0 slots) slave
  7.    replicates fe4efdc11de7ac0d44636fdde10b6da41fea74f9
  8. M: fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005
  9.    slots:[682-5460] (4779 slots) master
  10.    1 additional replica(s)
  11. S: 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004
  12.    slots: (0 slots) slave
  13.    replicates 9eee9fdef750a76681c9a434a5b38998d33765dc
  14. M: 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003
  15.    slots:[11605-16383] (4779 slots) master
  16.    1 additional replica(s)
  17. M: 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002
  18.    slots:[6144-10922] (4779 slots) master
  19.    1 additional replica(s)
  20. M: 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007
  21.    slots:[0-681],[5461-6143],[10923-11604] (2047 slots) master
  22. S: 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006
  23.    slots: (0 slots) slave
  24.    replicates 24c158d00fb92da6d0ba39ad5eef21db6f5fab00
  25. [OK] All nodes agree about slots configuration.
  26. >>> Check for open slots...
  27. >>> Check slots coverage...
  28. [OK] All 16384 slots covered.
  29. >>> Send CLUSTER MEET to node 192.168.1.11:7008 to make it join the cluster.
  30. Waiting for the cluster to join
  32. >>> Configure node as replica of 192.168.1.11:7007.
  33. [OK] New node added correctly.

查看节点信息，7008作为7007的从节点加入了集群

  1. 127.0.0.1:7005> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 myself,master - 0 1644648434000 7 connected 682-5460
  3. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644648433000 3 connected 11605-16383
  4. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644648432505 3 connected
  5. 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007@17007 master - 0 1644648434551 8 connected 0-681 5461-6143 10923-11604
  6. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644648433525 2 connected 6144-10922
  7. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644648435000 7 connected
  8. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644648432000 2 connected
  9. 57a0b8b52c3b806a8573029ee3563a2be6f38b3a 192.168.1.11:7008@17008 slave 8725b260b9f78627eedc9b479580b386aa186ae0 0 1644648435575 8 connected

6 删除节点

6.1 删除从节点

先删除从节点7008

  1. # redis-cli -a 123456 --cluster del-node 192.168.1.11:7001 57a0b8b52c3b806a8573029ee3563a2be6f38b3a 
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Removing node 57a0b8b52c3b806a8573029ee3563a2be6f38b3a from cluster 192.168.1.11:7001
  4. >>> Sending CLUSTER FORGET messages to the cluster...
  5. >>> Sending CLUSTER RESET SOFT to the deleted node.

查看集群信息

  1. 127.0.0.1:7005> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 myself,master - 0 1644649135000 7 connected 682-5460
  3. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644649131249 3 connected 11605-16383
  4. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644649135336 3 connected
  5. 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007@17007 master - 0 1644649133293 8 connected 0-681 5461-6143 10923-11604
  6. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644649133000 2 connected 6144-10922
  7. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644649134313 7 connected
  8. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644649133000 2 connected

6.2 删除主节点槽位

在删除主节点之前，需要先将已经分配给主节点7007的槽位分配出去。这里将7007的1686个槽位分配给7005master节点。

  1. # redis-cli -a 123456 --cluster reshard 192.168.1.11:7001
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Performing Cluster Check (using node 192.168.1.11:7001)
  4. S: e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001
  5.    slots: (0 slots) slave
  6.    replicates fe4efdc11de7ac0d44636fdde10b6da41fea74f9
  7. M: fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005
  8.    slots:[0-360],[682-5460],[6144-6985],[11605-12447] (6825 slots) master
  9.    1 additional replica(s)
  10. S: 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004
  11.    slots: (0 slots) slave
  12.    replicates 9eee9fdef750a76681c9a434a5b38998d33765dc
  13. M: 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003
  14.    slots:[12448-16383] (3936 slots) master
  15.    1 additional replica(s)
  16. M: 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002
  17.    slots:[6986-10922] (3937 slots) master
  18.    1 additional replica(s)
  19. M: 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007
  20.    slots:[361-681],[5461-6143],[10923-11604] (1686 slots) master
  21. S: 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006
  22.    slots: (0 slots) slave
  23.    replicates 24c158d00fb92da6d0ba39ad5eef21db6f5fab00
  24. [OK] All nodes agree about slots configuration.
  25. >>> Check for open slots...
  26. >>> Check slots coverage...
  27. [OK] All 16384 slots covered.
  28. How many slots do you want to move (from 1 to 16384)? 1686
  29. What is the receiving node ID? fe4efdc11de7ac0d44636fdde10b6da41fea74f9
  30. Please enter all the source node IDs.
  31.   Type 'all' to use all the nodes as source nodes for the hash slots.
  32.   Type 'done' once you entered all the source nodes IDs.
  33. Source node #1: 8725b260b9f78627eedc9b479580b386aa186ae0
  34. Source node #2: done

查看7007 master节点已经没有槽位了

  1. 127.0.0.1:7005> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 myself,master - 0 1644649532000 9 connected 0-6985 10923-12447
  3. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644649534761 3 connected 12448-16383
  4. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644649535767 3 connected
  5. 8725b260b9f78627eedc9b479580b386aa186ae0 192.168.1.11:7007@17007 master - 0 1644649533000 8 connected
  6. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644649533000 2 connected 6986-10922
  7. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644649534000 9 connected
  8. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644649532768 2 connected

6.3删除节点

  1. # redis-cli -a 123456 --cluster del-node 192.168.1.11:7001 8725b260b9f78627eedc9b479580b386aa186ae0
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Removing node 8725b260b9f78627eedc9b479580b386aa186ae0 from cluster 192.168.1.11:7001
  4. >>> Sending CLUSTER FORGET messages to the cluster...
  5. >>> Sending CLUSTER RESET SOFT to the deleted node.

查看集群信息

  1. 127.0.0.1:7005> CLUSTER NODES
  2. fe4efdc11de7ac0d44636fdde10b6da41fea74f9 192.168.1.11:7005@17005 myself,master - 0 1644649649000 9 connected 0-6985 10923-12447
  3. 9eee9fdef750a76681c9a434a5b38998d33765dc 192.168.1.11:7003@17003 master - 0 1644649649433 3 connected 12448-16383
  4. 4e3c5f793a98e274c105a946350f6b887bb1833d 192.168.1.11:7004@17004 slave 9eee9fdef750a76681c9a434a5b38998d33765dc 0 1644649648410 3 connected
  5. 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 192.168.1.11:7002@17002 master - 0 1644649650448 2 connected 6986-10922
  6. e28d175a242a4eddeec24d2cf3862fb8e23467dc 192.168.1.11:7001@17001 slave fe4efdc11de7ac0d44636fdde10b6da41fea74f9 0 1644649648000 9 connected
  7. 88628ef7f9d403a9526f6709833630aa27bdbbe1 192.168.1.11:7006@17006 slave 24c158d00fb92da6d0ba39ad5eef21db6f5fab00 0 1644649647395 2 connected

7 其他

查看集群内所有key信息

  1. ]# redis-cli -c -a 123456 --cluster call 192.168.1.11:7001 keys \*
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. >>> Calling keys *
  4. 192.168.1.11:7001: k2
  5. 192.168.1.11:7005: k2
  6. 192.168.1.11:7004: k1
  7. 192.168.1.11:7003: k1
  8. 192.168.1.11:7002: 
  9. 192.168.1.11:7006: