rocketmq deploy

简介

  1. 1. 名称服集群务 NameServer
  2.   Nameserver 提供了轻量级的服务发现和路由。每个 NameServer 服务记录完整的路由信息, 提供一致的读写服务，支持快速存储扩展.
  3.   主要包含两个功能:
  4.     1) broker 管理, NameServer 接受来自 Broker 集群的注册信息并提供心跳来检测他们是否可用。
  5.     2) 路由管理 每一个 NameServer 都持有关于 Broker 集群和队列的全部路由信息，用来向客户端提供查询。
  7. 2. 代理服务集群 Broker Cluster
  8.   Broker 通过提供轻量级主题和队列机制来处理消息存储。它们支持 Push 和 Pull 模型，包含容错机制(2个副本或3个副本)，提供了极强的峰值处理里能力和按照时间顺序存储数以百万记的消息存储能力，此外，代理提供了灾难恢复、丰富的度量统计和警报机制.
  9.   Broker Server 负责消息的存储传递，消息查询，保证高可用等:
  10.     1) remoting（远程） 模块，broker 的入口，处理从客户端发起的请求。
  11.     2) client manager（客户端管理） 管理各个客户端（生产者/消费者）还有维护消费者主题订阅。
  12.     3) store（存储服务），提供简单的 api 来在磁盘保持或者查询消息。
  13.     4) HA 高可用服务 提供主从 broker 的数据同步。
  14.     5) index(索引服务)为消息建立索引提供消息快速查询。
  16. 3. 生产者集群 Producer Cluster
  17.   Producer 支持分布式部署，分布式的 Producer 通过 Broker 集群提供的各种负载均衡策略将消息发送到 Broker 集群中。发送过程支持快速失败是低延迟的.
  18.   Producer 生产消息 Topic
  19.     1. 先请求 Nameserver 拿到这个 Topic 的路由信息
  20.     2. 查找 Topic 在哪个 Broker, Broker 上有哪些队列
  21.     3. 拿到这些请求后再把消息发送到指定的 Broker 中
  23. 4. 消费者集群 Consumer Cluster
  24.   消费者也支持在推送和者拉取模式下分布式部署，它还支持集群消费和消息广播。提供实时的消息订阅机制，能够满足大多数消费者的需求。RocketMQ的网站为感兴趣的用户提供了一个简单的快速入门指南.
  25.   Consumer 消费消息 Topic, 也会经历 3(Producer 生产消息 Topic) 这个流程

下载

安装

配置

  1. ### ---------- 2 master, 2 slave, 同步模式 -----------
  3. # broker-a.properties master-a
  4. # 这个经纪人属于哪个集群
  5. brokerClusterName=DefaultCluster
  6. # broker 姓名
  7. brokerName=broker-a
  8. # broker ID，0 表示 master，正整数表示 slave
  9. brokerId=0
  10. # 何时删除超出保留时间的提交日志
  11. deleteWhen=04
  12. # 在删除之前保留 commitlog 的小时数
  13. fileReservedTime=48
  14. # SYNC_MASTER(同步主) / ASYNC_MASTER(异步主) / SLAVE(SLAVE)
  15. brokerRole=SYNC_MASTER
  16. # SYNC_FLUSH 同步模式会在响应每次生产者前写入磁盘, ASYNC_FLUSH 异步模式会提高处理生产者组的提交处理能力
  17. flushDiskType=ASYNC_FLUSH
  20. # broker-a-s.properties slave-a
  21. brokerClusterName=DefaultCluster
  22. brokerName=broker-a
  23. brokerId=1
  24. deleteWhen=04
  25. fileReservedTime=48
  26. brokerRole=SLAVE
  27. flushDiskType=ASYNC_FLUSH
  30. # broker-b.properties master-b
  31. brokerClusterName=DefaultCluster
  32. brokerName=broker-b
  33. brokerId=0
  34. deleteWhen=04
  35. fileReservedTime=48
  36. brokerRole=SYNC_MASTER
  37. flushDiskType=ASYNC_FLUSH
  39. # broker-b-s.properties slave-b
  40. brokerClusterName=DefaultCluster
  41. brokerName=broker-b
  42. brokerId=1
  43. deleteWhen=04
  44. fileReservedTime=48
  45. brokerRole=SLAVE
  46. flushDiskType=ASYNC_FLUSH
  50. ### ---------- 2 master, 2 slave, 异步模式 -----------
  52. # broker-a.properties master-a
  53. brokerClusterName=DefaultCluster
  54. brokerName=broker-a
  55. brokerId=0
  56. deleteWhen=04
  57. fileReservedTime=48
  58. brokerRole=ASYNC_MASTER
  59. flushDiskType=ASYNC_FLUSH
  62. # broker-a-s.properties slave-a
  63. brokerClusterName=DefaultCluster
  64. brokerName=broker-a
  65. brokerId=1
  66. deleteWhen=04
  67. fileReservedTime=48
  68. brokerRole=SLAVE
  69. flushDiskType=ASYNC_FLUSH
  72. # broker-b.properties master-b
  73. brokerClusterName=DefaultCluster
  74. brokerName=broker-b
  75. brokerId=0
  76. deleteWhen=04
  77. fileReservedTime=48
  78. brokerRole=ASYNC_MASTER
  79. flushDiskType=ASYNC_FLUSH
  82. # broker-b-s.properties slave-b
  83. brokerClusterName=DefaultCluster
  84. brokerName=broker-b
  85. brokerId=1
  86. deleteWhen=04
  87. fileReservedTime=48
  88. brokerRole=SLAVE
  89. flushDiskType=ASYNC_FLUSH
  93. ### ---------- Broker GC 优化参数 ----------
  94. bin/runbroker.sh
  96. -server -Xms8g -Xmx8g -Xmn4g
  97. -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25
  98. -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0
  99. -XX:SurvivorRatio=8 -XX:+DisableExplicitGC
  100. -verbose:gc -Xloggc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails
  101. -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime
  102. -XX:+PrintAdaptiveSizePolicy
  103. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m

部署

  1. # -------------------- 启动 nameserver --------------------
  3. # 创建日志目录
  4. sudo mkdir -p /opt/logs/rocketmqlogs
  5. sudo chown $USER:$USER /opt/logs/rocketmqlogs
  7. # rocketmq nameserver
  8. 172.17.19.2     rocketmq-nameserver1
  9. 172.17.19.51    rocketmq-nameserver2
  10. 172.17.19.52    rocketmq-nameserver3
  12. # 启动 nameserver
  13. nohup ./bin/mqnamesrv &
  14.   jps | grep NamesrvStartup
  15.   netstat -tunlp | grep 9876
  17. # 关闭 nameserver
  18. ./bin/mqshutdown namesrv
  21. # -------------------- 第 a 组, 1 主 2 从 --------------------
  23. # 主节点 broker-a.properties
  24. brokerClusterName=DefaultCluster
  25. brokerName=broker-a
  26. brokerId=0
  27. deleteWhen=04
  28. fileReservedTime=48
  29. brokerRole=SYNC_MASTER
  30. flushDiskType=ASYNC_FLUSH
  32. # 从节点 broker-a-s1.properties
  33. brokerClusterName=DefaultCluster
  34. brokerName=broker-a
  35. brokerId=1
  36. deleteWhen=04
  37. fileReservedTime=48
  38. brokerRole=SLAVE
  39. flushDiskType=ASYNC_FLUSH
  41. # 从节点 broker-a-s2.properties
  42. brokerClusterName=DefaultCluster
  43. brokerName=broker-a
  44. brokerId=2
  45. deleteWhen=04
  46. fileReservedTime=48
  47. brokerRole=SLAVE
  48. flushDiskType=ASYNC_FLUSH
  50. # 启动 broker master
  51. nohup sh ./bin/mqbroker -c ./conf/custom/broker-a.properties -n "rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876" &
  53.   jps | grep BrokerStartup
  54.   netstat -tunlp | grep 10911
  56. # 启动 broker slave1
  57. nohup sh ./bin/mqbroker -c ./conf/custom/broker-a-s1.properties -n "rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876" &
  59.   jps | grep BrokerStartup
  60.   netstat -tunlp | grep 10911
  62. # 启动 broker slave2
  63. nohup sh ./bin/mqbroker -c ./conf/custom/broker-a-s2.properties -n "rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876" &
  65.   jps | grep BrokerStartup
  66.   netstat -tunlp | grep 10911
  68. # 关闭 broker
  69. ./bin/mqshutdown broker
  72. # -------------------- 第 b 组, 1 主 2 从 --------------------
  74. # 主节点 broker-b.properties
  75. brokerClusterName=DefaultCluster
  76. brokerName=broker-b
  77. brokerId=0
  78. deleteWhen=04
  79. fileReservedTime=48
  80. brokerRole=SYNC_MASTER
  81. flushDiskType=ASYNC_FLUSH
  83. # 从节点 broker-b-s1.properties
  84. brokerClusterName=DefaultCluster
  85. brokerName=broker-b
  86. brokerId=1
  87. deleteWhen=04
  88. fileReservedTime=48
  89. brokerRole=SLAVE
  90. flushDiskType=ASYNC_FLUSH
  92. # 从节点 broker-b-s2.properties
  93. brokerClusterName=DefaultCluster
  94. brokerName=broker-b
  95. brokerId=2
  96. deleteWhen=04
  97. fileReservedTime=48
  98. brokerRole=SLAVE
  99. flushDiskType=ASYNC_FLUSH
  101. # 启动 broker master
  102. nohup sh ./bin/mqbroker -c ./conf/custom/broker-b.properties -n "rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876" &
  104.   jps | grep BrokerStartup
  105.   netstat -tunlp | grep 10911
  107. # 启动 broker slave1
  108. nohup sh ./bin/mqbroker -c ./conf/custom/broker-b-s1.properties -n "rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876" &
  110.   jps | grep BrokerStartup
  111.   netstat -tunlp | grep 10911
  113. # 启动 broker slave2
  114. nohup sh ./bin/mqbroker -c ./conf/custom/broker-b-s2.properties -n "rocketmq-nameserver1:9876;rocketmq-nameserver2:9876;rocketmq-nameserver3:9876" &
  116.   jps | grep BrokerStartup
  117.   netstat -tunlp | grep 10911
  119. mqadmin clusterlist

工具

  • tools.sh
  1. # 测试集群
  2. export NAMESRV_ADDR=rocketmq-nameserver1:9876
  4. # 生产测试
  5. ./bin/tools.sh org.apache.rocketmq.example.quickstart.Producer
  7. # 消费测试
  8. ./bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer

-