jetlinks以及所需中间件的集群部署
::: tip 注意：
仅专业版以上支持。
本文档以172.16.4.13、172.16.4.12、172.16.4.11三台服务器来搭建集群。
:::

集群架构图

集群架构图

redis集群安装配置

这里准备在一台服务器上搭建一个三主三从的redis集群。

  1. 安装redis
    先下载redis,本文档以redis5.0.5为例。
    下载完成后进行解压、编译、安装。

::: tip 注意：
redis是由C语言编写的，它的运行需要C环境，所以编译前需安装 gcc。
:::

  1. 新建一个cluster文件夹，用来存放集群节点目录。
    分别在172.16.4.11上创建7000、7003、7001、7004、7002、7005六文件夹，，这些节点分别使用7000、7003、7001、7004、7002、7005端口，以7000节点为例配置如下： ``` port 7000 bind 172.16.4.11 cluster-enabled yes cluster-config-file nodes_7000.conf cluster-node-timeout 5000 appendonly no daemonize yes 
  1. 其他节点只需修改端口和文件名，依次按此进行配置即可，配置完成后启动节点。<br />
  2. ::: tip 注意：<br />
  3. 建议appendonly在从节点开启，主节点关闭。<br />
  4. :::
  6. 2. 主从节点分配

redis-cli —cluster create 172.16.4.11:7000 172.16.4.11:7001 172.16.4.11:7002 172.16.4.11:7003 172.16.4.11:7004 172.16.4.11:7005 —cluster -replicas 1

  1. ::: tip 注意：<br />
  2. 　其中-replicas 1表示每个主节点1个从节点<br />
  3. :::
  5. 3. 查看集群节点信息

ops@jetlinks-server-3:~/data/redis-5.0.5$ redis-cli -h 172.16.4.11 -p 7000 cluster nodes 562558e5afa0575d1059c47b9531a37cd75a9190 172.16.4.11:7003@17003 slave 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 0 1594368796612 4 connected bd974dbdd2f8447b27375c832c4c9c99328f4487 172.16.4.11:7005@17005 slave 3ab893f4cdcfaa52254a8cece2b54b561de29990 0 1594368797514 6 connected 9174a3ebcdbda174ec9189fbae0e38d9bbeeff5f 172.16.4.11:7002@17002 master - 0 1594368797615 3 connected 10923-16383 3ab893f4cdcfaa52254a8cece2b54b561de29990 172.16.4.11:7001@17001 master - 0 1594368797000 2 connected 5461-10922 238585a079196b0ab15ebb47ac681d42c083cdaf 172.16.4.11:7000@17000 myself,master - 0 1594368796000 1 connected 0-5460 d1ae6c623694c5f1fafc64d7abbeac4a9926ff49 172.16.4.11:7004@17004 slave 238585a079196b0ab15ebb47ac681d42c083cdaf 0 1594368796000 5 connected

  1. redis集群搭建请参考[redis官方文档](https://redis.io/topics/cluster-tutorial)
  2. ## elasticsearch集群安装配置
  4. 1. 分别在三台服务器上安装elasticsearch<br />
  5. 前往官网[下载](https://www.elastic.co/cn/downloads/elasticsearch)。<br />
  6. 下载完成后安装es。
  7. 1. 配置elasticsearch.yml文件

ops@jetlinks-server-3:~/elasticsearch/config$ sudo vi elasticsearch.yml

======================== Elasticsearch Configuration =========================

#

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

#

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

#

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

#

————————————————— Cluster —————————————————-

#

Use a descriptive name for your cluster:

# cluster.name: es-cluster #

—————————————————— Node ——————————————————

#

Use a descriptive name for the node:

# node.name: node-3 #

Add custom attributes to the node:

#

node.attr.rack: r1

#

—————————————————- Paths ——————————————————

#

Path to directory where to store the data (separate multiple locations by comma):

#

path.data: /path/to/data

#

Path to log files:

#

path.logs: /path/to/logs

#

—————————————————- Memory —————————————————-

#

Lock the memory on startup:

#

bootstrap.memory_lock: true

#

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

#

Elasticsearch performs poorly when the system is swapping the memory.

#

————————————————— Network —————————————————-

#

Set the bind address to a specific IP (IPv4 or IPv6):

# network.host: 172.16.4.11

transport.tcp.port: 9300 #

Set a custom port for HTTP:

# http.port: 9200 #

For more information, consult the network module documentation.

#

————————————————- Discovery —————————————————

#

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

# discovery.zen.ping.unicast.hosts: [“172.16.4.13:9300”, “172.16.4.12:19300”,”172.16.4.11:9300”] #

Prevent the “split brain” by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

# discovery.zen.minimum_master_nodes: 1 #

For more information, consult the zen discovery module documentation.

#

————————————————— Gateway —————————————————-

#

Block initial recovery after a full cluster restart until N nodes are started:

#

gateway.recover_after_nodes: 3

#

For more information, consult the gateway module documentation.

#

————————————————— Various —————————————————-

#

Require explicit names when deleting indices:

#

action.destructive_requires_name: true

  1. 三台服务器不同的配置：

172.16.4.11

node.name: node-3
network.host:172.16.4.11 

  1. ```
  2. #172.16.4.12
  3. node.name: node-2  
  4. network.host:172.16.4.12
  1. #172.16.4.13
  2. node.name: node-1  
  3. network.host:172.16.4.13

3 在三台服务器上分别启动es

  1. ./bin/elasticsearch

数据库启动

本文档案例使用docker启动postgresql，可参考docker-compose.yml文件中的postgresql配置。

jetlinks集群安装配置

1.修改application.yml,并将jetlinks打成jar包
application.yml文件：

  1. server:
  2.   port: 8844
  3.   max-http-header-size: 64KB
  4. spring:
  5.   profiles:
  6.     active: dev
  7.   application:
  8.     name: jetlinks-platform
  9.   jackson:
  10.     date-format: yyyy-MM-dd HH:mm:ss
  11.     time-zone: Asia/Shanghai
  12.     serialization:
  13.       WRITE_DATES_AS_TIMESTAMPS: true
  14.     default-property-inclusion: non_null
  15.   resources:
  16.     static-locations: file:./static,classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/, classpath:/public/
  17.   redis:   
  18.     # redis集群配置
  19.     cluster: 
  20.       nodes:
  21.         - 172.16.4.11:7000
  22.         - 172.16.4.11:7001
  23.         - 172.16.4.11:7002
  24.         - 172.16.4.11:7003
  25.         - 172.16.4.11:7004
  26.         - 172.16.4.11:7005
  27.       max-redirects: 3
  28.     lettuce:
  29.       pool:
  30.         max-active: 1024
  31.     timeout: 20s
  32.     serializer: jdk # 设置fst时,redis key使用string序列化,value使用 fst序列化.
  33.     database: 0
  34.   r2dbc:
  35.     url: r2dbc:postgresql://172.16.4.12:5432/jetlinks    #数据库地址 本文档数据库为docker启动
  36.     username: postgres
  37.     password: jetlinks
  38.     pool:
  39.       max-size: 32
  40.   codec:
  41.     max-in-memory-size: 50MB
  42.   rabbitmq: # 开启了device.message.writer.rabbitmq.enabled=true时生效
  43.     host: localhost
  44.     port: 5672
  45.     username: admin
  46.     password: jetlinks
  47.   kafka:    # 开启了device.message.writer.kafka.enabled=true时生效
  48.     consumer:
  49.       client-id: ${spring.application.name}-consumer:${server.port}
  50.       group-id: ${spring.application.name}
  51.       max-poll-records: 1000
  52.     producer:
  53.       client-id: ${spring.application.name}-producer:${server.port}
  54.       acks: 1
  55.       retries: 3
  56.     bootstrap-servers: ["127.0.0.1:9092"]
  57.   reactor:
  58.     debug-agent:
  59.       enabled: true # 开启调试代理,在打印异常时将会生成调用踪栈信息
  60. easyorm:
  61.   default-schema: public # 数据库默认的schema
  62.   dialect: postgres #数据库方言
  63. elasticsearch:
  64.   client:
  65.     max-conn-total: 128
  66.     connect-timeout: 5000
  67.     socket-timeout: 5000
  68.     connection-request-timeout: 8000
  69.     hosts:                      # es集群配置
  70.       - http://172.16.4.13:9200
  71.       - http://172.16.4.12:19200
  72.       - http://172.16.4.11:9200
  73.   index:
  74.     default-strategy: time-by-month #默认es的索引按月进行分表
  75.     settings:
  76.       number-of-shards: 1 # es 分片数量
  77.       number-of-replicas: 0 # 副本数量
  78. device:
  79.   message:
  80.     writer:
  81.       time-series:
  82.         enabled: true # 直接写出设备消息数据到时序数据库
  83.       kafka:
  84.         enabled: false  # 推送设备消息到kafka
  85.         consumer: true  # 从kafka订阅消息并写入到时序数据库
  86.         topic-name: device.message
  87.       rabbitmq:
  88.         enabled: false  # 推送设备消息到rabbitMQ
  89.         consumer: true  # 从rabbitMQ订阅消息并写入到时序数据库
  90.         thread-size: 4 # 消费线程数
  91.         auto-ack: true  # 自定应答,为true可能导致数据丢失,但是性能最高。
  92.         topic-name: device.message  # exchange名称
  93.         ignore-message-types:
  94.         include-message-types:
  95. hsweb:
  96.   cors:
  97.     enable: true
  98.     configs:
  99.       - path: /**
  100.         allowed-headers: "*"
  101.         allowed-methods: ["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"]
  102.         allowed-origins: ["*"]
  103.         allow-credentials: true
  104.         max-age: 1800
  105.   dict:
  106.     enum-packages: org.jetlinks
  107.   file:
  108.     upload:
  109.       static-file-path: ./static/upload
  110.       static-location: http://demo2.jetlinks.org:8080/upload   # 此地址为服务器域名，将通过nginx代理到固定服务器上
  111.   webflux:
  112.     response-wrapper: enabled #开启响应包装器(将返回值包装为ResponseMessage)
  113.   #  auth:   #默认的用户配置
  114.   #    users:
  115.   #      admin:
  116.   #        username: admin
  117.   #        password: admin
  118.   #        name: 超级管理员
  119.   authorize:
  120.     auto-parse: true
  121.   cache:
  122.     type: redis
  123.     redis:
  124.       local-cache-type: guava
  125. jetlinks:
  126.   server-id: ${spring.application.name}:${server.port} #设备服务网关服务ID,不同服务请设置不同的ID
  127.   logging:
  128.     system:
  129.       context:
  130.         server: ${spring.application.name}
  131.   protocol:
  132.     spi:
  133.       enabled: false # 为true时开启自动加载通过依赖引入的协议包
  134. rule:
  135.   engine:
  136.     server-id: ${jetlinks.server-id}
  137.     server-name: ${spring.application.name}
  138. logging:
  139.   level:
  140.     org.jetlinks: info
  141.     rule.engine: info
  142.     org.hswebframework: debug
  143.     org.springframework.transaction: debug
  144.     org.springframework.data.r2dbc.connectionfactory: debug
  145.     io.micrometer: warn
  146.     org.hswebframework.expands: error
  147.     system: warn
  148.     org.jetlinks.rule.engine: info
  149.     org.jetlinks.gateway: info
  150.     org.springframework: warn
  151.     org.apache.kafka: warn
  152.     org.jetlinks.pro.device.message.writer: debug
  153.   #    org.elasticsearch: error
  154.   config: classpath:logback-spring.xml
  155. vertx:
  156.   max-event-loop-execute-time-unit: seconds
  157.   max-event-loop-execute-time: 30
  158.   max-worker-execute-time-unit: seconds
  159.   max-worker-execute-time: 30
  160.   prefer-native-transport: true
  162. micrometer:
  163.   time-series:
  164.     tags:
  165.       server: ${spring.application.name}
  166.     metrics:
  167.       default:
  168.         step: 30s

::: tip 注意：
在多台服务器上启动jetlinks，配置中的jetlinks.server-id必须互不相同。
:::
在项目根目录执行：

  1. mvn clean package -DskipTests
  1. 分别在三台服务器上启动jetlinks
    文档以脚本方式启动，脚本如下： ```

    !/bin/bash

    nohup java -jar -Dspring.application.name=jetlinks-cluster-test-3 jetlinks-standalone.jar >jetlinks-pro.log 2>&1 &
  1. ::: tip 注意：<br />
  2. 三台服务启动的jetlinks.server-id应该互不相同，在application.yml中jetlinks.server-id引用了spring.application.name，所以此处的name应不相同。<br />
  3. :::
  4. ## nginx配置
  5. 通过nginx来代理前后端。
  7. 1. 安装nginx<br />
  8. 前往官网[下载](https://nginx.org/download/)<br />
  9. 下载完成后解压、编译、安装。
  10. 1. 修改配置文件
  12. 配置文件如下：

upstream iotserver { server 172.16.4.11:8844; server 172.16.4.12:8844; server 172.16.4.13:8844; }

upstream webserver { server 172.16.4.11:9000; }

upstream fileserver { server 172.16.4.11:8844; #此处指定文件上传到该服务器上 }

server {

listen 8080; server_name demo2.jetlinks.cn;

location ^~/upload/ { proxy_pass http://fileserver; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; }

location ^~/jetlinks/file/static { proxy_pass http://fileserver/file/static; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_send_timeout 30m; proxy_read_timeout 30m; client_max_body_size 100m; }

location ^~/jetlinks/ { proxy_pass http://iotserver/; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection “upgrade”; proxy_connect_timeout 1; proxy_buffering off; chunked_transfer_encoding off; proxy_cache off; proxy_send_timeout 30m; proxy_read_timeout 30m; client_max_body_size 100m; }

location / { proxy_pass http://webserver/; } }

  2. 3. 启动

systemctl status nginx.service

  1. ## tcp负载均衡配置方式
  2. 用于代理设备连接。此处使用nginx作为演示,也可以使用LVS,haProxy等方式。<br />配置文件：

load_module /usr/lib/nginx/modules/ngx_stream_module.so; user root; worker_processes 1;

error_log /etc/nginx/log/error.log warn; pid /var/run/nginx.pid;

events { worker_connections 1024; }

stream {

upstream tcp-test { hash $remote_addr consistent; server 172.16.4.13:1889 max_fails=3 fail_timeout=10s;

  1.    #server 172.16.4.12:1889 max_fails=3 fail_timeout=10s;
  2.    server 172.16.4.11:1889 max_fails=3 fail_timeout=10s;

}

server {

  1.    listen 1884;
  3.    proxy_pass tcp-test;
  5.    proxy_connect_timeout 30s;
  7.    proxy_timeout 30s;

} }

``` ::: tip 注意：
此配置监听的MQTT的1884端口，代理到服务器上的1889。如需代理自定义端口需保持代理端口与jetlinks平台中网络组件开启的端口一致。
如此处平台中MQTT服务网络组件设置的端口应为1889。
:::