Elasticsearch需要运行在Java 8 及以上java环境一键部署 初步成功（数据录入且在kibana上查看），还有很多待优化的 es全家桶版本都是7.11.1

系统环境 开发电脑

  • 💻 windows
    • kibana

服务器两台

  • 💻linux
    • springboot
    • filebeat
  • 💻linux
    • elasticearch

项目三个

  • springboot

参考

数据流向

软件安装 elasticearch 👉 kibana 👉 项目 👉 filebeat

安装elasticearch

我是本机下的tar然后传到服务器的， 没有使用 wget，也没有使用rmp 解压： tar -zxvf elasticsearch-xxx cd 解压目录 修改配置文件：vim config/elasticsearch.yml

  • network.host: 0.0.0.0
  • http.port: 9200

设置jvm参数，防止服务器性能太垃圾启动不了或坚持不久 vim config/jvm.options （我设置的512m）

  • -Xms1g
  • -Xmx1g elaticsearch 不能使用root运行所以需要新增用户> - adduser elastic (注意使用其他名字防止被不法分子使用)
  • passwd elastic (注意使用其他密码防止被不法分子使用)
    • 输入密码
  • chmod -R 777 es的解压目录（eg: /tn/es/elasticsearch-7.11.1）
  • su elastic
  • ./bin/elasticsearch
    • ./bin/elasticsearch -d （养成好习惯，建议使用 exit 退出终端）
  • 途中有错参考：https://www.cnblogs.com/hellxz/p/11057234.html
  • 查看所有索引：http://127.0.0.1:9200/_cat/indices
    • 我没安装 elasticsearch-head 或 cerebro

  • 测试 ： curl 127.0.0.1:9200

    1. {
    2. "name" : "node-1",
    3. "cluster_name" : "elasticsearch",
    4. "cluster_uuid" : "yRT36f5DQhmaLgIgj16gqQ",
    5. "version" : {
    6.   "number" : "7.11.1",
    7.   "build_flavor" : "default",
    8.   "build_type" : "tar",
    9.   "build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a",
    10.   "build_date" : "2021-02-15T13:44:09.394032Z",
    11.   "build_snapshot" : false,
    12.   "lucene_version" : "8.7.0",
    13.   "minimum_wire_compatibility_version" : "6.8.0",
    14.   "minimum_index_compatibility_version" : "6.0.0-beta1"
    15. },
    16. "tagline" : "You Know, for Search"
    17. }

  • yaml详情 ```yaml

    ======================== Elasticsearch Configuration =========================

    #

    NOTE: Elasticsearch comes with reasonable defaults for most settings.

    Before you set out to tweak and tune the configuration, make sure you

    understand what are you trying to accomplish and the consequences.

    #

    The primary way of configuring a node is via this file. This template lists

    the most important settings you may want to configure for a production cluster.

    #

    Please consult the documentation for further information on configuration options:

    https://www.elastic.co/guide/en/elasticsearch/reference/index.html

    #

    ————————————————— Cluster —————————————————-

    #

    Use a descriptive name for your cluster:

    #

    cluster.name: my-application

    #

    —————————————————— Node ——————————————————

    #

    Use a descriptive name for the node:

    # node.name: node-1 #

    Add custom attributes to the node:

    #

    node.attr.rack: r1

    #

    —————————————————- Paths ——————————————————

    #

    Path to directory where to store the data (separate multiple locations by comma):

    #

    path.data: /path/to/data

    #

    Path to log files:

    #

    path.logs: /path/to/logs

    #

    —————————————————- Memory —————————————————-

    #

    Lock the memory on startup:

    #

    bootstrap.memory_lock: true

    #

    Make sure that the heap size is set to about half the memory available

    on the system and that the owner of the process is allowed to use this

    limit.

    #

    Elasticsearch performs poorly when the system is swapping the memory.

    #

    ————————————————— Network —————————————————-

    #

    Set the bind address to a specific IP (IPv4 or IPv6):

    # network.host: 0.0.0.0 #

    Set a custom port for HTTP:

    # http.port: 9200 #

    For more information, consult the network module documentation.

    #

    ————————————————- Discovery —————————————————

    #

    Pass an initial list of hosts to perform discovery when this node is started:

    The default list of hosts is [“127.0.0.1”, “[::1]”]

    #

    discovery.seed_hosts: [“host1”, “host2”]

    #

    Bootstrap the cluster using an initial set of master-eligible nodes:

    # cluster.initial_master_nodes: [“node-1”] #

    For more information, consult the discovery and cluster formation module documentation.

    #

    ————————————————— Gateway —————————————————-

    #

    Block initial recovery after a full cluster restart until N nodes are started:

    #

    gateway.recover_after_nodes: 3

    #

    For more information, consult the gateway module documentation.

    #

    ————————————————— Various —————————————————-

    #

    Require explicit names when deleting indices:

    #

    action.destructive_requires_name: true

  1. <a name="umeg6"></a>
  2. # 安装 kibana
  3. > 由于数据流是单向的我就安装在了本机上 （windows 10）
  4. > 右击解压
  5. > 修改配置文件 
  6. > - kibana-7.11.2-windows-x86_64\config\kibana.yml
  7. > 
  8. 启动bat
  9. > - kibana-7.11.2-windows-x86_64\bin\kibana.bat
  10. > 
  11. 访问：127.0.0.1:5601
  12. > - 我没设置安全所以不需要登录<br />
  14. - 配置文件详情（kibana.yml）
  15. > # 设置es地址
  16. > elasticsearch.hosts: ["http://es.tannn.cn:80"]
  17. > # 看上面英文介绍把
  18. > kibana.index: ".kibana"
  19. > # 设置中文
  20. > i18n.locale: "zh-CN"
  21. > # 默认端口
  22. > server.port: 5601
  24. ```yaml
  25. # Kibana is served by a back end server. This setting specifies the port to use.
  26. server.port: 5601
  28. # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
  29. # The default is 'localhost', which usually means remote machines will not be able to connect.
  30. # To allow connections from remote users, set this parameter to a non-loopback address.
  31. server.host: "localhost"
  33. # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
  34. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
  35. # from requests it receives, and to prevent a deprecation warning at startup.
  36. # This setting cannot end in a slash.
  37. #server.basePath: ""
  39. # Specifies whether Kibana should rewrite requests that are prefixed with
  40. # `server.basePath` or require that they are rewritten by your reverse proxy.
  41. # This setting was effectively always `false` before Kibana 6.3 and will
  42. # default to `true` starting in Kibana 7.0.
  43. #server.rewriteBasePath: false
  45. # Specifies the public URL at which Kibana is available for end users. If
  46. # `server.basePath` is configured this URL should end with the same basePath.
  47. #server.publicBaseUrl: ""
  49. # The maximum payload size in bytes for incoming server requests.
  50. #server.maxPayloadBytes: 1048576
  52. # The Kibana server's name.  This is used for display purposes.
  53. #server.name: "your-hostname"
  55. # The URLs of the Elasticsearch instances to use for all your queries.
  56. # 设置es地址
  57. elasticsearch.hosts: ["http://es.tannn.cn:80"]
  59. # Kibana uses an index in Elasticsearch to store saved searches, visualizations and
  60. # dashboards. Kibana creates a new index if the index doesn't already exist.
  61. # 看上面英文介绍把
  62. kibana.index: ".kibana"
  64. # The default application to load.
  65. #kibana.defaultAppId: "home"
  67. # If your Elasticsearch is protected with basic authentication, these settings provide
  68. # the username and password that the Kibana server uses to perform maintenance on the Kibana
  69. # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
  70. # is proxied through the Kibana server.
  71. #elasticsearch.username: "kibana_system"
  72. #elasticsearch.password: "pass"
  74. # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
  75. # These settings enable SSL for outgoing requests from the Kibana server to the browser.
  76. #server.ssl.enabled: false
  77. #server.ssl.certificate: /path/to/your/server.crt
  78. #server.ssl.key: /path/to/your/server.key
  80. # Optional settings that provide the paths to the PEM-format SSL certificate and key files.
  81. # These files are used to verify the identity of Kibana to Elasticsearch and are required when
  82. # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
  83. #elasticsearch.ssl.certificate: /path/to/your/client.crt
  84. #elasticsearch.ssl.key: /path/to/your/client.key
  86. # Optional setting that enables you to specify a path to the PEM file for the certificate
  87. # authority for your Elasticsearch instance.
  88. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
  90. # To disregard the validity of SSL certificates, change this setting's value to 'none'.
  91. #elasticsearch.ssl.verificationMode: full
  93. # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
  94. # the elasticsearch.requestTimeout setting.
  95. #elasticsearch.pingTimeout: 1500
  97. # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
  98. # must be a positive integer.
  99. #elasticsearch.requestTimeout: 30000
  101. # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
  102. # headers, set this value to [] (an empty list).
  103. #elasticsearch.requestHeadersWhitelist: [ authorization ]
  105. # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
  106. # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
  107. #elasticsearch.customHeaders: {}
  109. # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
  110. #elasticsearch.shardTimeout: 30000
  112. # Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
  113. #elasticsearch.logQueries: false
  115. # Specifies the path where Kibana creates the process ID file.
  116. #pid.file: /run/kibana/kibana.pid
  118. # Enables you to specify a file where Kibana stores log output.
  119. #logging.dest: stdout
  121. # Set the value of this setting to true to suppress all logging output.
  122. #logging.silent: false
  124. # Set the value of this setting to true to suppress all logging output other than error messages.
  125. #logging.quiet: false
  127. # Set the value of this setting to true to log all events, including system usage information
  128. # and all requests.
  129. #logging.verbose: false
  131. # Set the interval in milliseconds to sample system and process performance
  132. # metrics. Minimum is 100ms. Defaults to 5000.
  133. #ops.interval: 5000
  135. # Specifies locale to be used for all localizable strings, dates and number formats.
  136. # Supported languages are the following: English - en , by default , Chinese - zh-CN .
  137. # 设置中文
  138. i18n.locale: "zh-CN"

安装 filebeat

Service方式启动 我是本机下的tar然后传到服务器的， 没有使用 wget，也没有使用rmp 解压： tar -zxvf filebeat-xxx cd 解压目录 修改配置文件：vim filebeat.yml 启动：

  • cd （注意yml路径,我实在同目录下的）
  • nohup ./filebeat -e -c filebeat.yml &
    • nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &
  1. filebeat.inputs:
  2.  - type: log
  3.    paths:
  4.     - /doc/program/smart/logs/smart-appointment/*/*.log
  5.    multiline.pattern: '^\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}'
  6.    multiline.negate: true
  7.    multiline.match: after
  8.    fields:
  9.     index: 'smart-appointment'
  10.  - type: log
  11.    paths:
  12.     - /doc/program/smart/logs/smart-user/*/*.log
  13.    multiline.pattern: '^\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}'
  14.    multiline.negate: true
  15.    multiline.match: after
  16.    fields:
  17.     index: 'smart-user'   
  18.  - type: log
  19.    paths:
  20.     - /doc/program/smart/logs/smart-solr/*/*.log
  21.    multiline.pattern: '^\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}'
  22.    multiline.negate: true
  23.    multiline.match: after
  24.    fields:
  25.     index: 'smart-solr'  
  26. setup.template.name: "my-log"
  27. setup.template.pattern: "my-log-*"    
  28. output.elasticsearch:
  29.   hosts: ["es.tannn.cn:80"]
  30.   indices:
  31.     - index: "smart-appointment-%{+yyyy.MM.dd}"
  32.       when.contains:
  33.         fields:
  34.           index: "smart-appointment"
  35.     - index: "smart-user-%{+yyyy.MM.dd}"
  36.       when.contains:
  37.         fields:
  38.           index: "smart-user"
  39.     - index: "smart-solr-%{+yyyy.MM.dd}"
  40.       when.contains:
  41.         fields:
  42.           index: "smart-solr"

使用kibana 查看数据

http://localhost:5601

image.pngimage.pngimage.png

image.pngimage.png

  • 数据好像不自动刷新需要手动点击右上角刷新按钮

image.png

注意

  • 🧨 我都没设置自启

  • 🧨 数据在外网上需要加上安全验证 - 我测试是没有加任何安全验证的

    ps -ef | grep elastic kill -9 pid ./bin/elasticsearch -d

    • es设置密码 参考
      • 一定要先重启，在输入设置密码的命令 ./bin/elasticsearch-setup-passwords interactive ```

        进入es安装目录下的config目录

        vim elasticsearch.yml

配置X-Pack

http.cors.enabled: true http.cors.allow-origin: “*” http.cors.allow-headers: Authorization xpack.security.enabled: true xpack.security.transport.ssl.enabled: true

  2.       - kibana 设置es的登录密码 （一般用elastic 账户登录es）
  3. ```yaml
  5. # If your Elasticsearch is protected with basic authentication, these settings provide
  6. # the username and password that the Kibana server uses to perform maintenance on the Kibana
  7. # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
  8. # is proxied through the Kibana server.
  9. #elasticsearch.username: "kibana_system" #elastic
  10. #elasticsearch.password: "pass"
  1.   - filebeat 设置es登录密码
  1. output.elasticsearch:
  2.   username: elastic
  3.   password: elastic
  4.   hosts: ["es.tannn.cn:80"]
  5.   indices:
  6.     - index: "smart-appointment-%{+yyyy.MM.dd}"
  7.       when.contains:
  8.         fields:
  9.           index: "smart-appointment"
  10.     - index: "smart-user-%{+yyyy.MM.dd}"
  11.       when.contains:
  12.         fields:
  13.           index: "smart-user"
  14.     - index: "smart-solr-%{+yyyy.MM.dd}"
  15.       when.contains:
  16.         fields:
  17.           index: "smart-solr"