安装一些依赖

  1. yum -y install lrzsz vim curl wget java ntpdate && ntpdate -u cn.pool.ntp.org

这里java环境是非常重要的，如果不通过yum安装，源码方式也是可以的。但要注意配置好环境变量

安装elasticsearch

从这里下载ElasticSearch

比如可以通过curl下载

  1. # curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.12.0-linux-x86_64.tar.gz
  2.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  3.                                  Dload  Upload   Total   Spent    Left  Speed
  1. # tar zxvf /opt/elasticsearch-7.12.0-linux-x86_64.tar.gz
  • 增加elasticSearch用户

必须创建一个非root用户来运行ElasticSearch(ElasticSearch5及以上版本，基于安全考虑，强制规定不能以root身份运行。)
如果你使用root用户来启动ElasticSearch，则会有如下错误信息：

  1. # cd elasticsearch-7.12.0/
  2. [root@VM-0-14-centos elasticsearch-7.12.0]# ./bin/elasticsearch
  3. [2021-04-05T21:36:46,510][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [VM-0-14-centos] uncaught exception in thread [main]
  4. org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
  5.         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.0.jar:7.12.0]
  6.         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.0.jar:7.12.0]
  7.         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.0.jar:7.12.0]
  8.         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.0.jar:7.12.0]
  9.         at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.0.jar:7.12.0]
  10.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.0.jar:7.12.0]
  11.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.0.jar:7.12.0]
  12. Caused by: java.lang.RuntimeException: can not run elasticsearch as root
  13.         at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:101) ~[elasticsearch-7.12.0.jar:7.12.0]
  14.         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:168) ~[elasticsearch-7.12.0.jar:7.12.0]
  15.         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.12.0.jar:7.12.0]
  16.         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.0.jar:7.12.0]
  17.         ... 6 more
  18. uncaught exception in thread [main]
  19. java.lang.RuntimeException: can not run elasticsearch as root
  20.         at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:101)
  21.         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:168)
  22.         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397)
  23.         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
  24.         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
  25.         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
  26.         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
  27.         at org.elasticsearch.cli.Command.main(Command.java:79)
  28.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
  29.         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
  30. For complete error details, refer to the log at /opt/elasticsearch-7.12.0/logs/elasticsearch.log
  31. 2021-04-05 13:36:46,979269 UTC [8846] INFO  Main.cc@106 Parent process died - ML controller exiting

所以我们增加一个独立的elasticsearch用户来运行

# 增加elasticsearch用户
[root@VM-0-14-centos elasticsearch-7.12.0]# useradd elasticsearch
[root@VM-0-14-centos elasticsearch-7.12.0]# passwd elasticsearch
Changing password for user elasticsearch.
New password: 
BAD PASSWORD: The password contains the user name in some form
Retype new password: 
passwd: all authentication tokens updated successfully.

# 修改目录权限至新增的elasticsearch用户
[root@VM-0-14-centos elasticsearch-7.12.0]# chown -R elasticsearch /opt/elasticsearch-7.12.0
# 增加data和log存放区，并赋予elasticsearch用户权限
[root@VM-0-14-centos elasticsearch-7.12.0]# mkdir -p /data/es
[root@VM-0-14-centos elasticsearch-7.12.0]# chown -R elasticsearch /data/es
[root@VM-0-14-centos elasticsearch-7.12.0]# mkdir -p /var/log/es
[root@VM-0-14-centos elasticsearch-7.12.0]# chown -R elasticsearch /var/log/es

然后修改上述的data和log路径，vi /opt/elasticsearch-7.12.0/config/elasticsearch.yml

# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/es
#
# Path to log files:
#
path.logs: /var/log/es
  • 修改Linux系统的限制配置

启动服务

su elasticsearch

$ ./bin/elasticsearch -d
[2021-04-05T22:03:38,332][INFO ][o.e.n.Node               ] [VM-0-14-centos] version[7.12.0], pid[13197], build[default/tar/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z], OS[Linux/3.10.0-862.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-04-05T22:03:38,348][INFO ][o.e.n.Node               ] [VM-0-14-centos] JVM home [/opt/elasticsearch-7.12.0/jdk], using bundled JDK [true]
[2021-04-05T22:03:38,348][INFO ][o.e.n.Node               ] [VM-0-14-centos] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-17264135248464897093, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1894m, -Xmx1894m, -XX:MaxDirectMemorySize=993001472, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/opt/elasticsearch-7.12.0, -Des.path.conf=/opt/elasticsearch-7.12.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]



curl localhost:9200

调整一下配置文件：

[root@elk ~]$egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /logs/elasticsearch6
path.logs: /logs/elasticsearch6/log
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["elk-node1"]
discovery.zen.minimum_master_nodes: 1
xpack.security.enabled: false
  • cluster.name：自定义集群名，相同集群内的节点设置相同的集群名
  • node.name：自定义节点名，建议统一采用节点hostname
  • path.data：data存储路径，这里更改成自定义以应对日志的big。
  • path.logs：log存储路径，是为es自己的日志。

  • 注意创建上边两项定义的两个文件目录。否则会启动失败。

    mkdir -p /logs/elasticsearch6/log
cd /logs
chown -R elasticsearch.elasticsearch elasticsearch6/

  • 注意要更改对应目录的权限，否则es启动会报如下错误。

    [root@elk logs]$systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
 Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
 Active: failed (Result: exit-code) since Fri 2018-12-14 15:12:48 CST; 5min ago
   Docs: http://www.elastic.co
Process: 79428 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 79428 (code=exited, status=1/FAILURE)
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,084 main ERROR Null object returned for RollingFile in Appenders.
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,084 main ERROR Unable to locate appender "rolling" for logger config "root"
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,084 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,084 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security....gAuditTrail"
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,084 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,084 main ERROR Unable to locate appender "deprecated_audit_rolling" for logger config "org.elasticsearch.xpac...gAuditTrail"
Dec 14 15:12:48 elk elasticsearch[79428]: 2018-12-14 15:12:48,085 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Dec 14 15:12:48 elk systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 14 15:12:48 elk systemd[1]: Unit elasticsearch.service entered failed state.
Dec 14 15:12:48 elk systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

  • network.host：es监听地址，采用”0.0.0.0”，表示允许所有设备访问。

  • http.port：es监听端口，可不取消注释，默认即此端口。
  • discovery.zen.ping.unicast.hosts：集群节点发现列表，也可采用ip的形式
  • discovery.zen.minimum_master_nodes：如果暂时是单节点部署，可以设置成1
  • xpack.security.enabled：添加这条，这条是配置kibana的安全机制，暂时关闭。

重启es

systemctl restart elasticsearch.service
systemctl status elasticsearch.service

安装logstash

直接yum安装。

yum -y install logstash

配置logstash

[root@elk ~]$egrep -v "^#|^$" /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash

这个地方重要的是第二条配置，同许多应用类似的，这里定义了一个include的目录，以后我们的多个应用实例就可以直接放置在这个目录下了，这里作为了解，后边并不会应用配置好的logstash，原因在后边会说。
然后定义此logstash不再使用系统管理启动，而是以后通过命令行来起对应的logstash实例。 同时创建软链接，从而让系统能够在/usr/share/logstash下读取到相对应的logstash配置信息。

systemctl disable logstash.service
ln -s /etc/logstash /usr/share/logstash/config

安装kibana

从这里下载Kibana

# tar -vxzf kibana-7.12.0-linux-x86_64.tar.gz
  • 使用elasticsearch用户权限 ```java

chown -R elasticsearch /opt/kibana-7.12.0-linux-x86_64

配置Kibana的远程访问

vi /opt/kibana-7.12.0-linux-x86_64/config/kibana.yml

server.host: 0.0.0.0


- **启动**

需要切换至elasticsearch用户
```java
# su elasticsearch
[elasticsearch@VM-0-14-centos opt]$ cd /opt/kibana-7.12.0-linux-x86_64/
[elasticsearch@VM-0-14-centos kibana-7.12.0-linux-x86_64]$ ./bin/kibana
  log   [22:30:22.185] [info][plugins-service] Plugin "osquery" is disabled.
  log   [22:30:22.283] [warning][config][deprecation] Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0."
  log   [22:30:22.482] [info][plugins-system] Setting up [100] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsLegacy,kibanaLegacy,translations,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,observability,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,visualizations,visTypeVislib,visTypeVega,visTypeTimelion,features,licenseManagement,watcher,canvas,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,tileMap,regionMap,visTypeXy,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,maps,lens,reporting,lists,encryptedSavedObjects,dashboardMode,dataEnhanced,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,eventLog,actions,alerts,triggersActionsUi,stackAlerts,ml,securitySolution,case,infra,monitoring,logstash,apm,uptime]
  log   [22:30:22.483] [info][plugins][taskManager] TaskManager is identified by the Kibana UUID: xxxxxx
  ...

配置kibana

[root@elk ~]$egrep -v "^#|^$" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.100.120.82:9200"
kibana.index: ".newkibana"
  • kibana.index：原来默认是”.kibana”，但是新版本的kibana启动之后发现无法访问，访问之后抛出一个异常：kibana server is not ready yet，那么回来把配置更改成”.newkibana”，然后重启kibana，再次访问，即可成功。

另外：配置发现，当我把上边配置写入kibana，然后启动，看状态是正常的，但是访问起来总是会报 Kibana server is not ready yet，这似乎是一个经典的错误，却又让人无从下手解决。经过我的一些测试，获得以小经验。
那就是，此处配置文件，不建议直接把原来配置内容清空，然后添加当前内容的方式，尽管在上边配置elasticsearch以及logstash的时候，都这么做了，两个应用都没有发生什么奇怪的问题，但是这在kibana这里，似乎是不可行的，于是如果已经陷入上边那个报错之中了，那么我的建议是首先把当前kibana卸载，然后重新安装，接着在原有配置文件中，比照着上边的四项配置文件进行更改即可，配置完毕之后，启动kibana，等个两三分钟之后再访问会发现，问题就神奇的消失了。
启动kibana。

systemctl enable kibana.service
systemctl restart kibana
systemctl status  kibana

配置密码

使用基本许可证时，默认情况下禁用Elasticsearch安全功能。由于我测试环境是放在公网上的，所以需要设置下密码访问。相关文档可以参考这里

  1. 停止kibana和elasticsearch服务
  2. 将xpack.security.enabled设置添加到ES_PATH_CONF/elasticsearch.yml文件并将值设置为true
  3. 启动elasticsearch (./bin/elasticsearch -d)
  4. 执行如下密码设置器，./bin/elasticsearch-setup-passwords interactive来设置各个组件的密码
  5. 将elasticsearch.username设置添加到KIB_PATH_CONF/kibana.yml 文件并将值设置给elastic用户： elasticsearch.username: “elastic”
  6. 创建kibana keystore, ./bin/kibana-keystore create
  7. 在kibana keystore 中添加密码 ./bin/kibana-keystore add elasticsearch.password
  8. 重启kibana 服务即可 nohup ./bin/kibana &