14 Filebeat Logstash新架构

Filebeat和Logstash说明
1. Filebeat：轻量级，但不支持正则、不能移除字段等
2. Logstash：比较重，但支持正则、支持移除字段等

搭建架构演示
3. Logstash -> Elasticsearch -> Kibana
4. Filebeat -> Elasticsearch -> Kibana
5. Filebeat -> Logstash -> Elasticsearch -> Kibana

部署服务介绍
6. 192.168.10.11 Kibana ES
7. 192.168.10.12 Logstash Filebeat

Filebeat配置发往Logstash
[root@server12 ~]# vim /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
tail_files: true
backoff: “1s”
paths:
- /usr/local/nginx/logs/access.log

output:
logstash:
hosts: [“192.168.10.12:5044”]
[root@server12 ~]# nohup /usr/local/filebeat-6.6.0/filebeat -e -c /usr/local/filebeat-6.6.0/filebeat.yml &

Logstash配置监听在5044端口，接收Filebeat发送过来的日志
[root@server12 ~]# vim /usr/local/logstash-6.6.0/config/logstash.conf
input {
beats {
host => ‘0.0.0.0’
port => 5044
}
}
[root@server12 ~]# kill -1 ps aux | grep logstash | awk '{print $2}' &>/dev/null
#以下操作看情况而定
[root@server12 ~]# nohup /usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf &
[root@server12 ~]# /usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf
[2022-03-13T03:08:50,732][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>”0.0.0.0:5044”}
[2022-03-13T03:08:50,792][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>”main”, :thread=>”#“}
[2022-03-13T03:08:50,999][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-03-13T03:08:51,178][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2022-03-13T03:08:51,919][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2022-03-13T03:08:59,592][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>[“index”, {:_id=>nil, :_index=>”logstash-2022.03.13”, :_type=>”doc”, :routing=>nil}, #], :response=>{“index”=>{“_index”=>”logstash-2022.03.13”, “_type”=>”doc”, “_id”=>”7K0bgn8BrMOq2Sw_3nOg”, “status”=>400, “error”=>{“type”=>”mapper_parsing_exception”, “reason”=>”failed to parse field [host] of type [text]”, “caused_by”=>{“type”=>”illegal_state_exception”, “reason”=>”Can’t get text on a START_OBJECT at 1:298”}}}}}
[root@server12 ~]# netstat -pantul | grep 5044
tcp 0 0 192.168.10.12:50100 192.168.10.12:5044 ESTABLISHED 8674/filebeat
tcp6 0 0 :::5044 ::: LISTEN 8833/java
tcp6 0 0 192.168.10.12:5044 192.168.10.12:50100 ESTABLISHED 8833/java

再次测试
先执行
DELETE /logstash-2022.03.13

再刷新nginx页面

查看logstash状态


Kibana上查看数据
8. GET /xxx/_search?q=

9. 创建索引查看数据

Logstash上移除不必要的字段
10. Filebeat发过来的无用字段比较多
11. remove_field => [“message”,”@version”,”path”,”beat”,”input”,”log”,”offset”,”prospector”,”source”,”tags”]
[root@server12 ~]# vim /usr/local/logstash-6.6.0/config/logstash.conf
[root@server12 ~]# /usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf

DELETE /logstash-2022.03.13

刷新nginx页面

GET /_cat/indices?v



由以上结果可知，很多字段已被移除！

Filebeat批量部署比Logstash要方便得多
12. Logstash监听在内网
13. Filebeat发送给内网的Logstash

新架构
Filebeat(多台)
Filebeat(多台) -> Logstash(正则) -> Elasticsearch(入库) -> Kibana展现
Filebeat(多台)