16 Filebeat采集多个日志

采集多个日志
1. 收集单个Nginx日志
2. 如果有采集多个日志的需求

Filebeat采集多个日志配置

[root@server12 ~]# cat /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
tail_files: true
backoff: “1s”
paths:
- /usr/local/nginx/logs/access.json.log

output:
logstash:
hosts: [“192.168.10.12:5044”]
[root@server12 ~]# vim /usr/local/filebeat-6.6.0/filebeat.yml
filebeat.inputs:
- type: log
tail_files: true
backoff: “1s”
paths:
- /usr/local/nginx/logs/access.json.log
fields:
type: access
fields_under_root: true
- type: log
tail_files: true
backoff: “1s”
paths:
- /var/log/secure
fields:
type: secure
fields_under_root: true
output:
logstash:
hosts: [“192.168.10.12:5044”]
[root@server12 ~]# ps aux | grep filebeat
root 8974 0.0 0.9 434860 17524 pts/0 Sl Mar13 0:09 /usr/local/filebeat-6.6.0/filebeat -e -c /usr/local/filebeat-6.6.0/filebeat.yml
root 9780 0.0 0.0 112652 956 pts/0 R+ 19:59 0:00 grep —color=auto filebeat
[root@server12 ~]# kill -1 8974
DELETE /logstash-2022.03.13（注意日期变化）


刷新nginx页面



使/var/log/secure内容发生变化
[root@server12 ~]# ssh 192.168.10.12
The authenticity of host ‘192.168.10.12 (192.168.10.12)’ can’t be established.
ECDSA key fingerprint is e1:44:0f:74:12:4b:c2:bf:d1:0f:71:d9:4d:46:48:8d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘192.168.10.12’ (ECDSA) to the list of known hosts.
root@192.168.10.12’s password:
Last login: Sat Mar 12 22:32:37 2022 from 192.168.10.1
[root@server12 ~]# exit
logout
Connection to 192.168.10.12 closed.
[root@server12 ~]# cat /var/log/secure
Mar 14 09:38:05 server12 sshd[11620]: Accepted password for root from 192.168.10.12 port 51424 ssh2
Mar 14 09:38:05 server12 sshd[11620]: pam_unix(sshd:session): session opened for user root by (uid=0)
Mar 14 09:38:09 server12 sshd[11620]: Received disconnect from 192.168.10.12: 11: disconnected by user
Mar 14 09:38:09 server12 sshd[11620]: pam_unix(sshd:session): session closed for user root

在Kibana查看数据


Logstash如何判断两个日志
3. Filebeat加入一字段用来区别
4. Logstash使用区别字段来区分

Logstash通过type字段进行判断
[root@server12 ~]# cat /usr/local/logstash-6.6.0/config/logstash.conf
input {
beats {
host => ‘0.0.0.0’
port => 5044
}
}
filter {
json {
source => “message”
remove_field => [“message”,”@version”,”path”,”beat”,”input”,”log”,”offset”,”prospector”,”source”,”tags”]
}
}
output {
elasticsearch {
hosts => [“http://192.168.10.11:9200“]
}
}



[root@server12 ~]# vim /usr/local/logstash-6.6.0/config/logstash.conf
input {
beats {
host => ‘0.0.0.0’
port => 5044
}
}

filter {
if [type] == “access” {
json {
source => “message”
remove_field => [“message”,”@version”,”path”,”beat”,”input”,”log”,”offset”,”prospector”,”source”,”tags”]
}
}
}

output{
if [type] == “access” {
elasticsearch {
hosts => [“http://192.168.10.11:9200“]
index => “access-%{+YYYY.MM.dd}”
}
}
else if [type] == “secure” {
elasticsearch {
hosts => [“http://192.168.10.11:9200“]
index => “secure-%{+YYYY.MM.dd}”
}
}
}
[root@server12 ~]# /usr/local/logstash-6.6.0/bin/logstash -f /usr/local/logstash-6.6.0/config/logstash.conf

DELETE /logstash-2022.03.14


刷新nginx页面

注意：如果看不懂access和secure数据，需要多次重启filebeat和logstash后再观察！



网页上建立索引
5. access索引

6. secure索引