Nginx日志说明
    1. message:192.168.10.1 - - [12/Mar/2022:20:48:17 -0500] “GET / HTTP/1.1” 304 0 “-“ “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36” @version:1 host:server12 path:/usr/local/nginx/logs/access.log @timestamp:March 13th 2022, 09:48:18.048 _id:Fq32gH8BrMOq2Sw_R21o _type:doc _index:logstash-2022.03.13 _score: -“

    192.168.10.1 - - [12/Mar/2022:20:47:18 -0500] “GET /jaking HTTP/1.1” 404 571 “-“ “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36”

    2. 访问IP地址
    3. 访问时间
    4. 请求方式(GET/POST)
    5. 请求URL
    6. 状态码
    7. 响应body大小
    8. Referer
    9. User Agent

    Logstash正则提取日志
    10. 需要懂得正则，Logstash支持普通正则和扩展正则
    11. 需要了解Grok，利用Kibana的Grok学习Logstash正则提取日志

    Grok提取Nginx日志
    12. Grok使用(?提取内容)来提取xxx字段
    13. 提取客户端IP： (?[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})
    1.png
    14. 提取时间： [(?[^ ]+ +[0-9]+)]

    message:192.168.10.1 - - [13/Mar/2022:10:02:53 +0800] “GET / HTTP/1.1” 304 0 “-“ “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36” @version:1 host:server12 path:/usr/local/nginx/logs/access.log @timestamp:March 13th 2022, 10:02:54.358 _id:Jq0DgX8BrMOq2Sw_pm1- _type:doc _index:logstash-2022.03.13 _score: -
    2.png
    3.png


    Grok提取Nginx日志
    15. (?[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}) - - [(?[^ ]+ +[0-9]+)] “(?[A-Z]+) (?[^ ]+) HTTP/\d.\d” (?[0-9]+) (?[0-9]+) “[^”]+” “(?[^”]+)”
    4.png
    16. 提取Tomcat等日志使用类似的方法

    Logstash正则提取Nginx日志
    [root@server12 ~]# vim /usr/local/logstash-6.6.0/config/logstash.conf
    input {
    file {
    path => “/usr/local/nginx/logs/access.log”
    }
    }
    filter {
    grok {
    match => {
    “message” => ‘(?[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}) - - [(?[^ ]+ +[0-9]+)] “(?[A-Z]+) (?[^ ]+) HTTP/\d.\d” (?[0-9]+) (?[0-9]+) “[^”]+” “(?[^”]+)”‘
    }
    }
    }
    output {
    elasticsearch {
    hosts => [“http://192.168.10.11:9200“]
    }
    }

    [root@server12 ~]# kill -1 ps aux | grep logstash | awk '{print $2}'
    5.png
    注意正则提取失败的情况
    [root@server12 ~]# echo “jaking” >> /usr/local/nginx/logs/access.log
    6.png

    [root@server12 ~]# while true;do curl http://192.168.10.12;sleep 1;done
    <!DOCTYPE html>






    Welcome to nginx!


    If you see this page, the nginx web server is successfully installed and
    working. Further configuration is required.



    For online documentation and support please refer to
    nginx.org.
    ](http://nginx.org/">nginx.org.
    )
    Commercial support is available at
    nginx.com.

    ](http://nginx.com/">nginx.com.

    )

    Thank you for using nginx.




    7.png
    [root@server12 ~]# while true;do curl -d ‘jaking=legend’ http://192.168.10.12;sleep 1;done



    405 Not Allowed


    nginx/1.14.2



    8.png

    Logstash正则提取出错就不输出到ES
    [root@server12 ~]# vim /usr/local/logstash-6.6.0/config/logstash.conf
    output{
    if “_grokparsefailure” not in [tags] and “_dateparsefailure” not in [tags] {
    elasticsearch {
    hosts => [“http://192.168.10.11:9200“]
    }
    }
    }
    [root@server12 ~]# kill -1 ps aux | grep logstash | awk '{print $2}'
    9.png


    10.png

    [root@server12 ~]# echo 666 >> /usr/local/nginx/logs/access.log
    11.png

    [root@server12 ~]# vim /usr/local/logstash-6.6.0/config/logstash.conf
    output{
    elasticsearch {
    hosts => [“http://192.168.10.11:9200“]
    }
    }

    [root@server12 ~]# kill -1 ps aux | grep logstash | awk '{print $2}'
    [root@server12 ~]# echo 666 >> /usr/local/nginx/logs/access.log
    12.png