需求是小日志量下，通过filebeat收集多套日志，发送到logstash的5044端口，通过logstash进行分离清洗，然后分别建立索引，好了上配置文件。
    日志格式如下

    1. 2020-05-06 20:45:04.918  INFO 4857 --- [http-nio-8081-exec-4] c.s.s.controller.XytSysController        : method=csoUserInfoSysDTOList=[{"action":"1","channel":"DM","idCard":"4130261024873111111","licensed":"41302619811111111","name":"张某","number":"30042711","organizationCode":"235","telephone":"13526811111"}]
    2. 2020-05-06 20:51:41.748  INFO 4857 --- [http-nio-8081-exec-2] c.s.s.controller.XytSysController        : method=webAccountSys=[{"action":"1","name":"李某","organizationCode":"224","password":"00000000","role":"4","usercode":"CHN0022339"},{"action":"1","name":"李某","organizationCode":"224","password":"00000000","role":"5","usercode":"CHN0022339"}]
    3. 2020-05-06 20:55:05.130  INFO 4857 --- [http-nio-8081-exec-3] c.s.s.controller.XytSysController        : method=csoUserInfoSysDTOList=[{"action":"1","channel":"DM","idCard":"413026198801111111","licensed":"41302619881111111","name":"张某","number":"30042711","organizationCode":"235","telephone":"135268711111"}]

    fileteat的配置文件

    1. - type: log
    2.   enabled: true
    3.   paths:
    4.    - /usr/local/seektruth/situ-microservice-citic/logs/*/info.*.log
    5.   multiline:
    6.    pattern: '^\d+'
    7.    negate:  true
    8.    match:   after
    9.   fields:
    10.    appname: online-microservice-citicpru-info-log
    12. - type: log
    13.   enabled: true
    14.   paths:
    15.    - /usr/local/seektruth/situ-microservice-citic/logs/*/access.*.log
    16.   multiline:
    17.    pattern: '^\d+'
    18.    negate:  true
    19.    match:   after
    20.   fields:
    21.    appname: online-microservice-citicpru-access-log
    23. - type: log
    24.   enabled: true
    25.   paths:
    26.    - /usr/local/seektruth/situ-microservice-citic/logs/*/error.*.log
    27.   multiline:
    28.    pattern: '^\d+'
    29.    negate:  true
    30.    match:   after
    31.   fields:
    32.    appname: online-microservice-citicpru-error-log

    logstash的清洗配置

    1. input {
    2.         beats {
    3.                 port => 5044
    4.         }
    5. }
    6. ###########################################################################################################################
    8. filter {
    9.         if [fields][appname] =~ ".*info-log" {
    10.                 grok {
    11.                         match => {  "message" => ["(%{TIMESTAMP_ISO8601:Request_time}) \|(%{DATA:TraceId})\|(%{DATA:SpanId})\|(%{DATA:ParentSpanId})\|(%{DATA:Level})\|(%{DATA:PID})\|(%{DATA:Threading})\|(%{DATA:Code_Line})\|(%{DATA:Class_Name})\|(%{GREEDYDATA:Message})"] }
    12.                 }
    13.                 if [message] == ";" {
    14.                 drop {}
    15.                 }
    16.         }
    18.         if [fields][appname] =~ ".*error-log" {
    19.                 grok {
    20.                         match => {  "message" => ["(%{TIMESTAMP_ISO8601:Request_time}) \|(%{DATA:TraceId})\|(%{DATA:SpanId})\|(%{DATA:ParentSpanId})\|(%{DATA:Level})\|(%{DATA:PID})\|(%{DATA:Threading})\|(%{DATA:Code_Line})\|(%{DATA:Class_Name})\|(%{GREEDYDATA:Message})"] }
    21.                 }
    22.         }
    24.         if [fields][appname] =~ ".*access-log" {
    25.                 grok {
    26.                         match => {  "message" => ["(%{TIMESTAMP_ISO8601:Request_time}|-)\;(%{IPORHOST:Client_ip}|-)\;(%{DATA:User_name}|-)\;(%{DATA:Request_id}|-)\;(%{PATH:Request_uri}|-)\;(%{INT:Response_time}|-)\;(%{INT:Status_code}|-)\;(%{NOTSPACE:Message}|-)"] }
    27.                 }
    28.                 if [message] == ";" {
    29.                 drop {}
    30.                 }
    31.         }
    32.         date {
    33.                 match => [ "Request_time" , "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
    34.         }
    35. }
    36. ###########################################################################################################################
    38. output {
    39.         if [fields][appname] == "online-microservice-citicpru-access-log" {
    40.                 elasticsearch {
    41.                         hosts => ["172.17.0.3:9200"]
    42.                         user => "elastic"
    43.                         password => "situ1234"
    44.                         index => "online-microservice-citicpru-access-log-%{+YYYY.MM.dd}"
    45.                 }
    46.         }
    47.         if [fields][appname] == "online-microservice-citicpru-info-log"{
    48.                 elasticsearch {：
    49.                         hosts => ["172.17.0.3:9200"]
    50.                         user => "elastic"
    51.                         password => "situ1234"
    52.                         index => "online-microservice-citicpru-info-log-%{+YYYY.MM.dd}"
    53.                 }
    54.         }
    55.         if [fields][appname] == "online-microservice-citicpru-error-log"{
    56.                 elasticsearch {
    57.                         hosts => ["172.17.0.3:9200"]
    58.                         user => "elastic"
    59.                         password => "situ1234"
    60.                         index => "online-microservice-citicpru-error-log-%{+YYYY.MM.dd}"
    61.                 }
    62.         }
    63. }

    效果图
    image.png