问题描述

filebeat 和 log4j appender 同时到 kafka, logstash在启动时报错, 错误如下:

  1. javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=logstash-0

问题原因及解决

input 消费kafka时, 分别指定不同的 client_id.

  1. kafka {
  2.         bootstrap_servers => ["192.168.103.43:9092"] # 注意这里配置的kafka的broker地址不是zk的地址
  3.         client_id => "kafka_client_1"
  4.         group_id => "logstash"
  5.         topics => ["ipaynow_log"]  # kafka topic 名称
  6.         consumer_threads => 5
  7.         decorate_events => true
  8.         type => "string"
  9.         codec => "json"
  10. }
  11. kafka {
  12.         bootstrap_servers => ["192.168.103.43:9092"] # 注意这里配置的kafka的broker地址不是zk的地址
  13.         client_id => "kafka_client_2"
  14.         group_id => "logstash"
  15.         topics => ["ipaynow-hunter"]  # kafka topic 名称
  16.         consumer_threads => 5
  17.         decorate_events => true
  18.         type => "string"
  19.         codec => plain { charset=>"UTF-8" }
  20. }