1. Kafka监控

2. Kafka Eagle

先关闭所有kafka

修改kafka启动命令

cd /opt/module/kafka/bin
vim kafka-server-start.sh

修改以下内容

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
    #export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

xsync /opt/module/kafka/bin/kafka-server-start.sh

再启动kafka

安装eagle

tar -zxvf kafka-eagle-bin-1.3.7.tar.gz
cd kafka-eagle-bin-1.3.7
tar -zxvf kafka-eagle-web-1.3.7-bin.tar.gz -C /opt/module/
cd /opt/module/
mv kafka-eagle-web-1.3.7/ eagle
cd eagle/bin/
chmod +x ke.sh
vim ke.sh

修改 ke.sh

######################################
# multi zookeeper&kafka cluster list
######################################
kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=hadoop102:2181,hadoop103:2181,hadoop104:2181
######################################
# kafka offset storage
######################################
cluster1.kafka.eagle.offset.storage=kafka
######################################
# enable kafka metrics
######################################
kafka.eagle.metrics.charts=true
kafka.eagle.sql.fix.error=false
######################################
# kafka jdbc driver address
######################################
kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://hadoop102:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=000000

添加环境变量

vim /etc/profile.d/my_env.sh

export KE_HOME=/opt/module/eagle
export PATH=$PATH:$KE_HOME/bin

source /etc/profile.d/my_env.sh

启动

ke.sh start

访问http://192.168.130.102:8048/ke 用户admin 密码123456

3. Kafka Monitor

目前已经停止更新了 不推荐使用

https://github.com/Morningstar/kafka-offset-monitor/releases

mkdir -p /opt/module/kafka-offset-console
cd /opt/module/kafka-offset-console

将KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar上传到集群中的 /opt/module/kafka-offset-console

创建脚本文件 start.sh

vim start.sh

#!/bin/bash
nohup java -cp KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--offsetStorage kafka \
--kafkaBrokers hadoop102:9092,hadoop103:9092,hadoop104:9092 \
--kafkaSecurityProtocol PLAINTEXT \
--zk hadoop102:9092,hadoop103:9092,hadoop104:9092 \
--port 8086 \
--refresh 10.seconds \
--retain 2.days \
--dbName offsetapp_kafka >/dev/null 2>&1 &

chmod +x start.sh

访问http://hadoop102:8086/查看web

4. Kafka Manager(CMAK)

https://github.com/yahoo/CMAK

CMAK v3.0.0.5需要JDK11以上，如果你想用JDK1.8去支持此版本是行不通的，会报错提示找不到有些class文件

yum -y install java-11-openjdk-devel.x86_64

修改配置文件

suod vim /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el7_9.x86_64
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

source /etc/profile
java -version

上传解压

mkdir -p /opt/module/kafkamanager
cd /opt/module/kafkamanager

修改配置

vim conf/application.conf

kafka-manager.zkhosts="hadoop102:2181,hadoop103:2181,hadoop104:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="hadoop102:2181,hadoop103:2181,hadoop104:2181"
cmak.zkhosts=${?ZK_HOSTS}

启动

chmod +x bin/cmak
bin/cmak

访问http://hadoop102:9000/