前置环境

  • 更新系统
    yum install epel-release -y
    yum update -y
  • jdk环境搭建【参考jdk环境搭建】
  • zookeeper环境搭建【参考zookeeper环境搭建】

    官网下载

    wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.0.0/kafka_2.12-3.0.0.tgz
    tar -zvxf kafka_2.12-3.0.0.tgz -C /opt/bigdata/
    ln -s kafka_2.12-3.0.0 kafka

    配置环境变量

    vim /etc/profile.d/kafka_env.sh
    1. #set kafka environment
    2. export KAFKA_HOME=/opt/bigdata/kafka
    3. export PATH=$PATH:$KAFKA_HOME/bin
    source /etc/profile

    kafka集群搭建

    mkdir -p /opt/bigdata/kafka/logs
    cd /opt/bigdata/kafka/config

    vim server.properties

    broker.id=1 
log.dirs=/opt/bigdata/kafka/logs
zookeeper.connect=master01:2181,node01:2181,node02:2181
    注意：各个节点broker.id不能重复

    vim zookeeper.properties

    dataDir=/var/bigdata/zookeeper

    kafka服务启动

  1. 启动zookeeper集群： zkServer.sh start
  2. 启动kafka集群：
    kafka-server-start.sh -daemon /opt/bigdata/kafka/config/server.properties

    Kafka集群一键启动

    vim nodes

    master01
node01
node02

    vim kafka-cluster.sh

    #!/bin/bash
case $1 in
"start"){
cat /root/nodes | while read line  
do
 {
    echo ---------kafka $line 启动---------------
    echo $line
    ssh $line "nohup kafka-server-start.sh ${KAFKA_HOME}/config/server.properties >/dev/nul* 2>&1 & "
 }&
wait
done
}
;;
"stop"){
cat /root/nodes | while read line 
do
 {    
   echo ---------kafka $line 停止---------------
      echo $line
      ssh $line "source /etc/profile;jps |grep Kafka |cut -d' ' -f1 |xargs kill -s 9"
 }&
 wait
 done
}
;;
esac

    验证服务

    jps
    image.png

    zkCli.sh

    get /brokers/ids/${broker.id}
    image.png

    创建topic发送消息

    kafka3.x测试命令

    1、创建topic
    kafka-topics.sh —create —bootstrap-server master01:9092 —replication-factor 1 —partitions 3 —topic test
    2、查看topic列表
    kafka-topics.sh —list —bootstrap-server master01:9092
    kafka-topics.sh —describe —bootstrap-server master01:9092 —topic test

kafka-topics.sh —delete —zookeeper master01:2181 —topic baidu-search-log
3、生产者
kafka-console-producer.sh —bootstrap-server master01:9092 —topic test
4、消费者
kafka-console-consumer.sh —bootstrap-server master01:9092 —topic test —from-beginning

kafka2.x测试命令

1、创建topic
kafka-topics.sh —create —zookeeper master01:2181 —topic test —replication-factor 2 —partitions 3
2、查看topic列表
kafka-topics.sh —list —zookeeper master01:2181
3、生产者
kafka-console-producer.sh —broker-list master01:9092 —topic test
4、消费者
kafka-console-consumer.sh —bootstrap-server master01:9092 —topic test —from-beginning

Kafka-Eagle环境搭建

官网地址：https://www.kafka-eagle.org/
Kafka Eagle是一款结合了目前大数据Kafka监控工具的特点，重新研发的一块开源免费的Kafka集群优秀的监控工具。它可以非常方便的监控生产环境中的offset、lag变化、partition分布、owner等。

前置环境

  • JDK环境搭建

    开启Kafka JMX端口

    cd ${KAFKA_HOME}
export JMX_PORT=9988
nohup bin/kafka-server-start.sh config/server.properties &

    下载安装

    wget https://github.com/smartloli/kafka-eagle-bin/archive/v2.1.0.tar.gz
    tar -zvxf v2.1.0.tar.gz -C /opt/bigdata/
    mv kafka-eagle-bin-2.1.0 kafka-eagle-2.1.0
    ln -s kafka-eagle-2.1.0 kafka-eagle

    配置环境变量

    vim /etc/profile.d/kafka_env.sh ```shell

    set kafka-eagle environment

    export KE_HOME==/opt/bigdata/kafka-eagle/efak-web export PATH=$PATH:$KE_HOME/bin
source /etc/profile
<a name="YWEgS"></a>
## vim conf/system-config.properties
```properties
# 修改第4行，配置kafka集群别名
kafka.eagle.zk.cluster.alias=cluster1
# 修改第5行，配置ZK集群地址
cluster1.zk.list=node01:2181,node02:2181,master01:2181
# 注释第6行
#cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181

# 注释第69行，取消sqlite数据库连接配置
#kafka.eagle.driver=org.sqlite.JDBC
#kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
#kafka.eagle.username=root
#kafka.eagle.password=www.kafka-eagle.org

# 修改第77行，开启mys
kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://192.168.0.6:3306/hadoop?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=iflash

服务启动

  • ./bin/ke.sh start | stop | status
  • 注意kafka必须配置JMX_PORT=9988

image.png

测试验证

http://172.16.179.150:8048/ 用户密码查看上图