实战环境
192.168.10.12 Logstash Kafka Filebeat
Kafka
1. Kafka依赖于Zookeeper
2. 两个都依赖于Java
Kafka依赖于Zookeeper
ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件。它是一个为分布式应用提供一致性服务的软件,提供的功能包括:配置维护、域名服务、分布式同步、组服务等。
ZooKeeper的目标就是封装好复杂易出错的关键服务,将简单易用的接口和性能高效、功能稳定的系统提供给用户。
ZooKeeper包含一个简单的原语集,提供Java和C的接口。
3. 官方网站:https://zookeeper.apache.org/
4. 下载ZK的二进制包
5. 解压到对应目录完成安装
ZK的安装命令
tar xvf zookeeper-3.4.13.tar.gz -C /usr/local
cp /usr/local/zookeeper-3.4.13/conf/zoo_sample.cfg /usr/local/zookeeper-3.4.13/conf/zoo.cfg
ZK的启动
6. 更改配置:clientPortAddress=0.0.0.0
[root@server12 ~]# vim /usr/local/zookeeper-3.4.13/conf/zoo.cfg
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPortAddress=0.0.0.0
clientPort=2181
# the maximum number of client connections.
7. 启动:/usr/local/zookeeper-3.4.13/bin/zkServer.sh start
[root@server12 ~]# /usr/local/zookeeper-3.4.13/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.4.13/bin/../conf/zoo.cfg
Starting zookeeper … STARTED
[root@server12 ~]# netstat -pantul | grep java
tcp6 0 0 127.0.0.1:9600 ::: LISTEN 18372/java
tcp6 0 0 :::2181 ::: LISTEN 20951/java
tcp6 0 0 :::37190 ::: LISTEN 20951/java
tcp6 0 0 192.168.10.12:36038 192.168.10.11:9200 ESTABLISHED 18372/java
tcp6 0 0 192.168.10.12:44542 192.168.10.12:6379 ESTABLISHED 18372/java
tcp6 0 0 192.168.10.12:36036 192.168.10.11:9200 ESTABLISHED 18372/java
Kafka下载地址
8. Kafka官网:http://kafka.apache.org/
9. 下载Kafka的二进制包
10. 解压到对应目录完成安装
Kafka的安装命令
tar xvf kafka_2.11-2.1.1.tgz
mv kafka_2.11-2.1.1 /usr/local/kafka_2.11
Kafka的启动
11. 更改kafka的配置:更改监听地址、更改连接zk的地址
[root@server12 ~]# vim /usr/local/kafka_2.11/config/server.properties
zookeeper.connect=192.168.10.12:2181
listeners=PLAINTEXT://192.168.10.12:9092
12. 前台启动:/usr/local/kafka_2.11/bin/kafka-server-start.sh /usr/local/kafka_2.11/config/server.properties
13. 启动kafka:nohup /usr/local/kafka_2.11/bin/kafka-server-start.sh /usr/local/kafka_2.11/config/server.properties >/tmp/kafka.log 2>&1 &
[root@server12 ~]# netstat -pantul | grep java
tcp6 0 0 :::44377 ::: LISTEN 21604/java
tcp6 0 0 127.0.0.1:9600 ::: LISTEN 18372/java
tcp6 0 0 192.168.10.12:9092 ::: LISTEN 21604/java
tcp6 0 0 :::2181 ::: LISTEN 20951/java
tcp6 0 0 :::37190 ::: LISTEN 20951/java
tcp6 0 0 192.168.10.12:2181 192.168.10.12:33320 ESTABLISHED 20951/java
tcp6 0 0 192.168.10.12:36038 192.168.10.11:9200 ESTABLISHED 18372/java
tcp6 0 0 192.168.10.12:44542 192.168.10.12:6379 ESTABLISHED 18372/java
tcp6 0 0 192.168.10.12:9092 192.168.10.12:58276 ESTABLISHED 21604/java
tcp6 0 0 192.168.10.12:36036 192.168.10.11:9200 ESTABLISHED 18372/java
tcp6 0 0 192.168.10.12:33320 192.168.10.12:2181 ESTABLISHED 21604/java
tcp6 0 0 192.168.10.12:58276 192.168.10.12:9092 ESTABLISHED 21604/java