安装配置

下载

wget https://github.com/zendesk/maxwell/releases/download/v1.24.0/maxwell-1.24.0.tar.gz

配置kafka版本

复制jar包到maxwell

cp /opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/libs/kafka-clients-2.2.1-kafka-4.1.0.jar ./lib/kafka-clients/

配置文件

• 文档 http://maxwells-daemon.io/filtering/
• config.properties

log_level=info
kafka.compression.type=snappy
kafka.retries=0
kafka.acks=1
filter= exclude: *.*, include: test.*, include: /yx_log.*/.*

分配 mysql 权限

GRANT Replication client,Replication slave,select ON *.* TO 'maxwell'@'%' IDENTIFIED BY 'maxwell';
GRANT ALL ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY  'maxwell';
flush privileges;

命令执行

• --kafka_version 对应 kafka版本

  bin/maxwell --user='root' --password='youai123' --host='master' \
   --producer=kafka --kafka.bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092 --kafka_topic=maxwell \
   --kafka_version=2.2.1-kafka-4.1.0 --client_id=1 --output_ddl=true --config=config.properties

案例：
同步MySQL binlog 到 kafka

bin/maxwell --user='root' --password='mhtx123123' --host='10.19.200.2'  --port=3336  --producer=kafka --kafka.bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092 --kafka_topic=ylyh_test    --kafka_version=2.2.1-kafka-4.1.0 --client_id=ylyh_test --output_ddl=true --config=config.properties

kafka 同步到 kudu

python3 kafka2kudu.py --kafka 'kafka1:9092,kafka2:9092' --kudu_master 'master:7051' --set_pruduct ylyh --set_version test