1. 修改mapred-site.xml

   
   mapreduce.framework.name
   yarn
   指定mr框架为yarn方式
   
   
   
   mapreduce.jobhistory.address
   bigdata101:10020
   历史服务器端口号
   
   
   mapreduce.jobhistory.webapp.address
   bigdata101:19888
   历史服务器的WEB UI端口号
   
   （6）分发yarn-site.xml
   [later@bigdata101 hadoop]# scp yarn-site.xml bigdata101:/opt/module/hadoop-3.1.3/etc/hadoop/
   [later@bigdata101 hadoop]# scp yarn-site.xml bigdata102:/opt/module/hadoop-3.1.3/etc/hadoop/
   [later@bigdata101 hadoop]# scp mapred-site.xml bigdata101:/opt/module/hadoop-3.1.3/etc/hadoop/
   mapred-site.xml
   [later@bigdata101 hadoop]# scp mapred-site.xml bigdata102:/opt/module/hadoop-3.1.3/etc/hadoop/
   mapred-site.xml
   （7）启动hadoop历史服务器
   [later@bigdata101 hadoop]# mr-jobhistory-daemon.sh start historyserver
   （8）配置spark-default.conf
   [later@bigdata101 hadoop]# cd /opt/module/spark-2.4.5-bin-hadoop2.7/conf/
   [later@bigdata101 conf]# mv spark-defaults.conf.template spark-defaults.conf
   [later@bigdata101 conf]# vim spark-defaults.conf
   spark.yarn.historyServer.address=bigdata101:18080
   spark.yarn.historyServer.allowTracking=true
   spark.eventLog.dir=hdfs://mycluster/spark_historylog
   spark.eventLog.enabled=true
   spark.history.fs.logDirectory=hdfs://mycluster/spark_historylog

2. 启动spark历史服务节点,（我将spark分发到了3台机器上，非必须步骤）

[later@bigdata101 hadoop]# start-history-server.sh

11 安装hbase

1 HBase的解压

[later@bigdata101 software]$ tar -zxvf hbase-2.0.0-bin.tar.gz -C /opt/module/

2 HBase的配置文件

修改HBase对应的配置文件。
1）hbase-env.sh修改内容：
export JAVA_HOME=/opt/module/jdk1.8.0_211
export HBASE_MANAGES_ZK=false
2）hbase-site.xml修改内容：


hbase.rootdir
hdfs://bigdata101:8020/HBase

<property><br />        <name>hbase.cluster.distributed</name><br />        <value>true</value><br />    </property>

hbase.master.port
16000

<property>   <br />        <name>hbase.zookeeper.quorum</name><br />        <value>bigdata101,bigdata102,bigdata103</value><br />    </property>
<property>   <br />        <name>hbase.zookeeper.property.dataDir</name><br />        <value>/opt/module/apache-zookeeper-3.5.7-bin/zkData</value><br />    </property>
<property><br />        <name>hbase.unsafe.stream.capability.enforce</name><br />        <value>false</value><br />    </property><br /></configuration><br />3）regionservers：(在hbase2.X版本中，HMaster节点不能启动HRegionServer)<br />bigdata102<br />bigdata103<br />4）软连接hadoop配置文件到HBase：<br />[later@bigdata101 module]$ ln -s /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml /opt/module/hbase-2.0.0/conf/core-site.xml<br />[later@bigdata101 module]$ ln -s /opt/module/hadoop-3.1.3/etc/hadoop/hdfs-site.xml /opt/module/hbase-2.0.0/conf/hdfs-site.xml

3 HBase远程发送到其他集群

[later@bigdata101 module]$ xsync.sh hbase-2.0.0/

4 HBase服务的启动

1．启动方式
[later@bigdata101 hbase]$ bin/start-hbase.sh
对应的停止服务：
[later@bigdata101 hbase]$ bin/stop-hbase.sh
提示：如果集群之间的节点时间不同步，会导致regionserver无法启动，抛出ClockOutOfSyncException异常。
修复提示：
a、同步时间服务
请参看帮助文档：《大数据技术之Hadoop入门》
b、属性：hbase.master.maxclockskew设置更大的值

5 查看HBase页面

启动成功后，可以通过“host:port”的方式来访问HBase管理页面，例如：
http://bigdata101:16010

12 安装Kylin(之后再安装phoenix)

1）将apache-kylin-2.6.5-bin-hadoop3.tar.gz上传到Linux
2）解压apache-kylin-2.6.5-bin-hadoop3.tar.gz到/opt/module
[later@bigdata101 sorfware]$ tar -zxvf apache-kylin-2.6.5-bin-hadoop3.tar.gz -C /opt/module/
注意：
1、需要在/etc/profile文件中配置HADOOP_HOME，HIVE_HOME，HBASE_HOME并source使其生效。
2、Kylin的新版本不在自带spark，所以在启动kylin的时候需要指定自己的spark_home，或者下载配套的spark，官网有解释

export SPARK_HOME=/path/to/spark
或运行脚本进行下载：
$KYLIN_HOME/bin/download-spark.sh
3）启动
[later@bigdata101 kylin]$ bin/kylin.sh start
启动之后查看各个节点进程：
——————————- hadoop102 ————————
3360 JobHistoryServer
31425 HMaster
3282 NodeManager
3026 DataNode
53283 Jps
2886 NameNode
44007 RunJar
2728 QuorumPeerMain
31566 HRegionServer
——————————- hadoop103 ————————
5040 HMaster
2864 ResourceManager
9729 Jps
2657 QuorumPeerMain
4946 HRegionServer
2979 NodeManager
2727 DataNode
——————————- hadoop104 ————————
4688 HRegionServer
2900 NodeManager
9848 Jps
2636 QuorumPeerMain
2700 DataNode
2815 SecondaryNameNode

注意：启动Kylin之前要保证HDFS，YARN，ZK，HBASE相关进程是正常运行的。
在http://bigdata101:7070/kylin查看Web页面(大概1分钟左右)

用户名为：ADMIN，密码为：KYLIN（系统已填）
4）关闭
[later@bigdata101 kylin]$ bin/kylin.sh stop
5）踩坑
1、安装kylin，如果不能访问http://bigdata101:7070/kylin，先看kylin的日志
如果为缺spark相关的jar包，检查spark_home用的是自己的spark还是系统推荐下载的，使用官网推荐的spark就可以了

1. Kylin启动时Tomcat的kylin.war包报错，7070后台无法访问，这种就是版本不匹配，kylin有匹配hadoop2和hadoop3的版本

13 安装Phoenix

1）上传并解压tar包
[later@bigdata101 module]$ tar -zxvf /opt/software/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz -C /opt/module

1. 复制server包并拷贝到各个节点的hbase/lib

只传这一个jar包就可以了，别的不需要
[later@bigdata101 module]$ cd module/apache-phoenix-5.0.0-HBase-2.0-bin/

[later@bigdata101 phoenix]$ cp phoenix-5.0.0-HBase-2.0-server.jar /opt/module/hbase-2.0.0/lib/
[later@bigdata101 phoenix]$ scp phoenix-5.0.0-HBase-2.0-server.jar bigdata102:/opt/module/hbase-2.0.0/lib/
[later@bigdata101 phoenix]$ scp phoenix-5.0.0-HBase-2.0-server.jar bigdata103:/opt/module/hbase-2.0.0/lib/

3）配置环境变量
#phoenix
export PHOENIX_HOME=/opt/module/apache-phoenix-5.0.0-HBase-2.0-bin
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin
4）启动Pheonix
[later@hadoop101 phoenix]$ bin/sqlline.py bigdata101,bigdata102,bigdata103:2181

14 安装Flume

1）将apache-flume-1.9.0-bin.tar.gz上传到linux的/opt/software目录下
2）解压apache-flume-1.9.0-bin.tar.gz到/opt/module/目录下
[later@bigdata101 software]$ tar -zxf apache-flume-1.9.0-bin.tar.gz -C /opt/module/
3）将apache-flume-1.9.0-bin/conf下的flume-env.sh.template文件修改为flume-env.sh，并配置flume-env.sh文件
[later@bigdata101 conf]$ mv flume-env.sh.template flume-env.sh
[later@bigdata101 conf]$ vi flume-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144

4）在/opt/module/flume/目录下创建job文件夹
[later@bigdata101 flume]$ mkdir job
[later@bigdata101 job]$ vim file-flume-kafka.conf(采集日志)

a1.sources=r1
a1.channels=c1 c2

configure source
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /opt/module/apache-flume-1.9.0-bin/position/log_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /tmp/logs/app.+
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1 c2

interceptor
a1.sources.r1.interceptors = i1 i2
a1.sources.r1.interceptors.i1.type = com.dianjue.flume.MyInterceptor$Builders
a1.sources.r1.interceptors.i2.type = com.dianjue.flume.TypeInterceptor$BuilderType

a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.type = replicating
a1.sources.r1.selector.header = topic
a1.sources.r1.selector.mapping.topic_start = c1
a1.sources.r1.selector.mapping.topic_event = c2

configure channel
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = bigdata101:9092,bigdata102:9092,bigdata103:9092
a1.channels.c1.kafka.topic = topic_start
a1.channels.c1.parseAsFlumeEvent = false
a1.channels.c1.kafka.consumer.group.id = flume-consumer

a1.channels.c2.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c2.kafka.bootstrap.servers = bigdata101:9092,bigdata102:9092,bigdata103:9092
a1.channels.c2.kafka.topic = topic_event
a1.channels.c2.parseAsFlumeEvent = false
a1.channels.c2.kafka.consumer.group.id = flume-consumer

1. 在bigdata102把日志从kafka发到hdfs，所以把flume发送到bigdata102

[later@bigdata101 module]$ scp apache-flume-1.9.0-bin/ bigdata102:/opt/module
[later@bigdata101 job]$ vim kafka-flume-hdfs.conf(发送日志)
注意这里要使用a2,因为这个脚本需要和采集日志脚本同时启动，避免冲突
## 组件
a2.sources=r1 r2
a2.channels=c1 c2
a2.sinks=k1 k2

source1
a2.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a2.sources.r1.batchSize = 5000
a2.sources.r1.batchDurationMillis = 2000
a2.sources.r1.kafka.bootstrap.servers = bigdata101:9092,bigdata102:9092,bigdata103:9092
a2.sources.r1.kafka.topics=topic_start

source2
a2.sources.r2.type = org.apache.flume.source.kafka.KafkaSource
a2.sources.r2.batchSize = 5000
a2.sources.r2.batchDurationMillis = 2000
a2.sources.r2.kafka.bootstrap.servers = bigdata101:9092,bigdata102:9092,bigdata103:9092
a2.sources.r2.kafka.topics=topic_event

channel1
a2.channels.c1.type = file
a2.channels.c1.checkpointDir = /opt/module/apache-flume-1.9.0-bin/checkpoint/behavior1
a2.channels.c1.dataDirs = /opt/module/apache-flume-1.9.0-bin/data/behavior1
a2.channels.c1.maxFileSize = 2146435071
a2.channels.c1.keep-alive = 6

a2.channels.c1.type = memory
#a2.channels.c1.capacity = 1000000
#a2.channels.c1.transactionCapacity = 10000
#a2.channels.c1.byteCapacityBufferPercentage = 20
#a2.channels.c1.byteCapacity = 800000

a2.channels.c2.type = memory
#a2.channels.c2.capacity = 1000000
#a2.channels.c2.transactionCapacity = 10000
#a2.channels.c2.byteCapacityBufferPercentage = 20
#a2.channels.c2.byteCapacity = 800000
## channel2
a2.channels.c2.type = file
a2.channels.c2.checkpointDir = /opt/module/apache-flume-1.9.0-bin/checkpoint/behavior2
a2.channels.c2.dataDirs = /opt/module/apache-flume-1.9.0-bin/data/behavior2/
a2.channels.c2.maxFileSize = 2146435071
a2.channels.c2.capacity = 1000000
a2.channels.c2.keep-alive = 6

sink1
a2.sinks.k1.type = hdfs
a2.sinks.k1.hdfs.path = hdfs://bigdata101:8020/djBigData/origin_data/log/topic_start/%Y-%m-%d
a2.sinks.k1.hdfs.filePrefix = logstart-
a2.sinks.k1.hdfs.round = true
a2.sinks.k1.hdfs.roundValue = 10
a2.sinks.k1.hdfs.roundUnit = second

sink2
a2.sinks.k2.type = hdfs
a2.sinks.k2.hdfs.path = hdfs://bigdata101:8020/djBigData/origin_data/log/topic_event/%Y-%m-%d
a2.sinks.k2.hdfs.filePrefix = logevent-
a2.sinks.k2.hdfs.round = true
a2.sinks.k2.hdfs.roundValue = 10
a2.sinks.k2.hdfs.roundUnit = second

不要产生大量小文件
a2.sinks.k1.hdfs.rollInterval = 10
a2.sinks.k1.hdfs.rollSize = 134217728
a2.sinks.k1.hdfs.rollCount = 0

a2.sinks.k2.hdfs.rollInterval = 10
a2.sinks.k2.hdfs.rollSize = 134217728
a2.sinks.k2.hdfs.rollCount = 0

控制输出文件是原生文件。
a2.sinks.k1.hdfs.fileType = CompressedStream
a2.sinks.k2.hdfs.fileType = CompressedStream

a2.sinks.k1.hdfs.codeC = lzop
a2.sinks.k2.hdfs.codeC = lzop

拼装
a2.sources.r1.channels = c1
a2.sinks.k1.channel= c1

a2.sources.r2.channels = c2
a2.sinks.k2.channel= c2

在这里要使用lzop压缩的话，还需要先配置lzo，因为本身的hadoop不支持lzo压缩，详见编译安装lzo
6）采集日志脚本f1.sh
#! /bin/bash

case $1 in
“start”){
for i in bigdata101
do
echo “ ————启动 $i 采集flume———-“
ssh $i “nohup /opt/module/apache-flume-1.9.0-bin/bin/flume-ng agent —conf-file /opt/module/apache-flume-1.9.0-bin/job/file-flume-kafka.conf —name a1 -Dflume.root.logger=INFO,LOGFILE >/opt/module/apache-flume-1.9.0-bin/test1 2>&1 &”
done
};;
“stop”){
for i in bigdata101
do
echo “ ————停止 $i 采集flume———-“
ssh $i “ps -ef | grep file-flume-kafka | grep -v grep |awk ‘{print \$2}’ | xargs kill”
done

};;
esac

7）发送日志脚本f2.sh
#! /bin/bash

case $1 in
“start”){
for i in bigdata102
do
echo “ ————启动 $i 采集flume———-“
ssh $i “nohup /opt/module/apache-flume-1.9.0-bin/bin/flume-ng agent —conf conf/ —conf-file /opt/module/apache-flume-1.9.0-bin/job/kafka-flume-hdfs.conf —name a2 -Dflume.root.logger=INFO,LOGFILE>/opt/module/apache-flume-1.9.0-bin/test1 2>&1 &”
done
};;
“stop”){
for i in bigdata102
do
echo “ ————停止 $i 采集flume———-“
ssh $i “ps -ef | grep kafka-flume-hdfs | grep -v grep |awk ‘{print \$2}’ | xargs kill”
done
};;
esac

15 安装Datax (Python3.6+MySql 8)

16 解压

17 把mysql-connector-java-8.0.19.jar分别复制到下面

/datax/plugin/reader/mysqlreader/libs
datax/plugin/writer/mysqlwriter/libs

18 把/datax/bin 下三个文件替换，主要是为了Python3.6执行，如果是2版本可以不用

datax.py
dxprof.py
perftrace.py

19 运行自检脚本(注意路径)

Python3.6 datax.py datax/job/job.json

20 增加kafka组件

6 DataX 增加kafka组件

21 安装Canal + UI + RDS_MySQL

1 安装准备

canal.admin-1.1.4.tar.gz
canal.deployer-1.1.4.tar.gz
下载地址 https://github.com/alibaba/canal/releases/tag/canal-1.1.4/

2 安装canal-admin

A 创建文件夹与安装
mkdir canal_admin
tar -zxvf canal.admin-1.1.4.tar.gz -C /export/servers/canal_admin

B 修改配置application.yml
server:
port: 8899
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8

spring.datasource:
address: bigdata101:3306
database: canal_manager
username: root
password: Dj123456+
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false
hikari:
maximum-pool-size: 30
minimum-idle: 1

canal:
adminUser: admin
adminPasswd: 123456
C 执行/canal_admin/conf下的sql语句
D 启动canal-admin
canal_admin/bin
sh startup.sh
点击集群管理，新建集群，配置zk

选择编辑集群配置，下拉选择主配置

################################################
######### common argument #############
#################################################
# tcp bind ip
canal.ip =
# register ip to zookeeper
canal.register.ip =
canal.port = 11111
canal.metrics.pull.port = 11112
# canal instance user/passwd
canal.user = canal
canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458

canal admin config
canal.admin.manager = bigdata101:8899
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = xxxxxxx

canal.zkServers = 192.168.1.96:2181,192.168.1.12:2181,192.168.1.14:2181
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, RocketMQ
canal.serverMode = kafka
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true

detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size = 1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false

binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

binlog ddl isolation
canal.instance.get.ddl.isolation = false

parallel parser config
canal.instance.parser.parallel = true
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
#canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256

table meta tsdb info
canal.instance.tsdb.enable = true
canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername = canal
canal.instance.tsdb.dbPassword = 123456
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval = 24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire = 360

aliyun ak/sk , support rds/mq
canal.aliyun.accessKey =
canal.aliyun.secretKey =

###########################################
######### destinations #############
#################################################
canal.destinations = user_topic
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = manager
canal.instance.global.lazy = false
canal.instance.global.manager.address = ${canal.admin.manager}
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
#canal.instance.global.spring.xml = classpath:spring/file-instance.xml
canal.instance.global.spring.xml = classpath:spring/default-instance.xml

############################################
######### MQ #############
##################################################
canal.mq.servers = 192.168.1.96:9092,192.168.1.12:9092,192.168.1.14:9092
canal.mq.retries = 0
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
canal.mq.lingerMs = 100
canal.mq.bufferMemory = 33554432
canal.mq.canalBatchSize = 50
canal.mq.canalGetTimeout = 100
canal.mq.flatMessage = true
canal.mq.compressionType = none
canal.mq.acks = all
#canal.mq.properties. =
canal.mq.producerGroup = test
# Set this value to “cloud”, if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
# aliyun mq namespace
#canal.mq.namespace =

############################################
######### Kafka Kerberos Info #############
##################################################
canal.mq.kafka.kerberos.enable = false
canal.mq.kafka.kerberos.krb5FilePath = “../conf/kerberos/krb5.conf”
canal.mq.kafka.kerberos.jaasFilePath = “../conf/kerberos/jaas.conf”

3安装canal服务端

mkdir canal
tar -zxvf canal.deployer-1.1.4.tar.gz -C /export/servers/canal

将canal_local.properties文件内容覆盖到canal.properties，并编辑canal.properties

register ip
canal.register.ip =

canal admin config
canal.admin.manager = bigdata101:8899
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = xxxxx
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster = canal_cluster
分别启动服务器上的canal服务
/canal/bin
sh startup.sh
刷新浏览器，点击server管理，发现已自动注入三个实例

4 创建实时同步任务

点击instance管理，新建instance，输入instance名称user_topic，选择canal_cluster集群，点击后面的载入模板，并修改
#################################################
## mysql serverId , v1.0.26+ will autoGen
canal.instance.mysql.slaveId=23

enable gtid use true/false
canal.instance.gtidon=false

position info
canal.instance.master.address=ddddd:3306
canal.instance.master.journal.name=
canal.instance.master.position=0
canal.instance.master.timestamp=
canal.instance.master.gtid=

rds oss binlog
canal.instance.rds.accesskey=LTAI4Gxxxxxx2jDAYZFk
canal.instance.rds.secretkey=G1fyU1fD62xxxxxxxxXvdMQtieqZ
canal.instance.rds.instanceId=rm-uf68kxxxxxx6h0no3

table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

username/password
canal.instance.dbUsername=appuser
canal.instance.dbPassword=Ctm56@Long
canal.instance.defaultDatabaseName = test
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

table regex
canal.instance.filter.regex=test\.student
# table black regex
canal.instance.filter.black.regex=
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

mq config
canal.mq.topic=user_topic
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\..,.\..*
#canal.mq.partition=0
# hash partition config
canal.mq.partitionsNum=1
canal.mq.partitionHash=test.student:id

22 编译与安装

1安装maven+git（配置编译环境）

maven（安装maven，配置环境变量，修改sitting.xml加阿里云镜像）
# tar -zxvf apache-maven-3.6.1-bin.tar.gz

sudo vim /etc/profile.d/maven.sh
export MAVEN_HOME=/opt/software/apache-maven-3.6.1
export PATH=$PATH:$MAVEN_HOME/bin
MAVEN_OPTS=-Xmx2048m
export JAVA_HOME MAVEN_HOME MAVEN_OPTS JAVA_BIN PATH CLASSPATH

修改maven源为阿里源或网易源
# pwd
/opt/software/apache-maven-3.6.1/conf
# vim settings.xml


nexus-aliyun
central
Nexus aliyun
http://maven.aliyun.com/nexus/content/groups/public

jboss-public-repository-group
central
JBoss Public Repository Group
https://repository.jboss.org/nexus/content/groups/public

安装git(务必安装)
yum install -y git

2编译安装lzo

需要安装
gcc-c++
zlib-devel
autoconf
automake
libtool
通过yum安装即可，yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下载、安装并编译LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz

tar -zxvf lzo-2.10.tar.gz

cd lzo-2.10

./configure -prefix=/usr/local/hadoop/lzo/

make

make install

1. 编译hadoop-lzo源码

2.1 下载hadoop-lzo的源码，下载地址：https://github.com/twitter/hadoop-lzo/archive/master.zip
2.2 解压之后，修改pom.xml
3.1.3
2.3 声明两个临时环境变量
export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include
export LIBRARY_PATH=/usr/local/hadoop/lzo/lib
2.4 编译
进入hadoop-lzo-master，执行maven编译命令
mvn package -Dmaven.test.skip=true
2.5 进入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件

2.6）将编译好后的hadoop-lzo-0.4.20.jar 放入hadoop-3.1.3/share/hadoop/common/
[later@bigdata101 common]$ pwd
/opt/module/hadoop-3.1.3/share/hadoop/common
[later@bigdata101 common]$ ls
hadoop-lzo-0.4.21-SNAPSHOT.jar
3）同步hadoop-lzo-0.4.21-SNAPSHOT.jar到bigdata102、bigdata103
[later@bigdata101 common]$ xsync.sh hadoop-lzo-0.4.21-SNAPSHOT.jar
4）core-site.xml增加配置支持LZO压缩
<?xml version=”1.0” encoding=”UTF-8”?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

io.compression.codecs

org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec