canal.properties配置

  1. canal.id = 1
  2. canal.ip =
  3. canal.port= 11111
  4. canal.metrics.pull.port = 11112
  5. #指定zk集群的位置
  6. canal.zkServers = xxxxxx:2181
  7. canal.zookeeper.flush.period = 1000
  8. canal.withoutNetty = false
  9. # 使用rocketMQ
  10. canal.serverMode = rocketmq
  11. # flush meta cursor/parse position to file
  12. canal.file.data.dir = ${canal.conf.dir}
  13. canal.file.flush.period = 1000
  14. ## memory store RingBuffer size, should be Math.pow(2,n)
  15. canal.instance.memory.buffer.size = 16384
  16. ## memory store RingBuffer used memory unit size , default 1kb
  17. canal.instance.memory.buffer.memunit = 1024 
  18. ##内存模式
  19. canal.instance.memory.batch.mode = MEMSIZE
  20. canal.instance.memory.rawEntry = true
  22. ## canal检测instance的一些配置
  23. canal.instance.detecting.enable = false
  24. canal.instance.detecting.sql = select 1
  25. canal.instance.detecting.interval.time = 3
  26. canal.instance.detecting.retry.threshold = 3
  27. canal.instance.detecting.heartbeatHaEnable = false
  29. # 支持最大事务 这些无所谓
  30. canal.instance.transaction.size =  1024
  31. canal.instance.fallbackIntervalInSeconds = 60
  33. # 网络连接的一些配置
  34. canal.instance.network.receiveBufferSize = 16384
  35. canal.instance.network.sendBufferSize = 16384
  36. canal.instance.network.soTimeout = 30
  38. # binlog filter config 
  39. canal.instance.filter.druid.ddl = true
  40. canal.instance.filter.query.dcl = true
  41. canal.instance.filter.query.dml = false
  42. canal.instance.filter.query.ddl = true
  43. canal.instance.filter.table.error = true
  44. canal.instance.filter.rows = false
  45. canal.instance.filter.transaction.entry = true
  47. # 只支持binlog的 ROW模式
  48. canal.instance.binlog.format = ROW
  49. canal.instance.binlog.image = FULL,MINIMAL,NOBLOB
  51. # binlog ddl isolation
  52. canal.instance.get.ddl.isolation = false
  54. # parallel parser config
  55. canal.instance.parser.parallel = true
  56. ## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
  57. #canal.instance.parser.parallelThreadSize = 16
  58. ## disruptor ringbuffer size, must be power of 2
  59. canal.instance.parser.parallelBufferSize = 256
  61. # table meta tsdb info
  62. canal.instance.tsdb.enable=false
  63. canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
  64. canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
  65. canal.instance.tsdb.dbUsername=canal
  66. canal.instance.tsdb.dbPassword=canal
  67. # dump snapshot interval, default 24 hour
  68. canal.instance.tsdb.snapshot.interval=24
  69. # purge snapshot expire , default 360 hour(15 days)
  70. canal.instance.tsdb.snapshot.expire=360
  72. # rds oss binlog account
  73. canal.instance.rds.accesskey =
  74. canal.instance.rds.secretkey =
  76. #指定具体的instance
  77. canal.destinations = account,balance,distribution,inventory,member,product,order
  78. # conf root dir
  79. canal.conf.dir = ../conf
  80. # 自动检测也开启
  81. canal.auto.scan = true
  82. canal.auto.scan.interval = 5
  84. canal.instance.tsdb.spring.xml=classpath:spring/tsdb/h2-tsdb.xml
  85. #canal.instance.tsdb.spring.xml=classpath:spring/tsdb/mysql-tsdb.xml
  87. #配置文件模式是spring
  88. canal.instance.global.mode = spring
  89. canal.instance.global.lazy = false
  90. #使用default-instance.xml这个xml
  91. canal.instance.global.spring.xml = classpath:spring/default-instance.xml

总结： 整体看下来canal.properties的配置的关键点也能读懂，主要是以下几点

  • binlog只支持ROW模式
  • instance配置了account,balance,….,order等
  • 指定了zk

其他的配置可能由于我还不太熟练，所以影响不深

account/instance.properties解析

  1. #指定slaveId
  2. canal.instance.mysql.slaveId=1101
  3. # 直接关掉 gtidon
  4. canal.instance.gtidon=false
  6. # 监听的数据库
  7. canal.instance.master.address=10.11.7.24:3306
  8. canal.instance.master.journal.name=
  9. canal.instance.master.position=
  10. canal.instance.master.timestamp=
  11. canal.instance.master.gtid=
  13. # rds oss binlog
  14. canal.instance.rds.accesskey=
  15. canal.instance.rds.secretkey=
  16. canal.instance.rds.instanceId=
  18. # table meta tsdb info
  19. canal.instance.tsdb.enable=false
  20. #canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
  21. #canal.instance.tsdb.dbUsername=canal
  22. #canal.instance.tsdb.dbPassword=T9UKK3ZL
  24. #canal.instance.standby.address =
  25. #canal.instance.standby.journal.name =
  26. #canal.instance.standby.position =
  27. #canal.instance.standby.timestamp =
  28. #canal.instance.standby.gtid=
  30. #配置用户名密码
  31. canal.instance.dbUsername=canal
  32. canal.instance.dbPassword=canal-rfaDv5X3Vk92
  33. canal.instance.connectionCharset = utf8
  34. canal.instance.defaultDatabaseName = account_db
  35. # enable druid Decrypt database password
  36. canal.instance.enableDruid=false
  37. #canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==
  39. # 同步的表数据
  40. canal.instance.filter.regex = account_db.sys_tenant,account_db.sys_user,account_db.sys_user_tenant
  41. # table black regex
  42. canal.instance.filter.black.regex=

总结：instance.properties的配置相对而言好读，配置少，除了个别，比如gtid到底是个啥？tsdb？等其他都简单易懂

  • 指定了监听的表信息：sys_tenant，sys_user,sys_user_tenant
  • 指定数据库的连接地址用户名和密码

MQ的配置呢？

canal中已经配置rocketMQ模式，在canal-1.1.5已经可以在instance.properties中配置啦，但是这里锥智使用的是canal-1.1.1版本，所以需要在mq.yml中配置，这个yml也是canal自带的。

  1. servers: xxxxxx:9876
  2. retries: 0
  3. batchSize: 16384
  4. lingerMs: 1
  5. bufferMemory: 33554432
  7. # Canal的batch size, 默认50K, 由于kafka最大消息体限制请勿超过1M(900K以下)
  8. canalBatchSize: 50
  9. # Canal get数据的超时时间, 单位: 毫秒, 0为不限超时
  10. canalGetTimeout: 100
  11. flatMessage: true
  13. canalDestinations:
  14.   - canalDestination: product
  15.     topic: product-sync-canal
  16.     partition: 0
  17.   - canalDestination: order
  18.     topic: order-sync-canal
  19.     partition: 0
  20.   - canalDestination: member
  21.     topic: member-sync-canal
  22.     partition: 0
  23.   - canalDestination: balance
  24.     topic: balance-sync-canal
  25.     partition: 0
  26.   - canalDestination: inventory
  27.     topic: inventory-sync-canal
  28.     partition: 0
  29.   - canalDestination: distribution
  30.     topic: distribution-sync-canal
  31.     partition: 0
  32.   - canalDestination: account
  33.     topic: account-sync-canal
  34.     partition: 0
  35. #  #对应topic分区数量
  36. #  partitionsNum: 3
  37. #  partitionHash:
  38. #    #库名.表名: 唯一主键
  39. #    mytest.person: id

其他

其他的instance配置都类似就不一个一个看啦