001 数据迁移过程中出现的错误

  1. 19/03/28 10:24:09 ERROR zookeeper.ZooKeeperWatcher: hconnection-0x69ddcb11-0x269c20ff7830004, quorum=192.168.1.36:2181,192.168.1.35:2181,192.168.1.34:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
  2. org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/meta-region-server
  3. 原因
  4. 从以上日志可以看出，session超时，其实服务是好的，只是被zk认为死了，所以rs自己也就把自己kill了
  6. 仔细查看一下log信息
  7. 19/03/28 10:48:29 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x269c21ba162000c, likely server has closed socket, closing socket connection and attempting reconnect
  8. 19/03/28 10:48:30 INFO zookeeper.ClientCnxn: Opening socket connection to server c6/192.168.1.36:2181. Will not attempt to authenticate using SASL (unknown error)
  9. 19/03/28 10:48:30 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.1.251:54564, server: c6/192.168.1.36:2181
  11. 19/03/28 10:48:30 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x269c21ba162000c, likely server has closed socket, closing socket connection and attempting reconnect
  12. 19/03/28 10:48:32 INFO zookeeper.ClientCnxn: Opening socket connection to server c5/192.168.1.35:2181. Will not attempt to authenticate using SASL (unknown error)
  13. 19/03/28 10:48:32 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.1.251:41374, server: c5/192.168.1.35:2181
  15. 19/03/28 10:48:33 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x269c21ba162000c, likely server has closed socket, closing socket connection and attempting reconnect
  16. 19/03/28 10:48:33 INFO zookeeper.ClientCnxn: Opening socket connection to server c4/192.168.1.34:2181. Will not attempt to authenticate using SASL (unknown error)
  17. 19/03/28 10:48:33 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.1.251:41424, server: c4/192.168.1.34:2181
  19. 19/03/28 10:48:33 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x269c21ba162000c, likely server has closed socket, closing socket connection and attempting reconnect
  20. 19/03/28 10:48:34 INFO zookeeper.ClientCnxn: Opening socket connection to server c6/192.168.1.36:2181. Will not attempt to authenticate using SASL (unknown error)
  21. 19/03/28 10:48:34 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.1.251:54570, server: c6/192.168.1.36:2181
  23. 19/03/28 10:48:34 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x269c21ba162000c, likely server has closed socket, closing socket connection and attempting reconnect
  24. 19/03/28 10:48:36 INFO zookeeper.ClientCnxn: Opening socket connection to server c5/192.168.1.35:2181. Will not attempt to authenticate using SASL (unknown error)
  25. 19/03/28 10:48:36 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.1.251:41380, server: c5/192.168.1.35:2181
  27. 19/03/28 10:48:36 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x269c21ba162000c, likely server has closed socket, closing socket connection and attempting reconnect
  28. 19/03/28 10:48:37 INFO zookeeper.ClientCnxn: Opening socket connection to server c4/192.168.1.34:2181. Will not attempt to authenticate using SASL (unknown error)
  29. 19/03/28 10:48:37 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.1.251:41430, server: c4/192.168.1.34:2181
  30. 19/03/28 10:48:37 INFO zookeeper.ClientCnxn: Session establishment complete on server c4/192.168.1.34:2181, sessionid = 0x269c21ba162000c, negotiated timeout = 60000

优化方向

  1. HBase写优化方面：
  3. 1.做HBase简单的写优化：增加HBase的write buffer至55%，减小HBase的read
  4.     //增加hbase.client.write.buffer:2M
  6. buffer至25%（write buffer与read buffer的和不能超过80%）
  8. 2.增加HBase的regionserver handler数至最大。
  9. // 增加了rs的数量，从30调至40
  11. 3.HBase表的预分区修改(未做处理）
  13. Zk方面：
  14. HBase的zk超时检测延长。