概述

NameNode启动流程分三步：启动各种服务、加载元数据、开关安全模式。

0x01 启动各种服务

启动NameNode metrics system
启动JVM pause monitor
启动WebServer

0x02 加载fsimage和edit logs

加载fsimage

读fsimage，默认本地有两个fsimage文件会加载最新的，
Saving image file /var/hadoop/name/current/fsimage.ckpt_0000000000000204159 using no compression
FSEditLog: Starting log segment at 204160

回放 edit logs

读edit logs
把edit logs保存成新的fsimage
启动RPC Server

0x03 开启和关闭安全模式

STATE* Safe mode ON.

DataNode 注册
上报Block
NameNode对每一个BlockReport的RPC请求处理都需要持有全局锁，也就是说对于BlockReport类型RPC请求实际上是串行处理；

The reported blocks 0 needs additional 122 blocks to reach the threshold 0.9990 of total blocks 123.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

2019-11-01 08:51:31,787 INFO org.apache.hadoop.hdfs.StateChange: BLOCK registerDatanode: from DatanodeRegistration(192.168.124.13:50010, datanodeUuid=687f5076-4aa2-45d3-84a3-b8ec49b319be, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-51756656-8694-49fe-ac82-20acf74800b1;nsid=952757189;c=1568193706425) storage 687f5076-4aa2-45d3-84a3-b8ec49b319be
2019-11-01 08:51:31,788 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.124.13:50010
2019-11-01 08:51:31,788 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockReportLeaseManager: Registered DN 687f5076-4aa2-45d3-84a3-b8ec49b319be (192.168.124.13:50010).
2019-11-01 08:51:31,871 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor: Adding new storage ID DS-e1b8b4cf-8fd1-4e77-a9bd-77a65b50a595 for DN 192.168.124.13:50010
2019-11-01 08:51:31,957 INFO BlockStateChange: BLOCK processReport 0x9468455f1446703: from storage DS-e1b8b4cf-8fd1-4e77-a9bd-77a65b50a595 node DatanodeRegistration(192.168.124.13:50010, datanodeUuid=687f5076-4aa2-45d3-84a3-b8ec49b319be, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-51756656-8694-49fe-ac82-20acf74800b1;nsid=952757189;c=1568193706425), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0
The number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:tail1

待汇总的数据量达到预设比例（dfs.namenode.safemode.threshold-pct）后安全模式关闭。

Rolling edit logs

2019-11-07 12:33:33,462 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 55905, 56157

2019-11-07 12:33:33,462 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 254 Total time for transactions(ms): 6 Number of transactions batched in Syncs: 72 Number of syncs: 183 SyncTimes(ms): 55
2019-11-07 12:33:33,463 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /var/hadoop/name/current/edits_inprogress_0000000000000055905 -> /var/hadoop/name/current/edits_0000000000000055905-0000000000000056158

相关类

FSNamesystem管理文件元数据

  • FSDirectory：管理文件系统的命名空间
  • BlockManager：管理文件到数据块的映射和数据块到数据节点的映射

https://tech.meituan.com/2017/03/17/namenode-restart-optimization.html