Hadoop 完全分布式模式

配置参考：Hadoop 伪分布式模式

角色规划

免密登录

配置 node01 作为管理，用于启动 start-dfs.sh 和关闭 stop-dfs.sh，故 node01 需要把公钥发送给其他主机

scp id_rsa.pub node01:/root/.ssh/node01.pub
scp id_rsa.pub node02:/root/.ssh/node01.pub
scp id_rsa.pub node03:/root/.ssh/node01.pub
scp id_rsa.pub node04:/root/.ssh/node01.pub

其他主机需要先生成 .ssh 目录（ssh localhost），然后将读取公钥并创建 authorized_keys 文件

cat node01.pub >> authorized_keys
cat node02.pub >> authorized_keys
cat node03.pub >> authorized_keys
cat node04.pub >> authorized_keys

测试 node01 免密登录

ssh node02
ssh node03
ssh node04

相关配置

按照“角色规划”进行配置

JDK 绝对路径

vim hadoop-env.sh
export JAVA_HOME=/usr/java/default

NameNode 启动位置

vim core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://node01:9000</value>
    </property>
</configuration>

SecondaryNameNode 启动位置

vim hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/var/bigdata/hadoop/full/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/var/bigdata/hadoop/full/dfs/data</value>
    </property>
    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>/var/bigdata/hadoop/full/dfs/secondary</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>node02:9870</value>
    </property>
</configuration>

DataNode 启动位置

vim slaves
node02
node03
node04

分发配置

scp -r ./bigdata node02:`pwd`
scp -r ./bigdata node03:`pwd`
scp -r ./bigdata node04:`pwd`

格式化

注意：只有第一次启动时需要格式化，在 namenode 节点执行

hdfs namenode -format

启动

start-dfs.sh
Starting namenodes on [node01]
node01: starting namenode, logging to /opt/bigdata/hadoop-2.10.0/logs/hadoop-root-namenode-node01.out
node03: starting datanode, logging to /opt/bigdata/hadoop-2.10.0/logs/hadoop-root-datanode-node03.out
node02: starting datanode, logging to /opt/bigdata/hadoop-2.10.0/logs/hadoop-root-datanode-node02.out
node04: starting datanode, logging to /opt/bigdata/hadoop-2.10.0/logs/hadoop-root-datanode-node04.out
Starting secondary namenodes [node02]
node02: starting secondarynamenode, logging to /opt/bigdata/hadoop-2.10.0/logs/hadoop-root-secondarynamenode-node02.out

验证

[root@node01 ~]# jps
6372 Jps
5311 NameNode
[root@node02 ~]# jps
5363 SecondaryNameNode
6251 Jps
5294 DataNode
[root@node03 ~]# jps
9868 Jps
5567 DataNode
[root@node04 ~]# jps
5190 DataNode
9501 Jps