0. 软件包准备

  • JDK

版本：jdk-8u221-linux-x64.tar.gz
下载：https://www.oracle.com/technetwork/pt/java/javase/downloads/jdk8-downloads-2133151.html?printOnly=1

  • Hadoop

版本：hadoop-2.7.2.tar.gz
下载：http://archive.apache.org/dist/hadoop/core

1. 获取镜像

  1. # 查看可用的稳定版本
  2. docker search centos
  3. docker pull centos:centos7.7.1908
  4. docker image ls |grep centos

2. 单机伪分布式

1. 资源规划

image.png

hadoop01 hadoop02 hadoop03 hadoop04 hadoop05
CentOS-7.7 CentOS-7.7 CentOS-7.7 CentOS-7.7 CentOS-7.7
JDK-1.8.221 JDK-1.8.221 JDK-1.8.221 JDK-1.8.221 JDK-1.8.221
NTP Client NTP Server NTP Client NTP Client NTP Client
NameNode SecondaryNameNode/DataNode DataNode DataNode DataNode
ResourceManager NodeManager NodeManager NodeManager NodeManager
JobHistoryServer N.A N.A N.A N.A

2. 创建容器

1. 手动搭建方式

下面会用到systemctl管理服务，需要加上参数—privileged来增加权，并且不能使用默认的bash，换成/usr/sbin/init。

  1. sudo docker run -d --name hadoop01 --hostname hadoop01 --net hadoop-network --ip 172.24.0.101 -P -p 50070:50070 -p 8088:8088 -p 19888:19888 --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  2. sudo docker run -d --name hadoop02 --hostname hadoop02 --net hadoop-network --ip 172.24.0.102 -P -p 50090:50090 --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  3. sudo docker run -d --name hadoop03 --hostname hadoop03 --net hadoop-network --ip 172.24.0.103 -P --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  4. sudo docker run -d --name hadoop04 --hostname hadoop04 --net hadoop-network --ip 172.24.0.104 -P --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  5. sudo docker run -d --name hadoop05 --hostname hadoop05 --net hadoop-network --ip 172.24.0.105 -P --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init

2. 脚本搭建方式

  1. vi /share/hadoop-cluster.sh

shell脚本内容如下：

  1. #!/bin/bash
  2. # description: Batch start Containers Script
  3. # author: polaris
  5. if [ $# -ne 1  ];then
  6.         echo "You need to start several containers explicitly."
  7.         echo "Some like './hadoop-cluster.sh 3' or 'sh hadoop-cluster.sh 3'"
  8.         exit 1
  9. fi
  10. # 要启动的容器数量
  11. NUM_CONTAINERS=$1
  12. # 自定义网络名称
  13. NETWORK_NAME=hadoop-network
  14. # Hadoop基础镜像ID或者名称
  15. IMAGE_ID=registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2
  16. # 前缀
  17. PREFIX="0"
  18. for (( i=1;i<=$NUM_CONTAINERS;i++ ))
  19.         do
  20.                 if [ $i -eq 1  ];then
  21.                         sudo docker run -d --name hadoop$PREFIX$i --hostname hadoop$PREFIX$i --net ${NETWORK_NAME} --ip 172.24.0.$[$i+100] -P -p 50070:50070 -p 8088:8088 -p 19888:19888 --privileged $IMAGE_ID /usr/sbin/init
  22.                 elif [ $i -eq 2 ];then
  23.                         sudo docker run -d --name hadoop$PREFIX$i --hostname hadoop$PREFIX$i --net ${NETWORK_NAME} --ip 172.24.0.$[$i+100] -P -p 50090:50090 --privileged $IMAGE_ID /usr/sbin/init
  24.                 else
  25.                         sudo docker run -d --name hadoop$PREFIX$i --hostname hadoop$PREFIX$i --net ${NETWORK_NAME} --ip 172.24.0.$[$i+100] -P --privileged $IMAGE_ID /usr/sbin/init
  26.                 fi
  27.         done
  28. echo "$NUM_CONTAINERS containers started!"
  29. echo "==================================="
  30. sudo docker ps | grep hadoop
  31. echo "============IPv4==================="
  32. sudo docker inspect $(sudo docker ps -q) | grep -i ipv4

启动5个容器：

  1. sh /share/hadoop-cluster.sh 5

3. 容器配置

1. 进入容器

  1. docker exec -ti hadoop01 /bin/bash
  2. docker exec -ti hadoop02 /bin/bash
  3. docker exec -ti hadoop03 /bin/bash
  4. docker exec -ti hadoop04 /bin/bash
  5. docker exec -ti hadoop05 /bin/bash
  6. su hadoop

2. 配置映射

  1. # 所有节点执行
  2. sudo vi /etc/hosts

节点ip分配如下：

  1. hadoop01=172.24.0.101
  2. hadoop02=172.24.0.102
  3. hadoop03=172.24.0.103
  4. hadoop04=172.24.0.104
  5. hadoop05=172.24.0.105

3. 免密配置

  1. # 所有节点执行-生成公私钥（-N：无密码）
  2. ssh-keygen -t rsa -N '' -C "450733605@qq.com"
  3. # 所有节点执行-提供公钥给别的节点
  4. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop01
  5. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop02
  6. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop03
  7. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop04
  8. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop05
  9. # 所有节点执行-验证免密
  10. ssh hadoop01
  11. ssh hadoop02
  12. ssh hadoop03
  13. ssh hadoop04
  14. ssh hadoop05

4. 时间同步

1. 安装NTP服务

  1. # 所有节点执行
  2. rpm -qa | grep ntp
  3. sudo yum -y install ntp

2. 设置时间配置文件

  1. # hadoop02节点执行
  2. sudo vi /etc/ntp.conf

配置如下：

  1. # 修改一（设置本地网络上的主机不受限制-新增）
  2. restrict 172.24.0.0 mask 255.255.255.0 nomodify notrap
  3. # 修改二（添加默认的一个内部时钟数据，使用它为局域网用户提供服务-新增）
  4. server 127.127.1.0
  5. fudge 127.127.1.0 stratum 10
  6. # 修改三（设置为不采用公共的服务器-注释）
  7. #server 0.centos.pool.ntp.org iburst
  8. #server 1.centos.pool.ntp.org iburst
  9. #server 2.centos.pool.ntp.org iburst
  10. #server 3.centos.pool.ntp.org iburst

3. 设置BIOS与系统时间同步

  1. # hadoop02节点执行
  2. sudo vim /etc/sysconfig/ntpd

配置如下：

  1. # 删除如下内容
  2. OPTIONS="-g"
  3. # 增加如下内容（让硬件时间与系统时间一起同步）
  4. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
  5. SYNC_HWCLOCK=yes

4. 启动ntp服务并测试

  1. # hadoop02节点执行
  2. sudo systemctl start ntpd
  3. sudo systemctl restart ntpd
  4. # 设置ntp服务开机自启
  5. sudo systemctl enable ntpd.service
  7. # 查看NTP服务状态
  8. systemctl status ntpd
  9. ntpstat
  10. sudo ntpq -p
  12. # 查看时区
  13. timedatectl
  14. # 设置时区（所有节点执行，服务端）
  15. sudo timedatectl set-timezone Asia/Shanghai

5. 客户端与时间服务器同步时间

注意：先关闭非时间服务器节点的ntpd服务。

  1. # 非hadoop02节点执行
  2. sudo systemctl stop ntpd

设置时区：

  1. # 设置时区（所有节点执行，客户端）
  2. sudo timedatectl set-timezone Asia/Shanghai

客户端安装crontabs以定时同步时间：

  1. # 非hadoop02节点执行
  2. sudo yum -y install vixie-cron crontabs

添加定时任务：

  1. # 非hadoop02节点执行
  2. sudo vi /etc/crontab

任务内容如下：

  1. */1 * * * * /usr/sbin/ntpdate hadoop02

任务生效：

  1. # 非hadoop02节点执行
  2. sudo /bin/systemctl start crond
  3. sudo /bin/systemctl stop crond
  4. sudo /bin/systemctl restart crond
  5. sudo /bin/systemctl reload crond
  6. sudo /bin/systemctl status crond
  7. # 使任务生效
  8. sudo crontab /etc/crontab

6. 测试

  1. # 修改时间服务器时间
  2. sudo date -s "2020-12-31 12:12:12"
  3. # 服务器时间同步修正
  4. sudo ntpdate -u ntp.api.bz
  5. # 客户端手动同步时间
  6. sudo /usr/sbin/ntpdate hadoop02
  7. ###### NTP服务器(上海) ：ntp.api.bz
  8. ###### 中国国家授时中心：210.72.145.44
  9. ###### 美国：time.nist.gov
  10. ###### 复旦：ntp.fudan.edu.cn
  11. ###### 微软公司授时主机(美国) ：time.windows.com
  12. ###### 台警大授时中心(台湾)：asia.pool.ntp.org

5. Hadoop配置

注意：主节点上配置完成后分发至其他节点。

  • 创建所需文件夹

    1. # 主节点（hadoop01）执行
    2. su hadoop
    3. mkdir -p /opt/software/hadoop-2.7.2/tmp
    4. mkdir -p /opt/software/hadoop-2.7.2/dfs/namenode_data
    5. mkdir -p /opt/software/hadoop-2.7.2/dfs/datanode_data
    6. mkdir -p /opt/software/hadoop-2.7.2/checkpoint/dfs/cname

    1. hadoop-env.sh

    1. vi ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh

    配置如下：

    1. export JAVA_HOME=/opt/moudle/jdk1.8.0_221
    2. export HADOOP_CONF_DIR=/opt/software/hadoop-2.7.2/etc/hadoop

    2. core-site.xml

    1. vi ${HADOOP_HOME}/etc/hadoop/core-site.xml

    配置如下：

    1. <configuration>
    2.       <property>
    3.           <!--用来指定hdfs的master，namenode的地址-->
    4.           <name>fs.defaultFS</name>
    5.           <value>hdfs://hadoop01:9000</value>
    6.       </property>  
    7.       <property>
    8.           <!--用来指定hadoop运行时产生文件的存放目录-->   
    9.           <name>hadoop.tmp.dir</name>
    10.           <value>/opt/software/hadoop-2.7.2/tmp</value>
    11.       </property>
    12.       <property>
    13.           <!--设置缓存大小，默认4kb-->
    14.           <name>io.file.buffer.size</name>
    15.           <value>4096</value>
    16.       </property>
    17. </configuration>

    3. hdfs-site.xml

    指定副本系数和hdfs操作权限。

    1. vi ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml

    配置如下：

    1. <configuration>
    2.       <property>
    3.          <!--数据块默认大小128M-->
    4.          <name>dfs.block.size</name>
    5.          <value>134217728</value>
    6.       </property>
    7.       <property>
    8.           <!--副本数量，不配置的话默认为3-->
    9.           <name>dfs.replication</name> 
    10.           <value>1</value>
    11.       </property>
    12.       <property>
    13.                <!--定点检查--> 
    14.                <name>fs.checkpoint.dir</name>
    15.                <value>/opt/software/hadoop-2.7.2/checkpoint/dfs/cname</value>
    16.       </property>
    17.       <property>
    18.           <!--namenode节点数据（元数据）的存放位置-->
    19.           <name>dfs.name.dir</name> 
    20.           <value>/opt/software/hadoop-2.7.2/dfs/namenode_data</value>
    21.       </property>
    22.       <property>
    23.           <!--datanode节点数据（元数据）的存放位置-->
    24.           <name>dfs.data.dir</name> 
    25.           <value>/opt/software/hadoop-2.7.2/dfs/datanode_data</value>
    26.       </property>
    27.       <property>
    28.           <!--指定secondarynamenode的web地址-->
    29.           <name>dfs.namenode.secondary.http-address</name> 
    30.           <value>hadoop02:50090</value>
    31.       </property>
    32.       <property>
    33.           <!--hdfs文件操作权限,false为不验证-->
    34.           <name>dfs.permissions</name> 
    35.           <value>false</value>
    36.       </property>
    37. </configuration>

    4. mapred-site.xml

    1. cp ${HADOOP_HOME}/etc/hadoop/mapred-site.xml.template ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
    2. vi ${HADOOP_HOME}/etc/hadoop/mapred-site.xml

    配置如下：

    1. <configuration>
    2.       <property>
    3.           <!--指定mapreduce运行在yarn上-->
    4.           <name>mapreduce.framework.name</name>
    5.           <value>yarn</value>
    6.       </property>
    7.       <property>
    8.           <!--配置任务历史服务器地址-->
    9.           <name>mapreduce.jobhistory.address</name>
    10.           <value>hadoop01:10020</value>
    11.       </property>
    12.       <property>
    13.           <!--配置任务历史服务器web-UI地址-->
    14.           <name>mapreduce.jobhistory.webapp.address</name>
    15.           <value>hadoop01:19888</value>
    16.       </property>
    17. </configuration>

    5. yarn-site.xml

    1. vi ${HADOOP_HOME}/etc/hadoop/yarn-site.xml

    配置如下：

    1. <configuration>
    2.       <property>
    3.       <!--指定yarn的主节点，resourcemanager的地址-->
    4.       <name>yarn.resourcemanager.hostname</name>
    5.       <value>hadoop01</value>
    6.       </property>
    7.       <property>
    8.           <name>yarn.resourcemanager.address</name>
    9.           <value>hadoop01:8032</value>
    10.       </property>
    11.       <property>
    12.           <name>yarn.resourcemanager.webapp.address</name>
    13.           <value>hadoop01:8088</value>
    14.       </property>
    15.       <property>
    16.           <name>yarn.resourcemanager.scheduler.address</name>
    17.           <value>hadoop01:8030</value>
    18.       </property>
    19.       <property>
    20.           <name>yarn.resourcemanager.resource-tracker.address</name>
    21.           <value>hadoop01:8031</value>
    22.       </property>
    23.       <property>
    24.           <name>yarn.resourcemanager.admin.address</name>
    25.           <value>hadoop01:8033</value>
    26.       </property>
    27.       <property>
    28.           <!--NodeManager获取数据的方式-->
    29.           <name>yarn.nodemanager.aux-services</name>
    30.           <value>mapreduce_shuffle</value>
    31.       </property>
    32.       <property>
    33.           <!--开启日志聚集功能-->
    34.           <name>yarn.log-aggregation-enable</name>
    35.           <value>true</value>
    36.       </property>
    37.       <property>
    38.           <!--配置日志保留7天-->
    39.           <name>yarn.log-aggregation.retain-seconds</name>
    40.           <value>604800</value>
    41.       </property>
    42. </configuration>

    6. master

    1. vi ${HADOOP_HOME}/etc/hadoop/master

    配置如下：

    1. hadoop01

    7. slaves

    1. vi ${HADOOP_HOME}/etc/hadoop/slaves

    配置如下：

    1. hadoop02
    2. hadoop03
    3. hadoop04
    4. hadoop05

    6. 分发

    1. sudo scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop02:/opt/software/
    2. sudo scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop03:/opt/software/
    3. sudo scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop04:/opt/software/
    4. sudo scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop05:/opt/software/

    4. 启动Hadoop

    1. HDFS初始化

    1. # 主节点（hadoop01）执行
    2. hdfs namenode -format

    2. 启动HDFS和YARN

    1. # 主节点（hadoop01）执行
    2. # 启动dfs服务
    3. start-dfs.sh
    4. # 启动yarn服务
    5. start-yarn.sh
    6. # 启动任务历史服务器
    7. mr-jobhistory-daemon.sh start historyserver

    5. 验证

  • 进程查看 ```bash [hadoop@hadoop01 ~]$ jps NameNode ResourceManager JobHistoryServer

[hadoop@hadoop02 ~]$ jps

SecondaryNameNode

DataNode

NodeManager

[hadoop@hadoop03 ~]$ jps

DataNode

NodeManager

[hadoop@hadoop04 ~]$ jps

DataNode

NodeManager

[hadoop@hadoop05 ~]$ jps

DataNode

NodeManager

  2. - **Web UI**
  3.    - **NameNode WebUI**：[http://192.168.0.99:50070](http://192.168.0.99:50070)
  4.    - **SecondaryNameNode WebUI**：[http://192.168.0.99:50090](http://192.168.0.99:50090)
  5.    - **ResourceManager WebUI**：[http://192.168.0.99:8088](http://192.168.0.99:8088)
  6.    - **HistoryServer WebUI**：[http://192.168.0.99:19888](http://192.168.0.99:19888)
  7. <a name="SBOBn"></a>
  8. ## 6. 测试集群
  9. 提交 Hadoop 内置的计算 Pi 的示例程序为例，在任何一个节点上执行都可以，命令如下：
  10. ```bash
  11. hadoop jar /opt/software/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 11 24

任务完成后可以在任务历史服务器（端口19888）中找到相应的job：http://192.168.0.99:19888。

7. 提交镜像到公仓

  1. # 登录公仓
  2. docker login --username=alypolarisbigdata registry.cn-qingdao.aliyuncs.com
  3. # 镜像tag
  4. docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop01" hadoop01 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop01
  5. docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop02" hadoop02 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop02
  6. docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop03" hadoop03 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop03
  7. docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop04" hadoop04 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop04
  8. docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop05" hadoop05 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop05
  9. docker images |grep hadoop
  10. # 镜像推送
  11. docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop01
  12. docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop02
  13. docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop03
  14. docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop04
  15. docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop05
  16. # 拉取镜像
  17. docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop01
  18. docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop02
  19. docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop03
  20. docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop04
  21. docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-cluster:hadoop05

3. 单机伪分布式HA

1. 资源规划

hadoop01 hadoop02 hadoop03 hadoop04 hadoop05
CentOS-7.7 CentOS-7.7 CentOS-7.7 CentOS-7.7 CentOS-7.7
JDK-1.8.221 JDK-1.8.221 JDK-1.8.221 JDK-1.8.221 JDK-1.8.221
NTP Client NTP Server NTP Client NTP Client NTP Client
NameNode/DFSZKFailoverController/DataNode/JournalNode NameNode/DFSZKFailoverController/DataNode/JournalNode DataNode/JournalNode DataNode/JournalNode DataNode/JournalNode
ResourceManager NodeManager ResourceManager/NodeManager NodeManager NodeManager
JobHistoryServer N.A N.A N.A N.A
QuorumPeerMain QuorumPeerMain QuorumPeerMain N.A N.A

2. 创建容器

  1. sudo docker run -d --name hadoop-ha-01 --hostname hadoop-ha-01 --net hadoop-network --ip 172.24.0.101 -P -p 50070:50070 -p 8088:8088 -p 19888:19888 --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  2. sudo docker run -d --name hadoop-ha-02 --hostname hadoop-ha-02 --net hadoop-network --ip 172.24.0.102 -P -p 50072:50070 --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  3. sudo docker run -d --name hadoop-ha-03 --hostname hadoop-ha-03 --net hadoop-network --ip 172.24.0.103 -P -p 8083:8088 --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  4. sudo docker run -d --name hadoop-ha-04 --hostname hadoop-ha-04 --net hadoop-network --ip 172.24.0.104 -P --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init
  5. sudo docker run -d --name hadoop-ha-05 --hostname hadoop-ha-05 --net hadoop-network --ip 172.24.0.105 -P --privileged registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-single:2.7.2 /usr/sbin/init

3. 进入容器

  1. sudo docker exec -ti hadoop-ha-01 /bin/bash
  2. sudo docker exec -ti hadoop-ha-02 /bin/bash
  3. sudo docker exec -ti hadoop-ha-03 /bin/bash
  4. sudo docker exec -ti hadoop-ha-04 /bin/bash
  5. sudo docker exec -ti hadoop-ha-05 /bin/bash
  6. su hadoop

4. 容器配置

1. 配置映射

  1. # 所有节点执行
  2. sudo vi /etc/hosts

节点ip分配如下：

  1. hadoop-ha-01=172.24.0.101
  2. hadoop-ha-02=172.24.0.102
  3. hadoop-ha-03=172.24.0.103
  4. hadoop-ha-04=172.24.0.104
  5. hadoop-ha-05=172.24.0.105

2. 免密配置

  1. # 所有节点执行-生成公私钥（-N：无密码）
  2. ssh-keygen -t rsa -N '' -C "450733605@qq.com"
  3. # 所有节点执行-提供公钥给别的节点
  4. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-01
  5. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-02
  6. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-03
  7. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-04
  8. ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop-ha-05
  9. # 所有节点执行-验证免密
  10. ssh hadoop-ha-01
  11. ssh hadoop-ha-02
  12. ssh hadoop-ha-03
  13. ssh hadoop-ha-04
  14. ssh hadoop-ha-05

3. 时间同步

1. 安装NTP服务

  1. # 所有节点执行
  2. rpm -qa | grep ntp
  3. sudo yum -y install ntp

2. 设置时间配置文件

  1. # hadoop-ha-02节点执行
  2. sudo vi /etc/ntp.conf

配置如下：

  1. # 修改一（设置本地网络上的主机不受限制-新增）
  2. restrict 172.24.0.0 mask 255.255.255.0 nomodify notrap
  3. # 修改二（添加默认的一个内部时钟数据，使用它为局域网用户提供服务-新增）
  4. server 127.127.1.0
  5. fudge 127.127.1.0 stratum 10
  6. # 修改三（设置为不采用公共的服务器-注释）
  7. #server 0.centos.pool.ntp.org iburst
  8. #server 1.centos.pool.ntp.org iburst
  9. #server 2.centos.pool.ntp.org iburst
  10. #server 3.centos.pool.ntp.org iburst

3. 设置BIOS与系统时间同步

  1. # hadoop02节点执行
  2. sudo vi /etc/sysconfig/ntpd

配置如下：

  1. # 删除如下内容
  2. OPTIONS="-g"
  3. # 增加如下内容（让硬件时间与系统时间一起同步）
  4. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"
  5. SYNC_HWCLOCK=yes

4. 启动ntp服务并测试

  1. # hadoop02节点执行
  2. sudo systemctl start ntpd
  3. sudo systemctl restart ntpd
  4. # 设置ntp服务开机自启
  5. sudo systemctl enable ntpd.service
  7. # 查看NTP服务状态
  8. systemctl status ntpd
  9. ntpstat
  10. sudo ntpq -p
  12. # 查看时区
  13. timedatectl
  14. # 设置时区（所有节点执行，服务端）
  15. sudo timedatectl set-timezone Asia/Shanghai

5. 客户端与时间服务器同步时间

注意：先关闭非时间服务器节点的ntpd服务。

  1. # 非hadoop-ha-02节点执行
  2. sudo systemctl stop ntpd

设置时区：

  1. # 设置时区（所有节点执行，客户端）
  2. sudo timedatectl set-timezone Asia/Shanghai

客户端安装crontabs以定时同步时间：

  1. # 非hadoop-ha-02节点执行
  2. sudo yum -y install vixie-cron crontabs

添加定时任务：

  1. # 非hadoop-ha-02节点执行
  2. sudo vi /etc/crontab

任务内容如下：

  1. */1 * * * * /usr/sbin/ntpdate hadoop-ha-02

任务生效：

  1. # 非hadoop-ha-02节点执行
  2. sudo /bin/systemctl start crond
  3. sudo /bin/systemctl stop crond
  4. sudo /bin/systemctl restart crond
  5. sudo /bin/systemctl reload crond
  6. sudo /bin/systemctl status crond
  7. # 使任务生效
  8. sudo crontab /etc/crontab

6. 测试

  1. # 修改时间服务器时间
  2. sudo date -s "2020-12-31 12:12:12"
  3. # 服务器时间同步修正
  4. sudo ntpdate -u ntp.api.bz
  5. # 客户端手动同步时间
  6. sudo /usr/sbin/ntpdate hadoop-ha-02
  7. ###### NTP服务器(上海) ：ntp.api.bz
  8. ###### 中国国家授时中心：210.72.145.44
  9. ###### 美国：time.nist.gov
  10. ###### 复旦：ntp.fudan.edu.cn
  11. ###### 微软公司授时主机(美国) ：time.windows.com
  12. ###### 台警大授时中心(台湾)：asia.pool.ntp.org

4. ZooKeeper安装配置

1. 上传安装包

  1. # 宿主机执行，先将zk安装包上传至/share，随后上传至容器
  2. sudo docker cp /share/zookeeper-3.4.10.tar.gz hadoop-ha-01:/opt/software/

2. 解压缩

  1. tar -zxvf /opt/software/zookeeper-3.4.10.tar.gz -C /opt/software/

3. 创建所需目录

  1. mkdir /opt/software/zookeeper-3.4.10/zoo_data
  2. mkdir /opt/software/zookeeper-3.4.10/zoo_logs

4. 配置zoo.cfg

  1. cd /opt/software/zookeeper-3.4.10/conf
  2. cp zoo_sample.cfg zoo.cfg
  3. vi zoo.cfg

配置如下：

  1. tickTime=2000
  2. initLimit=10
  3. syncLimit=5
  4. dataDir=/opt/software/zookeeper-3.4.10/zoo_data
  5. dataLogDir=/opt/software/zookeeper-3.4.10/zoo_logs
  6. clientPort=2181
  7. server.1=hadoop-ha-01:2888:3888
  8. server.2=hadoop-ha-02:2888:3888
  9. server.3=hadoop-ha-03:2888:3888

6. 分发

  1. scp -r /opt/software/zookeeper-3.4.10 hadoop@hadoop-ha-02:/opt/software/
  2. scp -r /opt/software/zookeeper-3.4.10 hadoop@hadoop-ha-03:/opt/software/

7. 修改myid

  1. # hadoop-ha-01
  2. echo 1 >> /opt/software/zookeeper-3.4.10/zoo_data/myid
  3. # hadoop-ha-02
  4. echo 2 >> /opt/software/zookeeper-3.4.10/zoo_data/myid
  5. # hadoop-ha-03
  6. echo 3 >> /opt/software/zookeeper-3.4.10/zoo_data/myid

8. 启动zk集群

  1. # 分别在zk的各节点上执行
  2. cd /opt/software/zookeeper-3.4.10/bin
  3. ./zkServer.sh start
  4. ./zkServer.sh stop
  5. ./zkServer.sh status

5. Hadoop配置

注意：主节点上配置完成后分发至其他节点。

  • 创建所需文件夹

    1. # 主节点（hadoop-ha-01）执行
    2. su hadoop
    3. # 若非第一次创建，则需要先清理相关目录
    4. rm -rf /opt/software/hadoop-2.7.2/tmp/ 
    5. rm -rf /opt/software/hadoop-2.7.2/dfs/ 
    6. rm -rf /opt/software/hadoop-2.7.2/checkpoint/ 
    7. rm -rf /opt/software/hadoop-2.7.2/logs/
    8. mkdir -p /opt/software/hadoop-2.7.2/tmp
    9. mkdir -p /opt/software/hadoop-2.7.2/dfs/journalnode_data
    10. mkdir -p /opt/software/hadoop-2.7.2/dfs/edits
    11. mkdir -p /opt/software/hadoop-2.7.2/dfs/datanode_data
    12. mkdir -p /opt/software/hadoop-2.7.2/dfs/namenode_data

    1. hadoop-env.sh

    1. vi ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh

    配置如下：

    1. export JAVA_HOME=/opt/moudle/jdk1.8.0_221
    2. export HADOOP_CONF_DIR=/opt/software/hadoop-2.7.2/etc/hadoop

    2. core-site.xml

    1. vi ${HADOOP_HOME}/etc/hadoop/core-site.xml

    配置如下：

    1. <configuration>
    2.   <property>
    3.       <!--指定hadoop集群在zookeeper上注册的节点名-->
    4.       <name>fs.defaultFS</name>
    5.       <value>hdfs://hacluster</value>
    6.   </property>  
    7.   <property>
    8.       <!--用来指定hadoop运行时产生文件的存放目录-->   
    9.       <name>hadoop.tmp.dir</name>
    10.       <value>/opt/software/hadoop-2.7.2/tmp</value>
    11.   </property>
    12.   <property>
    13.       <!--设置缓存大小，默认4kb-->
    14.       <name>io.file.buffer.size</name>
    15.       <value>4096</value>
    16.   </property>
    17.   <property>
    18.       <!--指定zookeeper的存放地址 -->
    19.       <name>ha.zookeeper.quorum</name>
    20.       <value>hadoop-ha-01:2181,hadoop-ha-02:2181,hadoop-ha-03:2181</value>
    21.   </property>
    22. </configuration>

    3. hdfs-site.xml

    指定副本系数和hdfs操作权限。

    1. vi ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml

    配置如下：

    1. <configuration>
    2.   <property>
    3.          <!--数据块默认大小128M-->
    4.          <name>dfs.block.size</name>
    5.          <value>134217728</value>
    6.   </property>
    7.   <property>
    8.           <!--副本数量，不配置的话默认为3-->
    9.           <name>dfs.replication</name> 
    10.           <value>1</value>
    11.   </property>
    12.   <property>
    13.           <!--namenode节点数据（元数据）的存放位置-->
    14.           <name>dfs.name.dir</name> 
    15.           <value>/opt/software/hadoop-2.7.2/dfs/namenode_data</value>
    16.   </property>
    17.   <property>
    18.           <!--datanode节点数据（元数据）的存放位置-->
    19.           <name>dfs.data.dir</name> 
    20.           <value>/opt/software/hadoop-2.7.2/dfs/datanode_data</value>
    21.   </property>
    22.   <property>
    23.           <name>dfs.webhdfs.enabled</name>
    24.           <value>true</value>
    25.   </property>
    26.   <property>
    27.           <name>dfs.datanode.max.transfer.threads</name>
    28.           <value>4096</value>
    29.       </property>
    30.   <property>
    31.           <name>dfs.nameservices</name>
    32.           <value>hacluster</value>
    33.   </property>
    34.   <property>
    35.           <!-- hacluster集群下有两个namenode，分别为nn1,nn2 -->
    36.           <name>dfs.ha.namenodes.hacluster</name>
    37.           <value>nn1,nn2</value>
    38.   </property>
    39.   <!-- nn1的rpc、servicepc和http通信 -->
    40.   <property>
    41.           <name>dfs.namenode.rpc-address.hacluster.nn1</name>
    42.           <value>hadoop-ha-01:9000</value>
    43.   </property> 
    44.   <property>
    45.           <name>dfs.namenode.servicepc-address.hacluster.nn1</name>
    46.           <value>hadoop-ha-01:53310</value>
    47.   </property>
    48.   <property>
    49.           <name>dfs.namenode.http-address.hacluster.nn1</name> 
    50.           <value>hadoop-ha-01:50070</value>
    51.   </property>
    52.   <!-- nn2的rpc、servicepc和http通信 -->
    53.   <property>
    54.           <name>dfs.namenode.rpc-address.hacluster.nn2</name>
    55.           <value>hadoop-ha-02:9000</value>
    56.   </property>
    57.   <property>
    58.           <name>dfs.namenode.servicepc-address.hacluster.nn2</name>
    59.           <value>hadoop-ha-02:53310</value>
    60.   </property>
    61.   <property>
    62.           <name>dfs.namenode.http-address.hacluster.nn2</name>
    63.           <value>hadoop-ha-02:50070</value>
    64.   </property>
    65.   <property>
    66.           <!-- 指定namenode的元数据在JournalNode上存放的位置 -->
    67.           <name>dfs.namenode.shared.edits.dir</name>
    68.           <value>qjournal://hadoop-ha-01:8485;hadoop-ha-02:8485;hadoop-ha-03:8485;hadoop-ha-04:8485;hadoop-ha-05:8485/hacluster</value>
    69.   </property>
    70.   <property>
    71.           <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
    72.           <name>dfs.journalnode.edits.dir</name>
    73.           <value>/opt/software/hadoop-2.7.2/dfs/journalnode_data</value>
    74.   </property>
    75.   <property>
    76.           <!-- namenode操作日志的存放位置 -->
    77.           <name>dfs.namenode.edits.dir</name>
    78.           <value>/opt/software/hadoop-2.7.2/dfs/edits</value>
    79.   </property>
    80.   <property>
    81.           <!-- 开启namenode故障转移自动切换 -->
    82.           <name>dfs.ha.automatic-failover.enabled</name>
    83.           <value>true</value> 
    84.   </property>
    85.   <property>
    86.           <!-- 配置失败自动切换实现方式 -->
    87.           <name>dfs.client.failover.proxy.provider.hacluster</name>
    88.           <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    89.   </property>    
    90.   <property>
    91.           <!-- 配置隔离机制 -->
    92.           <name>dfs.ha.fencing.methods</name>
    93.           <value>sshfence</value>
    94.   </property>
    95.   <property>
    96.           <!-- 使用隔离机制需要SSH免密登录 -->
    97.           <name>dfs.ha.fencing.ssh.private-key-files</name>
    98.           <value>/home/hadoop/.ssh/id_rsa</value>
    99.   </property>
    100.   <property>
    101.           <!--hdfs文件操作权限,false为不验证-->
    102.           <name>dfs.permissions</name> 
    103.           <value>false</value>
    104.   </property>
    105. </configuration>

    4. mapred-site.xml

    cp ${HADOOP_HOME}/etc/hadoop/mapred-site.xml.template ${HADOOP_HOME}/etc/hadoop/mapred-site.xml
vi ${HADOOP_HOME}/etc/hadoop/mapred-site.xml

    配置如下：

    <configuration>
  <property>  
      <!--指定mapreduce运行在yarn上-->
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
  <property>
      <!--配置任务历史服务器地址-->
      <name>mapreduce.jobhistory.address</name>
      <value>hadoop-ha-01:10020</value>
  </property>
  <property>
      <!--配置任务历史服务器web-UI地址-->
      <name>mapreduce.jobhistory.webapp.address</name>
      <value>hadoop-ha-01:19888</value>
  </property>
  <property>
      <!--开启uber模式-->
      <name>mapreduce.job.ubertask.enable</name>
      <value>true</value>
   </property>
</configuration>

    5. yarn-site.xml

    vi ${HADOOP_HOME}/etc/hadoop/yarn-site.xml

    配置如下：

    <configuration>
  <property>
          <!-- 开启Yarn高可用 -->
          <name>yarn.resourcemanager.ha.enabled</name>
          <value>true</value>
  </property>
  <property>
          <!-- 指定Yarn集群在zookeeper上注册的节点名 -->
          <name>yarn.resourcemanager.cluster-id</name>
          <value>hayarn</value>
  </property>
  <property>
          <!-- 指定两个ResourceManager的名称 -->
          <name>yarn.resourcemanager.ha.rm-ids</name>
          <value>rm1,rm2</value>
  </property> 
  <property>
          <!-- 指定rm1的主机 -->
          <name>yarn.resourcemanager.hostname.rm1</name>
          <value>hadoop-ha-01</value>
  </property>              
  <property>
          <!-- 指定rm2的主机 -->
          <name>yarn.resourcemanager.hostname.rm2</name>
          <value>hadoop-ha-03</value>
  </property> 
  <property>
          <!-- 配置zookeeper的地址 -->
          <name>yarn.resourcemanager.zk-address</name>
          <value>hadoop-ha-01:2181,hadoop-ha-02:2181,hadoop-ha-03:2181</value>
      </property>    
  <property>
          <!-- 开启Yarn恢复机制 -->
          <name>yarn.resourcemanager.recovery.enabled</name>
          <value>true</value>
  </property> 
  <property>
          <!-- 配置执行ResourceManager恢复机制实现类 -->
          <name>yarn.resourcemanager.store.class</name>
          <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  </property>
  <property>
          <!--指定主resourcemanager的地址-->
          <name>yarn.resourcemanager.hostname</name>
          <value>hadoop-ha-03</value>
  </property>
  <property>
          <!--NodeManager获取数据的方式-->
          <name>yarn.nodemanager.aux-services</name>
          <value>mapreduce_shuffle</value>
  </property>
  <property>
          <!--开启日志聚集功能-->
          <name>yarn.log-aggregation-enable</name>
          <value>true</value>
  </property>
  <property>
          <!--配置日志保留7天-->
          <name>yarn.log-aggregation.retain-seconds</name>
          <value>604800</value>
  </property>
</configuration>

    6. slaves

    vi ${HADOOP_HOME}/etc/hadoop/slaves

    配置如下：

    hadoop-ha-01
hadoop-ha-02
hadoop-ha-03
hadoop-ha-04
hadoop-ha-05

    6. 分发

    scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop-ha-02:/opt/software/
scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop-ha-03:/opt/software/
scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop-ha-04:/opt/software/
scp -r /opt/software/hadoop-2.7.2/ hadoop@hadoop-ha-05:/opt/software/

    5. 启动集群

    1. 启动ZooKeeper集群

    # 分别在zk的各节点上执行
cd /opt/software/zookeeper-3.4.10/bin
./zkServer.sh start
./zkServer.sh stop
./zkServer.sh status

    2. 启动JournalNode集群

    # 分别到各节点执行
hadoop-daemon.sh start journalnode

    3. HDFS格式化

    # NameNode主节点执行
hdfs namenode -format

    4. NameNode元数据同步

    # 将NameNode主节点的元数据同步至NameNode副节点
scp -r /opt/software/hadoop-2.7.2/dfs/namenode_data/current/ hadoop@hadoop-ha-02:/opt/software/hadoop-2.7.2/dfs/namenode_data/

    5. 启动ZKFC服务

    # NameNode主节点、NameNode副节点，二者只能选择一个节点执行
hdfs zkfc -formatZK

    6. 安装psmisc服务

    # NameNode主节点、NameNode副节点
sudo yum -y install psmisc

    7. 启动HDFS主节点

    # NameNode主节点执行
start-dfs.sh

    8. 启动YARN主节点

    # YARN主节点执行
start-yarn.sh

    9. 启动YARN副节点及历史任务服务器

    # hadoop-ha-01
yarn-daemon.sh start resourcemanager
mr-jobhistory-daemon.sh start historyserver

    6. 验证

  • 进程查看 ```bash [hadoop@hadoop-ha-01 ~]$ jps QuorumPeerMain DFSZKFailoverController ResourceManager NameNode NodeManager DataNode JournalNode JobHistoryServer

[hadoop@hadoop-ha-02 ~]$ jps

QuorumPeerMain

DFSZKFailoverController

NameNode

NodeManager

DataNode

JournalNode

[hadoop@hadoop-ha-03 ~]$ jps

QuorumPeerMain

ResourceManager

NodeManager

DataNode

JournalNode

[hadoop@hadoop-ha-04 ~]$ jps

NodeManager

DataNode

JournalNode

[hadoop@hadoop-ha-05 ~]$ jps

NodeManager

DataNode

JournalNode


- **Web UI**
   - **NameNode WebUI**：[http://192.168.0.99:50070](http://192.168.0.99:50070)
   - **NameNode WebUI**：[http://192.168.0.99:50072](http://192.168.0.99:50072)
   - **ResourceManager WebUI**：[http://192.168.0.99:8088](http://192.168.0.99:8088)
   - **ResourceManager WebUI**：[http://192.168.0.99:8083](http://192.168.0.99:8083)
   - **HistoryServer WebUI**：[http://192.168.0.99:19888](http://192.168.0.99:19888)
<a name="3c2k9"></a>
## 7. 测试集群
提交 Hadoop 内置的计算 Pi 的示例程序为例，在任何一个节点上执行都可以，命令如下：
```bash
hadoop jar /opt/software/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 11 24

任务完成后可以在任务历史服务器（端口19888）中找到相应的job：http://192.168.0.99:19888。

8. 提交镜像到公仓

# 登录公仓
docker login --username=alypolarisbigdata registry.cn-qingdao.aliyuncs.com
# 镜像tag
docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop-ha-01" hadoop-ha-01 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-01
docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop-ha-02" hadoop-ha-02 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-02
docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop-ha-03" hadoop-ha-03 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-03
docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop-ha-04" hadoop-ha-04 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-04
docker commit -a "polaris<450733605@qq.com>" -m "This is backup for hadoop-2.7.2 hadoop-ha-05" hadoop-ha-05 registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-05
docker images |grep hadoop-2.7.2-ha
# 镜像推送
docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-01
docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-02
docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-03
docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-04
docker push registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-05
# 拉取镜像
docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-01
docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-02
docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-03
docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-04
docker pull registry.cn-qingdao.aliyuncs.com/polaris-docker/hadoop-ha:hadoop-2.7.2-ha-05

参考

B站：Docker环境下DIY你的Hadoop镜像
https://www.bilibili.com/video/BV1C64y1T7FZ?p=1
微信：Docker环境下Hadoop单机伪分布式
https://mp.weixin.qq.com/s/pkS2nWI8RY22AnsHWRTZug
微信：Docker环境下Hadoop完全分布式
https://mp.weixin.qq.com/s/vfUdLohUwmsD4ecm1ldNJw
微信：Docker之镜像拉取、删除及重命名
https://mp.weixin.qq.com/s/7L9aelZRbPl-67jyntdNXg
微信：Docker环境下HA(高可用)-Hadoop集群
https://mp.weixin.qq.com/s/PTLRXsaPj7jtVFiSJPYrSQ
微信：DockerCompose搭建Zookeeper集群
https://mp.weixin.qq.com/s/9wNzofl8hdJlZtqijq7PNQ