配置

  • centos7
  • jdk8
  • zookeeper3.5.5
  • hadoop2.8.5

角色规划

服务器名称 服务器ip 角色
bigdata01 192.168.0.105 namenode\datanode\zookeeper
bigdata02 192.168.0.106 namenode\datanode\zookeeper
bigdata03 192.168.0.107 datanode\zookeeper

一、环境准备

1.1 物理环境

  • 3台或以上服务器
  • 给服务器配置静态ip（动态ip也行）
  1. vim /etc/sysconfig/network-scripts/ifcfg-eth0
  1. BOOTPROTO=static
  2. ONBOOT=yes
  3. IPADDR=192.168.0.105
  4. NETMASK=255.255.255.0
  5. GATEWAY=192.168.0.1
  • 配置hostname
  1.  vim /etc/hostname

image.png

  • 配置hosts
  1. vim /etc/hosts
  1. 192.168.0.105 bigdata01
  2. 192.168.0.106 bigdata02
  3. 192.168.0.107 bigdata03
  • 节点间ssh免密钥
  1. ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  2. ssh-copy-id + 主机名

image.png

  • 时间同步
  1. # 检查是否安装ntp
  2. rpm -q ntp
  3. # 安装
  4. yum -y install ntp
  5. # 开机自启
  6. systemctl enable ntpd
  7. # 启动
  8. systemctl start ntpd
  9. # 同步服务器
  10. ntpdate -u ntp1.aliyun.com
  12. # 配置
  13. # 主节点
  14. vim /etc/ntp.conf

image.png
从节点
image.png

  1. # 重新启动
  2. service ntpd restart
  3. # 开机启动
  4. chkconfig ntpd on
  5. 查看时间同步情况
  6. ntpq -p

**

1.2 软件环境

  • jdk8+
  1. # 安装步骤不会的自行百度
  2. # 配置环境变量
  3. export JAVA_HOME=/usr/java/jdk1.8
  4. export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
  5. export PATH=$PATH:${JAVA_HOME}/bin
  • zookeeper

安装

  1. # 下载解压
  2. tar zxvf apache-zookeeper-3.5.5-bin.tar.gz -C /opt
  3. cd /opt
  4. # 重命名
  5. mv apache-zookeeper-3.5.5-bin zookeeper

配置

  1. # 1、zoo.cfg
  2. cp zoo_sample.cfg zoo.cfg
  3. vim zoo.cfg

image.png

  1. # 2、myid
  2. mkdir -p /var/zookeeper
  3. echo 1 > /var/zookeeper/myid

配置环境变量

  1. 配置环境变量
  2. vim /etc/profile
  3. export ZOOKEEPER_HOME=/opt/zookeeper
  4. export PATH=$PATH:$ZOOKEEPER_HOME/bin
  6. source /etc/profile
  9. # 启动
  10. zkServer.sh start
  12. # 查看状态
  13. zkServer.sh status

二、开始安装hadoop

2.1 解压安装

  1. tar zxvf hadoop-2.8.5.tar.gz -C /opt

2.2 配置

2.2.1 hadoop-env.sh

  1. export JAVA_HOME=/usr/java/jdk1.8

2.2.2 mapred-env.sh

  1. export JAVA_HOME=/usr/java/jdk1.8

2.2.3 yarn-env.sh

  1. export JAVA_HOME=/usr/java/jdk1.8

2.2.4 hdfs-site.xml

  1. <configuration>
  3. <property>
  5.   <name>dfs.replication</name>
  7.   <value>2</value>
  9. </property>
  11. <property>
  13.   <name>dfs.nameservices</name>
  15.   <value>mycluster</value>
  17. </property>
  19. <property>
  21.   <name>dfs.ha.namenodes.mycluster</name>
  23.   <value>nn1,nn2</value>
  25. </property>
  27. <property>
  29.   <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  31.   <value>bigdata01:8020</value>
  33. </property>
  35. <property>
  37.   <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  39.   <value>bigdata02:8020</value>
  41. </property>
  43. <property>
  45.   <name>dfs.namenode.http-address.mycluster.nn1</name>
  47.   <value>bigdata01:50070</value>
  49. </property>
  51. <property>
  53.   <name>dfs.namenode.http-address.mycluster.nn2</name>
  55.   <value>bigdata02:50070</value>
  57. </property>
  59. <property>
  61.   <name>dfs.namenode.shared.edits.dir</name>
  63.   <value>qjournal://bigdata01:8485;bigdata02:8485;bigdata03:8485/mycluster</value>
  65. </property>
  67. <property>
  69.   <name>dfs.journalnode.edits.dir</name>
  71.   <value>/var/hadoop/ha/jn</value>
  73. </property>
  75. <property>
  77.   <name>dfs.client.failover.proxy.provider.mycluster</name>
  79.   <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  81. </property>
  83. <property>
  85.   <name>dfs.ha.fencing.methods</name>
  87.   <value>sshfence</value>
  89. </property>
  91. <property>
  93.   <name>dfs.ha.fencing.ssh.private-key-files</name>
  95.   <value>/root/.ssh/id_rsa</value>
  97. </property>
  99. <property>
  101.    <name>dfs.ha.automatic-failover.enabled</name>
  103.    <value>true</value>
  105. </property>
  107. </configuration>

2.2.5 core-site.xml

<configuration>

<property>

  <name>fs.defaultFS</name>

  <value>hdfs://mycluster</value>

</property>

<property>

  <name>hadoop.tmp.dir</name>

  <value>/var/hadoop/ha</value>

</property>

<property>

   <name>ha.zookeeper.quorum</name>

   <value>bigdata01:2181,bigdata02:2181,bigdata03:2181</value>

</property>

</configuration>

2.2.6 mapred-site.xml

<property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

</property>

2.2.7 yarn-site.xml

<property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

<property>

   <name>yarn.resourcemanager.ha.enabled</name>

   <value>true</value>

</property>

<property>

   <name>yarn.resourcemanager.cluster-id</name>

   <value>cluster1</value>

</property>

<property>

   <name>yarn.resourcemanager.ha.rm-ids</name>

   <value>rm1,rm2</value>

</property>

<property>

   <name>yarn.resourcemanager.hostname.rm1</name>

   <value>bigdata02</value>

</property>

<property>

   <name>yarn.resourcemanager.hostname.rm2</name>

   <value>bigdata03</value>

</property>

<property>

   <name>yarn.resourcemanager.zk-address</name>

   <value>bigdata01:2181,bigdata02:2181,bigdata03:2181</value>

</property>

2.2.8 slaves

bigdata01
bigdata02
bigdata03

2.3 分发到各节点

scp -r hadoop-2.8.5 bigdata02:/opt
scp -r hadoop-2.8.5 bigdata03:/opt

2.4 启动zookeeper（所有节点）

zkServer.sh start

2.5 启动journalnode（所有节点）

/opt/hadoop-2.8.5/sbin/hadoop-daemon.sh start journalnode

2.6 格式化（主节点）

/opt/hadoop-2.8.5/bin/hdfs namenode -format

格式化完成后主节点启动namenode
/opt/hadoop-2.8.5/sbin/hadoop-daemon.sh start namenode

2.7 备节点同步主节点元数据（备节点）

/opt/hadoop-2.8.5/bin/hdfs namenode -bootstrapStandby

2.8 格式化zookeeper(主节点)

/opt/hadoop-2.8.5/bin/hdfs zkfc -formatZK

2.9 配置环境变量(所有节点)

vim /etc/profile

export HADOOP_HOME=/opt/hadoop-2.8.5
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

source /etc/profile

2.10 启动

# 启动hdfs
start-dfs.sh
# 启动resourcemanager（2、3节点）
yarn-daemon.sh start resourcemanager

之后每次的启动顺序

启动zookeeper

zkServer.sh start(node01 node02 node03)

启动hdfs

start-dfs.sh

启动yarn

start-yarn.sh

yarn-daemon.sh start resourcemanager

yarn-daemon.sh start resourcemanager

hdfs-namenodeweb界面
http://192.168.0.105:50070/
yarn-resourcemanagerweb界面
http://192.168.0.106:8088/cluster