一：基础设施

操作系统，环境，网络，必须软件 vmware虚拟机，虚拟网络编辑器为桥接模式，虚拟机也需要设置成桥接模式 需要环境 centos7 ，jdk-8u181-linux-x64.rpm ， hadoop-2.6.5.tar.gz
 链接：https://pan.baidu.com/s/1IpwgtLCK8kvJ_7f105n1hw 提取码：h3jj

1：设置IP

vi /etc/sysconfig/network-scripts/ifcfg-eth0

  1. DEVICE=eth0
  2. #HWADDR=00:0C:29:42:15:C2
  3. TYPE=Ethernet
  4. ONBOOT=yes
  5. NM_CONTROLLED=yes
  6. BOOTPROTO=static
  7. IPADDR=192.168.0.200
  8. NETMASK=255.255.255.0
  9. GATEWAY=192.168.0.2
  10. DNS1=223.5.5.5
  11. DNS2=114.114.114.114

2：设置主机名

vi /etc/sysconfig/network

  1. NETWORKING=yes
  2. HOSTNAME=node01

3：设置本机的ip到主机名的映射关系

vi /etc/hosts

  1. 192.168.0.200 node01
  2. 192.168.0.12 node02

4：关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

5：关闭 selinux

vi /etc/selinux/config

  1. SELINUX=disabled

6：时间同步

yum install ntp -y
vi /etc/ntp.conf

  1. server ntp1.aliyun.com

service ntpd start
chkconfig ntpd on

7：安装JDK

rpm -i jdk-8u181-linux-x64.rpm
cd /usr/java
ll
vi /etc/profile

  1. export  JAVA_HOME=/usr/java/default
  2. export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile

8：ssh免密

ssh localhost
ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
如果A 想 免密的登陆到B：
A：
ssh-keygen -t dsa -P ‘’ -f ~/.ssh/id_dsa
B：
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

二：hadoop配置

1：规划路径

mkdir /opt/bigdata
tar xf hadoop-2.6.5.tar.gz
mv hadoop-2.6.5 /opt/bigdata/
pwd
/opt/bigdata/hadoop-2.6.5
vi /etc/profile

  1. export  JAVA_HOME=/usr/java/default
  2. export HADOOP_HOME=/opt/bigdata/hadoop-2.6.5
  3. export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source /etc/profile
**

2：配置hadoop的角色：

cd $HADOOP_HOME/etc/hadoop
#必须给hadoop配置javahome要不ssh过去找不到
vi hadoop-env.sh

  1. export JAVA_HOME=/usr/java/default

vi core-site.xml

  1. <property>
  2.     <name>fs.defaultFS</name>
  3.     <value>hdfs://node01:9000</value>
  4. </property>

vi hdfs-site.xml

  1.     <property>
  2.         <name>dfs.replication</name>
  3.         <value>1</value>
  4.     </property>
  5.     <property>
  6.         <name>dfs.namenode.name.dir</name>
  7.         <value>/var/bigdata/hadoop/local/dfs/name</value>
  8.     </property>
  9.     <property>
  10.         <name>dfs.datanode.data.dir</name>
  11.         <value>/var/bigdata/hadoop/local/dfs/data</value>
  12.     </property>
  13.     <property>
  14.         <name>dfs.namenode.secondary.http-address</name>
  15.         <value>node01:50090</value>
  16.     </property>
  17.     <property>
  18.         <name>dfs.namenode.checkpoint.dir</name>
  19.         <value>/var/bigdata/hadoop/local/dfs/secondary</value>
  20.     </property>

vi slaves

  1. node01

3：格式化node(只需format一次)

hdfs namenode -format

4：启动

start-dfs.sh

三：FAQ

1：datanode未启动成功

访问192.168.0.200:50070发现datanode未启动

  1. 1:先查看namenode日志
  2. cat /opt/bigdata/hadoop-2.6.5/logs/hadoop-root-datanode-node01.log
  4. 2020-12-24 18:24:51,058 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /var/bigdata/hadoop/local/dfs/data: namenode clusterID = CID-c97578ff-5b0f-409a-8fd0-b285acb557e5; datanode clusterID = CID-808b44b6-6f31-4c00-ae8e-6f7627c537c7
  6. 2:原因,重启前后执行多次hdfs namenode -format ， 导致namenode clusterID和datanode clusterID不对应
  8. 3:解决
  9. 在hdfs-site.xml中找到
  10. <name>dfs.datanode.data.dir</name>
  11. <value>/var/bigdata/hadoop/local/dfs/data</value>
  12. 在/var/bigdata/hadoop/local/dfs/data/current下找到VERSION配置文件
  13. vi VERSION
  14. #修改clusterID为clusterID = CID-c97578ff-5b0f-409a-8fd0-b285acb557e5
  15. clusterID=CID-c97578ff-5b0f-409a-8fd0-b285acb557e5
  17. 4:重新启动