集群规划

	linux102	linux103	linux104
HDFS	NameNode DataNode	DataNode	SecondaryNameNode DataNode
YARN	NodeManager	ResourceManager NodeManager	NodeManager

配置集群

配置文件在$HADOOP_HOME/etc/hadoop

| 核心配置文件
core-site.xml | ```xml <?xml version=”1.0” encoding=”UTF-8”?> <?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

fs.defaultFS hdfs://linux102:8020 hadoop.tmp.dir /opt/module/hadoop-3.1.3/data hadoop.http.staticuser.user linux

 |
| --- | --- |
| HDFS配置文件<br />`hdfs-site.xml` | ```xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- nn web 端访问地址-->
    <property>
         <name>dfs.namenode.http-address</name>
         <value>linux102:9870</value>
     </property>
    <!-- 2nn web 端访问地址-->
     <property>
         <name>dfs.namenode.secondary.http-address</name>
         <value>linux104:9868</value>
     </property>
</configuration>

| | YARN配置文件
yarn-site.xml | ```xml <?xml version=”1.0”?>

<!-- 指定 MR 走 shuffle -->
 <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
 </property>

<!-- 指定 ResourceManager 的地址-->
 <property>
     <name>yarn.resourcemanager.hostname</name>
     <value>linux103</value>
 </property>

 <!-- 环境变量的继承 -->
 <property>
     <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
 </property>

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<!-- 设置日志聚集服务器地址 -->
<property>
    <name>yarn.log.server.url</name>
    <value>http://linux102:19888/jobhistory/logs</value>
</property>

<!-- 设置日志保留时间为7天 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>

 |
| MapReduce配置文件<br />`mapred-site.xml` | ```xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <!-- 指定MapReduce程序运行在Yarn上 -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

    <!-- 历史服务器端地址 -->
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>linux102:10020</value>
    </property>

    <!-- 历史服务器web地址 -->
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>linux102:19888</value>
    </property>

</configuration>

| | 配置workers
$HADOOP_HOME/etc/hadoop/workers | ```java linux102 linux103 linux104

注意：新增数据时，内容结尾不需要有空格，不能有空行 |



<a name="GTgMq"></a>
# 集群启动/暂停
> 如果集群是第一次启动，需要在linux102节点格式化NameNode（注意：格式化NameNode，会产生新的集群id，导致NameNode和DataNode的集群id不一致，集群找不到以往的数据。如果集群在运行过程中报错，需要重新格式化NameNode的话，一定要先停止namenode和datanode进程，并且要删除所有机器上的data和logs目录，然后再进行格式化。）
> `hdfs namenode -format`

| 模块 | 启动 | 暂停 |
| --- | --- | --- |
| HDFS | `$HADOOP_HOME/sbin/start-dfs.sh` | `$HADOOP_HOME/sbin/stop-dfs.sh` |
| YARN | `$HADOOP_HOME/sbin/start-yarn.sh` | `$HADOOP_HOME/sbin/stop-yarn.sh` |
| 服务组件 | 启动 | 停止 |
| HDFS | `hdfs --daemon start namenode/datanode/secondarynamenode` | `hdfs --daemon stop namenode/datanode/secondarynamenode` |
| YARN | `yarn --daemon start resourcemanager/nodemanager` | `yarn --daemon stop resourcemanamger/nodemanamger` |
| HistoryServer | `mapred --daemon stop historyserver`<br />在指定的机器上执行 | `mapred --deamon stop historyserver` |
| 集群脚本 | ```shell
#!/bin/bash

if [ $# -lt 1 ]
then 
    echo "No Args Input..."
    exit
fi

case $1 in
"start")
    echo "======================== 启动 hadoop 集群 ========================"

    echo "--------------- 启动 hdfs ---------------"
    ssh linux102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"
    echo "--------------- 启动 yarn ---------------"
    ssh linux103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"
    echo "--------------- 启动 historyserver ---------------"
    ssh linux102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver"
;;
"stop")
    echo "======================== 关闭 hadoop 集群 ========================"

  echo "--------------- 关闭 historyserver ---------------"
  ssh linux102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
  echo "--------------- 关闭 yarn ---------------"
  ssh linux103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"
  echo "--------------- 关闭 hdfs ---------------"
  ssh linux102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;;
*)
    echo "Input Args Error..."
;;
esac

| |

大数据

A-hadoop集群规划

集群规划

配置集群