一、Hadoop

创建hadoop用户

  1. sudo useradd -m hadoop -s /bin/bash

设置hadoop用户密码（输入两次）

  1. sudo passwd hadoop

添加权限

  1. sudo adduser hadoop sudo

换Mirror源

  1. sed -i 's/security.ubuntu/mirrors.aliyun/g' /etc/apt/sources.list
  2. sed -i 's/archive.ubuntu/mirrors.aliyun/g' /etc/apt/sources.list

切换到hadoop用户(这里要输入刚刚设置的hadoop密码)

  1. su hadoop

更新源缓存

  1. sudo apt-get update -y

更新软件

  1. sudo apt-get upgrade -y

安装ssh服务

  1. sudo apt-get install openssh-server -y

重新生成 key

  1. rm -r /etc/ssh/ssh*key
  3. dpkg-reconfigure openssh-server

启动ssh服务

  1. sudo service ssh start

注意：每次重启wsl时都需要启动ssh服务

配置ssh免密登录

  1. mkdir -p ~/.ssh/
  3. cd ~/.ssh/
  5. ssh-keygen -t rsa -P ""

ssh-keygen执行后一直回车至结束

  1. cat ./id_rsa.pub >> ./authorized_keys

配置hosts文件

切换至root用户

  1. su root

添加hosts记录

  1. sudo echo '
  2. 127.0.0.1 Master' >> /etc/hosts

退回hadoop用户

  1. exit

安装JDK8

  1. sudo apt-get install openjdk-8-jdk -y

配置Java环境

  1. echo '
  2. export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> ~/.bashrc
  4. source ~/.bashrc

安装Maven

  1. sudo apt-get install maven -y

下载hadoop

  1. cd ~
  3. wget https://mirrors.huaweicloud.com/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz

解压hadoop

  1. sudo tar -zxf hadoop-3.2.1.tar.gz -C /usr/local

重命名hadoop目录

  1. cd /usr/local/
  3. sudo mv ./hadoop-3.2.1/ ./hadoop

授权hadoop目录

  1. sudo chown -R hadoop ./hadoop

验证hadoop

  1. cd /usr/local/hadoop
  2. ./bin/hadoop version

配置hadoop环境变量

  1. echo '
  2. export HADOOP_HOME=/usr/local/hadoop
  3. export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin' >> ~/.bashrc
  5. source ~/.bashrc

创建hadoop相关文件目录（为后面的xml做准备）

  1. mkdir -p /usr/local/hadoop/dfs/name && \
  2. mkdir -p /usr/local/hadoop/dfs/data && \
  3. mkdir -p /usr/local/hadoop/dfs/tmp && \
  4. mkdir -p /usr/local/hadoop/tmp

配置hadoop的java环境变量，在hadoop-env.sh和yarn-env.sh的首行都写入JAVA_HOME路径

  1. sed -i '1iexport JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' /usr/local/hadoop/etc/hadoop/hadoop-env.sh && \
  2. sed -i '1iexport JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' /usr/local/hadoop/etc/hadoop/yarn-env.sh

修改hadoop配置

core-site.xml

  1. vi /usr/local/hadoop/etc/hadoop/core-site.xml

core-site.xml内容如下：

  1. <configuration>
  2.     <property>
  3.         <name>fs.default.name</name>
  4.         <value>hdfs://Master:9000</value>
  5.     </property>
  7.     <property>
  8.         <name>hadoop.tmp.dir</name>
  9.         <value>/usr/local/hadoop/tmp</value>
  10.     </property>
  11. </configuration>

hdfs-site.xml

vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml

hdfs-site.xml内容如下：

<configuration>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>Master:9001</value>
    </property>

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/dfs/name</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/dfs/data</value>
    </property>

    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

mapred-site.xml

vi /usr/local/hadoop/etc/hadoop/mapred-site.xml

mapred-site.xml内容如下：

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

yarn-site.xml

vi /usr/local/hadoop/etc/hadoop/yarn-site.xml

yarn-site.xml内容如下：

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>Master</value>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>  
    </property>

    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
</configuration>

执行 NameNode 的格式化

hdfs namenode -format

启动hadoop

start-all.sh
mr-jobhistory-daemon.sh start historyserver

关闭hadoop

stop-all.sh
mr-jobhistory-daemon.sh stop historyserver

二、Hbase

下载hbase

cd ~
wget https://mirrors.huaweicloud.com/apache/hbase/2.2.6/hbase-2.2.6-bin.tar.gz

解压hhbase

sudo tar -zxf hbase-2.2.6-bin.tar.gz -C /usr/local

重命名hbase

cd /usr/local
sudo mv hbase-2.2.6/ hbase

授权hbase目录

sudo chown -R hadoop ./hbase

配置hbase环境变量

echo '
export PATH=$PATH:/usr/local/hbase/bin' >> ~/.bashrc

source ~/.bashrc

配置hbase

echo '
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HBASE_MANAGES_ZK=true
' >> /usr/local/hbase/conf/hbase-env.sh

vi /usr/local/hbase/conf/hbase-site.xml


<configuration>
    <property>
        <name>hbase.unsafe.stream.capability.enforce</name>
        <value>false</value>
    </property>
    <property>
        <name>hbase.rootdir</name>
        <value>file:///usr/local/hbase/hbase-tmp</value>
    </property>
</configuration>

三、Spark

下载scala

cd ~
wget https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz

解压scala

sudo tar -zxf scala-2.11.12.tgz -C /usr/local

重命名scala目录

sudo mv /usr/local/scala-2.11.12/ /usr/local/scala

授权scala目录

sudo chown -R hadoop /usr/local/scala/

配置scala环境变量

echo '
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin' >> ~/.bashrc
source ~/.bashrc

验证scala

scala -version

下载spark

cd ~
wget https://mirrors.bfsu.edu.cn/apache/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz

解压spark

sudo tar -zxf spark-3.1.1-bin-hadoop3.2.tgz -C /usr/local

重命名spark目录

sudo mv /usr/local/spark-3.1.1-bin-hadoop3.2/ /usr/local/spark

授权spark目录

sudo chown -R hadoop /usr/local/spark/

配置spark环境变量

echo '
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin' >> ~/.bashrc

source ~/.bashrc

验证spark

spark-shell

输入:quit可退出spark-shell

配置spark

cd /usr/local/spark/conf

cp spark-env.sh.template spark-env.sh
echo '
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

HADOOP_HOME=/usr/local/hadoop
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

SCALA_HOME=/usr/local/scala

SPARK_HOME=/usr/local/spark
SPARK_MASTER_IP=master
SPARK_EXECUTOR_MEMORY=1G
' >> spark-env.sh

启动hadoop

start-all.sh

启动Spark

（注意./，因为spark的很多脚本和hadoop名称一样，所以这里没有为spark的脚本添加环境变量）

cd /usr/local/spark/sbin
./start-all.sh

测试spark

cd /usr/local/spark/bin
./run-example SparkPi 2>&1 | grep "Pi is"