1、安装准备:
下载安装介质:
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari-2.7.3.0-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/HDP-3.1.0.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.0.0/HDP-GPL-3.1.0.0-centos7-gpl.tar.gz
2、配置Linux主机域名(所有服务器)
执行:
vim /etc/hosts
添加如下内容:
10.10.195.53 bigdata-1.mid.neu.com bd1
10.10.195.54 bigdata-2.mid.neu.com bd2
10.10.195.55 bigdata-3.mid.neu.com bd3
10.10.195.56 bigdata-4.mid.neu.com bd4
测试:
ping bd1
ping bd2
ping bd3
ping bd4
3、解除 Linux 系统限制(所有服务器)
执行:
vim /etc/security/limits.conf
添加如下内容:
soft nofile 32768
hard nofile 1048576
soft nproc 65536
hard nproc unlimited
hard memlock unlimited
soft memlock unlimited
执行:
vim /etc/profile
添加如下内容:
ulimit -u 65535
ulimit -n 65535
执行:
source /etc/profile
4、关闭防火墙和selinux(所有服务器)
a)关闭防火墙:
查看防火墙状态:
firewall-cmd —state
临时停止firewall:
systemctl stop firewalld.service
永久禁止开启启动firewall:
systemctl disable firewalld.service
b)关闭SELinux
查看:
getenforce 或者 /usr/sbin/sestatus -v
临时禁用:
setenforce 0
永久禁用:
执行:
vim /etc/selinux/config
将SELINUX=enforce改成SELINUX=disabled
保存,重启
5、时钟同步(所有服务器)
NTP服务设置
节点1配置:
执行:
vim /etc/ntp.conf
注释掉原先的默认配置:
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
添加如下配置(配置时钟同步地址为时钟同步服务器):
server 10.10.195.106
fudge 10.10.195.106 stratum 2
restrict 127.0.0.1
logfile /var/log/ntp.log
重启ntp服务
systemctl restart ntpd.service
其他节点配置:
执行:
vim /etc/ntp.conf
注释掉原先的默认配置:
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
添加如下配置:
server 10.10.195.53
fudge 10.10.195.106 stratum 3
restrict 127.0.0.1
logfile /var/log/ntp.log
重启ntp服务
systemctl restart ntpd.service
查看ntp状态:
ntpq -p
开启自启动:
systemctl enable ntpd
6、配置swap:(所有服务器)
echo vm.swappiness = 10 >> /etc/sysctl.conf
sysctl -p
7、配置内存透明大页面:(所有服务器)
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
将如下脚本添加到/etc/rc.d/rc.local文件中
vim /etc/rc.d/rc.local
添加:
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
8、免密码登陆:(master服务器)
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub bigdata-1.mid.neu.com
ssh-copy-id -i ~/.ssh/id_rsa.pub bigdata-2.mid.neu.com
ssh-copy-id -i ~/.ssh/id_rsa.pub bigdata-3.mid.neu.com
ssh-copy-id -i ~/.ssh/id_rsa.pub bigdata-4.mid.neu.com
cd .ssh/
scp id_rsa id_rsa.pub bigdata-2.mid.neu.com:/root/.ssh/
scp id_rsa id_rsa.pub bigdata-3.mid.neu.com:/root/.ssh/
scp id_rsa id_rsa.pub bigdata-4.mid.neu.com:/root/.ssh/
9、安装mysql并配置驱动:(master服务器)
卸载centos7原先自带的mariadb
rpm -qa|grep mariadb
rpm -e —nodeps mariadb-libs
rpm -e —nodeps mariadb-devel
rpm -e —nodeps mariadb-server
rpm -e —nodeps mariadb
按照顺序执行如下命令:
rpm -ivh mysql-community-common-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.17-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.17-1.el7.x86_64.rpm
启动mysql:
systemctl start mysqld.service
systemctl enable mysqld
查找初始密码:
grep ‘temporary password’ /var/log/mysqld.log
登陆mysql:
mysql -uroot -pRoot,123
修改初始密码,并允许其他服务器登陆:
create database ambari;
create database hive;
create database oozie;
set global validate_password_policy=0;
set global validate_password_length=1;
set password = password(‘root123’);
update mysql.user set host=”%” where Host=’localhost’ and user = “root”;
flush privileges;
配置驱动:
mkdir /usr/share/java
mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
chmod +x /usr/share/java/mysql-connector-java.jar
10、安装jdk:(master服务器)
解压缩:
tar -xvf jdk-8u211-linux-x64.tar -C /opt
设置环境变量:
vim /etc/profile
添加如下内容:
export JAVA_HOME=/opt/jdk1.8.0_211
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
生效:
source /etc/profile
验证:
java -version
复制到其他节点:
scp -r /opt/jdk1.8.0_211/ bd2:/opt
scp -r /opt/jdk1.8.0_211/ bd3:/opt
scp -r /opt/jdk1.8.0_211/ bd4:/opt
其他节点设置环境变量:
vim /etc/profile
添加如下内容:
export JAVA_HOME=/opt/jdk1.8.0_211
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
生效:
source /etc/profile
验证:
java -version
11、配置本地yum源:(master服务器)
注意:需要操作系统光盘软件安装源,请自行准备,并复制到/var/www/html/目录下,如centos
启动httpd:
systemctl start httpd
解压缩:
tar -xvf ambari-2.7.3.0-centos7.tar -C /var/www/html/
tar -xvf HDP-3.1.0.0-centos7-rpm.tar -C /var/www/html/
tar -xvf HDP-UTILS-1.1.0.22-centos7.tar -C /var/www/html/
tar -xvf HDP-GPL-3.1.0.0-centos7-gpl.tar -C /var/www/html/
验证:
http://10.10.195.53/ambari/
配置yum:
vim /etc/yum.repos.d/ambari.repo
填写:
[ambari-repo]
name=ambari_repo
baseurl=http://bigdata-1.mid.neu.com/ambari/centos7/2.7.3.0-139/
enable=true
gpgcheck=false
vim /etc/yum.repos.d/hdp.repo
[hdp-repo]
name=hdp_repo
baseurl=http://bigdata-1.mid.neu.com/HDP/centos7/3.1.0.0-78/
enable=true
gpgcheck=false
vim /etc/yum.repos.d/hdp-utils.repo
[hdp-utils-repo]
name=hdp-utils_repo
baseurl=http://bigdata-1.mid.neu.com/HDP-UTILS/centos7/1.1.0.22/
enable=true
gpgcheck=false
vim /etc/yum.repos.d/hdp-gpl.repo
[hdp-gpl-repo]
name=hdp-gpl_repo
baseurl=http://bigdata-1.mid.neu.com/HDP-GPL/centos7/3.1.0.0-78/
enable=true
gpgcheck=false
[centos-repo]
name=centos_repo
baseurl=http://bigdata-1.mid.neu.com/centos/
enable=true
gpgcheck=false
移走旧文件:
mkdir /etc/yum.repos.d/bak
mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d/bak
验证:
yum repolist
scp /etc/yum.repos.d/ambari.repo bd2:/etc/yum.repos.d
scp /etc/yum.repos.d/ambari.repo bd3:/etc/yum.repos.d
scp /etc/yum.repos.d/ambari.repo bd4:/etc/yum.repos.d
scp /etc/yum.repos.d/hdp.repo bd2:/etc/yum.repos.d
scp /etc/yum.repos.d/hdp.repo bd3:/etc/yum.repos.d
scp /etc/yum.repos.d/hdp*.repo bd4:/etc/yum.repos.d
scp /etc/yum.repos.d/centos.repo bd2:/etc/yum.repos.d
scp /etc/yum.repos.d/centos.repo bd3:/etc/yum.repos.d
scp /etc/yum.repos.d/centos.repo bd4:/etc/yum.repos.d
11—-安装libtirpc-devel
下载:libtirpc-devel
http://mirror.centos.org/centos/7/os/x86_64/Packages/libtirpc-0.2.4-0.16.el7.x86_64.rpm
http://mirror.centos.org/centos/7/os/x86_64/Packages/libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm
rpm -e libtirpc —nodeps
rpm -ivh libtirpc-0.2.4-0.16.el7.x86_64.rpm
rpm -ivh libtirpc-devel-0.2.4-0.16.el7.x86_64.rpm
12、安装ambari:
yum -y install ambari-server
初始化mysql:
mysql -uroot -pRoot,123
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
配置ambari:
ambari-server setup
[root@c1 ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux… SELinux status is ‘disabled’ Customize user account for ambari-server daemon [y/n] (n)? Adjusting ambari-server permissions and ownership… Checking firewall status… Checking JDK… Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /opt/jdk1.8.0_211
Validating JDK on Ambari Server…done.
Check JDK version for Ambari Server…
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement…
Completing setup…
Configuring database…
Enter advanced database configuration [y/n] (n)? y
Configuring database…
Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere
[7] - BDB
Enter choice (3): 3 Hostname (localhost): Port (3306): Database name (ambari): Username (bigdata): bigdata Enter Database Password (123456789): Configuring ambari database… Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y Configuring remote database connection properties… WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views… ambari-admin-2.7.3.0.139.jar …. Ambari repo file doesn’t contain latest json url, skipping repoinfos modification Adjusting ambari-server permissions and ownership… Ambari Server ‘setup’ completed successfully.
启动:
ambari-server start
访问http://10.10.195.53:8080 用户名密码:admin/admin 开始配置
详见文件:/root/.ssh/id_rsa
ambari-server setup —jdbc-db=mysql —jdbc-driver=/usr/share/java/mysql-connector-java.jar
常见问题:
原因:YARN Registry DNS的默认端口53本其它进程占用了。
方案一:
1.查询占用53端口的进程/服务
lsof -i:53
查询到PID和User,这里的User一般是systemd-resolved
2.停止占用53端口的进程/服务
systemctl stop systemd-resolved
3.在Ambari界面启动YARN Registry DNS,成功。
方案二:
直接在Ambari界面,YARN -> configs -> advanced -> RegistryDNS Bind Port,修改端口后保存。
问题:
解决:
检查mysql字符集:
show variables like “character_set_database”;
是否是latin1,如果是,则更新为utf8。
alter database druid character set utf8 collate utf8_general_ci;
重启Druid coordinator。
