1、免密登陆W
生成公钥和私钥(生成路径是 /root/.ssh/ 下):ssh-keygen -t rsa
拷贝公钥到同一台主机:ssh-copy-id 目标地址IP
这个命令也能实现拷贝公钥到其他机器:scp /root/.ssh/authorized_keys node1:/root/.ssh/
2、时钟同步
安装:yum -y install ntp
启动:systemctl start ntpd
开机自启动:systemctl enable ntpd
指定互联网的时间服务器,定时同步(每分钟一次): */1 * * * * * /usr/sbin/ntpupdate ntp4.aliyun.com
设置本机时间: date -s "2021-08-15 12:00:00"
1、修改 /etc/ntp.conf 配置
新增:restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
表示允许192.168.0.0网段的机器访问该服务器的ntp
修改:注释掉server开头的配置,表示关闭连接互联网外部的时钟
新增:server 127.127.1.0
fudge 127.127.1.0 stratum 10
表示当该节点网络丢失还可以采用本地时间为集群中的其他机器提供时间同步服务
2、修改 /etc/sysconfig/ntpd 配置
新增:SYNC_HWCLOCK=yes
表示让硬件时间和系统时间一起同步
3、其他服务器关闭ntp服务,systemctl stop ntpd;systemctl disable ntpd
每分钟同步一次hadoop01的服务器:*/1 * * * * * /usr/sbin/ntpupdate hadoop01
3、文件同步
增量同步(只需要同步拷贝有更新的文件),不会像scp那样全量:
rxync -av 源文件路径 用户名@目标IP:目标地址
-a:表示规定拷贝
-v:显示复制过程
4、Yarn常用命令
#Yarn状态的查询,除了可以在hadoop103:8088页面查看外,还可以通过命令操作。常见的命令操作如下所示:
#需求:执行WordCount案例,并用Yarn命令查看任务运行情况。
#[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
1.5.1 yarn application查看任务
(1)列出所有Application:
[atguigu@hadoop102 hadoop-3.1.3]$ yarn application -list
2021-02-06 10:21:19,238 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
(2)根据Application状态过滤:yarn application -list -appStates
(所有状态:ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED)
[atguigu@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
2021-02-06 10:22:20,029 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
application_1612577921195_0001 word count MAPREDUCE atguigu default FINISHED SUCCEEDED 100% http://hadoop102:19888/jobhistory/job/job_1612577921195_0001
(3)Kill掉Application:
[atguigu@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1612577921195_0001
2021-02-06 10:23:48,530 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Application application_1612577921195_0001 has already finished
1.5.2 yarn logs查看日志
(1)查询Application日志:yarn logs -applicationId <ApplicationId>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1612577921195_0001
(2)查询Container日志:yarn logs -applicationId <ApplicationId> -containerId <ContainerId>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1612577921195_0001 -containerId container_1612577921195_0001_01_000001
1.5.3 yarn applicationattempt查看尝试运行的任务
(1)列出所有Application尝试的列表:yarn applicationattempt -list <ApplicationId>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1612577921195_0001
2021-02-06 10:26:54,195 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of application attempts :1
ApplicationAttempt-Id State AM-Container-Id Tracking-URL
appattempt_1612577921195_0001_000001 FINISHED container_1612577921195_0001_01_000001 http://hadoop103:8088/proxy/application_1612577921195_0001/
(2)打印ApplicationAttemp状态:yarn applicationattempt -status <ApplicationAttemptId>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1612577921195_0001_000001
2021-02-06 10:27:55,896 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Application Attempt Report :
ApplicationAttempt-Id : appattempt_1612577921195_0001_000001
State : FINISHED
AMContainer : container_1612577921195_0001_01_000001
Tracking-URL : http://hadoop103:8088/proxy/application_1612577921195_0001/
RPC Port : 34756
AM Host : hadoop104
Diagnostics :
1.5.4 yarn container查看容器
(1)列出所有Container:yarn container -list <ApplicationAttemptId>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1612577921195_0001_000001
2021-02-06 10:28:41,396 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total number of containers :0
Container-Id Start Time Finish Time State Host Node Http Address
(2)打印Container状态: yarn container -status <ContainerId>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn container -status container_1612577921195_0001_01_000001
2021-02-06 10:29:58,554 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Container with id 'container_1612577921195_0001_01_000001' doesn't exist in RM or Timeline Server.
注:只有在任务跑的途中才能看到container的状态'
1.5.5 yarn node查看节点状态
列出所有节点:yarn node -list -all
[atguigu@hadoop102 hadoop-3.1.3]$ yarn node -list -all
2021-02-06 10:31:36,962 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Total Nodes:3
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
hadoop103:38168 RUNNING hadoop103:8042 0
hadoop102:42012 RUNNING hadoop102:8042 0
hadoop104:39702 RUNNING hadoop104:8042 0
1.5.6 yarn rmadmin更新配置
加载队列配置:yarn rmadmin -refreshQueues
[atguigu@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues
2021-02-06 10:32:03,331 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8033
1.5.7 yarn queue查看队列
打印队列信息:yarn queue -status <QueueName>
[atguigu@hadoop102 hadoop-3.1.3]$ yarn queue -status default
2021-02-06 10:32:33,403 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
Queue Information :
Queue Name : default
State : RUNNING
Capacity : 100.0%
Current Capacity : .0%
Maximum Capacity : 100.0%
Default Node Label expression : <DEFAULT_PARTITION>
Accessible Node Labels : *
Preemption : disabled
Intra-queue Preemption : disabled