1、免密登陆W

  1. 生成公钥和私钥（生成路径是 /root/.ssh/ 下）：ssh-keygen -t rsa
  2. 拷贝公钥到同一台主机：ssh-copy-id 目标地址IP
  3. 这个命令也能实现拷贝公钥到其他机器：scp /root/.ssh/authorized_keys node1:/root/.ssh/

2、时钟同步

  1. 安装：yum -y install ntp
  2. 启动：systemctl start ntpd
  3. 开机自启动：systemctl enable ntpd
  4. 指定互联网的时间服务器，定时同步(每分钟一次)： */1 * * * * * /usr/sbin/ntpupdate ntp4.aliyun.com
  5. 设置本机时间： date -s "2021-08-15 12:00:00"
  7. 1、修改 /etc/ntp.conf 配置
  8. 新增：restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
  9. 表示允许192.168.0.0网段的机器访问该服务器的ntp
  10. 修改：注释掉server开头的配置，表示关闭连接互联网外部的时钟
  11. 新增：server 127.127.1.0
  12. fudge 127.127.1.0 stratum 10
  13. 表示当该节点网络丢失还可以采用本地时间为集群中的其他机器提供时间同步服务
  15. 2、修改 /etc/sysconfig/ntpd 配置
  16. 新增：SYNC_HWCLOCK=yes 
  17. 表示让硬件时间和系统时间一起同步
  19. 3、其他服务器关闭ntp服务，systemctl stop ntpd;systemctl disable ntpd
  20. 每分钟同步一次hadoop01的服务器：*/1 * * * * * /usr/sbin/ntpupdate hadoop01

3、文件同步

  1. 增量同步（只需要同步拷贝有更新的文件），不会像scp那样全量：
  2. rxync -av 源文件路径 用户名@目标IP:目标地址
  3. -a：表示规定拷贝
  4. -v：显示复制过程

4、Yarn常用命令

  1. #Yarn状态的查询，除了可以在hadoop103:8088页面查看外，还可以通过命令操作。常见的命令操作如下所示：
  2. #需求：执行WordCount案例，并用Yarn命令查看任务运行情况。
  4. #[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
  6. 1.5.1 yarn application查看任务
  7. （1）列出所有Application：
  8. [atguigu@hadoop102 hadoop-3.1.3]$ yarn application -list
  9. 2021-02-06 10:21:19,238 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  10. Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
  11.                 Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                           Tracking-URL
  13. （2）根据Application状态过滤：yarn application -list -appStates 
  14. （所有状态：ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED）
  15. [atguigu@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
  16. 2021-02-06 10:22:20,029 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  17. Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
  18.                 Application-Id        Application-Name        Application-Type          User         Queue                 State           Final-State           Progress                           Tracking-URL
  19. application_1612577921195_0001              word count               MAPREDUCE       atguigu       default              FINISHED             SUCCEEDED               100%    http://hadoop102:19888/jobhistory/job/job_1612577921195_0001
  21. （3）Kill掉Application：
  22. [atguigu@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1612577921195_0001
  23. 2021-02-06 10:23:48,530 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  24. Application application_1612577921195_0001 has already finished
  27. 1.5.2 yarn logs查看日志
  28. （1）查询Application日志：yarn logs -applicationId <ApplicationId>
  29. [atguigu@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1612577921195_0001
  31. （2）查询Container日志：yarn logs -applicationId <ApplicationId> -containerId <ContainerId> 
  32. [atguigu@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1612577921195_0001 -containerId container_1612577921195_0001_01_000001
  35. 1.5.3 yarn applicationattempt查看尝试运行的任务
  36. （1）列出所有Application尝试的列表：yarn applicationattempt -list <ApplicationId>
  37. [atguigu@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1612577921195_0001
  38. 2021-02-06 10:26:54,195 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  39. Total number of application attempts :1
  40.          ApplicationAttempt-Id                   State                        AM-Container-Id                           Tracking-URL
  41. appattempt_1612577921195_0001_000001                FINISHED    container_1612577921195_0001_01_000001    http://hadoop103:8088/proxy/application_1612577921195_0001/
  43. （2）打印ApplicationAttemp状态：yarn applicationattempt -status <ApplicationAttemptId>
  44. [atguigu@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1612577921195_0001_000001
  45. 2021-02-06 10:27:55,896 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  46. Application Attempt Report : 
  47.     ApplicationAttempt-Id : appattempt_1612577921195_0001_000001
  48.     State : FINISHED
  49.     AMContainer : container_1612577921195_0001_01_000001
  50.     Tracking-URL : http://hadoop103:8088/proxy/application_1612577921195_0001/
  51.     RPC Port : 34756
  52.     AM Host : hadoop104
  53.     Diagnostics :
  56. 1.5.4 yarn container查看容器
  57. （1）列出所有Container：yarn container -list <ApplicationAttemptId>
  58. [atguigu@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1612577921195_0001_000001
  59. 2021-02-06 10:28:41,396 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  60. Total number of containers :0
  61.                   Container-Id              Start Time             Finish Time                   State                    Host       Node Http Address    
  62. （2）打印Container状态：    yarn container -status <ContainerId>
  63. [atguigu@hadoop102 hadoop-3.1.3]$ yarn container -status container_1612577921195_0001_01_000001
  64. 2021-02-06 10:29:58,554 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  65. Container with id 'container_1612577921195_0001_01_000001' doesn't exist in RM or Timeline Server.
  66.     注：只有在任务跑的途中才能看到container的状态'
  69. 1.5.5 yarn node查看节点状态
  70. 列出所有节点：yarn node -list -all
  71. [atguigu@hadoop102 hadoop-3.1.3]$ yarn node -list -all
  72. 2021-02-06 10:31:36,962 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  73. Total Nodes:3
  74.          Node-Id         Node-State    Node-Http-Address    Number-of-Running-Containers
  75.  hadoop103:38168            RUNNING       hadoop103:8042                               0
  76.  hadoop102:42012            RUNNING       hadoop102:8042                               0
  77.  hadoop104:39702            RUNNING       hadoop104:8042                               0
  80. 1.5.6 yarn rmadmin更新配置
  81. 加载队列配置：yarn rmadmin -refreshQueues
  82. [atguigu@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues
  83. 2021-02-06 10:32:03,331 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8033
  86. 1.5.7 yarn queue查看队列
  87. 打印队列信息：yarn queue -status <QueueName>
  88. [atguigu@hadoop102 hadoop-3.1.3]$ yarn queue -status default
  89. 2021-02-06 10:32:33,403 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  90. Queue Information : 
  91. Queue Name : default
  92.     State : RUNNING
  93.     Capacity : 100.0%
  94.     Current Capacity : .0%
  95.     Maximum Capacity : 100.0%
  96.     Default Node Label expression : <DEFAULT_PARTITION>
  97.     Accessible Node Labels : *
  98.     Preemption : disabled
  99.     Intra-queue Preemption : disabled