1、免密登陆W

  1. 生成公钥和私钥(生成路径是 /root/.ssh/ 下):ssh-keygen -t rsa
  2. 拷贝公钥到同一台主机:ssh-copy-id 目标地址IP
  3. 这个命令也能实现拷贝公钥到其他机器:scp /root/.ssh/authorized_keys node1:/root/.ssh/

2、时钟同步

  1. 安装:yum -y install ntp
  2. 启动:systemctl start ntpd
  3. 开机自启动:systemctl enable ntpd
  4. 指定互联网的时间服务器,定时同步(每分钟一次): */1 * * * * * /usr/sbin/ntpupdate ntp4.aliyun.com
  5. 设置本机时间: date -s "2021-08-15 12:00:00"
  6. 1、修改 /etc/ntp.conf 配置
  7. 新增:restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
  8. 表示允许192.168.0.0网段的机器访问该服务器的ntp
  9. 修改:注释掉server开头的配置,表示关闭连接互联网外部的时钟
  10. 新增:server 127.127.1.0
  11. fudge 127.127.1.0 stratum 10
  12. 表示当该节点网络丢失还可以采用本地时间为集群中的其他机器提供时间同步服务
  13. 2、修改 /etc/sysconfig/ntpd 配置
  14. 新增:SYNC_HWCLOCK=yes
  15. 表示让硬件时间和系统时间一起同步
  16. 3、其他服务器关闭ntp服务,systemctl stop ntpd;systemctl disable ntpd
  17. 每分钟同步一次hadoop01的服务器:*/1 * * * * * /usr/sbin/ntpupdate hadoop01

3、文件同步

  1. 增量同步(只需要同步拷贝有更新的文件),不会像scp那样全量:
  2. rxync -av 源文件路径 用户名@目标IP:目标地址
  3. -a:表示规定拷贝
  4. -v:显示复制过程

4、Yarn常用命令

  1. #Yarn状态的查询,除了可以在hadoop103:8088页面查看外,还可以通过命令操作。常见的命令操作如下所示:
  2. #需求:执行WordCount案例,并用Yarn命令查看任务运行情况。
  3. #[atguigu@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
  4. 1.5.1 yarn application查看任务
  5. 1)列出所有Application
  6. [atguigu@hadoop102 hadoop-3.1.3]$ yarn application -list
  7. 2021-02-06 10:21:19,238 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  8. Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
  9. Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
  10. 2)根据Application状态过滤:yarn application -list -appStates
  11. (所有状态:ALLNEWNEW_SAVINGSUBMITTEDACCEPTEDRUNNINGFINISHEDFAILEDKILLED
  12. [atguigu@hadoop102 hadoop-3.1.3]$ yarn application -list -appStates FINISHED
  13. 2021-02-06 10:22:20,029 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  14. Total number of applications (application-types: [], states: [FINISHED] and tags: []):1
  15. Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
  16. application_1612577921195_0001 word count MAPREDUCE atguigu default FINISHED SUCCEEDED 100% http://hadoop102:19888/jobhistory/job/job_1612577921195_0001
  17. 3KillApplication
  18. [atguigu@hadoop102 hadoop-3.1.3]$ yarn application -kill application_1612577921195_0001
  19. 2021-02-06 10:23:48,530 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  20. Application application_1612577921195_0001 has already finished
  21. 1.5.2 yarn logs查看日志
  22. 1)查询Application日志:yarn logs -applicationId <ApplicationId>
  23. [atguigu@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1612577921195_0001
  24. 2)查询Container日志:yarn logs -applicationId <ApplicationId> -containerId <ContainerId>
  25. [atguigu@hadoop102 hadoop-3.1.3]$ yarn logs -applicationId application_1612577921195_0001 -containerId container_1612577921195_0001_01_000001
  26. 1.5.3 yarn applicationattempt查看尝试运行的任务
  27. 1)列出所有Application尝试的列表:yarn applicationattempt -list <ApplicationId>
  28. [atguigu@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -list application_1612577921195_0001
  29. 2021-02-06 10:26:54,195 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  30. Total number of application attempts :1
  31. ApplicationAttempt-Id State AM-Container-Id Tracking-URL
  32. appattempt_1612577921195_0001_000001 FINISHED container_1612577921195_0001_01_000001 http://hadoop103:8088/proxy/application_1612577921195_0001/
  33. 2)打印ApplicationAttemp状态:yarn applicationattempt -status <ApplicationAttemptId>
  34. [atguigu@hadoop102 hadoop-3.1.3]$ yarn applicationattempt -status appattempt_1612577921195_0001_000001
  35. 2021-02-06 10:27:55,896 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  36. Application Attempt Report :
  37. ApplicationAttempt-Id : appattempt_1612577921195_0001_000001
  38. State : FINISHED
  39. AMContainer : container_1612577921195_0001_01_000001
  40. Tracking-URL : http://hadoop103:8088/proxy/application_1612577921195_0001/
  41. RPC Port : 34756
  42. AM Host : hadoop104
  43. Diagnostics :
  44. 1.5.4 yarn container查看容器
  45. 1)列出所有Containeryarn container -list <ApplicationAttemptId>
  46. [atguigu@hadoop102 hadoop-3.1.3]$ yarn container -list appattempt_1612577921195_0001_000001
  47. 2021-02-06 10:28:41,396 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  48. Total number of containers :0
  49. Container-Id Start Time Finish Time State Host Node Http Address
  50. 2)打印Container状态: yarn container -status <ContainerId>
  51. [atguigu@hadoop102 hadoop-3.1.3]$ yarn container -status container_1612577921195_0001_01_000001
  52. 2021-02-06 10:29:58,554 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  53. Container with id 'container_1612577921195_0001_01_000001' doesn't exist in RM or Timeline Server.
  54. 注:只有在任务跑的途中才能看到container的状态'
  55. 1.5.5 yarn node查看节点状态
  56. 列出所有节点:yarn node -list -all
  57. [atguigu@hadoop102 hadoop-3.1.3]$ yarn node -list -all
  58. 2021-02-06 10:31:36,962 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  59. Total Nodes:3
  60. Node-Id Node-State Node-Http-Address Number-of-Running-Containers
  61. hadoop103:38168 RUNNING hadoop103:8042 0
  62. hadoop102:42012 RUNNING hadoop102:8042 0
  63. hadoop104:39702 RUNNING hadoop104:8042 0
  64. 1.5.6 yarn rmadmin更新配置
  65. 加载队列配置:yarn rmadmin -refreshQueues
  66. [atguigu@hadoop102 hadoop-3.1.3]$ yarn rmadmin -refreshQueues
  67. 2021-02-06 10:32:03,331 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8033
  68. 1.5.7 yarn queue查看队列
  69. 打印队列信息:yarn queue -status <QueueName>
  70. [atguigu@hadoop102 hadoop-3.1.3]$ yarn queue -status default
  71. 2021-02-06 10:32:33,403 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
  72. Queue Information :
  73. Queue Name : default
  74. State : RUNNING
  75. Capacity : 100.0%
  76. Current Capacity : .0%
  77. Maximum Capacity : 100.0%
  78. Default Node Label expression : <DEFAULT_PARTITION>
  79. Accessible Node Labels : *
  80. Preemption : disabled
  81. Intra-queue Preemption : disabled