image.png

集群机器

创建三台机器node101，node102，node103

附加组件

https://flink.apache.org/downloads.html
image.png

下载hadoop组件放到flink lib目录

image.png

启动集群-Hadoop

假设已经部署和启动好Hadoop集群

image.png

启动Flink-YARN-Session

  1. ## -n(--container)：TaskManager 的数量，可以不设置，由YARN分配资源
  2. ## -s(--slots)： 每个TaskManager的slot数量，默认1个core对应1个slot，我们的例子中，1个core对应2个slot
  3. ## -jm：JobManager 的内存（单位 MB)
  4. ## -tm：每个 taskmanager 的内存，单位 MB
  5. ## -nm：显示在yarn web ui的appName
  6. ## -d：在后台执行
  7. [java@node101 bin]$ ./yarn-session.sh -n 2 -s 2 -jm 1600 -tm 1024 -nm flinkapp -d
  8. --------------------------------------------------------------------------------------
  9. 2021-07-19 20:29:22,065 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.rpc.address, node101
  10. 2021-07-19 20:29:22,069 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.rpc.port, 6123
  11. 2021-07-19 20:29:22,069 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.memory.process.size, 1600m
  12. 2021-07-19 20:29:22,069 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.memory.process.size, 1728m
  13. 2021-07-19 20:29:22,070 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.numberOfTaskSlots, 2
  14. 2021-07-19 20:29:22,070 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: parallelism.default, 1
  15. 2021-07-19 20:29:22,070 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.execution.failover-strategy, region
  16. 2021-07-19 20:29:22,495 WARN  org.apache.hadoop.util.NativeCodeLoader                      [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  17. 2021-07-19 20:29:22,606 INFO  org.apache.flink.runtime.security.modules.HadoopModule       [] - Hadoop user set to java (auth:SIMPLE)
  18. 2021-07-19 20:29:22,638 INFO  org.apache.flink.runtime.security.modules.JaasModule         [] - Jaas file will be created as /tmp/jaas-56853801744925321.conf.
  19. 2021-07-19 20:29:22,663 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink-1.13.1/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
  20. 2021-07-19 20:29:22,735 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at node102/192.168.89.102:8032
  21. 2021-07-19 20:29:22,888 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
  22. 2021-07-19 20:29:22,910 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (172.800mb (181193935 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
  23. 2021-07-19 20:29:23,167 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
  24. 2021-07-19 20:29:23,302 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured JobManager memory is 1600 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 448 MB may not be used by Flink.
  25. 2021-07-19 20:29:23,302 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - The configured TaskManager memory is 1728 MB. YARN will allocate 2048 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 320 MB may not be used by Flink.
  26. 2021-07-19 20:29:23,306 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Cluster specification: ClusterSpecification{masterMemoryMB=1600, taskManagerMemoryMB=1728, slotsPerTaskManager=2}
  27. 2021-07-19 20:29:33,322 INFO  org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (160.000mb (167772162 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
  28. 2021-07-19 20:29:33,339 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1626694871823_0001
  29. 2021-07-19 20:29:34,194 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1626694871823_0001
  30. 2021-07-19 20:29:34,195 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated
  31. 2021-07-19 20:29:34,199 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED
  32. 2021-07-19 20:29:50,062 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - YARN application has been deployed successfully.
  33. 2021-07-19 20:29:50,064 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface node102:36065 of application 'application_1626694871823_0001'.
  34. JobManager Web Interface: http://node102:36065

查看YARN

  1. http://node102:8088/cluster/apps

image.png

image.png

NC监听

  1. [java@node101 bin]$ nc -lkv 7777
  2. Ncat: Version 7.70 ( https://nmap.org/ncat )
  3. Ncat: Listening on :::7777
  4. Ncat: Listening on 0.0.0.0:7777

提交任务

  1. [java@node101 bin]$ ./flink run -c online.javabook.flink.quickstart.SocketWordCount /opt/module/flink-jobs/javabook.apache.flink-1.0-SNAPSHOT.jar --host 192.168.89.101 --port 7777
  3. 2021-07-19 20:47:12,813 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-java.
  4. 2021-07-19 20:47:12,813 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Found Yarn properties file under /tmp/.yarn-properties-java.
  5. 2021-07-19 20:47:13,357 WARN  org.apache.flink.yarn.configuration.YarnLogConfigUtil        [] - The configuration directory ('/opt/module/flink-1.13.1/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
  6. 2021-07-19 20:47:13,463 INFO  org.apache.hadoop.yarn.client.RMProxy                        [] - Connecting to ResourceManager at node102/192.168.89.102:8032
  7. 2021-07-19 20:47:13,621 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
  8. 2021-07-19 20:47:13,623 WARN  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
  9. 2021-07-19 20:47:13,745 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface node103:33459 of application 'application_1626694871823_0002'.
  10. Job has been submitted with JobID e5206edf1f6902c7cc7906c56fb2b1fb

杀掉任务

[java@node101 bin]$ yarn application --kill [Yarn-applicatinId]