一、前提条件

  • jdk8+
  • zookeeper
  • hadoop on yarn搭建完成


hadoop on yarn 搭建参考https://www.yuque.com/docs/share/d152d0a8-1fef-41f7-ba82-81372f263764?#

二、修改yarn配置

2.1 停止hdfs和yarn

2.2 vim yarn-site.xml

  1. <property>  
  2.     <name>yarn.nodemanager.vmem-check-enabled</name>  
  3.     <value>false</value>  
  4. </property> 
  5. <property>
  6.     <name>yarn.scheduler.minimum-allocation-mb</name>
  7.     <value>1024</value>
  8.   </property>
  9.   <property>
  10.     <name>yarn.scheduler.maximum-allocation-mb</name>
  11.     <value>4096</value>
  12.   </property>
  13.   <property>
  14.     <name>yarn.nodemanager.resource.memory-mb</name>
  15.     <value>4096</value>
  16.   </property>
  17. <property>
  18.        <name>yarn.nodemanager.resource.cpu-vcores</name>
  19.        <value>3</value>
  20.     </property>
  21. <property>
  22.   <name>yarn.resourcemanager.am.max-attempts</name>
  23.   <value>4</value>
  24.   <description>
  25.     The maximum number of application master execution attempts.
  26.   </description>
  27. </property>

2.3 vim mapred-site.xml

  1. <property>
  2.         <name>mapreduce.map.memory.mb</name>
  3.         <value>4096</value>
  4. </property>
  5. <property>
  6.         <name>mapreduce.reduce.memory.mb</name>
  7.         <value>4096</value>
  8. </property>
  9. <property>
  10.         <name>mapreduce.map.java.opts</name>
  11.         <value>4096</value>
  12. </property>
  13. <property>
  14.        <name>mapreduce.reduce.java.opts</name>
  15.        <value>4096</value>
  16. </property>

2.4 重启hdfs和yarn

三、解压安装flink

3.1 下载

http://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.7.2/flink-1.7.2-bin-hadoop28-scala_2.11.tgz
**

3.2 解压

tar zxvf flink-1.7.2-bin-hadoop28-scala_2.11.tgz -C /opt
**

3.3 配置

cd /opt/flink-1.7.2/conf

3.3.1 vim masters

  1. bigdata02:8081
  2. bigdata03:8081

3.3.2 vim slaves

  1. bigdata01
  2. bigdata02
  3. bigdata03

3.3.3 vim flink-conf.yaml

  1. high-availability: zookeeper
  2. high-availability.zookeeper.quorum: bigdata01:2181,bigdata02:2181,bigdata03:2181
  3. high-availability.zookeeper.path.root: /flink
  4. #high-availability.cluster-id: bigdata01
  5. high-availability.storageDir: hdfs:///flink/recovery
  6. pplication-attempts: 10

四、添加环境变量

  1. export FLINK_HOME=/opt/flink-1.7.2
  2. export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
  3. export PATH=$FLINK_HOME/bin:$PATH

五、使用

  • 启动

    1. yarn-session.sh -n 2 -jm 1024m -tm 4096m

  • 提交任务

  1. flink run -m yarn-cluster -yjm 1024m -ytm 1024m -ys 2 ./examples/batch/WordCount.jar --input hdfs:///data/input/wc.txt --output hdfs:///data/output