image.png

Component Configuration options Description
Framework Heap Memory taskmanager.memory.framework.heap.size JVM heap memory dedicated to Flink framework (advanced option)
Task Heap Memory taskmanager.memory.task.heap.size JVM heap memory dedicated to Flink application to run operators and user code
Managed memory taskmanager.memory.managed.size
taskmanager.memory.managed.fraction		 Native memory managed by Flink, reserved for sorting, hash tables, caching of intermediate results and RocksDB state backend
Framework Off-heap Memory taskmanager.memory.framework.off-heap.size Off-heap direct (or native) memory dedicated to Flink framework (advanced option)
Task Off-heap Memory taskmanager.memory.task.off-heap.size Off-heap direct (or native) memory dedicated to Flink application to run operators
Network Memory taskmanager.memory.network.min
taskmanager.memory.network.max
taskmanager.memory.network.fraction		 Direct memory reserved for data record exchange between tasks (e.g. buffering for the transfer over the network), it is a capped fractionated component of the total Flink memory
JVM metaspace taskmanager.memory.jvm-metaspace.size Metaspace size of the Flink JVM process
JVM Overhead taskmanager.memory.jvm-overhead.min
taskmanager.memory.jvm-overhead.max
taskmanager.memory.jvm-overhead.fraction		 Native memory reserved for other JVM overhead: e.g. thread stacks, code cache, garbage collection space etc, it is a capped fractionated component of the total process memory

JVM Parameters

Flink根据配置的或派生的内存组件大小，在启动任务执行程序进程时显式添加以下与内存相关的JVM参数：

JVM Arguments Value
-Xmx and -Xms Framework + Task Heap Memory
-XX:MaxDirectMemorySize Framework + Task Off-Heap + Network Memory
-XX:MaxMetaspaceSize JVM Metaspace

Configure Total Memory

taskmanager.memory.process.size

  • TM 的总内存: 粗粒度所有内存,不需要配置一般启动命令指定TM 内存
  • 通常我们配置 TM 内存为 1g、2g 或 4g，对应的就是这个参数
  • 注：请不要配置该参数，⼀般通过启动命令指定 TM 内存
  • 命令：flflink run -yjm 4096 -ytm 4096

taskmanager.memory.flink.size

  • TM 内 Flink 使⽤的总内存
  • 计算规则： process.size - JVM Metaspace - JVM Overhead
  • 计算 flink.size 没有实质的意义，仅仅是为了计算其他参数，
  • 例如 Heap 的⼤⼩就依赖 flink.size
  • 注：请不要配置该参数，该参数会⾃动根据 TM 总内存⽽计算出来
for TaskManager: for JobManager:
taskmanager.memory.flink.size jobmanager.memory.flink.size
taskmanager.memory.process.size jobmanager.memory.process.size
taskmanager.memory.task.heap.size
and taskmanager.memory.managed.size		 jobmanager.memory.heap.size

参考