1. Task submission process

1. Submit an independent cluster task

2. Manage cluster resources in YARN mode

A brief overview of this approach will be followed by a detailed explanation.

2. Task scheduling principle

3. The TaskManager and slots

1. The overview map

  1. In Flink, each TaskManager is a JVM process, and each Task slot starts a thread, which may execute one or more subtasks on a separate thread, with each subTask occupying a Task slot.
  2. In order to control how many tasks a TaskManager can receive, the TaskManager controls this through task Slot (each TaskManager has at least one slot).

2. The figure in detail

  1. By default, Flink allows subtasks to share slots, even if they are subtasks of different tasks. The result is that a single slot can hold the entire pipe of a job.
  2. Task Slot is a static concept and refers to the concurrent execution capability of TaskManager

3. Analyze examples in detail

  1. Configure the number of slots in taskManager in the configuration file

2. Set the case that only one slot is required for the task3. Set the case that only two slots are required for the task4. Set the situation that only nine slots are required for the task5. Set different parallelism for different operators

The parallelism is less than the number of slots