The article directories

    • I. Basic concepts
      • 1.1 Related Components
      • 1.2 Logical Layers
      • 1.3 Two-Layer Resource Scheduling
    • Ii. Mechanism and Strategy (Flink1.10)
      • 2.1 What resources does TaskManager have?
      • 2.2 What Resources does the Slot have?
      • 2.3 TaskManager management
      • 2.4 Cluster -> Job Resource scheduling process
      • 2.5 Job -> Task Resource scheduling process
      • 2.6 Resource Tuning

I. Basic concepts

1.1 Related Components

A Flink Cluster consists of a Flink Master and multiple Task Managers. Flink Master and Task Manager are process-level components, while other components are in-process components.

A FlinkMaster has multiple JobManagers in a ResourceManager. In a FlinkMaster, each JobManager manages a specific Job independently. The Scheduler component in JobManager is responsible for scheduling all tasks in the DAG that executes the Job and sending out resource requests, which is the starting point of the entire resource scheduling. The Slot Pool component in JobManager holds all resources of the Job. In addition, the only ResourceManager in FlinkMaster schedules resources in the entire Flink Cluster and interconnects with external scheduling systems, such as Kubernetes, Mesos, and Yarn.

Task Manager is responsible for executing tasks. Slot is a subset of Task Manager resources and the basic unit of Flink resource management. The concept of Slot is used throughout the resource scheduling process.

1.2 Logical Layers

After introducing related components, we need to understand the logical relationships between these components, which are divided into four layers as follows.

  • The Operator Operator is the most basic unit of data processing
  • Chained Operators are the minimum units that the Task Flink Runtime actually dispatches. If two operators belong to the same Task, one Operator is not already running and the other Operator is not yet scheduled.
  • Job corresponds to a Job Graph
  • Flink Cluster 1 Flink Master + N Task Managers

JobManager, Secheduler, and SlotPool correspond to the Job level. ResourceManager, SlotManager, and TaskManager correspond to the FlinkCluster level.

Chaining between an Operator and a Task is how to compose a Task with an Operator. Slot Sharing between tasks and jobs refers to how multiple tasks share a Slot resource, which does not happen across jobs. Slot Allocation between Flink clusters and jobs refers to how slots in Flink clusters are allocated to different jobs.

1.3 Two-Layer Resource Scheduling

Resource scheduling of Flink is a classic two-layer model, in which the allocation process from Cluster to Job is completed by SlotManager, and the internal allocation process to Task resources is completed by Scheduler. The Scheduler issues a SlotRequest to a SlotPool. If the SlotPool fails to meet the Resource requirement, it requests the Resource Manager.

Ii. Mechanism and Strategy (Flink1.10)

2.1 What resources does TaskManager have?

The resource type

  • memory
  • CPU
  • Other development resources GPU

TM resources are determined by configuration

  • In Standalone deployment mode, TM resources may be different
  • In other deployment modes, all TM resources are the same

2.2 What Resources does the Slot have?

  • TM has a fixed number of Slot resources
  • The number of slots is determined by configuration
  • Slot resources are determined by TM resources and the number of slots
  • There is no difference between slots on the same TM

Slot calculation Management

2.3 TaskManager management

Standalone Deployment mode

In Standalone deployment mode, the number of Task Managers is fixed. If the cluster is started with the start-cluster.sh script, the number of TM can be determined by modifying the configuration in the following file. You can also start a TM by manually executing the TaskManager.sh script.

<FLINK_DIR>/conf/slaves
Copy the code

ActiveResourceManager deployment mode

  • Kubernetes, Yarn, Mesos
  • The number of slots is allocated on demand, and the TaskManager is started based on the number of Slot Requests
  • If the TaskManager is idle for a period of time, the TaskManager will be released
  • In the on-YARN deployment mode, a fixed number of TaskManagers and a specified number of TM are no longer supported. That is, the following command parameters are invalid.
yarn-session.sh -n <num>
flink run -yn <num>
Copy the code

2.4 Cluster -> Job Resource scheduling process

The red arrow

The Scheduler sends a request to the Slot Pool. If the Slot Pool resources are sufficient, the Scheduler allocates the request to the Slot Manager. In this case, the Job requests resources from the Cluster. If the Slot Manager determines that there are enough resources in the cluster to meet the demand, it sends the Assign directive to the Task Manager, which provides slots to Slot pools. Slot pools are then used to satisfy Scheduler’s resource requests.

Blue arrow

In the Active Resource Manager Resource deployment mode, when the Resource Manager determines that there are not enough resources in the Flink Cluster to meet the demand, it will further request resources from the underlying Resource scheduling system. The scheduling system starts the new TaskManager and the TaskManager registers with the Resource Manager to fill the new Slot.

2.5 Job -> Task Resource scheduling process

Scheduler

  • Based on the Execution Graph and the Execution status of the Task, determine which Task to schedule next

  • Initiate SlotRequest

    • Task + Slot -> Allocate TaskSlot

  • Determines the allocation between tasks/slots

Slot Sharing

  • Slot Sharing Group Tasks can share slots
  • By default, all nodes are in a Slot Sharing Group
  • Only one task can exist in a Slot

advantages

  • The maximum number of concurrent slots required to run a job
  • Relative load balancing

Slot Sharing process (each row is multiple concurrent tasks A, B, AND C from bottom to top). The parallelism of TASKS A, B, and C is 4, 4, and 3, respectively. These tasks belong to the same Slot Sharing Group. So different tasks can run in the same Slot, as shown on the right side of Figure 7, where three slots are ABC and the fourth Slot is AB. We can easily calculate that the number of slots required by the Job is 4, which is also the maximum number of concurrent slots.

2.6 Resource Tuning

Through the mechanism introduced above, we can easily find that Flink adopts top-down resource management. We configure the overall resources of jobs, while Flink controls the number of slots and load balancing through Slot Sharing mechanism. Adjust Task Manager/Slot resources to fit the resource requirements of a Slot Sharing Group. Flink’s resource management and configuration are simple and easy to use. It is suitable for jobs with simple topology or small scale.

reference

Cloud.tencent.com/developer/n…