0. Component diagram

Generally distributed architectures are master-slave architectures with only master-slave nodes, each of which serves a different purpose. In Flink, the job manager is equivalent to Master, and the task manager is equivalent to slave, similar to Executor in Spark

1. JobManager (JobManager,master node)

  1. The main process that controls the execution of an application, that is, each application has a corresponding JobManager.
  2. JobManager will first accept the application to be executed, which will include: JobGraph, Logical Dataflow Graph, and all the classes, libraries, and dependent JAR packages packaged.
  3. JobGraph is converted into a physical data flow graph, called an ExecutionGraph, that contains all tasks that can be executed concurrently.
  4. The JobManager requests resources necessary for performing tasks from ResourceManager (Flink’s ResourceManager), that is, slots on the TaskManager. Once it has acquired enough resources, it distributes the execution diagrams to the TaskManager that actually runs them. During the runtime, JobManager takes charge of any operations that require central coordination, such as checkpoints (which I capture).

2. TaskManager

  1. Work process in Flink. There are usually multiple TaskManagers running in Flink, and each TaskManager has a number of slots. The number of slots limits the number of tasks a TaskManager can perform.
  2. Once started, TaskManager registers its slots with the resource manager; Upon receiving the resource manager’s instruction, The TaskManager provides one or more slots to the JobManager call. JobManager can then assign tasks to the slot to execute.
  3. During execution, a TaskManager can exchange data with other TaskManagers running the same Flink application.

3. ResourceManager

  1. The TaskManager slot is a processing resource unit defined in Flink.
  2. Flink provides different resource managers for different environment and resource management tools, such as YARN, Mesos, K8s, and standalone deployment.
  3. When JobManager applies for a slot resource, Flink’s resource manager assigns a TaskManager with a vacant slot to JobManager. If Flink’s resource manager does not have enough slots to accommodate JobManager requests, it can also initiate a session to the resource provider platform to provide a container to start the TaskManager process.

4. The dispenser

  1. It can be run across jobs and provides a REST interface for application submission.
  2. When an application is submitted for execution, the distributor starts and hands the application over to a JobManager.
  3. The Dispatcher also launches a Web UI to easily display and monitor information about job execution.
  4. Dispatcher may not be necessary in the architecture, depending on how the application submission runs.