Hello everyone, I am a senior programmer.

Starting today, we will open a new column, and in the following days, we will share some knowledge about big data with you.

Let’s start today with YARN, the resource manager. If you are interested in big data related technologies, please click on it.

First impression

YARN is a universal resource management module that manages and schedules resources for various applications. On YARN, you can run the MapReduce program, Spark, FLink, and other computing frameworks. YARN uses the Master/Slave architecture and includes ResourceManager and NodeManager. ResourceManager manages and schedules resources on each NodeManager.

  1. ResourceManager(RM)

RM is a global resource manager that manages and allocates resources for the entire system. It mainly consists of a Scheduler and an Application Manager. The scheduler allocates resources in the system to running jobs based on capacity, queue, and other constraints. YARN provides many different schedulers. The system includes Fair Scheduler, FIFO Scheduler, and Capacity Scheduler. The application manager is responsible for managing all applications throughout the system, including application submission, negotiating resources with the scheduler to start ApplicationMaster, monitoring ApplicationMaster’s health, and restarting it if it fails.

  1. ApplicationMaster(AM)

Each application submitted by the user contains an AM. Its main functions include.

(1) Negotiate with the RM scheduler to obtain resources.

(2) Communicate with NM to start/stop tasks.

(3) Further assign the obtained tasks to internal tasks.

(4) Monitor the running status of the task.

  1. NodeManager(NM)

NM manages resources and tasks on each node. On the one hand, NM periodically reports resource usage on the node and the running status of each Container to RM. On the other hand, it receives and processes requests such as Container start/stop requests from the AM.

  1. Container

Container is a resource abstraction in YARN. It encapsulates multi-dimensional resources on a node, such as memory, CPU, disk, and network resources. When AM applies for resources from RM, the resources returned by RM are represented by Container.

YARN Workflow

When a user submits an application to YARN, YARN runs the application in two phases. The first stage is to start ApplicationMaster; In the second phase, the ApplicationMaster creates the application, allocates resources for it, and monitors its entire run until it is complete.

Let’s take a look at the entire process of a task from submission to execution to completion.

1. Users submit applications to ResouceManager.

2.ResourceManager allocates the first Container for the application and communicates with the NodeManager to start ApplicationMaster of the application in this Container.

3. After ApplicationMaster is started, register it with ResourceManager. It then requests resources for each task and monitors its running status until the end of the run, repeating steps 4-7.

4.ApplicationMaster uses RPC to apply for and obtain resources from ResourceManger.

5. Once ApplicationMaster has requested the resource, it communicates with the corresponding NodeManager asking it to start the task.

6.NodeManger starts tasks.

7. Each task reports its status and progress to ApplicationMaster through RPC. This enables ApplicationMaster to know the running status of each task and restart the task if the task fails.

8. After the application is executed, ApplicationMaster ResourceManager logs out and shuts itself.

Resource scheduling model

YARN uses a two-layer resource scheduling model. In the first layer, the resource scheduler in ResourceManager allocates resources to applicationMasters. In the second layer, ApplicationMaster further allocates resources to its internal tasks. Here we mainly talk about the scheduling problem of the first layer. The scheduling policy for layer 2 is entirely up to the user application.

In YARN, the resource allocation process is described in the following seven steps.

  1. NodeManager periodically reports node information through heartbeat.

  2. ResourceManager returns a heartbeat response for NodeManager, including releasing the Container list.

  3. When ResourceManager receives information from NodeManager, it triggers a Node_Update event.

  4. After receiving the Node_Update event, ResourceScheduler allocates resources (released in Step 2) on this node to the application program based on certain policies and puts the allocation result into a memory data structure.

  5. ApplicationMaster periodically sends heartbeat messages to ResourceManager to collect the newly allocated Container.

  6. When ResourceManger receives the heartbeat message from ApplicationMaster, the allocated Container is returned to ApplicationMaster in the form of a heartbeat reply.

  7. When the ApplicationMaster receives a list of newly assigned Containers, it further assigns those containers to its internal tasks.

The resource scheduler focuses on the strategy taken in Step 4, which is how the free resources on the nodes are allocated to the various applications.

Resource scheduling

The resource scheduler is one of the most core components in YARN. It manages and allocates resources for the entire cluster. It provides three resource schedulers available: FIFO Scheduler, Capacity Scheduler, and Fair Scheduler.

  1. FIFO Scheduler

FIFO Scheduler, or first come, first served, is a scheduling mechanism in which all jobs are uniformly submitted to a queue and run in the order they are submitted.

  1. Capacity Scheduler

Capacity Scheduler is a multi-user Scheduler. It divides resources by queue. Each queue can set a certain proportion of minimum guarantee and upper limit of resource usage. In addition, each user can set a certain upper limit of resource usage to prevent resource abuse. If the resources in one queue are available, the remaining resources can be temporarily shared with other queues.

  1. Fair Scheduler

The Fair Scheduler is also a multi-user Scheduler that, like the Capacity Scheduler, divides resources in queues. Each queue can set a certain proportion of the minimum guarantee and upper limit of resource usage, and each user can set a certain upper limit of resource usage to prevent resource abuse. If the resources in one queue are available, the remaining resources can be temporarily shared with other queues. It differs from the Capacity Scheduler in the following respects.

(1) Fair sharing of resources. Within each queue, Fair Scheduler can allocate resources to an application based on FIFO, Fair, or DRF policies. The Fair policy is adopted by default. This means that if there are two applications running in the queue at the same time, each application gets 1/2 of the resources.

(2) Support resource preemption. When there are resources left in one queue, the scheduler shares them with other queues, and when a new application is submitted in that queue, the scheduler reclaims resources for it.

(3) Load balancing. Fair Scheduler provides a load balancing mechanism based on the number of tasks, which distributes tasks evenly among nodes in the system as much as possible. In addition, users can also design load balancing mechanisms according to their own requirements.

(4) Scheduling policy configuration is flexible. Fair Scheduler offers a more diverse set of scheduling policies, which allow each queue to be configured individually. Currently, you can choose from three scheduling policies: FIFO, Fair, and DRF.

(5) Improve applets response time. Because of the max-min fairness algorithm, small jobs can quickly acquire resources and run.

The last

So far, we’ve talked about YARN in Hadoop. If you enjoyed it, please click on YARN.

For more interesting knowledge, please pay attention to the public number ** [senior programmer]. I have prepared hundreds of learning materials for you, including Python, Java, data structures and algorithms. If necessary, please pay attention to the public number [senior programmer], reply to [information] **, you can get.

The more you know, the more your mind opens up. See you next time.