This is the fifth day of my participation in the November Gwen Challenge. Check out the details: The last Gwen Challenge 2021

Disadvantages of MRv1

  1. JobTracker is prone to single points of failure
  2. JobTracker is a heavy burden, responsible for both resource management and job scheduling; Excessive resource consumption occurs when too many tasks need to be handled.
  3. A large number of MapReduce jobs will cause a large memory overhead. On TaskTracker, the number of MapReduce tasks is too simple to represent resources, and CPU and memory usage are not taken into account. If two tasks with large memory consumption are scheduled together, OutOfMemory exceptions can easily occur.
  4. On the TaskTracker side, the resource is forcibly divided intomap task slotandreduce task slot, if the system has onlymap taskOr onlyreduce taskWill cause a waste of resources.

YARN architecture

The basic components

1, the ResourceManager

There are two components, one is a Scheduler (Scheduler) and the other is an application manager (ASM). RM is a central service that schedules, starts the ApplicationMaster for each Job, and monitors the ApplicationMaster’s presence.

2, NodeManager

The framework agent on each node is responsible for starting the containers required by the application, monitoring resource usage (memory, CPU, disk, network, etc.) and reporting it to the scheduler.

3, ApplicaionManager

It is responsible for receiving jobs, negotiating the first container for executing ApplicationMaster and providing services for restarting the failed AM Container.

4, ApplicationMaster

Responsible for all jobs in a Job’s life cycle, similar to the old Job Tracker framework. Note that each Job (but not each type) has an Application Master, which can run on a machine other than ResourceManager.

YARN operation process analysis

1. Clien submits jobs to ResourceManager. It usually specifies what ApplicationMaster is, what start command ApplicationMaster is, and how much CPU, memory, and external resources are required

2. ResourceManager uses an internal scheduler to find resources for ApplicationMaster based on requirements. Tell the NodeManager what command to use to start the ApplicationMaster, how much CPU is required, and so on.

NodeManager starts ApplicaionMaster after receiving this command.

ApplicaionMaster is a MapReduce application that splits job submission data into maps and Reduces. Then summarize the resource requirements of each Map and Reduce, and communicate with ResourceManager. For example, apply for 10 cpus and 10 GB memory. ResourceManager allocates resources to ApplicaionMaster based on the resource usage of each node. (Not all at once, but in a dynamic process.)

If you have a CPU and 1 GB of memory free, allocate it to ApplicaionMaster. After allocating the CPU, ApplicaionMaster communicates with NodeManager to tell ApplicaionMaster to start a task.

6. After NodeManager receives the command, based on the description (what are the start commands and environment variables for the task), NodeManager starts the task, encapsulates it in a separate environment of the Container, and then the task runs.

Structure and function of RM

The user interaction

1, the ClientService

Is a service provided for ordinary users. It handles various RPC requests from clients, such as submitting applications, terminating applications, obtaining application running status, etc

2, AdminService

YARN provides an independent set of service interfaces for administrators to prevent management commands sent by administrators from starving due to a large number of common user requests. Administrators can use these interfaces to manage clusters, for example, dynamically update node lists, ACL lists, and queues.

NM management

1, NMLivelinessMonitor

Monitors whether NM is alive and if a NodeManager does not report heartbeat information within a certain period of time (10 minutes by default), it is considered dead and removed from the cluster.

2, NodesListManager

Maintains lists of healthy and abnormal nodes, and manages exlude (similar to a blacklist) and Inlude (similar to a whitelist) node lists, both of which are set in the configuration file and can be loaded dynamically.

3, ResourceTrackerService

Handles requests from NodeManager, which consist of two main types of requests: registration and heartbeat. Registration is a behavior that occurs when NodeManager is started. The request packet contains node IDS, upper limit of available resources, and other information. Heartbeat is a periodic behavior that contains the running status of each Container, a list of running applications, and node health status (which can be set by a script). The ResourceTrackerService returns a list of containers, applications, and so on to be released for NM.

AM management

1, AMLivelinessMonitor

Monitors whether the AM is alive. If an ApplicationMaster does not report a heartbeat within a certain amount of time (default: 10 minutes), it is considered dead and all running Containers on it are considered dead. The AM itself is reassigned to another node (the user can specify the number of attempts per ApplicationMaster, which is 1 by default).

2, ApplicationMasterLauncher

Communicate with the NodeManager and ask it to start ApplicationMaster for an application.

3, ApplicationMasterService

Processing requests from ApplicationMaster includes two main types of requests: registration and heartbeat. Registration is the behavior that occurs when ApplicationMaster is started, including the node contained in the request packet, the RPC port number and tracking URL. Heartbeat is a periodic behavior, containing the type description of the requested resource and the list of containers to be released, for which AMS returns information such as newly allocated containers and failed containers.

Application management

1, ApplicationACLsManager

You can manage application access permissions, including viewing and modifying permissions. Viewing allows you to view basic application information, and modifying allows you to modify application priorities and kill applications.

2, RMAppManager

Manage the startup and shutdown of applications.

3, ContainerAllocationExpirer

YARN does not allow AM to not use a Container for a long time after it is acquired. This reduces the cluster utilization. After an AM receives a Container allocated by RM, it must start the Container on NM within a certain period (10 minutes by default). Otherwise, RM will reclaim the Container.

NM structure

YARN advantage

  1. The design of YARN reduces the resource consumption of JobTracker and makes the program that monitors the status of each Job subtask more secure and elegant.
  2. In the new Yarn, ApplicationMaster is a changeable part. Users can write their own AppMst for different programming models, so that more types of programming models can run in the Hadoop cluster.
  3. Resources are represented in memory, rather than in the number of slots left.
  4. A big burden of JobTracker in MRv1 is to monitor the health of tasks under jobs. This task is now left to ApplicationMaster, and ResourceManager has a module called ApplicationManager. It monitors the health of ApplicationMaster and restarts it on another machine if something goes wrong.
  5. Container is used as a resource isolation component of YARN to schedule and control resources.