This is the seventh day of my participation in the August More text Challenge. For details, see:August is more challenging

The body of the

YARN is a classic master/slave structure, as shown in the following figure.

In general, the YARN service consists of a ResourceManager (RM) and multiple NodeManagers (NM). ResourceManager is the master node and NodeManager is the slave node.

ApplicationMaster can run any type of task within a Container.

Before Hadoop3.x, Container only supported memory and CPU resources. Hadoop3.x supports user-defined resource types, which greatly enhanced resource management capabilities.

How do the above components interact?

ResourceManager manages and schedules resources on each NodeManager in a unified manner. When you submit an application, you need to provide ApplicationMaster to track and manage the application. ApplicationMaster applies for resources from ResourceManager and requires NodeManger to start tasks that occupy certain resources.

Because the different ApplicationMasters are distributed on different nodes, they do not interact with each other.

Each application submitted by the Client to ResourceManager must have an ApplicationMaster. After resources are allocated by ResourceManager, the ApplicationMaster runs in a Container on a Slave node. The specific Task that does the Task also runs in a Container with a Slave node.

supplement

ResourceManager

RM is a global resource manager. There is only one cluster. RM is responsible for resource management and allocation of the entire system, including processing client requests, starting/monitoring ApplicationMaster, monitoring NodeManager, resource allocation and scheduling.

It consists of two main components: the Scheduler and the ApplicationManager.

Scheduler

The scheduler allocates resources in the system to each running application based on capacity, queue, and other constraints (such as a certain amount of resources per queue, maximum number of jobs to perform, etc.). It is important to note that this scheduler is a “pure scheduler” that does any application-specific work. It is not responsible for monitoring or tracking application execution status, and it is not responsible for restarting tasks that fail due to application execution failures or hardware failures. This is all done by the application-specific ApplicationMaster.

The scheduler only allocates resources according to the Resource requirements of each application, and the Resource allocation unit is represented by an abstract concept “Resource Container” (Container for short). A Container is a dynamic Resource allocation unit. It encapsulates resources such as memory, CPU, disk, network, and so on to limit the amount of resources used by each task.

ApplicationManager

The application manager is responsible for managing all applications in the system, receiving job submission requests, assigning the first Container to the application to run ApplicationMaster, This includes application submission, negotiating resources with the scheduler to start ApplicationMaster, monitoring ApplicationMaster’s health status, and restarting it if it fails.

ApplicationMaster

Manages each instance of an application running in YARN. ApplicationMaster processes manage jobs and applications. Yarn allows us to develop ApplicationMaster for our own applications.

function

• Data segmentation;
• Requesting resources for the application and further assigning them to internal tasks;
• Task monitoring and fault tolerance;
• Coordinates resources from ResourceManager and uses NodeManager to monitor easy execution and resource usage.

The communication between ApplicationMaster and ResourceManager is the core part of the Yarn application from submitting to running. It is the fundamental step for Yarn to dynamically manage resources for the entire cluster. ApplicationMaster from multiple applications dynamically communicates with ResourceManager to continuously apply for, release, and re-apply for, and release resources.

NodeManager

There are multiple NodeManagers across the cluster, responsible for the resources and usage on each node.

NodeManager is a Slave service. It receives resource allocation requests from ResourceManager and allocates specific Containers to applications. In addition, the ResourceManager monitors and reports Container usage information to ResourceManager. Working with ResourceManager, the NodeManager allocates resources in the Hadoop cluster.

function

• NodeManager Monitors the resource usage of the node and the running status of each Container (CPU and memory resources).
• Receives and processes command requests from ResourceManager and assigns Container to a task of an application.
• Periodically report to RM to ensure the smooth running of the entire cluster. RM tracks the health status of the entire cluster by collecting the report information of each NodeManager, while NodeManager is responsible for monitoring its own health status.
• Handle requests from ApplicationMaster;
• Manage logs on each node;
• Perform some additional services applied on Yarn, such as the Shuffle process of MapReduce.

When a node starts, it registers with ResourceManager and informs ResourceManager of the available resources. During the running period, the NodeManager and ResourceManager work together to update the cluster information and ensure that the cluster works in the best state. The NodeManager is only responsible for managing its own Container; it is not aware of the information about applications running on it. The component responsible for managing application information is ApplicationMaster

Container

Container is a resource abstraction in YARN. It encapsulates multi-dimensional resources on a node, such as memory, CPU, disk, and network resources. When AN AM requests resources from RM, THE RESOURCES returned by RM are represented by Container. YARN assigns a Container to each task, and the task can use only the resources described in the Container.

The relationship between containers and cluster nodes is as follows: A node can run multiple Containers, but a Container does not cross nodes. Each job or application must run in one or more Containers. In the Yarn framework, ResourceManager only tells ApplicationMaster which Containers can be used. ApplicationMaster also needs to go to the NodeManager and ask for specific containers.

Note that a Container is a dynamic unit of resource allocation and is dynamically generated based on application requirements. Up to now, YARN supports only CPU and memory resources and uses the lightweight resource isolation mechanism Cgroups to isolate resources.

function

• Abstraction of the Task environment;
• Describe a set of information;
• A collection of resources (CPU, memory, IO, etc.) for the task to run;
• Task running environment

Resource Request

Yarn is designed to allow applications to use the entire cluster in a shared, secure, and multi-tenant manner.

In addition, to ensure efficient cluster resource scheduling and data access, Yarn must be aware of the entire cluster topology.

To achieve these goals, the Scheduler of ResourceManager defines flexible protocols for the Resource requests of applications to better schedule applications running in clusters. Resource Requests and Containers are created.

An application sends a resource request that meets its requirements to ApplicationMaster. ApplicationMaster then sends the resource request to the ResourceManager Scheduler in the form of resource-request. Scheduler then returns the resource description Container from the original resource-request.

Each ResourceRequest can be regarded as a serializable Java object and contains the following field information:

<! -- resource-name: Specifies the name of the resource. At this stage, it refers to the host or rack where the resource is located. Later, VMS or more complex network structures may be supported. The number of containers that meet the requirements -->
<resource-name, priority, resource-requirement, number-of-containers>
Copy the code

JobHistoryServer

Run the following command to start the job history service on the data node in the cluster without any configuration. After the job history service is started successfully, the JobHistoryServer process is displayed. Run the JPS command to view the JobHistoryServer process. Described below), and log details can be viewed from port 19888

Two ways to start JobHistoryServer (Hadoop 3.x)

[hadoop@node1 ~]$ mr-jobhistory-daemon.sh start historyserver
WARNING: Use of this script to start the MR JobHistory daemon is deprecated.
WARNING: Attempting to execute replacement "mapred --daemon start" instead.
Copy the code

[hadoop@node1 ~]$ mapred --daemon start historyserver
[hadoop@node1 ~]$ 
Copy the code

TimelineServer

It is used to write log service data. Generally, it is used to write log service data combined with the third party (such as Spark). According to the description on the JobHistoryServer website, it is an effective supplement to the JobHistoryServer function

JobHistoryServer can only record information about MapReduce jobs. In addition to JobHistoryServer’s ability to record information about job running, there are more granular information, such as which queue the job is running in. Which user is set when the task is run.

According to the explanation on the official website, JobHistoryServer can only record records of MapReduce applications. TimelineServer is more powerful and can record records of third-party computing engines. However, TimelineServer does not replace JobHistoryServer. The two are complementary to each other.

practice

Start JobHistoryServer on each of the three machines by executing the following commands

mapred --daemon start historyserver
Copy the code

At this point, after starting JobHistoryServer on all three nodes, we run the WordCount program here

We use the official WordCount example.

[hadoop @ node1 ~] $hadoop jar/opt/bigdata/hadoop - 3.2.2 / share/hadoop/graphs/hadoop - graphs - examples - 3.2.2. Jar Wordcount /test/words /test/output 2021-06-17 22:06:18,348 INFO client.rmproxy: Connecting to the ResourceManager at node1/172.16.68.201:18040 2021-06-17 22:06:19, 108 INFO graphs. JobResourceUploader: Disabling Erasure Coding for path: / TMP/hadoop - yarn/staging/hadoop /. The staging/job_1614270487333_0009 22:06:19 2021-06-17, 346: INFO input. FileInputFormat: Total Input files to process :1 2021-06-17 22:06:19,844 INFO Mapreduce.JobSubmitter: Number of Splits :1 2021-06-17 22:06:20,026 INFO Mapreduce.JobSubmitter: plan-electronics for Job: Job_1614270487333_0009 2021-06-17 22:06:20,028 INFO Mapreduce.JobSubmitter: Executing with tokens: [] 2021-06-17 22:06:20,253 INFO conf.Configuration: Resource-types.xml not found 2021-06-17 22:06:20,253 INFO Resource.ResourceUtils: Unable to find 'resource-types.xml'. 2021-06-17 22:06:20,356 INFO imp.yarnclientimpl: Unable to find 'resource-types.xml'. 2021-06-17 22:06:20,356 INFO imp.yarnclientimpl: Application application_1614270487333_0009 2021-06-17 22:06:20,403 INFO MapReduce.Job: The url to track the job: http://node1:18088/proxy/application_1614270487333_0009/ 2021-06-17 22:06:20, 404 INFO graphs. Job: Running the Job: Job_1614270487333_0009 2021-06-17 22:06:26,536 INFO Mapreduce.Job: Job job_1614270487333_0009 False 2021-06-17 22:06:26,538 INFO mapreduce.Job: Map 0% reduce 0% 2021-06-17 22:06:31,614 INFO Mapreduce.Job: Map 100% reduce 0% 2021-06-17 22:06:36,653 INFO Mapreduce.Job: Map 100% reduce 100% 2021-06-17 22:06:36,666 INFO Mapreduce. Job job_1614270487333_0009 completed successfully 2021-06-17 22:06:36,763 INFO mapreduce. 54 File System Counters FILE: Number of bytes read=54 FILE: Number of bytes written=470051 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=115 HDFS: Number of bytes written=28 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2460 Total time spent by all reduces in occupied slots (ms)=2545 Total time spent by all map tasks (ms)=2460 Total time spent by all reduce tasks (ms)=2545 Total vcore-milliseconds taken by all map tasks=2460 Total vcore-milliseconds taken by all reduce tasks=2545 Total megabyte-milliseconds taken by  all map tasks=2519040 Total megabyte-milliseconds taken by all reduce tasks=2606080 Map-Reduce Framework Map input records=2 Map output records=6 Map output bytes=46 Map output materialized bytes=54 Input split bytes=93 Combine input records=6 Combine output records=5 Reduce input groups=5 Reduce shuffle bytes=54 Reduce input records=5 Reduce output records=5 Spilled Records=10 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=154 CPU time spent (ms)=1330 Physical memory (bytes) snapshot=546705408 Virtual memory (bytes) snapshot=5198422016 Total committed heap usage (bytes)=368574464 Peak Map Physical memory (bytes)=319201280 Peak Map Virtual memory (bytes)=2592329728 Peak Reduce Physical memory (bytes)=227504128 Peak Reduce Virtual memory (bytes)=2606092288 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=22 File Output Format Counters Bytes Written=28Copy the code

At this point, access the page:http://node1:18088/cluster (note that in the machine configuration host node1)

Click the History link to jump to a new page. At the bottom of the page, you will see the Map and Reduce tasks listed in TaskType. Total indicates the map and Reduce task data required for the Execution of the MapReduce program this time.

Click the Map link in the TaskType column below

As shown in the figure below, we can see the relevant information of the Map task, such as execution status, start time and completion time.

We can use the same approach to view the details of reduce job execution, which will not be covered here.

From the above operations, we can see that JobHistoryServer is a record of the historical running information during the operation of a job, which is convenient for us to analyze the job.