“This is the 17th day of my participation in the Gwen Challenge in November. See details of the event: The Last Gwen Challenge in 2021”.

1. Yarn common commands

You can query the Yarn status by running commands in addition to hadoop103:8088. The common command operations are as follows:

Requirements: Run the WordCount case and run the Yarn command to view the task running status.

[MOE @ hadoop102 hadoop - 3.1.3] $hadoop jar share/hadoop/graphs/hadoop - graphs - examples - 3.1.3. Jar wordcount/input /outputCopy the code

1. Yarn Application View the task

1.1. List all applications

[moe@hadoop102 hadoop-3.1.3]$YARN Application -list 2021-11-16 22:15:26/809 INFO client.rmProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 the Total number of applications (application - types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):1 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1637071605761_0002 word count MAPREDUCE moe default RUNNING UNDEFINED 50% http://hadoop103:42721Copy the code

1.2. Filter according to Application state

Yarn application-list-appStates (ALL states: ALL, NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED)

[moe@hadoop102 hadoop-3.1.3]$YARN Application-list-AppStates FINISHED 2021-11-16 22:18:09.194 INFO client.rmProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 the Total number of applications (application - types: [], states: [FINISHED] and tags: []):2 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1637071605761_0001 word count MAPREDUCE moe default FINISHED SUCCEEDED 100% http://hadoop102:19888/jobhistory/job/job_1637071605761_0001 application_1637071605761_0002 word count MAPREDUCE moe default FINISHED SUCCEEDED 100% http://hadoop102:19888/jobhistory/job/job_1637071605761_0002Copy the code

1.3. Kill Application

[moe@hadoop102 hadoop-3.1.3]$YARN Application -kill Application_1637071605761_0002 2021-11-16 22:19:49,567 INFO client.RMProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 Application application_1637071605761_0002 has already finishedCopy the code

2. Yarn logs View logs

2.1. Query Application logs

yarn logs -applicationId <ApplicationId>

[moe@hadoop102 hadoop-3.1.3]$YARN Logs -applicationId Application_1637071605761_0002Copy the code

2.2. Query Container logs

yarn logs -applicationId <ApplicationId> -containerId <ContainerId>

[moe@hadoop102 hadoop-3.1.3]$YARN Logs -applicationId Application_1637071605761_0002 -containerId container_1637071605761_0002_01_000003Copy the code

Yarn ApplicationAttempt View the attempted task

3.1 List all Application attempts

yarn applicationattempt -list <ApplicationId>

[moe@hadoop102 hadoop-3.1.3]$YARN ApplicationAttempt -list application_1637071605761_0002 2021-11-16 22:29:59:269 INFO client.RMProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 the Total number of application attempts: 1 ApplicationAttempt-Id State AM-Container-Id Tracking-URL appattempt_1637071605761_0002_000001 FINISHED container_1637071605761_0002_01_000001 http://hadoop103:8088/proxy/application_1637071605761_0002/Copy the code

3.2. Print the ApplicationAttempt state

yarn applicationattempt -status <ApplicationAttemptId>

[moe@hadoop102 hadoop-3.1.3]$YARN ApplicationAttempt -status appatTEMPt_1637071605761_0002_000001 2021-11-16 22:35:08, 505 INFO client. RMProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 Application Attempt Report: ApplicationAttempt - Id: appattempt_1637071605761_0002_000001 State : FINISHED AMContainer : container_1637071605761_0002_01_000001 Tracking-URL : http://hadoop103:8088/proxy/application_1637071605761_0002/ RPC Port : 36294 AM Host : hadoop103 Diagnostics :Copy the code

4. Yarn Container View the container

4.1. List all Containers

yarn container -list <ApplicationAttemptId>

[moe@hadoop102 hadoop-3.1.3]$YARN container -list appatTEMPt_1637071605761_0002_000001 2021-11-16 22:38:05.848 INFO client.RMProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 the Total number of containers: 0 Container - Id Start Time Finish Time State Host Node Http Address LOG-URLCopy the code

4.2. Print the Container status

yarn container -status <ContainerId>

[moe@hadoop102 hadoop-3.1.3]$YARN container-status container_1637071605761_0002_01_000001 2021-11-16 22:39:22.407 INFO  client.RMProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 Container with id 'container_1637071605761_0002_01_000001' doesn't exist in RM or Timeline Server.Copy the code

Note: You can only see the container status during the task run

5. Yarn node Check the node status

List all nodes: yarn node-list-all

[moe@hadoop102 hadoop-3.1.3]$YARN node-list-all 2021-11-16 22:40:33.889 INFO client.rmproxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 Total Nodes: 3 Node - Id Node - State Node - Http - the Address Number-of-Running-Containers hadoop104:33986 RUNNING hadoop104:8042 0 hadoop102:41945 RUNNING hadoop102:8042 0 hadoop103:34927 RUNNING hadoop103:8042 0Copy the code

6. Yarn rmadmin Updates the configuration

Load queue configuration: YARN rmadmin-refreshqueues

[moe@hadoop102 hadoop-3.1.3]$YARN rmadmin-refreshqueues 2021-11-16 22:41:56,852 INFO Client.rmProxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8033Copy the code

7. Yarn Queue View queues

Print queue information: yarn queue-status

[moe@hadoop102 hadoop-3.1.3]$yarn queue-status default 2021-11-16 22:42:42,329 INFO client.rmproxy: Connecting to the ResourceManager at hadoop103/192.168.85.103:8032 Queue Information: the Queue Name: default State: RUNNING Capacity: 100.0% Current Capacity:.0% Maximum Capacity: 100.0% Default Node Label expression: <DEFAULT_PARTITION> Accessible Node Labels : * Preemption : disabled Intra-queue Preemption : disabledCopy the code

2. Yarn production environment core parameters

3. Yarn case practice

1. Set core parameters in the Yarn production environment

1) Requirement: Count the frequency of occurrence of each word from 1G data. Three servers, each equipped with 4G memory, 4-core CPU, and 4 threads.

2) Demand analysis:

1G / 128M = 8 MapTasks; 1 ReduceTask; 1 mrAppMaster

On average, each node runs 10/3 ≈ 3 tasks (4 3 3)

3) Modify the yarn-site. XML configuration parameters as follows:

<! -- Select scheduler, default capacity -->
<property>
	<description>The class to use as the resource scheduler.</description>
	<name>yarn.resourcemanager.scheduler.class</name>
	<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

<! -- ResourceManager Number of threads to process scheduler requests. The default value is 50. If the number of submitted tasks is greater than 50, you can increase this value, but do not exceed 3 * 4 threads = 12 threads (excluding other applications can not exceed 8) -->
<property>
	<description>Number of threads to handle scheduler interface.</description>
	<name>yarn.resourcemanager.scheduler.client.thread-count</name>
	<value>8</value>
</property>

<! The default value is false. If there are many other applications on this node, you are advised to manually configure yarn. If there are no other applications for this node, use automatic -->
<property>
	<description>Enable auto-detection of node capabilities such as
	memory and CPU.
	</description>
	<name>yarn.nodemanager.resource.detect-hardware-capabilities</name>
	<value>false</value>
</property>

<! Default: false; use physical CPU cores.
<property>
	<description>Flag to determine if logical processors(such as
	hyperthreads) should be counted as cores. Only applicable on Linux
	when yarn.nodemanager.resource.cpu-vcores is set to -1 and
	yarn.nodemanager.resource.detect-hardware-capabilities is true.
	</description>
	<name>yarn.nodemanager.resource.count-logical-processors-as-cores</name>
	<value>false</value>
</property>

<! -- Virtual kernel and physical kernel multiplier, default is 1.0 -->
<property>
	<description>Multiplier to determine how to convert phyiscal cores to vcores. This value is used if yarn.nodemanager.resource.cpu-vcores is set to -1(which implies auto-calculate vcores) and yarn.nodemanager.resource.detect-hardware-capabilities is set to true. The number of vcores will be calculated as number  of CPUs * multiplier.</description>
	<name>yarn.nodemanager.resource.pcores-vcores-multiplier</name>
	<value>1.0</value>
</property>

<! -- NodeManager (default: 8GB, changed to 4GB) -->
<property>
	<description>Amount of physical memory, in MB, that can be allocated 
	for containers. If set to -1 and
	yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
	automatically calculated(in case of Windows and Linux).
	In other cases, the default is 8192MB.
	</description>
	<name>yarn.nodemanager.resource.memory-mb</name>
	<value>4096</value>
</property>

<! -- Number of CPU cores for nodeManager (default: 8)
<property>
	<description>Number of vcores that can be allocated
	for containers. This is used by the RM scheduler when allocating
	resources for containers. This is not used to limit the number of
	CPUs used by YARN containers. If it is set to -1 and
	yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
	automatically determined from the hardware in case of Windows and Linux.
	In other cases, number of vcores is 8 by default.</description>
	<name>yarn.nodemanager.resource.cpu-vcores</name>
	<value>4</value>
</property>

<! -- Container minimum memory, default 1G -->
<property>
	<description>The minimum allocation for every container request at the RM	in MBs. Memory requests lower than this will be set to the value of this	property. Additionally, a node manager that is configured to have less memory	than this value will be shut down by the resource manager.
	</description>
	<name>yarn.scheduler.minimum-allocation-mb</name>
	<value>1024</value>
</property>

<! -- Container Max memory, default 8GB, changed to 2G -->
<property>
	<description>The maximum allocation for every container request at the RM	in MBs. Memory requests higher than this will throw an	InvalidResourceRequestException.
	</description>
	<name>yarn.scheduler.maximum-allocation-mb</name>
	<value>2048</value>
</property>

<! -- Minimum number of CPU cores in container, default 1 -->
<property>
	<description>The minimum allocation for every container request at the RM	in terms of virtual CPU cores. Requests lower than this will be set to the	value of this property. Additionally, a node manager that is configured to	have fewer virtual cores than this value will be shut down by the resource	manager.
	</description>
	<name>yarn.scheduler.minimum-allocation-vcores</name>
	<value>1</value>
</property>

<! -- Maximum number of CPU cores in a container (default: 4) -->
<property>
	<description>The maximum allocation for every container request at the RM	in terms of virtual CPU cores. Requests higher than this will throw an
	InvalidResourceRequestException.</description>
	<name>yarn.scheduler.maximum-allocation-vcores</name>
	<value>2</value>
</property>

<! -- Virtual memory check, default enabled, changed to disabled -->
<property>
	<description>Whether virtual memory limits will be enforced for
	containers.</description>
	<name>yarn.nodemanager.vmem-check-enabled</name>
	<value>false</value>
</property>

<! -- Virtual memory to physical memory set ratio, default 2.1 -->
<property>
	<description>Ratio between virtual memory to physical memory when	setting memory limits for containers. Container allocations are	expressed in terms of physical memory, and virtual memory usage	is allowed to exceed this allocation by this ratio.
	</description>
	<name>yarn.nodemanager.vmem-pmem-ratio</name>
	<value>2.1</value>
</property>
Copy the code

4) Distribute the configuration.

Note: If the hardware resources in the cluster are inconsistent, configure each NodeManager separately

5) Restart the cluster

[moe@hadoop103 hadoop-3.1.3]$sbin/stop-yarn.sh [moe@hadoop103 hadoop-3.1.3]$sbin/start-yarn.shCopy the code

6) Execute the WordCount program

[MOE @ hadoop102 hadoop - 3.1.3] $hadoop jar share/hadoop/graphs/hadoop - graphs - examples - 3.1.3. Jar wordcount/input /outputCopy the code

7) View the Yarn task execution page

http://hadoop103:8088/cluster/apps

2. Capacity scheduler multi-queue submission cases

1) How to create queues in the production environment?

(1) The scheduler has only one default queue by default, which cannot meet the production requirements.

(2) According to the frameworks: Hive/Spark/Flink, the tasks of each framework are put into a specified queue (enterprises do not use many frameworks).

(3) According to business modules: login registration, shopping cart, placing orders, Business department 1, business department 2

2) What are the benefits of creating multiple queues?

(1) All resources will be exhausted by writing recursive endless loop code for fear that employees will not be careful.

(2) To achieve degraded use of tasks and ensure sufficient resources of important task queues in special periods. 11.11 6.18

Business Department 1 (Major) = business Department 2 (Minor) = Order (Minor) = Shopping Cart (Minor) = Login registration (minor)

2.1, requirements,

Requirement 1: The Default queue accounts for 40% of the total memory and the maximum resource capacity accounts for 60% of the total resource capacity. The Hive queue accounts for 60% of the total memory and the maximum resource capacity accounts for 80% of the total resource capacity.

Requirement 2: Configure queue priority

2.2. Configure a multi-queue capacity scheduler

1) The configuration in capacity-scheduler. XML is as follows:

(1) Modify the following configurations

```xml <! -- Specify multiple queues, Increase The hive queue - - > < property > < name > yarn. The scheduler. Capacity. The root, The queues < / name > < value > default, hive < value > / < description > The queues at the this level (root is the root queue). </description> </property> <! -- Reduce the rated capacity of the default queue resource to 40%. Default 100% - > < property > < name > yarn. The scheduler. Capacity. Root. Default. Capacity < / name > < value > 40 < value > / < / property > <! -- Reduce the maximum capacity of the default queue resource to 60%. Default 100% - > < property > < name > yarn. The scheduler. Capacity. Root. The default. The maximum - capacity < / name > < value > 60 < value > / < / property > (2) Add the necessary attributes to the new queue: <! - specifies the hive queue resources rated capacity - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. Capacity < / name > < value > 60 < value > / < / property > <! -- The maximum number of resources a user can use in the queue, 1 - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. User - limit - factor < / name > < value > 1 < value > / < / property > <! - specifies the hive queue resource capacity - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. Maximum - capacity < / name > < value > 80 < / value > </property> <! - start the hive queue - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. State < / name > < value > RUNNING < value > / < / property > <! - which users shall have the right to submit to the queue assignments - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. Acl_submit_applications < / name > < value > * < / value > </property> <! Which users have the right to operate the queue, Administrator rights (view/kill) -- - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. Acl_administer_queue < / name > < value > * < / value > </property> <! - configuration which users have the right to submit task priority - > < property > < name > yarn. The scheduler. Capacity. Root. Hive. Acl_application_max_priority < / name > <value>*</value> </property> <! Yarn Application -appID appId -updatelifeTime Timeout https://blog.cloudera.com/enforcing-application-lifetime-slas-yarn/ --> <! If an application specifies a timeout, the maximum timeout that an application submitted to the queue can specify cannot exceed this value. --> <property> <name>yarn.scheduler.capacity.root.hive.maximum-application-lifetime</name> <value>-1</value> </property> <! If the application does not specify a timeout, Use the default value of default-application-lifetime --> <property> <name>yarn.scheduler.capacity.root.hive.default-application-lifetime</name> <value>-1</value> </property> ```Copy the code

2) Distribute the configuration file

Restart Yarn or run Yarn rmadmin-refreshqueues to refreshQueues.

2.3. Submit a job to the Hive queue

1) Hadoop JAR approach

[MOE @ hadoop102 hadoop - 3.1.3] $hadoop jar share/hadoop/graphs/hadoop - graphs - examples - 3.1.3. Jar wordcount - D mapreduce.job.queuename=hive /input /outputCopy the code

Note: -d indicates changing parameter values during runtime

2) Jar package method

The default task submission is to the default queue. If you want to submit tasks to other queues, you need to declare in the Driver:

public class WcDrvier {

    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {

        Configuration conf = new Configuration();

        conf.set("mapreduce.job.queuename"."hive");

        //1. Obtain a Job instanceJob job = Job.getInstance(conf); . ./ / 6. Submit the Job
        boolean b = job.waitForCompletion(true);
        System.exit(b ? 0 : 1); }}Copy the code

This way, the task will be submitted to the Hive queue when the cluster commits:

2.4. Task priority

Capacity scheduler: supports task priority configuration. When resources are scarce, tasks with higher priorities obtain resources first. By default, Yarn limits the priority of all tasks to 0. To use the priority function of tasks, you need to enable the priority function.

1) Modify the yarn-site. XML file and add the following parameters

<property>
    <name>yarn.cluster.max-application-priority</name>
    <value>5</value>
</property>
Copy the code

2) Distribute the configuration and restart Yarn

[moe@hadoop102 hadoop]$xsync yarn-site. XML [moe@hadoop103 hadoop-3.1.3]$sbin/stop-yarn.sh [moe@hadoop103 Hadoop - 3.1.3] $sbin/start - yarn. ShCopy the code

3) Simulate a resource-constrained environment and submit the following tasks continuously until the newly submitted tasks cannot apply for resources.

[MOE @ hadoop102 hadoop - 3.1.3] $hadoop jar The/opt/module/hadoop - 3.1.3 / share/hadoop/graphs/hadoop - graphs - examples - 3.1.3. Jar PI 5, 2000000Copy the code

4) Resubmit the high-priority task again

[MOE @ hadoop102 hadoop - 3.1.3] $hadoop jar The/opt/module/hadoop - 3.1.3 / share/hadoop/graphs/hadoop - graphs - examples - 3.1.3. Jar PI - D graphs. Job. Priority = 5 5 2000000Copy the code

5) You can also run the following command to change the priority of the task being executed:

Yarn Application -appID <ApplicationID> -updatepriority Priority

[moe@hadoop102 hadoop-3.1.3]$YARN Application -appID Application_1611133087930_0009 -updatePriority 5Copy the code

3. Fair scheduler case

3.1, requirements,

Create two queues, test and MOE (named after the group to which the user belongs). The following effects are expected: If a queue is specified when a user submits a task, the task is submitted to the specified queue for execution; If no queue is specified, the tasks submitted by the test user are run in the root.group.test queue, and the tasks submitted by the atguigu user are run in the root.group. MOE queue (note: group indicates the group to which the user belongs).

The configuration of the fairness scheduler involves two files: yarn-site. XML and the fairness scheduler queue allocation file fair-scheduler. XML (the file name can be customized).

(1) Configuration file Reference materials:

Hadoop.apache.org/docs/r3.1.3…

(2) Task queue placement rules reference:

Blog.cloudera.com/untangling-…

3.2. Configure a fair scheduler for multiple queues

1) Modify the yarn-site. XML file and add the following parameters

<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description>Configure to use the Fairness scheduler</description>
</property>

<property>
    <name>yarn.scheduler.fair.allocation.file</name>
    <value>/ opt/module/hadoop - 3.1.3 / etc/hadoop/fair - scheduler. XML</value>
    <description>Specifies the fair scheduler queue allocation profile</description>
</property>

<property>
    <name>yarn.scheduler.fair.preemption</name>
    <value>false</value>
    <description>Prohibit resource preemption between queues</description>
</property>
Copy the code

2) Configure fair-scheduler.xml

      
<allocations>
  <! -- Maximum proportion of resources occupied by Application Master in a single queue. The value ranges from 0 to 1.
  <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
  <! Test MOE default -->
  <queueMaxResourcesDefault>4096mb,4vcores</queueMaxResourcesDefault>

  <! -- add queue test -->
  <queue name="test">
    <! -- Queue minimum resource -->
    <minResources>2048mb,2vcores</minResources>
    <! -- Queue maximum resource -->
    <maxResources>4096mb,4vcores</maxResources>
    <! Maximum number of concurrent applications in the queue, default 50, based on the number of threads.
    <maxRunningApps>4</maxRunningApps>
    <! The maximum number of resources in the queue occupied by Application Master -->
    <maxAMShare>0.5</maxAMShare>
    <! -- The queue resource weight, default value 1.0 -->
    <weight>1.0</weight>
    <! Resource allocation policy inside queue -->
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>
  <! -- Add queue MOE -->
  <queue name="moe" type="parent">
    <! -- Queue minimum resource -->
    <minResources>2048mb,2vcores</minResources>
    <! -- Queue maximum resource -->
    <maxResources>4096mb,4vcores</maxResources>
    <! Maximum number of concurrent applications in the queue, default 50, based on the number of threads.
    <maxRunningApps>4</maxRunningApps>
    <! The maximum number of resources in the queue occupied by Application Master -->
    <maxAMShare>0.5</maxAMShare>
    <! -- The queue resource weight, default value 1.0 -->
    <weight>1.0</weight>
    <! Resource allocation policy inside queue -->
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>

  <! -- Task queue allocation policy, can configure multiple rules, from the first rule matching, until the match is successful -->
  <queuePlacementPolicy>
    <! -- Specify a queue when submitting a task. If no queue is specified, the next rule will be matched. False: if the specified queue does not exist, it cannot be created automatically -->
    <rule name="specified" create="false"/>
    <! Username queue. If root.group does not exist, it cannot be created automatically. If root.group.user does not exist, it can be created automatically -->
    <rule name="nestedUserQueue" create="true">
        <rule name="primaryGroup" create="false"/>
    </rule>
    <! The last rule must be reject or default. Reject: Reject, default: commit to default queue -->
    <rule name="reject" />
  </queuePlacementPolicy>
</allocations>
Copy the code

3) Distribute the configuration and restart Yarn

[moe@hadoop102 hadoop]$ xsync yarn-site.xml [moe@hadoop102 hadoop]$ xsync fair-scheduler.xml [moe@hadoop103 Hadoop-3.1.3]$sbin/stop-yarn.sh [moe@hadoop103 hadoop-3.1.3]$sbin/start-yarn.shCopy the code

4. Friendship links

Big data Hadoop-YARN Learning journey 1

Big data Hadoop-MapReduce learning journey 6

Big data Hadoop-MapReduce learning journey chapter 5

Big data Hadoop-MapReduce learning journey chapter 4

Big data Hadoop-MapReduce learning journey chapter 3

Big data Hadoop-MapReduce learning journey chapter 2

Big data Hadoop-MapReduce learning journey first