This is the third day of my participation in the Gwen Challenge

Graphs is defined

MapReduce is a distributed computing framework first proposed by Google in 2004 and contributed to the Apache Foundation.

MR version

  • MapReduce 1.0, early version of Hadoop (MR model only)
  • MapReduce 2.0, Hadoop 2.X (After introducing the YARN resource scheduling framework, MapReduce supports other computing models in addition to MR

MR V1 execution flow

  1. JobTracker has been waiting for JobClient to submit jobs;
  2. TaskTracker sends JobTracker a heartbeat every 3 seconds to ask if there is a task available and if so, to assign it to perform;
  3. This is a pull process: the slave actively pulls business from the master;

MR V2 vs MR V1 process

The operating mechanism of MR V2 is improved, and YARN is started

  1. Added ResourceManager and NodeManager.
  2. ResourceManager(RM) : ApplicationManager and ResourceScheduler. That is, application management and resource scheduling are separated.
  3. NodeManager(NM) runs nodes. When a task is received, the Container starts to complete the Map or Reduce task.
  4. ResourceManager corresponds to v1 JobTracker.
  5. NodeManager corresponds to v1’s TaskTracker

MR V2 Execution process

  1. Users submit applications to YARN, including the MR App Mstr program, commands to start ApplicationMaster, and data.
  2. ResourceManager allocates the first Container for the application and communicates with the corresponding NodeManager to ask it to start the ApplicationMaster of the application in this Container.
  3. ApplicationMaster registers with ResourceManager so that users can view the running status of applications using ResourceManager. Then, ApplicationMaster applies for resources for each task and monitors the running status until the running is complete. Repeat Steps 4 to 7
  4. ApplicationMaster applies for and obtains resources from ResourceManager through RPC in polling mode.
  5. Once the ApplicationMaster has requested the resource, it communicates with the corresponding NodeManager asking it to start the task.
  6. After setting the running environment (including environment variables, JAR packages, and binaries) for a task, NodeManager writes the task startup command to a script and runs the script to start the task
  7. Each task reports its status and progress to ApplicationMaster through an RPC protocol so that ApplicationMaster can keep track of the running status of each task and restart the task if it fails. While the application is running, the user can query the current running state of the application from the ApplicationMaster via RPC at any time.
  8. After the application is complete, ApplicationMaster logs out of ResourceManager and shuts itself down.

Resource Container: Resource Container

1) Cluster node encapsulates its own memory, CPU, disk and other resources in one abstract concept;

2) The scheduler conducts scheduling according to application resource requirements and resource capacitors of each cluster node;

3) The application manager of RM is responsible for receiving the job, negotiating to obtain the task subject of the first resource container for executing the application, and allocating containers for the application subject that fails to restart;

4) The application subject of THE DN node is responsible for obtaining resource capaciters from the scheduler through negotiation, and tracking the state of these containers and application execution

Mapper task process

① Read the input file and parse it into key and value pairs. Each line of the input file (k1,v1) is parsed into multiple key and value pairs (k2,v2). Call map function once for each (k1, v1) key-value pair;

② Write the code logic to process the input K1 and V1 into the new k2 and V2 output;

③ Partition the output K2 and v2 (if no default partition is specified, the partition number is 0);

④ Sort and group the data of different partitions according to K2. The same k2 v2 is placed in the set v2 ‘={v21,v22,v23… V2n};

⑤ (Optional) The grouped data is combiner;

⑥ Mapper results are written to the local disk

Reducer Task Flow

(1) The output of multiple Map tasks is copied to different Reduce nodes over the network based on different partitions (the result data of the Map end is obtained).

② A Reduce task processes a partition data;

③ Merge and sort the output results of multiple MAP tasks in the current partition. Write the code of reduce function, process k2,v2 ‘of the input, and convert it into new K3, V3 output;

④ Save the reduce output to a file. The file name is part-R-0000n

Core concepts of MapReduce

  • Different Map tasks do not communicate with each other
  • No information is exchanged between Reduce tasks
  • Users cannot explicitly send messages from one machine to another
  • All data exchange is done through the MapReduce framework itself

File InputFormat InputFormat

Defines how data files are split and read. InputFile provides the following functions

  • Select a file or other object to use as input
  • Define the InputSplits a file between the different missions
  • Provide a factory for RecordReader to read this file

For example, there is an abstract FileInputFormat class that reads all the files from that directory and splits those files into multiple InputSplits.

Input data split InputSplits

InputSplit defines the input data to be entered into a single Map task

  • A MapReduce program is collectively referred to as a Job and may consist of hundreds of jobs
  • InputSplit splits the file into 128MB sizes

Data records are read into RecordReader

InputSplit defines a data split, but does not define how to read the data record. RecordReader actually defines detailed methods for converting a data record into a (key,value) pair and passing the data record to a Mapper class.

For example, TextInputFormat provides LineRecordReader, which reads a textual row data record

Mapper

Each instance of the Mapper class generates a Java process that processes the data on a given InputSplit

  • In V1, there are two additional parameters, the OutputCollector, which collects intermediate results, and Reporter, which gets the environment parameters and sets the current execution status.
  • The current version of V2 provides each Mapper function with Context, which provides the functions of the two objects above

Combiner

Merge the output on the Map side, that is, merge key-value pairs of the same key, to reduce the partitioner data communication overhead. You don’t have to specify it; If you specify a merge class, its nature is similar to a Reducer for local execution, which can be executed only if certain conditions are met

Partitioner & Shuffle&Sort

  • Partitioner & Shuffle

After the Map work is completed, the results of each Map function will be transmitted to the node that the Reducer should have. At this point, the user can provide a Partitioner class that determines which node a given (key,value) pair will be sent to.

  • Sort

The keys and value pairs transferred to each Reducer node that will be received by all Reduce functions will be automatically sorted by Hadoop (that is, when the results generated by Map are sent to a certain node, they will be automatically sorted)

Shuffle

The Shuffle process is between the output from the Map terminal and the input from the Reduce terminal. Is the core of Hadoop!!

Shuffle the Map

  1. Merge before the Map task is complete
  2. Merge to get large files on local disk
  3. During file merging, if the number of overwritten files exceeds the preset value (3 by default), the Combiner can be started again. If the number of overwritten files is less than 3, the Combiner is not required
  4. JobTracker monitors Map job execution and notifies Reduce job to collect data

Shuffle the Reduce

  1. The Reduce task queries JobTracker through RPC to check whether the Map task is complete. If so, JobTracker obtains data
  2. Reduce data is stored in the cache, collected from different Map machines, and then merged into disks
  3. Multiple overwrite files are merged into one or more large files in which key-value pairs are sorted
  4. When the data is small, it is directly merged in the cache and output to Reduce without overwriting the data to disks

Reducer

Perform user-defined Reduce operations. The programming interface is Context. A partition corresponds to a Reducer. Each partition corresponds to an output file part-R-0000x.

File OutputFormat OutputFormat

All outputFormats written to HDFS inherit from FileOutputFormat. Each Reducer writes a file to a common output directory, named part-nnnnn, where NNNNN is a number (partition ID) associated with each Reducer.

Summary of MapReduce- Task execution

  1. Submit a job
  2. Initialization job
  3. Assigned tasks
  4. Perform a task
  5. Update the task progress and status

Speculative execution – If tasks on a node are running too slowly, backup tasks are started

YARN is defined

YARN is a resource management system in Hadoop 2.0. It is a universal resource management module that can manage and schedule resources for various applications.

YARN Three components

  1. ResourceManager – Processes client requests
  • Start/monitor ApplicationMaster
  • Monitor the NodeManager
  • Resource allocation and scheduling
  1. NodeManager – Resource management on a single node
  • Process commands from ResourceManager
  • Process commands from ApplicationMaster
  1. ApplicationMaster
  • Data segmentation
  • Apply for resources for the application and assign them to internal tasks
  • Task monitoring and fault tolerance

YARN resource scheduler

  • FIFO, first in, first out scheduler
  • Fair Scheduler
  • Capacity Scheduler