1. Inconsistent use of class types (Mapper and Reduce)

20/01/13 16:03:42 INFO conf.Configuration: found resource resource-types.xml at file:/tmp/hadoop-unjar5833957221661452217/resource-types.xml 20/01/13 16:03:42 INFO  resource.ResourceUtils: Adding resource type - name = rm1, units = G, type = COUNTABLE 20/01/13 16:03:42 INFO resource.ResourceUtils: Adding resource type - name = rm2, units = , type = COUNTABLE 20/01/13 16:03:42 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE 20/01/13 16:03:42 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE 20/01/13 16:03:43 INFO impl.YarnClientImpl: Submitted application application_1578714121919_0001 20/01/13 16:03:43 INFO mapreduce.Job: The url to track the job: http://node04:8088/proxy/application_1578714121919_0001/ 20/01/13 16:03:43 INFO mapreduce.Job: Running job: job_1578714121919_0001 20/01/13 16:03:51 INFO mapreduce.Job: Job job_1578714121919_0001 running in uber mode : false 20/01/13 16:03:51 INFO mapreduce.Job: map 0% reduce 0% 20/01/13 16:03:57 INFO mapreduce.Job: map 100% reduce 0% 20/01/13 16:04:01 INFO mapreduce.Job: Task Id : attempt_1578714121919_0001_r_000000_0, Status : FAILED Error: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.util.bloom.Key at com.ideatech.hadoop.mapreduce.MyReduce.reduce(MyReduce.java:14) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171) 20/01/13 16:04:06 INFO mapreduce.Job: Task Id : attempt_1578714121919_0001_r_000000_1, Status : FAILEDCopy the code

This is because the inheritance classes of Mapper and Reduce use different T parameters. For example, if map uses Text and Reduce uses Key, the type conversion exception will occur

2:2.10.0 lacks configuration file (unlike earlier 2.x)

2.10.0 Yarn Resource Management Parameters Need to Be Configured This parameter is different from that of 2.x earlier version

Three: AM resource allocation problem, always waiting for allocation of resources

Refer to the article: www.cnblogs.com/yjt1993/p/9…

The configuration file of resource-types. XML is missing. The earlier version 2.x is not required. The higher version 2.x and 3.x are required.

　　

<? xml-stylesheet type="text/xsl" href="configuration.xsl"? > <configuration> <property> <name>yarn.resource-types</name> <value>rm1, rm2</value> </property> <property> <name>yarn.resource-types.rm1.units</name> <value>G</value> </property> <property> <! -- Maximum resources to allocate to application masters If this is too high application masters can crowd out actual The work -- -- > < name > yarn. The scheduler. Capacity. The maximum - am - resource -percent < / name > < value > 0.6 < / value > < / property > <! -- <property>--> <! -- <name>yarn.scheduler.minimum-allocation-mb</name>--> <! -- <value>1</value>--> <! -- </property>--> <! -- <property>--> <! -- <name>yarn.scheduler.maximum-allocation-mb</name>--> <! -- <value>1</value>--> <! -- </property>--> <! -- <property>--> <! -- <name>yarn.resource-types.rm2.minimum</name>--> <! -- <value>1</value>--> <! -- </property>--> <! -- <property>--> <! -- <name>yarn.resource-types.rm2.maximum</name>--> <! -- <value>1</value>--> <! -- </property>--> </configuration>Copy the code

5: Yarn resource management allocation fails to start. (The following figure shows the HA mode.)

* : Note Memory Total and Apps Completed, VCores Used

The configuration file is incorrectly configured

  • Summary: The Hadoop series, 1.x, 2.x and 3.x versions are quite different, if you want to operate, it is best to use 2.x medium version, 3.x low version, such as 2.6, 2.7, or 3.2 or 3.1 (time: 20200113). It is best to explore the history of the three stages of change and change.