In CDH, yarn scheduling spark, container resources are tilted, and Containers are concentrated on one or several machines

In CDH 6.0.1, the Spark task is submitted. It is found that containers allocated by YARN are concentrated on several nodes, and other nodes are not allocated. This obviously leads to high load on individual machines, which can affect the overall performance of the cluster.

Because yarn. The scheduler. Fair. Assignmultiple: Whether to allow multiple container assignments in one heartbeat. Defaults to false.

This configuration item determines whether to allocate multiple containes in a heartbeat allocation request. The default value is true in CDH and false in YARN native.

Yarn. The scheduler. Fair. Dynamic. Max. Assign for the assignmultiple is true, whether to a dynamically allocated, if it is true the node is a most distribution not half of the allocation of resources.

The solution set of CDH configuration yarn. The scheduler. Fair. Assignmultiple to false, restart the CDH effect, so will not tilt the distribution of resources, evenly distributed to multiple nodes.