Flink on YARN (bottom) : FaQs and troubleshooting methods

Common client Problems and troubleshooting methods

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CLASSPATH=`${HADOOP_HOME}/bin/hadoop classpath`Copy the code

RULE test
CLASS org.apache.flink.client.cli.CliFrontend
METHOD getActiveCustomCommandLine
AT EXIT
IF TRUE
DO traceln("------->CliFrontend#getActiveCustomCommandLine return: "+ $!) ; ENDRULECopy the code

export BYTEMAN_HOME=/path/to/byte-home
export TRACE_SCRIPT=/path/to/script
export JVM_ARGS="-javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:${TRACE_SCRIPT}"Copy the code

——->CliFrontend#getActiveCustomCommandLine return: org.apache.flink.yarn.cli.FlinkYarnSessionCli@25ce9dc4

Flink Cluster FaQs and Troubleshooting Methods

Env.java. opts=-verbose:class // Configure jobManager &TaskManager env.java.opts. Jobmanager =-verbose:class // Configure JobManager Env. Java. Opts. Taskmanager = verbose: taskmanager class / / configurationCopy the code

• If the NEW_SAVING state is in the persistent application information state, you need to check whether the RM state storage service (usually the ZooKeeper cluster) is running properly.
• If the state is in the SUBMITTED state, the possible cause is that some time-consuming operations of the hold read/write lock occur in RM, resulting in event accumulation. You need to locate the fault based on the YARN cluster logs.
• If the STATUS is ACCEPTED, check whether the AM is normal and go to Step 2.
• If the JOB is in the RUNNING state but not all resources are available, go to Step 3.

• Check cluster or queue Resources. The following Resources are displayed in the tree of the scheduler page: Effective Max Resource, Used Resources: (1) Check whether cluster Resources, queue Resources or its parent queue Resources are Used up. (2) Check whether a dimension resource of the leaf queue approaches or reaches its upper limit;
• Check for resource fragments: (1) Check the proportion of the sum of the Used resources and Reserved resources in the total resources of the cluster. When the cluster resources are nearly full (for example, more than 90%), there may be resource fragmentation, and the application allocation speed will be affected and slow, because most machines have no resources, and insufficient available resources will be Reserved. When the reserved resources reach a certain scale, most of the machine resources may be locked, and the subsequent allocation may be slow. (2) Check the distribution of available NM resources. Even if the cluster resource usage is low, it may be caused by different resource distribution in different dimensions. For example, memory resources on half of the nodes are nearly used up, and CPU resources on half of the nodes are nearly used up, and CPU resources on half of the nodes are nearly used up, and CPU resources on half of the nodes are nearly used up, and CPU resources on half of the nodes are nearly used up. If the resource value of a dimension is too large, the resource cannot be applied for.
• Check for high-priority applications that frequently apply for and immediately release resources. In this case, the scheduler is too busy meeting the resource requests of this application to take care of other applications.
• Check whether the Container fails to be started or automatically exits after the Container is started. You can view Container logs (including localize logs and launch logs), YARN NM logs, or YARN RM logs.

• If the process exits, an error may occur or the process is affected by the preemption mechanism on YARN RM or NM. You need to trace the TaskManager logs or YARN RM/NM logs.
• The process is still running and the cluster is disconnected due to a network fault. If the connection times out, JobManager automatically exits. After the fault occurs, JobManager automatically applies for resources again and starts a new TaskManager.
• If the GC process takes a long time, memory leaks or improper configuration of memory resources may be caused. Locate the cause based on logs or memory analysis.

The resources