Big Data Learning path Sharing Hadoop phase high availability configuration, what is Hadoop HA mechanism

Ha mechanism is the high availability of Hadoop (7*24 hours uninterrupted service)

HA has been officially introduced since hadoop2.0, which was not available in previous versions

Hadoop-ha is strictly divided into the HA mechanism of each component, HDFS HA and YARN HA

Details about the HA mechanism of HDFS

The HA of HDFS is mainly realized through the coordination of two Namenodes

Key points of dual Namenode coordination:

A. Metadata management mode needs to be changed:

Each store a copy of metadata in memory

Only one Edits log can be created. Only namenode nodes in the Active state can write Edits logs

Both Namenodes can read edits

Shared edits are managed in a shared storage (qJournal and NFS are the two main implementations)

B, need a state management function module

Zkfailover is implemented, which is resident on each node where the Namenode resides

Each ZKFailover is responsible for monitoring its own Namenode node and using ZK for status identification

When a state switch is required, ZkFailover is responsible for the switch

It is necessary to prevent brain split when switching



The operation principle of Namenode

There is a Namenode on both servers. One namenode is in active state and the other is in standby state. Data is shared between the two servers. On both servers, only the Active Namenode server can write to Edit, and only the other server can read edit. The shared Edit is managed in a shared store. Shared storage is implemented by file management systems QJournal and NFS.

Zookeeper Failover Controller (ZKFC) is required to manage the active standby status of two servers. Each ZKFC is responsible for monitoring its own Namenode node and using ZK for status identification. When a state switch is required, ZkFailover is responsible for the switch

It is necessary to prevent brain split when switching.

What is split-brain phenomenon

Split brain is when two Namenodes are active and conflict. Hadoop high availability configuration should be careful to resolve the split state.

How does split-brain state occur

When an active Namenode server is in suspended animation, ZKFC of another Namenode server receives a message and changes its Namenode state to active. The first Namdenode in suspended animation wakes up again, resulting in split brain.

How to solve the split brain

Use SSH kill -9 namenode to directly kill the Namenode of the first server for knife repair. If the knife repair fails, ZKFC will enter the first server. Direct call user custom script/home/Hadoop/kill/poweroff. The namenode sh kill off false – death.