1. HDFS primary/secondary architecture

Namenode: nn namenode

  • Contents:
    • File name
    • The directory structure of the file
    • File properties (permissions, number of copies, creation time)
    • Which data blocks (including the number of copies) a file is cut into –> Which Datanodes are distributed in the file
  • Mapping relationships: Blockmaps are maintained in memory and NN does not persist such mappings. When the cluster is up and running, DN regularly reports blockreports to NN, and then maintains this mapping dynamically in memory.
  • Manage the namespace of a file system. It maintains files and folders in the file system tree
  • Save read and write operations: fsimage (mirror file), Editlogs (editlog file)

Secondary namenode: SNN secondary namenode

  • Fsimage, editlog file to merge, backup, push to nn.
  • In order to solve THE SINGLE point of failure (SPOF) of NN in the early stage, we added a CHECKPOINT of SNN and 1 million reads and writes per hour. Although it can reduce the risk of data loss caused by single point of failure, I do not use SNN in production, but High Availability. More on that later.
  • Checkpoint on the basis of:
    • dfs.namenode.checkpoint.period 3600
    • dfs.namenode.checkpoint.txns 1000000

Datanode: indicates the dn datanode

  • The checksum (blk_XXX.meta) used to store data blocks and data blocks (blk_XXX)
  • Send block Reports at regular intervals
    • On the basis of:dfs.blockreport.intervalMsec 21600000=6h
  • Communicate with namenode:
    • Heartbeat packets are sent every 3 seconds
    • Block location information is sent every 10 heartbeats
  • Purpose: Data blocks used to read and write files.

HDFS read and write

  • HDFS reads and writes are insensitive to users
  • Write an output
  • Reading is the input

HDFS writing process

  • Description:
    • The HDFS Client invokes FileSystem. Create (filePath) to communicate with namenode through RPC. Namenode checks whether the file exists and has permission to create it. After the check passes, a new file is created with no data and no blocks associated. Namenode calculates the number of blocks to be uploaded and the corresponding Datanodes based on the file size, block size, and number of copies. Eventually this information is returned to the client as the FSDataOutputStream object.
    • The Client invoks the write method of the FSDataOutputStream object on the Client, writes the first copy of the first block to DN1, copies it to DN2, and then copies it to DN3 based on the copy placement policy. When DN3 is finished writing, it sends an ACK package to DN2. DN2 receives an ACK and then sends an ACK to DN1. DN1 receives an ACK and then sends an ACK to the client FSDataOutputStream to tell it that the first block triple copy is finished. And so on.
    • When all blocks are written, the Client calls the close method of the FSDataOutputStream object to close the output stream. Call filesystem.plete to tell namenode that the file is successfully written.

HDFS reading process

  • Description:
    • The HDFS Client calls FileSystem. Open (filePath) and communicates with namenode through RPC to return a partial or complete block list of the file, that is, the FSDataInputStream object.
    • The Client schedules the read method of the FSDataInputStream object, reads from the nearest DN node of the first block, checks after reading, and closes the communication with the DN after passing. If no, the block and DN information is recorded. If the DN is read next time, the DN is automatically skipped and the DN is read from the next DN. And so on. It then reads from the nearest DN of the second block, and so on. FileSystem is called to fetch the list of blocks for the next batch from NN, assuming that the list of blocks has been read before the file is finished.
    • The Client calls the close method of the FSDataInputStream object to close the input stream.

3. SNN workflow

  • Description:
      1. When the SNN checkpoint action is performed, the SNN stops using the current Edit file 515-516 and temporarily records the read and write operations to a new Edit file 517.
      1. SNN downloads NN’s fsimage 514 and EDits files 515-516 remotely to the local.
      1. SNN loads fsimage 514 into memory, executes the contents of EDits files 515-516 in memory, and creates a new fsimage file 516.
      1. SNN pushes the new Fsimage 516 to NN.
      1. NN Receives fsimage 516. The checkpoint scroll is fsimage 516, and the new edit17.new file is edit517, which is the latest file.

HDFS_DownLoad

HDFS_Upload

Working mechanism of namenode

Working mechanism of datanode