Today’s share is about HBASE design experience and what you have learned.

Today is the second article about HBASE merge, split, and failover.

– a,Introduction of HBASE

– 2. Detailed HBASE read and write, read magnification, merge, and fault recovery

– 3. Use HBASE alarm information

– 4. HBASE optimization experience

What is HBASE?

HBase is a distributed, column-oriented open source database. This technology is derived from The Google paper “Bigtable: A Distributed Storage System for Structured Data” written by Fay Chang. Just as Bigtable leveragesthe distributed data store provided by Google’s File System, HBase provides capabilities similar to Bigtable on top of Hadoop.

HBase is a subproject of the Apache Hadoop project. Unlike common relational databases, HBase is a database suitable for unstructured data storage. Another difference is that HBase is column-based rather than row-based.

(1) Data model

(2) Logical architecture

(3) System architecture

(4) How do you read it?

(5) What is Region Server and what does it store?

(6) How to write data?

(7) What is a MemStore?

(8) cache write how to brush disk, always write disk up?

(9) What is HFILE?

The last article addressed these issues! The update is coming, so I’ll just get back to it

(10) When will read merge be triggered?

Region Server has read cache, write cache, and HFILE. Region Server has read cache, write cache, and HFILE

So if we read a piece of data, it may be reading the Cache (LRU), it may have just been written to the Cache, or it may not be writing to the Cache, HBase uses the Block Cache index and bloom filter to load the corresponding HFile into memory. Therefore, data may come from the read cache, the scanner read and write cache, and the HFILE. This is called the left HBASE read merge

As mentioned earlier, the write cache may have multiple Hfiles, so a read request may read multiple files and affect performance, which is also known as read amplification.

(11) Since there is read magnification, is there a way to read fewer files? – write to merge

In simple terms, HBase automatically merges some small HFiles and overwrites them into a small number of larger HFiles

It uses merge sort algorithm to merge small files into large files, effectively reducing the number of hfiles

This process is called a minor compaction.

(12) Which hfiles does write merge target?

1. It merges and overwrites all HFiles under each Column Family

2. In this process, deleted and expired cells are physically deleted, which improves read performance

3. Since a major compaction rewrites all hfiles, it incurs a significant amount of hard drive I/O and network overhead. This is called Write Amplification.

4. Automatic HBASE scheduling is performed by default. You are advised to perform it in the early morning or on weekends because of write enlargement

5. When a Major compaction occurs, data migration caused by Server crashes or load balancing can be moved back to a location away from the Region Server, restoring local data.

(13) The Region Server does not manage multiple regions. How many regions does it manage? When will regions be expanded? – the Region division

Let’s review the concept of region:

  • The HBase Table is horizontally cut into one or several regions. Each region contains a continuous, ordered row bounded by the start key and end key.
  • The default size of each region is 1GB.
  • The Region Server reads and writes data in the Region and interacts with the client.
  • Each Region Server can manage about 1000 regions (which may come from one or more tables).

At first, each table has only one region by default. When a region becomes large, it is split into two sub-regions. Each sub-region contains half of the data of the original region. The two sub-regions are created on the original region server in parallel. The split action is reported to the HMaster. For load balancing purposes, the HMaster may migrate the new region to another Region server.

(14) Due to split, read amplification may occur in multiple Region servers for load balancing, until the write merge comes, and the region server is relocated or merged to a place near the Region Server node

(15) How is HBASE data backed up?

All reads and writes occur on the primary DataNode of the HDFS. HDFS automatically backs up WAL (logs before write) and HFile blocks. HBase relies on HDFS to ensure data integrity and security. When data is written to the HDFS, one copy is written to the local node and the other two copies are written to other nodes.

WAL and HFiles are persisted to hard disks and backed up. How does HBase recover data from MemStore that has not been persisted to HFile?

(16) How is HBASE data recovered after downtime?

  1. If a Region Server crashes, its regions cannot be accessed until the crash is detected and fault recovery is complete. Zookeeper detects node faults by heartbeat detection, and HMaster receives a notification of region Server faults.

  2. If the HMaster detects that a Region server is faulty, it allocates regions managed by the Region Server to other healthy Region Servers. To recover the data in the MemStore of the faulty Region Server that has not been persisted to HFile, HMaster divides WAL into several files and stores them on the new Region Server. Each region server then plays back the data from the WAL fragment it took to create a MemStore for the new region to which it was assigned.

  3. WAL contains a series of modification operations, each of which represents a PUT or DELETE operation. These changes are written in chronological order, and when persisted they are written to the end of the WAL file.

  4. What if the data is still in MemStore and has not been persisted to HFile? WAL files will be played back. The way to do this is to read WAL files, sort and add all changes to MemStore, and finally MemStore will be written to HFile.

This is the end of HBASE basic explanation, tomorrow to explain the actual combat drill!

❤ ️ ❤ ️ ❤ ️ ❤ ️

Thank you very much talent can see here, if this article is written well, feel that there is something to ask for praise 👍 for attention ❤️ for share 👥 for handsome Oba I really very useful!!

If there are any mistakes in this blog, please comment, thank you very much!

At the end of this article, I recently compiled an interview material “Java Interview Process Manual”, covering Java core technology, JVM, Java concurrency, SSM, microservices, databases, data structures and so on. How to obtain: GitHub github.com/Tingyu-Note… , more attention to the public number: Ting rain notes, one after another.