HBase Advanced - HBase architecture and core processes

1. The RegionServer architecture

StoreFile
- Physical files for storing data. Storefiles are stored in the HDFS as hfiles. Each Store has one or more storefiles (hfiles).
MenStore
- Write memory, so write data will be stored in MenStore first, sorted, and written to HDFS when the write mechanism starts. Each write will generate a new HFile.
WAL
- WAL solves this problem by writing data to a write-ahead logfile and then to MenStore. In the event of a failure, data can be recovered from this logfile.
BlockCache
- Read cache. After each query, data is stored in the BlockCache for future query.

Client Accesses ZooKeeper and obtains hbase: RegionServer where the Meta table resides.
Access the RegionServer, obtain the hbase: Meta table, and query the Region in which the target data resides based on the Namespace: Table/Rowkey of the read request. Region information of the table and meta table location information are cached in meta Cache of the client for next access.
Communicates with the target RegionServer to add data.
Data appended to WAL.
Data is written to the MenStore and sorted in memory.
Wait until the time is right for MenStore to be swiped to HFile

Client Accesses ZooKeeper and obtains hbase: RegionServer where the Meta table resides.
Access the RegionServer, obtain the hbase: Meta table, and query the Region in which the target data resides based on the Namespace: Table/Rowkey of the read request. Region information of the table and meta table location information are cached in meta Cache of the client for next access.
Communicates with the target RegionServer to query data.
Search for data in BlockCache, MenCache, and StoreFile and merge the found data.
The queried new data blocks (HFile data storage unit, default size: 64KB) are cached to the Block Cache.
The final result of the merge is returned to the client.

When the size of a memstore reached the hbase. Hregion) memstore. Flush. The size (the default value of 128 m), the region’s all memstore will flash.
When the size of the memstore reached the hbase) hregion) memstore. Flush. The size (the default value of 128 m) hbase. Hregion) memstore. Block. The multiplier (the default value (4), Prevents further writing to the memStore.
The total size of memStore in Region Server reaches javA_heapsize Hbase. Regionserver. Global. Memstore. The size (the default value of 0.4) hbase. Regionserver. Global. Memstore. Size. The lower. The limit (the default value of 0.95), The region writes memstores in descending order until the total size of memstores in the Region Server decreases below the preceding value. When the region server memstore the total size of the achieve java_heapsize hbase. Regionserver. Global. Memstore. The size (the default value of 0.4), will continue to stop all the memstore write data.
Memstore Flush is also triggered when automatic flush is reached. Automatically refresh interval by this property to configure hbase. The regionserver. Optionalcacheflushinterval 1 hour (the default).
When the number of WAL file over hbase. Regionserver. Max. Logs, region will be carried out in accordance with the time sequence flash, until a WAL file number to hbase. Regionserver. Max. Logs the following (the attribute name has been abandoned, now don’t need to manually, Maximum value is 32)

Because memStore generates a new HFile every time it is flushed, and different versions (TIMESTAMP) and different types (Put/Delete) of the same field may be distributed in different Hfiles, all hfiles need to be traversed during query. To reduce the number of hfiles and clean up stale or deleted data, a StoreFile Compaction occurs.
There are two types of Compaction: Minor Compaction and Major Compaction. Minor Compaction consolidates several nearby, smaller Hfiles into a single, larger HFile, and wipes out some of the expired or deleted data. A Major Compaction compacts all hfiles from a Store into a single HFile and wipes out all expired or deleted data

When a region in a certain Store all StoreFile. The total size of the over hbase hregion). Max filesize, the region will split before (version 0.94).
When the total size of all storefiles in a Store of a region exceeds Min(initialSize), the value of storefiles in a Store of a region exceeds Min(initialSize)R ^ 3, hbase hregion. Max. Filesize “), the Region will split. The default value of initialSize is 2Hbase) hregion) memstore. Flush. The size, R for the current Region belongs to the Table in the Server (version 0.94) after the amount of the Region.

The specific segmentation strategy is as follows:

First split: 1^3 * 256 = 256MB

Second split: 2^3 * 256 = 2048MB

Third split: 3^3 * 256 = 6912MB

Fourth split: 4^3 * 256 = 16384MB > 10GB, so take the smaller value 10GB

The size of each subsequent split is 10GB.
Hbase 2.0 introduces a new split policy: If the current RegionServer on the table only one Region, according to the 2 * hbase. Hregion) memstore. Flush. The size divided, otherwise in accordance with the hbase. Hregion). Max filesize division.