Hbase and Hive are at different positions in the big data architecture. Hbase mainly deals with real-time data query, while Hive mainly deals with data processing and computing.

First, differences:

  1. Hbase: short for Hadoop Database. It is a NoSQL database that is used for random real-time query of detailed volume data (billions or billions), such as log details, transaction lists, and track behaviors.
  2. Hive: Hive is a Hadoop data warehouse. Strictly speaking, Hive is not a database. Hive allows developers to use SQL to calculate and process structured data in the Hadoop Distributed File System (HDFS).
  3. Metadata is used to describe structured text data in Hdfs. Generally speaking, a table is defined to describe structured text data in Hdfs, including column data names and data types, which facilitates data processing. Many SQL ON Hadoop computing engines use Hive metadata. For example, Spark SQL and Impala.
  4. Based on the first point, HDFS data is processed and calculated using SQL, and Hive translates SQL into Mapreduce to process data.

Second, the relationship between

In the big data architecture, Hive and HBase collaborate. Data flows are as follows:

  1. Extract data source to HDFS storage by ETL tool;
  2. Clean, process, and calculate raw data using Hive.
  3. HIve cleaning results can be stored in Hbase if they are for random query of massive data
  4. Data applications query data from HBase.

Hive is a data warehouse tool based on Hadoop. Hive maps structured data files to a database table, provides simple SQL query functions, and converts SQL statements to MapReduce jobs. HBase is a distributed, scalable storage of big data. It may be hard to see the difference between the two individually and literally, but don’t worry, let’s take a look at them in detail.

Characteristics of both

Hive helps people familiar with SQL to run MapReduce jobs. Because it is JDBC-compliant, it can also be integrated with existing SQL tools. Running Hive queries takes a long time because it iterates through all the data in the table by default. Despite this disadvantage, the amount of data traversed at a time can be controlled using Hive’s partitioning mechanism. Partitioning allows filtering queries to be run on datasets stored in different folders, traversing only the data in the specified folder (partition). This mechanism can be used, for example, to process only files within a certain time range, as long as the file names include a time format.

HBase works by storing keys and values. It supports four main operations: add or update rows, view a cell in a range, get the specified row, and delete the specified row, column, or version of the column. Version information is used to retrieve historical data (each row of historical data can be deleted and space can be freed using Hbase Compactions). Although HBase includes tables, schemas are only required by tables and column clusters, and columns do not require a schema. Hbase tables include the add and count functions.

limit

Hive currently does not support update operations. In addition, since Hive runs batch operations on Hadoop, it takes a long time, usually from minutes to hours, to obtain the results of a query. Hive must provide a predefined schema to map files and directories to columns, and Hive is not compatible with ACID.

HBase queries are written in a specific language, which needs to be relearned. Sql-like functionality can be implemented through Apache Phonenix, but this comes at the expense of having to provide a schema. Also, Hbase is not compatible with all ACID features, although it does support some. Last but not least – Zookeeper is required to run Hbase. Zookeeper is a distributed coordination service that includes configuration services, maintenance meta-information, and namespace services.

Application scenarios

Hive can be used to analyze and query data within a period of time, for example, to calculate trends or website logs. Hive should not be used for real-time queries. Because it takes a long time to return results.

Hbase is suitable for real-time query of big data. Facebook uses Hbase for messaging and real-time analysis. It can also be used to count connections to Facebook.

conclusion

Hive and Hbase are two different hadoop-based technologies – Hive is an SQL engine that runs MapReduce jobs, and Hbase is a NoSQL Key/ Vale database on top of Hadoop. Of course, the two tools can be used together. Hive can be used for statistical queries, HBase can be used for real-time queries, and data can be written from Hive to HBase, and Settings can be written from HBase back to Hive.