1. Hbase Overview

1. Basic description

Hadoop’s native feature is to solve the offline batch processing scenario of large-scale data. HDFS has powerful storage capacity, but does not provide a strong data query mechanism. The HBase component provides services similar to BigTable based on the HDFS file system.

HBase is a distributed and scalable NoSQL database that supports massive structured data storage. HBase provides capabilities similar to Bigtable on top of Hadoop, with a column-based storage mode instead of a row-based mode. Data storage features: For unstructured or loose semi-structured data, storage of large tables naturally requires the ability to scale horizontally and process massive data based on service clusters.

2. Data model

Basic description of Hbase data structure;

Table-table: consists of rows and columns divided into several column families.
Row-row: rows identified by keys representing data objects.
Column family: Column family supports dynamic expansion and is stored as a string.
Column identifiers: Data in a column family is located by column identifiers;
Cell: Row keys, column families, and column identifiers work together to identify a cell;
Cell data: Data stored in a cell is called cell data;
Timestamp: Version identification is based on the timestamp by default.

The HBase data model is similar to that of a relational database. Data is stored in a table with rows and columns. However, the underlying physical storage structure of HBase is more like a Map (K-V) collection.

Data management is based on the characteristics of column storage;
A simple data model where content is stored as a string;
No complex table relationship, simple add and delete operations;

In the data model as a whole, HBase is a sparse, multidimensional, sorted mapping table whose indexes are row keys, column families, column qualifiers, and timestamps. Each value is an uninterpreted string.

2. Build a cluster environment

1. Unzip the files

The tar - ZXVF hbase - 1.3.1 - bin. Tar. GzCopy the code

2. Configure environment variables

Vim /etc/profile export HBASE_HOME=/opt/hbase-1.3.1 export PATH=$PATH:$HBASE_HOME/bin source /etc/profileCopy the code

3. Configure hbase-env

Vim /opt/hbase-1.3.1/conf/hbase-env.sh export JAVA_HOME=/opt/jdk1.8 export HBASE_MANAGES_ZK=falseCopy the code

4. Configuration: hbase-site

Vim/opt/hbase - 1.3.1 / conf/hbase - site. XML<configuration>
    <! - HDFS storage - >
	<property>
		<name>hbase.rootdir</name>
		<value>hdfs://hop01:9000/HBase</value>
	</property>
    <! -- Start cluster -->
	<property>
		<name>hbase.cluster.distributed</name>
		<value>true</value>
	</property>
	<! -- -- -- > port
	<property>
		<name>hbase.master.port</name>
		<value>16000</value>
	</property>
    <! - ZK cluster -- -- >
	<property>   
		<name>hbase.zookeeper.quorum</name>
	     <value>hop01,hop02,hop03</value>
	</property>
    <! - ZK data - > 
	<property>   
		<name>hbase.zookeeper.property.dataDir</name>
	     <value>/data/zookeeper/data/</value>
	</property>
</configuration>
Copy the code

5. Configuration: RegionServers

Vim /opt/hbase-1.3.1/conf/ RegionServers HOP01 HOP02 Hop03Copy the code

6. Configuration: soft connection

Soft connection hadoop configuration file to HBase

Ln -s/opt/hadoop2.7 / etc/hadoop/core - site. XML/opt/hbase - 1.3.1 / conf/core - site. XML ln -s / opt/hadoop2.7 / etc/hadoop/HDFS - site. XML/opt/hbase - 1.3.1 / conf/HDFS - site. XMLCopy the code

7. Synchronize the cluster service environment

You can also configure the cluster manually or use synchronization commands.

xsync hbase/
Copy the code

8. Start the cluster

Start at hop01.

The/opt/hbase - 1.3.1 / bin/start - hbase. ShCopy the code

Startup log:

Hop03: Starting RegionServer, logging to /opt/hbase-1.3.1/bin/.. Out HOP02: Starting regionServer, logging to /opt/hbase-1.3.1/bin/.. /logs/ hbase-root-regionServer-hop02. out HOP01: Starting regionServer, logging to /opt/hbase-1.3.1/bin/.. /logs/hbase-root-regionserver-hop01.outCopy the code

9. Check the status

JPS HMaster: primary node HRegionServer: partition nodeCopy the code

10. Stop the cluster

Stop at hop01.

The/opt/hbase - 1.3.1 / bin/stop - hbase. ShCopy the code

11. View the interface

http://hop01:16010
Copy the code

Basic Shell commands

1. Access the client

The/opt/hbase - 1.3.1 / bin/hbase shellCopy the code

2. Look at the table

hbase(main):002:0> list
Copy the code

Create table

Hbase (main):003:0> create 'user','info' 0 row(s) in 2.7910 seconds => hbase :: table-userCopy the code

4. View the table structure

hbase(main):010:0> describe 'user'
Copy the code

5. Add data

put 'user','id01','info:name','tom'
put 'user','id01','info:age','18'
put 'user','id01','info:sex','male'
put 'user','id02','info:name','jack'
put 'user','id02','info:age','20'
put 'user','id02','info:sex','female'
Copy the code

6. View table data

hbase(main):010:0> scan 'user'
ROW     COLUMN+CELL                                                                             
id01    column=info:age, timestamp=1594448524308, value=18                                      
id01    column=info:name, timestamp=1594448513534, value=tom                                    
id01    column=info:sex, timestamp=1594448530817, value=male                                    
id02    column=info:age, timestamp=1594448542631, value=20                                      
id02    column=info:name, timestamp=1594448536520, value=jack                                   
id02    column=info:sex, timestamp=1594448548005, value=female
Copy the code

These table structures and data are automatically synchronized across clusters.

7, query the specified column

hbase(main):012:0> get 'user','id01'
COLUMN      CELL                                                                                    
info:age    timestamp=1594448524308, value=18                                                       
info:name   timestamp=1594448513534, value=tom                                                       
info:sex    timestamp=1594448530817, value=male
Copy the code

8. Count rows

hbase(main):013:0> count 'user'
Copy the code

Delete row data

hbase(main):014:0> deleteall 'user','id02'
Copy the code

10. Clear the table

hbase(main):016:0> truncate 'user'
Copy the code

Drop table

hbase(main):018:0> disable 'user'
hbase(main):019:0> drop 'user'
Copy the code

4. JDBC basic query

1. Core dependencies

<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-client</artifactId>
    <version>1.3.1</version>
</dependency>
Copy the code

2. Basic configuration

Connect to the Address of the ZooKeeper cluster.

Zookeeper: address: indicates the cluster address. The urls are separated by commasCopy the code

Write HBase configurations and common tools.

@Component
public class HBaseConfig {

    private static String address;
    private static final Object lock=new Object();
    public static Configuration configuration = null;
    public static ExecutorService executor = null;
    public static Connection connection = null;

    /** * get the connection */
    public static Connection getConnection(a){
        if(null == connection){
            synchronized (lock) {
                if(null == connection){
                    configuration = new Configuration();
                    configuration.set("hbase.zookeeper.quorum", address);
                    try {
                        executor = Executors.newFixedThreadPool(10);
                        connection = ConnectionFactory.createConnection(configuration, executor);
                    } catch(Exception e) { e.printStackTrace(); }}}}return connection;
    }

    /** * Get HBaseAdmin */
    public static HBaseAdmin getHBaseAdmin(a){
        HBaseAdmin admin = null;
        try{
            admin = (HBaseAdmin)getConnection().getAdmin();
        }catch(Exception e){
            e.printStackTrace();
        }
        return admin;
    }
    /** * get Table */
    public static Table getTable(TableName tableName) {
        Table table = null ;
        try{
            table = getConnection().getTable(tableName);
        }catch(Exception e){
            e.printStackTrace();
        }
        return table ;
    }
    /** * Close the resource */
    public static void close(HBaseAdmin admin,Table table){
        try {
            if(admin! =null) {
                admin.close();
            }
            if(table! =null) { table.close(); }}catch(IOException e) { e.printStackTrace(); }}@Value("${zookeeper.address}")
    public void setAddress (String address) { HBaseConfig.address = address; }}Copy the code

3. Query cases

Query data refer to the scan results of all tables:

@RestController
public class HBaseController {

    /** * scan all tables */
    @GetMapping("/scanTable")
    public String scanTable (a) throws Exception {
        Table table = HBaseConfig.getTable(TableName.valueOf("user"));
        try {
            ResultScanner resultScanner = table.getScanner(new Scan());
            for(Result result : resultScanner) { printResult(result); }}finally {
            HBaseConfig.close(null, table);
        }
        return "success";
    }

    /** * Scan */ based on RowKey
    @GetMapping("/scanRowKey")
    public void scanRowKey(a) throws Exception {
        String rowKey = "id02";
        Table table = HBaseConfig.getTable(TableName.valueOf("user"));
        try {
            Result result = table.get(new Get(rowKey.getBytes()));
            printResult(result);
        } finally {
            HBaseConfig.close(null, table); }}/** * Outputs Result */
    private void printResult (Result result){
        NavigableMap<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> map = result.getMap();
        Set<Map.Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>>> set = map.entrySet();
        for (Map.Entry<byte[], NavigableMap<byte[], NavigableMap<Long, byte[]>>> entry : set) {
            Set<Map.Entry<byte[], NavigableMap<Long, byte[]>>> entrySet = entry.getValue().entrySet();
            for (Map.Entry<byte[], NavigableMap<Long, byte[]>> entry2 : entrySet) {
                System.out.print(new String(result.getRow()));
                System.out.print("\t");
                System.out.print(new String(entry.getKey()));
                System.out.print(":");
                System.out.print(new String(entry2.getKey()));
                System.out.print(" value = ");
                System.out.println(newString(entry2.getValue().firstEntry().getValue())); }}}}Copy the code

END

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Data warehouse components: HBase cluster environment construction and application case

1. Hbase Overview

2. Build a cluster environment

Basic Shell commands

4. JDBC basic query

Data warehouse components: HBase cluster environment construction and application case

1. Hbase Overview

2. Build a cluster environment

Basic Shell commands

4. JDBC basic query

Related Posts

Guess the number size | LeetCode brush notes

50 most commonly used Unix/Linux commands

Thinking Logic of Computer Programs (33) – joda-time