Foreword: I still remember that it was a summer in 2018. The weather was very hot. I was wiping sweat while listening to the leader’s bold talk about his future reform blueprint. By the end of the meeting, the core idea was: we’re going to create a big data pool, we’re going to put as much data as the company can into this big pool, and then we’re going to let other people scoop up the data in various poses. It took more than half a year and less than a year for the system to go online, and it also took more than a year for the system to run stably online. I want to make a quick summary today.

one background

It’s about 15 or 16 years old. It’s old. Because of this, there are too many iterations, too many lines of business, and too many interface calls. Sometimes A asks B for data, sometimes B asks C for interfaces, and sometimes C asks A for services. Each line of business each has its own property, their own camp, like a small vassal of self-respect, the cost of running errands and meeting costs are very expensive. Faced with this situation, we urgently need a wave of transformation.

This system (let’s call it Tianchi) is designed to integrate the resources of various business lines of the company and transform this intricate spider web into a simple linear bus. Avoid unnecessary interface calls, business interludes, meeting communication and the trouble of not knowing where to get data, not getting data, and getting data slowly. Of course, it saves the time of products and developers and improves the overall work efficiency of all business lines.

A few words to describe tianchi: steady, fast, large, economical and clear.

two The business card

After sorting out various lines of business of the company, the following common data output models are summarized:

  • Key-value Fast output type, the simplest KV query, concurrency may be very high, fast speed requirements. Risk control.

  • Key-map fast output type, directional output, such as the common by the article ID to obtain article details data, KV query upgrade version.

  • Multikey-map batch output, such as the common recommendation Feed stream display, key-map query upgrade version.

  • C-list Multi-dimensional query output type, which specifies multiple criteria for data filtering. The criteria may be flexible, and the data that meets the criteria is output in pages. This should be fairly common, such as filtering items that have been labeled or rated for recommendations, retrieving items that have been purchased by a given user in the past, and so on.

  • G-top Displays the statistics rankings grouped by certain dimensions. Such as the highest heat of a forum Top10 posts.

  • G-count Statistical analysis output type, data warehouse statistical analysis demand type.

  • Multi-table mixed output, and different tables have different query conditions. For example, the output content of the list page is mixed.

  • Term word segmentation output type

There may be more data models, which are not listed here. From the front end to the back end, no matter how many data models, in fact, can be converted to the form of index +KV output, even sometimes, I think index +KV>SQL.

Based on the analysis of the business data model and the company’s long-term use of ElasticSearch, we finally choose HBase + ElasticSearch to implement.

3. Architecture design and module introduction

First, take a look at the overall architecture diagram, as shown below:

The whole Tianchi system core is mainly divided into data access layer, policy output layer, metadata management, index establishment, platform monitoring and offline data analysis six sub-modules, which will be introduced respectively below.

1. Data access module

Data access module The hBase-Client API is lightly encapsulated twice. It supports online RESTFUL service interfaces and offline SDK packages to provide services externally, and is compatible with HBase native apis and HBase BulkLoad for mass data writing.

The online RESTFUL service provides services externally in HBase Connection mode. It has the following advantages: Easy cross-language operation without significant impact on performance, and more importantly, easy management. At this level, you can do a lot of things, such as permissions management, load balancing, failure recovery, dynamic scaling, data interface monitoring, and more, all thanks to the power of K8S.

2. Introduction to the policy output module

This module is mainly to connect with the various business needs summarized by the above business sorting module, and provide services from this module. As the name implies, the policy module is mainly used to configure policies for users or users themselves, and finally generates policy ids based on policies.

At this layer, we mainly encapsulate ElasticSearch and HBase. We use dynamic templates to convert user requests into ElasticSearch DSL statements, and then query ES to directly return data or obtain rowkey to query HBase and return results.

Using the metadata management center, you can determine whether required fields are overwritten by index fields and whether it is necessary to query the HBase result twice. The entire query process is not perceived by the user. All they need is a PolicyID.

Of course, we are also educating users on how to configure generation policies themselves in the background. Business parties with more cooperation can even configure everything in the test environment themselves to complete the self-service data retrieval work. All we need to do is one-click synchronize the test environment’s policies to the online environment and notify them that they are available online. After 5-10 minutes, a new interface is created.

Secondly, because ES is not as strong as HBase in working under pressure, our policy interface determines whether to enable caching based on service requirements. In fact, most interfaces accept data caching for short periods of time. Of course, simple KV, K-map, MK-Map are directly HBase, the demand is also quite large.

Up to now, most of the above business output models have supported dynamic policy configuration. Thanks to ElasticSearch’s syntax and ability to cover business scenarios, ElasticSearch is more of a business product in my opinion. When you dig into ES, you’ll see that it’s really more powerful than SQL in some ways; The policy platform now even supports word segmentation, bucket segmentation, multi-table federation, TopN, aggregate query, and many more composite queries thanks to ElasticSearch.

3. Metadata management module

As we all know, HBase is a no-Schema model. The metadata management team does a virtual Schema management for it and ES, and dynamically controls which fields are indexed. At the time of data access, we will use the metadata center to determine whether the data conforms to the rules (some rules we set ourselves); During data output, we control which policies need to go to cache and which policies do not need to go to HBase. Secondly, the maintenance of a set of metadata is convenient for us to do some simple page indicator monitoring, and there is a bus control for ES and HBase (such as table building and table deletion), this module is not mentioned.

4. Introduction to index building module

This module, in fact, is a relatively complex module. We use HBase + Kafka + ES instead of HBase + WAL + ES to synchronize index data. The WAL layer is difficult to control and monitor, THE ES consumption efficiency of WAL, and the data consistency of WAL layer is difficult to maintain.

Therefore, we put part of the work into the data access layer. After the data is written to HBase, it responds to Success externally and asynchronously pushes the data to the Kafak queue for ES to consume again. If writing fails, an exception is thrown. Ensure that writing HBase succeeds or fails.

In the ES consuming layer, we can dynamically specify the number of consuming threads. When the accumulation of Kafka Lag exceeds a certain threshold (a threshold for group-level adjustment and monitoring), an alarm is generated and the number of consuming threads is dynamically adjusted.

We also do a lot of work on data consistency, and we only guarantee final consistency. After data is successfully written to HBase, link tracing is performed for write Kafka and write ES. If writing fails in either link, the Failed Key will be written to the blacklist (Redis storage).

For blacklisted data, we will start a scheduled thread to scan these keys and perform automatic index filling. The latest data is written to the queue again from HBase. If we fail again at this time, we will put these keys into the final death list (Redis storage), and scan the death list through the scheduled thread. If there is a body, we will alarm, and then human intervention.

This layered processing mode is also based on the HBase LSM concept. Do not spray

I have drawn the process briefly to make it easier for you to understand, as shown below:

5. Platform monitoring module introduction

This module will not be detailed, mainly monitoring Hadoop cluster, HBase cluster, plus K8S platform monitoring. K8S monitoring platform is mainly based on Prometheus+Grafana+Fluent.

6. Offline data analysis module is introduced

This module relies on HBase Replication between clusters. After data is synchronized to an offline HBase cluster, it is used for data warehouse interconnection, Spark read/write analysis, and large-scale scan operations. The main purpose is to reduce the impact of analytical operations on the online real-time platform.

The six modules are briefly introduced here.

Four. tips

General feeling: IT feels good to use ES for HBase, ES is great, ES+HBase is really comparable to SQL.

It seems that ES and HBase are a family by nature. HBase supports dynamic columns, and ES also supports dynamic columns, which makes the combination of the two very harmonious. The powerful index function of ES is not provided by HBase. If service index fields are stored in ES, the volume is not large. Even in many cases, more than 60% of the business index fields are Term types, requiring no word segmentation at all. Although we do support participles, such as multi-label indexes.

Many designers may find the combination of HBase + Kafka + ES too heavy and costly to operate and maintain. But if you think about it from a different point of view, we are not just engaged in technology, so we can learn three mature products together! Now it seems that the gain is more than the cost.

As for who ES and Solr choose to do the secondary index, I don’t think there is much difference, just choose according to the current situation of their own companies.

Finally, give ElasticSearch a thumbs up! Nice product!

Five. Things to do in the future

  • The multi-tenant full link is established. Procedure
  • Policy layer SQL support
  • The system is constantly optimized and productized

Reprint please indicate the source! Welcome to follow my wechat official account [HBase Work Notes]