On January 7th, Hologres and the developer community jointly held the annual conference of real-time data warehouse. At the conference, Guo Bei, a senior technical expert from Ali Cloud, started from the core scene of Ali and deeply interpreted the new trend of real-time data warehouse technology development: one-stop, online and agile. At the press conference, Hologres product manager released Hologres annual key capacity based on the current new trend of data warehouse, helping enterprises to better build one-stop real-time data warehouse.

This article will summarize the key features of Hologres from multiple perspectives to help users better understand and use Hologres.

From business innovation, continuous business evolution

Hologres has been six years since its birth. It first proposed the concept of Hybrid Analytical Processing (HSAP) and gradually developed into an enterprise-level one-stop data warehouse. It not only serves many core businesses within Ali Group (such as Taobao, Tmall, 1688, Caibird, Local Life, Dharma Courtyard, etc.), but also provides continuous cloud service capacity output, helping more domestic and international enterprises to build enterprise-level real-time data warehouse. Up to now, the service has been provided in 13 regions.

New iteration, annual blockbuster release

Highlight

In 21 years, Hologres has made great efforts in engine capability. It has gone through three iterations of core versions, and the next generation product has been in internal testing, still maintaining the pace of rapid iteration. Features such as column and column coexistence, high availability deployment, read/write separation, and rapid Failover significantly improve the stability of data services and enrich service scenarios. 1. Continuous performance improvement: Each major version continues to optimize the engine performance, improving the optimizer, query engine and storage engine. Taking 32core TPCH 100G test results as an example, the performance of version 1.1 has been improved by 256% (query part) compared with last year’s 0.8 version, and the improvement effect is significant

2. Enhanced serving capacity: 100% better row and 30% better column throughput, column co-existence, multiple scenarios in one data set, AliORC format upgrade, higher compression ratio, significantly lower storage cost.

3. Focus on enterprise-level high availability deployment and enterprise-level governance capabilities, and improve self-operation and maintenance capabilities of enterprises by revealing more dimensions and finer granularity of observational indicators.

A more granular description of the key features is provided below.

Native JSON support

JSON is the field that Hologres focuses on recently, because data collection is becoming more and more flexible, and data processing is becoming more and more agile. JSON data that needed to be widened and flattened in the past often retain the original non-standard structure. So how you handle Schemeless scenarios is particularly important.

Traditionally, JSON as a data unit adopts the Schema on Read Schema, which provides flexibility in storage but limits efficiency in analysis. In order to access some nodes in JSON, you have to Read the entire JSON data structure, which is very inefficient and difficult to compress in storage.

Hologres innovates the storage method of JSON data, adopting the storage format similar to column storage in the bottom storage, mapping different node paths in JSON into virtual columns, providing native column compression and index capabilities, and automatically pushing down JSON access operator in the query layer, greatly accelerating the efficiency of JSON data filtering and statistics. In the protocol layer, fully compatible with the PG specification, support JSON, JSONB and other types, support PG native various construction, access, update operators. Based on these innovative capabilities, JSON became Hologres’s recommended data type for buried log analysis scenarios.

Binlog Full link event driver

Similar to the concept of Binlog in MySQL, in Hologres, Binlog is used to record changes to the database table data, such as Insert/Delete/Update operations. Through Flink and JDBC consumption Hologres BInlog, the whole link real-time development between data warehouse levels is realized, and the end-to-end delay of data processing is shortened under the premise of hierarchical governance. Typical application scenarios are as follows:

1. Real-time data replication and synchronization scenarios, a typical scenario is to convert a Hologres row storage table into another column storage table in real time, row storage supports point-check and point-write, column storage supports multi-dimensional analysis requirements, and synchronization logic is supported by Flink. This is a typical use of Hologres prior to V1.1. With Hologres 1.1’s support for row and column co-existence tables, one table can meet both row and column storage requirements, eliminating additional scheduling overhead.
2. Realize the full link drive of events: Through the consumption of Hologres Binlog by Flink, realize the development of event-driven processing and complete the full real-time processing operations from ODS to DWD and from DWD to DWS.
3. ** Data change monitoring: ** Monitors real-time changes of data, such as monitoring real-time changes of inventory data and sensitive fields, and triggers alarms.
4. Data synchronization between instances: Users can create multiple Hologres instances for common layer processing, business domain processing and other duties. Real-time data synchronization between different instances can be completed through Binlog, realizing real-time data flow and sharing.

​

Hologres+MaxCompute

MaxCompute is the scalable big data processing engine of Ali Cloud, with mature technology and stable service. Hologres and MaxCompute both use Pangu storage engine and have the same underlying technology. Hologres uses the native vectorized query engine to accelerate the query of MaxCompute data to achieve the query acceleration capability of MaxCompute. This scenario does not require data migration and data import, high-performance and fully compatible access to a variety of MaxCompute file formats. Establish a fast channel for interactive analysis between BI tools and big data warehouse. In the new version of Hologres, the native vector engine has been used by default to read MaxCompute directly, reducing the cross-engine RPC calls, less data serialization between engines, the same engine, which is conducive to cache reuse, performance improvement of 30%-80%.

One of the most important aspects of real-time offline integration is the simplification of data exchange, even without data movement. External query is a scenario that does not need to move data. However, due to the lack of external index and SSD optimization, the query performance and concurrency are lower than that of internal table scenarios. In performance-sensitive scenarios, it is still recommended that data be synchronized from MaxCompute to Hologres. At present, through the innovation of storage technology, the bidirectional synchronization between MaxCompute and Hologres of millions of rows per second is supported, simplifying the scenario of data development and data flushing.

The connection between MaxCompute and Hologres metadata is also a continuous work. In early days, Hologres supported batch import of metadata, which could import all tables of the whole project at one time or batch update. Recently, Hologres supports automatic metadata import and refresh. When a new table is created in MaxCompute or the table structure changes, the metadata can be automatically synchronized to Hologres, making the experience easier.

At the same time, Hologres provides a new service form: shared clusters for more user-friendly and agile support for MaxCompute acceleration scenarios. In this mode, users do not need to deploy instances in advance. Instead, users use the Serverless mode to charge based on the scan volume of the queried data. This mode only supports MaxComputeBI acceleration scenarios and cannot create internal tables and indexes. Users who used MaxCompute Lightning query acceleration in the past can smoothly switch to this shared cluster without having to rebuild tables or modify BI protocols. More connections are available than Lightning, and the service is more stable.

Accelerate data lake exploration

The traditional architecture of lake and warehouse is still relatively separate, some enterprises in the construction of data lake, data management and data management is relatively difficult, with the advancement of enterprise process, more and more enterprises began to explore the integration of lake and warehouse. Hologres focused on optimizing the integration scenario of lake and warehouse this year, supporting external direct access to OSS data and data backflow OSS, supporting DLF integration, accelerating data lake exploration, and facilitating the integration of lake and warehouse construction.

Deepen the traffic analysis scenario

Traffic analysis is Hologres continues to increase the scene, currently supports the high-performance Bitmap library RoaringBitmap, can be constant complexity, fast calculation of UV, is the key means to solve the Internet level of high base accurate weight removal, UV calculation, widely used in user portrait, tag screening and other typical scenes, It was deeply used in ali Mom’s advertising scenes.

Hologres also has more built-in functions such as funnel analysis, retention analysis and detailed circle people. Through these built-in functions, the development work is greatly simplified and the execution efficiency of complex operations is ensured.

Multi-load resource isolation

Resource group isolation is a means to effectively support mixed load. Hologres supports the design of single instance and multiple resource groups. In this design, the total computing resources are divided into several resource groups, and different resource groups represent certain CPU and memory resources. Users can use only the computing resources of the corresponding resource group. All unbound users use the Default resource group. Currently, resources are not shared or preempted between resource groups. Multiple users in a resource group share the same group of resources.

For example, on the right side of the figure, an instance of 256Core1024G is divided into three resource groups. Different application scenarios use different accounts. For example, OLAP analysis uses Default resource group, which allocates 50% of resources, online spot search requires 20% of resources, and real-time data writing allocates 30% of resources. The isolation capability of resource groups prevents Bad Query from eating up all resources and allows resources to be allocated according to business priorities.

High availability deployment

Resource group isolation is a kind of isolation of computing resources between different thread pools within a single instance. Failure isolation and 100% read and write isolation cannot be achieved. So Hologres continues to innovate better high availability solutions, and in V1.1, supports multi-instance deployment solutions for shared storage. In this scheme, the master instance has complete capability, data can be read and written, permissions and system parameters can be configured, while the sub-instance is in the read-only state, all changes are completed through the master instance, a data store is shared between the instances, and data between the instances is asynchronously synchronized in real time with a delay of milliseconds.

Application scenarios: 1. Read/write separation: this solution implements complete read/write separation and ensures SLA in different business scenarios. In scenarios such as high-throughput data writing and complex architecture jobs, OLAP, AdHoc queries, and online services, the loads are physically isolated from each other without query jitter. 2. Multi-type load fine-grained resource allocation: A master instance can be configured with multiple read-only slave instances, with different specifications between instances depending on the business situation, such as 256Core as a write and machining instance, 512Core as an OLAP read-only instance, 128Core as an online Serving instance, and 32Core as a development test instance.

3. Online service high availability: time of any system is not stable, the disk is damaged, the system Bug, and so on, in high availability support online service scenarios, need in the system level support redundant Yu Degao reliable capacity, failure in a service system, can quickly switch to the standby system, and try to short switching time and consistent state of data. Multiple subinstances can be deployed to provide access to multiple endpoints. When an instance fails, the application layer can quickly switch to other endpoints, implementing high availability and Dr Switching.

Enhance system observability

System observability is continuously visualized through HoloWeb, including slow query, execution plan, active connection, etc., to improve enterprise self-operation and maintenance capabilities:

• Slow Query logs: The system diagnoses and analyzes slow or failed Queries based on time, plan, and CPU consumption, and takes optimization measures to improve the self-diagnosis capability.
• Active connection: through the visual view of active connection, locate the connection status, connection number and user, so as to analyze the instance connection status and diagnose the purpose of running SQL.
• Execution plan: Execute analysis and execution analysis of Query, interpret detailed operators, and guide optimization suggestions through various visual presentation methods to avoid blind tuning, reduce the threshold of performance tuning, and quickly achieve the purpose of performance tuning.

Enterprise security capability upgrade

Enterprise-level security capability is upgraded again, supporting data encryption, access control, disaster recovery and backup.

• Encrypts and stores data based on the Key Management Service (KMS) to provide static data protection and meet enterprise regulation and security compliance requirements.
• Seamless access to MaxCompute, support for accelerating MaxCompute encryption data, further real-time offline integration.
• Data desensitization is supported and data protection umbrella is connected to improve data privacy.

2) Access control

• IP address whitelist is supported to achieve fine-grained security access.
• Interconnects with ActionTrial to support instance-level behavior auditing.

3) Dr And backup

• Cross-room Dr In the same city (private cloud)
• Support DISASTER recovery, remote live, and achieve read and write separation and high read concurrency, also can be based on multiple instances to achieve read high availability. In addition, you can perform version hot upgrade and storage system migration.

PG ecological compatibility

Support for richer data types, more diversified extension functions, and continuous efforts in BI visualization analysis, seamless integration with more BI tools, lowering the learning barrier, and becoming more PostgreSQL compatible.

Continuous focus on one-stop real-time data warehouse

In the future, Hologres will continue to focus on the reliability, efficiency and operational and maintenance capabilities of real-time Data warehouse. It is committed to real-time, focusing on One SQL and One Data, improving enterprise operational and maintenance, reducing the development threshold, and realizing the ultimate goal of one-stop real-time Data warehouse at enterprise level. Specific efforts will be made in the following aspects:

Real-Time

Simplify the development experience of real-time data warehouse, support the Pull mode, take the initiative to Pull data from the upstream message middleware, simplify the process of data falling into the database. We are developing the capability of real-time materialized view. Through real-time materialized view, users can describe complex processing logic as the definition of view. Hologres ensures real-time synchronization between view and table, expresses processing logic of data through SQL, and realizes real-time stratification of data and automation of data flow.

OneSQL

Improved Hologres SQL presentation capability and support for OLAP/Serving/AI scenarios to continuously improve stability and efficiency, not only fast but also stable. Continuously optimize the real-time offline integration experience to further reduce the need for data movement. Improve JSON indexing and compression efficiency. UDF capability will be supported to improve scalability and give users more extension possibilities. Flexibility to computing resources is also the focus.

OneData

Further connect with DLF, MaxCompute, EMR and other storage and metadata systems to avoid data islands and support the integration of lake and warehouse solutions. We are developing hot and cold storage layering, introducing HDD on the basis of SSD, which can significantly reduce the storage cost of users.

Ops

Finally, operation and maintenance will continue to focus on the observability and tunability of the system to avoid system black box, support system thermal upgrade and thermal expansion, and reduce operation and maintenance risks. Data governance capabilities will be greatly improved by partnering with DataWorks to support more metadata analysis and diagnostics.

Attachment: PPT download for the press conference

For PPT materials of this press conference, please go to Dingding user group to obtain