This article is compiled by Lu Peijie, a community volunteer, and shared by Wang Zhanxiong, senior engineer of Tencent Hotspot data team, in Flink Forward Asia 2020. Tencent Hotspot builds real-time data warehouse and real-time query system based on Flink. The contents include:

  1. background
  2. Architecture design
  3. Number of real-time warehouse
  4. Real-time data query system
  5. Summary of application achievements of real-time system

I. Background introduction

1. Business pain points that need to be addressed

Recommendation system

For the recommendation students, I want to know the recommendation effect of a recommendation strategy in different groups.

operating

For the operation of the students, want to know in Guangdong province users, the most popular guangdong regional content is what? Convenient for regional push.

audit

For those who have reviewed, I want to know the most reported content and accounts of games in the past 5 minutes, so that I can handle them in time.

Content creation

The author of the content wants to know how many views, likes and retweets the content has received so far today so he can adjust his strategy in time.

The boss decision

For your boss, wanting to know how many users have consumed content in the last 10 minutes gives you a big picture of who is consuming it.

These are the business scenarios we often encounter in our daily work, and the corresponding solutions will be provided in the following section.

2. Pre-development research

We did the following research prior to development.

2.1 Can offline data analysis platform meet these requirements

The conclusion of the survey is that offline data analysis platform cannot be satisfied for the following reasons:

  • First, the consumer behavior data is reported through Spark’s multi-layer offline calculation, and the final result is exported to MySQL or ES for the offline analysis platform to query. The delay of this process is at least 3-6 hours. At present, it is common to provide the query on the next day, so many business scenarios with high real-time requirements cannot be met.
  • Another problem is that the data volume of Tencent hotspot is too large, bringing relatively large instability, often unexpected delays, so offline analysis platform is unable to meet these needs.

2.2 Quasi-real-time data analysis platform

Tencent provides the function of quasi-real-time data query internally. The underlying technology is Kudu + Impala. Impala is a big data computing engine based on MPP architecture, and accesses Kudu which stores data in columns. However, for the real-time data analysis scenario, its query response speed and data delay are relatively high. For example, a query of real-time DAU returns can take at least a few minutes and does not provide a good interactive user experience.

Therefore, the speed advantage of Kudu+Impala, a general big data processing framework, is more than that of Spark and HDFS, an offline analysis framework, which cannot meet our higher real-time requirements. Therefore, development is required, which involves solution selection and architecture design.

3. Business process of Tencent’s Hotspot information flow

In the introduction of Tencent hotspot information flow business process, understand the business process, will be able to better understand the technical architecture of the scheme.

  • Step 1: Content creators publish content;

  • Step 2, the content will be enabled or removed by the content review system;

  • Step 3: The enabled content is sent to the recommendation system and operation system and distributed to c-side users;

  • Step 4: After the content is distributed to users on the C side, users will have various behaviors, such as exposure, click report, etc., and these behavioral data will be reported through buried points and connected to the message queue in real time.

  • Step 5, build a real-time data warehouse;

  • The sixth step is to build a real-time data query system.

Our work is mainly in step 5 and step 6. You can have a look at our business flow chart for further understanding.

In the business flow chart, we mainly do the two parts, which are colored in the figure:

  • In the orange part, we built a real-time data warehouse of Tencent Hotspot;
  • In green, we developed a real-time data analysis system based on OLAP storage computing engine.

Why build a real-time data warehouse? Because the amount of original data reported is very large, the peak of the reported data is over a trillion pieces in a day, and the reported format is very chaotic, lacking the dimension of content and the user’s portrait information, the downstream can not be directly used.

And we provide real-time data warehouse, it is according to read tencent information flow of the business scenario and the content dimension of associations, associations and a picture of the users of all kinds of particle size of aggregate, the downstream can be very convenient, the use of real-time data and real-time data warehouse can provide for the use of downstream users repeated consumption, can reduce duplication of work.

The multidimensional real-time data analysis system in the green part consumes the real-time data warehouse provided by us and uses the OLAP storage computing engine to store massive data efficiently and provide high-performance multidimensional real-time analysis function.

Second, architecture design

1. Design objectives and difficulties

First look at the data analysis system design objectives and difficulties. Our real-time data analysis system is divided into four modules:

  • Real-time computing engine;
  • Real-time storage engine;
  • Background service layer;
  • Front end display layer.

The difficulty lies mainly in the first two modules, real-time computing engine and real-time storage engine.

  • It is difficult to access the massive data of tens of millions of levels per second in real time and carry out extremely low delay dimension table association.
  • How real-time storage engines support high concurrency writes. Highly available distributed and high-performance index queries are difficult, so take a look at our system architecture design to see how these modules are implemented.

2. System architecture design

About the system architecture design, mainly from the following aspects.

2.1 Real-time Computing

  • The access layer mainly separates the microqueues of different business and behavior data from the original message queues of tens of millions of levels per second. Take QQ watch video content for example, after the split data is only a million per second.
  • The real-time computing layer is mainly responsible for the operation of “row to column” of multi-line behavior data and real-time association of user portrait data and content dimension data.
  • Real-time data warehouse storage layer is mainly designed to meet the business of the hotspot, the downstream easy to use real-time message queue.

We provisionally provide two message queues as two layers of a real-time warehouse:

  • The first layer is the DWM layer, which is granular aggregate of content ID and user ID, meaning that one piece of data contains content ID and user ID, then content dimension data on the B side, user behavior data on the C side, and user portrait data.
  • The second layer is the DWS layer, which is the aggregation of content ID granularity, that is, a piece of data contains content ID, B-side data and C-side data. You can see messages with content ID and user ID granularity, the queue traffic is further reduced to 100,000 levels per second, the content ID granularity is reduced to 10,000 levels per second, and the format is clearer and the dimension information is richer.

2.2 Real-time Storage

  • The real-time write layer is mainly responsible for Hash routing and writing data.
  • OLAP storage layer is a storage engine that uses MPP to design indexes and materialized views in line with business and store massive data efficiently.
  • Background interface layer is to provide efficient multidimensional real-time query interface.

2.3 Background Service

The background service is written based on the RPC background service framework developed by Tencent, and some secondary caching will be carried out.

2.4 Front-end Service

The front end uses the open source component Ant Design, using Nginx, reverse proxy browser requests to the background server.

3. Scheme selection

As for the selection of architectural design solutions, we compared the leading solutions in the industry and finally chose the solution that best fits our business scenario.

3.1 Selection of real-time data warehouse

We chose Lambda architecture, which is relatively mature in the industry. Its advantages are high maturity, high flexibility, low migration cost and so on. However, it has a disadvantage, real-time and offline use of two sets of code, there may be one caliber modified data, but the other is not modified, resulting in data inconsistency. Our solution does data reconciliation every day, and alarms will be generated if there is any abnormality.

3.2 Selection of real-time computing engine

We chose Flink as a real-time computing engine because Flink was originally designed for Streaming, Sparks Streaming is technically microbatch and Storm doesn’t use it much anymore. In addition, Flink has the characteristics of exact-once accuracy, lightweight fault tolerance mechanism, low delay and high throughput, and high applicability, so we choose Flink as a real-time computing engine.

3.3 Real-time Storage Engine

Our requirements are to have dimensional indexes, support high concurrency writes, and high performance multidimensional real-time OLAP queries. It can be seen that HBase, TiDB, and ES cannot meet the requirements. Druid has a bug in that it divides the segments by time sequence, which means that you can’t store all of the same content in the same Segment, so you can only calculate the global Top N value by approximation. We chose Clickhouse, an MPP database engine that has been popular in the last two years. I will introduce the advantages of Clickhouse in combination with our usage scenarios and the principles of the Clickhouse kernel.

Three, real-time count warehouse

Real-time data warehouse is also divided into three pieces to introduce:

  • The first is how to construct real-time data warehouse;
  • The second is the advantages of real-time data warehouse;
  • The third is based on real-time data warehouse, using Flink to develop real-time applications encountered some problems.

The difficulty of real-time data warehouse is that it is in a relatively new field, and the gap between the various businesses of each company is relatively large, how to design a convenient and convenient real-time data warehouse in line with the business scene of hotspot information flow is difficult.

1. How to construct real-time data warehouse

Let’s take a look at what real time stacks are going to do. The real-time data warehouse refers to several message queues externally. Different message queues store real-time data of different aggregation granularity, including content ID, user ID, user behavior data on the C side, content dimension data and user portrait data on the B side. Building a real-time data warehouse can be divided into three steps.

1.1 Data Cleaning

Firstly, clear real-time data can be obtained by complex data cleaning operations from massive original message queues. Its specific operation is actually in the real-time calculation link of Flink, the window is first aggregated according to the granularity of one minute, and the behavior data of multiple lines in the window is converted into the data format of multiple lines and columns.

1.2 High-performance Dimension Table Association

The second step is to carry out high-performance real-time dimension table association to supplement user portrait data and content dimension data. However, a large amount of user portrait data exists in HDFS, and content dimension data exists in HBase, so it is a technical challenge to achieve extremely low-latency dimension table association. This will be covered separately later.

1.3 Different particle size polymerization

The third step is to aggregate the calculated real-time data according to different granularity, and then put it into the corresponding message queue for storage, which can be provided to the downstream multi-user reuse. Here the real-time data warehouse is built.

Next, take a closer look at how high performance real-time dimension table association is handled in step 2.

Billions of user portrait data stored in HDFS cannot be associated with high-performance dimension tables, so it needs to be cached. Due to the large amount of data, the cost of local cache is unreasonable. Redis is used for cache. The detailed implementation is to read portrait data in HDFS in batches by Spark, update Redis cache every day, and store content dimension data in HBase.

In order not to affect online services, we access the HBase standby database. In addition, content dimensions change more frequently than user portraits. Therefore, we need to associate real-time HBase data as much as possible during dimension association.

If data in the one-minute window is directly associated with HBase, it takes more than 10 minutes, which results in task delay. We found that 1000 pieces of data can access HBase in seconds, while accessing Redis in milliseconds. The speed of accessing Redis is basically 1000 times that of accessing HBase. Therefore, we set up a Redis cache before accessing HBase content. The hbase-proxy write flow is monitored to ensure cache consistency.

Such one-minute window data, which used to take more than ten minutes to associate content dimension data, now becomes second level. To prevent the cache from wasting expired data, we set the expiration time of the cache to 24 hours.

Finally, there are some minor optimizations. For example, in the process of content data reporting, many unconventional content ids are reported. These content ids are not stored in HBase, resulting in cache penetration problems. So in real time, we filter out these content ids directly to prevent cache penetration and save some time. In addition, because the timing cache is set, a cache avalanche problem will be introduced, so we carried out the operation of peak clipping and valley filling in the real-time calculation process, stagger the setting time of cache to alleviate the cache avalanche problem.

2. Advantages of real-time data warehouse

We can look at the difference between developing a real-time application before and after we build a real-time warehouse.

When there is no data warehouse, we need to consume tens of millions of original queues per second, perform complex data cleaning, and then perform user portrait association and content dimension association to get real-time data in the required format. Development and expansion costs are high. If you want to develop a new application, you have to go through the process again. Now with real-time data warehouse, if you want to develop a real-time application of content ID granularity, you can directly apply for TPS ten thousand level per second DWS layer message pair column, development cost is much lower, resource consumption is much smaller, scalability is also much stronger.

Let’s look at a practical example. Developing a large real-time data screen for our system, which used to require all of the above operations to get the data, now only requires consuming DWS layer message queue to write a Flink SQL, which only consumes 2 CPU cores and 1GB of memory. 50 consumers, for example, the establishment of real-time before, during and after the number of positions downstream to develop a real-time application, can reduce 98% of the resource consumption, including computing resources, storage resources, manpower cost and developers learning access cost, etc., and with the more consumers save more, take the Redis store for this part, You can save millions of yuan a month.

3. Summary of problems encountered in the development of Flink

I have encountered many problems in the process of using Flink to develop real-time applications. Here I choose a few representative ones to share with you.

3.1 Real-time data large screen

The first one is the development of real-time data large screen, which is realized by Flink SQL at the beginning. The function is very simple, that is, to calculate the number of clicks accumulated on the day up to the current, and the realization method is also very simple. The input source table is the message queue of real-time data warehouse. The output sink table is Redis. Select sum(click) from sourceTable Group by day time.

This task appears to be ok, but in practice the data cannot be updated in real time, because the source table will increase the cumulative value of each click of data, and then write the latest data to Redis. Therefore, when the amount of data is too large, it will frequently write Redis, so that the network delay of writing Redis will be very high, and the back pressure data cannot be updated in real time.

We made a simple optimization, after executing SQL using the TABLE API, converting it to DataStream, and then printing only the latest cumulative value to Redis per second through a one-second data window, so that the data can be updated in real time.

3.2 TTL of Flink state

Version 1.6 of Flink introduced state TTL. After state TTL is enabled, Flink will add a timestamp field for each keyed state. By the timestamp field, it can determine whether the state is expired and whether it needs to be cleared. However, taking this literally can be problematic. Prior to version 1.10, Flink did not automatically clear expired states by default, although state TTL was enabled. Heap Memory Backend causes OOM problems. If rocksDB Backend is used, the state becomes larger and larger, which may take a long time to restart. After further investigation, we found that there are two ways to clear the expired state of Flink.

The first is manual cleaning, the second is automatic cleaning. We ultimately chose to manually trigger the cleanup of expired states. Every night in the middle of the night, when the business is at a low ebb, we will traverse the data in the state and clean the expired data when it is accessed.

The reason why we did not choose Flink’s automatic cleanup policy is because prior to Flink’s 1.8 release, there was only one automatic cleanup policy, clean Up in Full Snapshot. The name of this clearing strategy indicates that it will clear the state when the full snapshot is being created. However, it has a fatal flaw. It does not reduce the size of its state, but only puts the cleared state into the snapshot, and eventually it will still get OOM. Also, it can’t load the previously cleared state until it restarts, which causes it to restart frequently.

Although two automatic cleanup strategies were added after version 1.8, since it is asynchronous, the timing and usage of manual cleanup are not as flexible as manual cleanup, so we finally chose manual cleanup. After version 1.10, the automatic clearing policy is selected by default. However, users are required to have a good understanding of the automatic clearing policy timing and policy to better meet service requirements.

3.3 Experience summary of Flink valueState and mapState

ValueState can also be used to store map structured data. However, mapState should be used where mapState can be used. It is better not to use valueState to store map structured data. Because Flink is optimized for mapState, it is more efficient than valuState to store map-structured data.

For example, we encountered a problem where valueState was used to store map data and rocksDB Backend was used. We find that disk IO is getting higher and higher, and latency increases accordingly. In valueState, changing any key in a map will read out the data in the map and then write back the data. As a result, I/OS are too high. But with mapState, where each key is a separate key in rocksDB, disk IO costs are much lower.

3.4 Checkpoint Timeout Problem

We also encountered some problems, such as Checkpoint timeout. At that time, our first thought was that the timeout was caused by insufficient computing resources and insufficient parallelism. Therefore, we directly increased computing resources and increased parallelism, but the timeout situation was not alleviated. After further research, it was found that data skew caused the delay in issuing the barrier of a node. This can only be solved after rebalance.

In general, Flink has strong functions. It has perfect documents, abundant online information and active community. Generally, it can find solutions to problems quickly.

Real-time data query system

Our real-time query system, multi-dimensional real-time query system is implemented by Clickhouse, which is introduced in three parts. The first is distributed high availability, the second is massive data write, the third is high-performance query.

Click House has many table engines. Table engines determine how data is stored, how data is loaded, and what features data tables have. Clickhouse currently has more than 20 table engines including MergeTree, replaceingMerge Tree, AggregatingMergeTree, external memory, MEMORY, IO, etc. Merge Tree and its family of table engines are the only Clickhouse engines that support primary key indexes, data partitioning, and data duplicates. We are currently using Clickhouse’s Merge Tree and its family of table engines, and the rest of the introduction is based on that engine.

1. Distributed high availability

No matter how strong the performance of a single node is, with the growth of services, there will be a bottleneck one day sooner or later, and unexpected downtime is inevitable in the running of computers. Clickhouse scales the cluster horizontally by shards, dividing the total data level into m points, then storing one copy of data per shard to avoid the performance bottleneck of a single node, and then ensuring high availability of the cluster by duplicates, i.e. having several copies of data per shard.

In Clickhouse’s default high availability scenario, data is written to a distributed table, which then simultaneously writes data to all copies of the same shard. There is a problem here. If the write from copy 0 succeeds and the write from copy 1 fails, the data of different copies of the same shard will be inconsistent. Therefore, the default high availability scheme cannot be used in the production environment.

We are following Clickhouse’s official advice to implement high availability with Zookeeper. When data is written to a shard, only one copy is written to Zookeeper. Zookeeper tells other copies of the same shard to pull data. Ensure data consistency.

Let’s take a look at the underlying rationale for Clickhouse to implement this highly available solution through Clickhouse’s Replicated Merge Tree table engine. In the core code of the Replicated Merge Tree table engine, there is a lot of logic to interact with Zookeeper to achieve the synergy of multiple replicas, including the primary replicas’ election to write changes to the task queue and state changes of replicas. As you can see, external data written to a shard in Clickhouse is written to memory in a replica, sorted in memory according to specified criteria, and then written to a temporary directory on disk. Finally, the temporary directory is renamed to the name of the final directory. After the data is written, a series of interactions are performed through Zookeeper to achieve data replication.

Message queues are not used for data synchronization, because Zookeeper is more lightweight, and any copy can be written to Zookeeper to obtain consistent data. Even if other nodes fail to obtain data the first time, If it is found to be inconsistent with the data record on Zookeeper, the system tries to obtain data again to ensure data consistency.

2. Write massive data

2.1 Append + the Merge

The first problem with data writing is that writing Clickhouse directly to large amounts of data will fail. Clickhouse’s Merge Tree family of table engines is similar to LSM Trees in that data is written to an append and then the Merge thread is started to merge small data files. Looking at the writing process of the Clickhouse Merge Tree family of table engines, two problems emerge.

  • If too little data is written at a time, for example, one piece of data is written only once, a large number of file directories can be created. When the background merge thread is too late to merge, the number of file directories increases, causing Clickhouse to throw too Many Parts and fail to write.
  • In addition to the data itself, Clickhouse also needs to interact with Zookeeper about 10 times per write. Zookeeper is not capable of high concurrency, so we can see that Clickhouse QPS is too high. Zookeeper crashes.

The solution we adopted was to write in batch mode, write a batch of zooKeeper data, generate a data directory, and then interact with ZooKeeper once. So how big is batch set? If the batch is too small, it won’t relieve the pressure on Zookeeper. However, batch cannot be set too large, otherwise the upstream memory pressure and data latency will be large. Therefore, through the experiment, we finally chose batch with a size of several hundred thousand, which can avoid the problem caused by high QPS.

In fact, the current scheme still has room for optimization. For example, Zookeeper cannot be linearly expanded. I have learned that some teams in the industry do not write the information related to Mark and date part into Zookeeper. This reduces stress on Zookeeper. However, this involves modifying the source code, which can be costly for the average business team to implement.

2.2 Distributed table Writes

If the data is written by writing will encounter a single point of distributed table disk problem, first to introduce distributed table, distributed table is actually a logical table, it itself is not real data storage, can be understood as a proxy list, such as user query distributed table, distributed query requests will be issued to each subdivision of the query on the surface, The query results of each local table are then collected and returned to the user. In the scenario where users write to distributed tables, users write a large batch of data into distributed tables. Then, distributed means that the large batch of data is divided into several mini batches of data according to certain rules and stored in different slices.

Here is a very easy to misunderstand the place, we also thought that the distributed table is only in accordance with certain rules to do a network forwarding, that the bandwidth of the ten gigabit network card is enough, there will not be a single point of performance bottleneck. But actually Clickhouse does this. Let’s look at an example where there are three shards shard1, shard2, and shard3, where the distributed table is built on the node of Shard2.

  • In the first step, we write 300 pieces of data to the distributed table. The distributed table will group the data according to the routing rules. Suppose shard1 is divided into 50 pieces, Shard2 into 150 pieces, and Shard3 into 100 pieces.

  • Second, since the distributed table is on the same machine as Shard2, shard2’s 150 entries are written directly to disk. Then shard1’s 50 and SharD3’s 100 are not forwarded directly to them, but are also written to the disk’s temporary directory first on the distributed table machine.

  • Third, the distributed table node shard2 will initiate a remote connection request to shard1 and Shard3 respectively, and send the data of the corresponding temporary directory to Shard1 and Shard3.

It can be seen that all the data of shard2, the node where the distributed table is located, will fall on the disk first. We know that the read and write speed of the disk is not fast enough, and it is easy to have a single point of disk performance bottleneck. For example, tens of billions of dollars of data can be written to a single QQ video every day. If you write a distributed table, it is easy to create disk bottlenecks on a single machine. In particular, Clickhouse uses merge Tree, which causes write magnifies during the merge process.

We made two optimization schemes:

  • The first is to RAID the disk to improve the IO of the disk;

  • The second is that before writing, the upstream division of data table operation, directly separate write to different fragments, disk pressure directly changed into the original N one, so very good to avoid the disk single point bottleneck.

2.3 Local Top is not global Top

Although our write is divided by sharding, a common problem in distributed system is introduced here, that is, local Top is not global Top. For example, the data of the same content X falls on different shards, and when calculating the global Top100 clicks, the distributed table will send the query request to each shard, calculate the local Top100 clicks, and then summarize the results.

For example, content x is not Top100 on shard 1 and shard 2, so the click data for content x on shard 1 and shard 2 will be lost when the data is summarized.

Second, it may cause data errors. The optimization we did was to add a layer of routing before writing. We routed all the data with the same content ID to the same fragment, which solved the problem. For SQL commands with group BY and limit, only the group BY statement is sent to the local table for execution, and then the full result of each local table is sent to the distributed table. Perform the global group by operation again on the distributed table, and finally perform the limit operation.

In this way, although the correctness of global TOP N can be guaranteed, part of the execution performance is sacrificed. If you want to restore to higher execution performance, you can select the execution method using the distributed_group_by_NO_MERGE parameter provided by Clickhouse. All records with the same content ID are then routed to the same shard so that limits can be performed on the local table.

3. High-performance storage and query

A key to Clickhouse’s high-performance queries is sparse indexing. Sparse index design is very careful, good design can speed up the query, but bad design will affect the efficiency of the query. Because most of our queries are related to time and content ID, such as the performance of a certain content in various groups in the past N minutes, I built a sparse index according to the date-minute granularity time and content ID. For a certain content query, the establishment of a sparse index can reduce the file scanning by 99%.

Clickhouse high-performance queries the second point, is that we now the amount of data is too big, too many dimensions, for QQ to watch video content, the storage of water a day to have tens of billions, some dimensions have hundreds of categories, if at all dimensions of pre-aggregated query will slow, and the index will take up a lot of storage space. Our optimization is to set up corresponding pre-aggregate and materialized views for different dimensions, and exchange space for time, which can shorten the query time.

For example, a summary merge tree is used to build a aggregation of content ID granularity to accumulate the materialized view of PV, which is equivalent to the calculation of group BY in advance. When the aggregation results really need to be queried, the materialized view is directly queried. The data has been aggregated and calculated. And the amount of data scanned is only one thousandth of the original stream.

Another problem with distributed table query is that when a single content ID is queried, the distributed table will send the query request to all shards and then return the query results for summary. In fact, because of routing, a content ID only exists on one shard, and the rest of the shards are actually running empty. For this kind of query, our optimization is the background in accordance with the same rules of the first route, and then query the target fragment, which reduces the n-1 load of n, can greatly shorten the query time. And since we are providing OLAP queries, the data meets the final consistency. Therefore, read and write separation of the primary and secondary copies can further improve performance. We also did a one-minute data cache in the background so that queries for the same criteria could be returned directly in the background.

4. Clickhouse expansion solution

We investigated some common solutions in the industry:

  • For example, HBase original data is stored in the HDFS. Capacity expansion is only for region Server expansion and does not involve migration of original data.

  • However, each shard in Clickhouse is local and is more like RocksDB’s underlying storage engine, which cannot be easily expanded like HBase.

  • Then there’s Redis, which is a Hash slot, which is similar to a consistent Hash, which is a classic distributed cache scheme.

Although Redis slot may have temporary ASK unavailability during the Hash process, it is generally convenient to migrate. We migrated from h0 to H1 and then deleted H0, but Clickhouse is mostly OLAP batch queries, and since column storage does not support deletion, a consistent hash solution is not a good fit.

Our current capacity expansion plan is to consume an additional copy of data from the real-time store and write it to the new Clickhouse cluster. The two clusters run together for a period of time, because the live data is now saved for three days, and after three days, the backend service directly accesses the new Clickhouse cluster.

5. Summary of application results of real-time system

We have exported Tencent Watch’s real-time data warehouse, two message queues of DWM layer and DWS layer, and launched Tencent Watch’s real-time data analysis system, which can respond to multi-dimensional conditional query requests at sub-second level. In case of a cache miss:

  • 99% of content queries in the last 30 minutes took less than a second;
  • 90% of the content query requests in the past 24 hours took less than 5 seconds, and 99% took less than 10 seconds.

For more technical problems related to Flink, you can scan the code to join the community nail nail exchange group for the first time to obtain the latest technical articles and community dynamics, please pay attention to the public number ~