The background,

As mobile Internet, Internet of things, big data, and other industries, the rapid development of data in the continued at the speed of exponential growth, for example, we use a mobile phone access the behavior of network data, submitted to the state data of all kinds of wearable equipment, factory equipment sensors to collect index data, the monitoring data of the traditional Internet company. In fact, the data that records the state changes of systems and devices in chronological order are all Time Series, which are widely used in the Internet, Internet of Things and IT infrastructure.

Thanks to the rapid development of software and hardware technology, the cost of handling such a large sequential data set in continue to reduce, more companies began to continue to collect, analyze data, used in exception handling, trend prediction, precise marketing, risk control, such as scene, hoping to use the potential value of the data and improve company profitability and competitiveness. Here are two examples:

1. The picture below shows the borrowing and return of a shared bike in a popular area in San Francisco. By analyzing historical vehicle return data in the area, bike companies can optimize vehicle replenishment during hot spots.

2. The following figure shows the historical inbound and outbound traffic of an Internet service. It can be seen from the figure that the incoming traffic (blue line) is burr in a certain period of time. The service provider can check whether the service is abnormal based on this period. Alarms can be generated based on traffic monitoring, enabling O&M personnel to handle online problems in a timely manner.

Second, traditional time series data solutions have a lot of problems

Traditional time series data solutions and problems are as follows:

1. MySQL relational database:

  • Low write throughput: The single write throughput is low, which is difficult to meet the write pressure of tens of millions of sequential data;
  • High storage cost: Poor compression of sequential data takes up a lot of machine resources;
  • High maintenance cost: single-machine system, need to manually divide the database and table in the upper layer, high maintenance cost;
  • Poor query performance: Applicable to transaction processing, poor performance of mass data aggregation analysis.

2. Batch processing systems such as Hadoop and Spark

  • High data delay: offline batch processing system, data from generation to analysis, time-consuming hours or even days;
  • Poor query performance: Cannot make good use of indexes, relies on batch tasks, and takes more than minutes to query.

3. HBase

  • Poor multidimensional analysis capability: HBase can meet the write pressure well, but the query performance for non-RowKey prefixes is poor.
  • Maintenance cost: HBase requires maintenance of both HBase and Hadoop systems, and HBase stability increases maintenance costs.

Three, write, storage, query multi-link optimization, timing database has obvious advantages

1. Time series data model and characteristics

Before introducing timing database, it is necessary to understand the model and characteristics of timing data.

1.1 Mathematical model of time series data

The previous section introduces the scenario of sequential data, and also explains the significance and traditional scheme of analyzing sequential data. So how should sequential data be stored? Data storage should consider its mathematical model and characteristics, and time series data is certainly no exception. Here, taking a time series data as an example, the mathematical model of time series data is introduced.

The following data records the incoming and outgoing traffic of each port on each machine in a cluster over a period of time, and records an observation value every half hour:

  • Metric: An associated set of data, similar to a table in a relational database;
  • Point: A sequential data point, similar to a row in a relational database;
  • Timestamp: timestamp, representing the time point when the sequential data was generated;
  • Tag: Dimension column, used to describe the attributes of the device/system, indicating which device/module is generated, generally does not change over time;
  • Field: indicator column, used to describe the device/system status changes and fluctuate smoothly over time.

1.2 Characteristics of time series data

  • Data mode: Time series data increases over time, dimension information of the same device/system remains unchanged, and indicators change smoothly, which can be seen from the data changes in the Network table above.
  • Write: Continuous high concurrent write, no update operation: The sequential database often faces the real-time data write of millions or even tens of millions of terminal devices (for example, the number of mobike vehicles in China in 2017 was tens of millions), but most of the data represent the status of the equipment, and will not be updated after writing.
  • Query: Statistical analysis is performed on indicators in different dimensions, and there are obvious hot and cold data. In general, only recent data is frequently queried.

2. Timing database

2.1 Timing Database

Timing database is a specialized database for managing timing data, and the process of writing, storing and querying is optimized according to the characteristics of timing data, so as to solve the problem of timing data processing:

  • Storage cost:

O Use dimension repetition, time increasing, index smooth change and other characteristics, reasonable selection of coding compression algorithm, improve the data compression ratio;

O Prereduce the accuracy to aggregate historical data and save storage space.

  • High concurrent write:

O Write data in batches to reduce network overhead;

O Data is first written to the memory and then periodically dumped to immutable files to improve the write speed.

  • Low query delay, high query concurrency:

O Optimize common query modes to reduce query latency through indexing and other technologies;

O Improve query concurrency through caching and routing.

2.2 Comparison of open source timing database

At present, the popular open source timing database products in the industry include InfluxDB, OpenTSDB, Prometheus, Graphite, etc., whose product characteristics are shown in the figure below:

As can be seen from the above table, the open source timing database has the following problems:

  • There is no free and easy-to-use distributed version (OpenTSDB supports distributed deployment, but relies on too many systems and has high maintenance costs).
  • The aggregation ability is generally weak, and most of the time series data need to be used for statistical analysis.
  • There is no free permission management;
  • There is no multi-dimensional comparative analysis tool for time series.

Iv. Technical architecture of Tencent Cloud CTSDB after trillions of writes per day

Tencent Cloud Time Series Database (CTSDB) is a distributed and high-performance temporal sequence Database, which has made a lot of optimizations for high concurrent write of temporal data, obvious hot and cold data, IoT user scenarios, etc. Meanwhile, it also supports log parsing and storage of various industries. In Tencent internal support of Tencent cloud and other daily trillions of write to swallow the scene, after harsh pressure polished. Its architecture is shown in the figure below:

1. Main features of CTSDB

  • High performance :(refer to the test section later for specific performance data)

O Support batch write, high concurrency query, and strong analysis aggregation capability;

O Linearly improve system performance through horizontal scaling;

O Sharding and routing are supported to accelerate query.

  • High reliability:

O Distributed system, supporting multiple copies;

O Rack-aware, automatically stagger rack allocation of primary and secondary copies.

  • Easy to use:

O Rich data types, REST interface, data write query using JSON format;

O Native distributed, elastic and scalable, automatic data balancing;

O Permission system: a permission system that supports user names, passwords, and machine whitelists.

  • Low cost:

O Support column storage, high compression ratio (about 0.1), reduce storage cost;

O Data pre-reduction precision: Reduces storage cost and improves query performance.

O The number of copies can be adjusted as required.

  • Compatible with open source ecology:

O Compatible with Kibana/Logstash/Beat components, convenient data collection and visual analysis;

O Support data synchronization from MySQL, Kafka and other open source ecosystems for easy migration.

2. Competitive character comparison test

Here, a popular InfluxDB is used for performance comparison test with CTSDB.

2.1 Write performance test

(1) Comparison of write performance between single-node CTSDB cluster and single-node InfluxDB

Abscissa: concurrency (number of writing threads), ordinate: QPS (unit: ten thousand times /s)

Conclusion: The maximum write performance of single CTSDB node is 19W, and the InfluxDB is 15W.

(2) Comparison of write performance between CTSDB single-node cluster and CTSDB two-node cluster

Abscissa: concurrency (number of writing threads), ordinate: QPS (unit: ten thousand times /s)

Conclusion: The maximum write performance of CTSDB is 20W in single-node cluster and 34W in two-node cluster.

2.2 Querying performance Tests

(1) Comparison of query performance between the SINGLE-node CTSDB cluster and the single-node InfluxDB

Abscissa: number of concurrent queries (number of query threads), ordinate: QPS (times /s)

Conclusion:

  • The overall query performance of CTSDB is much better than that of InfluxDB. When the number of concurrent requests is high (40), the query performance of CTSDB is nearly 4 times higher than that of InfluxDB, reaching about 2W.
  • When the number of concurrent threads reaches 50, a link error occurs on the InfluxDB and query requests are rejected. In this case, CTSDB can be queried normally.

(2) CTSDB single-node cluster and two-node cluster query performance comparison

Abscissa: number of concurrent queries (number of query threads), ordinate: QPS (times /s)

Conclusion: When the number of concurrent requests is high, the query performance of two-node cluster is significantly improved compared with that of single-node cluster, showing a trend of linear expansion of query performance.

About us

1. Our current situation

As the only time series database of Tencent, CTSDB supports more than 20 core businesses of Tencent (wechat Lottery, Tenpay, cloud monitoring, cloud database, cloud load, etc.). Among them, the cloud monitoring system records the real-time status of various internal software and hardware systems of Tencent, and CTSDB bears all its data storage. It runs stably under the writing pressure of ten million series data points per second and the writing of 20TB+ data per day, which is sufficient to prove that CTSDB can stably support the massive data scene of the Internet of Things.

2. Our future

CTSDB has officially started public testing on Tencent Cloud to provide technical services for timing data processing. We will further optimize CTSDB in terms of reducing storage cost, improving ease of use and enriching functionality.

We welcome students who are interested in timing database and distributed storage to join us!