Abstract: With the growth of Internet of Things data volume, “timing database” has become the “required answer” for enterprises.
According to research firm Forrester, the value of the Internet of Things is 30 times higher than that of the Internet, and China’s Internet of Things industry will grow into a huge market worth more than $5 trillion by 2020.
The integration of 5G, AI, blockchain and other next-generation information technologies with the Internet of Things is accelerating. In the vision of smart Connectivity, data collected by machines, devices and sensors in iot systems is analyzed and correlated with artificial intelligence technologies to serve users in more meaningful ways. However, as the volume of data in the Internet of Things grows, “sequential database” is becoming a “must answer” for enterprises.
High-performance sequential database is the data storage base of the Internet of Things
Timing database is a vertically optimized database for timing data, which is specially used to store and manage timing data. For example, xiang yu said that 200 rooms are checked in at 8:00 p.m. in a hotel, and the number 200 stored at 8:00 p.m. is time sequence data.
In the Internet of things and operational monitoring scenarios, such as server CPU and memory usage, the operation condition of the electric car data, or application service business indicators and so on, all kinds of data collected index up to thousands or even hundreds of millions of, or even a collection of index data is likely to more than 10 gb, all these data must be within the prescribed time written to the database. In addition, the index data is usually collected every few seconds. Such a massive amount of time sequence data must have large concurrent writing capability and high data compression efficiency.
In addition, sequential services usually need to retrieve data from the database and present it in a nearly real-time visual way for convenient analysis and decision-making, which also has strict requirements on the query performance of the database. In this scenario, the biggest problem of the traditional relational database is the lack of data compression and low query efficiency. The sequential database was designed for high throughput, low latency and high data compression rate from the very beginning to meet the demands for performance and storage cost in the Internet of Things and operation and maintenance monitoring scenarios. Because of these characteristics, sequential database is widely used in manufacturing, banking and finance, social media, energy, smart home and other industries.
More than 10 years of hardware and software technology experience, the future challenges
According to an IDC white paper, the global data volume will reach 175 zabytes by 2025, of which 30 percent will be time-series data. Sequential databases have only really grown in the last decade, and there are more than 30 of them on the DBEngines website alone.
Compared to relational databases, sequential databases are slightly simpler, with no complex transaction support and no updates and deletes for individual pieces of data, Xiang said. However, it is not easy to build a timing database well, just like building a car. To build a car well, it is far from enough to simply buy parts, assemble and test, but also need to consider quality, performance, comfort, functionality, safety, etc. A car is the crystallization of human wisdom and culture.
To build a timing database, we need to condense decades of hardware and software technology and experience in the database field, such as storage, security, distributed system, compilation, algorithm, data structure, architecture design and so on, and make the system safe, reliable, stable, efficient and versatile in multiple scenarios. “In the future, more and more enterprises hope to mine more valuable information by using sequential database, which is facing great technical challenges in massive time line management, data compression, read and write performance, etc.” “Said Xiang Yu.
Cloud original survival and computing separation architecture, Huawei cloud data innovation Lab practice
The sequential database, as the data storage base of the entire Internet of Things, is also an important part of the cloud vendor’s infrastructure. As a global cloud service provider, the rapid development of Huawei Cloud is behind the expansion of a large number of infrastructure. How to fully monitor all the infrastructure and cloud services is a technical problem that the operation and maintenance team has to solve. The existing open source timing database has been unable to meet the increasing demands of Huawei cloud monitoring data. The number of monitoring indicators has rapidly increased from millions to billions, and the number of data writes per second has rapidly increased from hundreds of millions to billions. Therefore, a self-developed timing database is urgently needed to support the monitoring system of the operation and maintenance team.
Since 2018, Huawei Cloud Data Innovation Lab, where Xiang Yu works, began to focus on temporal data storage and management in the future Internet of Things and operation and maintenance monitoring scenarios, and self-developed temporal database GaussDB (for Influx). Following internal validation, GaussDB (for Influx) went live for commercial use in 2020.
GaussDB(forInflux) uses a cloud native storage and computing separation architecture, supports minute-scale elastic node scaling, and gets things done without migrating data; Support 100 million time line, every day 100 billion data writing is not a problem; Support lossless data compression, using adaptive data compression algorithm, the data compression ratio increased to 1:20; Using MPP architecture, vectorization, pre-aggregation and other related technologies, compared with open source timing databases such as OpenTSDB and InfluxDB, the performance of common queries in timing databases, such as single timeline conditional query and multidimensional aggregation query, is greatly improved.
After switching from Cassandra to GaussDB(for Influx), huawei cloud reduced the total number of compute nodes by four times from 39 (18 hot clusters, 9 cold clusters, and 12 big data analysis clusters) to nine nodes, Xiang said. The storage space consumption is reduced from 1TB to less than 100GB per day, reducing the storage space consumption by 10 times.
Currently, huawei cloud timing database GaussDB(forInflux) has served more than 15 internal and external customers and has become an important part of Huawei cloud infrastructure.
There is no ready reference answer for the road of research and development.
Reflecting on the development process for GaussDB (for Influx), Xiangyu said that a system’s journey from birth to maturity is often accompanied by a long period of Bug fixes and continuous optimization of specific scenarios. Because no one can think of and test cover all application scenarios in advance, GaussDB(for Influx) is no exception.
During the initial development of GaussDB(forInflux), one of the first problems the Sangwoo team encountered was that process OOM (operating system protection mechanism triggered by memory exhaustion) quit. As we all know, OOM is only possible for two reasons, one is memory leak, and the other is the actual use of too much memory.
As we all know, the data in the database is stored in disk files, efficient data retrieval often requires the establishment of file index in memory, convenient and fast location of data in the file. In a sequential database, the longer the data is kept in the database, the larger the data files will be and the more files there will be. During the process of program restart, metadata of each data file needs to be read into the memory organization as index. The metadata here mainly includes how many timelines are stored in the current file, and the offset of data of each timeline in the file, etc. In the scenario of operation and maintenance monitoring, the number of timelines increases exponentially. The time line of the sequential database exceeds 100 million levels, and the virtual machine specifications remain unchanged. As a result, the metadata cannot be fully stored in the memory and then converted into indexes, the program cannot be restarted in OOM.
To further elaborates and temporal database is difficult here, because the vast majority of users or scenario problems won’t reach the timeline and the amount of data, in the face of computing resources are limited, and the data quantity is too big, in the industry, there is no effective method available, solve this problem, often need to combine technology and experience. For example, during the process of program restart, metadata is loaded. In order to avoid storing too much data in memory, you can choose the mode of flow limiting. Then, how to set the threshold of data volume for each processing depends on long-term system development experience.
“Having a problem is not scary. Having no problem is scary. When problems occur, our choice is positive ‘hard hard’, one appears to eliminate one!” “Said Xiang Yu. It is not hard to see that they have developed GaussDB(for Influx), an important part of Huawei’s cloud infrastructure, with the spirit of “ingenuity,” which has already served 15+ internal and external customers.
Click to follow, the first time to learn about Huawei cloud fresh technology ~