In order to build a more complete big data processing ecosystem for the Internet of Things and support simple and efficient migration from other sequential databases to TDengine, we developed taosAdapter.

TDengine is a big data platform designed and optimized by Taos data for Internet of Things, Internet of vehicles, industrial Internet, IT operation and maintenance. In addition to the timing database function, which is more than 10 times faster than the core, it also provides caching, data subscription, streaming computing and other functions to minimize the complexity of research and development, operation and maintenance, and the core code including cluster function are all open source. Since TDengine was announced as open source in July 2019, there has been very positive feedback on GitHub, with more than 17,300 people contributing to Star and more than 4,100 people forking the code. More and more users are adopting TDengine.

TaosAdapter is a new stand-alone program that can be included in TDengine 2.3.0.0 and above. The core purpose of taosAdapter is to address users’ pain points and reduce migration costs.

In areas such as the Internet of Things and operations monitoring, some users are still using traditional solutions or older products for timing data processing, such as OpenTSDB. OpenTSDB is also an open source distributed timing database. It does not have its own storage engine, but is based entirely on HBase. Because of its early generation, many o&M monitoring projects choose this system.

Sf Express Technology, for example, uses OpenTSDB+HBase as a storage solution for full monitoring data on the big data monitoring platform. But as more and more data was connected to the platform, they encountered many pain points, such as high system dependency, high usage cost and unsatisfactory performance.

Specifically:

  • High dependency but low stability: The big data monitoring platform is the underlying infrastructure and relies on big data components such as Kafka, Spark, and HBase for data storage. The data processing link will be very long, and the longer the data link, the greater the challenge to ensure the reliability of the system. If there is a problem with the monitoring system itself, there is no way to find and locate problems with the business system based on it.
  • High cost of use: Monitoring data is written in large volumes, and they need to keep full monitoring data for more than six months in order to track historical issues. Data storage costs remain high.
  • Performance fails to meet requirements: As a storage solution for fully monitored data, OpenTSDB meets requirements in writing performance, but fails to meet requirements in large-span and high-frequency daily queries.

In order to solve these pain points, sf Express engineers thought it was necessary to upgrade the full monitoring data storage solution. They researched a number of timing database products and decided on TDengine. They then adapted the system based on TDengine. After the transformation, TDengine cluster can easily carry the full monitoring data write, and it runs stably at present.

The number of server physical servers is reduced from 21 to 3, and the required daily storage space is only 1/10 of that of OpenTSDB+HBase under the same conditions. The hardware cost is greatly reduced. In terms of query performance, in the case of using the predictive function, the query P99 is within 0.7 seconds, which has been able to meet most of the daily query requirements. In the case of large-span (6 months) non-predictive query, the first query takes about 10 seconds, and the subsequent similar query takes much less time (2-3s).

Ruixin Internet of Things platform also encountered similar problems. They previously used OpenTSDB to store timing data, which was functionally able to meet requirements. However, due to the complex architecture and heavy volume of OpenTSDB, the development, testing, installation, deployment, operation and maintenance management and other work has brought a lot of trouble. With the development of business scale, the problem becomes more and more serious. After the same research, they also chose TDengine. They need to retain historical data during the upgrade, so they need to migrate the historical data from OpenTSDB to TDengine. They also developed and tested a data migration tool specifically for this purpose.

User demand is the driving force of product evolution. The TDengine team began to wonder: Since so many users have this kind of migration requirement, could there be an official solution?

TaosAdapter is our answer. The taosAdapter provides the following functions:

  • RESTful interface
  • Compatible with InfluxDB v1 Write interface
  • Compatible with OpenTSDB JSON and Telnet format writing
  • Seamless connection to Telegraf, CollectD, StatsD, Icinga, TCollector

It is compatible with Telnet/JSON writing protocol of OpenTSDB. For o&M monitoring services, users can directly push data collected by CollectD and StatsD to TDengine through taosAdapter. After the data is properly written to TDengine, Grafana can be adapted to visually present the data written to TDengine. TDengine also provides connection plug-ins for Grafana. If historical data is to be migrated, Taos data has also developed a plug-in for DataX, a data synchronization tool, to help users automatically write data to TDengine. Users don’t need to change a single line of code, just a few configuration changes, and the migration is seamless.

Next, taosAdapter will continue to be refined to support migration from more platforms to TDengine.

🌟 Click the link to explore the taosAdapter!