The author | JiaoHaiBo, the director of research and development of information

Jiangsu Xingnuo Information Is an Internet of Things technology company providing vehicle positioning and fleet management services based on intelligent vehicle positioning terminals. It has more than 10 years of service and operation experience in the industry, and rich experience in software and hardware development. Haiguanche is a fleet management SaaS tool based on big data artificial intelligence + cloud services developed by Sinuo Information. Good tube car is committed to build an intelligent pipe tools, based on the unique precision hardware platform + intelligent big data algorithm, provided to the enterprise management team and an integrated management and logistics enterprise customer service solutions, covering accurate mileage, vehicle positioning playback, energy management, mergers, billing, security monitoring, intelligent equipment and statistical analysis, such as the team operation entire process, Empower the fleet and logistics enterprises, improve the overall efficiency of the industry.

A few considerations for choosing TDengine

In December 2018, I had the honor to meet Mr. Jianhui Tao, the founder of TDengine, at the Global Architect Summit held at InfoQ. At that time, Mr. Jianhui Tao gave a speech on “Quickly Build a super high-performance Temporal Space Big Data Processing platform”. As a good management vehicle system in the early development stage, Many advanced concepts and design ideas in TDengine are very in line with the needs of an Internet of vehicles project. No matter from the perspective of business scenarios or data characteristics related to the Internet of vehicles, it was felt to be very compatible with our good management of vehicles project, and there is another very important reason. At the beginning of the new project development TDengine for system development costs and can do part of the lower operational costs, after all, for a start-up project, product verification and experiment is the main target, can quick, quick, late in the database and the future of the big data processing platform can have a more solid foundation must be the best choice.

After that, we started in-depth communication and cooperation with Taos data, and started the private deployment version test within one month. After the initial debugging and docking of the local deployment version, we started the opening and testing of the cloud service version in early March 2019.

Scenario fit, data value added

After some research on TDengine, we realize that TDengine is a big data processing software specially facing the analysis of time series structured data storage. For the data of the Internet of vehicles, most of them are reported by TBox positioning terminals. These data, including GPS coordinates, fuel consumption, mileage, acceleration, start and stop information, are fully structured sequential data streams that are ideal for writing to TDengine. The system construction ideas of Xingnuo good management car are as follows:

The SaaS service platform of Sinogood Management Vehicle conducts a series of real-time and offline data analysis based on the vehicle running status data reported by terminals. The data reported by each TBox is distributed to the back-end real-time analysis server through load balancing. After preliminary analysis, the data is directly written into the database through the interface provided by TDengine. TDengine we built a super table to store in the basis of GPS data, and for each affair a child table, the table gpsinfo tag is used to store the vehicle child table of static information, such as vehicle number, license plate number, belongs to the user, by grouping, these don’t often change information, child table is used to store the instantaneous state of vehicle information, Including driving and stopping information, such as speed, direction, latitude and longitude, alarm status code, signal strength and more than ten kinds.

As for the real-time analysis of data, in addition to the real-time analysis of reported data, we also used the message subscription service function of TDengine to notify the new records received and inserted into the database in the form of message queue for the alarm data of abnormal vehicle status, considering the stability of core parsing applications. Several alarm information parsing servers at the back end are specially responsible for subscribing and receiving notifications for further alarm analysis and processing. In view of the later historical data analysis, scheduled tasks are carried out regularly, and daily data statistical analysis is carried out on time or with the regularity of vehicle travel.

Experience with

TDengine uses standard SQL syntax, similar to MYSQL. It costs almost nothing to get started on the project. According to its characteristics, we build a separate table for each data collection point, maintain common data fields through the super table, and provide a variety of query functions such as aggregation selection, which makes it very convenient to integrate multi-dimensional data query. According to TDengine and our company’s business, mainly to achieve vehicle information collection and processing.

Set table gpsinfo (‘ gps_time ‘, ‘lngitude’); , and the corresponding label of the vehicle, the vehicle number VID and other information, and the area molecular table information

-- create table gpsinfo (gps_time TIMESTAMP, lngdouble, latitude DOUBLE, altitude INT) tags ( vid BINARY(9), pno_64 BINARY(40) ); Create table db_gps_info.vid(..) create table db_gps_info.vid(..) tags(...)Copy the code

2. Business use: Aggregate data to achieve corresponding business through super table query.

Select first(gps_time) from gpsinfowhere tbname in (vid) and gps_time >= "xxxx-xx-xx xx:xx:xx" and  gps_time <= "xxxx-xx-xx xx:xx:xx"
group by vid
Copy the code

Problems encountered and solutions

Out-of-order writes of reissued data

In actual scenarios, as TBox uses 4G network to send collected data, vehicles often enter areas with poor signal. Without signal, TBox cannot report data in real time and can only be cached in the local cache of the terminal. After the network signal is recovered, the TBox terminal will report the latest status first and then send the cached data. At the beginning, we found that many of the reissued historical data could not be directly inserted into TDengine, because the reissued data timestamp is usually not the largest timestamp in the table, and TDengine does not allow the insert of these records. By using import, the problem of writing reissued historical data is solved.

Vision of the future

The future in view of the data stored in the database we will try to use more TDengine real-time streaming computing characteristics, in view of the single car or under the same customer car real-time data flow aggregation, statistical computing, and calculate the derivative of the data as a new analysis results into TDengine, late for further operations, statistics and presentation.

About TDengine

TDengine is a high-performance, scalable, highly reliable and zero-management Internet of Things big data platform software with independent intellectual property rights of TAos Data, which can fully integrate database, cache, message queue, streaming computing and other functions. Because according to the characteristics of the Internet of things big data made all sorts of optimization, TDengine data insert, query performance is better than gm’s big data platform more than 10 times, has been greatly save the storage space, using SQL interface, can seamlessly integrate with third party software, greatly simplifies the system architecture of Internet of things platform, significantly reduce the complexity and cost of research and development and operations. TDengine can be widely used in Internet of Things, Internet of vehicles, industrial big data and other fields. On July 12, 2019, TDengine opened source and ranked number one on GitHub’s Global Trends list for several consecutive days.