In this series of articles, we will share with you how to build a reliable, efficient and industry-specific Internet of vehicles platform based on EMQ’s practical experience in the field of Internet of vehicles, from theoretical knowledge such as protocol selection to practical operation such as platform architecture design.
preface
With the advancement of the new four modernizations (electrification, intelligence, interconnection and sharing) in the whole automobile travel field, various automobile manufacturers are gradually building intelligent driving and intelligent network connection as the core of the Internet of vehicles system. The new generation of Internet of vehicles system has put forward higher requirements for the underlying platform architecture of message collection, transmission and processing.
As mentioned in the previous article of this series “MQTT Protocol in The Internet of Vehicles”, MQTT protocol is the most suitable communication protocol for the data platform construction in the Internet of vehicles. Based on this, in this paper, we will continue to discuss MQTT message collection and delivery in the scenario of Internet of vehicles, and how to build a ten-million-level MQTT message platform for Internet of vehicles, in order to provide a reference for platform architecture design for enterprise users who are engaged in Internet of vehicles business.
The basis of the Internet of vehicles: message collection and delivery
Evolution of Internet of Vehicles transport protocol
As we all know, Vehicle-to-everything (V2X) refers to the dynamic mobile communication system that realizes the network communication between vehicles and the public through the interaction between vehicles and the cloud, vehicles and the Internet, vehicles and vehicles, vehicles and roads, vehicles and people, and vehicles and sensing devices. It is a heterogeneous communication network established to meet the efficiency, safety, management and other elements in every link related to the car. And the communication protocol running in it becomes the key and core of the Internet of vehicles system construction.
In the course of the development of the Internet of vehicles, there are mainly two mainstream communication technologies, which promote the overall development of the Internet of vehicles:
DSRC (DeDICated Short Range CommunICation) : It was developed by ASTM for ETC business scenarios in 1992, and evolved into IEEE(802.1x) Internet of Vehicles communication technology standard after years of improvement and iteration. For quite a long time, DSRC technology has been the mainstream communication protocol for the Internet of vehicles used in the international automotive mainstream production and consumption markets.
**C-V2X (Cellular Vehicle to Everything) : **C-V2X relies on the existing cellular base station. In addition to supporting the direct communication of PC5, rSUS and vehicles can be connected to the V2X platform through 4/5G channel (using Uu interface) to achieve vehicle-road cooperative communication. Compared to DSRC, C-V2X is technically superior, enhances communication security and confidentiality, supports high network capacity, and can support high bandwidth and large data volume requirements.
The competition between DSRC and C-V2X is fierce, with both aiming to become the dominant communication standard for connected vehicles. At present, China has the most perfect 5G communication network infrastructure, so it is more inclined to adopt C-V2X(LTE-V, 5G-V2X) communication technology, through the systematic construction of V2X road system + single-bike intelligent system, to achieve a new generation of vehicle network architecture based on autonomous driving.
The significance of message platform construction for the Internet of vehicles
Today, with the rapid development of the Internet of vehicles, all oems have formed a consensus: the purpose of the Internet of vehicles is not for the sake of the Internet, nor for the sake of on-board entertainment, but for the sake of data. With the Internet of cars, you have data. With data, complemented by a complete data governance and application architecture, you have it all.
And the target data of this business is not limited to the data related to the car end. In the V2X framework, it is necessary to solve the vehicle-to-vehicle (V2V), vehicle-to-road (V2R), vehicle-to-network (V2I), vehicle-to-cloud (V2C), vehicle-to-human (V2H) interconnection, and achieve comprehensive data collection and analysis for vehicles, roads, clouds, networks, and people. The C-V2X protocol and communication mode based on 5G provide basic capability guarantee for the construction of the whole system.
From traditional OTA applied to intelligent cockpit, highly precise maps adapter, cm-level positioning, car machine side long connection, mobile phone and news gathering, car, road cloud road together many new intelligent application scenarios, such as car networking business demand for message platform and data processing system is from the original car cloud extends human – – road – net – cloud the overall architecture of the building, Therefore, higher requirements are put forward for the construction of the whole message platform.
How to build a huge connection, high concurrency, throughput and low delay of message communication and transmission system architecture, to ensure that the whole system of generic in nature, convenience, high availability, reliability, security, and high concurrency, became the automatic car driving and road based collaborative scenarios key of the construction of a new generation of car networking system.
Architecture design of ten-million-level Vehicle Internet messaging platform
Next, we will take EMQ’s Internet of vehicles messaging platform and overall data processing solution as an example to introduce how to build a ten-million-level Internet of vehicles messaging platform.
Business challenges
Safe access of vehicle and machine, road test unit and mobile terminal system
The auto terminal needs to cover the new auto auto business such as auto-machine data reporting, POI delivery, file push, configuration delivery, message push, operation care, etc. The massive message topics generated need more secure and stable access and transmission to realize message subscription and release. RSU security access, message collection and transmission, map data transmission, etc.
Real-time and reliability of large concurrent messaging
Application scenarios such as high-precision map, centimeter-level positioning and vehicle-road cooperation need to guarantee the millisecond level low delay and highly reliable transmission capacity of massive vehicle-road map messages, and the message processing platform needs to be equipped with high performance, low delay and high reliability to support ten million connections and one million concurrent business scenarios.
Rich application scenario integration
In the Internet of vehicles system with autonomous driving as the core, it is necessary to use the message platform to connect various applications based on people, road, map and cloud. The vehicle end data is connected with application scenarios such as high-precision map, centimeter-level positioning, vehicle-road collaboration, and mobile phone end connection through the messaging platform. The consumption supply of applications is guaranteed through the messaging platform, and the data architecture with high performance, low latency and high reliability is provided.
Mass data storage, processing and distribution
From the people, vehicles, road, cloud, figure, net of massive Internet data is collected, the need for these large scale real-time data flow, storage, processing, distribution for the whole life cycle management, for the application to provide continuous data flow dynamic database support, support application depth using car networking data for the consumer, business decisions.
Total solution
In the scheme, we mainly adopt EMQX, the cloud native distributed Internet of Things access platform under EMQ, to realize the data connection, movement and processing of vehicle end, human end and road end in the Internet of vehicles system. EMQX’s integrated distributed MQTT messaging service and powerful IoT rules engine provide a foundation for highly reliable and high-performance real-time data movement, processing and integration of the Internet of Things, helping enterprises quickly build business-critical IoT platforms and applications.
Message processing for the car end
EMQX uses MQTT protocol to access the vehicle union system. The vehicle-machine end is connected with EMQX distributed cluster through load balancing, and the horizontal expansion capability of EMQX can realize the data communication capability of tens of millions of vehicle-machine connections and millions of concurrent responses. The rules engine provides one-stop capabilities such as massive message bridging message queue, persistent entry, offline message storage, and rich northbound integration of API atomic capabilities.
In terms of security, EMQX not only supports TLS/DTLS or GMSSL security protocols, but also ensures the reliability and stability of the system. It also provides multiple guarantee mechanisms such as heartbeat monitoring, testament message and QoS level, and realizes real-time, safe and reliable vehicle-machine message communication in complex network environment through offline message storage.
Message processing for people and terminals
EMQX provides a message collection and processing platform for terminals such as mobile apps and RSU. Based on the slicing capability of 5G network, the traffic information service with ultra-low delay can be realized through the nearby access of personal terminals and roadside units. Through MQTT and other protocols, the road condition information perceived by the human end and roadside facilities is pushed to the cloud control platform, and the cloud control platform integrates V2X algorithm to realize the intelligent traffic scenes such as road collaborative awareness, safety reminder and remote collaborative control.
In terms of security, it supports TLS/DTLS encryption of international standards or GMSSL encryption of national secret algorithm, and ensures secure communication between people, vehicles and road information systems through the extension of PKI/CA certificate authentication system.
Ten million message access framework model
For the new generation of Internet of vehicles, the overall message access and data processing platform reference architecture of EMQ with ten-million-level connection scale and one-million-level concurrency is as follows:
- Business scenario: Vehicles, mobile APP terminal, RSU and other devices in the Internet of vehicles system are connected through MQTT to achieve concurrent access capability to terminals of more than ten million magnitude.
- ** System architecture: ** Terminal devices are accessed through MQTT, HTTP and other protocols, and connected to the distributed message platform EMQ X through load balancing components. The distributed multi-cluster deployment can meet the requirements of tens of millions of concurrent connections. According to the million-level message throughput capacity, Kafka cluster can be connected to achieve data forwarding through the rule engine. The vehicle network service platform, high-precision map service, V2X cloud control service, location service and other vehicle network related applications can be directly consumed by subscription Kafka data. Meanwhile, EMQ provides three southbound interface services, REST, MQTT and MQ message queue, to realize bidirectional communication of vehicle control (remote control) messages.
Based on the above reference framework, EMQ can realize the business requirements of 10 million connections and 1 million concurrent processing in the scenario of Internet of vehicles through the EMQX cloud native distributed Internet of Things access platform.
Ten-million-level message access testing
Test environment and purpose
An automobile company plans to verify the following capabilities of EMQX cluster based on the test environment in the scenario of Internet of vehicles, so as to prepare corresponding technical architecture and capacity support for subsequent business growth:
- Supports 10 million concurrent connections, and supports 100,000-150,000 QoS 0 messages with payload of 100 bytes per second to receive Kafka through the rule engine bridge.
- 10 million concurrent connections subscribe and consume OTA broadcast topics;
- 3 million users connecting at the same time will not cause cluster avalanche and test the connection time.
In addition, after all of the above tests are completed, continue to explore the maximum throughput that can be supported for sending and bridging messages to Kafka simultaneously under the current configuration of 10 million concurrent messages (increasing client message sending frequency based on EMQX cluster resource usage), And test the maximum message throughput with a QoS of 2 and an average response time of 50 milliseconds under 10 million connections.
Test preparation
The client connects to the load balancer ELB through TLS encryption, and then terminates the client through TLS in HAProxy, and finally connects to the EMQX cluster through TCP. The capability of EMQX cluster can be improved by terminating TLS on HAProxy, and the processing capability of EMQX in this deployment mode is exactly the same as that of clients directly connected via MQTT TCP. On the other hand, compared with THE MQTT TCP connection, the client also needs to consume more resources through TLS connection. However, the test scale of this test is tens of millions and the number of testing machines required is large. In order to reduce the required testing resources and not affect the test target of EMQX cluster, TCP connection will be directly used in this test.
| service | The number of | version | OS | The instance specification | CPU | RAM | The network card | port |
|---|---|---|---|---|---|---|---|---|
| Load balancing cloud services | 1 | Network/Large II | 18083/1883/8081 | |||||
| EMQX node | 10 | V4.3.4 | Centos7.8 | C6.16 xlarge.2 Common block storage | 64C | 128G | 1 | 18083/1883/8081/8883 |
| Kafka cloud services | 4 | 2.3.0 | Centos7.8 | C6.4 Xlarge.2 Ultra High IO block storage | 16 | 32G | 1 | 9092 |
| XMeter pressure test control node | 2 | 3.0 | Centos7.8 | C6.4 xlarge. 2 | 16 | 32G | 1 | 443/80/3000/8086 The Intranet is disconnected |
| XMeter pressure test node | 43 | 3.0 | Centos7.8 | C6.4 xlarge. 2 | 16 | 32G | 5 | Put through Intranet |
Test scenarios
| The serial number | The scene name | describe | Expect the result |
|---|---|---|---|
| 1 | 10 million connection + message throughput | 10 million MQTT TCP concurrent connections, heartbeat interval 200s. Each user reports a QOS 0 message every 15 seconds. The payload is 100B. The message gets to Kafka through the rule engine bridge. First test for 1 hour, 24 hours after passing the stability test | The internal network test success rate is 100%, no cancellation backlog, CPU and memory performance is stable during the test, no large jitter. |
| 2 | News broadcast | 10 million MQTT TCP concurrent connections. All connections are subscribed to the same OTA broadcast topic (QoS 0, payload 100B). An MQTT client was simulated to broadcast a message to the topic every 10 minutes for 30 minutes | The Intranet test success rate was 100%, and all subscribing clients successfully consumed 3 messages |
| 3 | 3 million concurrent instantaneous connections | 3 million MQTT clients initiate connections at the same time and test how long it takes for all connections to complete | With 3 million clients successfully connected, the cluster will not collapse |
| 4 | Explore the highest message throughput under 10 million connections | Maximum throughput (Qos 0, payload 100B/1kB) for kafka with 10 million connections | 2 hours after the maximum message throughput rate is reached, the Intranet test success rate is 100%, no message backlog, CPU and memory performance is stable during the test, without large jitter |
| 5 | TPS at average response time | For 10 million connections, the message is QoS2, payload 100B, and the maximum message TPS supported by the average response time is within 50 mm | It can achieve a throughput of no less than 200,000 TPS |
The test results
The following is the result of the test:
| The serial number | scenario | Mean response time | CPU usage of EMQX node | EMQX CPU IDLE | EMQX Node Memory Usage (G) | Bandwidth required for LB (MB) |
|---|---|---|---|---|---|---|
| 1 | 10 million connections + 200,000 messages, QoS 0, payload 100B | 1.5 ms | On average, 47% 31% ~ 48% | On average, 47% 37% ~ 54% | 2:27.7 |
45 |
| 2 | Message broadcast under 10 million connections | 100ms | Up to 21% | A minimum of 69% | Used highest 32.3 Free lowest 87.9 | 200 |
| 3 | 3 million client instant connections | 3 minutes to complete the connection | Up to 25% | A minimum of 63% | Used highest 14.7 Free lowest 108.2 | 400 |
| 4 | Maximum throughput: 10 million connections + 1.2 million messages, QoS 0, payload 1kB | 164.3 ms | On average, 46% 23% ~ 64% | On average, 43% 20% ~ 64% | Used: 33 |
1350 |
| 5 | Payload 1 million payload 1 million payload 2 | 51.4 ms | On average, 41% 3% ~ 51% | On average, 43% 31% ~ 53% | 2:22.2 |
95 |
summary
As shown in the above results, the current deployment architecture can meet the verification requirements of the vehicle enterprise for 10 million concurrent connections + 200 thousand message bridge to Kafka, message broadcast, and 3 million instantaneous concurrent connections. In the exploration test, the maximum TPS(QoS 0 and payload 1kB) of 1.2 million messages was tested under 10 million connections. The test lasted for 10 hours. The EMQX cluster was stable, CPU idle was at least 20%, and memory usage was stable.
As can be seen from the above, EMQX has outstanding performance and stable and reliable architecture in supporting tens of millions of connections in the scenario of Internet of vehicles.
Introduction and use of stress testing tools
Due to the large number of testing machines required and complex management, the commercial version of EMQ testing software XMeter performance test platform and Jmeter-MQTT plug-in were used for this test.
XMeter is a performance testing platform based on the open source testing tool JMeter extension. In view of the characteristics of the Internet of Things, such as large access scale, flexible expansion requirements, multiple access protocols, and mixed scenarios, XMeter has modified JMeter to support large-scale and high-concurrency performance tests, such as realizing ten-million-level MQTT concurrent connection and message throughput tests. In addition to testing the MQTT protocol, it can also support testing of mainstream applications such as HTTP/HTTPS.
The Jmeter-MQTT plug-in is an open source MQTT performance testing plug-in implemented by XMeter. It is used in numerous projects and is currently the most popular MQTT plug-in in the JMeter community.
Write in the last
In this paper, we introduce the architecture design of MQTT messaging platform for ten-thousand-level Internet of vehicles based on cloud native distributed Internet of Things access platform EMQX, and verify the performance of the architecture in ten-thousand-level concurrent connection scenarios, providing a possible design reference for the construction of messaging data platform for the Internet of vehicles system.
As the world’s leading Internet of Things data infrastructure software provider, EMQ is committed to building high-performance, low latency, high availability and high reliability products, providing overall solutions for message collection, movement, processing and analysis for the new generation of Internet of vehicles system. It provides infrastructure service guarantee for auto manufacturers, T1 suppliers, aftermarket service providers, travel service companies and government regulatory agencies for autonomous driving and intelligent connected vehicle business, and realizes intelligent connection of people, vehicles, roads and clouds.
About the author
Rongjie Fan is director of Solution Architecture at EMQ. With rich experience in the field of IT infrastructure, He has been deeply engaged in the fields of enterprise digital transformation and Internet of Things architecture construction, data platform construction, agile development, lean operation and DevOps concept for many years, and is committed to introducing advanced cloud computing and Internet of Things products technology and services to the market.
Other articles in this series
- MQTT protocol in Internet of Vehicles scenario
Copyright: EMQ
Original link: www.emqx.com/zh/blog/mqt…