With the advent of 5G, the great idea of the Internet of everything is becoming a reality. The number of internet-connected Iot devices has reached 7 billion in 2018 [1], and in the next two years, smart water and electricity meters alone will exceed 1 billion [2].

Massive device access and device management pose challenges to network bandwidth, communication protocols, and platform service architecture. For iot protocol, several key problems of iot device communication must be targeted to solve: its network environment is complex and unreliable, its memory and flash memory capacity is small, its processor power is limited.

MQTT protocol is based on the publish/subscribe mode of the Internet of Things communication protocol, with simple and easy to implement, support for QoS, small message and other characteristics, occupy half of the Internet of things protocol:

Birth of the MQTT protocol

MQTT was created by Andy Stanford-Clark of IBM, and Arlen Nipper (then of Arcom Systems, later CTO of Eurotech).[3]

According to Arlen Nipper on an IBM Podcast, MQTT was originally called MQTT. Note the space between MQ and TT. MQ Telemetry Transport, a real-time data transfer protocol he developed while working on Conoco Phillips’ Pipeline SCADA system in the early 1990s. Its purpose is to allow the sensor to communicate with IBM’s MQ Integrator via the bandwidth-limited VSAT. As Nipper is a remote sensing and data acquisition and monitoring professional, so according to the industry convention to give the name OF MQ TT.

MQTT design principles

According to Nipper, MQTT must be simple and easy to implement, must support QoS(the device network environment is complex), must be lightweight and bandwidth efficient (because bandwidth is expensive at that time), must be data-independent (does not care about Payload data format), and must be continuously session-aware (always know if the device is online). Several of the core features of MQTT (version 3.1.1) are described below, which correspond to implementations of these design principles.

Flexible publish subscribe and theme design

The publis-subscribe pattern is a decoupling solution to the traditional Client/Server pattern. The publisher communicates with the consumer through a Broker, whose role is to correctly send the received message to the consumer through certain filtering rules. The benefits of the publish/subscribe model over the client/server model are:

  • Publishers and consumers do not need to know each other’s existence in advance, such as the need to communicate each other’s IP Address and Port in advance
  • Publishers and consumers don’t have to run at the same time. The Broker is always running.

In the MQTT protocol, the filtering rule mentioned above is Topic. For example, all messages published to the Topic News are forwarded by the Broker to subscribers who have subscribed to the news:

In the figure above, the subscriber presubscribing to the news, then the publisher publishes a message “some MSG “to the Broker and specifies to publish to the news Topic, and the Broker decides to forward the message to the subscriber by Topic matching.

MQTT topics are hierarchical and support wildcard characters + and #:

  • +Is to match a single wildcard. Such asnews/+Can matchnews/sports.news/+/basketballCan be matched to thenews/sports/basketball.
  • #Is a wildcard with one to multiple levels. Such asnews/#Can matchnews,news/sports,news/sports/basketballAs well asnews/sports/basketball/xAnd so on.

MQTT topics are not pre-created, but are automatically created by the Broker when a publisher sends a message to a topic, or when a subscriber subscrises to a topic.

Minimum bandwidth consumption

The MQTT protocol minimizes the extra consumption of the protocol itself, with a message header of as little as 2 bytes.

The MQTT message format is divided into three parts:

Fixed length header, 2 bytes, for all message types
Variable length headers, available only in certain message types
Payload, only for certain message types

The main types of MQTT messages are:

  • CONNECT / CONNACK
  • PUBLISH / PUBACK
  • SUBSCRIBE / SUBACK
  • UNSUBSCRIBE / UNSUBACK
  • PINGREQ / PINGRESP
  • DISCONNECT

The PINGREQ/PINGRESP and DISCONNECT packets do not require variable headers and do not have Payload. In other words, their packet sizes only consume 2 bytes.

In the variable length header of a CONNECT packet, there is a Protocol Version field. To save space, only one byte. So instead of storing the version number as a string “3.1.1”, use the number 4 to represent version 3.1.1.

Three optional QoS levels

In order to adapt to the different network environment of the device, MQTT designs three QoS levels, 0, 1, 2:

  • At most once (0)
  • At least once (1)
  • Exactly once (2)

QoS 0 is a “fire and forget” message sending mode: after the Sender (perhaps the Publisher or Broker) sends a message, it doesn’t care whether it was sent or not, and it doesn’t set up any resending mechanism.

QoS 1 includes a simple resend mechanism, where the Sender waits for an ACK from the receiver after sending a message, and resends the message if it does not receive an ACK. This pattern guarantees that messages will arrive at least once, but it does not guarantee that they will be repeated.

QoS 2 designs a slightly more complex mechanism for resend and discovery of repeated messages, ensuring that messages reach each other and strict values arrive once.

Keep the session

MQTT does not assume that the device or Broker uses THE TCP Keepalive mechanism [4]. Instead, the Keepalive mechanism is designed at the protocol layer: the Keepalive field can be set in the CONNECT message to set the interval for sending the PINGREQ/PINGRESP heartbeat packets. When a device’s PINGREQ is not received for a long time, the Broker considers the device offline.

In general, Keepalive has two functions:

  • The peer device is dead or the network is interrupted
  • To keep a connection from being disconnected by a network device when no message is exchanged for a long period of time

MQTT designs persistent connections for devices that want to rereceive messages they missed while offline: Set CleanSession to False in the CONNECT message, and the Broker will store it for the terminal:

  • All subscriptions for the device
  • QoS1 and QoS messages that have not been confirmed by the device
  • Missed messages when the device was offline

Status awareness

MQTT designs Last Will messages so that the Broker helps the device publish a Last Will message to a specified topic if it finds that the device is offline abnormally.

In fact, in some IMPLEMENTATIONS of MQTT servers (such as EMQ X), the Broker issues device status updates through some system topics when the device is online or offline, which is more realistic.

How to select an open source MQTT server

There are several popular open source MQTT servers so far:

  1. Eclipse Mosquitto

    MQTT server implemented using C language. The Eclipse organization also includes a large number of MQTT client projects: www.eclipse.org/paho/#

  2. EMQ X

    MQTT server developed in Erlang language, built-in powerful rules engine, support many other IoT protocols such as MQTT-SN, CoAP, LwM2M, etc.

  3. Mosca

    MQTT server developed with Node.js, easy to use.

  4. VerneMQ

    MQTT server also developed using Erlang.

In terms of MQTT 5.0 support, stability, scalability, and clustering capability, EMQ X should perform the best:

  • Developed using Erlang OTP, good fault tolerance (a proven language in the telecommunications field, used to make 99.9999999% available switch equipment [5])
  • Officially, there are a number of extensions available. There are many authentication and Backend plug-ins to choose from. Supports various relational databases, NoSQL databases, and common message queues such as Kafka, RabbitMQ, Pulsar, etc
  • Supports clusters and horizontal node expansion
  • A single node supports 2000K concurrent connections
  • Support for rule engine and codec

Quick experience with MQTT protocol

MQTT online server

The EMQ X MQTT Internet of Things Cloud service provides an online public MQTT 5.0 server that allows you to quickly start learning, testing, or prototyping the MQTT protocol without any installation.

For detailed access information about the MQTT server, see the EMQ website page: Free ONLINE MQTT Server.

MQTT online client

EMQ also provides a browser-accessible MQTT online client tool that allows connections to the MQTT server via normal or encrypted WebSocket ports, as well as caching connections for next access.

Copyright notice: this article is the original EMQ, reprint please indicate the source.

Link: www.emqx.io/cn/blog/wha…

  1. The number of connected devices that are in use worldwide now exceeds 17 billion, with the number of IoT devices at 7 billion… Iot-analytics.com/state-of-th… ↩ ︎

  2. The estimated installed base of smart meters (electricity, Gas and water) is expected to surpass the 1 billion mark within the next 2 years. iot-analytics.com/smart-meter… ↩ ︎

  3. Github.com/mqtt/mqtt.g… ↩ ︎

  4. www.cnblogs.com/softidea/p/… ↩ ︎

  5. Pragprog.com/articles/er… ↩ ︎