“This is the second day of my participation in the November Gwen Challenge. See details of the event: The last Gwen Challenge 2021”.

1. Principle of MQTT protocol

1. Implementation of MQTT protocol

The implementation of MQTT protocol requires the communication between the client and the server. In the communication process, MQTT protocol has three identities: Publish, Broker and Subscribe. The publisher and subscriber of the message are both clients, and the cancellation agent is the server. The publisher of the message can be both subscribers.

The messages transmitted by MQTT are divided into Topic and payload.

  • (1) Topic, which can be interpreted as the type of message. After a subscriber subscribes, he/she receives a message payload of the Topic.
  • (2) The payload is the specific content that subscribers use.

2. Network transmission and application messages

MQTT builds the underlying network transport: it establishes a client-to-server connection, providing an ordered, lossless, bidirectional byte stream based transport between the two.

As application data is sent over the MQTT network, the MQTT associates the associated quality of service (QoS) with the Topic name (Topic).

3. MQTT client

An application or device using the MQTT protocol that always establishes a network connection to the server. The client can:

  • (1) Publish information that other clients may subscribe to;
  • (2) Subscribe to messages published by other clients;
  • (3) Message of unsubscribing or deleting the application;
  • (4) Disconnect from the server.

4. MQTT server

An MQTT server, called a “message Broker,” can be an application or a device. It is located between the message publisher and the subscriber. It can:

  • (1) Accept network connections from customers;
  • (2) Accept application information released by customers;
  • (3) Handle subscription and unsubscribe requests from clients;
  • (4) Forward application messages to subscribing customers.

5. Publish/subscribe, topics, sessions

MQTT is based on Publish/Subscribe mode for communication and data exchange, which is fundamentally different from HTTP Request/Response mode.

A Subscriber subscribes to a Topic from the message Broker. After a successful subscription, the message server forwards messages for that topic to all subscribers.

Topics are separated by a ‘/’ to distinguish between different levels. Topics that contain the wildcard ‘+’ or ‘#’ are also called Topic Filters; Topic Names that do not contain wildcards are Topic Names.

1. chat/room/1 2. sensor/10/temperature 3. sensor/+/temperature 4. $SYS/broker/metrics/packets/received 5. $SYS/broker/metrics/# 1. '+': matching a/+, a/ X, A/Y 2. '#': matching a/#, A /x, A /b/c/d 3. Note: '+' matches one level, '#' matches multiple levels (must be at the end).Copy the code

Publisher can only publish messages to ‘topic names’, Subscriber can subscribe to’ topic filters’ to wildcard multiple topic names.

Session After each client establishes a connection with the server, it is a Session. There is stateful interaction between the client and the server. Sessions exist across a network and may span multiple consecutive network connections between clients and servers.

6. Methods in MQTT protocol

The MQTT protocol defines methods (also known as actions) that represent actions to be performed on identified resources. This resource can represent pre-stored data or dynamically generated data, depending on the server implementation. In general, resources refer to files or output on the server. The main methods are:

  • (1) CONNECT: The client connects to the server
  • (2) CONNACK: connection confirmation
  • (3) PUBLISH the message
  • (4) PUBACK: Release confirmation
  • (5) PUBREC: The published message has been received
  • (6) PUBREL: Release the message has been released
  • (7) PUBCOMP: Release completed
  • (8) SUBSCRIBE
  • (9) SUBACK: subscription confirmation
  • (10) UNSUBSCRIBE
  • (11) UNSUBACK: unsubscribe confirmation
  • PINGREQ: indicates that the client sends the heartbeat
  • (13) PINGRESP: indicates the heartbeat response of the server
  • (14) DISCONNECT: Disconnects
  • (15) AUTH: Certification

2. MQTT protocol packet structure

1. Structural composition

The structure of the MQTT protocol package is specified in the official documentation: mqtt.org/documentati…

In MQTT protocol, an MQTT packet consists of Fixed header, Variable header, and message body payload. MQTT packet structure is as follows:

  • (1) Fixed header. It exists in all MQTT packets and represents the packet type and group class identification of packets, such as connection, publication, subscription, heartbeat, etc. Fixed headers are required, and must be included in all types of MQTT protocols.

  • (2) Variable headers Exists in partial MQTT packets, and the packet type determines the presence and content of the variable header. Variable headers are not optional, but are present in some protocol types and not in others.

  • (3) The Payload. Exists in partial MQTT packets and represents specific content received by the client. As with mutable headers, there are message contents in some protocol types and none in others.

2. Fixed header

A fixed header is present in all MQTT packets and contains two parts, the first byte (byte 1) and the remaining message message length (1-4 bytes from the second byte), which is the number of bytes remaining in the current packet, including variable headers and data in the payload. The residual length does not contain the bytes used to encode the residual length.

The remaining length is encoded using a variable length structure that uses a single byte to represent values from 0 to 127. Values greater than 127 are treated as follows. The lower seven bits of each byte are used to encode data, and the highest bits are used to indicate whether there are any subsequent bytes. So each byte can encode 128 values, plus an identifier bit. The remaining length can be represented by up to four bytes.

1. Packet type

Position: 7-4 bits (bit [7-4]) in the first Byte (Byte 1), representing four unsigned values

The type of message packet can be determined by the high 4 bits of the first byte. The 4 bits can determine 16 types, among which 0000 and 1111 are reserved fields. The types of MQTT messages are as follows:

2. The sign bit

Position: 0-3 bits (bit [3-0]) in the first byte. The byte Bit[3-0] is used as the packet identifier.

The lower four bits of the first byte (bit3~bit0) are used to represent the control fields of some packet types. In fact, only a few packet types have control bits, as shown in the following figure:

  • (1) : Bit[3] is the DUP field. If the value is 1, it indicates that this packet is a repeated message. Otherwise, the packet is advertised for the first time.
  • (2) : Bit[2-1] is Qos field:
    • If Bit 1 and Bit 2 are both 0, QoS 0: at most one time.
    • If Bit 1 is 1, QoS 1: at least one time.
    • If Bit 2 is 1, QoS 2 is performed only once.
    • If both Bit 1 and Bit 2 are set to 1, the client or server considers this an invalid message and closes the current connection.

Currently Bit[3-0] is only available in the PUBLISH protocol and is indicated in the table as MQTT version 3.1.1. For other VERSIONS of the MQTT protocol, the content may be different. For all protocol types with the fixed header marked “reserved”, the Bit[3-0] must be consistent with that in the table. For example, the Bit 1 of the SUBSCRIBE protocol must be 1. If the recipient receives an invalid message, the current connection is forcibly closed.

3. MQTT message QoS

MQTT publish message Quality of Service (QoS) is not end-to-end, it is client-to-server. The QoS level at which a subscriber receives an MQTT message ultimately depends on the QoS of the published message and the QoS to which the topic subscribes.

1. Qos0 message publishing and subscription

2. Qos1 message publishing and subscription

3. Qos2 message publishing and subscription

Bit[0] is the RETAIN field, which publishes the retention flag, indicating that the server will RETAIN this push information. If a new subscriber appears, the server will push this message to it. If there is no new subscriber, the server will push this message to the current subscriber and release it.

3. Variable headers

Mutable headers mean changeable message headers. Some packet types contain variable headers and some do not. The variable header is between the fixed header and the dissipated content, and the content varies according to the message type.

1. The agreement

The protocol name is a utF-8 encoded string representing the protocol name MQTT. Subsequent versions of the MQTT specification do not change the offset and length of this string.

A server that supports multiple protocols uses the protocol name field to determine whether the data is an MQTT packet. The protocol name must be utF-8 character string “MQTT”. If a server does not want to accept CONNECT but wishes to identify itself as an MQTT server, it can send a CONNACK message with reason code 0x84 (unsupported protocol version) and must then close the network connection

2. Protocol version

Bit unsigned value indicates the version level of the client. The protocol level of version 3.1.1 is 4, and the protocol version field of MQTT V5.0 is 5 (0x05)

3. MQTT Session (Clean Session)

When an MQTT client makes a CONNECT request to the server, the Session can be set up with the ‘Clean Session’ flag.

  • If Clean Session is set to 0, a persistent Session is created. When the client is disconnected, the Session is kept and the offline message is saved until the Session times out and is logged out.
  • If Clean Session is set to 1, a new temporary Session is created. The Session is automatically destroyed when the client is disconnected.
1.Will Flag/Will Qos/Will Retain

If the Will Flag is set to 1, it means that if the connection request is accepted, the server must store a Will Message and associate it with the network connection. After that, the Will Message must be published when the network connection is disconnected, unless the server receives the DISCONNECT packet and deletes the Will Message

Will Message Will be published under certain circumstances, including but not limited to:

  • The server detects an I/O error or a network failure.
  • Description The client failed to communicate with the client during the Keep Alive period.
  • The client closes the network connection without sending the DISCONNECT packet.
  • The server closed the network connection due to a protocol error. Procedure

If Will Flag is set to 1, the Will QoS and Will Retain fields in the connection identifier Will be used by the server

Will QoS The two bits indicate the QoS level used when the Will Message is published

Will Retain This bit indicates whether the Will Message needs to be retained after publication.

If Will Flag is set to 0, then Will Retain must be 0

If Will Flag is set to 1:

  • If Will Retain is set to 0, the server must publish the Will Message without saving it
  • If Will Retain is set to 1, the server must publish the Will Message and save it
2. User Name Flag
  • If the User Name Flag is set to 0, the User Name does not have to appear in the payload
  • If the User Name Flag is set to 1, the User Name must appear in the payload
3. Password Flag
  • If Password Flag is set to 0, the Password does not have to appear in the payload
  • If Password Flag is set to 1, the Password must appear in the payload
  • If the User Name Flag is set to 0, the Password Flag must be set to 0
4. MQTT connection keeps heartbeat alive

What the heartbeat does:

The PINGREQ packet is sent from the client to the server. It can be used as follows: -1: Informs the server of the client's survival status when no other control packet is sent from the client to the server. - 2: requests a response from the server to confirm whether the server is alive. - 3: Indicates that the network connection is valid. The PINGRESP packet is sent from the server to the client in response to the PINGREQ packet. It represents that the server is alive.Copy the code

When the MQTT client initiates a CONNECT request to the server, the KeepAlive period is set with the KeepAlive parameter.

Keep Alive is an interval in seconds. Expressed in 2 bytes, it refers to the maximum interval between the time the client sends a control packet and the time it starts sending the next one. It is the client’s responsibility to ensure that the interval between the two control packets sent does not exceed the Keep Alive value. If no other control packets are available, the client must send a PINGREQ packet

The client can send PINGREQ packets at any time, regardless of the value of Keep Alive. PINGRESP is used to check whether the network connection with the server is normal.

If the value of Keep Alive is not 0 and the server does not receive the control packet from the client within one half of the Keep Alive period, the server must disconnect from the network as a network fault

If a client does not receive a PINGRESP packet within a reasonable amount of time after sending a PINGREQ, the client should close the network connection to the server.

If the value of Keep Alive is 0, the maintenance mechanism is turned off. This means that the server does not disconnect the silent client in this case.

5. MQTT Last Will

When an MQTT client sends a CONNECT request to the server, it can set whether to send a Will Message flag, and Topic and content Payload.

When the MQTT client goes offline abnormally (the client does not send a DISCONNECT message to the server before disconnecting), the MQTT messaging server will publish a bequest message.

A common variable header such as Packet Identifier

A message ID contains two bytes, with the high byte first and the low byte last. The types of protocols that contain Packet identifiers include:

PUBLISH(QoS > 0), PUBACK, PUBREC, PUBREL, PUBCOMP, SUBSCRIBE, SUBACK, UNSUBSCRIBE, UNSUBACK

The message ID starts from 1 and increments by default. If a message ID is used up, the message ID can be reused. For PUBLISH (QoS 1), if the sender receives a PUBACK, the message ID is used up. For PUBLISH(QoS 2), if the recipient receives PUBCOMP, the message ID is used up. For SUBSCRIBE and UNSUBSCRIBE, message ID usage completes the mark that the sender received the corresponding SUBACK and UNSUBACK.

In addition, the client and server message ids are assigned independently. The client and server can use the same message ID at the same time.

4. Payload

Some packet types contain Payload. Payload refers to the message carrier

For example, the published Payload refers to the message content (the content of the message published by the application). The Payload of the CONNECT contains information such as the Client Identifier, Will Topic, Will Message, Username, and Password.

The types of packets containing payload are as follows:

5. To summarize

MQTT message formats include Fixed Header, Variable Header, and Payload. Because the MQTT message format is very compact, data can be transferred efficiently.

Fixed Header contains the first byte. The high four bits indicate the packet type, and the low four bits indicate the type control. Currently only PUBLISH uses the type control field. Other control fields are reserved and must be consistent with the protocol definition.

The Fixed Header also contains Remaining Length, which is the Remaining message Length. The maximum Length is 4 bytes. Theoretically, an MQTT can transmit up to 256MB of data. Remaining Length=Variable Header+Payload Length.

Variable Header is a Variable Header. Some packet types need to contain Variable headers. Variable headers vary according to packet types. For example, Packet Identifier is used in publish, subscribe/unsubscribe packets.

Payload is the content of a message. It occurs only in certain packet types. The content and format of the message vary according to the packet type.