An overview of the

Connect is an important part of the MQTT protocol because it is the basis for clients to communicate with MQTT brokers and provides many useful features that can be used in many scenarios.

Or theory is combined with practice to tell it, or worry about small partners to see sleep. ~ ~ ~

As mentioned earlier, MQTT is a publishing-subscription-based messaging protocol, so an MQTT publishing client can publish messages to one or more subscribing clients. The model is much like a television or radio broadcast, where the station distributes the program and millions of people in the house receive the program. All messages are sent to the MQTT Broker, which forwards the messages to its subscribers. In this process, the following points need to be noted:

  • All clients have a unique Id, which is just a tag, not a client address. When a publisher publishes a message, it can only send the message to a topic, but not to a specific address or Id.
  • The client Id must be unique. If there is A Client A connected to the MQTT Broker and then another Client B connected to the MQTT Broker, the MQTT Broker will disconnect Client A. Since the MQTT Client has automatic connection, Client A disconnects, tries to reconnect to the MQTT Broker, Client B disconnects, then Client B reconnects, and the two clients enter A disconnection-connection-disconnect loop.
  • The MQTT Broker is responsible for receiving messages, filtering them, and forwarding them to clients subscribed to the topic.
  • Publisher and Subscriber are not directly related. Both connect only to MQTT brokers.
  • A client can either send or receive messages.
  • Typically, MQTT Brokers do not store messages.

You should now have a better understanding of the MQTT Client and Broker. Let’s discuss the format of MQTT Connect.

CONNECT

Fixed Header

The fixed header of MQTT contains the first byte and variable length. The first four bits (bit7 to bit4) indicate the packet type, and 1 indicates connect. Other marker bytes (bit3~bit0) are 0, as shown in the following table.

Bit 7 6 5 4 3 2 1 0
Byte 1 0 0 0 1 0 0 0 0
Byte 2… Remaining Length
Variable Header

In the Connect message, the variable header contains 10 bytes, as shown in the following table:

Byte Description bit7 bit6 bit5 bi4 bit3 bit2 bit1 bit0
Byte 1 Length MSB (0) 0 0 0 0 0 0 0 0
Byte 2 Length LSB (4) 0 0 0 0 0 1 0 0
Byte 3 ‘M’ 0 1 0 0 1 1 0 1
Byte 4 ‘Q’ 0 1 0 1 0 0 0 1
Byte 5 ‘T’ 0 1 0 1 0 1 0 0
Byte 6 ‘T’ 0 1 0 1 0 1 0 0
Byte 7 Level(4) 0 0 0 0 0 1 0 0
Byte 8 Connect Flag User Name Password Will Retain Will QoS Will QoS Will Flag Clean Session Reserved
Byte 9 Keep Alive MSB 0 0 0 0 0 0 0 0
Byte 10 Keep Alive MSB 0 1 1 0 0 0 0 0

Variable headers include Protocol Name, Protocol Level, Connect Flags, and Keep Alive time. The following are introduced respectively:

Protocol Name: bytes 1-6 are fixed. Byte 1 and byte 2 represent the length of the Protocol Name. The value is 0x04. Bytes 3-byte 6 represent the UTF-8 encoding of the protocol name “MQTT”.

Protocol Level: byte 7: indicates the Protocol Level. The Protocol Level of MQTT 3.1.1 is 4.

Connect Flag: byte 8, connection Flag, each bit represents a Flag, Bit0 is reserved Flag. The value ranges from Bit1 to Bit7, indicating Clean Session, Will Flag, etc. These markers determine whether the Payload contains the corresponding information. For example, if Bit7 and Bit6 are both 1, the Payload for this connection contains the User Name and Password. The purpose of each tag will be described later.

Keep Alive: byte 9 and byte 10, the heartbeat interval between the client and the server. The high byte is first and the low byte is last. The unit is S. The default value is 60S. Things to note about Keep Alive include:

  • When there is no message transmission between the client and the server, the client sends PINGREQ data packets to the MQTT Broker every 60 seconds (keep alive value). The server needs to reply to PINGRESP data packets.
  • If the client does not receive a PINGRESP packet after sending a PINGREQ packet for some time, the client disconnects.
  • If the Keep Alive value is set to greater than 0(say 60S), if the server does not receive PINGREQ packets after 1.5 times of Keep Alive time (90S), the server will disconnect from the current client in the case of no data interaction.
  • Keep Alive can be set to 0 so that the client does not send a PINGREQ packet and the server does not disconnect the client if it does not receive a PINGREQ.

Let’s test Keep Alive by observing the PINGREQ and PINGRESP packets. Enter the following command on the command line terminal to obtain the corresponding result:

$ mosquitto_sub -t topic001 -k 5 -d
Client mosqsub|16532-SCNWCL012 sending CONNECT
Client mosqsub|16532-SCNWCL012 received CONNACK (0)
Client mosqsub|16532-SCNWCL012 sending SUBSCRIBE (Mid: 1, Topic: topic001, QoS: 0)
Client mosqsub|16532-SCNWCL012 received SUBACK
Subscribed (mid: 1): 0
Client mosqsub|16532-SCNWCL012 sending PINGREQ
Client mosqsub|16532-SCNWCL012 received PINGRESP
Client mosqsub|16532-SCNWCL012 sending PINGREQ
Client mosqsub|16532-SCNWCL012 received PINGRESP
Client mosqsub|16532-SCNWCL012 sending PINGREQ
Client mosqsub|16532-SCNWCL012 received PINGRESP
Copy the code

In the preceding command, -t topic001 indicates that topIC001 is subscribed to. -k 5 indicates that the keep Alive interval is set to 5S. -d indicates that the Debug mode is enabled. You can see from the output that every 5S, the client sends a PINGREQ and receives a PINGRESP returned from the server.

Connect Flag

Since Connect Flag has a lot of content, it is introduced in a separate section.

Reserved

Byte 8 of Bit0. Reserved bit, must be 0.

Clean Session

The 8th bit of Bit1 indicates whether the Session needs to be cleared. If the value is 0, it indicates that the Session is retained. If the value is 1, it indicates that the Session is cleared.

Clients and brokers can store Session state information for reliable transmission of messages. We know that MQTT guarantees the reliable transmission of messages by defining QoS levels, so the most important part of Session state information is the status of QoS messages. Take the Broker as an example. If the Broker fails to deliver A message to Client A with A QoS of 1 or 2, the message status remains. When Client A reconnects next time, the server redelivers the failed message based on the Session status.

So, if the client publishes or subscribes to a topic and QoS > 0 is set, Clean Session must be set to 0.

Will Flag

Byte 8 of Bit2, which is used to indicate whether to send a will message.

A will message is a message that a client wishes to send to a specific topic notifying it that it is offline when the client is abnormally disconnected. One application scenario for bequest messages is device offline alerts, where subscribers (usually background servers) know that the device has gone offline when it has gone offline.

When the Will Flag is set to 1, the Will message needs to be set. The Broker stores the Will message and sends the Will message to the specified topic if the client is abnormally offline.

If the Will Flag is set to 1, there must be Will Topic(Will Topic) and Will Message(Will Message), and Will QoS and Will Retain are also read. Will message can also set QoS, so as to ensure the reliable delivery of will message.

When does a Broker send a Will Message? The Broker sends a wish message when any of the following occurs.

  • Network anomalies
  • The server did not receive a PINGREQ after the Keep Alive timed out
  • The client does not send DISCONNECT before closing the connection
  • The server closed the client connection because of a protocol error

Note that if the client closes the connection normally and sends DISCONNECT before closing the connection, the will message will not be sent. When the client sends a DISCONNECT request, the message is deleted by the Broker.

Will QoS

Byte 8 Bit3 and Bit4 are used to set the QoS of the will message. Since QoS has three values, 0,1 and 2, it is represented by two digits, with the highest value first and the lowest value second.

Will Retain

Byte 8 of Bit5, which is used to indicate whether to keep the will message.

The Will Retain flag is also useful because if set to 1, all subsequent new subscribers Will receive a Will message when they subscribe to Will Topic after the client is disconnected. QoS > 0 Can only be received by the newspaper’s previously subscribed subscribers, and Will Retain ensures that new subscribers also receive messages.

User Name Flag

It is used to mark whether the Payload contains User Name information. If the payload is set to 1, the payload must contain the User Name.

Password Flag

It is used to mark whether the Payload contains Password information. If the payload is set to 1, the payload must contain Password information. Note that if the User Name Flag is set to 0, the Password Flag must be set to 0. However, the value can contain only the User Name but not the Password. Therefore, when the User Name Flag is set to 1, the Password Flag can also be set to 0.

Keep Alive

Bytes 9 and 10 in the mutable header are used to indicate the Keep Alive time. This is the heartbeat time of MQTT in seconds, the default is 60S. When there is no data interaction between Client and Broker, the Client needs to send PINGREQ to the Broker, which replies with PINGRESP to check if the Client is online. A few things to note about Keep Alive:

  • The client is responsible for sending a heartbeat PINGREQ, and the server just replies with a PINGRESP when it receives a heartbeat.
  • If the client does not receive a reply after sending PINGREQ for some time, the client will close the connection.
  • The server will also close the connection if it does not receive any data from the client, including PINGREQ, after 1.5 hours of Keep Alive.
  • Keep Alive can be set to 0, indicating that the heartbeat mechanism is not enabled. In this case, neither the client nor the server will close the connection because no heartbeat or reply is received.
The sample

Let’s look at an example of the variable header of a CONNECT message to deepen our understanding.

describe 7 6 5 4 3 2 1 0
Association, discussion class type
1 byte Type Length High (0) 0 0 0 0 0 0 0 0
2 bytes Type Length low (4) 0 0 0 0 0 1 0 0
The byte 3 ‘M’ 0 1 0 0 1 1 0 1
4 bytes ‘Q’ 0 1 0 1 0 0 0 1
Byte 5 ‘T’ 0 1 0 1 0 1 0 0
Byte 6 ‘T’ 0 1 0 1 0 1 0 0
Association, discussion Etc. level
Word 7 Level (4)
even Pick up mark remember
8 bytes See note 1 1 0 0 1 1 0 0
Keep Alive
9 bytes Keep Alive high 0 0 0 0 0 0 0 0
10 bytes Keep Alive the low 0 0 0 0 1 0 1 0

The above bytes 8 represent User Name Flag (1), Password Flag (1), Will Retain (0), Will QoS (01), Will Flag (1), Clean Session (1), Reserved (0)

Payload

Having introduced mutable headers, let’s look at the body of the message.

The CONNECT protocol contains the Message body, which is the Client Id/[Will Topic]/[Will Message]/[User Name]/[Password].

The sequence of the preceding types cannot be changed. Except for the Client Id, all the other types are optional. The Payload contains related information only when the value of the Connect Flag is 1. For example, Will Topic and Will Message appear in Payload only when the Will Flag is 1 in the CONNECT Flag part of the variable header.

For the content of any Payload, the first two bits are the length, followed by the data. We will verify this later with Wire Shark capture.

Client Id

The Client Id is unique to a Client. This Id cannot be repeated. If two clients use the same Id, they will kick each other. Note the following about the Client Id:

  • The Client Id must be unique, and it should be meaningful and readable. For example, < Client type >_< device number > is used as the Client Id. As mentioned earlier, the Client Id is useful for troubleshooting, although you cannot send messages directly to the Client Id. For example, the background management tool of EMQX can directly determine whether a Client is online based on the Client Id and track all messages sent and received by a certain Client Id. This is great for troubleshooting problems online.
  • The Client Id can be empty. If the Client Id is empty, the server automatically assigns an Id and uses the assigned Id until the network is disconnected. However, this is not recommended for production environments.
  • Client ids are associated with sessions, so if you use QoS > 0 in your project, you cannot use random Client ids.
Will Topic & Will Message

When the Will Flag is set to 1, the Payload contains Will Topic and Will Message. Will Topic and Will Message have already been discussed and Will not be repeated here.

User Name & Password

If these two Connection flags are set to 1, the Payload contains the User Name and Password. The User Name and Password can be used to authenticate and authorize clients. Authentication can determine whether a client is allowed to connect, and authorization can restrict client access to certain resources (such as topic). This is the client-managed part of the Broker, which we’ll cover later.

CONNACK

After the client sends a CONNECT packet, the server replies a CONNACK packet. If the client sends CONNECT to the Broker and does not receive CONNACK for a period of time, the client needs to close the current connection.

Fixed Header

CONNACK fixed head as follows:

Bit 7 6 5 4 3 2 1 0
Byte 1 0 0 1 0 0 0 0 0
Byte 2 0 0 0 0 0 0 1 0

Again, recall the format of the fixed header, where the first byte is 4 bits high for the protocol type, 2 for CONNACK, and 4 bits low for some protocol flags. For CONNACK, this is a reserved field.

The Remaining Length of the fixed header is Remaining Length, CONNACK’s protocol, which is 2 bytes in Length.

Variable Header

The variable header contains two bytes in the following format:

Bit 7 6 5 4 3 2 1 0
Byte 1 0 0 0 0 0 0 0 X
Byte 2 X X X X X X X X

The first byte bit7-bit1 is the reserved Bit, and Bit 0 is used to represent Session Present.

The second byte is Return Code. Let’s introduce them separately.

Session Present

Session Present indicates whether the server has Session status. As mentioned earlier, when Clean Session is set to 0, the server retains some state of the Session. When the Client reconnects, the server determines whether the Session Status exists based on the Client Id. If so, the value is 1; otherwise, it is 0. There are two things to note about Session Present:

  • When a client initiates a connection and sets Clean Session to 1, Session Present is always 0 regardless of whether the server has Session Status.

  • If the Return Code from the Broker is not 0, the Session Present must be 0.

Return Code

The second byte of the CONNACK variable header represents Return Code, indicating the Return status of the connection. 0 indicates success, and other numbers indicate exceptions. The Return Code list is as follows:

value describe
0X00 normal
0X01 The server does not support the current protocol version
0X02 The Client Id does not have connection permission
0X03 Connection denied. Server unavailable
0X04 Connection rejected. User name and password incorrect
0X05 Connection denied. No authorization
0X06-0XFF keep

The difference between 0X02 and 0X05 is that the authentication fails and the connection is rejected.

Payload

CONNACK has no Payload.

test

Said so much, we now capture the package test, verify the content we described above.

  1. To start Wire Shark and listen for local loopback network adapters, enter tcp.port==1883

  2. Open the terminal and enter the following command to send a message.

    mosquitto_pub -d -t topic1 --will-qos 2 --will-topic "will_topic" --will-payload "I'm offline!" -u "zengbiaobiao" -P "password" -m "Hello MQTT"
Copy the code

  3. Go back to Wire Shark and look at the CONNECT packet as shown below.

When we look at the CONNECT protocol, we can see the detailed data content,

The first field shows the fixed header and the message length, Msg Len: 85, followed by the protocol name length and the protocol name.

The second area shows CONNECT Flags, which corresponds to the parameters we set to send messages.

The third area shows the content of the Payload. Each item is preceded by two bytes.

conclusion

Today we introduced the CONNECT and CONNACK protocol types in MQTT. The CONNECT protocol type contains the protocol header, variable header, and message body. It focuses on CONNECT Flags and some of the important features in connection messages, such as Clean Session, Will Topic, Keep Alive, User/Password, etc. These features are useful.

In addition, CONNACK was introduced, and finally a packet capture test was carried out through an experiment to verify what we have said.

If you have

All posts are synced on Github, and you can also visit my personal blog