What the front end needs to know about RTMP

An overview of the

RTMP is short for Real Time Messaging Protocol. It is an application-level Protocol over time-based transport layer protocols (such as TCP) designed to packet multimedia transport streams (such as video, audio and interactive content).

introduce

RTMP protocol is an application layer protocol proposed by Adobe. The introduction part of RTMP official document is quoted above. Its purpose is to provide real-time two-way flow of information to both sides of communication such as video, audio and data messages. RTMP implements different weights for different types of messages so that the underlying flow can be queued when the amount of transmission is limited.

define

  • Payload: data contained in a packet

  • Packet: A Packet consists of a fixed header and payload data. Some underlying protocols may need to encapsulate the defined Packet

  • Port: indicates a TCP/IP Port. The host number identifies multiple addresses

  • Transport Address: A combination of a network address and a port

  • Message Stream: The logical channel through which messages flow

  • Message Stream ID: Each Message has an ID that is expected to match the Message flow

  • Chunk: a fragment of a message. Messages are divided into small parts before being transmitted by the network. All messages are transmitted through the peer end of the multi-routing time sequence

  • Chunk stream: A logical channel of communication that allows blocks to be sent in a specific direction, either from client to server or from server to client

  • Chunk stream ID: Each Chunk stream has an ID that matches it

  • Multiplexing: The process by which separate audio and video data can be played continuously, allowing simultaneous transmission of audio and video

  • DeMultiplexing: The reverse process of multiplexing that aggregates interlaced audio and video data into raw audio and video data

  • Remote Procedure Call: A request that allows a client or server to Call another server or client program or child process

  • Metadata: a description of data. The movie metadata includes the movie title, length, release date, and so on.

  • Application Instance: a server Instance that a client can connect to and send connection requests to

  • Action Message Format: compressed binary Format for serializing AS ActionScript object Graphs. Available in AMF0 and AMF3 versions

Byte order & Alignment & time format

All entire fields are transmitted in byte order and are big-endian, and are in decimal unless otherwise specified. The timestamp is in milliseconds, starts at 0, and is 32 bits long. The time interval is expressed as an unsigned integer in milliseconds and can be 24 or 32 bits long

RTMP block flow

Although RTMP is designed to transmit messages in real time, it can also be used to process any message. RTMP is ideal for a wide range of audio and video applications, and while it does not provide any optimization, it is optimized for higher-level protocols, such as a live video server that may choose to discard video messages in case of a slow client connection to ensure that audio messages arrive in real time.

The message format

The message format can be broken up into blocks to provide multiplexing, and these messages need to include the following fields:

The field name meaning The size of the
Time stamp (Timestamp) The timestamp of the message 4 bytes
Length (Length) The length of the message payload, which must be included if the header cannot be omitted 3 bytes
Type of ID (Type ID) Type ID is used for message control 1 byte
Message Stream ID Any numeric value, little endian storage 4 bytes

Shake hands

The RTMP handshake differs from other protocols in that it contains three fixed-size blocks instead of a variable-size block

Shake hands with the order

To start, the client sends C0 and C1

The client must receive S1 before it can send C2 and S2 before it can send any other data

The server must wait until it receives C0 before sending S0 and S1, preferably after receiving C1. The server cannot send S2 until it receives C1. The server cannot send other data until it receives C2

The format is c0&s0

C0 and S0 are an 8-bit group and are considered to be an 8-bit integer segment. For C0, this field is considered to be the version of the client RTMP request; For S0, this field is considered to be the RTMP version chosen by the server. The default is 3, 0-2 is deprecated, 4-31 is used for future releases, and 32-255 is not allowed

 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|   version     |
+-+-+-+-+-+-+-+-+
Copy the code

Format: C1 & S1

C1 and S1 are a 1536-bit length group containing the following fields:

The field name meaning The size of the
Time (Time) Contains a timestamp, which can be 0 or any numeric value, for time identification 4 bytes
0 (Zero) They all have to be 0 4 bytes
Random data Any value, no encryption required 1528 bytes 
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | time (4 bytes) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | zero (4 bytes) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | random bytes | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | random bytes | | (cont) | | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Copy the code

C2&s2 format

C2 and S2 are a 1536-bit length group, corresponding to the replies of S1 and C1, containing the following fields:

The field name meaning The size of the
Time (1) Time1 Timestamp, S1(C2 sent) or C1(S2 sent) 4 bytes
Time 2 (Time2) Timestamp, before S1 or C1 4 bytes
Random data Any value to quickly verify the connection bandwidth or delay 1528 bytes 
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | time (4 bytes) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | time2 (4 bytes) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | random bytes | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | random bytes | | (cont) | | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Copy the code

A diagram to shake hands

  • Uninitialized: The protocol version is sent in this phase. The client and server are Uninitialized. The client sends the protocol version in the C0 packet. If the server supports this version, it will send S0 and S1. If the server does not support it, the server will take appropriate actions. In RTMP, this action is to abort the connection.

  • Version Sent: Both the client and the server are in the Version Sent phase after the uninitialized phase. The client is waiting for S1 and the server is waiting for C1. After receiving the corresponding waiting packet, the client sends C2 and the server sends S2, and then enters the acknowledgement sending phase.

  • Ack Sent: wait S2 and C2 for client and server responses

  • Handshake Done: Client and server exchange messages

+---------------+                             +---------------+
|    Client     |        TCP/IP Network       |    Server     |   
+---------------+              |              +---------------+
        |                      |                      |
  Uninitialized                |                 Uninitialized
        |          C0          |                      |
        | -------------------> |          C0          |
        |                      | -------------------> |
        |          C1          |                      | 
        | -------------------> |          S0          |
        |                      | <------------------- |
        |                      |          S1          |
   Version sent                | <------------------- |
        |          S0          |                      |
        | <------------------- |                      |
        |          S1          |                      |
        | <------------------- |                 Version sent
        |                      |          C1          |
        |                      | -------------------> |
        |          C2          |                      |
        | -------------------> |          S2          |
        |                      | <------------------- |
     Ack sent                  |                    Ack sent
        |          S2          |                      |
        | <------------------- |                      |
        |                      |          C2          |
        |                      | -------------------> |
  Handshake Done               |                Handshake Done
        |                      |                      |
Copy the code

block

After the handshake, the connection can be multiplexed into block streams. Each block carries one type of information flow, and each stream has a unique ID, the block stream ID. When transmitted over the network, each block must be fully loaded and reassembled according to the ID at the receiver. Chunking allows fragmentation of small messages over higher-order protocols, such as blocking small high-priority messages (such as audio or control) by blocking large low-priority messages (such as video). Chunking also allows for less indirect cost of sending small messages, such as bulk containing compressed information that represents content, or vice versa. The size of the block is configurable. Large blocks reduce CPU usage, but may cause low bandwidth delays for other content due to more write operations. Small blocks are not suitable for high bit streams.

Block format

Each block contains the header and the data. The header contains three parts:

The field name meaning The size of the
Basic Header Block flow ID and block type 1 or 3 bytes
Message Header The main content of the message 0,3,7 or 11 bytes
Extended Timestamp A timestamp in a situation 0 or 4 bytes
Chunk Data Block of load Variable byte 
+---------------+----------------+--------------------+----------------+
|  Basic Header | Message Header | Extended Timestamp |   Chunk Data   |
+---------------+----------------+--------------------+----------------+
|                                                     |
| <------------------ Chunk Header -----------------> |
                              Chunk Format
Copy the code

1. Block basic head

There are three types:

The field name meaning The size of the
Block flow id id (cs) Numerical 2 ~ 63 six
Format (FMT) Format identifier two
Block flow ID-64 (cs ID-64) Minus 64 8 or 16 bits

Block foundation head 1:

 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|fmt|   cs id   |
+-+-+-+-+-+-+-+-+
Copy the code

Block foundation head 2:

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|fmt|     0     |   cs id - 64  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Copy the code

Block foundation head 3:

0 1 2 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |fmt| 1 | cs id - 64 | + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - +Copy the code

2. Block header

There are four types:

Type 0:11 bytes

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp | message length| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | message length (cont) |messagetype id| msg stream id |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            message stream id (cont)           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                 Chunk Message Header - Type 0
Copy the code

Type 1:7 bytes

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp delta | message length| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | message length (cont) |messagetype id| 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                 Chunk Message Header - Type 1
Copy the code

Type 2:3 bytes

 0                   1                   2      
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              timestamp delta                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
         Chunk Message Header - Type 2
Copy the code

Type 3: No head

Public header fields:

The field name meaning The size of the
Timestamp interval (timestamp delta) Type 1 or type 2 3 bytes
Message Length Type 0 or type 1 3 bytes
Message Type ID Type 0 or type 1 1 byte
Message Stream ID Type 0 4 bytes

3. Extend the timestamp

Used when the timestamp is greater than 16777215 (0xFFFFFF)

Protocol control message

The message type ids 1, 2, 3, 5, and 6 are protocol control messages, which need to be based on the block flow protocol

Setting block size

Use the prompt to set the maximum byte, maximum 128 bytes, server and client can change this value, and upgrade the size of the other end

Suspend the message

Used to prompt for a completion message and then discard part of the received message

identify

When receiving bytes of the same size as the window, the client or server must send an acknowledgement message to its counterpart

Window recognition size

When the acknowledgement is made, the client or server sends the corresponding window size to the corresponding side

Setting the End Bandwidth

Clients and servers send output restrictions to their counterparts

RTMP Message format

RTMP is available with a variety of transport layer protocols

RTMP message

Messages can contain audio, video, data, and other message formats

1. The message header

Contains message type, length, timestamp, message flow ID

2. Message payload

The actual message bearer content

User control message

The message type is 4

RTMP command message

Messages on the client and server interoperate

Message type

1. Command message

The value in AMF0 is 20. The value in AMF3 is 17

2. Data messages

The value in the AMF0 format is 18. The value in the AMF3 format is 15

3. Share object messages

The value in AMF0 is 19. The value in AMF3 is 16

4. Audio messages

A value of eight

5. Video message

A value of 9

6. Aggregate messages

Value is 22

7. User control message events

Type the command

1. Network connection commands

Connection calls create flow

2. Network flow commands

Play Play 2 Delete Stream receive Audio Receive video release Address pause

reference

  • RTMP official document
  • RTMP H5 live stream technical analysis
  • Take you through RTMP
  • Full advanced H5 live
  • Parsing a Web live stream
  • Learn MP4 well and make live broadcasting more powerful
  • A little advice on audio and video testing
  • No more fragmented learning, quickly master H5 live technology
  • WebRTC peer to peer live
  • Nginx-rtmp Push stream (video)
  • Web Player
  • Videojs
  • Flv.js
  • MediaElement.js
  • h5player.bytedance.com
  • Streaming media
  • Write a Web video player from scratch
  • Media Source Extensions API
  • WebRTC API
  • ArrayBuffer