“This is the 35th day of my participation in the First Challenge 2022. For details: First Challenge 2022”

Pion/RTCP packets

RTCP is a sister protocol to RTP. Pion/RTCP implements RFC3550 and RFC5506. RTCP uses out-of-band(ooB, out-of-band data, not the same channel as data transmission) and provides statistics and control information for some RTP sessions.

The MAIN function of RTCP is to provide qos feedback, such as bytes transmitted/number of packets/number of packets lost/delay/RTT delay, etc. App may use these data to control service parameters, such as traffic limiting or switching encoding format.

/* Decoding RTCP packets: pkt, err := rtcp.Unmarshal(rtcpData) // ... switch p := pkt.(type) { case *rtcp.CompoundPacket: ... case *rtcp.PictureLossIndication: ... default: ... } Encoding RTCP packets: pkt := &rtcp.PictureLossIndication{ SenderSSRC: senderSSRC, MediaSSRC: mediaSSRC } pliData, err := pkt.Marshal() // ... * /Copy the code

The following is an example of serialization and deserialization of RTCP packets.

Each RTCP packet is small, so multiple RTCP packets are always aggregated and transmitted during transmission.

Pion/RTCP Specifies the RTCP packet type supported

There are many types of RTCP packets, with the current pion/ RTCP implementation supporting RTCP types ranging from 200 to 206

  • 200 sr
  • 201 rr
  • 202 sdes
  • 203 bye
  • 204 app
  • 205 rtpfb
  • 206 psfb

RTPFB is transport-dependent feedback and PSFB is load-dependent feedback.

PSFB also contains the following specific message types:

  • The PLI, the decoder, tells the coder that it has lost an indeterminate amount of video-encoded data
    • The sender will then re-send a keyframe
  • The SLI, the decoder, tells the encoder that several macroblocks are missing
    • The sender will then re-send a keyframe
  • FIR is also a message requesting a key frame
  • REMB, congestion control algorithm, receiver predicted maximum bit rate

RTPFB also contains the following specific message types:

  • TLN, NACK, negative feedback, this feedback is to tell the sender that some RTP packets have been lost
  • RRR, fast synchronization request, mostly used in MCU
  • TCC, also known as TWCC, estimates bandwidth at the sender

rtcp.Header

RTCP packets have a common header,4 bytes, regardless of type

type Header struct {
  Padding bool
  Count uint8
  Type PacketType
  Length uint16
}
Copy the code

They correspond to several pieces of information in the RTCP header, where Count is 5bit and has different meanings depending on Type.

func (h Header) Marshal() ([]byte, error) { /* * 0 1 2 3 * 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * |V=2|P| RC | PT=SR=200 | length | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */ rawPacket := make([]byte, headerLength) rawPacket[0] |= rtpVersion << versionShift if h.Padding { rawPacket[0] |= 1 << paddingShift } if h.Count >  31 { return nil, errInvalidHeader } rawPacket[0] |= h.Count << countShift rawPacket[1] = uint8(h.Type) binary.BigEndian.PutUint16(rawPacket[2:], h.Length) return rawPacket, nil }Copy the code

The contents of the serialization are relatively simple.

func (h *Header) Unmarshal(rawPacket []byte) error { if len(rawPacket) < headerLength { return errPacketTooShort } version := rawPacket[0] >> versionShift & versionMask if version ! = rtpVersion { return errBadVersion } h.Padding = (rawPacket[0] >> paddingShift & paddingMask) > 0 h.Count = rawPacket[0] >> countShift & countMask h.Type = PacketType(rawPacket[1]) h.Length = binary.BigEndian.Uint16(rawPacket[2:]) return nil }Copy the code

Deserialization is also very simple and violent.

rtcp.Packet

RTCP packets

type Packet interface {
  DestinationSSRC() []uint32

  Marshal() ([]byte, error)
  Unmarshal(rawPacket []byte) error
}
Copy the code

Serialization and deserialization are conversions between RTCP Packet objects and byte arrays []byte. The DestinationSSRC method returns the SSRC associated with the RTCP packet.

func Unmarshal(rawData []byte) ([]Packet, error) { var packets []Packet for len(rawData) ! = 0 { p, processed, err := unmarshal(rawData) if err ! = nil { return nil, err } packets = append(packets, p) rawData = rawData[processed:] } switch len(packets) { // Empty packet case 0: return nil, errInvalidHeader // Multiple Packets default: return packets, nil } }Copy the code

Unmarshal was used to resolve the RTCP. Packet one by one.

func unmarshal(rawData []byte) ( packet Packet, bytesprocessed int, Err = h.nmarshal (rawData) if err! {var h Header // Will resolve the first 4 bytes, which is the common Header of RTCP. = nil { return nil, 0, err } bytesprocessed = int(h.Length+1) * 4 if bytesprocessed > len(rawData) { return nil, 0, ErrPacketTooShort} inPacket := rawData[:bytes] // After receiving the data, construct the Packet according to the Packet type. packet = new(SenderReport) case TypeReceiverReport: packet = new(ReceiverReport) case TypeSourceDescription: packet = new(SourceDescription) case TypeGoodbye: Packet = new (Goodbye) / / you can see in RTPFB, is to distinguish the TLN/RRR by Count/TCC case TypeTransportSpecificFeedback: switch h.Count { case FormatTLN: packet = new(TransportLayerNack) case FormatRRR: packet = new(RapidResynchronizationRequest) case FormatTCC: packet = new(TransportLayerCC) default: Packet = new(RawPacket)} // In PSFB, Count is used to distinguish /PLI/SLI/REMB/FIR // REMB and TCC belong to the same congestion control algorithm, one in the load feedback case and the other in the transmission feedback case TypePayloadSpecificFeedback: switch h.Count { case FormatPLI: packet = new(PictureLossIndication) case FormatSLI: packet = new(SliceLossIndication) case FormatREMB: packet = new(ReceiverEstimatedMaximumBitrate) case FormatFIR: packet = new(FullIntraRequest) default: packet = new(RawPacket) } default: Err = packet.Unmarshal(inPacket) return packet, err = packet. bytesprocessed, err }Copy the code

In the unmarsharl function count, RTCP packets are processed one at a time.

Serialize multiple RTCP packets into an array of characters:

func Marshal(packets []Packet) ([]byte, error) { out := make([]byte, 0) for _, p := range packets { data, err := p.Marshal() if err ! = nil { return nil, err } out = append(out, data...) } return out, nil }Copy the code

You can see the serialization here, using the serialization of rTCP. Packet directly. In the deserialization process, a complete RTCP Packet data was used when the character array was used to deserialize the Packet, so a complete RTCP Packet data was also used in the deserialization process here.

To summarize, the pion/ RTCP Packet first exposes the Packet Packet and two packet-based functions: serialize multiple packets into character arrays; Invert the character array into multiple packets.

The rest is the implementation of each RTCP type to Packet interface.

RawPacket

Raw packets are unparsed RTCP packets. In this case, only the RTCP packet type or specific message type is not parsed, and the header information is still there.

type RawPacket []byte var _ Packet = (*RawPacket)(nil) func (r RawPacket) Marshal() ([]byte, error) { return r, nil } func (r *RawPacket) Unmarshal(b []byte) error { if len(b) < (headerLength) { return errPacketTooShort } *r = b var  h Header return h.Unmarshal(b) } func (r RawPacket) Header() Header { var h Header if err := h.Unmarshal(r); err ! = nil { return Header{} } return h } func (r *RawPacket) DestinationSSRC() []uint32 { return []uint32{} }Copy the code

For raw RTCP, the processing is pretty simple,

func (r RawPacket) String() string {
  out := fmt.Sprintf("RawPacket: %v", ([]byte)(r))
  return out
}
Copy the code

Finally, a printing support was added.

SenderReport

To clarify the difference between sr and RR:

  • Sr/RR is used to feedback reception quality
  • Sr contains 20 bytes more data than RR, and these 20 bytes contain information about the participant
  • When to issue sr, when to issue RR
    • If RTP data is sent after the last report is sent, sr is sent
    • If RTP data is sent within a report period, sr is sent
    • Other cases: If no RTP data is sent within a report period, rr is sent
  • Rr is sent to other participants who send RTP data: how many packets I received, how many I lost, when…
  • Sr is sent to other RTP participants: How much have I received,… , but also how much I sent, when…

The types of SenderReport are as follows:

type SenderReport struct {
  SSRC uint32
  NTPTime uint64
  RTPTime uint32
  PacketCount uint32
  OctetCount uint32
  Reports []ReceptionReport
  ProfileExtensions []byte
}
Copy the code

According to the structural description in Section 6.4.1 of RFC3550, the sequence is as follows:

  • 4-byte SSRC
  • An 8-byte NTP timestamp
  • A 4-byte RTP timestamp
  • 4 bytes Total number of RTP packets sent
  • 4 bytes Total length of the payload sent
  • Multiple report blocks Report blocks
  • Profile – specific extensions

In the header,Count represents the number of report blocks, and PT is 200.

func (r SenderReport) Marshal() ([]byte, error) { rawPacket := make([]byte, r.len()) packetBody := rawPacket[headerLength:] binary.BigEndian.PutUint32(packetBody[srSSRCOffset:], r.SSRC) binary.BigEndian.PutUint64(packetBody[srNTPOffset:], r.NTPTime) binary.BigEndian.PutUint32(packetBody[srRTPOffset:], r.RTPTime) binary.BigEndian.PutUint32(packetBody[srPacketCountOffset:], r.PacketCount) binary.BigEndian.PutUint32(packetBody[srOctetCountOffset:], R.octetcount) // The entire header of the sr is 24 bytes offset := srHeaderLength for _, rp := range R.reports {data, err := rp.marshal () if err! = nil { return nil, err } copy(packetBody[offset:], If len(r.read) > countMax {if len(r.read) > countMax {if len(r.read) > countMax { Return nil, errTooManyReports} copy(packetBody[offset:], r.profileExtensions) err := r.Header().Marshal() if err ! = nil { return nil, err } copy(rawPacket, hData) return rawPacket, nil }Copy the code

Serialization is a relatively simple way to form SenderReport data into a character array.

Func (r *SenderReport) Unmarshal(rawPacket []byte) error {// 4-byte RTCP public header + 24-byte SR header if len(rawPacket) < (headerLength +)  srHeaderLength) { return errPacketTooShort } var h Header if err := h.Unmarshal(rawPacket); err ! = nil { return err } if h.Type ! = TypeSenderReport { return errWrongType } packetBody := rawPacket[headerLength:] r.SSRC = binary.BigEndian.Uint32(packetBody[srSSRCOffset:]) r.NTPTime = binary.BigEndian.Uint64(packetBody[srNTPOffset:]) r.RTPTime = binary.BigEndian.Uint32(packetBody[srRTPOffset:]) r.PacketCount = binary.BigEndian.Uint32(packetBody[srPacketCountOffset:]) r.OctetCount = binary.BigEndian.Uint32(packetBody[srOctetCountOffset:]) offset := srReportOffset for i := 0; i < int(h.Count); i++ { rrEnd := offset + receptionReportLength if rrEnd > len(packetBody) { return errPacketTooShort } rrBody := packetBody[offset : offset+receptionReportLength] offset = rrEnd var rr ReceptionReport if err := rr.Unmarshal(rrBody); err ! = nil { return err } r.Reports = append(r.Reports, If offset < len(packetBody) {r.troFileExtensions = packetBody[offset:] if uint8(len(r.Reports)) ! = h.Count { return errInvalidHeader } return nil }Copy the code

As with serialization, report blocks are not parsed in deserialization.

func (r *SenderReport) DestinationSSRC() []uint32 {
  out := make([]uint32, len(r.Reports)+1)
  for i, v := range r.Reports {
    out[i] = v.SSRC
  }
  out[len(r.Reports)] = r.SSRC
  return out
}
Copy the code

The purpose of DestinationSSRC is to get the SSRC in the report block.

func (r SenderReport) String() string {
  out := fmt.Sprintf("SenderReport from %x\n", r.SSRC)
  out += fmt.Sprintf("\tNTPTime:\t%d\n", r.NTPTime)
  out += fmt.Sprintf("\tRTPTIme:\t%d\n", r.RTPTime)
  out += fmt.Sprintf("\tPacketCount:\t%d\n", r.PacketCount)
  out += fmt.Sprintf("\tOctetCount:\t%d\n", r.OctetCount)

  out += "\tSSRC    \tLost\tLastSequence\n"
  for _, i := range r.Reports {
    out += fmt.Sprintf("\t%x\t%d/%d\t%d\n",
      i.SSRC, i.FractionLost, i.TotalLost, i.LastSequenceNumber)
  }
  out += fmt.Sprintf("\tProfile Extension Data: %v\n", r.ProfileExtensions)
  return out
}
Copy the code

Printing is also supported.

ReceptionReport

Report block, which is used to tell the stream pusher the quality information of the local stream.

This report is based on the SSRC.

type ReceptionReport struct {
  SSRC uint32
  FractionLost uint8
  TotalLost uint32
  LastSequenceNumber uint32
  Jitter uint32
  LastSenderReport uint32
  Delay uint32
}
Copy the code

According to rfc3550 6.4.1, the length of this report block is 24 bytes:

  • SSRC 4 bytes, indicating which source the receive report is for
  • FractionLost, 1 byte, RTP packet loss rate since the last SR/RR packet was sent, expressed as a decimal
  • TotalLost, 3 bytes, the total number of RTP packets lost
  • LastSequenceNumber: 4 bytes. The lower 16 bits are the maximum RTP number received. The higher 16 bits are the number of times the RTP number is reset
  • Jitter, 4 bytes, a variance estimate of the time between the arrival of RTP packets, also known as Jitter
  • LastSenderPeport, 4 bytes, 32 bits in the NTP timestamp in the latest SR package received from the source, or 0 if no SR was received
  • Delay, 4 bytes, in 1/2 ^ 16 seconds, is the time between receiving an SR packet from the source and sending it, or 0 if the source does not send an SR packet

This report block contains a lot of information. How to use it depends on how the upper business is scheduled. Let’s first look at the method of the report block:

func (r ReceptionReport) Marshal() ([]byte, error) { rawPacket := make([]byte, receptionReportLength) binary.BigEndian.PutUint32(rawPacket, r.SSRC) rawPacket[fractionLostOffset] = r.FractionLost if r.TotalLost >= (1 << 25) { return nil, ErrInvalidTotalLost} // High bytes are saved in low address, big-ende mode tlBytes := rawPacket[totalLostOffset:] tlBytes[0] = byte(R.totalLost >> 16) tlBytes[1] = byte(r.TotalLost >> 8) tlBytes[2] = byte(r.TotalLost) binary.BigEndian.PutUint32(rawPacket[lastSeqOffset:],  r.LastSequenceNumber) binary.BigEndian.PutUint32(rawPacket[jitterOffset:], r.Jitter) binary.BigEndian.PutUint32(rawPacket[lastSROffset:], r.LastSenderReport) binary.BigEndian.PutUint32(rawPacket[delayOffset:], r.Delay) return rawPacket, nil } func (r *ReceptionReport) Unmarshal(rawPacket []byte) error { if len(rawPacket) < receptionReportLength { return errPacketTooShort } r.SSRC = binary.BigEndian.Uint32(rawPacket) r.FractionLost = rawPacket[fractionLostOffset] tlBytes := rawPacket[totalLostOffset:] r.TotalLost = uint32(tlBytes[2]) | uint32(tlBytes[1])<<8 | uint32(tlBytes[0])<<16 r.LastSequenceNumber = binary.BigEndian.Uint32(rawPacket[lastSeqOffset:]) r.Jitter = binary.BigEndian.Uint32(rawPacket[jitterOffset:]) r.LastSenderReport = binary.BigEndian.Uint32(rawPacket[lastSROffset:])  r.Delay = binary.BigEndian.Uint32(rawPacket[delayOffset:]) return nil }Copy the code

Because of the RFC standard, serialization and deserialization of report blocks are easy, but the challenge is to use the information at the application level.

ReceiverReport

As the receiver of a stream, the feedback information is sent to the sender. Rr is a short part of SR, and most of the data and structure are similar.

As an open source author, I never use the same move in two identical places, but various shows.

Since some of the code logic of SR and RR is similar, let’s just pick out some of the differences.

When serializing, we take into account the possibility that the final extension will be less than 4 bytes:

for (len(pe) & 0x3) ! = 0 { pe = append(pe, 0) }Copy the code

The population problem is not considered in deserialization because it is not considered in deserialization.

SourceDescription

RTCP packet of sDES type, source description packet.

Here are a few constraints on the RTCP packet composition:

  • Sr/RR packets should be sent as often as bandwidth permits, and must be reported for each transmission cycle
  • Each composite package should contain the SDES package and, more specifically, the Cname in the SDES package
    • The new receiver can identify the SSRC by the Cname and perform stream synchronization
  • The length of RTCP composite packets is limited by THE MTU, so pay attention to the number of composite packets

In addition to the common header, the rest of THE SDES-type RTCP packet is a chunk, which contains SSRC/CSRC information and specific SDES information, so it is also called three-layer structure. The Count in the public header indicates the number of chunks.

1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * header |V=2|P| SC | PT=SDES=202 | length | * +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ * chunk | SSRC/CSRC_1 | * 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SDES items | * | ... | * +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ * chunk | SSRC/CSRC_2 | * 2 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SDES items | * | ... | * + + + + + = = = = = + + + + + = = = = = + + + + + = = = = = + + + + + = = = = = + + + + + = = = = = + + + + + = = = = = + + = = + * /Copy the code

Each of the SSRC/CSRC takes up 4 bytes. Here are the sDES items:

Sdes Items describes a number of items:

  • Cname Specifies the specification name
  • Nmae user name
  • Email
  • Phone call
  • The Location address
  • Tool App or Tool name
  • Note prompt
  • Private Private information

They all have a common beginning:

1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | CNAME=1 | length | user and domain name ... * + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + * /Copy the code

Take a look at the processing:

type SourceDescription struct {
  Chunks []SourceDescriptionChunk
}
Copy the code

SourceDescription implements the Packet interface, using SourceDescriptionChunk for chunk serialization and deserialization.

type SourceDescriptionChunk struct {
  Source uint32
  Items  []SourceDescriptionItem
}
Copy the code

In the serialization and deserialization of the SourceDescriptionChunk, the SourceDescriptionItem is used:

type SourceDescriptionItem struct {
  Type SDESType
  Text string
}
Copy the code

During serialization, there is a special treatment for chunk length Settings, as specified in the RFC:

func (s SourceDescriptionChunk) len() int { len := sdesSourceLen for _, Len += sdesTypeLen it := range s.items {len += it.len()} it := range s.items {len += it.len() getPadding(len) return len }Copy the code

The following serialization process is analyzed from the beginning:

Func (s SourceDescription) Marshal() ([]byte, error) {// Calculate the total length of the RTCP packet. s.len()) packetBody := rawPacket[headerLength:] chunkOffset := 0 for _, c := range s.Chunks { data, err := c.Marshal() if err ! = nil { return nil, err } copy(packetBody[chunkOffset:], data) chunkOffset += len(data) } if len(s.Chunks) > countMax { return nil, errTooManyChunks } hData, err := s.Header().Marshal() if err ! = nil { return nil, err } copy(rawPacket, hData) return rawPacket, nil }Copy the code

Next look at the serialization of each chunk:

func (s SourceDescriptionChunk) Marshal() ([]byte, error) { rawPacket := make([]byte, sdesSourceLen) binary.BigEndian.PutUint32(rawPacket, s.Source) for _, it := range s.Items { data, err := it.Marshal() if err ! = nil { return nil, err } rawPacket = append(rawPacket, data...) Uint8 (SDESEnd)) rawPacket = Append (rawPacket, uint8(SDESEnd)) rawPacket = Append (rawPacket, make(]byte, getPadding(len(rawPacket)))...) return rawPacket, nil }Copy the code

Chunk is divided into SSRC/CSRC and ITEM, whose serialization is as follows:

func (s SourceDescriptionItem) Marshal() ([]byte, error) { if s.Type == SDESEnd { return nil, errSDESMissingType } rawPacket := make([]byte, SdesTypeLen +sdesOctetCountLen) // item Setting Type rawPacket[sdesTypeOffset] = Uint8 (s.type) txtBytes := [] Byte (s.ext) octetCount := len(txtBytes) if octetCount > sdesMaxOctetCount { return nil, ErrSDESTextTooLong} // item setting Length rawPacket[sdesOctetCountOffset] = Uint8 (octetCount) // item setting content rawPacket = append(rawPacket, txtBytes...) return rawPacket, nil }Copy the code

The deserialization of RTCP of type SDES is similar.

func (s *SourceDescription) DestinationSSRC() []uint32 {
  out := make([]uint32, len(s.Chunks))
  for i, v := range s.Chunks {
    out[i] = v.Source
  }
  return out
}
Copy the code

The DestinationSSRC of the sdes is the number of chunks retrieved.

Goodbye

An RTCP packet of type BYE, used to indicate that some source is no longer active.

1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * |V=2|P| SC | PT=BYE=203 | length | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * | SSRC/CSRC | * + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + * :... : * +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ * (opt) | length | reason for leaving ... * + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + * /Copy the code

Inactive sources can be multiple SSRC or CSRC, and optional reasons can be included in the BYE package. The processing of the final reason for leaving is consistent with the processing of chunk filling in SDES.

type Goodbye struct { Sources []uint32 Reason string } func (g Goodbye) Marshal() ([]byte, Error) {// Calculate the length of the bye packet rawPacket := make([]byte, g.len()) packetBody := rawPacket[headerLength:] if len(g.Sources) > countMax { return nil, ErrTooManySources} / / copy SSRC for I, s: = range g.S ources {binary. BigEndian. PutUint32 (packetBody [I * ssrcLength:], S)} // If g.eason! = "" { reason := []byte(g.Reason) if len(reason) > sdesMaxOctetCount { return nil, errReasonTooLong } reasonOffset := len(g.Sources) * ssrcLength packetBody[reasonOffset] = uint8(len(reason)) Copy (packetBody[reasonOffset+1:], reason)} // Set common header hData, err := g.ader ().marshal () if err! = nil { return nil, err } copy(rawPacket, hData) return rawPacket, nil }Copy the code

Deserialization is similar.

TransportLayerNack

RTPFB TLN, packet loss.

There is currently only one generic NACK for RTP transport layer nACK feedback. Its function is to notify that one or more RTP packets have been lost.

Once the underlying transport protocol provides a similar “feedback to the sender” mechanism, generic NACK should not be used.

The feedback packet in RTCP consists of three levels:

  • Transport layer feedback
  • Load layer feedback
  • The app layer feedback

The unified format of the feedback package is as follows:

// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // |V=2|P| FMT | PT | length | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | SSRC of packet sender | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | SSRC of media source | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | feedback control information(FCI) |Copy the code

In contrast to the generic RTCP header, the Count of the generic header is used to represent the number of SSRC or Chunk or report packets, and in the feedback packet,FMT is used to represent the fine classes under PT. For TLN, PT = 205, FMT = 15.

SSRC of media source nACK packet sender: sender of media source nACK packet sender: sender of media source NACK packet FCI,4 bytes, determines what information.

// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | PID | BLP | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Copy the code

This is the FCI of the generic NACK.

PID,2 bytes, is the serial number of a missing RTP packet. BLP, which is the mask bit of subsequent lost packets, is calculated based on PID.

As you can see, every time one RTP packet and the next 16 packets are returned, if the second packet is missing, the second is set to 1. Note: setting to 0 does not mean that the corresponding RTP has been received, only that bit X is not marked as missing in this feedback.

type PacketBitmap uint16

type NackPair struct {
  PacketID uint16
  LostPackets PacketBitmap
}

type TransportLayerNack struct {
  SenderSSRC uint32
  MediaSSRC uint32
  Nacks []NackPair
}
Copy the code

RFC also stipulates that the length of the feedback message is N +2, where N is the number of Nacks (a NACK is an FCI). In the code,n +2 cannot exceed 255. The specific basis has not been found, which corner of the RFC is estimated.

func (p TransportLayerNack) Marshal() ([]byte, error) { if len(p.Nacks)+tlnLength > math.MaxUint8 { return nil, errTooManyReports } rawPacket := make([]byte, nackOffset+(len(p.Nacks)*4)) binary.BigEndian.PutUint32(rawPacket, p.SenderSSRC) binary.BigEndian.PutUint32(rawPacket[4:], p.MediaSSRC) for i := 0; i < len(p.Nacks); i++ { binary.BigEndian.PutUint16(rawPacket[nackOffset+(4*i):], p.Nacks[i].PacketID) binary.BigEndian.PutUint16(rawPacket[nackOffset+(4*i)+2:], uint16(p.Nacks[i].LostPackets)) } h := p.Header() hData, err := h.Marshal() if err ! = nil { return nil, err } return append(hData, rawPacket...) , nil }Copy the code

TLN’s simple structure makes serialization easy.

There are several separate methods for NackPair:

func (n *NackPair) Range(f func(seqno uint16) bool) { more := f(n.PacketID) if ! more { return } b := n.LostPackets for i := uint16(0); b ! = 0; i++ { if (b & (1 << i)) ! = 0 { b &^= (1 << i) more = f(n.PacketID + i + 1) if ! more { return } } } }Copy the code

Looking at this method alone, I don’t have any clue, but look at it together with the following methods:

func (n *NackPair) PacketList() []uint16 {
  out := make([]uint16, 0, 17)
  n.Range(func(seqno uint16) bool {
    out = append(out, seqno)
    return true
  })
  return out
}
Copy the code

Use the closure to get a uint16. Combined with Range, the number of lost packets is obtained, and the maximum number is not more than 17.

func NackPairsFromSequenceNumbers( sequenceNumbers []uint16) (pairs []NackPair) { if len(sequenceNumbers) == 0 { return []NackPair{}} // NackPair := &NackPair{PacketID: sequenceNumbers[0]} for I := 1; i < len(sequenceNumbers); I ++ {m := sequenceNumbers[I] // Create a new NACK package if m-nackpair.PacketID > 16 {pairs = append(pairs, *nackPair) nackPair = &NackPair{PacketID: M} the continue} / / the current iteration of RTP, the serial number can be added to the nack nackPair. LostPackets | = 1 < < (m - nackPair PacketID - 1)} / / processing last nack, added to the nack queue pairs = append(pairs, *nackPair) return }Copy the code

As you can see, the function that generates the NACK queue is pretty neat. Simple is beautiful.

These are the low-level operations of NACK that the upper layer uses to compose more complex logic.

RapidResynchronizationRequest

Fast synchronous request feedback is also RTP transport layer packet. The receiver notifies the coder that one or more images are missing.

The RFC says that when a media receiver is unable to synchronize some media streams, it sends an RRR to the media sender, expecting the media sender to send an SR packet as soon as possible.

type RapidResynchronizationRequest struct {
  SenderSSRC uint32
  MediaSSRC uint32
}
Copy the code

RRR fci.

func (p RapidResynchronizationRequest) Marshal() ([]byte, error) { rawPacket := make([]byte, p.len()) packetBody := rawPacket[headerLength:] binary.BigEndian.PutUint32(packetBody, p.SenderSSRC) binary.BigEndian.PutUint32(packetBody[rrrMediaOffset:], p.MediaSSRC) hData, err := p.Header().Marshal() if err ! = nil { return nil, err } copy(rawPacket, hData) return rawPacket, nil }Copy the code

The length of the RRR is a 4-byte public header and an 8-byte feedback header.

Other parts are similar to other types of RTCP, so I won’t go into details.

TransportLayerCC

TCC, also known as TWCC, and REMB are called bandwidth adaptive algorithms.

TWCC has two advantages:

  • RTP packets, rather than media streams, are more suitable for congestion control
  • Earlier packet loss detection and recovery can be performed

If you want to use TWCC, first you need to extend the RTP header, second you need to say “TWCC enabled” in the SDP, and third you need RTCP support.

Since PION/RTP supports TCC but is not used in pion’s other projects, RTCP’s TCC is not analyzed yet.

PictureLossIndication

Pli, load layer feedback.

type PictureLossIndication struct { SenderSSRC uint32 MediaSSRC uint32 } func (p PictureLossIndication) Marshal() ([]byte, error) { rawPacket := make([]byte, p.len()) packetBody := rawPacket[headerLength:] binary.BigEndian.PutUint32(packetBody, p.SenderSSRC) binary.BigEndian.PutUint32(packetBody[4:], p.MediaSSRC) h := Header{ Count: FormatPLI, Type: TypePayloadSpecificFeedback, Length: pliLength, } hData, err := h.Marshal() if err ! = nil { return nil, err } copy(rawPacket, hData) return rawPacket, nil }Copy the code

This kind of notification feedback does not need FCI, just like RRR. Pli is 12 bytes, and most of the logic is similar to RRR.

SliceLossIndication

If an image is layered into several small pieces, numbered 1-N from left to right and top to bottom, SLI is feedback which pieces are missing. Pli is more violent, and it is directly lost.

The FCI of SLI is as follows:

// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | First | Number | PictureID | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Copy the code

Description:

  • First, 13 bits, the First missing block
  • Number, 13 digits, how many pieces are missing
  • PictureID, the lower 6 digit ID of an image, depends on the encoding format

The corresponding types are as follows:

type SLIEntry struct {
  First uint16
  Number uint16
  Picture uint8
}

type SliceLossIndication struct {
  SenderSSRC uint32
  MediaSSRC uint32
  SLI []SLIEntry
}
Copy the code

In serialization and deserialization, it is similar to other types of RTCP packages, except that the usual copy is changed to append, with minor changes.

FullIntraRequest

Fir, which is also a request for key frames, is similar to PLI, but PLI is used to recover from errors, and fir is used when new participants enter, they send FIR, they merge, they also use FIR, if they recover from errors, they don’t use FIR, they use PLI.

// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | SSRC | // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ // | seq nr. | reserved |Copy the code

Seq nr. Is the command sequence number. It is 1 byte and can have multiple Fcis.

type FIREntry struct {
  SSRC           uint32
  SequenceNumber uint8
}

type FullIntraRequest struct {
  SenderSSRC uint32
  MediaSSRC  uint32
  FIR []FIREntry
}
Copy the code

Serialization and deserialization are not attached, the logic is similar to other packages.

ReceiverEstimatedMaximumBitrate

Congestion control algorithm.

Remb and TWCC are algorithms on the sender side and on the receiver side, and their effects are similar.

Remb is to estimate the total available bandwidth of a session.

Remb packets are notifications to multiple media senders: my total estimated available bandwidth. The sender must not send more than the estimated total bandwidth and needs to respond quickly to changes in the transmission bandwidth.

In addition to RTCP packets, you also need to tell SDP that REMB is enabled.

1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P| FMT=15 | PT=206 | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC of packet sender | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC of media source | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Unique identifier 'R' 'E' 'M' 'B' | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Num SSRC | BR Exp | BR Mantissa | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC feedback | + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + |... | * /Copy the code

Here are the corresponding types:

type ReceiverEstimatedMaximumBitrate struct { SenderSSRC uint32 Bitrate uint64 SSRCs []uint32 } func (p Marshal ReceiverEstimatedMaximumBitrate) () (buf byte [], err error) {/ / rebm length is + 4 * number of SSRC buf = 20 bytes make (byte [], p.MarshalSize()) n, err := p.MarshalTo(buf) if err ! = nil { return nil, err } if n ! = len(buf) { return nil, errWrongMarshalSize } return buf, nil }Copy the code

The specific serialization is placed above MarshalTo:

func (p ReceiverEstimatedMaximumBitrate) MarshalTo( buf []byte) (n int, err error) { size := p.MarshalSize() if len(buf) < size { return 0, errPacketTooShort } buf[0] = 143 // v=2, p=0, fmt=15 buf[1] = 206 length := uint16((p.MarshalSize() / 4) - 1) binary.BigEndian.PutUint16(buf[2:4], length) binary.BigEndian.PutUint32(buf[4:8], p.SenderSSRC) binary.BigEndian.PutUint32(buf[8:12], 0) // always zero buf[12] = 'R' buf[13] = 'E' buf[14] = 'M' buf[15] = 'B' buf[16] = byte(len(p.SSRCs)) // // Shift := uint(64-bits. LeadingZeros64(p.bitrate)) // Here we use exponents and mantisses Var mantissa uint var exp uint if shift <= 18 {// if shift <= 18 {// if shift <= 18 {// if shift <= 18 { mantissa = uint(p.Bitrate) exp = 0 } else { mantissa = uint(p.Bitrate >> (shift - 18)) exp = shift - 18 } buf[17] = byte((exp << 2) | (mantissa >> 16)) buf[18] = byte(mantissa >> 8) buf[19] = byte(mantissa) n = 20 for _, ssrc := range p.SSRCs { binary.BigEndian.PutUint32(buf[n:n+4], ssrc) n += 4 } return n, nil }Copy the code

There’s a lot of new stuff going on in serialization.

Finally in the printing, reflected a lot of fun units.

Combination packages CompoundPacket

RTCP packets are very small, so many are aggregated and sent together. There are some rules for sending:

  • The first package in a composite package must be the report package SR or RR
    • Send an empty RR packet even if there is no data
    • Add an empty RR package even if there is only one BYE package to combine
  • Each composite package should contain an SDES package with a CNAME entry

The type is []Packet

type CompoundPacket []Packet var _ Packet = (*CompoundPacket)(nil) // assert is a Packet func (c CompoundPacket) Marshal() ([]byte, error) { if err := c.Validate(); err ! = nil { return nil, err } p := []Packet(c) return Marshal(p) }Copy the code

When we analyze RTCP, the first object we analyze is Packet and the serialization and deserialization based on Packet array, which happens to be used here.

func (c *CompoundPacket) Unmarshal(rawData []byte) error { out := make(CompoundPacket, 0) for len(rawData) ! = 0 { p, processed, err := unmarshal(rawData) if err ! = nil { return err } out = append(out, p) rawData = rawData[processed:] } *c = out if err := c.Validate(); err ! = nil { return err } return nil }Copy the code

Now let’s analyze the last one, the RFC defined checksum process:

Func (c CompoundPacket) Validate() error {if len(c) == 0 {return errEmptyCompound} the first packet must be a report packet (SR/RR) switch c[0].(type) { case *SenderReport, *ReceiverReport: // ok default: Return errBadFirstPacket} // Sdes. Cname must contain a sdes. Cname can only be preceded by a Sdes. pkt := range c[1:] { switch p := pkt.(type) { case *ReceiverReport: continue case *SourceDescription: var hasCNAME bool for _, c := range p.Chunks { for _, it := range c.Items { if it.Type == SDESCNAME { hasCNAME = true } } } if ! hasCNAME { return errMissingCNAME } return nil default: return errPacketBeforeCNAME } } return errMissingCNAME }Copy the code

At this point,pion/ RTCP package analysis is complete, except TCC not detailed analysis, other packages have been through.