preface
This series is Golang combat, from 0 to 1 to achieve a barrage system. In the process of development, some off-the-shelf wheels will be implemented, just for learning to use, in practical use, or use off-the-shelf wheels.
At present, the live broadcasting industry is on fire. There are large live broadcasting platforms such as Douyu and Huya, as well as live broadcasting with goods and e-commerce, etc. All of these live broadcasting have bullet screen system. Live broadcasting without bullet screen is soulless. Not just live streaming, but when you watch videos on major video websites, don’t you open a pop-up screen? !!!!! Sometimes bullets are better than video content.
Bullet screen system has a wide range of application scenarios, and its main characteristics are high real-time performance and large concurrency, especially concurrency. In the live broadcast scene, often a big anchor, a live broadcast down millions of bullets easily. Due to golang’s superior performance in concurrent scenarios, we chose to use Golang to implement the barrage system.
Websocket protocol implementation
For sure, webSocket protocol cannot be wrapped around the bullet-screen system. The system using bullet-screen basically has H5 application, and the bullet-screen system on H5 application must be realized by WebSocket.
There are many ready-made WebSocket class libraries in various languages, such as nodeJS socke. IO, PHP swoole, etc. At the beginning of the project, we did not use these ready-made libraries, we implemented a simple WebSocket protocol by ourselves, and learned the WebSocket protocol in the implementation process.
The websocket protocol is described in RFC6455 and is available in Chinese.
Without further ado, let’s begin.
Websocket establishes a handshake based on the HTTP service. We start an HTTP service:
func main(a) {
http.HandleFunc("/echo".func(writer http.ResponseWriter, request *http.Request) {
serveWs(writer, request)
})
err := http.ListenAndServe("localhost:9527".nil)
iferr ! =nil {
log.Fatal(err)
}
}
Copy the codeWe first use an Echo service to test the implementation of webSocket protocol, port 9527, serveWs function is very simple, establish socket connection, read information and write back, that is, an Echo service.
Handshake connection
Section 4 of the RFC6455 document details the process of establishing a handshake. We follow the documentation step by step.
The method of the request MUST be GET, and the HTTP version MUST be at least 1.1.
HTTP methods must be GET methods:
ifrequest.Method ! ="GET" {
return nil, fmt.Errorf("HTTP must be a GET method")}Copy the codeThe request MUST contain an
Upgradeheader field whose value MUST include the “websocket” keyword.
There must be a Upgradeheader and its value must be websocket:
if! httpHeaderContainsValue(request.Header,"Upgrade"."websocket") {
return nil, fmt.Errorf("Must contain an Upgrade Header field, which must be websocket")}Copy the codeAnd so on a series of checks, I won’t go over them here.
After the HTTP header is verified, we can deal with TCP connections. As we all know, HTTP is an application layer protocol on top of TCP. Normally, TCP will be disconnected after an HTTP request is completed. A Websocket connection is actually a TCP connection, so we can’t break the TCP connection in HTTP. We manage the TCP connection ourselves. How do I get a TCP connection in HTTP? Golang provides us with a Hijack method.
hijacker, ok := writer.(http.Hijacker)
if! ok {return nil, fmt.Errorf("HTTP.Hijacker not implemented")
}
netConn, buf, err := hijacker.Hijack()
iferr ! =nil {
return nil, err
}
Copy the codeAfter receiving the handshake request from the client, we must respond to the request from the front end to complete the whole handshake process.
If the status code received from the server is not 101, the
client handles the response per HTTP [RFC2616] procedures. In particular, the client might perform authentication if it receives a 401 status code; the server might redirect the client using a 3xx status code (but clients are not required to follow them), etc. Otherwise, proceed as follows.
We simply implement a WebSocket protocol and set the status code to 101:
var response []byte
response = append(response, "HTTP / 1.1 101 Switching separate Protocols \ r \ n"...).Copy the codeIf the response lacks an |Upgrade| header field or the |Upgrade|
header field contains a value that is not an ASCII case- insensitive match for the value “websocket”, the client MUST Fail the WebSocket Connection.
The response must contain Upgradeheader information and the value must be websocket (case insensitive) :
response = append(response, "Upgrade: websocket\r\n"...).Copy the codeIf the response lacks a |Connection| header field or the
|Connection| header field doesn’t contain a token that is an ASCII case-insensitive match for the value “Upgrade”, the client MUST Fail the WebSocket Connection.
Must have Connection and the value must be Connection:
response = append(response, "Connection: Upgrade\r\n"...).Copy the codeIf the response lacks a |Sec-WebSocket-Accept| header field or
the |Sec-WebSocket-Accept| contains a value other than the base64-encoded SHA-1 of the concatenation of the |Sec-WebSocket- Key| (as a string, not base64-decoded) with the string “258EAFA5- E914-47DA-95CA-C5AB0DC85B11” but ignoring any leading and trailing whitespace, the client MUST Fail the WebSocket Connection.
Sec-websocket-acceptheader is required. The value must be concatenated according to the value of sec-websocket-key and “258eAFa5-E914-47DA-95CA-C5AB0DC85B11”. Ignore all Spaces before and after the base64 SHA-1 encoding to obtain:
var acceptKeyGUID = []byte("258EAFA5-E914-47DA-95CA-C5AB0DC85B11")
generateAcceptKey(request.Header.Get("Sec-WebSocket-Key"))
func generateAcceptKey(key string) string {
h := sha1.New()
h.Write([]byte(key))
h.Write(acceptKeyGUID)
return base64.StdEncoding.EncodeToString(h.Sum(nil))
}
response = append(response, "Sec-WebSocket-Accept: "...). response =append(response, generateAcceptKey(request.Header.Get("Sec-WebSocket-Key"))...)
response = append(response, "\r\n\r\n"...).Copy the codeOther information that is not required will not be added. After the header is constructed, it responds to the client:
if_, err = netConn.Write(response); err ! =nil {
netConn.Close()
}
Copy the codeSince the TCP connection is managed by ourselves, we need to close the connection when the handshake fails.
Now, we’ve done the whole handshake. Test it out:
The handshake is successful, and then we begin the detailed sending and receiving processing.
Read the message
In the WebSocket protocol, data is transmitted through a series of data frames, which are described in detail in section 5 of RFC6455.
Basic data frame format:
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-------+-+-------------+-------------------------------+ |F|R|R|R| opcode|M| Payload len | Extended payload length | |I|S|S|S| (4) |A| (7) | (16/64) | |N|V|V|V| |S| | (if payload len==126/127) | | |1|2|3| |K| | | +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - + | Extended payload length continued, if payload len == 127 | + - - - - - - - - - - - - - - - +-------------------------------+ | |Masking-key, if MASK set to 1 | +-------------------------------+-------------------------------+ | Masking-key (continued) | Payload Data | +-------------------------------- - - - - - - - - - - - - - - - + : Payload Data continued ... : + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + | Payload Data continued ... | +---------------------------------------------------------------+Copy the codeExtract some important information:
FIN: 1 bit
Indicates that this is the final fragment in a message. The first fragment MAY also be the final fragment. Copy the code
RSV1, RSV2, RSV3: 1 bit each
MUST be 0 unless an extension is negotiated that defines meanings for non-zero values. If a nonzero value is received and none of the negotiated extensions defines the meaning of such a nonzero value, the receiving endpoint MUST _Fail the WebSocket Connection_. Copy the code
Opcode: 4 bits
Defines the interpretation of the "Payload data". If an unknown opcode is received, the receiving endpoint MUST _Fail the WebSocket Connection_. The following values are defined. * %x0 denotes a continuation frame * %x1 denotes a text frame * %x2 denotes a binary frame * %x3-7 are reserved for further non-control frames * %x8 denotes a connection close * %x9 denotes a ping * %xA denotes a pong * %xB-F are reserved for further control frames Copy the code
Mask: 1 bit
Defines whether the "Payload data" is masked. If set to 1, a masking key is present in masking-key, and this is used to unmask the "Payload data" as per Section 5.3. All frames sent from client to server have this bit set to 1. Copy the code
Payload length: 7 bits, 7+16 bits, or 7+64 bits
The length of the "Payload data", in bytes: if 0-125, that is the payload length. If 126, the following 2 bytes interpreted as a 16-bit unsigned integer are the payload length. If 127, the following 8 bytes interpreted as a 64-bit unsigned integer (the most significant bit MUST be 0) are the payload length. Multibyte length quantities are expressed in network byte order. Note that in all cases, the minimal number of bytes MUST be used to encode the length, for example, the length of a 124-byte-long string can't be encoded as the sequence 126, 0, 124. The payload length is the length of the "Extension data" + the length of the "Application data". The length of the "Extension data" may be zero, in which case the payload length is the length of the "Application data". Copy the code
Masking-key: 0 or 4 bytes
All frames sent from the client to the server are masked by a 32-bit value that is contained within the frame. This field is present if the mask bit is set to 1 and is absent if the mask bit is set to 0. See Section 5.3 for further information on client- to-server masking. Copy the code
Payload data: (x+y) bytes
The "Payload data" is defined as "Extension data" concatenated with "Application data". Copy the code
Before we start parsing data frames, let’s define some data structures and methods:
type Conn struct {
conn net.Conn
br *bufio.Reader
writeBuf []byte / / write cache
readLength int64 // Data length
maskKey [4]byte // mask key
}
// Data frame bit
/ / RFC6455 section 5.2
const (
finalBit = 1 << 7
rsv1Bit = 1 << 6
rsv2Bit = 1 << 5
rsv3Bit = 1 << 4
opCode = 0xf
maskBit = 1 << 7
pladloadLen = 0x7f
)
// Message type
RFC6455 section 5.2 or 11.8
const (
ContinuationMessage = 0
TextMessage = 1
BinaryMessage = 2
CloseMessage = 8
PingMessage = 9
PongMessage = 10
)
func (c *Conn)read(n int) ([]byte, error) {
// Read n bytes of data
p, err := c.br.Peek(n)
// Discard n bytes of data
c.br.Discard(len(p))
return p, err
}
func newConn(conn net.Conn, br *bufio.Reader) *Conn {
c := &Conn{
conn:conn,
br:br,
writeBuf:make([]byte.128), // Write to death, only 128 bytes of data are accepted
}
return c
}
Copy the codeRead data frame header information
In a network, data is transferred in bytes. In websocket, all important header information is in the first 2 bytes, read the first 2 bytes:
p, err := c.read(2)
iferr ! =nil {
return err
}
// Parse data frame RFC6455 section 5.2
// Force to press 0, regardless of whether there is extended information
if rsv := p[0] & (rsv1Bit | rsv2Bit | rsv3Bit); rsv ! =0 {
return fmt.Errorf("RSV must be 0")}// indicates that this is the last fragment of the message. The first clip could be the last.
// The FIN bit information is not considered for the moment
// final := p[0]&finalBit ! = 0
frameType := int(p[0]&opCode)
// Check whether the FIN and opcode match
/ / RFC6455 section 5.4
// todo
switch frameType {
case ContinuationMessage:
case TextMessage, BinaryMessage:
case CloseMessage, PingMessage, PongMessage:
default:
return fmt.Errorf("Unknown Opcode")}Copy the codeAll frames sent from client to server have this bit set to 1.
The mask bit in the data frame of the client must be 1:
mask := p[1]&maskBit ! =0
if! mask {return fmt.Errorf("Mask bit must be marked as 1")}Copy the codeThen get the length of the application data
Payload length: 7 bits, 7+16 bits, or 7+64 bits
c.readLength = int64(p[1]&pladloadLen)
Copy the codeIf the data length is less than or equal to 125, the actual value is the application data length. If equal to 126, then the next two bytes are interpreted as a 16-bit unsigned integer as the length of the application data. If equal to 127, then the next 8 bytes are interpreted as a 64-bit unsigned integer as the length of the application data.
// Get the length of the data
/ / https://tools.ietf.org/html/rfc6455#section-5.2
// The length of the "Payload data", in bytes: if 0-125, that is the
// payload length. If 126, the following 2 bytes interpreted as a
// 16-bit unsigned integer are the payload length. If 127, the
// following 8 bytes interpreted as a 64-bit unsigned integer (the
// most significant bit MUST be 0) are the payload length. Multibyte
// length quantities are expressed in network byte order.
c.readLength = int64(p[1]&pladloadLen)
switch c.readLength {
case 126:
p, err := c.read(2)
iferr ! =nil {
return err
}
c.readLength = int64(binary.BigEndian.Uint16(p))
case 127:
p, err := c.read(8)
iferr ! =nil {
return err
}
c.readLength = int64(binary.BigEndian.Uint64(p))
}
Copy the codeObtain the mask-key mask:
Masking-key: 0 or 4 bytes
p, err := c.read(4)
iferr ! =nil {
return err
}
Copy the codeWe only send the simplest data, other data frame information is not present, we will not be compatible with processing.
Reading Application Data
Reading application data is simple:
// The read length is increased by 1 to make it easier to read EOF directly
var p = make([]byte, c.readLength+1)
n, err := c.br.Read(p)
iferr ! =nil {
return nil, err
}
Copy the codeSince the data read is after the mask, we need to decode it. The mask and decoding algorithms are described in detail in Section 5.3.
Octet i of the transformed data (“transformed-octet-i”) is the XOR of octet i of the original data (“original-octet-i”) with octet at index i modulo 4 of the masking key (“masking-key-octet-j”):
j = i MOD 4 transformed-octet-i = original-octet-i XOR masking-key-octet-j Copy the code
// Only English and digits are supported
func maskBytes(key [4]byte, pos int, b []byte) int {
for i := range b {
b[i] ^= key[pos%4]
pos++
}
return pos%4
}
maskBytes(c.maskKey, 0, p[:n])
Copy the codeTest it out:
$ go run *.go
rsv msg WebSocket rocks
Copy the codeThe data is successfully read. Let’s start writing the message.
Writing Application Data
To keep things simple, only 125 bytes of data can be written. Because data exceeding 125 bytes, Payload len bits have to be treated specially. In this case, the main header information is in the first two bytes.
Enforces text information, and sets RSV to 0 and FIN to 1:
// First byte (first 8 bits)
// The default message is text
b0 := byte(TextMessage)
// FIN 1
b0 |= finalBit
Copy the codeThen set the MASK bit and the application data length:
b1 := byte(0)
b1 |= byte(len(msg))
Copy the codeYou can set the Payload len bit, because you can only write up to 125 bytes of data. Write cache:
c.writeBuf[0] = b0
c.writeBuf[1] = b1
Copy the codeWith the headers set, we can write application data:
func (c *Conn) WriteMessage(msg []byte) error{...copy(c.writeBuf[2:], msg)
_, err := c.conn.Write(c.writeBuf)
return err
}
Copy the codeLet’s test it again:
At this point, we have a super simple, can only learn to use webSocket protocol implementation.
We did not deal with websocket extension information and binary type information, ping, pong, etc., and we did not deal with data fragmentation, sticky packet processing, etc. So this is just learning.
The code address of the project is gitee.com/ask/danmaku
There is a good blog about websocket.