In 2020, due to the impact of the epidemic, all kinds of live streaming of e-commerce have flooded into people’s eyes, and many businesses sell their products through live streaming.

Live broadcast barrage is one of the core functions of live broadcast system. How to quickly make a missile barrage system with good scalability? How to deal with rapid business development? I’m sure many engineers/architects have their own ideas. Bilibili as one of the leading live video websites in China, today we take a look at how they do it.
** One to many live system development, focusing on these **
High concurrency real-time barrage is an interactive experience. For interaction, the most important considerations are: high stability, high availability, and low latency.
High stability, in order to ensure the real-time interaction, so the connection state is required to be stable.
High availability is equivalent to providing a backup solution. For example, if one machine is down during interaction, it must be connected to another machine at this time. In this way, the problem of uninterrupted user connection is solved from the side.
For low delay, the delay period of barrage is controlled within 1 second, and the response is relatively fast, so it can meet the needs of interaction.

** One to many live system development, ****GOIM system optimization road **
First, memory optimization
Memory optimization is mainly divided into the following three aspects:

1. A message must have only one piece of memory that uses Job aggregation messages, referenced by Comet Pointers.

2. As far as possible, a user’s memory is placed on the stack. Memory is created in the corresponding user’s Goroutine (Go program).

3. Memory control is mainly for Comet module optimization, you can view the module’s various memory allocation places, use the memory pool.

Second, module optimization

Ensure that each Comet communication channel is independent of each other and that message distribution is completely parallel and does not interfere with each other.

2. Concurrency must be controllable every Goroutine (Go coroutine) that needs to be opened for asynchronous processing must have a fixed number created in advance. If this is not controlled in advance, the Goroutine can explode at any time.

3. Global lock must be broken Socket link pool management, user online data management are multiple locks; The number of scatters usually depends on the CPU, and it is often necessary to consider the burden of CPU switching, not the more the better.

Three aspects of module optimization, the main consideration of the problem is that the distributed system will appear in the single point of problem, that is, when a user in the establishment of links, if there is a failure, the links established by other users can not be affected.
Testing is the most indispensable part of the practice process, at the same time, the test data is also used for reference and comparison of the best tool.
4. Replace short connection polling scheme with long connection
Long connection overall architecture diagram

The long connection service module is used to verify the user entering the live broadcast room.

The communication between the servers uses the TARDIS framework developed by the basic r&d team for service invocation, which is based on gRPC and uses ETCD for service discovery.
We adopt the subscription push model. For example, user 1 subscribes to A live broadcast, and A live broadcast has new messages:
After the push layer queries the subscription relationship, it knows that user 1 has subscribed to A live broadcast and user 1 is on the node of connection layer 1, so it will inform the connection layer of new messages. After receiving the notification message, connection layer 1 waits for a short period of time (millisecond level), pulls the message from user 1 again, and pushes the message to user 1.
If it is a large live broadcast room (with many subscribers), the notification/pull model of the push layer and the connection layer will be automatically downgraded to the broadcast model.
5. The final bottleneck of live broadcast barrage optimization is only flow

1. Adjust the user optimal A-Node

The Svrlist module is used to select the most stable node closest to the user, adjust the IP address segment, and access the node.

2.IDC’s service quality monitoring: call drop rate

To judge whether a node is stable, a large number of user link information needs to be collected continuously. In this case, you can use monitoring to query the disconnection rate, and then continuously tune and collect the final results to make a topology (nationwide). In the topology, you can determine the optimal line between the city and the equipment room.

3. Flow control

For barrage, when users reach a certain level, you need to consider problems or flow control, it is also for cost cost control, when buy the bandwidth of the room, is based on gigabit bandwidth pricing standard, when there is excessive, must cut exceeds bid section of the flow, to realize the flow control function.