Abstract: With the development of the Internet, more and more people like live broadcasting, and Baidu live broadcasting is also developing rapidly. In order to improve the user experience, this paper summarizes the complex process of Baidu Live broadcasting, and describes in detail a series of optimization work carried out.

The full text is 4,216 words and the estimated reading time is 9 minutes.

The background,

Baidu has two goals for livestreaming:

1. Copy the real world to online, so that you can have the same experience online and offline. Types of live broadcast include media, consulting, e-commerce, show, etc.

2. It is a perfect shaping of beautiful imagination. With the arrival of 5G, VR and AI, it brings us the space of imagination.

The first thing to do in live broadcasting is QOE (Quality of Experience). As technical students, we should not only ensure the business objectives, but also improve the user Experience.

There are many experiences of live broadcasting: first screen time, delay, picture quality, sound and picture synchronization, clarity, noise, echo, etc.

From the perspective of QOE, there are many standard QOS (Quality of Service) as a measure: push stream success rate, push stream slow flow, push slow speed ratio, end-to-end delay, CDN smooth rate, start time, pull stream slow rate, video bit rate, push slow speed ratio, memory consumption, CPU&GPU consumption and so on. These technical indicators constitute a unified whole as a standard to measure the quality of live streaming services.

However, from the perspective of live broadcast users, when users click live broadcast, they want to see the screen immediately, that is, the start speed of live broadcast should be fast. When swiping up and down an immersive studio, users will also expect the live stream they are looking at on the current screen to start playing quickly. As the user’s first perception of live broadcast, the time of starting broadcast is put into the primary optimization goal.

Second, the status quo

Baidu Live broadcasting is divided into medium service live broadcasting and medium entertainment live broadcasting. Medium service live broadcasting provides media live broadcasting, consultation live broadcasting, e-commerce live broadcasting and other services, while medium entertainment live broadcasting provides entertainment live broadcasting such as show field, audio broadcast and voice room.

Among them, the pan-service live broadcast is more complicated, and the starting process has the following characteristics:

1. Cumbersome process: it is divided into two sets of processes: external jump in the live broadcast room and immersive jump in the live broadcast room, among which there are many flow links;

2. Many live broadcast states: including live broadcast, playback, playback generated medium state

3. Wide range of involvement: involving multiple teams of Baidu App: live broadcast, player, kernel, network, CDN, etc

4. Changing wheels for driving cars: Baidu attaches great importance to live broadcasting and rapidly iterates its business. It needs to change wheels for driving cars.

After summarizing the characteristics of pan-service initiation, it is necessary to quantitatively analyze the whole process of initiation and measure the whole process of initiation with data.

3. Data analysis

Based on the data analysis of the whole start-up process, the start-up process can be roughly divided into three stages: ** time of live broadcasting business, time of player and time of kernel. ** Here is a rough breakdown:

In the actual data statistics, more detailed statistics will be carried out for each link. Briefly talk about the data statistics of live broadcasting business and the data statistics of the kernel:

Data quantification is carried out for each link of live streaming business, and the time consumption of each step is accurate, so that analysis can be carried out on the basis of data.

Take the kernel as an example:

You can then get a detailed chart of the kernel data.

From the user’s click to jump to the live broadcast room or slide to switch the live broadcast room within the immersive live broadcast room to the final success of the live broadcast, it is expected that more than 60 points will be tracked and analyzed in a series. Detailed tables of data show how long each step of the start process takes, and then optimize for where it takes more time.

First of all, the time of business scenarios accounts for the largest proportion in the report, accounting for more than 60%, so the first thing to solve is this time. The second phase with the most time is the time of pull flow; When the large time is solved, it is necessary to optimize the small time.

Optimization of service scenarios

After the fusion of the media live broadcast of Baidu App and the nationwide live broadcast of the show, they will be mixed and distributed in the studio. From the perspective of live jump, the jump in the live broadcast room can be divided into two types: one is the external jump to the live broadcast room through scheme; the other is the sliding jump in the immersive live broadcast room;

The following is a simple process for the external to jump to the broadcast room through Scheme:

There are two ways to jump to the broadcast room through Scheme:

1. If there is no roomID, first request the list interface, then continue to request the relevant information of the current broadcast room according to the roomID in the list, install components & plug-ins and render operations according to the relevant information, and start to create the player and setURL after the page rendering is completed, and initialize the kernel. Start play until play successful callback;

2. In the case of a roomID, information about the current broadcast room is directly requested and the same operation as in Step 1 is performed.

Here’s a slide jump in the immersive studio:

Compared with the external jump, the jump in the immersive live broadcast room has more users’ sliding operation. From the start of the slide stop to the start of the broadcast statistics until the successful callback, the duration of the whole broadcast is expected to be about 1700ms, while in the whole process, the whole live broadcast service takes about 1000ms. External jump: Internal jump = 1: N; N is much greater than 1. In this way, it is preferred to optimize the jump in the broadcast room.

After qualitative and quantitative analysis, the optimization plan is formulated:

Based on the consideration of page lag, there are two overall schemes A&B:

Solution A is implemented on iphone8 and above models: start creating the player at the beginning of the user’s slide, and directly call play.

Plan B is implemented on models below iphone8: the player is created and resources are prepared when the user slides, and the last broadcast room is destroyed after the user stops sliding and the next broadcast room is played.

The implementation of A&B scheme makes the lag rate of models around the iphone8 perform well, and will not bring the experience of degradation to users. For models, it is also a temporary balance sought by ABTest to constantly test the balance between the start and lag. In the future, continuous optimization will be made for the catton to avoid frequent creation and destruction of objects, so as to reduce the constant calculation of CPU, so as to continuously expand the models adapted to scheme A.

With the optimization scheme of the internal jump of the broadcast room, the jump of the external scheme is also obvious:

The communication scheme between components of Baidu App uses scheme for communication, so add the URL of live broadcast in Scheme. When jumping to the live broadcast page, directly create the player by the URL carried in scheme and start to play live broadcast, which is executed in parallel with the business logic.

4. DNS preresolution

The Domain Name System (DNS) is used to search for IP addresses based on Domain names. It is the prerequisite of THE HTTP protocol. The following processes can proceed only after the Domain Name is correctly resolved into an IP address. When the kernel processes the live stream, the live DNS takes 80 bits about 60ms, which is also a lot of time.

In Baidu AppAPP, to prevent DNS hijacking and reduce the network delay, we adopted the HTTPDNS solution:

To optimize the time of DNS resolution in live broadcast, the DNS pre-resolution policy is adopted to cache the CORRESPONDING IP addresses in advance and reduce the time of DNS resolution.

In the application of cold start 10s, network switchover, front and background switchover, the model determines whether to adopt the live DNS pre-resolution scheme according to the model. The model’s judgment criteria are as follows: user browsing live duration M within N days, current network status, back-end control, and so on. When the model is judged to pass, HTTPDNS will be called to asynchronously initiate the resolution of the live domain name to obtain a list of IP, respectively for IP speed measurement, speed measurement is not directly choose the best result of that one, but in the test results can be accepted within a certain range of random selection and cache the results of pre-resolution. A large number of users are clustered on a small number of nodes, which causes load imbalance among nodes.

The validity time of HTTPDNS cache IP is obtained from the server, which is 300s by default. When the live watershed name resolution starts, query the cache, if there is a valid IP(cache time <300s) directly return the corresponding IP, and call HTTPDNS asynchronously to initiate a request for update. If it does not exist, the update request will be asynchronously initiated and will be degraded to LocalDNS resolution.

When the network changes (Wifi <-> 4G), clear the current cached IP and re-initiate the pre-resolution of all domain names in the domain list.

Final benefit: PVS with live DNS taking less than 3ms account for more than 90% of the total live PVS with HTTPDNS pre-parsing.

Five, some kernel optimizations

A series of optimization has been carried out for the time-consuming stage in the report:

1, the first screen forced rendering: mainly is the first frame of the video decoding will be forced to go through the rendering process, not to do sound and picture synchronization;

2. In the case of weak network, start the low bit rate multicast strategy;

3. Load the next player kernel for high-end models, and prepare. When switching to live broadcast, carry out the follow-up frame operation. The logic is that if the buffer delay is longer than 3 seconds, it will lose 200ms of audio every 2 seconds until it is less than 3 seconds. If the buffer delay is longer than 16 seconds, it will lose all the audio and reconnect directly.

Vi. Optimization of live start

Optimization of live start – media information analysis module optimization

Video width and height, image encoding format and other information are required for the Hardware decoder Mediacodec on The Android platform and Video ToolBox on the IOS platform. Before configuring the hardware decoder on the platform, the Video width and height, image encoding format and other information should be prepared.

Some packaging formats (such as MP4) will describe the video width and height, image encoding format and other information in the head. For the case that the video container does not contain the above information (such as FLV), FFMPEG native process will loop download the video stream, and then through the software decoders to decode the video, to obtain the above information;

In H.264/AVC video coding standard, the whole system framework is divided into two levels: video coding level (VCL) and network abstraction level (NAL). Among them, the former is responsible for effectively representing the content of the video data, while the latter is responsible for formatting the data and providing header information to ensure that the data is suitable for transmission over various channels and storage media. SPS, PPS in NAL has width and height information, image coding format description, can be obtained by analyzing SPS, PPS necessary information, eliminating the process of soft solution video.

7. Live playback (HLS) M3U8 prefetch

Live video is usually encoded into HLS format and saved, which is called live playback. The 80th bit of video in HLS package format is 1250ms, and the playback speed is the core indicator reflecting the playback experience. The performance of live playback is significantly worse than that of live (HTTP-FLV) at 510ms.

The main reason for the slow start of HLS encapsulation is that you need to download the video index file (M3U8) first, and then parse the index file to get the real video file, which means that there is at least one more HTTP request than live (HTTP-FLV). To solve this problem, prefetch the HLS video index file (M3U8) to the local SDCard in advance; The time of M3U8 download is saved when the broadcast starts. AB experiment benefits: Compared with the hit and miss prefetch M3U8 experimental group, the hit prefetch benefits are 346ms.

References:

Chromium.googlesource.com/chromium/sr…

Tools.ietf.org/html/rfc785…

Github.com/bilibili/ij…

Recruitment Information:

Baidu – Live research and development department – pan-knowledge live group, the team aims to build the industry first-class live experience, technology-driven business, and constantly innovate in the live scene, combined with VR&AR&AI and other technologies, and constantly explore new gameplay. By playing the kernel, the team continuously improves the basic performance of the kernel, enhances the kernel capability, improves the index monitoring, improves the service capability, and finally realizes better user experience and product quality.

We sincerely invite iOS and Android friends to follow the public account with the same name. Click “Push” in the menu bar to join the search architecture department. We look forward to your joining!

Recommended Reading:

Baidu search stability analysis of the story (second)

Baidu search stability analysis of the story (on)

Baidu on the micro front-end architecture EMP exploration: ground production available micro front-end architecture

———- END ———-

Baidu said Geek

Baidu official technology public number online!

Technical dry goods · Industry information · online salon · Industry conference

Recruitment information · Internal promotion information · Technical books · Around Baidu

Welcome your attention