Audio and video learning (10) Build live broadcast server (RTMP, HTTP) based on Nginx

preface

In recent years, the field of live broadcast and short video is really hot, and the field of live broadcast is also very wide. It can be foreseen that audio and video technology will be applied to a wider range of scenes as a basic technology in the future. It can be combined with AR/VR, so that you can experience virtual and reality in the remote end, such as virtual clothing experience; It can also be combined with artificial intelligence to improve the quality of service, such as teaching to help teachers improve teaching quality; It can also be combined with the Internet of things for autonomous driving, home work and other applications. Can we participate in such a wide range of fields? Sure, we can. Below we use Nginx and nginx-HTTP-flV-Module to build a simple live server, of course, if the concurrency requirements are not too high, this can be satisfied.

Because the author has no background service development experience and some service environment configuration is not very understanding, so also referred to some blogs and finally summarized a set of installation and use tutorial, the following attached detailed steps.

Note: if http://ip:port appears in the following text, please replace it with your own IP or domain name.

Introduction to Live Broadcast Protocols

There are several common live broadcast protocols in China: RTMP, HLS, HDL (HTTP-FLV) and RTP. Let’s introduce them one by one.

RTMP

It is a proprietary agreement of Adobe, which is no longer supported by most foreign CDNS. It is very popular in China. There are several reasons for this:

1. Stable and complete support for open source software and open source libraries. For example, OBS software commonly used by Douyu anchors, open source Librtmp library, nginx-rtmp plug-in on the server side.

2. High installation rate of the playback end. As long as the browser supports FlashPlayer, it is very easy to play live RTMP. Compared with other protocols, the initial handshake process of RTMP is too complicated (the underlying layer is based on TCP, here we are talking about the interaction of RTMP itself). Depending on different network conditions, it will bring more than 100ms delay for the initial connection. Rtmp-based broadcast content generally has a delay of 2~5 seconds.

HTTP-FLV

That is to use HTTP protocol to transmit media content streaming. Compared to RTMP, HTTP is simpler and better known, without fear of being held hostage to Adobe’s patents. Content latency can also be 2-5 seconds, which is faster to open because HTTP itself does not have complex state interactions. So http-FLV is superior to RTMP in terms of latency.

HLS

Http Live Streaming is a media Streaming protocol based on Http proposed by Apple. HLS has one big advantage: HTML5 can play directly; This means that a live broadcast link can be forwarded and shared through wechat and other channels without the need to install any independent APP and only need to have a browser, so its popularity is very high. Social broadcast APP, HLS, can be said to be just needed, let’s analyze its principle.

The HLS based live stream URL is an M3U8 file that contains several recent video TS (a video encapsulation format, which will not be expanded here) files. For example, http://ip:port/live.m3u8 is a live broadcast retention link, and its content is as follows:

Assuming that the list contains five TS files, each of which contains five seconds of video content, the overall delay is 25 seconds. It is certainly possible to shorten the length of the list and the size of the individual TS files to reduce latency, and in extreme cases it is possible to shorten the m3u8 files with a list length of 1,1 seconds of content, but it is extremely vulnerable to network fluctuations causing lag. Through the verification of the public network, the best effect can be achieved by using the same-city network at present, which is the result of the comprehensive fluency and content delay.

RTP

The Real-Time Transport Protocol is a Transport layer Protocol for multimedia data flows on the Internet.

In actual application scenarios, RTP Control Protocol (RTCP) is often used together. RTCP transmits interactive Control signaling and RTP transmits actual media data.

RTP is widely used in video surveillance, video conferencing, and IP phone, because one important user experience of video conferencing and IP phone is real-time content.

One important difference between RTP and the above three or two protocols is that RTP uses UDP to transmit data by default, while RTMP and HTTP are TCP based. Why can UDP achieve such real-time effect? The difference between TCP and UDP analysis of a large number of articles, not to repeat here, a brief summary:

UDP: single datagram, do not need to establish a connection, simple, unreliable, packet loss, will be out of order;

TCP: streaming, need to establish connections, complex, reliable, orderly.

Real-time audio and video streaming scenes do not need reliable guarantee, so there is no need to have a retransmission mechanism, real-time view of the image and sound, network jitter lost some content, blurred and blurred screen, completely unimportant. TCP causes delay and unsynchronization for retransmission. For example, when a piece of content is retransmitted and arrives one second later, the entire conversation will be delayed by one second. With network jitter, the delay will increase to two or three seconds.

To summarize: In the selection of live protocols, RTMP or HTTP-FLV means a content delay of 2-5 seconds, but in terms of open delay, HTTP-FLV is superior to RTMP. HLS has a content delay of 5-7 seconds. If RTP is selected for live broadcast, the live broadcast delay can be achieved within 1 second. However, as far as we know, major CDN manufacturers do not support live broadcast based on RTP, so the mainstream in China is still RTMP or HTTP-FLV.

Open source streaming media server software

• Streaming media server

Environment to prepare

Originally, I planned to build it on my personal blog server, but finally I gave up, because the bandwidth and capacity were not very large before. I happened to meet Tencent Cloud and bought another server during activities these days, and the project about background service was basically deployed on this server.

The following is the environment I built and tested

Setting up the Server Environment

Cloud server: Tencent Cloud

System: centos

Live server: nginx

Nginx-flv-module (support RTMP, HTTP-FLV, http-hLS, etc.)

If NGINX is to support regular expressions, you need to install the PCRE library.

If NGINX is to support encrypted access, you need to install the OpenSSL library.

If NGINX is to support compression, you need to install the Zlib library.

Test environment:

Computer: MAC

Push stream software: OBS-Studio

MAC pull flow software: VLC

Android pull stream software: a few days ago, an Android player ykplayer is just available for pull stream testing

HTML5 FLV player: Bilibili open source flv.js

Nginx-rtmp-module: nginx-rtmp-module: nginx-rtmp-module: nginx-rtmp-module: nginx-rtmp-module It is based on the nginx-rtmp-module secondary development, so perfect inheritance of all functions of the RTMP module.

Server setup

1. download nginx

#Run the wget command to download the fileWget HTTP: / / http://nginx.org/download/nginx-1.17.8.tar.gz#Unpack theThe tar - ZXVF nginx - 1.17.8. Tar. GzCopy the code

2. download nginx-http-flv-module

Note: Refer to its introduction for more information

#Run the wget command to download the fileWget HTTP: / / https://github.com/winshining/nginx-http-flv-module/archive/v1.2.7.tar.gz#Unpack theThe tar - ZXVF v1.2.7. Tar. Gz#renameThe mv v1.2.7 nginx - HTTP - FLV - moduleCopy the code

3. Install the required environment for nginx

OpenSSL, PCRE, and Zlib errors must be installed if they are reported after configure

#Install openssl
yum install   openssl
#Installing pcre
yum install 	pcre-devel
#Install the zlib
yum install   zlib-devel
Copy the code

Compile nginx after they are installed

4. build nginx

Create and compile nginx and HTTP-FLV scripts in the current unzipped nginx directory

#! /bin/sh
#. / indicates the level above the current directoryHTTP_FLV_MODULE_PATH=.. / nginx - HTTP - FLV - module - 1.2.7 OpenSSL_PATH =). / openssl - 1.1.1 d
#--prefix=./bin indicates the path address output after compilation
#--add-module Adds extension modules to the current compilation
./configure --prefix=./bin \
--add-module=$HTTP_FLV_MODULE_PATH \
--with-openssl=$OpenSSL_PATH \
--with-debug

#Build by making install
make
make install
Copy the code

If no errors are reported along the way and the bin directory we specified is printed, then success is achieved. As shown below:

5. Configure nginx. Conf

Enter vim bin/conf/nginx.conf in the current directory to configure RTMP and HTTP live protocols. I paste my configuration directly

user root; worker_processes auto; # Worker_processes Auto set to 1 on Windows because Windows does not support Unix domain socket #worker_processes Auto; Worker_cpu_affinity 0001 0010 0100 1000; worker_CPU_affinity 0001 0010 0100 1000 Worker_cpu_affinity auto for FreeBSD and Linux Error_log logs/error.log error; If this module is compiled as a dynamic module and rTMP-related work is used, the following configuration item must be specified and it must precede the events configuration item. Otherwise, NGINX startup will not load this module or load fails #load_module modules/ngx_http_flv_live_module.so; events { worker_connections 4096; } http { include mime.types; default_type application/octet-stream; keepalive_timeout 65; server { listen 80; // Customize the HTTP port location / {root/ root/nginx/nginx-http-flv-module-1.2.7/test/ WWW; index index.html index.htm; } error_page 500 502 503 504/50x.html; location = /50x.html { root html; } location /flvjsplay {// test address root/ root/nginx/flv.js-1.5.0; index index.html; } location/FLV {flv_live on; Chunked_transfer_encoding on; Add_header 'access-Control-allow-origin' '*'; # add additional HTTP headers add_header 'access-control-allow-credentials' 'true'; Add additional HTTP headers #} the location/HLS {types {application/VND. Apple. Mpegurl m3u8; video/mp2t ts; } root/root/nginx/nginx - HTTP - FLV - module - 1.2.7; add_header 'Cache-Control' 'no-cache'; } location /dash {root/ root/nginx/nginx-http-flv-module-1.2.7; add_header 'Cache-Control' 'no-cache'; } location /stat {#push/pull configuration rtmp_stat all; rtmp_stat_stylesheet stat.xsl; } location /stat. XSL {root /root/nginx-nginx-http-flv-module-1.2.7; } # If you want JSON style stat, you don't need to specify stat. XSL # but you need to specify a new configuration item rtmp_stat_format #location /stat {# rtmp_stat all; # rtmp_stat_format json; #} location /control { rtmp_control all; Rtmp_auto_push on; rtmp_auto_push_reconnect 1s; Rtmp_socket_dir/root/nginx/nginx - HTTP - FLV - module - 1.2.7; rtmp { out_queue 4096; out_cork 8; max_streams 128; timeout 30s; drop_idle_publisher 30s; log_interval 5s; Log_size 1m; log_size 1m; Server {listen 1935; // Custom RTMP port # server_name www.test.*; Application devyk {live on; gop_cache on; } application HLS {live on; hls on; Hls_path/root/nginx/nginx - HTTP - FLV - module - 1.2.7 / HLS; } application dash { live on; dash on; Dash_path/root/nginx/nginx - HTTP - FLV - module - 1.2.7 / dash; }} # Can have multiple server configurations}Copy the code

See this article for details on Nginx configuration files

Nginx-rtmp-module configuration instruction

Enter bin/sbin/nginx -t in the root directory. If the following information is displayed, the configuration is successful.

6. Start the nginx service

#Open the service
bin/sbin/nginx

#Stop the service
bin/sbin/nginx -s stop

#Restart the service
bin/sbin/nginx -s reload
Copy the code

7. Check whether all web pages are displayed normally

1. Type directly on the web page:http://ip:portIf the following information is displayed, the home page and basic configurations are correct

2. Directly enter:http://ip:port/statIf the following monitoring page is displayed, the monitoring page is normal.

Now that the server is set up, it’s time to start testing.

The RTMP push flow

We stream directly with open-source OBS-Studio, which I’ve heard a lot of game hosts use as well.

Push current source setting:

As shown in the figure above, proving that the push flow has been successful, let’s test the pull flow.

Pull flow

FLV plays on Html5

Note: Other playback is also in the following format, using Html as an example:

Example:

Suppose the LISTEN configuration item in the HTTP configuration block is:

http {
    ...
    server {
        listen 8080; Not the default port 80. location /live { flv_live on; }}}Copy the code

The LISTEN configuration items in the RTMP configuration block are:

rtmp {
    ...
    server {
        listen 1935; # may not be the default 1935 port. application myapp { live on; }}}Copy the code

And the name of the published stream is mystream, then the http-FLV based playback URL is:

http://ip:8080/flv?port=1935&app=myapp&stream=mystream

Copy the code

Player here choose Bilibili open source flv.js, since we already have a server, then directly deploy the flv.js project on the server

1. Install the NPM

#Install the NPM
yum install npm
#Check whether the installation is successful. If yes, the installation is successful
npm --version

Copy the code

2. Download flv.js directly to the server

#Download via wgetWget HTTP: / / https://github.com/bilibili/flv.js/archive/v1.5.0.tar.gz#Unpack theThe tar - ZXVF v1.5.0. Tar. GzCopy the code

3. The installation

Go to the flv.js root directory and type NPM install. After the installation, a node_modules module will appear

4. Install the Build tool

To install it in the current root directory, enter the following command:

npm install -g gulp

Copy the code

5. Wrap and minimize js into the dist folder

#Enter the following command
gulp release

Copy the code

When this step is complete, the following files are generated:

6. Modify the playback page provided by Demo

Copy the two files (.ccs,.js) in demo to dist, and modify the flv.js path in HTML, as follows:

Finally, rename the dist folder flvjsPlay

7. The deployment of

Nginx. conf configures the page loading path:

Location /flvjsplay {// test address root/ root/nginx/flv.js-1.5.0; index index.html;#flv.js test playback home page
				}

Copy the code

Nginx: nginx: nginx: nginx: nginx

#Restart the
bin/sbin/nginx -s reload
Copy the code

8. Chrome load play

The pull flow is on the left and the push flow is on the right

It can be seen that the loading speed of the first screen is still relatively fast and the delay is between 2-5s. The picture delay is a little high because of my server and network.

VLC RTMP Pull stream playback

VLC Click File -> Open network enter RTMP pull stream address and click Play

// IP :host //rtmpPost: RTMP service port // appName: RTMP application name // streamName: Push the flow of time to fill in the password RTMP: / / IP: rtmpPort/appname/streamnameCopy the code

The pull flow is on the left and the push flow is on the right

HTTP – HLS

Playback format:

http://ip:port/hls/streamname.m3u8
Copy the code

Since the author is not engaged in H5 development, I am not very clear about the compatibility of browser playing HLS live stream. I directly used the video tag to test it with Chrome browser on my computer, but the result was not able to play. Native video tags only support MP4, WebM and Ogg formats. This can be done in open source projects such as videos.js, videojs-contrib-hls, etc. At present, the test has been successfully conducted in Android browser, PC Google Browser, IOS wechat and IOS Safari browser. The FOLLOWING Html code is shown as follows:

<html>
    <head>
        <meta charset="utf-8" />
        <title>Player</title>
        <link href="https://unpkg.com/video.js/dist/video-js.css" rel="stylesheet">
    </head>
 
    <body>
        <video id="video" class="video-js vjs-default-skin" controls autoplay="autoplay" width="640" height="320" data-setup='{}'>// Switch to your own live link<source src="http://ip:8082/hls/live1.m3u8" type="application/x-mpegURL" />
        </video>
 
        <script src="https://unpkg.com/video.js/dist/video.js"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/videojs-contrib-hls/5.12.1/videojs-contrib-hls.min.js"></script>
    </body>
</html>
Copy the code

Note: the js resources used here are all supported online. If you need to use it in your project, it is best to download it locally

For testing convenience, I also directly deployed the Html code to the cloud server, nginx.conf configuration is as follows:

Location /hlsplay {root/ root/nginx/nginx-http-flv-module-1.2.7/ HLS; index hlsplay.html; // Specify the home page, that is, the page we are playing. Hlsplay.html is the code above. }Copy the code

After restarting the nginx server, directly enter http://ip:port/hlsplay to play, the test effect is as follows:

VLC RTMP, flv.js HTTP-FLV, and Android RTMP simultaneously pull flow test

conclusion

So far, you have successfully set up a live broadcast server. Although there is no code in this article, you may say that you have not learned anything, but the process of setting up a server and deploying a simple live broadcast environment is also worth learning as a mobile or front-end developer.

reference

• Introduction to Live Broadcast Protocols
• Nginx + nginx-http-flV-module constructs the documentation

Thank you

• Igor Sysoev, author of NGINX.
• Roman Arutyunyan, author of nginx-Rtmp-Module.
• Winshining, author of nginx-HTTP-FLV-Module.
• Obsproject, OBS-Studio (for real-time streaming and screen recording).
• Bilibili, HTML 5 FLV. Js