Use the information

In the second half of 2017, wechat version 6.5.21 supported online audio and video functions. Developers can live broadcast, video call, voice call and other functions through two audio and video components.

The above functionality requires two of the two small program media components: live-pusher and live-player. The following sections briefly describe the basics of two of these components. Can only take you entry, if the master please detour.

Register applet

To register a small program, please click the wechat public platform. After registration, record the AppID of the small program in [Development] > [Basic Configuration] on the small program management page for later use.

Note: You must register as a non-personal principal type, otherwise you cannot open and these two tags.

The label permission is enabled

Live-pusher and live-player are feature labels within small programs that support audio and video uplink capabilities

For policy and compliance reasons, wechat is not releasing support for live-pusher and live-player tags for all small programs:

• The categories that currently support these two tags are shown in the table below (only non-personal subjects have the following categories) :

The main categories	categories	Applet content scenario
social	live	It involves entertainment nature, such as star live broadcast, life fun live broadcast, pet live broadcast, etc. After selecting this category, the first submission for code review shall be reviewed and confirmed by the local Internet authority, which is expected to take about 7 days
education	Online video courses	Online classes, online training, lectures and other live education
medical	Internet hospital, public hospital	Consultations, large health lectures and other live
financial	Banking, trust, fund, securities/futures, securities and futures investment consulting, insurance, credit investigation, New Third Board information service platform, Stock information service platform (Hong Kong/US stocks), consumer finance	Financial product video customer service claim settlement, financial product promotion live broadcast, etc
The car	Pre-sale service	Car presale, promotion live
Government principal account	–	Government related work to promote live broadcasting, leadership speech live, etc
tool	Video call	One-to-one video customer service that does not involve the above contents, such as one-to-one video service after enterprise sales

• Small programs that meet the requirements of the category need to open the permission to use the push and pull stream label in the [development] > [interface setting] of the small program management background

[Img/vni239TU-1586502758626] (/Users/ MXJ/Baidu Cloud synchronization disk/study notes/front-end full stack/wechat mini program /img/image-2020040907350] /img/image-2020040907350 6475.png)]

Note: If the above Settings are correct, but the small program still does not work properly, it may be that the cache inside wechat is not updated, please delete the small program and restart wechat, and then try again.

live-player

Live-player is a feature label within the applet that supports the downlink (play) capability of audio and video.

Versioning support

• Minimum requirement of wechat App iOS version: 6.5.21
• Minimum version requirement of wechat App for Android: 6.5.19.
• Minimum version requirement of small program base library: 1.7.0.

The current base library version information can be obtained by wx.getSystemInfo.

Attribute definitions

The property name	type	The default value	instructions
src	String	–	Playback URL for audio and video downlink. Supports protocols such as RTMP and FLV
mode	String	live	Mode. Live, RTC, lower latency
autoplay	Boolean	false	Auto Play or not
muted	Boolean	false	If quiet
orientation	String	vertical	Vertical, horizontal
object-fit	String	contain	Fill mode, optional values areThe long side of the image will fill the screen and the short side will be filled with black.FillCrop (images fill the screen, and parts outside the display area will be cut off)
background-mute	Boolean	false	When wechat cuts to the background, whether to turn off the sound playing
min-cache	Number	1	Minimum buffer delay, in seconds
max-cache	Number	3	Maximum buffer delay, in seconds
sound-mode	String	speaker	Voice output mode. A speaker B ear C speaker D speaker
auto-pause-if-navigate	Boolean	true	When jumping to other pages of this small program, whether to automatically pause the real-time audio and video playback of this page
auto-pause-if-open-native	Boolean	true	When jumping to other wechat native pages, whether to automatically pause the real-time audio and video playback on this page
picture-in-picture-mode	string/Array	–	Set small window modes: push, pop, or multiple modes via arrays (e.g., [“push”, “pop”]) (uncommon)
bindstatechange	EventHandler	–	Use to specify a javascript function to accept player events. Detail = {code}
bindfullscreenchange	EventHandler	–	Use to specify a javascript function that accepts full-screen change events, detail = {direction, fullScreen}
bindnetstatus	EventHandler	–	Network status notification, detail = {info}
bindaudiovolumenotify	EventHandler	–	Play volume or volume notifications, detail = {} (uncommon)
debug	Boolean	false	Whether to enable the debugging mode

The sample code

<view id='video-box'>  
    <live-player
        wx:for="{{player}}"
        id="player_{{index}}"
        mode="RTC"
        object-fit="fillCrop"
        src="{{item.playUrl}}" 
        autoplay='true'
        bindstatechange="onPlay">
   </live-player>
 </view> 
Copy the code

Ultra low delay

The RTC mode supports ultra-low latency links of less than 500ms and can be used in scenarios such as video calls and remote control. To use ultra-low latency playback, pay attention to the following points: (1) If the streaming terminal is a wechat applet, use the RTC mode. (2) If the streaming end is iOS or Android SDK, please use setVideoQuality MAIN_PUBLISHER mode. (3) If the push stream terminal is Windows, please do not use OBS, the delay is too high, can use our Windows SDK. (4) Do not set min-cache and max-cache by yourself. Use the default values. (5) Play address please use ultra-low delay play address, that is, with anti-linked-theft signature RTMP :// address, as follows:

Compare the project	The sample	Time delay
Common Live URL	rtmp://3891.liveplay.myqcloud.com/live/3891_test_clock_for_rtmpacc	>2s
Ultra-low latency URL	rtmp://3891.liveplay.myqcloud.com/live/3891_test_clock_for_rtmpacc?bizid=bizid&txTime=5FD4431C&txSerect=20e6d865f462dff6 1ada209d53c71cf9	< 500ms

Properties,

SRC Is used to play the URL of audio and video downlink. It supports RTMP (URL starts with RTMP ://) and FLV (URL starts with http:// and ends with.flv).

Description:

The tag does not support the HLS (M3U8) protocol, because the HLS (M3U8) playback protocol is supported. However, HLS (M3U8) protocol is not recommended for live viewing, and the delay is an order of magnitude higher than RTMP and FLV protocols.

Mode Live mode is mainly used for live broadcast scenarios, such as event live broadcast, online education, and distance training. In this mode, the modules inside the small program give priority to ensuring a smooth viewing experience. By adjusting the min-cache and max-cache properties, you can adjust the time delay experienced by the audience (playing) side. These two parameters are described in detail below.

RTC is mainly used for two-way video calls or multi-party video calls, such as financial meetings, online customer service, auto insurance damage determination, and training meetings. In this mode, the Settings for min-cache and max-cache do not work because the applet automatically controls the latency to a very low level (around 500ms).

Min – cache and Max – cache

These two parameters are used to specify the minimum and maximum buffer times on the viewing side, respectively. Buffer time, refers to the player in order to alleviate the impact of the network fluctuation on watch fluency and introducing a “reservoir”, when packets from the network card, or even stagnation, “reservoir” in the emergency water can let the players can still insist on for a short period of time, as long as the Internet connection in the short time back to normal, The player can render a smooth and smooth video continuously.

The more water in the “reservoir”, the better the ability to resist network fluctuations, but at the expense of the larger latency of the audience, so in different scenarios, use different configurations to achieve a balance of experience:

• For a live stream with a low bit rate (about 1Mbps, and the screen is mainly people), min-cache = 1 and max-cache = 3 are suitable.
• For a live stream with a high bit rate (2Mbps – 3Mbps HD game screen), min-cache = 3 and max-cache = 5 are appropriate.

These two parameters are invalid in RTC mode.

Background-mute Whether to continue playing sound after wechat is switched to the background. It is used to avoid the impact of screen lock on the video content being played by the current mini program.

Debug Enables debugging of audio and video related functions, which would be a nightmare without a good tool. Therefore, the small program supports debug mode for the live-pusher tag. When the debug mode is started, a semi-transparent log window is displayed on the window originally used to render the video. It is used to display various audio and video indicators and events, and reduce the difficulty of debugging related functions. The specific usage method is explained in detail in the FAQ.

The object operation

Wx. CreateLivePlayerContext () by wx. CreateLivePlayerContext () tag and javascript objects can be associated, immediately after the operation.

Play starts playing, and if the autoPlay property is set to false (the default), then play can be used to manually start playing.

Stop Stops playing.

Pause To pause the play.

Resume continues playing, used in conjunction with pause.

Mute Mute.

RequestFullScreen goes to full screen.

ExitFullScreen Exits the full screen.

The code examples

var player = wx.createLivePlayerContext('pusher');
player.requestFullScreen({
    success: function(){
            console.log('enter full screen mode success! ')}fail: function(){
            console.log('enter full screen mode failed! ')}complete: function(){
            console.log('enter full screen mode complete! ')}});Copy the code

Internal events

The bindStatechange property of the live-player tag allows you to bind an event handler that listens for internal events and exception notifications in the push-stream module.

The key events

code	An event definition	Meaning that
2001	PLAY_EVT_CONNECT_SUCC	The cloud server has been connected
2002	PLAY_EVT_RTMP_STREAM_BEGIN	The server starts to transmit audio and video data
2003	PLAY_EVT_RCV_FIRST_I_FRAME	The network has received the first audio and video data
2004	PLAY_EVT_PLAY_BEGIN	The video starts playing, and you can use the default image to represent the wait state before receiving the event
2006	PLAY_EVT_PLAY_END	End of video
2007	PLAY_EVT_PLAY_LOADING	Enters the buffered state where the player is waiting or storing data from the server
– 2301.	PLAY_ERR_NET_DISCONNECT	The network connection is disconnected and cannot be restored when reconnected. The player has stopped playing

For more details, see the official document

Description:

The PLAY_EVT_PLAY_END event will not be thrown if the audience sees the live stream break during the FLV address, because FLV does not define a stop event. You can only listen to PLAY_ERR_NET_DISCONNECT instead.

Warning events

Internal warning is not an unrecoverable error. The audio and video SDK inside the small program will start corresponding recovery measures. The purpose of warning is mainly to prompt developers or end users, such as:

code	An event definition	Meaning that
2101	PLAY_WARNING_VIDEO_DECODE_FAIL	Failed to decode the current video frame
2102	PLAY_WARNING_AUDIO_DECODE_FAIL	Current audio frame decoding failed
2103	PLAY_WARNING_RECONNECT	Network disconnection, automatic reconnection recovery enabled (PLAY_ERR_NET_DISCONNECT is thrown directly after three reconnections)
2104	PLAY_WARNING_RECV_DATA_LAG	The video stream is erratic, possibly because the viewer’s current Internet speed is not sufficient
2105	PLAY_WARNING_VIDEO_PLAY_LAG	The current video is playing slowly
2106	PLAY_WARNING_HW_ACCELERATION_FAIL	The hard solution fails to start. The soft solution is used
2107	PLAY_WARNING_VIDEO_DISCONTINUITY	The current video frame is not continuous. The video source may lose frames, which may lead to a splintered screen
3001	PLAY_WARNING_DNS_FAIL	DNS resolution failed (throwing when only RTMP:// address is played)
3002	PLAY_WARNING_SEVER_CONN_FAIL	Server connection failed (throw when playing RTMP:// address only)
3003	PLAY_WARNING_SHAKE_FAIL	Server handshake failed (throw when only RTMP:// address is played)

The code examples

Page({
    onPlay: function(ret) {
        if(ret.detail.code == 2004) {
                console.log('Start video play',ret); }},/** * life cycle function -- listen for page loading */
    onLoad: function (options) {
    / /...}})Copy the code

Network status data

Key name	instructions
videoBitrate	The bit rate output by the current video encoder/coder, in KBPS
audioBitrate	The bit rate output by the current audio encoder/coder, in KBPS
videoFPS	Current video frame rate
videoGOP	The GOP of the current video is the interval time of every two key frames (I frames), in s
netSpeed	Current send/receive speed
netJitter	Network jitter: 0 indicates that there is no jitter. A larger value indicates that the network jitter is greater and the network is more unstable
videoWidth	The width of a video frame
videoHeight	The height of a video frame

Small window features

The live-Player widget supports the following three trigger modes (set the picture-in-picture-mode property on the component) :

1. Push mode, i.e. a small window appears when jumping from the current page to the next page (page stack push)
2. Pop mode, which is triggered when leaving the page (page stack POP)
3. The above two routing behaviors trigger small Windows

In addition, small Windows support the following features:

• The size of small window container will be judged automatically according to the original component size
• Clicking on the small window will take the user back to the player page corresponding to the small window
• Small window, the user can click on the small window in the top right corner of the close button or call context. ExitPictureInPicture closed small window () interface

Special instructions

1. A component is a native component that is created by the client and is at the highest level. You cannot control the level through z-index. Can be used and overlaid on top.

2. Do not use components in…

3. The default width of live-Player is 300px and the height is 225px. You can set the width and height by WXSS.

4. Currently not supported in developer tools.

5. CSS animations are not valid for live-pusher components.

Q&A

Development and runtime environment requirements

• Minimum requirement of wechat App iOS version: 6.5.21
• Minimum version requirement of wechat App for Android: 6.5.19.
• Minimum version requirement of small program base library: 1.7.0.
• Since the wechat developer tool does not support native components (i.e., tags), it needs to be run on a real machine.

Why should I enable debug mode when debugging?

Enable debugging to skip the work of adding these domain names to the small program whitelist, otherwise you may encounter login failure, call can not connect to the problem.

live-pusher

Attribute definitions

The property name	type	The default value	instructions
url	String	–	Push-stream URL for audio and video uplink
mode	String	RTC	SD, HD, FHD, RTC
autopush	Boolean	false	Whether to automatically start push flow
muted	Boolean	false	If quiet
enable-camera	Boolean	true	Turn on/off camera
auto-focus	Boolean	true	Manual autofocus
orientation	String	vertical	Vertical and horizontal
beauty	Number	0	The value of beauty index ranges from 0 to 9, the larger the value, the more obvious the effect
whiteness	Number	0	Whitening index, the value is 0-9, the larger the value, the more obvious the effect
aspect	String	For if	3:4, 9:16
zoom	Boolean	false	Whether the focal length is normal. True: zoom in on the camera
device-position	String	front	Front camera, back camera
min-bitrate	Number	200	Minimum bit rate, which determines the worst sharpness of the picture
max-bitrate	Number	1000	The maximum bit rate, which determines the best sharpness of the picture
audio-quality	String	low	Low is suitable for voice calls, and high indicates high voice quality
waiting-image	String	–	When wechat cut to the background of the shim picture
waiting-image-hash	String	–	Check value of the shim picture when wechat cuts to the background
background-mute	Boolean	false	Whether to disable sound collection when wechat switches to background
bindstatechange	String	–	Use to specify a javascript function to receive audio and video events
debug	Boolean	false	Whether to enable the debugging mode

The sample code

<view id='video-box'>  
    <live-pusher
          id="pusher"
          mode="RTC"
          url="{{pusher.push_url}}" 
          autopush='true'
          bindstatechange="onPush">
    </live-pusher>  
 </view> 
Copy the code

Properties,

• Url is used for the upstream push url of audio and video. It starts with RTMP :// protocol prefix. For the obtaining method of Tencent Cloud push URL, see Quick Get URL Document.

  Description:

  The RTMP protocol used in small programs supports UDP acceleration. Under the same network conditions, the UDP version of RTMP has better upstream speed and anti-jitter capability than the open source version.

• Mode SD, HD, and FHD are mainly used for livestreaming scenarios, such as event livestreaming, online education, and distance training. SD, HD, and FHD correspond to three default definitions, respectively. In this mode, the applet pays more attention to clarity and smoothness, and doesn’t overemphasize low latency or sacrifice quality and smoothness for delay.

  The RTC is used for two-way video calls or multi-party video calls, such as financial meetings, online customer service, auto insurance damage determination, and training meetings. In this mode, the small program will pay more attention to reducing point-to-point delay, and will give priority to ensuring sound quality. When necessary, the clarity and smoothness of the picture will be reduced.

• Orientation and Aspect Horizontal mode is still vertical mode, which defaults to portrait mode with the home button down. At this time, the small program launched the screen aspect ratio is 3:4 or 9:16 these two vertical screen aspect ratio screen, that is, width < height. If you change the landscape mode, the small program launched the screen aspect ratio is changed to 4:3 or 16:9 this landscape screen aspect ratio screen, that is, width > height.

  The aspect ratio is determined by the aspect, which defaults to 9:16, but can also support 3:4. This is if the orientation attribute is vertical. If the orientation attribute is horizontal, then the effect of 3:4 is equivalent to 4:3 and the effect of 9:16 is equivalent to 16:9.

• Here we first want to popularize a concept – video bitrate, refers to the video encoder output per second of the video data. When the video resolution is determined, the higher the video bit rate, that is, the more data output per second, the better the corresponding picture quality.

  Therefore, the two attributes of min-bitrate and max-bitrate are used to determine the lowest and highest sharpness of the output picture respectively. These two numbers are not bigger is better, because the user’s network uplink is not infinite. But smaller is not better, because clarity is an important indicator of user experience in practical applications. The specific value setting will be described in ** “Parameter setting” ** section.

  The small program will automatically handle the relationship between resolution and bit rate. For example, if the bit rate is 2Mbps, the small program will choose the resolution of 720p for matching, and if the bit rate is 300kbps, the small program will choose a lower resolution to improve the coding efficiency. So you only need to pay attention to the min-bitrate and max-bitrate to control the picture quality.

• Waiting -image and Waiting -image-hash For the consideration of user privacy, after wechat is cut to the background, the small program wants to stop the camera’s picture collection. But for the other end, the screen goes black or freezes (stays at the last frame), which is a very bad experience. To solve this problem, we introduced the wait-image property, where you can set an image with the meaning “wait” (wait-image is the URL of the image, and wait-image-hash is the MD5 value of the image). When wechat is switched to the background, the small program will use the picture as a substitute for the camera screen, maintaining the video stream for 3 minutes with very low traffic.

• Debug Enables debugging of audio and video related functions, which would be a nightmare without a good tool. Therefore, the small program supports debug mode for the live-pusher tag. When the debug mode is started, a semi-transparent log window is displayed on the window originally used to render the video. It is used to display various audio and video indicators and events, and reduce the difficulty of debugging related functions. The specific usage method is explained in detail in the FAQ.

Parameter Settings

scenario	mode	min-bitrate	max-bitrate	audio-quality	instructions
The qing live	SD	300kbps	800kbps	high	Narrow band scenarios, such as outdoor or network instability
Hd broadcast	HD	600kbps	1200kbps	high	At present, the parameter setting adopted by mainstream apps is recommended for ordinary live broadcast scenarios
Super clear live	FHD	600kbps	1800kbps	high	For scenes with more demanding definition requirements, ordinary mobile phones can be used to watch HD
Video customer service (user)	RTC	200kbps	500kbps	high	This is a scene where the sound is dominant and the picture is auxiliary, so don’t set it too high
Vehicle Insurance Damage (owner)	RTC	200kbps	1200kbps	high	The upper limit will be set higher as you may want to see more details
Multi-party Meeting (speaker)	RTC	200kbps	1000kbps	high	The speaker’s picture quality can be appropriately high, and the participation quality can be set low
Multi-party Meeting (Participation)	RTC	150kbps	300kbps	low	As a conference participant, you don’t need to have high picture and sound quality

If you are not particularly confident about the bandwidth, do not select low for the audio quality option. The sound quality and latency will be much worse than high mode.

The object operation

• Wx. CreateLivePusherContext () by wx. CreateLivePusherContext () tag and javascript objects can be associated, immediately after the operation.
• Start starts the push. If the autopush property is set to false (the default), start can be used to manually start the push.
• Stop Stops the push flow.
• Pause Suspends the push stream.
• Resume Resumes the push stream. Pair it with the pause operation.
• SwitchCamera Switches the front and rear cameras.
• snapshotThe size of the screenshot is the same as that of the component. The temporary path of the successful screenshot isret.tempImagePath.

The code examples

var pusher = wx.createLivePusherContext('pusher'); pusher.start({ success: function(ret){ console.log('start push success! ') } fail: function(){ console.log('start push failed! ') } complete: function(){ console.log('start push complete! ')}});Copy the code

Internal events

The bindStatechange attribute of the tag binds to an event handler that listens for internal events and exception notifications in the push-stream module.

Regular event

code	An event definition	Meaning that
1001	PUSH_EVT_CONNECT_SUCC	The cloud server has been successfully connected
1002	PUSH_EVT_PUSH_BEGIN	Handshake with the server is complete, all is well and ready to start the upward push
1003	PUSH_EVT_OPEN_CAMERA_SUCC	The camera has been started, but cannot be opened when the camera is occupied or the permission is restricted

Serious mistakes

code	An event definition	Meaning that
– 1301.	PUSH_ERR_OPEN_CAMERA_FAIL	Failed to open the camera
– 1302.	PUSH_ERR_OPEN_MIC_FAIL	Failed to turn on the microphone
– 1303.	PUSH_ERR_VIDEO_ENCODE_FAIL	Video encoding failure
– 1304.	PUSH_ERR_AUDIO_ENCODE_FAIL	Audio coding failure
– 1305.	PUSH_ERR_UNSUPPORTED_RESOLUTION	Unsupported video resolution
– 1306.	PUSH_ERR_UNSUPPORTED_SAMPLERATE	Unsupported audio sampling rate
– 1307.	PUSH_ERR_NET_DISCONNECT	The network is disconnected and invalid after being reconnected for three times. You can give up and try again for more timesRestart the push flow

Warning events

Internal warning is not an unrecoverable error. The audio and video SDK inside the small program will start corresponding recovery measures. The purpose of warning is mainly to prompt developers or end users, such as:

• PUSH_WARNING_NET_BUSY The upstream network speed is poor. You are advised to prompt the user to improve the current network environment, for example, to get the user closer to the home router or switch to Wi-Fi before using the network.
• The PUSH_WARNING_SERVER_DISCONNECT request was rejected by the background. This is usually because the txSecret in the URL is miscalculated, or the URL is occupied by someone else (unlike playing, a push_stream URL can only be used by one user at a time).
• PUSH_WARNING_HANDUP_STOP When the user clicks the circle in the upper right corner of the applet or the back button, wechat will suspend the applet and the 5000 event will be thrown.

code	An event definition	Meaning that
1101	PUSH_WARNING_NET_BUSY	The upstream network speed is insufficient. You are advised to improve the current network environment
1102	PUSH_WARNING_RECONNECT	The network is disconnected and reconnection process has started (retry failure more than three times will be abandoned)
1103	PUSH_WARNING_HW_ACCELERATION_FAIL	Hard code startup failure, automatically switch to soft code
1107	PUSH_WARNING_SWITCH_SWENC	Due to machine performance problems, the hardware coding is automatically switched
3001	PUSH_WARNING_DNS_FAIL	The DNS resolution fails and the retry process is started
3002	PUSH_WARNING_SEVER_CONN_FAIL	The server connection fails, and the retry process is started
3003	PUSH_WARNING_SHAKE_FAIL	The server handshake fails, and the retry process is started
3004	PUSH_WARNING_SERVER_DISCONNECT	The server disconnects and starts the retry process
3005	PUSH_WARNING_SERVER_DISCONNECT	The socket link is abnormally disconnected, and the retry process is started
5000	PUSH_WARNING_HANDUP_STOP	Applet suspended by user

The code examples

Page({
    onPush: function(ret) {
        if(ret.detail.code == 1002) {
                console.log('Push flow worked',ret); }},/** * life cycle function -- listen for page loading */
    onLoad: function (options) {
    / /...}})Copy the code