This article first appeared on my blog: Audio and Video Mini Program Development Notes

Ps: Github. IO occasionally hangs, if not, please temporarily hang the agent, later will have to fix the inner loop 🐕.

preface

Cloud computer, cloud office, cloud meeting, cloud games and other cloud concepts, in the extraordinary 2020, still one after another, you sing me! What do they seem to be telling us: The 5G era won’t be ordinary? Or is it dormant for the application of a technology that could disrupt an industry?

Some time ago to contact the cloud game applet, record one or two, only for sharing.

Cloud games based on cloud computing, games run on the server side, no longer require users to have a high configuration of the client (graphics card and processor), through video transmission, control virtual remote desktop (VNC) to achieve the interaction on the game. Of course, this has a high requirement for network stability and low latency, which is why in the future, 5G is widely used in commercial conditions, the reason for higher expectations.

background

Q: Cloud gaming can be based on browser H5, but can it be based on applets?

  • In this article, wechat small program as an example, let us see the possibility of its realization and the key points pointed north.

First you need to know how to run through Demo, cloud game mini program implementation scheme based on TRTC two-person call.

TRTC interworking with the custom component is through the < trTC-room > tag, it is based on wechat small program native and two media components.

The main use here is live-player, which is used to play video streams of cloud games.

Architecture diagram

photo

As can be seen from the above picture: Wechat small program to achieve audio and video must be transcoded through Tencent video cloud (billing QAQ).

The point that

  1. Development of small program audio and video needs to pass category audit first. Categories range

  2. Because it needs to go through the agent of Tencent Cloud, it is necessary to register Tencent Cloud, apply for SDKAPPID and SECRETKEY, and make corresponding configurations. The front and back ends need to be unified.

Development of guidance

The code directory

The implementation of this scheme is based on small program native, without using the framework (such as UniAPP, Taro, etc.).

Native specification specific can see: wechat official documents – small programs

Initialize directory:

Development steps

  1. Configure SDKAPPID and SECRETKEY in GenerateTestUserSig. Js, which must be unified with the background.
  2. Open the video room in the background, and get the room number in the front end;
  3. Open the project in wechat developer tool, enable real machine debugging, scan code to test whether the video stream is pushed successfully;
  4. The front end acquires user click and slide operations;
  5. Websocket docking, byte stream encoding and decoding;
  6. Process improvement;

Encoding and decoding

You need to use base64 for the relay, and the encoding type is ASCII.

  • Receiving data (decoding) : ArrayBuffer is translated into String
  1. First convert Buffer to base64: WX. arrayBufferToBase64
  2. Then convert base64 to String: wechat applet base64 encoding and decoding and UTF-8 decoding
  • Send data (encoding) : String to ArrayBuffer
  1. First convert String to base 64;
  2. Then call wx.base64ToArrayBuffer to convert to ArrayBuffer

This partial method is encapsulated in the trtc-room-socket.js file

Real machine commissioning

Video streams cannot be seen on developer tools and must be enabled for debugging. Real machine debugging can see real-time view print information, view delay information;

Afterword.

summary

Based on the micro channel small program native label <live-player> can realize the cloud game video push stream, micro channel small program needs to do a layer of transcoding service through Tencent Cloud, the frame diagram has been explained. The implementation of the control flow needs to be encapsulated into VNC, such as obtaining the click and slide operation, and then transcoding ArrayBuffer to the background for the actual game control, server processing and then push the processed video stream.

Key and difficult points:

  1. WSS heartbeat, reconnection mechanism;
  2. Video screen rotation and zooming;
  3. Byte stream transmission;
  4. Breakpoint continuation, etc.;

This question is worth studying things are really quite a lot, a superficial taste, look forward to the application of 5G. TIME!

Document finishing

  • Run demo in one minute
  • Rapid integration component
  • Quick adjustment of basic functions
  • trtc-room api
  • Micro channel small program WebSocket heartbeat reconnection
  • On BASE64 encoding
  • A few things about Base64 in JS
  • Wechat small program ASCII to character
  • Base64 – Uses ASCII to encode binary data
  • Base64 notes

Error collecting

  • Exit room timer not cleared
  • SendSocketMessage: Fail taskID not exist (key)
  • Small program using BTOA, ATOB methods