RTSP H264/HEVC stream Wasm playback

This article will introduce how RTSP H264/HEVC raw stream plays in the front end of web page. It involves WebSocket proxy sending stream data, Wasm front-end decoding and so on.

• Code: github.com/ikuokuo/rts…

Related modules:

# RTSP WebSocket Proxy
RTSP/Webcam/File > FFmpeg open > Packets > WebSocket
# WS Wasm Player
WebSocket > Packets > Wasm FFmpeg decode to YUV > WebGL display
                                                > Wasm OpenGL display
# WS Local Player
WebSocket > Packets > FFmpeg decode to YUV > OpenGL display
Copy the code

• RTSP WebSocket Proxy: stream Proxy server (C++). HTTP request flow information (cross-domain support), WebSocket transmits stream data.
• WS Wasm Player: Front-end playback implementation (ES6) WebSocket, Wasm, WebGL and other packages provide WsClient interface.
• WS Local Player: Local Player implementation (C++). As with the front-end process, data is requested from the stream proxy server, decoded and displayed in OpenGL.

Front-end effect:

Back-end stream proxy service

Main process:

# RTSP WebSocket Proxy
RTSP/Webcam/File > FFmpeg open > Packets > WebSocket
Copy the code

• Open RTSP/Webcam/File, fetch packets (common/media/stream.cc)
• H264 / HEVC raw stream packets (RTSP-WS-proxy /stream_handler.cc)
• Boost.Beast implements WebSocket service, sending raw packets to subscribing clients (RTSP-WS-proxy/WS_ *)

Front-end FFMpeg Wasm decoding requires two structures:

• AVCodecParameters: encoding parameters. Serialized to JSON, HTTP Get request (common/net/json.h)
• AVPacket: indicates the flow packet. Serialized to binary, WebSocket for transmission (common/net/packet.h)

The service supports HTTP static servers that can directly deploy WS Wasm Player pages. However, when the front and back ends are deployed separately, services are required to be allowed to cross domains (common/net/ cers.h).

Finally, the configurable items are:

log:
  # true: stderr, false: logfiles
  logtostderr: true
  alsologtostderr: false
  colorlogtostderr: true
  # LOG(), 0: INFO, 1: WARNING, 2: ERROR, 3: FATAL
  minloglevel: 0
  # VLOG(N)
  v: 0
  log_prefix: true
  log_dir: "."
  max_log_size: 8
  stop_logging_if_full_disk: true

server:
  addr: "0.0.0.0"
  port: 8080
  threads: 3

  http:
    enable: true
    doc_root: ".. /ws-wasm-player/"

  cors:
    enabled: true
    allowed_origins: "*"
    allowed_methods: [ GET ]
    allowed_headers:
      - Content-Type
    allowed_credentials: false
    exposed_headers:
      - Content-Type
    debug: false

  stream:
    http_target: "/streams"
    ws_target_prefix: "/stream/"

streams:
  -
    id: "a"
    method: "file"
    input_url: ".. /data/test.mp4"
  -
    id: "b"
    method: "network"
    input_url: "RTSP: / / 127.0.0.1:8554 / test"

    max_delay: 1000000
    rtsp_transport: "tcp"
    stimeout: 5000000
  -
    id: "c"
    method: "webcam"
    input_url: "/dev/video0"

    input_format: "v4l2"
    width: 640
    height: 480
    framerate: 20
    pixel_format: "yuyv422"

# 25 = 1000 / 40 fps
stream_get_frequency: 25

# test only: multithreading glfw not coding stable now
stream_ui_enable: false
Copy the code

Configure the RTSP streams you want to broker into Streams and run the service.

Front-end decoding and playback

Main process:

# WS Wasm Player
WebSocket > Packets > Wasm FFmpeg decode to YUV > WebGL display
                                                > Wasm OpenGL display
Copy the code

• On the front-end page, fill in the service address, refresh and select a stream, and then open (WS-WASM-player /index.html).
• WebSocket retrieves stream data, decodes it to Wasm FFmpeg, and then transcodes it to YUV420p (ws-WASM-player/SRC /decoder.h)
• WebGL displays YUV420p, or displays it in Wasm OpenGL (ws-wASM-player/SRC /player.h)

Simple measurement:

• H264 1920×1080 25fps, front-end decoding and transcoding takes 80~120 ms, there is no time to process, resulting in stuck
• H264 1280×720 25fps, front-end decoding and transcoding time 10-30 ms, can be timely processing and display

For high resolution scenarios, consider MediaSource, WebCodecs and other hard solutions.

In addition, the RTSP recommends TCP. UDP, the back-end service reported a packet loss warning, front-end decoding reported a P frame warning, easy to screen, OOM.

conclusion

In addition to RTSP streaming, WebCam/File is also supported, so you can live broadcast WebCam or rotate an MP4 File.

For now, you’ll need to compile the code README yourself. In the future, it will probably be packaged for quick play.

GoCoding personal practice experience sharing, please pay attention to the public account!