The MediaStream API is an intermediate layer that connects the WebRTC API to the underlying physical flow. WebRTC processes audio and Video through Vocie/Video Engine, and then exposes the audio and Video to the upper layer through the MediaStream API.

A MediaStream object contains zero or more MediaStreamTrack objects, representing various sound and video tracks. Each Media Reamtrack may have one or more channels. This channel represents the smallest unit of a media stream, such as an audio signal corresponding to a corresponding speaker, such as the left or right channel in a stereo track. MediaTrack consists of Source and Sink. WebRTC does not have direct access to or control over the source; all control over the source can be exercised through the track control MediaTrackConstraints.

MediaStream API MDN documentation

concept

  • MediaStreamTrack
  • MediaStream
  • source
  • sink
  • MediaTrackConstraints

1. MediaStreamTrack

MediaStreamTrack is the basic media unit in WebRTC. A MediaStreamTrack contains a single type of media (such as audio, video) returned by a media source (media device or recorded content). A single track may contain more than one channel, such as a stereo sound source which, although composed of multiple audio tracks, can also be regarded as a single track.

MediaStreamTrack MDN document

2. MediaStream

MediaStream is a collection of mediastreamtracks that can contain >=0 Mediastreamtracks. MediaStream ensures that all tracks it contains are played simultaneously and that the tracks are single.

MediaStream MDN document

3. The source and sink

In the source code of MediaTrack, MediaTrack is composed of the corresponding source and sink.

//src\pc\video_track.cc
void VideoTrack::AddOrUpdateSink(rtc::VideoSinkInterface<VideoFrame>* sink, const rtc::VideoSinkWants& wants) {
  RTC_DCHECK(worker_thread_->IsCurrent());
  VideoSourceBase::AddOrUpdateSink(sink, wants);
  rtc::VideoSinkWants modified_wants = wants;
  modified_wants.black_frames = !enabled(a); video_source_->AddOrUpdateSink(sink, modified_wants);
}
 
void VideoTrack::RemoveSink(rtc::VideoSinkInterface<VideoFrame>* sink) {
  RTC_DCHECK(worker_thread_->IsCurrent());
  VideoSourceBase::RemoveSink(sink);
  video_source_->RemoveSink(sink);
}
Copy the code

The browser has a media channel from Source to sink. Source is responsible for producing media resources, including static resources such as multimedia files and Web resources, and dynamic resources such as audio resources collected by microphones and videos collected by cameras. Sink is responsible for consuming source and producing media resources, that is, passing,, and other media labels, or the source can be transmitted to the remote end through the network through RTCPeerConnection. RTCPeerConnection can play the role of source and sink at the same time. As sink, the obtained source can reduce bit rate, zoom, adjust frame rate, etc., and then be transmitted to the remote end. As source, the obtained remote code stream is transmitted to local rendering.

4. MediaTrackConstraints

MediaTrackConstraints describes the capabilities of MediaTrack and the one or more values that each capability can take to select and control sources. MediaTrackConstraints can be passed as a parameter to applyConstraints() to control the track properties, and you can call getConstraints() to see how recently the custom constraints have been applied.

const constraints = { width: {min: 640, ideal: 1280}, height: {min: 480, ideal: 720}, advanced: [ {width: 1920, height: }, {aspectRatio: 1.333}]}; //{video: true} is also a MediaTrackConstraints object that specifies the media type of the request and the corresponding parameters. navigator.mediaDevices.getUserMedia({ video: true }) .then(mediaStream => { const track = mediaStream.getVideoTracks()[0]; track.applyConstraints(constraints) .then(() => { // Do something with the track such as using the Image Capture API. })  .catch(e => { // The constraints could not be satisfied by the available devices. }); });Copy the code

More different constraints:

{audio: true, video: {width: 1280, height: 720}}Copy the code
{audio: true, video: {facingMode: "user"}}Copy the code
{audio: true, video: {facingMode: {exact: "environment"}}}Copy the code

How do I play MediaStream

The MediaStream object can be assigned directly to the srcObject property of the HTMLMediaElement interface.

video.srcObject = stream;
Copy the code

SrcObject MDN document

How do I get MediaStream

  1. Local device

Can by calling MediaDevices. GetUserMedia () to access the local media, call this method after the browser will prompt the user for permission to use the media input media input will produce a MediaStream, containing the request of the media type of orbit. This stream can contain A video track (from hardware or virtual video sources, such as cameras, video capture devices, screen sharing services, and so on), an audio track (also from hardware or virtual audio sources, such as microphones, A/D converters, and so on), or some other track type.

navigator.mediaDevices.getUserMedia(constraints)
  .then(function(stream) {
    /* 使用这个stream*/
    video.srcObject = stream;
  })
  .catch(function(err) {
    /* Handle error */
  });
Copy the code

Through MediaDevices. EnumerateDevices () we can get a native of the available media list of input and output devices, such as a microphone, camera, headphones equipment, etc.

// Get the media device
navigator.mediaDevices.enumerateDevices().then(res= > {
	console.log(res);
});
Copy the code

Each media input in the list can be used as a value of the corresponding type in MediaTrackConstraints, such as an audio device input audioDeviceInput that can be set to the value of the audio property in MediaTrackConstraints

cosnt constraints = { audio : audioDeviceInput }
Copy the code

The constraint value passed as a parameter to MediaDevices. The getUserMedia (constraints), can obtain the device MediaStream.

MediaDevices. EnumerateDevices (MDN) document MediaDevices. GetUserMedia (MDN) document

  1. Capture screen

Use MediaDevices. GetDisplayMedia () method, which can prompt the user to select and authorization to capture display content, or part of the content (e.g., a window), and the recorded content in a MediaStream.

MediaDevices. GetDisplayMedia (MDN) document

  1. HTMLCanvasElement.captureStream()

Using HTMLCanvasElement. CaptureStream () method returns the CanvasCaptureMediaStream is the flow of a real-time capture the canvas of animation.

//frameRate sets the double precision floating point value to the capture rate per frame.
// If not set, a new frame is captured each time the canvas changes.
// If set to 0, a single frame is captured.
cosnt canvasStream = canvas.captureStream(frameRate);
video.srcObject = canvasSream;
Copy the code

HTMLCanvasElement. CaptureStream (MDN) document CanvasCaptureMediaStream MDN document

  1. RTCPeerConnection

  2. Get it from another MediaStream

The constructor MediaStream() returns a newly created, empty MediaStream instance

newStream = new MediaStream();
Copy the code
  • The MediaStream object is passed in, and the data track for that MediaStream object is automatically added to the newly created stream. And these data tracks are not removed from the original stream, that is, become data shared by the two streams.
newStream = new MediaStream(otherStream);
Copy the code
  • The member of Array type passed to the MediaStreamTrack object, representing each data track added to the stream.
newStream = new MediaStream(tracks[]);
Copy the code
  • MediaStream.addTrack()Method adds a new track to the stream.
  • MediaStream.clone()Method to make a copyMediaStream. This newMediaStreamObject has a new oneid.

Achieve a simple recording screen tools

  • Get the MeidaStream for the capture screen
const screenStream = await navigator.mediaDevices.getDisplayMedia();
Copy the code
  • Get local audio and video data
const cameraStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: false });
const audioStream = await navigator.mediaDevices.getUserMedia({ video: false, audio: true });
Copy the code
  • Draw the screen picture and camera picture on canvas
function createStreamVideo(stream) {
	const video = document.createElement("video");
	video.srcObject = stream;
	video.autoplay = true;

	return video;
}

const cameraVideo = createStreamVideo(cameraStream);
const screenVideo = createStreamVideo(screenStream);
Copy the code
animationFrameHandler() {
      if (screenVideo) {
          canvas.drawImage(screenVideo, 0, 0, canvasWidth, canvasHeight);
      }

      if (cameraVideo) {
          canvas.drawImage(
              cameraVideo,
              canvasWidth - CAMERA_VIDEO_WIDTH,
              canvasHeight - CAMERA_VIDEO_HEIGHT,
              CAMERA_VIDEO_WIDTH,
              CAMERA_VIDEO_HEIGHT
          )
      }

      requestAnimationFrame(animationFrameHandler.bind(this));
  }
Copy the code
  • Get the MediaStream of the Canvas
const recorderVideoStream = await canvas.captureStream();
Copy the code
  • Merge the local audio track with the canvasStream video track to get the final picture of the MediaStream
const stream = new MediaStream();
audioStream.getAudioTracks().forEach(track => stream.addTrack(track));
recorderVideoStream.getVideoTracks().forEach(track => stream.addTrack(track));

video.srcObject = stream;
Copy the code
  • throughMediaRecorderRecord the picture
const recorder = new MediaRecorder(stream, { mineType: "video/webm; codecs=h264" }); recorder.ondataavailable = e => { recorderVideo.src = URL.createObjectURL(e.data); }; // Start recording recorder.start(); // Stop recording recorder.stop();Copy the code

The MediaRecorder MDN document