Demo source: github.com/ericlee33/h… If you find it helpful, please go to star👏

background

Since H5 is a system camera that takes photos by

Preparatory work

Current compatibility of all ends

Bottom line: android: chrome53 supports the API later. Ios: only supported by safari11+. Ios wechat built-in browser, Chrome, Edge and other browsers are not supported.

Consider alternatives

In the following cases, alternative schemes should be considered: Category 1: When the following conditions are met, scheme 1 with system camera should be adopted. The user does not provide the camera permission. 2. Any of the following errors are matched

  • AbortError[Abort error]
  • NotAllowedError[Reject error]
  • NotFoundError[No error found]
  • NotReadableError[Unable to read error]
  • OverConstrainedError[Unable to meet requirements error]
  • SecurityError[Security error]
  • TypeError[Type error]

3. User browsers do not support this API ** When ios users use non-Safari browsers to access the H5 page, since only Safari 11+ in ios can suspend the video stream of the rear camera, if ios users open the H5 landing page in non-Safari browsers, they should directly guide the user to copy the link to the Safari browser to open it. Avoid being unable to take a custom photo later. Here cattle do better, can imitate cattle do a guide button.

Positive start

This article does not cover the compatibility logic of the alternative, and can be used on its ownPromise.reject()Is processed accordingly.

Our protagonist is MediaDevices getUserMedia (), MDN introduction to the API is as follows

MediaDevices. GetUserMedia () will prompt the user for permission to use the media input media input will produce a MediaStream, containing the request of the orbit of media types. This stream can contain A video track (from hardware or virtual video sources, such as cameras, video capture devices, screen sharing services, and so on), an audio track (also from hardware or virtual audio sources, such as microphones, A/D converters, and so on), or some other track type.

Ability to detect

Different browsers implement different standards. Therefore, API capabilities must be compatible to prevent user browsers from invoking the API.

// A compatible method to access the user's media device
function getUserMedia(constrains) {
    if(navigator.mediaDevices? .getUserMedia) {// The latest standard API
        return navigator.mediaDevices.getUserMedia(constrains);
    } else if (navigator.webkitGetUserMedia) {
        // WebKit kernel browser
        return navigator.webkitGetUserMedia(constrains);
    } else if (navigator.mozGetUserMedia) {
        / / the Firefox browser
        return navigator.mozGetUserMedia(constrains);
    } else if (navigator.getUserMedia) {
        / / the old API
        returnnavigator.getUserMedia(constrains); }}Copy the code

Place a video element on the page

<video
   id="video"
   autoPlay
   muted
   playsInline
   style={{
    width: '100%',
  }}
></video>
Copy the code

There are a few caveats ⚠️

IOS 10 Safari allows you to automatically play two kinds of videos:

  • Trackless video;
  • Silent audio and video (setmutedAttribute);

For these two types of videos, you can use

  • Only providemutedProperty to mute the video<video autoplay> 或 video.play()There are two ways to play
  • You must provideplaysInlineProperty, otherwise only one frame will be played on ios

Call wrappedGetUserMedia to get the user’s media stream

Constrains, when called, can be given a number of different values to retrieve the various media streams underlying the user device.

  • video: true (Front-facing camera is used by default.)
  • In order to access the rear camera, we need to go throughfacingMode: { exact: 'environment' }Call ** (If the rear camera does not exist, it will fail to get the media stream
  • To get a video stream at a particular resolution, we can specify the correspondingwidth height(However, this method has defects, once the user device does not exist for the pixel stream, it will lead to the failure to obtain the media stream, so we do not customize the pixel, use the automatically obtained media stream pixel)
/** * This function needs to take a video DOM node as an argument */
function getUserMediaStream(videoNode) {
  	/** * Call the API callback successfully */
    function success(stream, video) {
        return new Promise((resolve, reject) = > {
            video.srcObject = stream;

            video.onloadedmetadata = function () {
                video.play();
                resolve();
            };
        });
    }
  
    // Call the user media device to access the camera
    return getUserMedia({
        audio: false.video: { facingMode: { exact: 'environment'}},// video: true,
        // video: { facingMode: { exact: 'environment', width: 1280, height: 720 } },
    })
        .then(res= > {
            return success(res, videoNode);
        })
        .catch(error= > {
            console.log('Failed to access user media device:', error.name, error.message);
            return Promise.reject();
        });
}

Copy the code

Current Effect:

Add crop boxes and external shadows

  • The crop box is written to the page as required and will be passed latergetBoundingClientRectGets the position of the cutting frame to cut.
  • Use of external shadowsbox-shadowCan be
<div className={styles['shadow-layer']} style={{ height: `${videoHeight}px`}} ><div id="capture-rectangle" className={styles['capture-rectangle']} ></div>
</div>
Copy the code
@function remB($px) {
    @return ($px/75) * 1rem;
}

.shadow-layer {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  z-index: 1;
  overflow: hidden;

  .capture-rectangle {
    margin: remB(200) auto 0;
    width: remB(700); // Write here the width we need to cutheight: remB(450); // Write here the height we need to cutborder: 1px solid #fff;
    border-radius: remB(20);
    z-index: 2;
    box-shadow: 0 0 0 remB(1000) rgba(0.0.0.0.7); // Outer shadow}}Copy the code

Current Effect:

Real-time photos are cropped and uploaded to the server for OCR recognition

Crop using the canvas.getContext(‘2d).drawImage ability. void ctx.drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight);

MDN describes this property as: image The element drawn to the context. Allows any Canvas image source (CanvasImageSource) such as: CSSImageValue (EN-US), HTMLImageElement, SVGImageElement (EN-US), HTMLVideoElement, HTMLCanvasElement, ImageBitmap or post screen anvas.

Developer.mozilla.org/zh-CN/docs/…

We can pass video as source for cutting. Notice here

  • sx 和 syThe corresponding distance is truevideoElements of thetop leftDistance, not in the pagevideoAfter getting the size of the cutting box position, you need to convert it before cutting, otherwise the cutting position will not match.

/** * Get the actual size of the video */
function getXYRatio() {
  // videoHeight is the actual height of the video
  // offsetHeight is the height of the video CSS
  const { videoHeight: vh, videoWidth: vw, offsetHeight: oh, offsetWidth: ow } = video;

  return {
    yRatio: height= > {
      return (vh / oh) * height;
    },
    xRatio: width= > {
      return(vw / ow) * width; }}; }Copy the code

After the call to getUserMediaStream succeeds, we start capturing the video stream, taking screenshots every few seconds and sending them to the server.

/** Cut the relevant core code */
const Photo = () = > {
		const [videoHeight, setVideoHeight] = useState(0);
    const ref = useRef(null);

    useEffect(() = > {
        const video = document.getElementById('video');
        const rectangle = document.getElementById('capture-rectangle');
        const _canvas = document.createElement('canvas');
        _canvas.style.display = 'block';

        getUserMediaStream(video)
            .then(() = > {
                setVideoHeight(video.offsetHeight);
                startCapture();
            })
            .catch(err= > {
                showFail({
                  text: 'Unable to adjust the rear camera, please click album to manually upload id card'.duration: 6}); });function startCapture() {
          ref.current = setInterval(() = > {
            const { yRatio, xRatio } = getXYRatio();
            /** Get the position of the crop box */
            const { left, top, width, height } = rectangle.getBoundingClientRect();

            const context = _canvas.getContext('2d');
            _canvas.width = width;
            _canvas.height = height;

            // void ctx.drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight);
            context.drawImage(
              video,
              xRatio(left + window.scrollX),
              yRatio(top + window.scrollY),
              xRatio(width),
              yRatio(height),
              0.0,
              width,
              height,
            );

            // Get the base64 encoding of the current screenshot
            const base64 = _canvas.toDataURL('image/jpeg');
            // Base64 compression can be done again based on the scenario
            // Call the OCR interface every 2 seconds and upload Base64 to the server for identification
          }, 2000);
        }

        /** Empty timer */
        return () = > clearInterval(ref.current); } []); }Copy the code

Note: sx and SY are relative to the root element, and the top and left values obtained by getBoundingClientRect are equivalent to viewports, so we need to add scroll values.

conclusion

In fact, getUserMedia has almost no problem running on Android and MacOs, but there is so little discussion about this API in the community that most people probably don’t even know the existence of this API. When debugging on ios real phones, it only shows one frame at the beginning and then stops. Errors don’t give developers a detailed warning, and I spent most of my initial time on ios trying to figure out why the API didn’t work. However, this kind of business scenario should be relatively common in APP, and this paper is only the h5 implementation of this business scenario.

Attached is a final rendering:

References

1. IOS13 getUserMedia not working on the chrome and edge stackoverflow.com/questions/6… Bugs.webkit.org/show_bug.cg… It prevents ALL other browsers on iOS to offer video-conferencing, while Safari can => it’s a nasty anti-competitive behaviour that will for sure be scrutinized by US House Antitrust Committee & EU Commission, and Apple should not accumulate evidence of evil conduct.

2. The MDN getUserMedia developer.mozilla.org/zh-CN/docs/…

3. Ios10 + video play new strategy imququ.com/post/new-vi…