preface

In the recent years of front-end development, Web applications have been developing towards the experience of Native APP. For example, PWA, the most talked about product this year, even Apple has gradually integrated Service Worker into its safari browser. With the support of major browser vendors, the Web is getting more and more powerful, and the experience of front-end applications is getting better as the hardware is upgraded.

However, in many cases, Native APP is still needed for low-level hardware calls and complex operations, which is provided to front-end application calls in the form of JSBridge in the Webview environment. As a result, many front-end applications can only rely on the customized Webview of Native APP. Of course, browsing vendors are also following suit, offering many low-level hardware and system-level interfaces directly to the Web, such as the Web Bluetooth API, Web Share API, Shape Detection API, and so on.

In this article, you’ll introduce Chrome’s integrated Shape Detection API.

Pattern recognition

Photos and images are the biggest part of the Internet, and a significant number of them contain recognizable features, such as faces, QR codes or text. As you can imagine, the computational overhead of identifying these features is very high, but there are some interesting scenarios, such as automatically tagging faces in photos or redirecting based on urls in images. Hardware vendors have supported these features for a long time, but Web applications have been slow to take advantage of them and must rely on hard-to-use libraries to do so.

Because image recognition requires system-level resources and computing power, only the native underlying API can handle it. In current Web applications, when it comes to the function of image recognition, pictures are generally uploaded to the server or sent to Native APP through JSBridge in Webview, and the results are returned to the Web for further processing. For the Web, the capabilities are thin and must rely on server-side processing or JSBridge, and the experience will be greatly improved if image processing becomes a standard API for the Web.

In Native development, both Android and IOS platforms directly provide some capabilities of image recognition for application development at the system level, such as qr code/bar code recognition. The Barcode API is available on Android and the CIQRCodeFeature API is available on iOS.

In Web applications, why not have a standard JS API to call the underlying capabilities of the system?

The Chrome team has been trying to integrate the Shape Detection API directly into the browser for Web development calls for 16 years. Recognition of the API is still in the incubation in WICG graphics and the experimental stage, if want to open this function, need to download the Canary version of Chrome (www.google.com/chrome/brow)… //flags/#enable-experimental-web-platform-features Click “Enable” on Experimental Web Platform Features to enable the Experimental feature.

Next verify that the browser supports the API by typing in Console:

window.FaceDetector
window.BarcodeDetector
window.TextDetector
Copy the code

If LOG out:

ƒ FaceDetector() {[native code]} ƒ BarcodeDetector() {[native code]} ƒ TextDetector() {[native code]}Copy the code

You can now invoke the Shape Detection API in your browser.

While the API is still experimental at the moment (experimental features are erratic and will not be fully integrated), as front-end developers of the new era, we are willing to give it a try.

In this example, you need to download the Canary version of Chrome and enable Experimental Web Platform features.

API

The Shape Detection API provides three interfaces to call:

  • FaceDetector: Face recognition;
  • BarcodeDetector: Qr code/barcode recognition;
  • TextDetector: Text recognition;

1. Face recognition

FaceDetector is a low-level acceleration platform component class for face recognition in images. It takes a configuration parameter, FaceDetectorOptions. We directly create an instance of FaceDetector:

const faceDetector = new FaceDetector({
  fastMode: true,
  maxDetectedFaces: 10
});
Copy the code

Parameters in FaceDetectorOptions:

  • FastMode: [Boolean], indicates whether speed first (over accuracy) mode is enabled, will be identified by smaller scale (closer to target pattern) or by looking for a larger target pattern;
  • MaxDetectedFaces: [Number], the maximum Number of faces recognized in the current scene;

The FaceDetector class has a method that accepts an Image or Blob object to detect the face in the Image and returns a Promise object:

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" EDGE "> <title>Shape Detection API Demo</title> <style> * { box-sizing: border-box; padding: 0; margin: 0; } </style> </head> <body> <img id="face-image" src="//opsqe9du9.bkt.clouddn.com/1.jpeg" alt="image" crossorigin="anonymous" style="width: 100%;" /> <script type="text/javascript"> window.onload = main(); function main () { const faceDetector = new FaceDetector({fastMode: true, maxDetectedFaces: 10}); const image = document.getElementById('face-image'); faceDetector.detect(image) .then(detectedFaces => { console.log(detectedFaces); }) .catch((e) => { console.error("Face Detection failed, boo.", e); }); } </script> </body> </html>Copy the code

In the above examples: An img tag is created in the HTML (note that crossorigin=”anonymous” if the Image is from a cross-domain source), and the Image object is retrieved. The faceDetector. Detect is called to detect the face in the Image. DetectedFaces is printed after successful detection.

(5) [DetectedFace, DetectedFace, DetectedFace, DetectedFace, DetectedFace] 0: DetectedFace boundingBox: DOMRect bottom: 275 height: 44 left: 710 right: 754 top: 231 width: 44 x: 710 y: 231 landmarks: Array(3) 0: location: {x: 729, y: 246} type: "eye" 1: {location: {... }, type: "eye"} 2: {location: {... }, type: "mouth"} length: 3 1: DetectedFace {boundingBox: DOMRect, landmarks: Array(3)} 2: DetectedFace {boundingBox: DOMRect, landmarks: Array(3)} 3: DetectedFace {boundingBox: DOMRect, landmarks: Array(3)} 4: DetectedFace {boundingBox: DOMRect, landmarks: Array(3)} length: 5Copy the code

DetectedFace === 5; Each element is a DetectedFace object with two properties:

  • BoundingBox: a DOMRect object containing the position (bottom, top, left, right, x, y) and size information (width, height) of the recognition area;
  • Landmarks: is an array containing the landmarks of the face, such as eyes, mouth, and position information (x, Y).

Complete Demo: tongchengqiu. Making. IO/mixin – Demo /…

2. Qr code/bar code recognition

BarcodeDetector is a low-level acceleration platform component class that recognizes qr codes or bar codes in images. We directly create examples:

const barcodeDetector = new BarcodeDetector();
Copy the code

Like FaceDetector, the detect function is used to detect images. This method also takes an Image or Blob object and returns a Promise object:

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" EDGE "> <title>Shape Detection API Demo</title> <style> * { box-sizing: border-box; padding: 0; margin: 0; } </style> </head> <body> <img id="qrcode-image" crossorigin="anonymous" src="//opsqe9du9.bkt.clouddn.com/img-2.png" alt="QRCODE"> <script type="text/javascript"> window.onload = main; function main () { const barcodeDetector = new BarcodeDetector(); const image = document.getElementById('qrcode-image'); barcodeDetector.detect(image) .then(detectedCodes => { console.log(detectedCodes); }) .catch((e) => { console.error("Barcode Detection failed, boo.", e); }) } </script> </body> </html>Copy the code

As in the example above, print out the detectedCodes obtained:

[DetectedBarcode] 0: DetectedBarcode boundingBox: DOMRect Bottom: 375.4765348434448 height: 362.06744384765625 left: Width: 362.0674743652344 x: 9.409090042114258 Y: CornerPoints: Array(4) 0: {x: 9.529978730623, y: 13.5299307307cornerpoints: Array(4) 0: {x: 9.529978730623, y: 13.5299307307307cornerpoints: Array(4) 0: {x: 9.529978730623, y: 13.409090995788574} 2: {x: 371.4765625, y: 375.4765319824219} 3: {x: 9.409090042114258, Y: 374.5908508300781} length: 4 rawValue: "https://qiutc.me/" length: 1Copy the code

Each element in the obtained array is a DetectedBarcode object of the recognized QR code, which contains three attributes:

  • DetectedBarcode: Location of qr code;
  • CornerPoints: a sequence of vertices with a bar code recognized;
  • RawValue: The real value represented by the TWO-DIMENSIONAL code, which is a String value;

Complete Demo: tongchengqiu. Making. IO/mixin – Demo /…

3. Word recognition

TextDetector is a low-level acceleration platform component that recognizes text in an image. We directly create an example:

const textDetector = new TextDetector();
Copy the code

Like FaceDetector, the detect function is used to detect images. This method also takes an Image or Blob object and returns a Promise object:

<! DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, <meta HTTP-equiv =" x-UA-compatible "content=" EDGE "> <title>Shape Detection API Demo</title> <style> * { box-sizing: border-box; padding: 0; margin: 0; } </style> </head> <body> <img id="text-image" crossorigin="anonymous" src="//opsqe9du9.bkt.clouddn.com/img-3.png" alt="TEXT"> <script type="text/javascript"> window.onload = main; function main () { const textDetector = new TextDetector(); const image = document.getElementById('text-image'); textDetector.detect(image) .then(detectedTexts => { console.log(detectedTexts); }) .catch((e) => { console.error("Text Detection failed, boo.", e); }) } </script> </body> </html>Copy the code

Like the above example, print out detectedTexts obtained:

(12) [DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText, DetectedText] 0: DetectedText boundingBox: DOMRect {x: 190.78125, y: 42.1875, width: 1527.984375, height: 31.21875, Top: 42.1875,... } cornerPoints: (4) [{...}, {...}, {...}, {...}] rawValue: "" 1: DetectedText {rawValue:" ", boundingBox: DOMRect, cornerPoints: Array(4)} 2: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 3: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 4: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 5: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 6: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 7: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 8: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 9: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 10: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} 11: DetectedText {rawValue: "", boundingBox: DOMRect, cornerPoints: Array(4)} length: 12Copy the code

Each element in the array is a DetectedText object that recognizes the text and contains three properties:

  • DetectedBarcode: Position of text;
  • CornerPoints: vertex position information;
  • RawValue: the real value of the text, which is empty from the demo, is not fully implemented.

Complete Demo: tongchengqiu. Making. IO/mixin – Demo /…

conclusion

At present, the Shape Detection API is still in the experimental stage and its functionality is not perfect, but it is expected to be improved and implemented in production environments over the next few years.

The Shape Detection API can greatly reduce application complexity and make Web applications more independent in many scenarios.

  • With face recognition, we can identify faces more accurately and give more accurate suggestions when users process pictures.
  • With the application of QR code/bar code recognition and the call of the camera, scanning information can be directly obtained at the front level, for payment, jump, add friends and other operations;
  • With the application of character recognition, we can directly realize the function of OCR on the client side.

At present, more and more low-level apis can be used directly in the browser environment, and front-end applications can directly use more system resources and call more hardware interfaces. This is also a trend of the future, as the browser gives the Web more power, the Web will also be able to provide a better experience for users.

reference

  • Github.com/WICG/shape-…
  • Wicg. Making. IO/shape – detec…
  • Developers.google.com/android/ref…
  • Developer.apple.com/documentati…
  • www.infoq.com/cn/news/201…