Suck the cat with code! This paper is participating in[Cat Essay Campaign]

Let’s start with a sneak peek:

Foreword: If you browse the Apple website, you’ll find a line at the bottom of each device’s introduction page that reads, “See *** with AUGMENTED reality.” After clicking on an Apple device, the device can be placed in the view of the user’s scene. When the camera of the phone rotates, different angles of objects can be seen, which feels like there is a real phone in front of you. (The effect is shown below. Note: You can’t view it on Android phones because it uses Apple’s own ARKit technology.)

You might be smart enough to wonder why you can only view it with an iPhone, but is there a universal Web AR technology that is pure front-end implementation?

Pure front-end solution

The implementation of pure front-end technology can be summarized as follows:

Take JSARToolKit as an example:

  • Use WebRTC to obtain camera information, and then draw the original picture on canvas canvas;
  • JSARToolKit computes the pose matrix and renders virtual information

Implement the core steps

  • (identification) WebRTC gets camera video stream;
  • Tracking.js, JSFeat, ConvNetJS, Deeplearn.js, keras.js;
  • (Render) A-frame, three. js, pixi. js, Babyl.js

A mature framework: ar.js

For example, there are mainstream development frameworks corresponding to each field. The relatively mature framework in Web AR field is AR.js, which provides the following three main functions in augmented reality:

  1. Image tracking. When the camera finds a 2D image, it can display something above or near it. The content can be 2D images, GIFs, 3D models (or animations), and 2D videos. Cases: artwork, study materials (books), leaflets, advertisements, etc.
  2. Location-based AR. This “augmented reality” technology uses real-world locations to display augmented reality content on a user’s device. Developers can leverage the library to give users real-world location-based experiences. Users can walk around (preferably outdoors) and view AR content from any location in the real world via their smartphone. As the user moves and rotates the phone, the AR content also reacts synchronously (so that some OF the AR content is “fixed” to its real location, with appropriate changes depending on how far it is from the user). Such a solution makes it possible for us to create interactive tour guides, such as visiting sites, museums, restaurants, hotels and so on in a new city. We can also improve learning experiences, such as scavenger hunts, biology or history learning games, and apply the technology to situational art, where visual art experiences are combined with specific real-world coordinates.
  3. Tag trace. When the camera finds a mark, some content can be displayed (this is the same as image tracking). The stability of the markings is not a problem; the limitations are shape, color, and size. It can be applied to experiences that require a lot of different markup and different content, such as :(enhanced books), flyers, ads, etc.

Start experimenting with AR.js

Enable HTTPS for development and debugging

Because the camera sensitive permission is used, debugging must be enabled in the HTTPS environment to run properly. If it is in the past, their own manual construction of HTTPS environment debugging is still more troublesome and time-consuming for many novices, fortunately, the latest vite+vue3 scaffolding construction project, you can easily open HTTPS access with a command. After scaffolding initialization, modify the package.json file and add –host to the dev command to expose the network request address (disabled by default), as shown below:

Then run the following command to enable HTTPS:

npm run dev -- --https
Copy the code

Run the official demo first and see how it works

The best way to learn a new framework or language is to run an official demo first.

Below is the case effect shown by the official code (note: the recorded GIF is large, please wait patiently for loading)

Wow ~ don’t you feel quite interesting? Next, enter the topic of the article formally, begin to masturbate the cat 🐱

start

As mentioned earlier, ar.js presents content in three ways, which will be implemented using Image Tracking. As the name implies, image tracking is based on an image. It identifies the image according to its feature points and tracks and displays the AR content. For example, when the camera captures the image, the content to be displayed can be displayed above the image.

Introducing dependency libraries

Ar.js has A new architecture from version 3 and uses JsarToolkit5 for tracking, while the render library is available in two ways: a-frame or three.js. The A-frame mode simplifies the creation of scene materials by using HTML tags. For example, to display an image, you can use < A-image >.

Modify the index.html file:

Comment out the VUE code injection first

Then introduce dependencies:

<script src='./src/sdk/ar-frame/aframe-master.min.js'></script>
<script src='./src/sdk/ar-frame/aframe-ar-nft.js'></script>
Copy the code

Cat stroking pose 1: Show pictures of cats

<body> <a-scene embedded arjs> <a-assets> <img id="my-image" src="./src/assets/cat.jpg"> </a-assets> <a-marker preset="hiro"> <a-image rotation="90 0 0" src="#my-image"></a-image> <! < span style =" max-width: 100%; box-sizing: border-box; color: green; ></a-box> --> </a-marker> <a-entity camera></a-entity> </a-scene> </body>Copy the code

A quick explanation of the above code:

  1. <a-scene>Declare a scene, which you can interpret as a body element, with other tag elements embedded in it;
  2. <a-marker>The label declares the identification picture, that is, when the camera recognizes the identification picture, it does the corresponding processing; The plugin’s default hiro image is used here, as can be seen in the following GIF
  3. use<a-assets>The material used by the package is equivalent to declaring the import of the material, followed by the<a-marker>The use of

Take a look at the effect:

Cat stroking position two: Play video

In addition to showing pictures, you can also show videos, first look at the effect:

The code is as follows:

<a-scene vr-mode-ui="enabled: false;" renderer='antialias: true; alpha: true; precision: mediump; ' embedded arjs='trackingMethod: best; sourceType: webcam; debugUIEnabled: false; '> <a-assets> <video src="https://ugcydzd.qq.com/uwMROfz2r57CIaQXGdGnC2ddPkb5Wzumid6GMsG9ELr7HeBy/szg_52471341_50001_d4615c1021084c03ad0df73c e2e898c8.f622.mp4?sdtfrom=v1010&guid=951847595ac28f04306b08161bb6d1f7&vkey=3A19FB37CFE7450C64A889F86411FC6CE939A42CCDAA6 B177573BBCB3791A64C441EFF5B3298E3ED4E99FFA22231772796F5E8A1FCC33FE4CAC487680A326980FFCC5C56EB926E9B4D20E8740C913D1F7EBF5 9387012BEC78D2816B17079152BC19FCEF09976A248C4B24D3A5975B243614000CAA333F06D850034DA861B01DCA1D53B546120B74F%22" preload="auto" id="vid" response-type="arraybuffer" loop crossorigin webkit-playsinline muted playsinline> </video> </a-assets> <a-nft videohandler type='nft' url='./src/assets/dataNFT/pinball' smooth="true" smoothCount="10" SmoothTolerance ="0.01" smoothThreshold="5"> < A-video SRC ="#vid" position='50 150-100 'Rotation ='90 0 180' width='300' height='175'> </a-video> </a-nft> <a-entity camera></a-entity> </a-scene> <script> window.onload = function () { AFRAME.registerComponent('videohandler', { init: function () { var marker = this.el; this.vid = document.querySelector("#vid"); Marker.addeventlistener ('markerFound', function () {// Identify the marker and start playing the video this.vid.play(); }.bind(this)); Marker.addeventlistener ('markerLost', function () {this.vid.pause(); this.vid.currentTime = 0; }.bind(this)); }}); }; </script>Copy the code

🐱 : Meow ~ is not feeling more cool and better play?

Cat stroke posture three: with sound net technology, and your cat across the air shout

If you are a front-end developer, I believe you must know ruan Yifeng this big guy. Once in his weekly science and technology magazine to see such an interesting thing: a piece in the amazon rain forest, the installation of the recording equipment, real-time will pick up to a bird to a web site, you can open the web site to hear real-time birds in the rainforest, simple said is the site can hear the rain forest birds “live”. (Unfortunately, I can’t find the website address right now.)

We cat lovers may have the same emotional needs: we are going on a business trip for several days and we can’t take good care of our cat. What if we want to see our cat in real time?

Buy a security camera

The solution, of course, is to open Sonnet and find video calls. (Here’s a thumbs up for sonnet documentation, which is very clear and organized, unlike some cloud products that look like they’re rummaging through garbage.)

Modify the document demo with vuE3 writing

Install dependency packages first:

"agora-rtc-sdk-ng": "latest"
Copy the code

Code in app.vue:

<script setup> import AgoraRTC from "agora-rtc-sdk-ng"; import { ref } from "vue"; const joinBtn = ref(null); const leaveBtn = ref(null); let rtc = { localAudioTrack: null, client: null, }; let options = { appId: "2e76ff53e8c349528b5d05783d53f53c", channel: "test", token: "0062e76ff53e8c349528b5d05783d53f53cIADkwbufdA1BIXWsCZ1oFKLEfyPRrCbL3ECbUg71dsv8HQx+f9gAAAAAEAACwxdSy/6RYQEAAQDK/pFh", uid: 123456, }; rtc.client = AgoraRTC.createClient({ mode: "rtc", codec: "vp8" }); rtc.client.on("user-published", async (user, mediaType) => { await rtc.client.subscribe(user, mediaType); console.log("subscribe success"); if (mediaType === "video") { const remoteVideoTrack = user.videoTrack; const remotePlayerContainer = document.createElement("div"); remotePlayerContainer.id = user.uid.toString(); remotePlayerContainer.textContent = "Remote user " + user.uid.toString(); remotePlayerContainer.style.width = "640px"; remotePlayerContainer.style.height = "480px"; document.body.append(remotePlayerContainer); remoteVideoTrack.play(remotePlayerContainer); } if (mediaType === "audio") { const remoteAudioTrack = user.audioTrack; remoteAudioTrack.play(); } rtc.client.on("user-unpublished", (user) => { const remotePlayerContainer = document.getElementById(user.uid); remotePlayerContainer.remove(); }); }); Const handleJoin = async () => {await rtc.client.join(options.appId, options.channel, options.token, options.uid ); rtc.localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack(); rtc.localVideoTrack = await AgoraRTC.createCameraVideoTrack(); await rtc.client.publish([rtc.localAudioTrack, rtc.localVideoTrack]); const localPlayerContainer = document.createElement("div"); localPlayerContainer.id = options.uid; localPlayerContainer.textContent = "Local user " + options.uid; localPlayerContainer.style.width = "640px"; localPlayerContainer.style.height = "480px"; document.body.append(localPlayerContainer); rtc.localVideoTrack.play(localPlayerContainer); console.log("publish success!" ); }; / / leave phone const handleLeave = async () = > {RTC. LocalAudioTrack. Close (); rtc.localVideoTrack.close(); rtc.client.remoteUsers.forEach((user) => { const playerContainer = document.getElementById(user.uid); playerContainer && playerContainer.remove(); }); await rtc.client.leave(); }; </script> <template> <div> <button ref="joinBtn" @click="handleJoin" type="button" id="join"> join </button> <button </button> < div> </template> <style></style>Copy the code

Running effect:

This is equivalent to installing a camera at home, and if we need a remote view, we can join the call through a test address provided by sonnet officials

Open the above url, enter your project appId and token, you can see successfully join the call:

The image below is captured by the phone’s camera. Forgive me for using cat photos instead of 😂

Make the video screen run in the AR.js screen

This is due to personal time, temporarily do not research to achieve. Here’s an idea: After all, the cat may not appear in the video. Combined with the display pictures mentioned in the cat stroking posture, in fact, we can arrange photo walls or other cool subjects around the video area in the AR scene. In this way, even if we can’t see the cat star person when we open the video, You can also look at its photos and things like that.

conclusion

This article borrow essay activity, simple start to understand the web AR related knowledge, in these days of learning process feel pretty fun, this article also should throw a brick to introduce, hope more developers know AR related knowledge.

AR is really cool in terms of experience and the future is worth looking forward to.

In recent years, Apple has been committed to promoting AR technology experience and bringing relevant landing products, such as radar scanning and spatial audio functions in order to improve THE AR experience. It is worth mentioning that in this year’s Apple autumn press conference, Apple’s invitation also made use of AR + space audio technology. Even if you are not an Apple fan, when you actually use the experience, you will still really feel from the heart: WOW ~ cool. You can check out this video.

However, compared with Apple’s own ARkit technology, the current Web AR technology still has some gaps in experience (such as performance problems and unstable recognition), and lacks the ecosystem. We hope that Web AR technology will develop rapidly in the future. After all, the cross-platform universal features of The Web end. Making everyone’s terminal running is the premise of realizing the large-scale application of AR scenarios.

Facebook is betting on the concept of the metasverse, and AR is actually part of it, so until the metasverse arrives, AR is something that every front-end developer should pay attention to and learn from.

eggs

If you ask me what my favorite cat is, I will say – “landlord’s cat”, ~ hahaha 🐱 ~

The resources

AR. Js’s official website

Ar.js Chinese translation document

Introduction and application of key technologies of cross-platform mobile Web AR

Sound network document