Evolution: Machine learning and Web-based experiences that are fast, real-time and fully interactive

The text/Takashi Kawashima

Illustration/Morgane Sanglier

Source | TensorFlow public number

There is no doubt that the emergence of machine learning (ML) was a breakthrough moment in modern computer science. As designers and users, we can already see its tangible impact: ML helps change the way medical diagnostics work, makes data centers more energy efficient, and even identifies a bowl of ramen through a store.

Machine learning (ML) has also helped develop the latest cutting-edge products and user experiences, creating countless exciting opportunities for web designers. In March, Google released TensorFlow.js, an open source framework for TensorFlow using JavaScript and ML. Tensorflow.js allows Web developers to train and deploy ML models in Web browsers such as Google Chrome. In other words, ML is open to the public and accessible to anyone with an Internet connection. But what does this mean for web designers?

Google Brand Studio recently released Emoji Scavenger Hunt, a fun mobile web game powered by Tensorflow.js. The game is simple: it shows you an emoji, and you need to use the camera in your phone to look up objects in the real world for a set amount of time. You need to find it in time to enter the next emoji.

Note: Emoji Scavenger Hunt link

emojiscavengerhunt.withgoogle.com/

Players captured more than two million emojis around the world; So far, they have found more than 85,000 different types of 💡 and 66,000 pairs of 👖. Finding ✋ seems pretty easy (2.91 seconds on average), but catching it is a bit more difficult (21.2 seconds on average). But how does the game accurately identify images? How does it know, for example, that the timepiece on your wrist is a watch? This is where ML comes in.

Browser-based machine learning is a game changer for web designers

Machine learning (ML) has shown us how to enhance the product experience; Similarly, machine learning (ML) in browsers has opened up many new and unheard-of interaction design opportunities for web designers. In the case of Emoji Scavenger Hunt, we wanted to create a fast-paced, fun and immediate experience — like the concept of communicating with Emojis — and web-based machine learning (ML) helped us achieve this mission.

Super fast real-time interaction

When playing Emoji Scavenger Hunt, you can point your phone or laptop’s camera at an object at varying distances, lighting and angles. It was impossible to predict using all the different ways to capture an object on a phone, but I was surprised to see how fast our ML model recognized the object; On my Pixel 2 phone, the image prediction algorithm ran 15 times per second, and on my laptop it ran even faster (60 times per second). The game’s algorithms run so fast that they constantly predict matches while moving the phone, significantly improving the accuracy of guesses. This results in a super fast real-time interaction experience that allows the game to flow smoothly without stalling and the player to enjoy the game.

One of the main reasons tensorflow.js is so fast is that it makes use of WebGL, a JavaScript API that allows you to render graphics in a browser using your device’s graphics processing unit (GPU). This speeds up the execution of the neural network while allowing you to run the ML model locally on individual devices without having to visit the server or travel back and forth to the back end. By accelerating the ML model, nearly 500 objects a day – from 👖 to 🐱 and from 🍔 to 🍲 – can be identified almost immediately.

Cacheable files and client-side calculations mean fast load times

If you’ve ever spent time waiting for a website to load, you know that speed is critical to a good Web experience. Even if your ML model is great, once it takes too long to load, users won’t engage with your experience. This is where the Tensorflow.js converter can help. It transforms the existing TensorFlow model into a cacheable file that can be run directly in a browser, while reducing the file size by up to 75%.

Note: Tensorflow.js converter link

Github.com/tensorflow/…

For example, our predictive model for Emoji Scavenger Hunt was only a few megabytes — about the size of a single image on a phone. Once loaded, the file is saved locally on the device to make the game run faster on subsequent loads.

Another benefit of browser-based ML is that it allows all ML computation — in this case, image recognition — to occur on the client side (for example, in their browser), whereas traditional ML experiences typically require a server-side with powerful processing power. For Emoji Scavenger Hunt, the server only needs to access web resources such as graphics and actual HTML files during the game. This makes back-end extensibility relatively simple and cost-effective.

When the power of the web meets the power of ML

Although most designers and developers today are focused on developing applications, the Web remains a very powerful medium. It is cross-platform and can be used with a variety of different devices, from mobile devices, tablets to desktop devices, and different operating systems (Android, iOS and Mac, Windows, etc.) with just one link. Unlike applications, it does not require download and installation, nor does it require complex configuration. Through the network, users only need to click, can be in-depth experience. And, of course, web-based content and experiences can be easily shared with others.

Today, people crave fast, fun experiences; Combining the power of the web with ML enables powerful new interactions that leverage the device’s own sensors.

Image recognition using the device’s camera is just one example. Using the Generic Sensor API, Web developers now have access to a range of device sensors, including accelerometers, microphones, and GPS. By combining device sensors with ML in the browser, you can imagine and design new interactive experiences at will.

Note: Common sensor API links

Developers.google.com/web/updates…

Machine learning and the future

We are just beginning to see the many ways ML can change Web development. There are still countless potential applications to explore, and I can’t wait to see the new interactive experiences people are designing. Are you working on a new project? Share with us using the hashtag # tensorFlowJS, or submit your project to the AI experiment. If you’re interested in the technical aspects of the project, all the code is available on GitHub.

Note: AI experiment links

Experiments.withgoogle.com/collection/…

Making a link

Github.com/google/emoj…

This work was made possible through a collaboration between Brand Studio and Google’s Tensorflow.js team. I also want to thank Jacques Bruwer, Jason Kafalas, Shuhei Iitsuka, Cathy Cheng, Kyle Gray, Blake Davidoff, Kyle Conerty, Daniel Smilkov, Nikhil Thorat, Ping Yu and Sarah Sirajuddin.

Takashi Kawashima is the designer and Creative director of Google Brand Studio. Prior to joining the team, he was the Art director of the Google Data Art team at Chrome Experiments for three years.