Sketch2Code is a Web-based solution that uses AI to turn hand-drawn user interface sketches into usable HTML code. Sketch2Code was developed by Microsoft in collaboration with Kabel and Spike Techniques. The code, solution development process, and other details related to Sketch2Code can be found on GitHub.

Sketch2Code project address: https://github.com/Microsoft/ailab/tree/master/Sketch2Code

The following figure illustrates the process of turning a hand-drawn sketch into code using Sketch2Code. Can do more to try on the Microsoft web site: https://sketch2code.azurewebsites.net/

Let’s take a look at the process of turning a hand-drawn sketch into HTML code using Sketch2Code:

  • Users upload pictures to the site.

  • The custom visual model predicts which HTML elements will appear in the image and marks their positions.

  • The handwriting text recognition service reads the text in the predicted element.

  • The layout algorithm generates the grid structure according to the border space information of the predicted elements.

  • The HTML generation engine uses the above information to generate HTML code.

The workflow is as follows:

Sketch2Code uses the following components:

  • Microsoft Custom Vision: This model is based on the training of different hand-drawn images and tagged with information related to common HTML elements such as text boxes, buttons, images, etc.

  • Microsoft Computer Vision Services: Used to identify text in design elements.

  • Azure Blob Storage: Holds information related to each step of the HTML generation process, including raw images, predicted results, layout and grouping information, etc.

  • Azure Function: It acts as a back-end entry point to coordinate the generation process by interacting with other services.

  • Azure Website: The front end of the user interface where users can upload design drawings and view the generated HTML.

The above components are grouped together using the following architecture:

Do you feel the urge to try?

You can find the open source code for Sketch2Code here:

https://github.com/Microsoft/ailab/tree/master/Sketch2Code

Can also to validate the actual effect of Sketch2Code here: https://sketch2code.azurewebsites.net/

Reference links:

https://blogs.technet.microsoft.com/machinelearning/2018/08/30/turn-whiteboard-ux-sketches-into-working-html-in-seconds- introducing-sketch2code/

http://www.alphr.com/microsoft/1009840/microsofts-ai-sketch2code-builds-websites