I’m sure you’ve had the wonderful idea of generating code directly from graphics when you were struggling with constantly changing UI designs, when you were pulling your hair out and tweaking the interface to make it look exactly like the design. Now, finally, such dark technology appears!

A project called Pix2Code uses deep neural networks trained to generate UI code directly from screenshots.

I won’t say anything. Let’s play the demo. The demo shows both iOS and Web UI screenshots that generate storyboards and HTML:

Pix2code, developed by a Copenhagen-based startup called UIzard, is now correct about 77% of the time, but the accuracy should get better as more data is available.

“We infer that deep learning used in this manner may eventually end the need to manually program [graphical user interfaces (GUIs)],” UIzard developer and founder Tony Beltramelli wrote in the research paper.

Arguably, with a lot of software being replaced, this innovation could at least allow some people to work in cooler areas. In his research paper, Beltramelli says it frees up developers to focus on the actual features they are building, rather than fiddling with the user interface for hours.

The neural network in Pix2Code is a convolutional neural network, a network capable of viewing images simultaneously at many different scales. This allows the network to learn by examining both the user interface and the underlying code to see the impact of each line of code on each element of the design.

To facilitate future research, Pix2Code’s datasets and code will be open sourced later this year. Beltramelli says the system is getting more accurate at scraping web pages to build a database of screenshots and HTML/CSS code. Once that happens, it won’t be long before you can map your website and have a neural network generate it directly.

However, the Demo does not show what the generated code will look like, and it may not be perfect enough. Hence the hilarious comment on Reddit:

And the funny ones:

Onlookers address: https://github.com/tonybeltramelli/pix2code

A recent article

Go back to Sohu and check out more