1. Background

With the rise of Zhongtai in recent years, the team has deconstructed, restructured and built the existing capabilities around the scene of zhongtai business, hoping to systematically connect the disassembled building blocks in series, so as to achieve the purpose of reducing cost and increasing efficiency.

For the e-commerce platform, every year needs to face a large number of promotion activities page demand, for how to improve the page output efficiency, we all adopt the “page visual construction” solution. Correspondingly, we also built a “Visual page building platform of Linglong”. However, in the past two years, the demand for customization of large-scale promotion activities has exploded. The limited component modules of the platform can no longer meet the requirements of product operation, and front-end engineers can no longer simply solve problems with the idea of “reuse”. When the business develops to a certain extent, the limited manpower and lengthy development process cannot meet the booming business needs.

We need to “seek change”. Can intelligent thinking be used to solve problems that cannot be solved by traditional human-intensive research and development? Along this direction, we aim at “front-end intelligence”, hoping to expand the front-end capability circle with the help of AI and machine learning, break through the design and research and development workflow, and achieve large-scale production.

2. Project introduction

Deco Intelligent code project is the team’s exploration in the direction of “front-end intelligence”. We try to start from the point of design code generation to complete the existing design to RESEARCH and development, so as to improve the efficiency of production and research.

In a daily requirement development process, it is often necessary to follow a fixed workflow. The product submits the requirement PRD, the interaction designer outputs the interaction draft according to the PRD, and then the visual designer outputs the product visual draft, and then enters the front-end development workflow. For front end engineers, the input source is visual draft + PRD, and the output is live page code.

Deco is expected to address the tasks of the above process that are relatively low value to front-end engineers and can be handled with reuse ideas:

  • UI visual draft restoration, namely page reconstruction, writing HTML + CSS;
  • Reusable business logic bindings;

“Design draft code generation” as the breakthrough point, we need to use intelligent solutions to replace the traditional manual page refactoring (analysis layer style + cut figure, etc.), is expected to extract structured data from visual version of the original information description, and then combined with intelligent layout algorithm, the output maintainable code page.

3. Technical solutions

The core of the intelligent code generation capability of the design document is how to generate a “structured data description” information, which is called D2C Schema.

Deco design draft intelligently generated code does two things:

  1. Extract “structured data description” from visual draft;
  2. Express “structured data description” into code;

In essence, Deco smart code extracts a structured data description (D2C Schema) from the original information of visual draft through design tool plug-ins, and then processes the D2C Schema by combining rules system, computer vision, intelligent layout, deep learning and other technologies. It is converted into D2C Schema JSON data with reasonable layout and semantics, and finally converted into multi-terminal code with the help of DSL parser.

The core link of Deco intelligent code constitutes the core technology system of the team’s exploration of “front-end intelligence”. Centering on the systematic construction of the production and research process, a benign production and research closed loop is achieved by combining Cloud IDE, Taro multi-terminal unified solution, design and development asset platform, and Intelligent design. Reduce cost and improve efficiency for the whole engineering link.

4, intelligent code technology stratification

At the present stage, Deco intelligent code core link mainly contains four layers of component recognition, layer processing, layout algorithm and semantic processing. The following is an analysis of the internal implementation principle centering on these subdivided layers.

4.1 Component Identification Layer

Component recognition layer is responsible for identifying elements in the design draft image, including business component recognition, basic component recognition and block recognition. Through the recognition ability, it outputs the parts of the design draft that match the components of the existing component library, carries out component recommendation and reuse, and assists the subsequent processing layer. For example, the semantic processing layer carries out semantic naming according to the component properties to improve the availability of the generated code.

4.1.1 Component Identification

The component recognition layer uses the AI capability to identify the components by using the deep learning target detection algorithm, input the exported pictures of the design draft, and output the categories and positions of components in the pictures.

4.1.2 data set

Data set is the main part of deep learning to deal with problems. We collected the activity page data of the Long platform, the design draft of the conference hall and the data of Relay platform, and constructed a component identification data set containing 2W + samples. Among them, automatic tagging is introduced to obtain absolutely accurate tagging data and reduce manual tagging cost through Dom structure analysis of the miniature page built with components.

In order to further enrich the data set and solve the problem of unbalanced number of components, an automatic sample generation scheme is adopted. Based on Quark official business component library and JDCOP atomic component library, ten types of samples are supported. Component templates with random attributes are used to form complete pages from components, which are automatically marked in the export process.

4.1.3 Block identification

Most mobile terminal design draft ratio is too long, there are more than 10:1, and uneven distribution, is not conducive to identify target detection, as a result, the first page is divided into blocks, block recognition algorithm based on traditional image processing process, the use of edge detection and flood water filling algorithm for connected area, set the filter threshold floor size identification box.

Block recognition algorithm is applied to layer processing layer automatic group, optimize the layer nesting structure, and help the layout algorithm to produce a more reasonable component structure tree. In addition, unrecognized areas are automatically divided into new floors. By fixed height restriction, adjacent blocks are grouped into a larger size block, which can achieve the effect of dividing designs with different heights into several areas with similar heights, and then put into target detection network for identification.

4.2 Layer handling Layers

The layer processing layer mainly separates, merges and extracts meta information from the layers in the design draft. At the same time, it intelligently groups and categorizes the components with the recognition layer to export the first Deco Schema DSL. A Deco workflow is like a design pattern for pipes and filters in software engineering. The design draft is the input of the pipe, the flow in the pipe is the DSL generated at each stage, and the output of the pipe is a piece of semantic code.

Layer processing layer by using component recognition ability of AI intelligent identification design draft for each block, synthesize block layer information within a group, and then automatically by the block matching algorithm matching blocks and layer, the design draft can be automatically group, the level of group data is advantageous to the layout algorithm judgement block information and parent-child relationships.

A Sketch is a compressed Document (file suffix “. Sketch “) composed of several layer meta-information (including Document and Pages, etc.) and resource files (mainly pictures). We obtain a DSL for layout algorithm service after processing layer meta-information.

Through the development of Sketch plug-in, the API provided by Sketch can help us operate the documents in Sketch, process and filter these data after obtaining the layer information. Layer information processing is mainly divided into two layers:

  1. Design draft processing layer:

    1. Copy an original design draft, in this copy of the design draft for various processing operations;
    2. Symbol decoupled, decoupled the Symbols in the document into the actual layer group;
    3. Filter invisible layers, filter out designer hidden layers and empty layer groups;
    4. Layer merge, combine some special layers or layers, and convert them into pictures;
    5. For mask processing, the image under the mask needs to be cropped if it is out of the scope of the mask, and the layer position and width information under the mask need to be reset.

  2. Layer information processing layer:

    1. Extract useful information from layers, such as style manipulation, text splitting, layer hierarchy, etc.
    2. Layer information conversion, such as image base64 character string data into CDN image address;
    3. Useless layer detection, remove some unstyled or transparent layers;
    4. Layer leveling process, layer data from the tree level level into a one-dimensional structure;
    5. Group information filtering, which classifies ungrouped data into grouped data by comparing its location and size.

The following is the layer information processing flow:

In addition to the basic processing of layer information, we set up a series of optimization rules for data export to increase the rationality of layout and semantics. For example, on some of the more aggressive designs, the complex background design may be composed of several vector shapes under one layer group (as shown below). Exporting these layers as they are would introduce a lot of complexity and uncertainty into the layout.

In the process of combining images, if all layers under a layer group are vector graphics, we will combine them into one image, which will greatly reduce the difficulty of layout. The final combination effect is as follows:

Of course, the optimization rules mentioned above are not sufficient in all cases, as designers are free. In order to improve the rationality of layout and semantics, we propose some standard protocols for designers and developers to use in the included design draft.

4.3 Layout algorithm layer

4.3.1 Why is a layout algorithm needed

The layout algorithm is based on the input source data conforming to the Deco Schema specification, which can be processed through the Deco Sketch plug-in for visual processing, resulting in the export of design element information.

The element data exported by the Deco Sketch plug-in is the element information based on the absolute positioning of the origin coordinates of the upper-left corner (0, 0). In general (no active grouping, no AI recognition, etc.), elements are flat, that is, there is no affiliation between elements.

In the front-end development process, absolute positioning layout is not scalable, readable development requirements, so if not solved, it will become one-time code. Therefore, layout algorithms are needed to improve the extensibility and readability of the generated code for subsequent secondary development.

4.3.2 Layout layer core algorithm

The design of layout algorithm layer consists of three layers: data structure transformation layer, layout derivation layer and style calculation layer.

4.3.2.1 Data conversion layer

The data structure transformation layer converts Deco Schema JSON data into a structure similar to DOM tree, which can be used for node insertion, deletion and lookup operations.

Here is the basic LayoutNode data structure:

LayoutNode { ... Omit node attributes... AppendChild (child) {} prependChild (child) {} insertAfter (insertedChild, afterChild) {} insertBefore (insertedChild, beforeChild) {} replaceChild (newChild, replacedChild) {} removeChild (child) {} get x () {} get y () {} get width () {} get height () {} get offsetLeft () {} get offsetTop () {} get previousSibling () {} get nextSibling () {} intersect (node) {} contains (node) {} disjoint (node) {} tangent (node) {} hitTest (node) { ... }Copy the code
4.3.2.2 Layout derivation layer

The layout deduction layer is to carry out the row and column segmentation deduction, which generally includes: spatial layout algorithm, projection layout algorithm, background layout algorithm, feature detection layout algorithm, coordinate deduction algorithm, background layer and redundant layer detection algorithm and so on.


Spatial layout algorithm

Projection layout algorithm


4.3.2.3 Style computing Layer

The style calculation layer is a series of calculations on the results obtained through the layout derivation layer, while the Deoc style uses Flexbox for most layouts, and some special cases require absolute positioning. After the Layout derivation, the Layout structure has a clear hierarchy and adjacency relationship.

Flexbox spindle and side axis can be obtained by coordinate calculation based on hierarchical relationship. Based on the adjacency relation, the margin and other styles between neighbors can be calculated.

4.4 Semantic Layer

When the layout algorithm is used to process the design data, we can obtain well-structured code, but we can find that the code is still not very readable due to the lack of corresponding semantic class names for node elements. In order to end up with code that can be redeveloped, we need to add a semantic processing layer after the layout algorithm layer to make the code have good semantics.

The first problem that the semantic layer solves is how to add semantically defined class names to element nodes.

To achieve this goal, we can review how we append class names to element nodes during development, using the following example of a single commodity.

The figure above is an example of a product diagram. We can identify a product based on the image, price, text below the image, etc. Then we can give the region the class name goods. The nodes within the region, such as the image, can be given the class name goods_pic. At the bottom of the image, the text can be given the class name goods_tit, and the price can be given the class name price, which is the general logic for adding the class name to the element node.

It can be seen that when we usually determine the semantics of a region or a component, we need to make a judgment according to the semantic combination of nodes in the region. For example, the commodity component above needs to determine the semantics by relying on internal elements such as pictures, prices and copywriting, so as to determine the class name. Therefore, the semantic processing method is to start from the child node of the container element, first determine the semantics of the child node, and then deduce the semantics of the container element, and finally deduce the complete semantics of the whole node tree.

At the semantic layer, our main processing object is the JSON Schema data processed by the layout algorithm layer, which is called the layout tree. At this time, the layout tree has a good structure, and we can carry out semantic inference operations on it. The process of inference starts at the leaves of the tree, bubbles up one layer at a time to the branches, and then bubbles back to the root.

At present, we make inferences mainly on the basis of node location, style, size, sibling nodes and other factors. At the same time, we combine some intelligent means for auxiliary inference based on different node types. For example, the minimum leaf node may be of two types: picture and text. For text, we can analyze the part of speech and semantics of the text through NLP. For some pictures, we can use the way of picture classification or recognition to determine the picture classification or extract the key information on the picture to determine the picture semantics.

In order to determine the semantics of each node, we need to combine a series of rules to deduce the existing facts (style, location and other information), and at the same time, new facts will be obtained after some rule inference, and the final result can be obtained only after other rule inference. Therefore, this is an inference system based on rule inference. We can help us to make inference decisions by implementing a forward chain inference engine.

For example, in the process of inferring the above commodity component, we first find the text node with price factor, named price, and then find the picture node near price, which is in a similar hierarchy in the tree and meets the requirements of commodity graph size. In this way, we can basically determine that the container that contains both the price and conforms to the characteristics of the commodity graph is a commodity container. Then, according to the number of elements in the container, whether there is a text near the picture, and NER analysis of the text, we can determine whether the text is the name of the commodity, so as to determine its semantic class name.

In the whole semantic layer, the above decision rules are just the tip of the iceberg. We combine the whole e-commerce scene, analyze a large number of design draft and online cases and summarize a large number of decision rules to help us rationalize semantic naming. Meanwhile, IN the semantic process, NLP analysis, image classification and recognition and other AI means are used. We will write relevant articles in the follow-up for you to carry out a specific introduction.

Of course, obtaining the node class name is only a preliminary achievement in the initial stage of semantization. In the future, we will continue to explore semantization to lay a solid foundation for the subsequent implementation of field logical binding.

5. Stage achievements

At present, the component layer, layer, processing layer, layout algorithm and the semantic layer of the four core modules we have already achieved breakthroughs, can already realize the analysis of the Sketch design draft, transforming it into a well structured, with semantic, can be secondary development code, preliminary realizes the design draft to refactor code reduction stage.

We have tested on a large number of e-commerce promotion design drafts, and the layout restoration degree can reach more than 90%, while the final output code availability rate can reach 80%. We have promoted the use of some internal businesses.

6. Future prospects

In the future, in the above core modules at the same time, we will join the online visual editor, allows developers to adjust the manual intervention in the generated code to, to get better code, at the same time we will also explore the join field binding and binding logic function, make the code can have the business logic and around to T3 Deco and the level of the engineer, Further improve the efficiency of production and research.

Deco is a seed, maybe it has just sprouting at this moment, but we have a lot of expectations for it. We hope to explore the road of front-end intelligence through Deco, explore various possibilities of AI and front-end integration, and more importantly, we hope to open the efficiency revolution of production and research through Deco. In the various front-end engineering, platform, methodology tends to improve the present, explore another way to reduce cost and increase efficiency for business.

The future of Deco is worth looking forward to.