Article/bourbon
Background & Trend
With the development of artificial intelligence in recent years, AI has emerged in all walks of life. In this Internet Web development and design, can also be seen in recent years, such as Rico (mobile client applications annotation data sets) of publicly available data sets, for the researchers to better in the mobile application (App) do some academic research under this scenario, intelligent front-end code generation is one of the good direction.
In addition to some data sets used for academic research, there are also some Internet products that directly attempt to provide intelligent code generation services, aiming to further optimize the service ability of code generation by collecting the quality evaluation of service code generation while providing code generation services.
- Imgcook: Specializes in design draft code generation services
- Fronty: Good at image generation website homepage
- Yotako: Good at PSD generated code services
The above three products generate code, input information channels mostly from the designer’s structured design draft or images, at the same time, the analysis of this input also determines the quality of the final output automatically generated code, so how to measure the quality of code generation in this? Let’s take a look at what our front-end code structure looks like in general.
The structure of the code
Typically, our front-end code is structured in two categories: static GUI code and dynamic logic code.
Static GUI code
Static GUI code typically consists of hypertext Markup Language (HTML) and cascading style sheets (CSS). The description of the code in visual style can compare with the filling style of each layer node in the design stage, but the description in hierarchy structure is usually unique to the front-end code and can not be reflected in the design stage.
Dynamic logic code
The front-end logic code (Javascript) is usually interactive logic code, which is often to give static GUI code some dynamic interactive logic behavior, which is also the most tedious part of the programming implementation requirements, but such interactive logic behavior is usually not directly reflected in the design phase (in static design tools).
Under the current hot trend of Serverless, another piece of programming code that the front end will face is the research and development of Server FaaS (Function as a Server), which includes the front end’s acquisition and processing of Server data in demand, and is closer to the coding processing of data flow behind the program.
The above three parts of the code occupy most of our front-end programming content, to realize the intelligent generation of these codes, we need to prepare for the following two aspects:
- The description of this code is structured and transformed through the data description
- Key information is extracted from related materials for structured data description
This article focuses on the second part of the information data acquisition thinking, let’s take a look at the information source to obtain the generated code consists of several parts.
Input source of information
In the actual business requirement scenario, the information we can access before development includes the following pieces: requirement draft of product manager, interaction draft of interaction designer, visual draft of visual designer and page image after finalization. If we compare our web (program) information flow process to the famous mathematician Claude Shannon’s information theory (source – encoder – channel – decoder) :
- The information source corresponds to the demand draft, which is the planning demand of the web page (program)
- The encoder corresponds to the interaction draft, that is, the designer interacts with the requirements obtained and arranges the overall design form of the page
- Channel corresponding design draft refers to the use of image, color, text, sound and other information transmission channels in the design
- The decoder corresponds to the information process that these channels are finally displayed in front of users to translate into their own understanding, such as understanding a specific functional component in the image
It can be seen from the above that the user’s complete information acquisition and understanding of an application comes from the flow of information in each stage. Only looking at a single channel is bound to lose other important information content, which is also the same for the acquisition of information in the development state. It is like that you only look at the visual draft instead of the requirement draft to do research and development. It is obvious that the requirements implemented by the delivered code are often substandard.
Requirement Draft (PRD)
The requirement draft usually contains the requirement function points of the foreground and server data, which is also the most original starting point of user information acquisition. But basically, the content of the requirement draft can not be structured description, which means that we can only artificially digest and understand the information here. What if we make constraints on the delivery of the requirement draft of the product experience so that it can be structured expression?
| Foreground data presentation | Foreground interaction logic | Background data logic | |
|---|---|---|---|
| Module A | Display fields A and B | Field A is hidden when not present; Module sends buried points when clicked | Field A is obtained from xx channel and treated as XXX |
It can be seen that we can more easily extract the information corresponding to specific modules in the three parts of front-end code for structurally expressed requirements, and analyze and structurally describe the important logic code we need by combining natural language processing (NLP) and domain-specific stock data model.
Visual Design
The content description of the design draft is relatively simple. Currently, some design tools used by designers, such as Sketch, Photoshop, XD and other software, all contain certain structured description. We only need to extract this part of structured information from the visual draft through the developer ability attached to the design tool. This translates into a structured description of what we need to populate our static GUI visual code.
Interactive draft (dynamic effect) design
Different from the description of static information of layer node in visual draft, interaction draft usually expresses the response between node states. Among some common design tools that can make interactive response (such as AE and Principle), If we can extract the structured description of the interaction made by the designer in the design tool, then we can also convert it into the corresponding interaction behavior in our programming, and take this part as the front-end interaction logic code implementation part.
Model information learning
In addition to obtaining data directly from the information carrier mentioned above, we can also enrich the data we extract through model training and learning based on our experience. For example, if common Web components cannot be described in the design draft, we can extract this part of information through the target detection model. For example, for the fields corresponding to the text content in the business domain, we can also automatically map the static text content in the design draft to the dynamic field through classification learning.
image
Different from the above information carrier, the image does not contain structured information data, so we can only intuitively obtain the pixel dot matrix information in the image. However, with current deep learning capabilities, we can extract the content of basic design elements (text, image, Shape) and style attributes from images.
In addition, we can also achieve the information content description to identify the object (component) in an image through training model.
The text
For some static text content in the design draft, if there is a dynamic field with interface issued by mapping relations, we can also analyze designers often depict the text content, combined with the specific business the content may be the corresponding data field to do, such as XXX flagship store, we might guess its shops header fields in a data source for the business data, This allows automatic mapping of fields in the interface during code generation.
Structure describing
Structural descriptions refer to the artifacts (JSON data descriptions) extracted from the above design draft. When we use GUI static code to describe a page or application, we often have a part of semantic structure, namely structural layout grouping, in addition to basic style information. In fact, the layout semantics are not included in the visuals, so we often need to use our previous experience of layout semantics to train the results of this part of the model to make the layout structure description of the GUI more semantic.
Deep connection of information
Intelligent generating code information sources, there are many, each channel with the resulting code is more or less, if not all digestion using this information, then the generated code is necessarily incomplete, so how will these come from different dimensions of information gathered in a line is also very critical.
Here are the different sources of information are we going to do correlation analysis of a try, as you can see, the dimension of the input channels we can use modules to do, that is, a module corresponding to the GUI static code, front desk logic code, the server-side data logic and potential model components within the module identification results. Connection through these information, the module code we generate in addition to the earliest phase only from static visual draft reduction in the UI code, can also be combined with other logic, semantic information will automatically be converted to dynamic part of static, so as to achieve the purpose of the complete code (field binding, module, material identification, rendering logic, etc.).
conclusion
Imgcook has experienced two years of experience and demonstrated that it is feasible to extract information from design draft to automatically generate part of GUI code. Then, how to obtain more information from other information channels in the next step, and make association analysis on these information points in a plane to establish a complete spectrum. To achieve coverage to generate more accurate code is a direction we continue to explore.