preface

UI (User Interface) is the medium of interaction and information exchange between software and users, realizing the transformation between the internal form of information and human acceptable form. UI development generally needs to go through UI design and UI implementation. UI design refers to the design of software interaction, operation logic and interface. UI designers and interaction designers usually complete a set of UI interface design according to users’ software requirements, and finally present it in the form of UI design draft (PSD, PNG, JPEG file, etc.). UI implementation is to code the UI design draft generated in the UI design stage, which is the task of the front-end engineer.

With the rapid development of the Internet, from the earliest simple hypertext document content, gradually developed into a rich and colorful flexible and dynamic experience platform, a variety of mobile phone APPS, PC applications and websites are more than to meet. From the earliest users only pay attention to the realization of software functions, to now not only need the realization of software functions, but also the overall UI interface of the software is very critical. At present, in order to meet the aesthetic needs of users, software UI is designed to be more and more complex. Whether it is layout or element style, front-end development is more and more laborious, development cost is higher and higher, and for a large number of pages that need to be on-line quickly, there is not enough manpower and material resources to develop.

In the end-to-end business of Bytedance live events, it is often necessary to develop event pages on multiple platforms. Active pages are usually laid out, logically similar, frequently required, and require rapid iteration. If the conventional development method is used to develop an activity page, the product, front-end, server, test and other parties need to participate, and each activity page online cycle is long, can not quickly respond to the needs of the product. For the development of the activity page, the optimal process is to use the page visual construction platform to achieve, that is, the Rubik’s Cube platform in the middle stage of the live event. The platform implements a componentalized UI editor based on DOM, and provides well-packaged UI components for operation students to use, so as to complete an activity page. From the previous 4 days to complete the development of an activity page, to 2 hours to drag out an activity page and online, greatly improving the efficiency of page development.

However, the Rubik’s Cube platform also has certain limitations. Since it only needs to target activity-related businesses, the platform can only be applied to the generation of active pages. An edited UI page is described in the form of schema definition by extending JSON. However, schema description based on JSON is limited, and the UI page can only be restored by the corresponding client to parse the schema, and it cannot be applied to other platforms.

Therefore, a more general UI editing App is proposed based on the Rubik’s Cube platform, which uses a more general DSL to describe the page dragged out, and can compile THE DSL code into the code of each platform. Similar to Imgcook, UI editor is implemented based on WebGL and compiled to multi-terminal code based on DSL to improve UI development efficiency.

Performance demonstration & technology used

Operation effect display

The main page: provides the base components on the left, the UI editor implemented using WebGL in the middle, and the properties of the selected UI components on the right

The code to compile

DSL edit page

The technology used

A normal drag-and-drop UI generation platform would be a website, but this article tries to implement it as a Electron App.

  • Electron: Electron is to use Web front-end technology (HTML/CSS/JavaScript/React, etc.) to create a native cross-platform desktop application framework. React can be developed quickly with the electric-react-Boilerplate template, but in this paper, the react environment is manually built, and resources are packed and App is built using Webpack and electric-Builder. Use Webpack/React to pack and build Electron.

  • Node.js: Node.js is an open source, cross-platform JavaScript runtime based on the Chrome V8 engine that allows JavaScript to run in server-side environments. Node.js uses a single-threaded, asynchronous, non-blocking IO, event-driven architecture, which makes Node.js extremely efficient in handling IO intensive tasks.

  • React: React is a JavaScript library for building Web UIs that allow developers to write UIs in a data-driven, component-based, declarative way.

  • WebGL: IS a 3D drawing protocol running on the Web side. This drawing protocol combines JavaScript and OpenGL ES2.0 to provide hardware-accelerated 3D rendering and render 3D scenes and models in the browser with the help of graphics cards. The birth of WebGL technology solves two key problems of existing Web 3D rendering: 1. Cross-platform, 3D rendering can be achieved using the native Canvas tag. 2. High rendering efficiency, graphics rendering based on the underlying hardware acceleration implementation.

  • Konva: A 2D JavaScript library based on Canvas, which can be easily used to achieve graphical interaction effects of desktop applications and mobile applications. It can efficiently realize animation, transformation, node nesting, local operations, filters, caching, events and other functions. Konva’s biggest feature is that graphics can interact. All Konva graphics can listen for events, which is similar to native DOM interaction. Event listener is implemented on the basis of Layer (Konva.Layer), each Layer has a foreground renderer used to display graphics and background renderer used to listen to events, by registering global events in the background renderer to determine the current triggered event graphics, and call the callback processing events. Konva borrows heavily from the browser DOM, for example, Konva defines stages to store all graphics, similar to HTML tags, and layers to display graphics, similar to the body tag. Node nesting, event listening, node lookup, and so on also borrow from DOM operations, which makes Konva framework easy to use for front-end developers.

Application design

Demand analysis

The core functions of App include WebGL UI editor, DSL code editor and DSL code compiler. The system functional requirements are shown below.

  • Basic functions: The system needs to implement basic login and registration functions, logout functions, global shortcut key binding and other functions.

  • UI Editor: Visual WebGL UI editor that provides a basic common UI component library and allows users to draw a UI page by dragging and dropping components from the basic common UI component library; Component toolbar allows users to copy, delete, paste, and redo components on a canvas. Provide component properties panel, allowing users to modify component background, border, position, size and other properties; Provides a DSL code build toolbar that allows users to generate UI pages on a canvas into DSL code, which in turn compiles DSL code into target platform code.

  • DSL code editor: provides an editor for writing DSL code, supporting code highlighting, copy, paste, save, and other functions. Provides a file system that allows users to create and delete a DSL code file; Provides code run tools to generate DSL code to UI pages or to generate object code.

  • Help center: DSL code syntax help, UI editor use help.

Overall architecture design

The system adopts Client/Server mode for architecture, and is developed by separating the front and back ends. The Client is Electron App, and the Server is implemented by Express.

  • Client, using Electron, React, Node.js to achieve a cross-platform PC App.

  • Server side, based on node.js Express written Server side, and exposed the corresponding API for the Client side to call. Integrated WebSocket service, running independently on node.js side, sharing corresponding database connection and other common classes and functions, providing Socket support. And build a static resource server based on Niginx to provide file storage services such as pictures.

  • The database uses MySQL/MongoDB databases, and MongoDB stores UI page information, such as UI element location, size, style, and other jSON-like information. MySQL stores basic information such as user information and component information.

Client architecture design

The Client is a PC application developed using Electron technology. Although Electron is a framework for creating cross-platform applications using front-end technology, it is not the same as traditional web development. Electron is based on the master-slave process model, which consists of a main process and multiple renderers that communicate with each other using IPC. Based on this process model, the system processes are divided into functions:

  • The main process is responsible for interprocess communication, window management, server requests, and native C++ plug-in loading
  • The renderer process is only responsible for rendering the Web page and concrete business logic

The render process is developed using Typescript/React/Redux. With the help of React Hooks, common UI logic can be removed to improve code reuse. The main process is developed using Typescript/C++, where node.js plug-ins are developed and packaged as.nodeFile, main process load.nodeFile to call C++ code. With the help of the Webpack compilation tool, all code of the renderer process is compiled toindex.html,renderer.js,style.cssAnd code compression and code segmentation optimization, improve the efficiency of code operation. The main process only compiles all code compilations to onemain.jsAnd, inmain.jsTo load the renderer processindex.htmlComplete the whole system operation. Final reuseelectron-builderPackage the compiled main and renderer code and other resource files into one.dmgApplication file, complete the construction of the whole system.

Main process design

The main Client process can be divided into three modules: Widget module, Services module, and compile module.

  • The Widget module is responsible for window creation and management, such as creating the Login window and implementing IPC calls such as minimizing and closing the Login window.

  • The Services module is responsible for providing basic system Services, including IPC call Services, for communication between the rendering process and the main process; Fetch service, which provides back-end interface invocation capability; Session service stores user sessions and records login information. Socket service provides back-end socket connection. The fileSave service provides the file saving function.

  • The Compile module performs DSL code compilation and implements multi-platform code building by implementing multiple compilers.

Render process Design

In the process of rendering process packaging, multi-page packaging design is adopted to separate some UI pages from a rendering process and design them into multiple independent new Windows (rendering process). During development, module hot update code is injected into each rendering process to achieve the development environment page hot update. Multiple page entries are added in the Entry field of Webpack to achieve independent packaging, and each packaged page uses HtmlWebpackPlugin to generate the corresponding HTML file. The main process instantiates a separate window to load the corresponding page packedindex.htmlA new window is created.

Among multiple Windows, the main window is the most core window of the system, and the modules and functions realized are relatively complex. Components developed using React Hooks cannot avoid communicating with each other, so Redux is adopted for global state management to optimize the communication flow between components.

In the Redux workflow, state is extracted and stored in the Redux state tree storedispatchThe action into thereducerTo updatestateAfter updating the state, React Render is triggered to update the view. The key point of designing a Redux state tree is to extract component state, to extract multiple component-dependent states into the Redux state tree and use them in componentsuseSelectorHooks Subscribes to a state in the state tree, useduseDispatchTo obtaindispatchTo update a state in the Redux state tree.

In the main window rendering process, including Redux module, Page module, Components module, WebGL module.

  • Page module, the main window Page is similar to a single Page application, each sub-page is implemented under the Page, including UI editor sub-page, DSL code editor sub-page and so on.

  • Redux module, realize Redux basic event flow store, Action, reducer, used for communication between components.

  • Components module, common UI component implementation, such as Toast, Modal and other common Components.

  • WebGL module, based on WebGL native JavaScript to achieve UI canvas and UI components and some related tool functions.

Sever terminal architecture design

The Server is built using the Node.js Express framework, which is encapsulated and extended on the basis of Express.

  • The Core layer encapsulates and extends Express, implementing App classes, Middleware abstract classes, Controller abstract classes, and defineRouter route decorator.

  • Services encapsulates basic Services and invokes third-party Services, such as file uploading and file downloading.

  • Socket is an abstraction of Socket services. It provides Socket classes to support Socket functions on the server. The underlying layer is developed based on SocketIO.

  • A Controller is an implementation of a concrete business logic Controller, using classes to abstract a business and routing decorators to decorate methods in the class to express a business logic.

  • The Database abstracts connections and operations between MySQL and MongoDB.

  • Model Provides the basic Model of database tables, including User table, WebGLPage table, and so on.

The server uses the Typescript programming language to compile all Typescript files into JavaScript and run them in node.js at runtime by running TSC commands based on tsconfig.json.

Database design

MongoDB is a key-value database with a storage structure similar to JSON. It has a certain hierarchical structure and can well represent the state of a UI page that is being edited. Therefore, the system uses this feature of MongoDB to store the information of each UI page being edited. The storage structure is as follows.

DSL syntax design

A Domain Specific Language (DSL) is a programming Language designed for a Specific Domain with limited expression. It includes internal DSLS and external DSLS.

1. External DSLS are different from traditional programming languages. External DSLS usually use custom syntax and use corresponding programming languages to parse DSL code. Regular expressions, SQL, configuration files, etc.

2. An internal DSL is a special syntax expression of a programming language. The code written with an internal DSL is a legitimate program, but with a specific style, and uses some features of the programming language, only for dealing with some specific problems of the system.

The system uses an external DSL definition that describes a UI page and parses the DSL to generate object code. The DSL syntax design references SCSS syntax and uses a nested structure to express UI page nesting relationships. Attribute abstraction of components in UI pages yields the following DSL syntax definitions:

1. Type. Name Indicates the Type and name of a component, starting with {and ending with}, and encapsulates component properties and related information.

2. Component attributes are defined into two categories: base attribute and style attribute. The base attribute keyword includes position, size, text and image; the style attribute is defined by the style keyword and wrapped with curly braces; the inner attribute includes background, border and shadow. Use “; “between attributes. To separate.

3. Separate the parameters of an attribute with Spaces and end with semicolons (;). The symbol terminates the definition of an attribute.

4. Express all the children of a component using the children keyword, wrapping all the children with “[” and”] “, separating the DSL code of the children with “, “.

A simple DSL component is defined as follows.

Function implementation

Main process-related service implementation

The Client uses the separation mode of main process and renderer process development, the main process to achieve Session management, Socket connection, server interface call, page communication and other services.

1. Implementation of the Session service The main process manages sessions globally and stores user login information. You can use the Session API in Electron to retrieve the current session

export const Session = {
  setCookies(name: string, value: string) {
    const Session = session.defaultSession; // Get the default session from the main process
    const Cookies = Session.cookies; / / get the cookies
    return Cookies.set({
      url: domain,
      name,
      value,
    });
  },
  getCookies(name: string | null = null) {
    const Session = session.defaultSession;
    const Cookies = Session.cookies;
    if(! name)return Cookies.get({ url: domain });
    return Cookies.get({ url: domain, name });
  },
  clearCookies(name: string) {
    const Session = session.defaultSession;
    returnSession.cookies.remove(domain, name); }};Copy the code

2.Socket connection implementation and encapsulation The server uses the SocketIO library to implement a Socket service, and the main process uses the SocketIO library to establish a Socket connection

class SocketService {
  static instance: SocketService | null = null;
  static getInstance() {
    return! SocketService.instance ? (SocketService.instance =new SocketService()) : SocketService.instance;
  }
  private socket: SocketIOClient.Socket;
  constructor() {
    this.socket = SocketIO(url);
    this.socket.on('connect'.(a)= > { / / the connection
      console.log('connect ! ');
    })
  }
  emit(event: string, data: any) {
    if (!this.socket.connected) this.socket.connect();
    this.socket.emit(event, data);
  }
  on(event: string, callback: Function) {
    this.socket.on(event, callback)
  }
}
Copy the code

In the main process, node.js request module is used to realize the server interface request, and the renderer process indirectly uses the Request module through IPC invocation, so as to realize the request of the server interface

export const fetch = {
  get(url: string, data: any) {
    return fetch.handle('GET', url, data);
  },
  post(url: string, data: any) {
    return fetch.handle('POST', url, data);
  },
  handle(method: 'GET' | 'POST', url: string, data: any) { // Encapsulate the request module
    return new Promise((resolve, reject) = > {
      constparams = { method, baseUrl, url, ... (method ==='GET' ? { qs: data } : { form: data })
      };
      request(params, (err, res, body) = > {
        try {
          if (err) {
            reject(err);
            return;
          }
          resolve(JSON.parse(res.body));
        } catch(e) { reject(e); }}); }); }};Copy the code

4.IPC interprocess communication realization and encapsulation The communication between the rendering process and the main process is the core of the whole system, reasonable definition of communication interface can improve the efficiency of the system. In the main process, Electron provides the ipcMain object to process the renderer’s messages; In the renderer process, ipcRenderer is used to process messages from the main process. For example, when a server requests a logical IPC call, the main process registers the IPC call with ipcmain.handle

export const handleFetch = (a)= > {
  ipcMain.handle(IpcEvent.FETCH, async (event, args: { method: 'GET' | 'POST', url: string, data: any= > {})return await fetch.handle(args.method, args.url, args.data); // fetch
  });
};
Copy the code

Renderer call

function fetch(method: 'GET' | 'POST', url: string, data: any = null) {
  return ipcRenderer.invoke(IpcEvent.FETCH, {
    method,
    data,
    url
  }).catch(console.error);
}

// fetch('GET', '/user/login', { email, password });
Copy the code

The renderer process sends the DSL code to the main process via an IPC call, and the main process calls the compilation service to complete the code compilation and return the results to the renderer. In general, THE parsing of DSL code is generated into the abstract syntax tree, and then the node modification of the abstract syntax tree is generated into the object code. However, given the simplicity of the designed DSL, you only need to use regular expressions to parse the corresponding properties and concatenate them into JSON

parser.id_index = 0;
export function parser(str: string) :any {
  let childrenMatch = str.match(/children\s*:\s*\[(.+)/);
  const childrenToken = childrenMatch ? childrenMatch[1].trim().replace(/\]\s*\}$/.' ').trim() : ' ';
  if (childrenMatch) {
    str = str.substring(0, childrenMatch.index);
  }

  const children = getChildrenToken(childrenToken); // Subcomponent token

  let nameMatch = str.match(/^[\w\d\.\s]+\s*{/); // Parse component type, name
  const [type, name] = nameMatch ? nameMatch[0].replace('{'.' ').trim().split('. ') :' '.' '];
  let positionMatch = str.match(/position\s*:([^;] +); /); // The component position property
  const [x = 0, y = 0] = positionMatch ? positionMatch[1].trim().split(' ').map(v= > Number.parseInt(v)) : [0.0];

  let sizeMatch = str.match(/size\s*:([^;] +); /); // The component size property
  const [width = 0, height = 0] = sizeMatch ? sizeMatch[1].trim().split(' ').map(v= > Number.parseInt(v)) : [0.0];

  let backgroundMatch = str.match(/background\s*:([^;] +); /); // Component background property
  const [fill = 'white', opacity = 0] = backgroundMatch ? backgroundMatch[1].trim().split(' ') :' '.' '];

  let shadowMatch = str.match(/shadow\s*:([^;] +); /); // Component shadow property
  let [offsetX = 0, offsetY = 0, blur = 0, shadowFill = 'white'] = shadowMatch ? shadowMatch[1].trim().split(' ').map((v, i) = > {
    if (i === 3) return v;
    return Number.parseInt(v); }) :0.0.0.' '];

  let borderMatch = str.match(/border\s*:([^;] +); /); // Component border property
  const [borderWidth = 0, radius = 0, borderFill = 'white'] = borderMatch ? borderMatch[1].trim().split(' ').map((v, i) = > {
    if (i === 2) return v;
    return Number.parseInt(v); }) :0.0.' '];


  let textMatch = str.match(/text\s*:([^;] +); /); // The component text property
  const textMatchRes = textMatch ? textMatch[1].trim() : ' ';
  let text = textMatchRes.match('/' (. +) /);
  if (text) {
    text = (text[0] as any).replace(/ / ^ '.' ').replace($/ / '.' ');
  }
  let textFill = textMatchRes.split(' ');
  textFill = (textFill[textFill.length - 1] as any).trim();

  let imageMatch = str.match(/image\s*:([^;] +); /); // The component image property
  const src = imageMatch ? imageMatch[1].trim().replace(/ / ^ '.' ').replace($/ / '.' ') : ' ';
  return { / / stitching JSON
    name,
    type: type.toLocaleUpperCase(),
    id: `${type.toLocaleUpperCase()}-${name}-${parser.id_index++}`, props: { position: { x , y }, size: { width, height }, ... (backgroundMatch ? { background: { fill, opacity: +opacity } } : {}), ... (shadowMatch ? { shadow: { offsetY, offsetX, blur, fill: shadowFill } } : {}), ... (borderMatch ? { border: { width: borderWidth, radius: radius, fill: borderFill } } : {}), ... (textMatch ? { text: { text, fill: textFill } } : {}), ... (imageMatch ? { image: { src } } : {}) }, children: children.map(str= > parser(str)) // Resolve the child token recursively
  };
}
// Calculate the child component token
function getChildrenToken(childrenToken: string) {
  let count = 0;
  let child = ' ';
  const result = [];
  for (let i = 0; i < childrenToken.length; i++) {
    child += childrenToken[i];
    if (childrenToken[i] === '{') {
      count++;
    }
    if (childrenToken[i] === '} ') {
      count--;
    }
    if ((childrenToken[i] === ', ' && count === 0) || (count === 0 && i === childrenToken.length - 1)) {
      result.push(child.replace($/ /,.' ').trim());
      child = ' '; }}return result;
}
Copy the code

The process of generating object code is conditional based on the component type of the JSON object

function compileToElementToken(obj: any) :any {
  switch (obj.type) {
    case TYPES.WIDGET: { // 
      return (`<div id="${obj.id}">${obj.children.map((v: any) => compileToElementToken(v)).join('\n')}</div>`);
    }
    case TYPES.BUTTON: {
      return (`<button id="${obj.id}">${obj.props.text ? obj.props.text.text : ''}</button>`);
    }
    case TYPES.SHAPE: {
      return (`<div id="${obj.id}">${obj.children.map((v: any) => compileToElementToken(v)).join('\n')}</div>`);
    }
    case TYPES.TEXT: {
      return (`<div id="${obj.id}">${obj.props.text ? obj.props.text.text : ''}</div>`);
    }
    case TYPES.INPUT: {
      return (`<input id="${obj.id}" placeholder="some text"/>`);
    }
    case TYPES.IMAGE: {
      return (`<img id="${obj.id}" src="${obj.props.image ? obj.props.image.src : ''}" alt="none"/>`); }}}Copy the code

Finally spliced into object code

const jsonObject = compileToJson(code);
let style = (`
* { box-sizing: border-box; margin: 0; padding: 0 }
html, body { height: 100%; width: 100% }
${compileToStyleToken(jsonObject)}`).replace(/\n(\n)*(\s)*(\n)*\n/g.'\n');
let div = compileToElementToken(jsonObject).replace(/\n(\n)*(\s)*(\n)*\n/g.'\n');
const html = (` <! DOCTYPE> <html lang="zh"> <head><title>auto ui</title></head> <style>${style}</style>
<body>${div}</body> 
</html>`);
Copy the code

Main process multi-window management

The Client App is composed of user information window, main window, login window, portrait selection window and other Windows. Each window is an independent rendering process, and the main process is responsible for managing all the Windows. Electron does not provide multi-window management, so you need to manually manage the status of each window and the interaction logic between Windows.

App abstracts each window into a Widget class. Due to the particularity of the window, each Widget class is designed based on the singleton pattern.

The parent Widget implements the IWidget interface and performs basic functions of a window, such as create() to create a window and close() to close a window. The subclass is a singleton class, obtained using the static method getInstance(). Each window is a frame window, which removes the status bar decorations of the operating system, so you need to manually close, minimize, maximize Windows, and drag and drop Windows. For window drag, you can use a line of CSS attribute -webkit-app-region: drag in Electron. Close, minimize, and maximize Windows are implemented by invoking IPC calls of registered close, minimize, and maximize Windows in the rendering process.

The Widget class’s create() method is the key method for creating Windows. Instantiate a window using Electron.BrowserWindow, render the page with an.html file that is loaded with the instance object’s loadURL() or loadFile(), and register the corresponding events

// DSL code preview window
export default class CodeWidget extends Widget {
  static instance: CodeWidget | null = null;
  static getInstance() {
    return CodeWidget.instance ? CodeWidget.instance : (CodeWidget.instance = new CodeWidget());
  }

  constructor() {
    super(a);// Window closing event
    onCloseWidget((event, args: { name: string }) = > {
      if (args.name === WidgetType.CODE) {
        if (this._widget) {
          this._widget.close();
        }
      }
    });
  }

  create(parent?: Electron.BrowserWindow, data?: any) :void {
    if (this._widget) return;
    // Instantiate window
    this._widget = newElectron.BrowserWindow({ ... CustomWindowConfig, parent, width:550,
      height: 600,
      resizable: false,
      minimizable: false,
      maximizable: false
    });
    // Load the.html file
    loadHtmlByName(this._widget, WidgetType.CODE);
    // Initial data
    if (data) {
      this._widget.webContents.on('did-finish-load'.(a)= > {
        this._widget? .webContents.send('code', data); }); } parent? .on('close'.(a)= > this.reset());
    this._widget.on('close'.(a)= > this.reset()); }}Copy the code

Multiple Windows cannot avoid communication between each other, such as the picture selection window and user information window communication. Click Modify profile picture in the user information window to open the profile picture selection window. After selecting an profile picture in the profile picture selection window, you need to send the selection result to the user information window.

The easiest way to communicate between Windows is to useipcMainThe objects andipcRendererObject to achieve, that is, in the rendering process of a window to send a message to the main process, the main process to send a message to the rendering process of another window, to achieve the communication of two Windows.

However, in this implementation mode, additional event names need to be defined and communication between the two Windows needs to be realized using the main process. As a result, Electron provides a more convenient remote module that can communicate without sending interprocess messages. The remote module of Electron is similar to Java’s RMI (Remote Method Invoke), a communication mechanism that uses remote objects to call each other to realize communication. For Windows with parent-child structure, communication only needs to use remote in the child window to send messages to the renderer process in the parent window

remote.getCurrentWindow().getParentWindow().webContents.send('avatar-data', { ...avatar });
Copy the code

The communication mechanism of remote is shown in the following figure.

Client UI canvas implementation

UI canvas is one of the core of the system and is implemented based on WebGL Konva framework.

When using Konva to implement a canvas, you only need to use konva. Stage to define the Stage and konva. Layer to define the drawing Layer

this.renderer = new Konva.Stage({
  container: container.id,
  width: CANVAS_WIDTH,
  height: CANVAS_HEIGHT
});
// Manage all the components in the canvas
this.componentsManager = new ComponentManager();
this.layer = new Konva.Layer();
// Redux dispatch is the core of webGL and React communication
this.dispatch = dispatch;
// Add an auxiliary line to the canvas
WebGLEditorUtils.addGuidesLineForLayer(this.layer, this.renderer);
this.renderer.add(this.layer);
Copy the code

When adding a UI component to a UI canvas, first bind the component to events in Konva, including events such as select, drag, and resize. Then draw the component to the Layer Layer; Then hide the anchor point of the previous component and show the anchor point of the dragged component. Detect whether the dragged component is in a component, if it is in a component, then the dragged component is added to the component inside, forming a nested structure; Notify dispatch to notify the React side and save the status of the current component. Finally re-paint the cloth.

addComponent(webGLComponent: WebGLComponent) {
  // Add events to the component
  this.addSomeEventForComponent(webGLComponent);
  // Add the component to the draw layer
  webGLComponent.appendToLayer(this.layer);
  this.componentsManager.pushComponent(webGLComponent);
  // Check whether the dragged component is inside a component
  const id = WebGLEditorUtils.checkInSomeGroup(
    this.layer,
    this.renderer,
    webGLComponent.getGroup()
  );

  if (id) {
    // If yes, add it to the corresponding component
    this.componentsManager.appendComponentById(id, webGLComponent);
  }
  // Notify the React side
  this.dispatch(selectComponent(
    webGLComponent.getId(),
    webGLComponent.getType(),
    webGLComponent.getName(),
    this.componentsManager.getPathOfComponent(webGLComponent).join('>'),
    getComponentProps(webGLComponent)
  ));
  // Re-paint the cloth
  this.render();
}
Copy the code

The corresponding addSomeEventForComponent() function is implemented as follows, adding selected events, drag events, and modify events

addSomeEventForComponent(component: WebGLComponent) {
  component.onSelected(e= > { // The component selects the event
    this.componentsManager.showCurrentComponentTransformer(
      component.getId()
    );
    component.moveToTop();
    this.dispatch(selectComponent(
      component.getId(),
      component.getType(),
      component.getName(),
      this.componentsManager.getPathOfComponent(component).join('>'),
      getComponentProps(component)
    ));
    this.render();
  });

  component.onDragEnd(e= > { // Component drag ends the event
    this.dispatch(dragComponent(e.target.position()));
  });

  component.onTransformEnd(e= > { // Component transform ends the event
    this.dispatch(transformComponent(component.getSize()));
  })

  component.onDragEnd(e= > { // Component drag ends the event
    const id = WebGLEditorUtils.checkInSomeGroup(
      this.layer,
      this.renderer,
      component.getGroup()
    );


    if (id) {
      this.componentsManager.appendComponentById(id, component);
    }
    this.render();
  });
}
Copy the code

3. Check whether a component is inside a component in the canvas. After the drag component event ends, check whether the dragged component is inside a component and moved to the corresponding target component to form a nested structure. First, obtain the coordinates and size information of all components in the canvas except the drag component, and store it in the format of {ID, W, H, x, Y} into the array points; Then get the coordinate and size information of the drag component, denoted as groupPoint, format {id, W, H, x, Y}; Iterate through the points array to see if it can contain the drag-and-drop component and add it to the includePoints array as follows:

const points = getAllGroupPoints();
const groupPoint = getGroupPoint(group);
const includePoints: PointType[] = [];
points.forEach(point= > {
  if (
    groupPoint.x >= point.x &&
    groupPoint.y >= point.y &&
    groupPoint.x + groupPoint.w <= point.x + point.w &&
    groupPoint.y + groupPoint.h <= point.y + point.h
  ) {
    includePoints.push(point);
  }
});
Copy the code

traverseincludePointsFor all items in the array, the component with the smallest distance from the drag component is selected as the parent component by Euclidean distance.

The algorithm flow for detecting whether a component is inside a component is as follows

let minDistance = Number.MAX_SAFE_INTEGER;
let id = ' ';
const distance = (p0: { x: number, y: number }, p1: { x: number, y: number }) = > {
  return Math.sqrt(Math.pow(p0.x - p1.x, 2) + Math.pow(p0.y - p1.y, 2));
};
includePoints.forEach(point= > {
  const diff =
        distance(
          { x: groupPoint.x, y: groupPoint.y },
          { x: point.x, y: point.y }
        ) +
        distance(
          { x: groupPoint.x + groupPoint.w, y: groupPoint.y },
          { x: point.x + point.w, y: point.y }
        ) +
        distance(
          { x: groupPoint.x, y: groupPoint.y + groupPoint.h },
          { x: point.x, y: point.y + point.h }
        ) +
        distance(
          { x: groupPoint.x + groupPoint.w, y: groupPoint.y + groupPoint.h },
          { x: point.x + point.w, y: point.y + point.h }
        );
  if(diff < minDistance) { minDistance = diff; id = point.id; }});Copy the code

4.WebGL communicates with ReactThe canvas drawn by WebGL has been removed from the browser DOM and elements are drawn line by line, unlike DOM. The communication between WebGL and React is implemented using the global state tree provided by Redux. The Dispatch function is passed when the WebGL canvas is constructed to trigger changes in the global state tree to notify React.

In HTML5, drag is defined as the movement of data, moving a piece of data to another area, so with this idea, you can realize the operation of dragging a component to the UI canvas

/ / drag
export function drag(type: string, name: string, event: DragEvent<any>) { event.dataTransfer? .setData('component'.JSON.stringify({type, name}));
}
/ / to put down
export function drop(callback: Function, event: DragEvent<any>) {
  event.preventDefault();
  const { type, name } = JSON.parse(event.dataTransfer? .getData('component'));
  callback({
    type,
    name,
    position: {
      clientX: event.clientX,
      clientY: event.clientY
    }
  });
}
Copy the code

The UI Canvas instantiates a component object based on the type and name of the component by resolving the dragged component type and name and adds it to the canvas

export function dropComponentToWebGLEditor(type: string, name: string, position: { x: number, y: number }, editor: CanvasEditorRenderer) {
  const cpn = new (ComponentMap as any) [type][name](position); // Instantiate the corresponding component based on type and name
  editor.addComponent(cpn);
  return cpn;
}
Copy the code

Client UI component implementation

The UI component is still implemented using the WebGL Konva framework and packaged as a Typescript class.

The IWebGLComponentProps interface abstracts the available properties of a component and methods for obtaining and setting properties, such as obtaining and setting location properties, and obtaining and setting background properties. The IWebGLComponentEvents interface abstracts the events that a component needs to bind, such as drag and drop events, selected events, etc. The WebGLComponent class encapsulates the basic structure of WebGL components, such as the children and parent properties that describe the component hierarchy, adding the component to the canvas appendToLayer() method, and implementing the IWebGLComponentProps() interface. Define a WebGL component property that implements the IWebGLComponentEvents interface and defines an event that the component needs to listen for. Each component is implemented by inheriting the WebGLComponent parent class, such as the WebGLRect class and the WebGLText class.

The general logic of a component is implemented by defining a WebGLComponent parent class. The basis of a component is the Group and transformer, which are the shape groups and freely changeable anchors of the canvas rendered to WebGL.

1. Draw UI componentsA UI component consists of several Konva shapes, such as a button component consisting of rectangles (Konva.Rect) and text (Konva.Text). Through to thegroupAdd several shapes to draw a component.

To remove a component, you only need to remove the current component from the parent component, remove the component’s group from the canvas, and remove the transformer of the component from the canvas

removeFromLayer() {  
  this.parent? .removeChild(this.getId());
  this.getGroup().remove();
  this.getTransformer().remove();
}
Copy the code

3. Parent component adds child componentTo add a component to another component simply add that component’sgroupandtransformerMove to the parent component and use it in the child componentparentReference to the parent component, used in the parent componentchildrenStores references to all child components.

Therefore, a hierarchy of parent and child components needs to be established when adding child components

appendComponent(component: WebGLComponent) {
  if (!this.isRawComponent) {
    const group = component.getGroup();
    const transformer = component.getTransformer();
    group.moveTo(this.getGroup()); // Move to the parent component
    transformer.moveTo(this.getGroup()); // Move to the parent component

    if (component.parent) { // Remove the original parent of the child component
      component.parent.removeChild(component.getId());
    }
    component.parent = this; // Redirect to the parent component
    this.appendChild(component); }}Copy the code

Client UI page and JSON conversion

The server uses MongoDB to store an edited UI page, so it is necessary to convert THE UI page to JSON, as well as JSON objects to UI page.

1. The conversion of UI page and JSON object starts from the root component, extracts the type, name, sub-component, style and other attributes, and then recursively parses the sub-component

export function webGLComponentToJsonObject(component: WebGLComponent) :TRawComponent {
  return {
    id: component.getId(),
    type: component.getType(),
    name: component.getName(),
    props: getComponentProps(component),
    children: component.getChildren().size ?
      [...component.getChildren().values()].map(value= > {
        returnwebGLComponentToJsonObject(value); }}) : []; }Copy the code

The breadth-first search is used to traverse the JSON object, instantiate the parent component and the corresponding child component in turn, set the component properties, add the child component to the parent component, record the root node, distinguish whether it is generated in the form of paste, and add it to the canvas

export function drawComponentFromJsonObject(jsonObject: TRawComponent, renderer: CanvasEditorRenderer, isPaste = false) :WebGLComponent {
  let root: WebGLComponent | null = null; // Record the root node
  const queue = [jsonObject]; // Breadth-first search queue
  const map = new Map<string, WebGLComponent>(); // Records whether the current component is instantiated

  while (queue.length) { // breadth-first search
    const front = queue.shift() as TRawComponent; // Record the parent node
    let parent;
    if (map.has(front.id)) { // If the parent component is instantiated, get the instantiated reference directly
      parent = map.get(front.id);
    } else { // If not instantiated, the component is instantiated and recorded in the map
      parent = new (ComponentMap as any)[front.type][front.name](
        front.props.position
      ) as WebGLComponent;
      setComponentProps(parent, front.props); // Set the properties
      map.set(front.id, parent);
    }

    if (root === null) { // Get the root node
      root = parent as WebGLComponent;
      renderer.addRootComponent(root as WebGLComponent); // Draw the root node to the UI canvas
    }

    for (let v of front.children) { // Iterate over the child components
      queue.push(v);
      const child = new (ComponentMap as any)[v.type][v.name](v.props.position, v.props.size) as WebGLComponent;
      setComponentProps(child, v.props);
      renderer.addComponentForParent(parent as WebGLComponent, child); // Draw to the parent componentmap.set(v.id, child); }}const component = root as WebGLComponent;
  // Whether to paste the form
  isPaste && component.setPosition({
    x: component.getPosition().x + 10,
    y: component.getPosition().y + 10}); renderer.getComponentManager().showCurrentComponentTransformer( root? .getId()as string
  );
  renderer.render(); // Re-render the UI canvas
  return component;
}
Copy the code

Client UI component editing function

React communicates with WebGL based on the Redux state tree, by calling Dispatch () on the WebGL side to notify React rendering, and by using useEffect Hooks when rendering the React Editor component.

To implement the edit function, record an edit state in the Redux state tree in the format {id, editType}, where id represents the component id and editType represents the editType.

The React Editor component uses useEffect Hooks to receive changes and uses the edit component methods provided by the CanvasEditorRenderer class to implement component editing

const editToolsDeps = [editToolsState.id, editToolsState.editType];
useEffect((a)= > {
  if (editToolsState.id) {
    const renderer = (webglEditor.current as CanvasEditorRenderer);
    switch (editToolsState.editType) {
      case 'delete': { // Delete the component
        // Remove the component with the corresponding Id from the canvas
        const rmCpn = removeComponentFromWebGLEditor(editToolsState.id, renderer);
        EventEmitter.emit('auto-save', webGLPageState.pageId); // Automatically save
        // Add edit history
        dispatch(addEditHistory(editToolsState.id, 'delete', {
          old: ' '.new: webGLComponentToJsonObject(rmCpn as WebGLComponent)
        }));
        return;
      }
      case 'paste': { // Paste the component
        const newCpn = pasteComponentToWebGLEditor(editToolsState.id, renderer);
        // Add edit history
        dispatch(addEditHistory(editToolsState.id, 'paste', { old: ' '.new: newCpn? .getId() })); EventEmitter.emit('auto-save', webGLPageState.pageId);
        return;
      }
      case 'save': { / / save
        savePage(webGLPageState.pageId, renderer.toJsonObject() as object).then((v: any) = > {
          if(! v.err) { toast('save! '); dispatch(resetComponent()); }});return;
      }
      case 'undo': { / / redo
        dispatch(removeEditHistory());
        dispatch(resetComponent());
        return;
      }
      default: {
        return;
      }
    }
  }
}, editToolsDeps);
Copy the code

1. When a component is removed from the CANVAS and selected from the UI canvas, the selected component ID is stored in the Redux state tree. With the component ID, the method to remove the corresponding ID is called by the CanvasEditorRenderer class

const cpn = this.componentsManager.getComponentById(id);
this.componentsManager.removeComponentById(id);
this.render();
this.dispatch(resetComponent());
return cpn;
Copy the code

2. Copy and paste a component. Record the component ID when copying a component. During pasting, search for the component with the corresponding ID and convert it into a JSON object

if (this.webGLComponentCollection.has(id)) {
  const cpn = this.webGLComponentCollection.get(id) as WebGLComponent;
  const json = webGLComponentToJsonObject(cpn); // Convert to JSON object
  return drawComponentFromJsonObject(json, renderer, true); // Generate a new component from the JSON object
}
return null;
Copy the code

3. The redo component implements the redo component logic by recording an edit history. The edit history is stored in an array. When an edit operation exists, the operation is stored in the array in the format of {ID, operator, data}, where ID indicates the component ID, operator indicates the operation name, and data indicates the data required by the reverse operation of the operator operation. When executing the redo command, the last item of the array is taken out and the corresponding operation of the item is reversed to achieve the redo effect.

Take the example of a paste component operation. Paste a component and add a paste operation to the array

dispatch(addEditHistory(editToolsState.id, 'paste', { old: ' '.new: newCpn? .getId() }));Copy the code

The reverse operation of the paste component operation is to delete the component, so get the ID of the paste component in data and delete it from the UI editor to achieve the effect of redo.

const { id, data } = editHistory.current;
renderer.removeComponent(data.new);
Copy the code

The Client modifies UI component attributes

Through an abstraction of WebGL component style attribute, the background attribute, border attribute, shadow attribute, text attribute and image attribute are abstracted. When modifying the properties of a component, determine the type of property to be modified, and then render the modified properties in the UI canvas. Properties panel when modifying propertiesPropspanelComponents throughdispatch()Modify the state of the Redux state tree, and then redraw the UI. The UIEditor component listens for state changes with the useEffect side effect and callsCanvasEditorRendererOf the classmodifyComponentPropsMethod to modify component properties.

conclusion

This project is my graduation project. I came into contact with rubik’s Cube platform during my internship. The UI editor of Rubik’s Cube platform is realized based on DOM technology, while the UI editor of Figma design software is realized by WebGL.

Existing deficiencies

  • UI components are difficult to implement in WebGL, and there are not many UI components available for implementation, so it is not possible to edit arbitrary UI

  • The object code is not readable when the DSL code compiles object code

  • The volume of packaged application packages is too large

The future planning

  • It will study how to use computer vision and machine learning algorithm to identify UI design draft and convert it into DSL representation of the system, so as to compile it into object code.

  • Explore how to parse the PSD file and convert the PSD into a DSL representation for compilation into object code

Write in the last

Time flies, four years of college life is coming to an end. In the four years of college, some people choose comfort, some choose to give up, but I choose to keep studying hard, leaving no regrets. In the four years of continuous learning, I changed from a computer rookie to a technical expert. I entered Bytedance as an intern and was recommended for admission to graduate school with excellent performance. “God rewards those who help themselves”, learning is not easy at all, in the four years of university study, there are too much happiness and tears, thank you very much for once insisted on their own. In the four years of study, the accumulation of knowledge has made me achieve today’s results. Life is still very long, graduation is not an end, the future in graduate or work still need to continue to work hard. Wish myself a happy graduation!!

reference

Electron development practices at Taro IDE

Share the six months ‘experience in Electron application development and optimization

Konvajs.Konva Tutorials

Github address: github.com/sundial-dre…