Image credit: unsplash.com

Author: Fei Yang

background

Online sample

Demo

As front-end development, especially c-side front-end developers (such as wechat small program), I believe that we have encountered sharing activity pictures, sharing posters similar functions

Generally, the solution to this requirement can be divided into the following categories:

  1. Rely on the server, such as writing a Node service, using puppeteer to access pre-written web pages for screenshots.

  2. Use the CanvasRenderingContext2D API directly or use a drawing aid such as React-Canvas.

  3. Use front-end page screenshot framework, such as HTML2Canvas, dom2Image, write the page structure with HTML, and then call frame API screenshot when necessary

Scheme analysis:

  1. Rely on the service side this scheme will consume a certain amount of server resources, especially the screenshot this service, the CPU and bandwidth consumption is very big, so it is possible that in some high concurrency or picture bigger of the two scenarios in this scheme will experience more bad, wait for a long time, this scheme has the advantage of reduction degree is very high, because the headless browser versions of a service is certain, So you can make sure that what you see is what you get, and there are no other learning costs in terms of development, and this is the most reliable solution if the business is not very large and the traffic is not very high.

  2. This solution is more hardcore and time-consuming, with a lot of code to calculate the placement of the layout, whether or not the text is wrapped, etc., and when the development is done, if there are some subsequent changes to the UI, you have to find the one you want to change in the code. The advantage of this solution is that the details can be controlled, and in theory all functions can be accomplished, if there is enough hair.

  3. This should also be the most widely used scheme on the Web end at present. Up to now, the amount of HTML2Canvas Star has reached 25K. The principle of HTML2Canvas is simply to traverse the attributes in the DOM structure and convert them to canvas for rendering, so it must depend on the host environment, so some old browsers may encounter compatibility problems, of course, it is good if it is encountered in development. After all we are universal front-end development (the head), can pass some hack to avoid, but c end products will run on a variety of equipment, it is hard to avoid after the release of compatibility in other user equipment, and out of the question unless the user, the general difficulty of monitoring, and users in the domestic small program base is very big, This scheme also cannot be used in applets. So this solution seems to be peaceful, but there are some compatibility problems.

In the past few years, I have basically met the need to share pictures in different jobs. Although the demand is not frequent, it is not very smooth in my impression. I have tried the above schemes, and there are more or less some problems.

Come up with ideas:

In a requirement review, I learned that there was a unified UI adjustment plan in the subsequent iterations, and several image sharing functions would be involved. At that time, the business involved small programs and H5. After the meeting, I opened the code, and saw a mountain of code for sharing pictures, interspersed with various compatible glue codes. Such a huge code is just to generate a layout of small cards. If it is AN HTML layout, it should be 100 lines, so I thought about how to reconstruct it.

Considering that there is still plenty of development time, I was wondering if there are other solutions that are more convenient, reliable and universal. Besides, I have always been interested in this area with an attitude of learning, so I came up with the idea of writing a library by myself. After consideration, I chose the implementation idea of React-Canvas. However, React-Canvas relies on react framework. In order to maintain universality, the engine developed by us does not rely on specific Web framework or DOM API, can generate layout rendering based on CSS style sheets, and supports advanced functions for interaction.

After sorting out the functions to be done, a simple canvas typesetting engine comes to mind.

What is a typesetting engine

A layout engine, also known as the Browser engine, rendering engine, or sample engine, is a software component. Take markup content (HTML, XML, image files, etc.), collate information (CSS, XSL, etc.), and output typeset content to monitor or printer. All Web browsers, E-mail clients, e-readers, and other applications that need to present content based on Presentational markup require a typography engine.

Webkit, Gecko, Gecko, webKit, Gecko, etc.

design

The target

The requirements carry the following goals:

  1. The framework supports “document flow layout”, which was our core requirement, without requiring the developer to specify the location of elements and automatic width and height.
  2. Instead of calling a cumbersome API to draw a graph, you can write a template to generate the graph.
  3. Cross-platform, by which I mean the ability to run on the Web and various applets, independent of a particular framework.
  4. Support for interaction, which means you can add events and make changes to the UI.

To sum up, you can write “web pages” in canvas.

API design

Originally, I intended to use a syntax like vue Template for structural style data, but doing so would add compilation costs and was a bit too far from the starting point for the core functionality I wanted to implement. In the end, we decided to use a syntax like React createElement and an API in the form of a Javascript style object, prioritizing core functionality.

Also note that our goal is not to implement browser standards in a Canvas, but to provide a solution for document flow layout that is as close to THE CSS API as possible.

The target API looks like this

// Create a layer
const layer = lib.createLayer(options);

// Create a node tree
// c(tag,options,children)
const node = lib.createElement((c) = > {
  return c(
    "view"./ / the node name
    {
      styles: {
        backgroundColor: "# 000".fontSize: 14.padding: [10.20],},/ / style
      attrs: {}, // Attributes such as SRC
      on: {
        click(e) {
          console.log(e.target); }},// events such as Click Load
    },
    [c("text", {}, "Hello World")] / / child nodes
  );
});

// Mount the node
node.mount(layer);
Copy the code

As shown above, the core of the API is the three parameters that create the node:

  1. tagNameNode names, here we support basic elements likeview.image.text.scroll-viewEtc., also supports custom tags through globalcomponentAPI to register a new component to facilitate expansion.
function button(c, text) {
  return c(
    "view",
    {
      styles: {
        // ...
      },
    },
    text
  );
}

// Register a custom tag
lib.component("button".(opt, children, c) = > button(c, children));

/ / use
const node = lib.createElement((c) = > {
  return c("view", {}, [c("button", {}, "This is the global component")]);
});
Copy the code

  1. Options (tag parameters) support styles,attrs, and on for styles, attributes, and events, respectively

  2. Children, children nodes, can also be words.

We expect the above API to render text in the Canvas and respond to events when clicked.

Process architecture

The first rendering of the frame will follow the following process, which will be explained in this order:

The key details in the flowchart will be described below. Some algorithms and data structures involved in the code need to be paid attention to.

The module details

pretreatment

Once the viewmodel is in hand (that is, the model written by the developer through createElementapi), it needs to be preprocessed first. This step is to filter the user input. The user input model only tells the framework the intended target, and cannot be used directly:

  1. Node preprocessing

    • Support for short string, this step requires converting the string toTextobject
    • Since we will need to visit sibling and parent nodes frequently later, it is important to store both sibling and parent nodes in the current node and mark their location in the parent container. This concept is similar to that in ReactFiberStructure, which is frequently used in subsequent calculations, and implemented for usInterruptible renderingLaid the foundation.

  2. Style preprocessing

    • Some styles support multiple abbreviations and need to be converted to target values. Such asPadding: [10, 20]In the preprocessor, you need to convert topaddingLeft,paddingRight,paddingTop,paddingBottomFour values.
    • Set the node default values, as shown inviewThe default nodedisplayProperties forblock
    • Inheriting value processing, such asfontSizeProperty inherits from the parent by default

  3. Outlier processing, the user fills in a value that does not meet the expectation is notified in this step.

  4. Initialize event mounts, resource requests, and so on.

  5. Other preparatory work for subsequent calculations and rendering (described below).

initStyles() {
    this._extendStyles()

    this._completeStyles()

    this._initRenderStyles()
}
Copy the code

Layout processing

After the pre-processing in the previous step, we get a node tree with a complete style. Then we need to calculate the layout, which is divided into the calculation of size and position. What we need to pay attention to is, why do we calculate the size first in the process? Think carefully, if we calculate the position first, such as text, picture and other nodes after, we need to refer to the calculation after the calculation of the position of the previous size. Therefore, this step is to calculate the location of all nodes after the size of all nodes is calculated in situ.

The whole process is animated as follows.

Computing size

A more professional statement should be the computing box model, talking about the box model we should be familiar with, basic interview almost must ask.

Image: mdn.mozillademos.org/files/16558…

In CSS, you can use the box-sizing property to use different box models, but we don’t support resizing this time. The default is border-box.

For a node, its size can be simplified into several cases:

  1. Refer to the parent node, as inwidth:50%.
  2. A specific value is set, as shown inwidth:100px.
  3. Refer to child nodes, such aswidth:fit-contentAnd likeimage textNodes are also sized by content.

Once we’ve sorted out these patterns, we can start traversal calculations, and we have multiple traversal patterns for a tree.

Breadth-first traversal:

Depth-first traversal:

Here we consider the above cases separately:

  1. Because we’re referring to the parent node we need to go from parent to child.
  2. There is no traversal order requirement.
  3. The parent node needs to wait for all the children nodes to complete the calculation, so it requires breadth-first traversal and is from child to parent.

There was a problem here, 1 and 3 species needed to traverse the way the conflict, but looking back on it traverse the preprocessing part of the father to son, so the part 1, 2, calculate the size of the task can be calculated in preprocessing part ahead of schedule, it arrived at this step only needs to calculate part 3, which calculated according to the child node.

class Element extends TreeNode {
  // ...

  // The parent calculates the height
  _initWidthHeight() {
    const { width, height, display } = this.styles;
    if (isAuto(width) || isAuto(height)) {
      // This step needs to be traversed
      this.layout = this._measureLayout();
    }

    if (this._InFlexBox()) {
      this.line.refreshWidthHeight(this);
    } else if (display === STYLES.DISPLAY.INLINE_BLOCK) {
      // If inline-block is used, only height is calculated
      this._bindLine(); }}// Calculate its own height
  _measureLayout() {
    let width = 0; // Need to consider the original width
    let height = 0;
    this._getChildrenInFlow().forEach((child) = > {
      // calc width and height
    });

    return { width, height };
  }

  // ...
}
Copy the code

The code iterates through the direct child nodes in the document stream to add the height and width. In addition, when there are multiple nodes in a row, such as inline-block and Flex, the Line object is added to help manage the objects in the current row. The child node is bound to a row instance until the Line instance reaches its maximum limit and cannot be added. When the parent node calculates the size, it reads the Line instance directly.

The _measureLayout method is overridden by the Text Image and other nodes that have their own contents. Text internally calculates the width and height of the newline, and Image internally calculates the scaled size.

class Text extends Element {
  // Calculate the size of the newline based on the set text size, etc
  _measureLayout() {
    this._calcLine();
    return this._layout; }}Copy the code

Calculation of position

After calculating the size, the position can be calculated. The traversal method here requires breadth-first traversal from parent to child. For an element, the position of itself can be determined as long as the position of the parent element and the position of the previous element is determined.

In this step, you only need to consider the location of the parent node based on the location of the previous node, or the nearest reference node if it is not in the document flow.

The complexity is that if it is a node bound to a Line instance, the calculation is done inside the Line instance, and inside the Line the calculation is similar, but with additional logic such as alignment and Line wrapping.

// The code only preserves the core logic
_initPosition() {
    // Initializes the CTX position
    if (!this._isInFlow()) {
      // Not processed in the document stream
    } else if (this._isFlex() || this._isInlineBlock()) {
      this.line.refreshElementPosition(this)}else {
      this.x = this._getContainerLayout().contentX
      this.y = this._getPreLayout().y + this._getPreLayout().height
    }
  }
Copy the code
class Line {
  // Calculate alignment
  refreshXAlign() {
    if (!this.end.parent) return;
    let offsetX = this.outerWidth - this.width;
    if (this.parent.renderStyles.textAlign === "center") {
      offsetX = offsetX / 2;
    } else if (this.parent.renderStyles.textAlign === "left") {
      offsetX = 0;
    }
    this.offsetX = offsetX; }}Copy the code

Once this is done, the layout processor is done, and the framework feeds the nodes into the renderer for rendering.

The renderer

There are several steps to draw a single node:

  • Draw the shadow, because the shadow is on the outside and needs to be drawn before clipping
  • Draw clipping and borders
  • Draw the background
  • Draws the child nodes and their contents, as shown in theText 和 Image

For rendering a single node, the function is normal, the renderer basic function is based on the input to draw different graphics, text, images, so we only need to implement these apis, and then the style of the node through these apis in order to render out here and he said to the order, then render this step we should be in what order? Here’s the answer depth-first traversal.

In default composition mode, canvas is drawn at the same position, and the post-rendered one will be overwritten on it, that is to say, the post-rendered node has a larger Z-index. (Due to complexity, there is currently no implementation like browser compositing layer processing, so manual setting of z-index is not supported for now.)

In addition, we also need to consider how to achieve overflow:hidden effect. For example, rounded corners, we need to crop the content beyond the canvas, but it is not suitable to crop only the parent node, in the browser, the parent node clipping effect can take effect on the child node.

A full clipping procedure call in canvas looks like this.

// save ctx status
ctx.save();

// do clip
ctx.clip();

// do something like paint...

// restore ctx status
ctx.restore();
//
Copy the code

Remember that the state in CanvasRenderingContext2D is stored as a stack data structure, and when we execute save multiple times, each time restore is restored to the most recent state

This means that the clipping of the parent node will be clipped only after the clip is clipped to the restore node. Therefore, if clipping of the parent node is to take effect for the child nodes, we cannot restore the node immediately after rendering. We need to wait until the inner child nodes are all rendered.

The following is illustrated by pictures

As shown, the numbers are rendered in order

  • Draws node 1, which cannot be restored immediately because there are child nodes
  • Draw node 2, with children, and draw node 3, which has no children, so execute restore
  • Draw node 4 with no child nodes and execute restore. Note that all nodes in node 2 have been drawn, so you need to execute restore again to restore the drawing context of node 1
  • If node 5 has no child nodes, run restore. At this point, all nodes in node 1 are drawn, and run restore again

Since we have implemented the Fiber structure in preprocessing and know the location of the parent node, we only need to determine how many times restore needs to be called after each node is rendered.

So far, after a long debug and reconstruction, the input node has been rendered normally, and what needs to be done is to add support for other CSS properties. At this time, I am very excited, but looking at the output of the rendering node in the console, I always feel that there is something more to be done.

Right! The model of each graph is saved, so can we modify and interact with these models? First, set a small goal to achieve the event system.

Event handler

Graphics in canvas cannot respond to events as DOM elements do. Therefore, dom events need to be proxied to determine the location of events on canvas and then distributed to the corresponding Canvas graphics node.

If we follow the conventional idea of event bus design, we only need to store different events in different List structures, and traverse whether the judgment point is in the node area when the trigger is triggered. However, this scheme definitely does not work, and the reason is the performance problem.

In the browser, event trigger into the capture and bubbling, that is to say, according to the hierarchy of nodes from top to bottom to perform first capture, touched the deepest node, and then in the opposite order bubbling process, cannot satisfy the List structure, traverse time complexity of the data structure will be very high, reflect on the user experience is operating with a delay.

After some brainstorming, I realized that events can also be saved in tree structure. The nodes monitored by events can be extracted to form a new tree, which can be called “event tree”, rather than stored in the original node tree.

As shown in the figure, mounting click events on nodes 1, 2, and 3 will generate another callback tree structure in the event handler. During the callback, only this tree needs to be traversed, and pruning optimization can be carried out. If the parent node does not trigger, the child elements under the parent node do not need to be traversed, improving performance.

Another important point is to determine whether the event point is inside the element. There are many mature algorithms for this problem, such as ray method:

Time complexity: O(n) Range of application: any polygon

Algorithm idea: taking the measured point Q as the endpoint, ray is taken in any direction (generally, ray is taken horizontally to the right), and the number of intersections between the ray and the polygon is counted. If it is odd, Q is in the polygon; If it is even, Q is outside the polygon.

However, for our scene, except for rounded corners, all of them are rectangles, and it is troublesome to process rounded corners. Therefore, rectangles are used for judgment in the first version, and then used as optimization points for improvement.

This is how we can implement our simple event handler.

class EventManager {
  // ...

  // Add event listener
  addEventListener(type, callback, element, isCapture) {
    // ...
    // Construct the callback tree
    this.addCallback(callback, element, tree, list, isCapture);
  }

  // The event is triggered
  _emit(e) {
    const tree = this[`${e.type}Tree`];
    if(! tree)return;

    /** * traverses the tree to check for callbacks * If the parent is not triggered, the children do not need to check either, skip to the next sibling node * perform the Capture callback to add the ON callback to the stack */
    const callbackList = [];
    let curArr = tree._getChildren();
    while (curArr.length) {
      walkArray(curArr, (node, callBreak, isEnd) = > {
        if (
          node.element.isVisible() &&
          this.isPointInElement(e.relativeX, e.relativeY, node.element)
        ) {
          node.runCapture(e);
          callbackList.unshift(node);
          // Nodes of the same level do not need to execute
          callBreak();
          curArr = node._getChildren();
        } else if (isEnd) {
          // The last one is still not detected, endcurArr = []; }}); }/** * execute on callback from child to parent */
    for (let i = 0; i < callbackList.length; i++) {
      if(! e.currentTarget) e.currentTarget = callbackList[i].element; callbackList[i].runCallback(e);// Handle prevent bubbling logic
      if (e.cancelBubble) break; }}// ...
}
Copy the code

After the event handler is completed, a scrollview can be implemented. The internal implementation principle is to use two views, the external width and height are fixed, and the internal can be split. The external event handler registers events to control the rendered transform value. The position of the child element is not the original position, so if the event is mounted on the child element, it will be offset. Here, the corresponding capture event is registered in the Scroll View. After the event is passed into the scroll View, the relative position of the event instance is modified to correct the offset.

class ScrollView extends View {
  // ...

  constructor(options, children) {
    // ...
    // Initialize a scrollView inside with adaptive height and fixed width and height
    this._scrollView = new View(options, [this]);
    // ...
  }

  // Register events for yourself
  addEventListener() {
    // Register the capture event, modify the relative position of the event
    this.eventManager.EVENTS.forEach((eventName) = > {
      this.eventManager.addEventListener(
        eventName,
        (e) = > {
          if (direction.match("y")) {
            e.relativeY -= this.currentScrollY;
          }
          if (direction.match("x")) {
            e.relativeX -= this.currentScrollX; }},this._scrollView,
        true
      );
    });

    // Handle the scroll
    this.eventManager.addEventListener("mousewheel".(e) = > {
      // do scroll...
    });

    // ...}}Copy the code

Rearrangement to redraw

In addition to generating static layout functions, the framework also has a redraw and rearrange process, which is triggered when the node attributes are modified. SetStyle and appendChild apis are provided to modify the style or structure, and the rearrangement is determined based on the attribute values. For example, changing width triggers redraw after rearrangement. Changing the backgroundColor will only trigger redrawing. For example, when the scrollview changes the transform value, it will only redraw.

compatibility

Although the framework itself does not rely on DOM and is drawn directly based on CanvasRenderingContext2D, compatibility is still required in some scenarios. Here are some examples.

  • The drawing picture API of wechat small program platform is different from the standard, so the platform is determined in the image component. If it is wechat, the specific API of wechat is called to obtain the image
  • Font thickness set by wechat small program platform does not take effect on iOS real phone. After internal judgment of platform, text will be drawn twice, and the second time will be offset on the basis of the first time to form bold effect.

Custom rendering

Although the framework itself already supports the layout of most scenarios, the business requirement scenarios are complex and changeable, so it provides the ability of custom drawing, that is, only the layout is carried out, and the drawing method is left to the developers to call by themselves, providing higher flexibility.

engine.createElement((c) = > {
  return c("view", {
    render(ctx, canvas, target) {
      // Here you can get CTX and layout information for developers to draw custom content}}); });Copy the code

Used in the Web framework

Although the API itself is relatively simple, it still requires some repetitive code, which is not easy to read when the structure is complex.

When used in modern web frameworks, corresponding framework version can be used, such as the vue version, internal vue node will be converted to API calls, to use will be more easy to read, but need to pay attention, due to internal node conversion process, compared with the direct use of there will be a loss of performance, the difference is more apparent in the complex structure.

<i-canvas :width="300" :height="600">
  <i-scroll-view :styles="{height:600}">
    <i-view>
      <i-image
        :src="imageSrc"
        :styles="styles.image"
        mode="aspectFill"
      ></i-image>
      <i-view :styles="styles.title">
        <i-text>Hello World</i-text>
      </i-view>
    </i-view>
  </i-scroll-view>
</i-canvas>
Copy the code

debugging

In view of simple service scenarios, the framework provides basic debugging tools. You can set the debug parameter to enable debugging of node layout. The framework draws the layout of all nodes. More comprehensive visual debugging tools will be provided after the core functions are improved.

results

After first-hand experience, the average page development is as efficient as writing HTML. To demonstrate the results, I’ve written a simple component library demo page.

The source code

Component library Demo

performance

The framework has achieved good performance after several refactorings, as shown below

Optimizations that have been made:

  • Optimization of traversal algorithm
  • Data structure optimization
  • Scroll view redrawing optimization
    • Scroll view redraws only elements within the range
    • Elements outside the viewable range of the scroll view will not be rendered
  • Image instance cache, although there is HTTP cache, but for the same image will generate multiple instances, internal instance cache

To be optimized:

  • Interruptible render as we have implemented similarFiberStructure, so later need to add this feature is also more convenient
  • The preprocessor also needs enhancements to improve compatibility with user-input styles and structures, and to increase robustness

conclusion

From the beginning, I wanted to achieve a simple image rendering function, and finally realized a simple Canvas layout engine. Although there are limited features and many details and bugs to be fixed, I still have the basic layout and interaction ability, but I still stepped in many holes and reconstructed many times. I can’t help but marvel at the power of the browser layout engine. And also realized the charm of algorithm and data structure, good design is the cornerstone of high performance, good maintenance, but also a lot of fun.

In addition, I think this mode still has a lot of imagination after improvement. In addition to simple picture generation, it can also be used for h5 game list layout, table rendering of massive data and other scenes. In addition, I have another idea in the later stage. So we want to separate the layout and the function of calculating newlines, zooming and so on into a single tool library, and integrate with other libraries for rendering.

My ability to express myself is limited, there may still be a lot of details have not been clarified, but also welcome everyone’s comments and exchanges.

Thank you for reading

This article is published from NetEase Cloud Music big front end team, the article is prohibited to be reproduced in any form without authorization. Grp.music – Fe (at) Corp.Netease.com We recruit front-end, iOS and Android all year long. If you are ready to change your job and you like cloud music, join us!