demand

Examples of use of UI components within the company have always shown code only in markDown format, and in more complex components the sample code can be too verbose to use a file directory structure to show code. So there has always been a need for this project:

  • Sample code for components can be organized and presented in a directory structure
  • Users can modify the sample code and see its effects in real time

First, analyze the requirements: together, the two are an online development platform for front-end developers, similar to Codepen, but better integrated into the current site.

Why do we need to implement such tools ourselves?

Now there are many tools, such as WebPackbin, codeSandBox, JsFiddle, Jsbin, etc. In addition to the support of embedding mode is not very good, the biggest problem is that our component is not open source, that is, the compilation process needs our own control, at this point we decided to develop by ourselves.

Identify the main ideas


Ugly interaction diagram

By default, only code can be displayed and files can be switched. When the editor changes to writable mode after clicking the Edit button and generates a unique link to save and display user code, CMD/CTRL +S and clicking the Save button both trigger the compilation process and asynchronously save the latest code to the database.


Technical solution

The overall data flow is shown in the figure above. The following points are deeply felt during the whole development process:

  • Why do YOU need to manage front-end state?
  • Where do you store the user’s temporary files for a more efficient compilation process?
  • Error handling: Compile-time errors should be able to be displayed on the web page
  • How do I determine the data capture of the code area based on the component page currently being viewed?
  • .

Why do YOU need to manage front-end state

When I set up the basic environment at the beginning, I did not directly add this part into the project. When it comes to development, the simpler the requirements are, the better. However, I later found that if I wanted to get/share/listen for a state globally, such as whether the current page is editable or not, it would affect whether the button in the upper right corner shows Edit or ‘save’ and whether the edit area can be read or written. Whether the current page has data, no need to specify the interface pull. Implement it yourself, such as an event mechanism, or a publish-subscribe model, to communicate between different components. However, when there are too many states that need to be monitored, the code used for monitoring is scattered in various components, and sometimes I can’t figure out which component to modify.

This is the development of the time are masked, later maintenance modification is more difficult, it seems that we need a tool to help us centralized management of the page state.

mobx VS redux

Because I have known about Redux before, redux is the first thing that comes to mind. However, when we want to write actions, reducer needs to connect components and states and inject corresponding states into components, which is really too much trouble. As it happens, I searched for other tools besides Redux and found mobx, which has a very simple syntax. Then I looked through its documentation and found that mobx is also relatively easy to use, and the subscription-based update mechanism for modifying affected nodes is very efficient. (PS: Specific use of numerous online resources, here will skip)

I think it would be irresponsible to say mobx is better than Redux, and anyone who talks about technology outside the context of use is a double rogue. So what’s the difference between the two? After my experience and thinking in using mobx, it is based on variable data and suitable for small projects and teams. Redux is based on immutable data, and its state is predictable. It is suitable for large projects, collaborative development of many people, and the relatively tedious process is to restrict or restrain some illegal activities. As in the real world, organizations of different sizes are bound to have different management styles. So when we make this kind of technical choice, if it is very tiring to use a tool in a small project, there is a high probability that it is the wrong choice.

Conclusion: The so-called state management is actually to achieve simple and convenient communication between components in today’s highly modular and componentized applications.

Where do you store the user’s temporary files for a more efficient compilation process?

Each time the user modifies code to trigger a save operation (CMD +S or click ‘save’ button), the code is submitted, saved to a temporary folder, and the compilation process begins. Every time you submit code, you save it on hard drive, right? Is too slow, especially when multiple users to edit at the same time, think about this process, a user to edit the code, create an own folder, the user submits the code, the background, receipt of a request to create the file and write the content, compilation, and generating two files (HTML, bundle. Js), written to disk again, then the program read the content is sent back to the page, Multiple read/write operations are involved, and all operations are limited by the disk read/write speed. Slow speed and high overhead.

If there was ever a good way to do it, all the others would have to do.

Consider the webpack-dev-server setup. The entry files are read locally, but the compilation results are placed in memory. Since each save triggers a compilation, temporary package files are constantly generated.

  • Since memory is faster than reading and writing from hard disk, the difference may not be significant for a single local user, but will be significant for multiple users working at the same time once the deployment is online.
  • There’s no need to add.gitignore, there’s nothing to clean up when the application dies and restarts.
  • Don’t worry too much about running out of memory. A file with hundreds of megabytes may not fit anywhere.

So we need entries (user-submitted code files) and outputs based on memory, but loaders are found locally in node_modules.

Webpack provides a custom file System, so the configuration is as follows:

compiler.inputFileSystem = memoryFs; compiler.outputFileSystem = memoryFs; compiler.resolvers.normal.fileSystem = memoryFs; / / the context need consistent with inputFileSystem compiler. Resolvers. Context. The fileSystem = fs; compiler.resolvers.loader.fileSystem =fs;Copy the code

Node_modules () : node_modules (); Node_modules is a large file, and it is not recommended to put it in memory if the machine configuration is limited, so the current problem is that the files to be compiled are stored in memory, but the dependencies are stored on physical hard disk. After a bit of exploration, I came up with an answer that at least solved the problem: rewrite the memory-FS method of reading files, and when you can’t find it, go to the physical hard drive. See here.

When only reacting components were compiled, the speed was acceptable in less than 2s, but when we compiled components based on our own company, time exploded. It takes 10+s every time, and that’s even though we pre-typed all the third-party packages into the DLL file. So why is it so slow? There are so many dependencies in our package that we have to switch from memory-FS to physical disk when we need to find dependencies at compile time. During a component build, we have to switch from memory-FS to physical disk over 5700 times! Maybe we can figure out what the program is looking for when it keeps switching file systems, and maybe we can shorten the time by helping it make path choices ahead of time. However, we will not consider all cases, and the decision on the path can only be made at a superficial level. To do this, we need to think differently.

When I was at a loss, one of my classmates mentioned Redis, which could not meet our needs, because in order to meet the compilation of Webpack, we had to write the user’s content into the file, so we could only store the file. However, Redis gave me some ideas. Why does caching remind him of Redis? It realizes data cache based on memory, extremely fast. Remember, memory-based reading and writing files is hundreds or even thousands of times faster than SSD. So the focus of the project is to implement memory based read and write! Why am I holding onto webpack’s js implementation of memory -fs? After opening up my ideas, I started to browse various articles about memory. Finally, TMPFS of Linux became my target. Based on this, we can directly use the memory system provided by the system to carry out read and write operations.

So my final solution is: Linux-based TMPFS can read and write temporary files of the user, as long as the entry and output are set to the path of the TMPFS, and Webpack is not aware of this. The only possible caveat is that when configuring Webpack, especially plugins in Options, you need to use require.resolve(” “), otherwise you will look in the entry path, which you will certainly not find. For example, with Babel:

module:{
    rules:[
        {
            test: /\.(jsx|js)$/,
            exclude: /node_modules/,
            use: [{
                loader:'babel-loader?cacheDirectory=true',
                options: {
                    presets: ['babel-preset-es2015', 'babel-preset-react','babel-preset-stage-0'].map(require.resolve),
                    plugins:[[require.resolve('babel-plugin-transform-react-jsx'),{
                      pragma: "require('react').createElement"
                     }]]
                }
            }]
        }
    ]
}
Copy the code

Tips: TMPFS is a virtual memory file system with storage space in the VM, consisting of real Memory (RM) and swap. RM is the physical memory, and swap is the memory space created by hard disks. The read/write speed is much slower than RM. When there is not enough RM, the data that is not commonly used in RM will be exchanged to Swap. When used again, the data will be exchanged to RM again. Example Add a Swap partition. The configured size of TMPFS is only the maximum occupied space. The actual occupied space is dynamically adjusted based on usage. TMPFS is half RM by default. 1. Run mount -t TMPFS -o size=500m TMPFS/TMP. 2. Write /etc/fstab to the /etc/fstab file.

TMPFS is Linux-only, and there are no direct commands available on the MAC, but clever programmers have already achieved the feat of ‘curving the nation’.

Error handling: Compile-time errors should be able to be displayed on the web page

Continue our compilation process, when the user submits the code, we store it in a temporary folder, start compiling, and compile the result as the response of the code submitted by the user. When the compilation is correct, the HTML file will be generated, and we can directly return it to the user. What if there is an error during compilation? The program should not hang at this point, and should return an HTML containing error information as normal. The compiler module in Webpack can be found in the official website

// During the compilation of webpack, Return new promise (function(resolve, reject) {compiler.run(function(err, stats) { if (err || stats.hasErrors()) { resolve(stats.compilation.errors); } else { resolve(stats.compilation.assets); }}); });Copy the code

Since compilation errors return only error messages, not HTML, we added an HTML template to the playground in order to display it correctly in the preview area:

export default function generateErrorTemplate(err) { const strToHtml = str => { return (str || "") .replace(/&/g, "&") .replace(/</g, "<") .replace(/>/g, ">") .replace(/"/g, """) .replace(/'/g, "'") .replace(/\[(\d+)m/g, "") .replace(/ /g, " ") .replace(/\n/g, "<br />"); }; let template = ` <! DOCTYPE html> <html> <head> </head> <body> <div> ${strToHtml(err.toString()) || ""} </div> </body> </html>`;  return template; }Copy the code

In routerHandler: Response.html = generateErrorTemplate(err);

How do I determine the data capture of the code area based on the component page currently being viewed?

The main task of this project is to implement the display in embedded mode, so more effort will be put into integrating playground with existing websites. It’s not hard to imagine that our playground is embedded as an iframe in the master site, but how can we automatically retrieve data (i.e., code) in the playground based on the page opened by the master site?

When website B is opened in the form of iframe in website A, there will be referer in the request header when the data is requested from website B, indicating the url of the invoked website. According to this information, we can obtain useful information in the URL of the host page through regular matching. Then the corresponding data can be obtained from B website according to this parameter. (It’s easier here, not to show the code)

When multiple users, how to minimize the memory footprint of the machine?

As mentioned earlier, we saved the user-submitted code in a folder mounted under TMPFS, that is, in memory. Memory based read and write efficiency is very high, of course, the space is also precious, we should promptly and efficiently clean up the useless data in this folder. Each user will create a folder named Unique ID when entering the page of different components to modify the code. When the content in this folder is not active (understand that the user is offline, because the system is for internal use and is not currently added to the user management system), this folder will be deleted. Since no session was added to the project, set a timer to delete the folder 30 minutes after the last compilation. (PS: But also because of this design, when the user changes the code submission, we must submit the folder in full, otherwise there will be an error when the user leaves the page for a period of time and submits the code again. Of course, because the sample code for each component is no more than a few folders, tests found that full commit code and incremental commit modified code files had no more than a few tens of milliseconds of impact on the overall compile time, thanks to fast memory-based reads and writes. When designing the delete task, record the delete task separately for each ID. It is better to create an object with gcTask={}, take unique ID as the key, and set the scheduled task as the value. Delete the key by delete gcTask[id] after the task result. To lighten the load on the global object gcTask.

In addition, within each folder, the user may perform multiple save operations on a component, generating multiple compilations of bundle.[hash].js. For this, it is best to add the clean-webpack-plugin to the webpack.config.js control compiler, New CleanWebpackPlugin(‘ ${tmpPath}/${ID}/bundle.*.js’,cleanOptions) removes the previous compilation results before each compilation. As there is no perfect in the world, such a brutal deleting files, will make the last no longer compile results cache is available, such as only modify the code, compile after error, undo modify, compile, again is such a case should be able to directly use the compilation of the results, just use this scenario probability is low, and less affected, so on balance, Choose to do your best to free up the machine’s memory.

How do I update the preview area?

Because playground was designed without the preview area as a separate app, it is only accessed as a secondary link to the Playground app. The only way to update the contents of the preview area is to update the child IFrame.

refreshIframe(html) {
    let frame = document.querySelector("#preview-iframe");
    let iframe =
        frame.contentWindow ||
        frame.contentDocument.document ||
        frame.contentDocument;
    iframe = iframe.document;
    iframe.open("text/htmlreplace");
    iframe.write(html);
    iframe.close();
Copy the code

This kind of update always brings a flash because the content is cleared and then rewritten. If you separate the iframe section as an application, you can directly modify the content in the preview area with document.body. HTML =toPreviewHtml without flickering after receiving the content to be updated via postMessage or other methods. Loading animations are currently the only way to fix the blinking problem caused by blank pages in the preview area.

A complement to the integration-time problem

  • Initialize the sample code to the database

    The data in the playground is all read from the database. The user’s modified code is saved directly to the database when submitted, but what about the code initially displayed when the component page opens? This code is a sample code written by the UI team to help developers understand component usage. It is displayed by default every time a component is used from the site, so you need to read the sample component code from the Git repository and save it to the database when the site is launched.

  • Automatically update the sample code via Webhook

    The data in the playground should be automatically updated as the UI team updates the sample components. Webhook is used to dynamically track the repository where the sample code resides and update the code of the component in the database after the sample code of the component changes.

  • Configure the dependency files of the project to be compiled separately

    Since the embedded playground is used to compile the internal components of the company, in order to prevent the dependencies of these components from colliding with those of developing playground, the dependency files used for compilation are separated into a folder. Prevent some problems caused by different versions of the same package.json. For example, I used react-router-dom@4 for Playground, but the company’s front-end components relied on react-router@3.

  • A separate implementation of the file directory in Playground: building an online compiler for the file directory operation

Write in the last

This is a very interesting project, and there is still a lot of work to be done, such as monitoring the running status of the project (memory footprint during concurrency, etc.), database read and write efficiency and space footprint, further improvement of compilation speed, etc.

PS: The playground was developed based on React + MOBx + KOA2 +webpack3, and a set of scaffolding koa-React-scaffold was also extracted. The configuration of Webpack has been optimized as far as possible (including DLL, distinguish dev and PROD mode), all the code (including node part) can be written in ES6, also includes the CRUD method for Mongoose, if you like, welcome to use and star~