It was first posted on my blog github.com/mcuking/blo…

background

In the past few months, I have been working on a project to share components/blocks across front-end projects in the company. The main idea is to develop on the basis of Bit. The main purpose of Bit is to realize different projects to share and synchronize components/blocks. The general idea is as follows:

In project A, the source code of the components/blocks that need to be shared can be pushed to the remote repository by executing the command line tool provided by Bit. Then in project B, the components/blocks stored in the remote repository can be pulled by using the command line tool provided by Bit. It sounds like Git, but the main difference is that in addition to pushing source code, Bit will also include component dependency map analysis, component version management and other functions. The following figure illustrates the implementation of Bit. For more details, see the official Bit documentation, Bit-docs

While Bit has open source command line tools, there is no open source shared component/block site like bit.dev. This means that the user cannot find the component/block code stored in the Bit remote repository by browsing the component/block’s built view. Bit website effect as shown below:

The next step is to implement a similar site of your own, and the hardest part is to implement an online IDE that displays component/block code and supports real-time build of code and capture of screenshots of built pages. The effect is shown below:

Problems with using the currently available online IDES

Looking at this, you may be wondering why you can’t just use an existing free online IDE. Examples include CodeSandbox, CodePen, Stackblitz, etc. The main reasons are as follows:

  1. Companies of a modest size will have their own proprietary NPM sources, and these NPM packages are not available to an online IDE;

  2. Specific configurations in front end project builds that are not supported by existing online ides;

    CodeSandbox, for example, can only set the type of build template –create-react-app, etc., and does not provide an EXTERNAL API to modify the specific build configuration. Given that your project uses less files, choosing the create-react-app template will not build that type of file.

  3. Special functions can not be realized, for example, click the button of the page, you can realize the screenshot of the page built on the right side of the online IDE, and transfer the picture data out;

  4. Using services provided by an online IDE generally means that your components/blocks are exposed to the public network, however there may be some code that is confidential and cannot be uploaded to the public network.

  5. Some build tools rely on files such as node_modules and won’t work in browsers without node_modules. For example, the Babel plug-in. This will be explained later in the customizing CodeSandbox feature section as an example.

So we need to build our own online IDE to solve the problems mentioned above. There are two options: develop an online IDE entirely from scratch, or find an open source project and customize it from there.

At the beginning, THE author chose to develop the IDE by himself, but after a period of development, I found that the IDE spent a lot of energy to achieve compared with the existing products, both in terms of function richness and ease of use, it was completely defeated. In addition to the fact that my main goal is to implement a platform for reuse of blocks across front-end projects, an online IDE is just a non-essential component (note: it is possible to display the shared component/block source code directly on the page, distinguished by the component/block name, although this is really low). In the end, I chose to secondary develop on the basis of an already open source online IDE.

CodeSandbox fundamentals

The author mainly studies Codesandbox and Stackblitz. Both of these are commercial projects. The core of Stackblitz is not open source, while most of the functions of CodeSandbox are open source, so CodeSandbox is the final choice.

For the sake of following up on how to customize and deploy CodeSandbox, here’s a brief overview of its fundamentals. Part of the previous article) :

The main feature of CodeSandbox is that it uses browser-side project building, which means that packaging and running do not depend on the server. Since there is no Node environment on the browser side, CodeSandbox has implemented a simplified version of WebPack that runs on the browser side.

CodeSandbox component

As shown in the figure below, CodeSandbox mainly contains three parts:

  • Editor: Mainly used to edit code, notifying Sandbox of code changes for translation

  • Sandbox code run Sandbox: Run in a separate IFrame that is responsible for compiling Transpiler and running Evaluation of code

  • Packager NPM online packer: Provides Sandbox with file contents from NPM packages

CodeSandbox Build project process

The build process consists of three main steps:

  • Packager– NPM package packaging phase: Download the NPM package and recursively find all referenced files, which are then provided to the next phase for compilation

  • Transpilation– Compilation phase: All code is compiled and the module dependency graph is built

  • Evaluation– Execution phase: Run the compiled code using Eval to implement a project preview

Packager– NPM package packaging phase

The code for the Packager phase is implemented in CodeSandbox’s repository dependency- Packager hosted on GitHub, a service based on the Express framework, And deployment with Serverless(based on AWS Lambda) makes the Packager service more scalable and flexible to cope with high concurrency scenarios. (Note: In a private deployment without a Serverless environment, you can comment out all AWS Lambda parts in the source code.)

The React package is used as an example to explain how the Packager service works. The Express framework receives the package name and package version in the request, for example, [email protected]. Use YARN to download the React and react dependent packages to the disk. Read the browser, module, main, and unpkg fields in the package.json file of the NPM package to find the NPM package entry file. All require statements in the AST are then parsed, the contents of the required file are added to the manifest file, and the previous steps are performed recursively to form the dependency graph. This achieves the goal of transferring the contents of the NPM package file to manifest.json, and also achieves the goal of weeding out unwanted files from the NPM module. Finally, it returns to the Sandbox for compilation. Here is an example of a manifest file:

{
    // Module contents
    "contents": {
        "/node_modules/react/index.js": {
            "content": "'use strict'; ↵ ↵ if...".// The content of the code
            "requires": [ // Other dependent modules
                "./cjs/react.development.js",]},/ /...
    },
    // Module installation version
    "dependencies": [{
        name: "@babel/runtime".version: "7.3.1"
    }, / *... * /].// Module alias, such as react as preact-compat alias
    "dependencyAliases": {},
    // Dependent dependencies, i.e., indirect dependencies on information. This information can be obtained from yarn.lock
    "dependencyDependencies": {
        "object-assign": {
            "entries": ["object-assign"].// Module entry
            "parents": ["react"."prop-types"."scheduler"."react-dom"]./ / parent module
        }
        / /...}}Copy the code

It is worth mentioning that in order to speed up the online packaging of NPM, CodeSandbox authors use S3 cloud storage service provided by AWS. When a version of the NPM package has already been packaged once, the packaged-manifest.json file is stored on S3. The next time you request the same version of the package, you can return the stored manifest.json file directly without having to repeat the above process. You can replace S3 with your own file storage service in a private deployment.

Transpilation– Compilation phase

When Sandbox receives source code for the front end project, NPM dependencies, and build template Preset from the Editor. Sandbox initializes the configuration, downloads the manifest file for the NPM dependencies from the Packager service, compiles the project from the entry file of the front-end project, and parses the AST to recursively compile the required files to form the dependency graph. Basically the same principle as WebPack).

Note that CodeSandbox supports Preset, a build template for externally predefined projects. Preset specifies which transpilers (the equivalent of Webpack loaders) are used to compile the files for a certain type of file. The available Preset options are UE -cli, create-react-app, create-react-app-typescript, parcel, Angular-cli, and Preact-CLI. There is no support for modifying a specific configuration in Preset, which is built into the CodeSandbox source code. The following is an example of Preset configuration:

import babelTranspiler from ".. /.. /transpilers/babel"; .const preset = new Preset(
  "create-react-app"["web.js"."js"."json"."web.jsx"."jsx"."ts"."tsx"] and {hasDotEnv: true.setup: manager= > {
      constbabelOptions = {... }; preset.registerTranspiler(module= >
          /\.(t|j)sx? $/.test(module.path) && !module.path.endsWith(".d.ts"),
        [{
          transpiler: babelTranspiler,
          options: babelOptions
        }],
        true); . }});Copy the code

Evaluation– Implementation phase

The Evaluation execution phase starts from the compiled module corresponding to the project entry file and recursively calls eval to execute all referenced modules.

Since this article focuses on how to build your own online IDE, see the following article for more implementation details on CodeSandbox:

  • How does CodeSandbox work? Ok,

  • How does CodeSandbox enable modules on NPM to run directly in the browser

Privatized deployment of CodeSandbox

With the basics of CodeSandbox behind you, it’s time to get to the core of this article: how to deploy CodeSandbox privately.

The online packaging service Packager

The first is the NPM online packaging service dependency packager. I deployed to my own server through an image.

The NPM source can be changed to the company’s private NPM source by either using the NPM config command in the image, such as the following Dockerfile:

FROM node:12-alpine

COPY . /home/app

# Set private NPM source
RUN cd /home/app && npm config set registry http://npm.xxx.com && npm install -f

WORKDIR /home/app

CMD ["npm"."run"."dev"]
Copy the code

The second option is to add the parameter –registry=http://npm.xxx.com to the command to download the NPM package through YARN in the source code. Relevant code in functions provides/packager/dependencies/install – dependencies. Ts file.

In addition, the service relies on Serverless provided by AWS Lambda, and uses S3 storage service provided by AWS to cache the packaged results of NPM packages. If the reader does not have these services, you can comment out this part of the source code or replace it with the corresponding services of other cloud computing vendors. Dependency – Packager is essentially a Node service based on the Express framework that can simply run directly on the server.

The Editor Editor

The standalone Packages/React-Sandpack project in the CodesandBox-client project is an editor provided by CodeSandbox based on the React implementation. Different from the editor implemented by the main project, this editor is mainly for users to customize, so the implementation is relatively simple, users can add their own needs on the basis of the editor according to their own needs. React-sandpack NPM package react-SmooshPack can be used directly if you don’t need a custom editor.

import React from 'react';
import { render } from 'react-dom';
import {
  FileExplorer,
  CodeMirror,
  BrowserPreview,
  SandpackProvider,
} from 'react-smooshpack';
import 'react-smooshpack/dist/styles.css';

const files = {
  '/index.js': {
    code: "document.body.innerHTML = `<div>${require('uuid')}</div>` ",}};const dependencies = {
  uuid: 'latest'};const App = () = > (
  <SandpackProvider 
      files={files} 
      dependencies={dependencies} 
      entry="/index.js" 
      bundlerURL= `http://sandpack-The ${version}.codesandbox.io` >
    <div style={{ display: 'flex', width: '100% ',height: '100'}} % >
      <FileExplorer style={{ width: 300}} / >
      <CodeMirror style={{ flex: 1}} / >
      <BrowserPreview style={{ flex: 1}} / >
    </div>
  </SandpackProvider>
);

render(<App />.document.getElementById('root'));
Copy the code

The sub-components FileExplorer, CodeMirror and BrowserPreview are respectively the file directory tree on the left, the code editing area in the middle and the page preview area after project construction on the right.

In addition to these three sub-components, SandpackProvider also inserts an iframe tag, which is used to display the page after the project is built. The Preview component in the Preview pane on the right, BrowserPreview, inserts this ifame into its own node, thus rendering the page the project builds in real time.

The default bundlerUrl loaded by iframe is http://sandpack-${version}.codesandbox.io, The service corresponding to this domain name is actually the core of CodeSandbox — a service that builds front-end projects on the browser side. The general principle has just been explained. The next section explains how to replace the officially provided build service with your own.

As for how code/dependencies in the code edit area are synchronized to the build service loaded in iframe, it actually relies on a separate library, Sandpack (the same directory as react-Sandpack), One Manager class Bridges the code editing area with the build service in the preview area on the right, using the Dispatch method provided by the CodesandBox-API package to communicate between the editor and the build service.

The code runs the SandBox SandBox

In case you get confused, the build service mentioned in the previous section is not a back-end service. It is actually a front end page built by CodeSandbox. The Fundamentals section has explained that CodeSandbox actually implements a WebPack in the browser, where the project is all built.

CodeSandbox front-end construction of the core part of the directory in codesandBox-client packages/ APP project, the principle of which has been described above, here only need to build the PROJECT to deploy the WWW folder to the server. Since the core library relies on other libraries, it also needs to be built first. Below, I write a build.sh file and place it in the level 1 directory of the entire project.

The Node 10 environment is required to run and build
nvm use 10

# install dependencies
yarn

# If it's the first build, build the entire project first and use the build artifacts later
# If the entire project has already been built once, there is no need to rebuild
yarn run build 

Build a dependency library
yarn run build:deps

Build into the core library Packages /app
cd packages/app

yarn run build:sandpack-sandbox

For some reason, some required static files need to be retrieved from the build directory of the overall project
# Build the whole project once before executing the shell script, i.e. Yarn Run Build (this build will take a long time)cp -rf .. /.. /www/static/* ./www/staticCopy the code

After executing the shell script above, you can deploy the WWW built in packages/app directory to the server.

FROM node:10.14.2 as build

WORKDIR /

ADD.

RUN /bin/sh build.sh

FROM nginx:1.16.1-alpine

COPY --from=build /packages/app/www /usr/share/nginx/html/
Copy the code

Note that a phased build image is used, that is, the CodeSandbox project is built first and then the image is built. In practice, however, the CodeSandbox project didn’t build well on the server, so I decided to build the project locally and upload the build to a remote Git repository, so that all I had to do was build an image and run it on the packaging machine.

The entire deployment was inspired by an issue of GitLab’s official warehouse: GitLab Hosted Codesandbox

Customize CodeSandbox functionality

The reader of the previous section may wonder why you just use the default build service provided by CodeSandbox. In order to customize the CodeSandbox build process, here are four examples.

Replace the Babel plug-in functionality that component styles automatically introduce

For company-built component libraries, plug-ins like babel-plugin-import will be developed to use components in code without importing additional component style files. The babel-plugin-import plug-in will automatically insert the introduced style code during JS compilation. However, such a plug-in might need to iterate through the dependencies in the component’s package.json to see if there are any other components, and if so, to import the other component’s style files into the compiled JS and recursively perform the procedure just described. This is where you need to read the relevant files in node_modules. But things like CodeSandbox and Stackblitz are built in the browser and don’t have node_modules.

To solve this problem, the author finally gave up using Babel plug-in to insert the style file code during JS compilation. Instead, the author obtained the component’s style file from NPM online packaging service during code execution, and then dynamically inserted the style file content into the head tag through the style tag. Here are the specific changes:

Online NPM packaging service

The online NPM package service generally only returns JS files, so you need to add a feature on top of this service to return additional style files when the requested NPM package is judged to be a built-in component. Here is the core code added in the exion-Packager project:

To provide a method for obtaining private component style files, create a new file called fetch-builtin-component-style.ts in the functions/packager/utils directory with the following core code:

// According to the component NPM package name and the NPM package path downloaded to disk through YARN, read the corresponding style file content and write it to the contents object of manifest.json
const insertStyle = (contents: any, packageName: string, packagePath: string) = > {
  const stylePath = `/node_modules/${packageName}/dist/index.css`;
  const styleFilePath = join(
    packagePath,
    `/node_modules/${packageName}/dist/index.css`,);if (fs.existsSync(styleFilePath)) {
    contents[stylePath] = {
      content: fs.readFileSync(styleFilePath, "utf-8"),
      isModule: false}; }};// Get the style file for the built-in component and write it to the manifest.json file returned to the Sandbox
const fetchBuiltinComponentStyle = (
  contents: any,
  packageName: string,
  packagePath: string,
  dependencyDependencies: any.) = > {
  // When the NPM package or its dependencies and dependent dependencies have built-in components, write the style file corresponding to that built-in component to the manifest.json file
  if (isBuiltinComponent(packageName)) {
    insertStyle(contents, packageName, packagePath);
  }

  Object.keys(dependencyDependencies.dependencyDependencies).forEach(
    (pkgName) = > {
      if(isBuiltinComponent(pkgName)) { insertStyle(contents, pkgName, packagePath); }}); };Copy the code

And in functions provides/packager/index. Ts file invoke this method. The code is as follows:

+  // For private components, the component style file is also written to the manifest.json file returned to the browser
+  fetchBuiltinComponentStyle(
+    contents,
+    dependency.name,
+    packagePath,
+    dependencyDependencies,
+  );

// return as a result
constresponse = { contents, dependency, ... dependencyDependencies, };Copy the code

Browser CodeSandbox side

The CodeSandbox side of the browser needs to provide a way to handle private component styles, mainly by dynamically inserting the style file content into the head tag during the Evaluation execution phase through the style tag. Can be in the packages/app/SRC/sandbox/eval/utils directory to create a new file insert – builtin – component – style. Ts, the following is the core code:

// Create a style tag based on the content of the style file and insert it into the head tag
const insertStyleNode = (content: string) = > {
  const styleNode = document.createElement('style');
  styleNode.type = 'text/css';
  styleNode.innerHTML = content;
  document.head.appendChild(styleNode);
}

const insertBuiltinComponentStyle = (manifest: any) = > {
  const { contents, dependencies, dependencyDependencies } = manifest;

  Filter the built-in components from dependencies and their dependencies based on NPM package names
  const builtinComponents = Object.keys(dependencyDependencies).filter(pkgName= > isBuiltinComponent(pkgName));
  dependencies.map((d: any) = > {
    if(isBuiltinComponent(d.name)) { builtinComponents.push(d.name); }});// Find the contents of the file according to the key assembled based on the NPM name of the built-in component, and call the insertStyleNode method to insert the contents into the head tag
  builtinComponents.forEach(name= > {
    const styleContent = contents[`/node_modules/${name}/dist/index.css`];
    if (styleContent) {
      const { content } = styleContent;
      if(content) { insertStyleNode(content); }}}); }Copy the code

The method is called in the Evaluation execution phase. The related files are in packages/sandpack-core/ SRC /manager.ts.

.setManifest(manifest? : Manifest) {
  this.manifest = manifest || {
    contents: {},
    dependencies: [].dependencyDependencies: {},
    dependencyAliases: {}}; + insertBuiltinComponentStyle(this.manifest); . }...Copy the code

Added the preview area screenshot function

In the block reuse platform project, when you click the save button, you not only save the edited code, but also take a screenshot of the built right preview area and save it. As shown below:

The preview area on the right shows the iframe inserted by the SandpackProvider component, so you just need to find the IFrame and communicate with the page inside the Iframe via postMessage. When the internal page of iframe receives the screenshot instruction, take a screenshot of the current DOM and send it out. Here, the author uses HTML2Canvas to take a screenshot. Below is CodeSandbox side code transformation, the file on the packages/app/SRC/sandbox/index in js, mainly in the end file add the following code:

const fetchScreenShot = async() = > {const app = document.querySelector('#root');
  const c = await html2canvas(app);
  const imgData = c.toDataURL('image/png');
  window.parent.postMessage({
    type: 'SCREENSHOT_DATA'.payload: {
      imgData
    }
  }, The '*');
};

const receiveMessageFromIndex = (event) = > {
  const {
    type
  } = event.data;
  switch (type) {
    case 'FETCH_SCREENSHOT':
      fetchScreenShot();
      break;
    default:
      break; }};window.addEventListener('message', receiveMessageFromIndex, false);
Copy the code

On the CodeSandbox side, you need to send a screenshot command to iframe when you need a screenshot. At the same time, the message sent by iframe needs to be monitored to screen out the instructions that return the screenshot data and obtain the screenshot data. Because the implementation is relatively simple, I won’t show the actual code here.

Added support for less file compilation to the create-react-app template

Is mainly to create the react – app make some changes, the preset configuration file address packages/app/SRC/sandbox/eval/presets/create – react – app/v1. Ts. Modify the code as follows:

. +import lessTranspiler from '.. /.. /transpilers/less';
+  import styleProcessor from '.. /.. /transpilers/postcss';

export default function initialize() {... + preset.registerTranspiler(module= > /\.less$/.test(module.path), [
  +    { transpiler: lessTranspiler },
  +    { transpiler: styleProcessor },
  +    {
  +      transpiler: stylesTranspiler,
  +      options: { hmrEnabled: true}, +}, +]); . }Copy the code

Modify the NPM package service address requested by CodeSandbox

The packaged NPM service can be replaced with the privatized deployed service above to solve problems such as the inability to obtain private NPM packages. Relevant documents in the packages/sandpack – core/SRC/NPM/preloaded/fetch – dependencies. Ts. Modify the code as follows:

 const PROD_URLS = {
   ...
// Replace it with your own online NPM package
-  bucket: 'https://prod-packager-packages.codesandbox.io',
+  bucket: 'http://packager.igame.163.com'}; .function dependencyToPackagePath(name: string, version: string) {-return `v${VERSION}/packages/${name}/${version}.json` ;
+  return `${name}@${version}` ;

}
Copy the code

These four examples are enough for readers to customize as much as they want. Customization isn’t that hard when you understand how the whole CodeSandbox works.

conclusion

At this point, the goal of privatizing the deployment of your own and customizable online IDE has been met. Of course, online IDE project construction is not limited to the browser, you can also put the entire build process on the server side, thanks to the cloud + container capabilities, so that the online IDE has almost exactly the same functionality as the native IDE. In fact, both of these are not used in many scenarios, and a browser-based build is more suitable for real-time preview of a single page project, while a server-based build is completely suitable for real project development, and not just for front-end projects. I am also trying to explore the possibility of building an IDE based on the server side, and I look forward to sharing some output with you later.

Next, if you are interested, you can read about a block platform project based on Bit and CodeSandbox implementation — Cross-project Block reuse practices

The resources

  • How does CodeSandbox work? Ok,
  • GitLab hosted Codesandbox