“This is the first day of my participation in the Gwen Challenge in November. Check out the details: The last Gwen Challenge in 2021”

With Protobuf, we have developed a complete tool chain for efficiency, including transformation engineering, PB Mock, various language filtering tools, etc. Today we will focus on filtering tools.

Proto files are converted into various languages. We currently support Java, GO, JS, Swift, Dart, Python and other languages. The conversion results are stored in their respective result repositories.

Different clients need different PB files. Full reference will increase the size of the code, which is very unfriendly to the client. Therefore, a tool is needed to do filtering. The tool finds the specified file from the result repository based on a developer-maintained JSON list and outputs it to the specified directory.

The first version of the filter tool is developed with Node, using the PKG package for executable file packaging, functional to meet the needs, but also in continuous optimization iteration. The biggest problem is that the output executable file after packaging is too big, up to 45M. If uploading to Git is cumbersome, if you do not upload it, you have to download the tool every time a new pull project, which is really troublesome. So let’s think about what we can optimize.

The idea of using Rust to write a new generation of tools came to mind when the team tested Rust and found that compiled executables were much smaller than Node PKG.

However, even if you use Rust to write, you still need to download it. Is there a more lazy way to do it? It occurs to me that Rust can be compiled into Node modules, so I wonder if it can be integrated into a project that can be used in VSCode plug-in projects and can be compiled into executable files.

Compilation and selection of Node module

When selecting the Rust to Node module build tool, two directions were considered:

  1. WebAssembly
  2. Node native module

WebAssembly

The tool used in the tests is wASM-pack, which allows you to compile Rust into multiple JS references, such as NodeJS, Web, Webpack, and so on. [See official website]

The generated WASM was tested to work, but since security does not support system-level file manipulation, regardless of Node or browser environments, this direction was abandoned.

By the way, the current MAC using Apple’s own chips (M1, M1X) will fail to install using the official website or NPM directly.

The only way it worked was to install using Cargo, which would have failed by default too, but after a lot of crawling it turned out that you needed to set the installation target to an older version of Mac OS so Cargo would think the platform was x86 and then compile and install.

Wasm-pack was installed in this way, and the result is no problem, or it may be a shallow test, friends to explore.

env MACOSX_DEPLOYMENT_TARGET=10.7 cargo install wasm-pack
Copy the code

Node native module

node_bindgen

It is easy to install, but the editor cannot find the module after installation, but it can compile the Node module.

Neon

Here we go, and it’s doing very well. Just add a connection layer under Rust, and you can call both Node and Rust successfully. It wasn’t until it was introduced into the VSCode plug-in project that the nightmare started. VSCode debugging kept reporting NODE_MODULE_VERSION as 89, while the Node module was introduced as 83.

A search revealed that the cause of this problem is that Electron does not use the ABI version of Node, but has a custom version, whereas VSCode was developed based on Electron. Even if you switch the local Node to Electron, you won’t be able to compile a usable Node module.

This problem bothered me for a while, until I found a Demo of Neon’s environment variables that you can use to compile a Node module before running it.

export NEON_NODE_ABI=89
exportNpm_config_target = 13.5.1export npm_config_disturl=https://atom.io/download/electron
# export nPM_config_ARCH =x64 M1/M1X chip is not needed
# export nPM_config_TARGEt_ARCH =x64 M1/M1X chip is not needed
export npm_config_runtime=electron
export npm_config_build_from_source=true
Copy the code

This works, but there is a problem. If VSCode updates the Electron version, the module we compiled will not work if the supported ABI version changes. There are three approaches in mind:

  1. Pre-built from 89-102 (currently highest)NodeThe module.
  2. Plug-ins are loaded based on the currentVSCodeBuild the corresponding version of the internal environmentNodeThe module.
  3. VSCodeUpdate the plug-in version as it is updated.

The first method is obviously a clunky one, and the plug-in will be very large and undesirable.

The second method requires the host environment to have Rust installed, otherwise the compilation cannot complete.

The third method is by far the most reliable, so expect the superhard Electron version to be a bit slower.

The project structure

After the tentative selection of technology, the structure of our project is as follows:

Bb-pb-tools ├─.eslintrc.json ├─.vsCode │ ├─ Download.json │ ├─ Download.json │ Settings │ ├─ Tasks.json ├─.vscodeIgnore ├─ Changelog.md ├─ Readme.md ├─ env.sh ├─ Native Rust │ ├─ Cargo. Lock │ ├─ Cargo. Toml │ ├─ Artifacts. Json │ ├─ build.rs │ ├─ index The Node module, Consider output to other directory behind │ └ ─ SRC │ ├ ─ lib │ │ └ ─ index. The rs │ ├ ─ lib. Rs │ └ ─ main. Rs ├ ─ package - lock. Json ├ ─ package. The json ├ ─ the SRC │ ├─ ├─ test.ts │ ├─ ├.test.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ ├.ts │ ├─ └ ├ ─ VSC - the extension - quickstart. Md └ ─ yarn. The lockCopy the code

Complete the process

Ok, friends, that’s all for today’s sharing, the tide is coming in soon.