Pipcook Contributor

Pipcook is a machine learning application framework for front-end developers developed by D2C team of Tao Department technology Department. We hope that Pipcook can become a platform for front-end personnel to learn and practice machine learning, so as to promote the process of front-end intelligence.

Boa is the technology behind Pipcook, which allows developers to call any Python function, including, of course: If you want to use TensorFlow, PyTorch, TVM and other Python libraries, you can use Numpy, SciKit-Learn, and Jieba.

Before implementing this functionality, Boa loads the Python library like this:

const boa = require('@pipcook/boa');
const { range, len } = boa.builtins();
const { getpid } = boa.import('os');
const numpy = boa.import('numpy');
Copy the code

Import the Python library by passing in the name of boa. Import () may be redundant if there are many packages to load. To do this, we developed a custom import semantics declaration, using ES Module to achieve a more concise import statement:

import { getpid } from 'py:os';
import { range, len } from 'py:builtins';
import {
  array as NumpyArray,
  int32 as NumpyInt32,
} from 'py:numpy';
Copy the code

The implementation of the above functionality relies on Node’s experimental feature –loader, which has an undisclosed alias –loader. When this feature was first introduced, it was actually loader. Experimental is added to later versions because it is considered to be experimental.

–experimental-loader

Custom loaders are implemented by specifying this flag when starting the program and accepting files with the specified suffix MJS. MJS files provide several hooks to intercept the default loader:

  • resolve
  • getFormat
  • getSource
  • transformSource
  • getGlobalPreloadCode
  • dynamicInstantiate

The order of execution is:

( Each `import` runs the process once ) |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -| ( run once ) | ---> dynamicInstantiate | getGlobalPreloadCode -> | resolve -> getFormat -> | | | ---> getSource -> transformSource | |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |Copy the code


getGlobalPreloadCode Hook

Run only once, used to run some code globally at application startup. Variables can be hung globally via the globalThis object. Currently only a getBuiltin (similar to require) function is provided to load the built-in module:

/** * @returns {string} Code to run before application startup */
export function getGlobalPreloadCode() {
  return `\ globalThis.someInjectedProperty = 42; console.log('I just set some globals! '); const { createRequire } = getBuiltin('module'); const require = createRequire(process.cwd() + '/
      
       '); `
      ;
}
Copy the code

The variable someInjectedProperty can be used in all modules.

resolve Hook

It intercepts import statements and the import() function that evaluates the imported string and returns the url of a particular logic:

const protocol = 'py:';

/** * @param {string} specifier * @param {object} context * @param {string} context.parentURL * @param {function} defaultResolve * @returns {object} response * @returns {string} response.url */
export function resolve(specifier, context, defaultResolve) {
  if (specifier.startsWith(protocol)) {
    return {
      url: specifier
    };
  }
  return defaultResolve(specifier, context, defaultResolve);
}
Copy the code

Parameter definition:

  • specifier – importStatements orimport()Strings in expressions
// eg:

import os from 'py:os'; // 'py:os'
async function load() {
  const sys = await import('py:sys'); // 'py:sys'
}
Copy the code
  • ParentURL – Imports the URL of the parent module of the module
// eg: 
// app.mjs

import os from 'py:os'; // file:///... /app.mjs
Copy the code

The first argument to defaultResolve accepts only three protocol strings:

  • data:– javascript wASM string
  • nodejs:– built – in the module
  • file:– Third-party or user-defined modules

The Boa custom py protocol does not pass the default resolve Hook argument check, so it returns the URL directly to the getFormat Hook.

getFormat Hook

There are several ways to define how to resolve urls passed from resolve Hook:

  • builtin– Node.js built-in module
  • commonjs– CommonJS module
  • dynamic– Dynamic instantiation, triggerdynamicInstantiate Hook
  • json– a JSON file
  • module– ECMAScript module
  • wasm– WebAssembly module

From the feature description, only Dynamic fits our requirements:

export function getFormat(url, context, defaultGetFormat) {
  // DynamicInstantiate hook triggered if boa protocol is matched
  if (url.startsWith(protocol)) {
    return {
      format: 'dynamic'}}// Other protocol are assigned to nodejs for internal judgment loading
  return defaultGetFormat(url, context, defaultGetFormat);
}
Copy the code

DynamicInstantiate Hook is triggered when the URL matches the BOA protocol, otherwise the default parser will decide what to load.

dynamicInstantiate Hook

Provides a different way to dynamically load modules than getFormat several parsing formats:

/** * @param {string} url * @returns {object} response * @returns {array} response.exports * @returns {function} response.execute */
export function dynamicInstantiate(url) {
  const moduleInstance = boa.import(url.replace(protocol, ' '));
  // Get all the properties of the Python Object to construct named export
  // const { dir } = boa.builtins();
  const moduleExports = dir(moduleInstance);
  return {
    exports: ['default'. moduleExports],execute: exports= > {
      for (let name ofmoduleExports) { exports[name].set(moduleInstance[name]); } exports.default.set(moduleInstance); }}; }Copy the code

Load the Python module using boa.import() and use the built-in dir function in Python Builtins to get all of the module’s properties. The hook needs to provide a list of exports passed to exports to support Named exports, plus default to support default exports. The execute function sets the property corresponding to the specified name when initializing the dynamic hook.

getSource Hook

Used to pass the source string, providing a different way to get the source than the default loader reads the file from disk, such as network, memory, and hard coding, etc:

export async function getSource(url, context, defaultGetSource) {
  const { format } = context;
  if (someCondition) {
    // For some or all URLs, do some custom logic for retrieving the source.
    // Always return an object of the form {source: <string|buffer>}.
    return {
      source: `export const message = 'Woohoo! '.toUpperCase(); `
    };
  }
  // Defer to Node.js for all other URLs.
  return defaultGetSource(url, context, defaultGetSource);
}
Copy the code

What is interesting here is that the source code can be obtained from different sources, such as the network in the same way as Deno.

transformSource Hook

After loading the source code, the Hook can modify the loaded source code:

export async function transformSource(source, context, defaultTransformSource) {
  const { url, format } = context;
  if (source && source.replace) {
    // For some or all URLs, do some custom logic for modifying the source.
    // Always return an object of the form {source: <string|buffer>}.
    return {
      source: source.replace(`'A message'; `.`'A message'.toUpperCase(); `)}; }// Defer to Node.js for all other sources.
  return defaultTransformSource(
    source, context, defaultTransformSource);
}
Copy the code

The example provided above replaces a particular string in the source code with a new string, and also allows instant compilation:

export function transformSource(source, context, defaultTransformSource) {
  const { url, format } = context;

  if (extensionsRegex.test(url)) {
    return {
      source: CoffeeScript.compile(source, { bare: true})}; }// Let Node.js handle all other sources.
  return defaultTransformSource(source, context, defaultTransformSource);
}
Copy the code


The last

Node.js can only load modules in a few built-in ways, but now the hooks are open to combine many interesting functions, and you can explore more interesting scenarios.

Refer to the link

Related articles