preface
To learn how rollup packaging works, I cloned the source code for the latest version (V2.26.5). Then I found that the packager was not quite what I expected. There was so much code that I had a headache just looking at the D.TS file. I wrote a script to see how many lines of source code there were and found 19,650 lines, crash…
Does that take away my determination to learn rollup? No way. The next best thing is, I downloaded the original rollup source code, which is only about 1000 lines.
My goal is to learn how rollup is packaged, how to do tree-shaking. The original source code already implements both of these functions (semi-finished), so it is sufficient to look at the original source code.
Okay, let’s start the text.
The body of the
Rollup uses the Acorn and Magic-String libraries. To read the rollup source code, you must know something about them.
I’ll take a quick look at what these two libraries do.
acorn
Acorn is a JavaScript syntax parser that parses JavaScript strings into a syntax abstraction tree AST.
For example:
export default function add(a, b) { return a + b }
Copy the codeWill be resolved to:
{
"type": "Program"."start": 0."end": 50."body": [{"type": "ExportDefaultDeclaration"."start": 0."end": 50."declaration": {
"type": "FunctionDeclaration"."start": 15."end": 50."id": {
"type": "Identifier"."start": 24."end": 27."name": "add"
},
"expression": false."generator": false."params": [{"type": "Identifier"."start": 28."end": 29."name": "a"
},
{
"type": "Identifier"."start": 31."end": 32."name": "b"}]."body": {
"type": "BlockStatement"."start": 34."end": 50."body": [{"type": "ReturnStatement"."start": 36."end": 48."argument": {
"type": "BinaryExpression"."start": 43."end": 48."left": {
"type": "Identifier"."start": 43."end": 44."name": "a"
},
"operator": "+"."right": {
"type": "Identifier"."start": 47."end": 48."name": "b"}}}]}}}],"sourceType": "module"
}
Copy the codeYou can see that the AST is of type Program, indicating that this is a program. The body contains the AST child nodes for all the following statements of the program.
Each node has a type of type, such as Identifier, indicating that the node is an Identifier. BlockStatement: a node is a BlockStatement. A ReturnStatement is a return statement.
For more details on AST nodes, see the article parsing JavaScript with Acorn.
magic-string
Magic-string is also a library of string operations written by the author of Rollup. Here’s an example on Github:
var MagicString = require( 'magic-string' );
var s = new MagicString( 'problems = 99' );
s.overwrite( 0.8.'answer' );
s.toString(); // 'answer = 99'
s.overwrite( 11.13.The '42' ); // character indices always refer to the original string
s.toString(); // 'answer = 42'
s.prepend( 'var ' ).append( '; ' ); // most methods are chainable
s.toString(); // 'var answer = 42; '
var map = s.generateMap({
source: 'source.js'.file: 'converted.js.map'.includeContent: true
}); // generates a v3 sourcemap
require( 'fs' ).writeFile( 'converted.js', s.toString() );
require( 'fs' ).writeFile( 'converted.js.map', map.toString() );
Copy the codeAs you can see from the examples, this library mainly encapsulates some common methods on strings. I won’t do any more introductions here.
Rollup source code structure
│ bundle. Js// Bundle. During packaging, a Bundle instance is generated, which is used to collect code from other modules, and then the collected code is packaged together.│ external -module.js ExternalModule External modules, such as the 'path' module, will generate an instance of ExternalModule.
│ module.js // Module Module, the developer's own code files, are Module instances. For example, the 'foo.js' file corresponds to an instance of module.│ a rollup. Js// the rollup function, where everything starts, is called for packaging.│ ├ ─ ast// The AST directory, which contains classes and functions related to the AST│ analyse. Js// Mainly used to analyze the scope and dependencies of AST nodes.│ Scope. Js// Generate Scope instance for each AST node when analyzing AST node, mainly record Scope of each AST node.│ walk. Js// Walk is a recursive call to the AST node for analysis.│ ├ ─ finalisers │ CJS. Js// Package mode. Currently, only code can be packaged in common.js format│ index. Js │ └ ─ utils// Some helper functions
map-helpers.js
object.js
promise.js
replaceIdentifiers.js
Copy the codeAbove is the directory structure of the original source code, please read the above comments carefully to understand the purpose of each file before going any further.
How does rollup pack?
In rollup, a file is a module. Each module generates an AST syntax abstraction tree based on the code of the file, and rollup needs to analyze each AST node.
Parsing an AST node is to see if it calls any functions or methods. If so, check to see if the called function or method is in the current scope, and if not, look up until you find the module’s top-level scope.
If this module is not found, it indicates that the function or method depends on other modules and needs to be imported from other modules.
For example, import foo from ‘./foo.js’, where foo() is retrieved from the./foo.js file.
During the introduction of the foo() function, if foo() is found to depend on other modules, it reads the other modules recursively, and so on until there are no dependent modules.
Finally, all the imported code is packaged together.
Example diagram of the above example:
Let’s start with a concrete example and walk through how rollup is packaged.
The following two files are code files.
// main.js
import { foo1, foo2 } from './foo'
foo1()
function test() {
const a = 1
}
console.log(test())
Copy the code// foo.js
export function foo1() {}
export function foo2() {}
Copy the codeHere is the test code:
const rollup = require('.. /dist/rollup')
rollup(__dirname + '/main.js').then(res= > {
res.wirte('bundle.js')})Copy the code1. Read a rollupmain.jsImport file.
Rollup () first generates an instance of the Bundle, the packer. Then read the file according to the entry file path, and finally generate a Module instance according to the file contents.
fs.readFile(path, 'utf-8'.(err, code) = > {
if (err) reject(err)
const module = new Module({
code,
path,
bundle: this./ / bundle instance})})Copy the code2. The New Moudle() process
When a Module instance is new, the Acorn library’s parse() method is called to parse the code into the AST.
this.ast = parse(code, {
ecmaVersion: 6.// The ECMA version of the JavaScript to parse, as ES6
sourceType: 'module'.// sourceType values are module and script. Module mode, which can use import/export syntax
})
Copy the codeNext, you need to analyze the generated AST.
The first step is to analyze the imported and exported modules and fill the imported and exported modules with corresponding objects.
Each Module instance has an imports and Exports object that fills in the imports and exports of the Module for code generation.
The imports and exports of the above example are:
// Key is the specific object to be imported, and value is the content of the AST node.
imports = {
foo1: { source: './foo'.name: 'foo1'.localName: 'foo1' },
foo2: { source: './foo'.name: 'foo2'.localName: 'foo2'}}// Empty because there are no exported objects
exports = {}
Copy the codeThe second step is to analyze the scope between each AST node and find the variables defined by each AST node.
Each time an AST node is traversed, a Scope instance is generated for it.
/ / scope
class Scope {
constructor(options = {}) {
this.parent = options.parent // Parent scope
this.depth = this.parent ? this.parent.depth + 1 : 0 // Scope level
this.names = options.params || [] // Variables in scope
this.isBlockScope = !! options.block// Whether to block scope
}
add(name, isBlockDeclaration) {
if(! isBlockDeclaration &&this.isBlockScope) {
// it's a `var` or function declaration, and this
// is a block scope, so we need to go up
this.parent.add(name, isBlockDeclaration)
} else {
this.names.push(name)
}
}
contains(name) {
return!!!!!this.findDefiningScope(name)
}
findDefiningScope(name) {
if (this.names.includes(name)) {
return this
}
if (this.parent) {
return this.parent.findDefiningScope(name)
}
return null}}Copy the codeThe Scope is simple in that it has an array of Names attributes that hold variables within this AST node. For example, the following code:
function test() {
const a = 1
}
Copy the codeAs you can see from the break point, the scoped object that it generates, the NAMES property, contains a. And because it is a function under a module, the scope level is 1 (the module’s top-level scope is 0).
Third, analyze the identifiers and find their dependencies.
What is an identifier? Variable names, function names, and attribute names are all classified as identifiers. When an identifier is resolved, rollup traverses its current scope to see if it exists. If not, look for its parent scope. If the top-level scope of the module is not found, the function or method depends on other modules and needs to be imported from other modules. If a function or method needs to be imported, add it to the _dependsOn object of the Module.
For example, if the variable a in the test() function is found in the current scope, it is not a dependency. Foo1 () is not found in the current module scope; it is a dependency.
Module _dependsOn property has foo1 in the break point.
This is the tree-shaking principle of Rollup.
Rollup doesn’t look at what functions you introduce, but what functions you call. If the function called is not in this module, it is imported from another module.
In other words, if you manually introduce a function at the top of the module but don’t call it. Rollup will not be introduced. Can be seen from our example, introduced the total foo1 foo2 () () two functions, only in the _dependsOn foo1 (), because of the introduction of foo2 () there is no call.
What does _dependsOn do? Later code generation will import the file based on the value in _dependsOn.
3. Read the corresponding file based on the dependency.
As you can see from the _dependsOn value, we need to introduce the foo1() function.
This is where the imports generated in step 1 come into play:
imports = {
foo1: { source: './foo'.name: 'foo1'.localName: 'foo1' },
foo2: { source: './foo'.name: 'foo2'.localName: 'foo2'}}Copy the codeRollup takes foo1 as the key and finds its corresponding file. This file is then read to generate a new Module instance. Since the foo.js file exports two functions, the exports attribute of the new Module instance looks like this:
exports = {
foo1: {
node: Node {
type: 'ExportNamedDeclaration'.start: 0.end: 25.declaration: [Node],
specifiers: [].source: null
},
localName: 'foo1'.expression: Node {
type: 'FunctionDeclaration'.start: 7.end: 25.id: [Node],
expression: false.generator: false.params: [].body: [Node]
}
},
foo2: {
node: Node {
type: 'ExportNamedDeclaration'.start: 27.end: 52.declaration: [Node],
specifiers: [].source: null
},
localName: 'foo2'.expression: Node {
type: 'FunctionDeclaration'.start: 34.end: 52.id: [Node],
expression: false.generator: false.params: [].body: [Node]
}
}
}
Copy the codeAt this point, the exports object of foo.js is matched with the key of foo1 imported by main.js. If the match is successful, the AST node corresponding to the foo1() function is extracted and placed in the Bundle. If the match fails, an error is reported indicating that foo.js did not export the function.
4. Generate code.
Because I’ve already introduced all the functions. You need to call the Bundle’s generate() method to generate code.
Also, during the packaging process, you need to do some additional operations on the introduced functions.
Remove extra code
For example, the code for the foo1() function imported from foo.js looks like this: export function foo1() {}. Rollup removes export and becomes function foo1() {}. Because they are going to be packaged together, there is no need for export.
rename
For example, if two modules have a function foo() with the same name, when packaged together, one of the functions is renamed _foo() to avoid collisions.
Okay, back to the text.
Remember the magic-String library mentioned at the beginning of this article? In generate(), the corresponding source code for each AST node is added to the magic-String instance:
magicString.addSource({
content: source,
separator: newLines
})
Copy the codeThis operation is essentially the same as composing a string:
str += 'This operation is equivalent to piecing together the source code of each AST as a string, as it is now'
Copy the codeFinally, the pieced code is returned.
return { code: magicString.toString() }
Copy the codeThat’s it. If you want to generate a file, you can call the write() method to generate a file:
rollup(__dirname + '/main.js').then(res= > {
res.wirte('dist.js')})Copy the codeThis method is written in the rollup() function.
function rollup(entry, options = {}) {
const bundle = newBundle({ entry, ... options })return bundle.build().then(() = > {
return {
generate: options= > bundle.generate(options),
wirte(dest, options = {}) {
const { code } = bundle.generate({
dest,
format: options.format,
})
return fs.writeFile(dest, code, err= > {
if (err) throw err
})
}
}
})
}
Copy the codeAt the end
This article abstracts the source code, so many implementation details are left out. If you are interested in implementation details, take a look at the source code. The code is on my Github.
I’ve stripped the original source code for Rollup and added a lot of comments to make it easier to read.