Recently, THE author is systematically combing the knowledge of native JS, because I think JS as the fundamental technology of front-end engineers, learn more than enough. I plan to make a series driven by a series of questions. Of course, there will also be questioning and expansion. The content is systematic and complete, which will greatly improve the beginner and intermediate players, and the senior players will also be reviewed and consolidated. This is the third article in this series

Chapter 1: the question of JavaScript memory — How is data stored?

The Internet basically says: Basic data types are stored on a stack, and reference data types are stored on a heap, which looks fine, but is actually problematic. Consider the closure case: if the variable is in the stack, then the function is destroyed at the top of the stack, and the closure variable is gone. Closure variables are stored in heap memory. Specifically, the following data types are stored on the stack:

  • null
  • undefined
  • number
  • boolean
  • string
  • bigint
  • symbol

All object data types are stored in the heap. It is worth noting that for assignment, the original type copies the value of the variable completely, while the object type copies the reference address. So here’s what happens:

let obj = { a: 1 }
let newObj = obj

newObj.a = 2
console.log(obj.a) // Becomes 2
Copy the code

The reason for this is that obj and newObj are the address of the same heap space. Changing newObj is equal to changing the common heap memory. In this case, obj will change the value of this memory

Of course, you might ask: why not just stack it all? First, for the system stack, besides saving variables, it also has the function of creating and switching the execution context of functions. Here’s an example:

function f(a) {
  console.log(a);
}
function func(a) {
  f(a);
}
func(1);
Copy the code

Assuming that ESP Pointer (Extended Stack Pointer register) is used to hold the current execution state, the following process occurs in the system Stack:

  1. Call func, pushing the context of the func function, ESP pointing to the top of the stack
  2. Run func and call f function to push the context of f function. ESP pointer moves up
  3. After f function is executed, move ESP down. The top space of f function is reclaimed
  4. After func is executed, ESP moves down and the func space is reclaimed

So as you can see, if you use a stack to store object data that is more complex than the primitive type, the overhead of context switching becomes huge! However, although the heap memory space is large, can store a large amount of data, but at the same time garbage memory recycling will bring more overhead, the next chapter will analyze how heap memory is garbage collection and optimization

Chapter 2: How does the V8 engine recycle garbage memory?

Unlike C/C++, which allows programmers to create or free memory, JS is similar to Java, which uses its own set of garbage collection algorithms for automatic memory management. As a senior front end engineer, the mechanism of JS memory reclamation needs to be very clear, so as to be able to analyze the bottleneck of system performance in extreme environment. On the other hand, learning the mechanism of this, also for our in-depth understanding of JS

  • Closure features
  • And efficient use of memory

Are of great help

V8 Memory limits

In other backend languages, such as Java/Go, there are no limitations on memory usage, but unlike JS, V8 can only use a portion of the system’s memory. Specifically, V8 can only allocate a maximum of 1.4 GB on 64-bit systems and 0.7 GB on 32-bit systems. If you think about it on the front end, it’s not a huge memory requirement, but on the back end, if NodeJS encounters a file of more than 2 gigabytes, it won’t be able to read it all into memory for various operations

We know that for stack memory, when the ESP pointer moves down, i.e. after a context switch, the space at the top of the stack is automatically reclaimed. However, heap memory is more complicated, and we will focus on the garbage collection of heap memory. As we mentioned in the last article, all object type data in JS is allocated through the heap. When we construct an object for assignment, the corresponding memory is already allocated to the heap. You can keep creating objects like this and let V8 allocate space for them until the heap reaches its maximum size

So why does V8 have a memory ceiling? Obviously my machine large dozens of gigabytes of memory, can only let me use so little? Basically, it is jointly determined by two factors, one is the execution mechanism of JS single thread, the other is the limitation of JS garbage collection mechanism. First, JS is single-threaded, which means that once garbage is collected, all other running logic is suspended. Garbage collection, on the other hand, is a very time-consuming operation, as described by V8:

Taking 1.5GB of garbage collection heap memory as an example, V8 takes more than 50ms to do a small garbage collection and even more than 1s to do a non-incremental garbage collection

It can be seen that it takes a long time, and in such a long time, our JS code execution will always have no response, resulting in application lag, resulting in a sharp decline in application performance and responsiveness. Therefore, V8 made a crude choice to limit the heap, which was a tradeoff, since most of the time you wouldn’t have to manipulate several gigabytes of memory. However, you can adjust the memory limit if you want. The configuration command is as follows:

// This is to adjust the old generation of this part of the memory, in MB. More on new generation and old generation memory later
node --max-old-space-size=2048 xxx.js 
Copy the code

or

// This is to adjust the memory of this part of the new generation, in KB.
node --max-new-space-size=2048 xxx.js
Copy the code

New generation memory reclamation

V8 divides heap memory into new generation memory and old generation memory for processing. As the name suggests:

  • A new generation is a temporary allocation of memory that has a short lifetime
  • Old generation is resident memory, live for a long time

V8 heap memory, which is the sum of two memoriesBased on these two different types of heap memory, V8 uses different reclamation strategies that are optimized for different scenarios. First, the memory of the new generation. I have just introduced the method of adjusting the memory of the new generation. What is its default memory limit? in

  • 32MB in a 64-bit system
  • 16MB on a 32-bit system

Small enough, but it makes sense that the variable in the new generation has a short lifetime and is not likely to cause too much memory burden, so it can be made small enough

So, what’s the next generation of recycling?

First, the memory space of the new generation is divided into two parts:The From part indicates the memory that is being used, and the To part indicates the memory that is currently idle

When garbage collection is performed, V8 checks the objects in the From section and copies them To To memory if they are alive (they were placed From scratch in order in To memory) or directly collects non-alive objects

The roles of From and To are reversed when all surviving objects From have been entered into To memory in sequence. From is now idle, To is in use, and so on

Now, you might ask, why do all of this when you can just recycle nonviable objects? Note that I specifically stated that the To memory is placed sequentially from the beginning, in order To deal with a scenario like this:The dark squares represent the living objects and the white parts represent the memory to be allocated. Since the heap is allocated continuously, this scattered space can cause slightly larger objects to be unable to allocate space. This scattered space is also calledMemory fragments. The new generation garbage collection algorithm I just introduced is also calledScavenge algorithm. The Scavenge algorithm is designed To take care of memory fragments, and the exploiture occurs when the space looks like this:Is it a lot cleaner? This greatly facilitates the subsequent allocation of continuous space. The advantage of the Scavenge avenge is that the insane only use half the amount of new generation memory, but only store objects with short lifetimes, which are usually very small, so the timing is very good

Old generation memory reclamation

Just introduced the recycling method of the new generation, so if the variables in the new generation still exist after repeated recycling, they will be put into the memory of the old generation. This phenomenon is called promotion. In fact, promotion is not only this kind of reason, let’s sort out what circumstances will trigger promotion:

  • Have been Scavenge once
  • The memory footprint of To (idle) space exceeds 25%

The Scavenge avenge is insane. The garbage collector is insane, the amount of variable space that can be accumulated is insane. So for old generation, what kind of strategy is adopted for garbage recycling?

  1. The tag-erase process, described in detail in JavaScript Advanced Programming (3rd Edition), is divided into two phases, the tag-erase phase and the tag-erase phase
    • It starts by going through all the objects in the heap and marking them
    • Variables used in the code environment and strongly referenced variables are then unmarked, and all that is left is the variables to be deleted, which are reclaimed in the subsequent cleanup phase

Of course, this leads to the problem of memory fragmentation, and the space discontinuities of living objects make subsequent space allocation difficult. How did old generation deal with this problem?

  1. V8’s solution to defragmentation is simple and crude: after the cleanup phase is over, all surviving objects are moved to one side

Since it’s a moving object, it’s not going to be fast, and in fact it’s the most time-consuming part of the process

Incremental tag

Due to the single-thread mechanism of JS, V8 garbage collection will inevitably block the execution of business logic, if the old generation garbage collection task is very heavy, then the time will be terrible, seriously affect the performance of the application. In order to avoid this problem, at this time the V8, the plan of the increment of the tag is a relief complete tag task is divided into many small parts, each finished a small part of the “rest”, js application logic to perform for a while, and then execute the following part, if circulation, until mark phase completed into memory fragments of the above. In fact, this process is similar to the React Fiber process, so I won’t elaborate on it here. After incremental marking, the garbage collection process blocks JS applications by a factor of 6, which is a very successful improvement. JS garbage collection principle is introduced here, in fact, it is very simple to understand, it is important to understand why it does so, not just how to do it, I hope this summary can inspire you

Chapter 3: Describe how V8 executes a piece of JS code?

Front end is a relatively new field, so the front-end framework and tools emerge in endlessly, make a person dazzling, especially the major manufacturers to launch small program after their respective standards, for more cumbersome front-end development work, in this context to wipe flat machine, the difference between the birth of compile tools/frameworks also countless. But in any case, want to catch up with these frameworks and tools update speed is very difficult, even if caught is difficult to generate its own technology accumulation, a better way is to study the nature of the knowledge, hold the upper application in the same underlying mechanism, so that we can easily understand not just passively using the framework of the upper, You can even build your own wheels in the right scenarios for development efficiency

Understanding the implementation mechanism from V8’s point of view can also help us understand many of the upper-layer applications, including Babel, Eslint, and the underlying mechanics of the front-end framework. So how exactly does a piece of JavaScript code perform in V8?

First of all, we need to understand that the machine is not able to read JS code, the machine can only understand specific machine code, so if we want JS logic to run on the machine, we must translate JS code into machine code, and then let the machine recognize. JS is an interpreted language. For interpreted languages, the interpreter performs the following analysis on the source code:

  • Generate AST(Abstract Syntax Tree) through lexical analysis and syntax analysis
  • Generate bytecode

The interpreter then executes the program according to the bytecode. However, the whole process of JS execution will be more complex than this, next to dismantle one by one.

1. Generate AST

There are two steps to building an AST — lexical analysis and syntax analysis. Lexical analysis is word segmentation, and its job is to break down lines of code into tokens. For example, the following line:

let name = 'sanyuan'
Copy the code

It breaks the sentence down into four parts:That resolves into four tokens, and that’s where lexical analysis comes in. In the next stage of grammar analysis, the generated token data is converted into an AST according to certain grammar rules. Here’s an example:

let name = 'sanyuan'
console.log(name)
Copy the code

The resulting AST looks like this:When an AST is generated, the compiler/interpreter relies on the AST for subsequent work rather than the source code. As a side note, Babel works by parsing ES6 code to generate AN ES6 AST, converting the ES6 AST into an ES5 AST, and finally converting the ES5 AST into specific ES5 code. Since this article focuses on the principle, the details of Babel compilation will not be expandedBabel articlesBack in V8 itself, the AST is generated, and the execution context is generated. For the execution context, see the previous article “JavaScript Memory: How is data stored?” In the context of the stack out of the stack process

2. Generate bytecode

As mentioned at the beginning, once the AST is generated, the bytecode is generated directly through the V8 interpreter (also called Ignition). butThe bytecodeInstead of running it directly, you might say, why convert it to bytecode if you can’t execute it, just convert the AST to machine code and let the machine execute it directly. True, this was done in the early days of V8, but the size of the machine code later caused serious memory problems. Here’s a comparison diagram to give you an intuitive sense of the difference between the following three code loads:It is easy to conclude that bytecode is much lighter code than machine code. So why does V8 use bytecode, and what exactly is bytecode?

Bytecode is code that is intermediate between AST and machine code, but independent of a particular type of machine code. Bytecode needs to be translated into machine code by an interpreter and then executed.

The bytecode still needs to be converted to machine code, but instead of converting the entire bytecode to machine code at once, the interpreter executes the bytecode line by line, eliminating the need to generate binary files, which greatly reduces memory stress

3. Execute the code

Next, it’s time for bytecode interpretation execution. If a section of code is found to repeat itself during bytecode execution, V8 records it as HotSpot and then compiles it into machine code. The compiler used is TurboFan, so the longer the code takes to execute, The execution becomes more efficient because more and more bytecodes are marked as hot code, and the corresponding machine code is executed directly when they are encountered, without being converted to machine code again

When you hear people say that JS is an interpreter language, there is something wrong with that statement. Because bytecode not only works with the interpreter, but also works with the compiler, JS is not a fully interpreted language. The fundamental difference between a compiler and an interpreter is that the former compiles and generates binaries but the latter does not

And this technique of combining bytecode with a compiler and interpreter is called just-in-time compilation, or JIT as we often hear it. This is the entire process of executing a piece of JS code in V8.

  1. First, the AST is generated through lexical analysis and grammar analysis
  2. Convert the AST to bytecode
  3. Bytecode is executed line by line by interpreter. When hot code is encountered, the compiler is started to compile and generate the corresponding machine code to optimize the execution efficiency

The disassembly of this problem is here, I hope to inspire you

Chapter 4: How to Understand EventLoop — Macro and Micro Tasks

Macrotasks are introduced

In JS, most tasks are executed on the main thread. Common tasks include:

  • Render event
  • User interaction events
  • Js script Execution
  • Network requests, file read/write completion events, and so on.

In order for these events to work properly, the JS engine needs to arrange the order in which they are executed. V8 actually stores these tasks in a queue. The simulation is as follows:

bool keep_running = true;
void MainTherad(){
  for(;;) {// Execute the tasks in the queue
    Task task = task_queue.takeTask();
    ProcessTask(task);
    
    // Execute the task in the delay queue
    ProcessDelayTask()
    if(! keep_running)// If the exit flag is set, exit the thread loop directly
        break; }}Copy the code

We’re using a for loop that pulls tasks out of the queue and executes them, which makes sense. But there are two types of task queues. In addition to the task queues mentioned above, there is also a delay queue that handles timer callbacks such as setTimeout/setInterval. As mentioned above, tasks in normal and delay queues are macro tasks

Introduction of microtasks

For each macro task, there is an internal queue of microtasks. So why microtasks? When do microtasks take place? Microtasks were originally introduced to solve the problem of asynchronous callbacks. How many different ways can asynchronous callbacks be handled? To sum up, there are two points:

  1. The asynchronous callback is enqueued to the macro task queue
  2. Place the asynchronous callback at the end of the current macro task

If the first method is used, the time to execute the callback should be after all the previous macro tasks have completed. If the current task queue is very long, the callback will not be executed, causing application lag. To avoid this problem, V8 introduced the second method, which is the microtask solution. A microtask queue is defined in each macro task. When the macro task is completed, the microtask queue will be checked. If it is empty, the next macro task will be directly executed

Common microtasks are

  • MutationObserver
  • Promise. Then (or reject).
  • And other technologies developed based on Promise (such as the FETCH API),
  • V8’s garbage collection process is also included

Chapter 5: What to Understand about EventLoop — the Browser

Theory is not easy to understand, so let’s start with an example:

console.log('start')

setTimeout(() = > {
  console.log('timeout')})Promise.resolve().then(() = > {
  console.log('resolve')})console.log('end')
Copy the code

Let’s break it down:

  1. The entire script is initially executed as a macro task, and the synchronous code is pushed directly onto the execution stack. ), so start and end are printed first
  2. SetTimeout is placed in the macro task queue as a macro task
  3. Promise.then is put into the microtask queue as a microtask
  4. When the macro task is complete, check the microtask queue and find a promise. then, execute first
  5. Next, go to the next macro task, setTimeout, and execute

So the final order is:

start
end
resolve
timeout
Copy the code

This gives you an intuitive sense of the EventLoop execution process in the browser environment. However, this is only part of the situation, let’s do a more complete summary

  1. The entire script is initially executed as the first macro task
  2. During the execution, the synchronized code is executed directly, the macro task enters the macro task queue, and the micro task enters the micro task queue
  3. When the current macro task is finished, check the microtask queue. If there are any microtask queues, execute them in sequence until the microtask queue is empty
  4. Performs rendering of the browser UI thread
  5. Check whether there are Web worker tasks and execute them
  6. Execute the new macro task at the head of the queue, return to 2, and repeat until both the macro and microtask queues are empty

I’ll leave you with a final exercise:

Promise.resolve().then(() = > {
  console.log("Promise1")
  setTimeout(() = > {
    console.log("setTimeout2")},0)})setTimeout(() = > {
  console.log("setTimeout1")
  Promise.resolve().then(() = > {
    console.log("Promise2")})},0)

console.log("start")

// start
// Promise1
// setTimeout1
// Promise2
// setTimeout2
Copy the code

Chapter 6: How to Understand EventLoop — NodeJS

Nodejs is quite different from eventLoop in the browser, so it’s worth mentioning if you’ve read some articles about eventLoop in NodeJS, or if you’ve been confused by these flowcharts:Don’t be alarmed by this, it’s a step by step breakdown of nodeJS’s event loop in the clearest way possible, leaving behind obscure flowcharts

1. Three key stages

First, let’s take a look at nodeJS’s three very important implementation phases:

  1. The stage in which the timer callback is executed. Check the timer and execute the callback if the time is up. These timers are setTimeout and setInterval. We’ll call this a timer
  2. Polling stage (poll). The main thread is notified of asynchronous operations such as file I/O, network I/O, etc. It is through data, connect and other events that the event cycle reaches the poll stage. After reaching this stage:
  • If a timer already exists and a timer is running out, the eventLoop will return to the Timer phase
  • If there is no timer, it will look at the callback function queue
    • If the queue is not empty, take out the methods in the queue and execute them in sequence
    • If the queue is empty, check if there is a callback for setImmdiate
      • Some go to the check phase (described below)
      • If no callback is queued, the callback will be queued for a certain amount of time, and will be executed immediately. After a period of time, the system automatically enters the check phase
  1. The check phase. This is a simpler stage, where the setImmdiate callback is performed directly.

These three stages are a circular process. Now eventLoop isn’t complete, so let’s go through all of them

2. Perfect

At the end of phase 1, nodeJS may not immediately wait for the response of the asynchronous event. At this time, nodeJS will enter the callback phase of THE I/O exception, such as ECONNREFUSED, which will execute the callback. At the end of the check phase, it also enters the callback phase that closes the event. If a socket or handle is suddenly closed, such as socket.destroy(), the callback to the close event is executed at this stage. Nodejs eventLoop is divided into the following stages:

  1. The timer period
  2. Abnormal I/O callback
  3. Idle, ready state (end of phase 2, before poll is triggered)
  4. Poll phase
  5. The check phase
  6. Close the callback phase of the event

Is it much clearer?

3. Example demonstration

Ok, let’s practice the last exercise:

setTimeout(() = >{
    console.log('timer1')
    Promise.resolve().then(function() {
        console.log('promise1')})},0)

setTimeout(() = >{
    console.log('timer2')
    Promise.resolve().then(function() {
        console.log('promise2')})},0)

/*
timer1
promise1
time2
promise2
*/
Copy the code

Node >= 11 will behave differently from node >= 11, which will behave the same as the browser. A timer will run as soon as it finishes running, while a timer under node 11 will behave differently:

If the first timer task out of the team and executed, it is found that the team leader task is still a timer, then the micro-task is temporarily saved, directly to execute the new timer task, when the new timer task is executed, and then one by one to execute the micro-task generated in the middle

So it prints something like this:

timer1
timer2
promise1
promise2
Copy the code

4. Main differences between NodeJS and browsers regarding eventLoop

The main difference between the two is that the microtasks in the browser are performed in each corresponding macro task, whereas the microtasks in NodeJS are performed in different stages

5. A note on Process. nextTick

Process. nextTick is a separate eventLoop task queue that is checked after each eventLoop phase is completed, and if there are tasks in the queue, those tasks are prioritized over microtasks

Chapter 7: How is asynchronous, non-blocking I/O implemented in NodeJS?

Asynchronous I/O and non-blocking I/O are two different technologies that are used to implement asynchronous I/O and non-blocking I/O.

What is I/O?

First, I think it is necessary to explain the concept of I/O. I/O is Input/Output. On the browser side, there is only one type of I/O, which is network I/O, sending a network request using Ajax and then reading the returned content. Going back to NodeJS, this kind of I/O scenario is actually more extensive, and can be divided into two main types:

  • File I/O For example, use the FS module to read and write files
  • Network I/O such as HTTP modules initiate network requests

Blocking and non-blocking I/O

Blocking and non-blocking I/O are really specific to the operating system kernel, not NodeJS itself. The characteristic of blocking I/O is that the call is not finished until the operating system has completed all operations, whereas non-blocking I/O is returned immediately after the call, without waiting for the operating system kernel to complete the operation

Blocking I/O While the operating system is performing I/O operations, our application is in a waiting state, doing nothing

Non-blocking I/O, our NodeJS application can do other things after the call returns, while the operating system is doing I/O. This makes the most of the waiting time and improves execution efficiency, but raises the question of how the NodeJS application knows that the operating system has completed the I/O operation.

To let NodeJS know that the operating system has finished the I/O operation, it needs to check the operating system repeatedly, which is polling. For polling, there are several options:

  1. Polling to check the I/O status until the I/O is complete. This is the most primitive way, and the least efficient, and will keep the CPU on hold. The effect is the same as blocking I/O
  2. The file descriptor (the file credential between the operating system and NodeJS during file I/O) is traversed to determine whether the I/O is complete, and the state of the file descriptor changes when the I/O is complete. But CPU polling is still very expensive
  3. Epoll mode. That is, the CPU will sleep if THE I/O is not completed at the time of polling, and wake up the CPU when it is finished

In short, the CPU is either double-checked for I/O, double-checked for file descriptors, or hibernated, which is not a good use. What we want is:

Nodejs applications can execute other logic directly after making I/O calls. The operating system silently completes the I/O and sends nodeJS a complete signal, and NodeJS performs callback operations.

This is the ideal situation and the effect of asynchronous I/O. How do you achieve this effect?

The nature of asynchronous I/O

Linux natively exists in such a way, namely (AIO), but with two fatal flaws:

  1. Asynchronous I/O support is not available on other systems.
  2. Unable to take advantage of system cache

Asynchronous I/O schemes in NodeJS

Is there nothing we can do? This is true in the case of single threads, but it becomes much easier to think outside the box and think about the problem in terms of multiple threads. We could have one process do the computation, and some others do the I/O calls, and then signal the I/O to the computation thread to perform the callback. Wouldn’t that be fine? Yes, asynchronous I/O is implemented using such thread pools. On Linux, this can be done directly using a thread pool, while on Windows, the IOCP system API(which is still done internally using a thread pool) is used. With operating system support, how does NodeJS interconnect with these operating systems for asynchronous I/O? For file I/O let’s take a piece of code as an example:

let fs = require('fs');
fs.readFile('/test.txt', function (err, data) {
    console.log(data); 
});
Copy the code

Execute the process

Here’s what happens when you execute code:

  1. First, fs.readfile calls Node’s core module fs.js
  2. Next, Node’s core module calls the built-in node_file.cc module to create the corresponding file I/O observer object.
  3. Finally, depending on the platform (Linux or Window), the built-in module makes system calls through the Libuv middle tier

Libuv call procedure unpacking

Here we go! How do you make system calls in Libuv? So what’s going on in uv_fs_open()?

1. Create a request object

In Windows, for example, we create a file I/O request object and inject the callback function into it

req_wrap->object_->Set(oncomplete_sym, callback);
Copy the code

Req_wrap is the request object, and the corresponding value of the onComplete_sym property of object_ in req_wrap is the callback function passed in our NodeJS application code

2. Push the thread pool and return the call

Once the object is wrapped, the QueueUserWorkItem() method pushes the object into the thread pool for execution. Now the JS call is returned directly, and our JS application code can continue to execute. Of course, the current I/O operation will also be executed in the thread pool, which completes the asynchron. The next step is to execute the callback notification

3. Callback notification

In fact, it doesn’t matter whether the I/O in the thread pool is blocked or not, because the purpose of asyncio has been accomplished. What matters is what happens after I/O is complete. Before the introduction to the subsequent story, to introduce the two important methods: GetQueuedCompletionStatus and PostQueuedCompletionStatus

  1. Remember eventLoop? In every Tick will calls GetQueuedCompletionStatus check thread pool for execution of the request, if you have said is ripe, you can perform the callback
  2. PostQueuedCompletionStatus method is submitted to the IOCP state, tell it to the current I/O completion

When the corresponding thread after completion of the I/O, the results obtained will be stored and saved to the corresponding request object, and then submit to the IOCP call PostQueuedCompletionStatus method completes, and thread to the operating system. Once the EventLoop polling operations, calls GetQueuedCompletionStatus detected in the finished state, would put the request object to the I/O observer (before the stage, now finally on) I/O observer is now out of the request object stored as a result, It also fetches its onComplete_sym property, the callback function (look back to step 1 if you don’t understand this property). Pass the former as a function argument to the latter, and execute the latter. Here, the callback is successfully executed! Conclusion:

  1. Blocking and non-blocking I/O are really specific to the operating system kernel. The characteristic of blocking I/O is that the call is not finished until the operating system has completed all operations, whereas non-blocking I/O is returned immediately after the call, without waiting for the operating system kernel to complete the operation.
  2. Asynchronous I/O in NodeJS adopts multi-threaded mode, from
    • EventLoop
    • The I/O observer
    • The request object
    • The thread pool

The four elements cooperate and realize together

Chapter 8: WHAT are the schemes of JS asynchronous programming? Why do these schemes exist?

JS single thread, EventLoop, and asynchronous I/O are some of the underlying features that we’ve dissected in detail. After exploring the underlying mechanics, we also need to understand how the code is organized, which is the part closest to our most routine development. The way asynchronous code is organized directly determines the efficiency of development and maintenance, and its importance cannot be underestimated. While the underlying mechanics remain the same, the way asynchronous code is organized has changed dramatically with the development of the ES standard. Then let’s find out!

Callback function era

Many nodeJS beginners have experienced this at one point or another, and many of node’s native apis look something like this:

fs.readFile('xxx', (err, data) => {
})
Copy the code

A typical higher-order function passes a callback function as a function parameter to readFile. Over time, however, this method of passing in callbacks has its drawbacks, such as the following:

fs.readFile('1.json'.(err, data) = > {
    fs.readFile('2.json'.(err, data) = > {
        fs.readFile('3.json'.(err, data) = > {
            fs.readFile('4.json'.(err, data) = >{}); }); }); });Copy the code

Callback nested callback, also known as callback hell. This code is very poorly readable and maintainable because there are so many nested levels. A serious problem is that each task may fail, and each failure needs to be dealt with in a callback, adding to the clutter of the code

Promise time

The new Promise in ES6 solves callback hell nicely, while incorporating error handling. The resulting code looks something like this:

readFilePromise('1.json').then(data => {
    return readFilePromise('2.json')
}).then(data => {
    return readFilePromise('3.json')
}).then(data => {
    return readFilePromise('4.json')});Copy the code

In the way of chain call to avoid a lot of nesting, but also in line with people’s linear way of thinking, greatly convenient asynchronous programming

Co + Generator mode

Use the coroutine to complete the Generator functions, and use the CO library to execute the code sequentially, writing it synchronously and allowing asynchronous operations to be executed sequentially

co(function* () {
  const r1 = yield readFilePromise('1.json');
  const r2 = yield readFilePromise('2.json');
  const r3 = yield readFilePromise('3.json');
  const r4 = yield readFilePromise('4.json');
})
Copy the code

Async + await

This is a new keyword in ES7. All async functions return a Promise object by default, and more importantly async + await allows asynchronous code to be written synchronously without the need for third-party library support.

const readFileAsync = async function () {
  const f1 = await readFilePromise('1.json')
  const f2 = await readFilePromise('2.json')
  const f3 = await readFilePromise('3.json')
  const f4 = await readFilePromise('4.json')}Copy the code

These four classic asynchronous programming approaches are briefly reviewed, but since I’m looking at the big picture from a bird’s eye view, I decided it was more important to know what it was than to know the details, so I didn’t expand on them. That’s ok, but let’s dive into the nature of asynchronous programming for these specific solutions

Chapter 9: Can we simply implement the node callback mechanism?

In fact, the way of callback function internal use of publish-subscribe model, here we simulate the implementation of node Event module as an example to write the mechanism of callback function

function EventEmitter() {
  this.events = new Map(a); }Copy the code

The EventEmitter needs to implement these methods: addListener, removeListener, once, removeAllListener, and EMIT

The first is addListener:

The once parameter indicates whether to fire only once
const wrapCallback = (fn, once = false) = > ({ callback: fn, once });

EventEmitter.prototype.addListener = function (type, fn, once = false) {
  let handler = this.events.get(type);
  if(! handler) {// Bind a callback to the type event
    this.events.set(type, wrapCallback(fn, once));
  } else if (handler && typeof handler.callback === 'function') {
    // Currently there is only one callback for type events
    this.events.set(type, [handler, wrapCallback(fn, once)]);
  } else {
    // Current number of type event callbacks >= 2handler.push(wrapCallback(fn, once)); }}Copy the code

RemoveLisener is implemented as follows:

EventEmitter.prototype.removeListener = function (type, listener) {
  let handler = this.events.get(type);
  if(! handler)return;
  if (!Array.isArray(handler)) {
    if (handler.callback === listener.callback) this.events.delete(type);
    else return;
  }
  for (let i = 0; i < handler.length; i++) {
    let item = handler[i];
    if (item.callback === listener.callback) {
      // Delete the callback. Note the collapse of the array, where the following elements move forward one bit. I want to --
      handler.splice(i, 1);
      i--;
      if (handler.length === 1) {
        // The length of the array is 1
        this.events.set(type, handler[0]); }}}} Copy codeCopy the code

The implementation of once is simple. First call addListener to add the once flag to the callback object, and then emit the once: true flag by iterating through the callback list

EventEmitter.prototype.once = function (type, fn) {
  this.addListener(type, fn, true);
}

EventEmitter.prototype.emit = function (type, ... args) {
  let handler = this.events.get(type);
  if(! handler)return;
  if (Array.isArray(handler)) {
    // Iterate over the list and execute the callback
    handler.map(item= > {
      item.callback.apply(this, args);
      // Once: true is removed directly
      if (item.once) this.removeListener(type, item); })}else {
    // Only one callback is executed
    handler.callback.apply(this, args);
  }
  return true;
}
Copy the code

Finally, removeAllListener:

EventEmitter.prototype.removeAllListener = function (type) {
  let handler = this.events.get(type);
  if(! handler)return;
  else this.events.delete(type);
}
Copy the code

Now let’s test it out:

let e = new EventEmitter();
e.addListener('type'.() = > {
  console.log("Type event triggered!");
})
e.addListener('type'.() = > {
  console.log("WOW! The type event is triggered again!);
})
function f() { 
  console.log("Type event I only trigger once"); 
}

e.once('type', f)
e.emit('type');
e.emit('type');
e.removeAllListener('type');
e.emit('type');


// type event triggered!
// WOW! The Type event is triggered again!
// I only trigger the type event once
// type event triggered!
// WOW! The Type event is triggered again!
Copy the code

This is a simple Event. Why is it simple? Because there are a lot of details that have not been considered:

  1. With fewer parameters, Call performs better than Apply, and vice versa. Therefore, call or apply can be invoked when the callback is executed, depending on the situation
  2. Considering the memory capacity, the maximum value of the callback list should be set. When the maximum value is exceeded, some callbacks should be selected for deletion
  3. The robustness needs to be improved. The verification of parameters is ignored in many places

However, the purpose of this case is just to take you to master the core principle, if you write three or four hundred lines here is not meaningful, interested can go to see the Node Event module source, in the face of various details and boundary cases to do a detailed processing

Chapter 10: Promise’s Questions — With what did Promise kill Callback Hell?

Question: What is callback hell

  1. The problem of multi-layer nesting.
  2. There are two possibilities (success or failure) for each task. You need to handle the two possibilities respectively after each task is executed.

These two problems are particularly prominent in the era of callback functions. Promise was created to address both of these issues

The solution

Promise uses three major techniques to solve callback hell:

  • The callback function delays binding
  • Return value penetration
  • Error bubble

Let’s start with an example:

let readFilePromise = (filename) = > {
  fs.readFile(filename, (err, data) = > {
    if(err) {
      reject(err);
    }else {
      resolve(data);
    }
  })
}
readFilePromise('1.json').then(data= > {
  return readFilePromise('2.json')});Copy the code

See, the callback function is not declared directly, but is passed in via the later THEN method, that is, deferred. This is the callback function delayed binding and then we do the following tweaks:

let x = readFilePromise('1.json').then(data= > {
  return readFilePromise('2.json')// This is the return Promise
});
x.then(/* Internal logic omits */)
Copy the code

We create different types of promises based on the incoming value of the callback function in THEN, and then pass the returned Promise through the outer layer for subsequent calls. The x here refers to the internally returned Promise, and then the chain calls can be made after the x. This is how the return value penetrates. The two techniques work together to write deep nested callbacks in the following form:

readFilePromise('1.json').then(data= > {
    return readFilePromise('2.json');
}).then(data= > {
    return readFilePromise('3.json');
}).then(data= > {
    return readFilePromise('4.json');
});
Copy the code

It’s a lot cleaner, and more importantly, it’s a lot more linear and a much better development experience. The two techniques combine to produce the effect of chain calls. This solves the problem of multiple layers of nesting, but what about the other problem of handling success and failure separately at the end of each task? Promise took the error bubble approach. In fact, it is very simple to understand, let’s look at the effect:

readFilePromise('1.json').then(data= > {
    return readFilePromise('2.json');
}).then(data= > {
    return readFilePromise('3.json');
}).then(data= > {
    return readFilePromise('4.json');
}).catch(err= > {
  // xxx
})
Copy the code

This way errors are passed backwards and caught, so you don’t have to check for errors as often

To solve the effect

  • Implementation of chain call, solve the problem of multi-layer nesting
  • One-stop processing after error bubbling is realized to solve the problem of incorrect judgment and increased code confusion in each task

Chapter 11: The Promise Question — Why did Promises introduce microtasks?

At this point, if you haven’t touched on promises yet, it’s important to look at the MDN documentation and understand how to use them, or you’ll get confused. The execution function in the Promise is synchronous, but there is an asynchronous operation that calls either resolve when the asynchronous operation ends or Reject when it fails, both of which enter the EventLoop as microtasks. But have you ever wondered why Promise introduced microtasks for callbacks?

The solution

Back to the problem itself, it’s really a matter of how to handle callbacks. To sum up, there are three ways:

  1. Use synchronous callbacks until the asynchronous task is complete before proceeding to subsequent tasks
  2. With asynchronous callbacks, place the callback function at the end of the macro task queue
  3. With asynchronous callbacks, place the callback function at the end of the current macro task

Advantages and disadvantages compared

The first way is obviously not desirable, because the synchronization problem is very obvious, will make the entire script blocked, waiting for the current task, the back of the task cannot be enforced, and that part of the waiting time is can be used to do other things, cause the CPU utilization is very low, and there is another deadly problem, is unable to achieve the effect of delayed binding

If you use the second approach, the resolve/reject callback should be executed after all the previous macro tasks are complete. If the current task queue is very long, the callback will not be executed, causing the application to stall

To address the above solution and the need for delayed binding, Promise took a third approach, introducing a microtask that placed the execution of the resolve(reject) callback at the end of the current macro task, using microtasks to address two major pain points:

  1. Using asynchronous callbacks instead of synchronous callbacks solves the problem of wasting CPU performance
  2. To solve the real-time problem of callback execution, the current macro task is executed last

Ok, the basic implementation idea of Promise has been clarified, and I believe you already know why it is designed this way, so let’s figure out how it is designed inside step by step

Chapter 12: The Promise Question — How does a Promise implement a chain call?

From now on, let’s start implementing a fully functional Promise, digging into the details step by step. Let’s start with the chain call

Simplified version implementation

Write the first version of the code first:

// Define three states
const PENDING = "pending";
const FULFILLED = "fulfilled";
const REJECTED = "rejected";

function MyPromise(executor) {
    let self = this; // Cache the current Promise instance
    self.value = null;
    self.error = null;
    self.status = PENDING;
    self.onFulfilled = null; // Successful callback function
    self.onRejected = null; // Failed callback function

    const resolve = (value) = > {
        if(self.status ! == PENDING)return;
        setTimeout(() = > {
            self.status = FULFILLED;
            self.value = value;
            self.onFulfilled(self.value);// Execute a success callback in resolve
        });
    };

    const reject = (error) = > {
        if(self.status ! == PENDING)return;
        setTimeout(() = > {
            self.status = REJECTED;
            self.error = error;
            self.onRejected(self.error);// Execute a success callback in resolve
        });
    };

    try {// An exception may occur during execution
        executor(resolve, reject);
    } catch (e) {
        reject(e);/ / promise failed
    }
}




MyPromise.prototype.then = function (onFulfilled, onRejected) {
    if (this.status === PENDING) {
        this.onFulfilled = onFulfilled;
        this.onRejected = onRejected;
    } else if (this.status === FULFILLED) {
        // If the status is fulfilled, execute the success callback directly and pass in the success value
        onFulfilled(this.value)
    } else {
        // If the status is Rejected, execute the failure callback directly and pass in the failure cause
        onRejected(this.error)
    }
    return this;
}
Copy the code

As can be seen, the essence of Promise is a finite state machine, which has three states:

  • PENDING (waiting)
  • FULFILLED (successful)
  • REJECTED (failure)

For a Promise, the state change is irreversible, that is, after the waiting state changes to another state, it cannot be changed again. Going back to the current version of Promise, though, there are some problems

Set the callback array

Only one callback function can be executed first, not multiple callback bindings, such as the following:

let promise1 = new MyPromise((resolve, reject) = > {
  fs.readFile('./001.txt'.(err, data) = > {
    if(! err){ resolve(data); }else{ reject(err); }})});let x1 = promise1.then(data= > {
  console.log("First demonstration", data.toString());    
});
let x2 = promise1.then(data= > {
  console.log("Second presentation", data.toString());    
});
let x3 = promise1.then(data= > {
  console.log("Third show", data.toString());    
});
Copy the code

What if I bind three callbacks that I want to execute together after resolve()? Ondepressing and onRejected need to be changed to an array. When you call resolve, you can take out the methods and perform them one by one

self.onFulfilledCallbacks = [];
self.onRejectedCallbacks = [];
Copy the code
MyPromise.prototype.then = function(onFulfilled, onRejected) {
  if (this.status === PENDING) {
    this.onFulfilledCallbacks.push(onFulfilled);
    this.onRejectedCallbacks.push(onRejected);
  } else if (this.status === FULFILLED) {
    onFulfilled(this.value);
  } else {
    onRejected(this.error);
  }
  return this;
}
Copy the code

Next modify the parts of the resolve and Reject methods that perform callbacks:

/ / resolve
self.onFulfilledCallbacks.forEach((callback) = > callback(self.value))

/ / reject
self.onRejectedCallbacks.forEach((callback) = > callback(self.error))
Copy the code

The chain call is complete

Let’s test with the current code:

let fs = require('fs');
let readFilePromise = (filename) = > {
  return new MyPromise((resolve, reject) = > {
    fs.readFile(filename, (err, data) = > {
      if(! err){ resolve(data); }else {
        reject(err);
      }
    })
  })
}
readFilePromise('./001.txt').then(data= > {
  console.log(data.toString());    
  return readFilePromise('./002.txt');
}).then(data= > {
  console.log(data.toString());
})

// 001. TXT contents
// 001. TXT contents
Copy the code

Yi? How to print two 001, the second time is not read 002 file? Here’s the problem:

MyPromise.prototype.then = function(onFulfilled, onRejected) {
  / /...
  return this;
}
Copy the code

I’ll write it this way every time I return the first Promise. The second Promise returned from the then function is simply ignored! The implementation of THEN needs to be improved. We now need to pay attention to the return value of THEN

MyPromise.prototype.then = function (onFulfilled, onRejected) {
  let bridgePromise;
  let self = this;
  
  if (self.status === PENDING) {
    return bridgePromise = new MyPromise((resolve, reject) = > {
      self.onFulfilledCallbacks.push((value) = > {
        try {
          // See? To get the result returned by the callback in then.
          let x = onFulfilled(value);
          resolve(x);
        } catch(e) { reject(e); }}); self.onRejectedCallbacks.push((error) = > {
        try {
          let x = onRejected(error);
          resolve(x);
        } catch(e) { reject(e); }}); }); }/ /...
}
Copy the code

If the current state is PENDING, add the above function to the callback array. When the Promise state changes, the corresponding callback array will be iterated and the callback will be executed. However, there are still some problems with this extent:

  1. The first case where the two parameters in then are not passed is not handled
  2. If the result returned by the callback in THEN (the x above) is a Promise, resolve is resolved, which we do not want

How to solve these two problems? First, judge the case that the parameters are not transmitted:

The successful callback does not pass it a default function
onFulfilled = typeof onFulfilled === "function" ? onFulfilled : value= > value;

// Throw an error for a failed callback
onRejected = typeof onRejected === "function" ? onRejected : error= > { throw error };
Copy the code

Then handle the case where a Promise is returned and then:

function resolvePromise(bridgePromise, x, resolve, reject) {
  If x is a promise
  if (x instanceof MyPromise) {
    // Unpack the promise until the return value is not a PROMISE
    if (x.status === PENDING) {
      x.then(y= > {
        resolvePromise(bridgePromise, y, resolve, reject);
      }, error= > {
        reject(error);
      });
    } else{ x.then(resolve, reject); }}else {
    // Resolve if no Promise is maderesolve(x); }}Copy the code

Then make the following changes in the then method implementation:

resolve(x)  ->  resolvePromise(bridgePromise, x, resolve, reject);
Copy the code

I want to emphasize what the recursive calls to resolve and reject are all about. They control the state of the bridgePromise that was originally passed in, which is very important. Next, We implement a logical success state call then when the Promise state is not PENDING:

if (self.status === FULFILLED) {
  return bridgePromise = new MyPromise((resolve, reject) = > {
    try {
      // When the status becomes successful, there is a corresponding self.value
      let x = onFulfilled(self.value);
      Resolve (x) resolve(x
      resolvePromise(bridgePromise, x, resolve, reject);
    } catch(e) { reject(e); }})}Copy the code

Failed state then:

if (self.status === REJECTED) {
  return bridgePromise = new MyPromise((resolve, reject) = > {
    try {
      // If the state becomes failed, there is self.error
      let x = onRejected(self.error);
      resolvePromise(bridgePromise, x, resolve, reject);
    } catch(e) { reject(e); }}); }Copy the code

As promised in Promise A+, both successful and failed callbacks are microtasks. Since JS in the browser cannot touch the distribution of the underlying microtasks, setTimeout(belonging to the category of macro tasks) can be directly used to simulate, and setTimeout can be used to wrap the tasks to be executed. Of course, The resolve implementation is the same, but it’s not really a microtask

if (self.status === FULFILLED) {
  return bridgePromise = new MyPromise((resolve, reject) = > {
    setTimeout(() = > {
      / /...})}Copy the code
if (self.status === REJECTED) {
  return bridgePromise = new MyPromise((resolve, reject) = > {
    setTimeout(() = > {
      / /...})} copy the codeCopy the code

Ok, now that we have basically implemented the THEN method, let’s test the code we just tested, and print it as follows:

001The content of the.txt002The content of the.txtCopy the code

As you can see, the chain call has been successfully completed

Error capture and bubbling mechanism analysis

Now implement the catch method:

Promise.prototype.catch = function (onRejected) {
  return this.then(null, onRejected);
}
Copy the code

Okay, so catch is the syntactic sugar for then. More important than implementation is understanding the error bubbling mechanism, which means that once an error occurs in the middle, it can be caught with a catch at the end. It’s not hard to look back at how Promise works, with a key line of code:

// Then
onRejected = typeof onRejected === "function" ? onRejected : error= > { throw error };
Copy the code

Once there is an error in the PENDING Promise state, the state will inevitably change to failure, and then the onRejected function will be executed. The onRejected function will throw an error, and the new Promise state will change to failure. The new Promise state changes to onRejected…… if it fails This continues until the error is caught and the drop is stopped. This is the error bubbling mechanism for Promises. So far, Promise has three magic tricks: callback function delay binding, callback return value penetration, and error bubbling

Chapter 13: The Question of Promises — Resolve, Reject, and Finally to implement promises

Realize the Promise. Resolve

There are three key elements to implementing the Resolve static method:

  1. Pass the parameter as a Promise and return it directly
  2. The parameter is passed as a Thenable object, and the returned Promise follows the object, taking its final state as its own
  3. Otherwise, a Promise object with that value as a success status is returned directly

The concrete implementation is as follows:

Promise.resolve = (param) => {
  if(param instanceof Promise) return param;
  return new Promise((resolve, reject) => {
    if(param && param.then && typeof param.then === 'function') {
      // A successful param state calls resolve, which changes the state of the new Promise to successful and vice versa
      param.then(resolve, reject);
    }else{ resolve(param); }})}Copy the code

Realize the Promise. Reject

The argument passed in promise.reject is passed down as a reason as follows:

Promise.reject = function (reason) {
    return new Promise((resolve, reject) => {
        reject(reason);
    });
}
Copy the code

Realize the Promise. Prototype. Finally

Regardless of whether the current Promise succeeds or fails, a call to finally executes the function passed in finally and passes the value down unchanged

Promise.prototype.finally = function(callback) {
  this.then(value => {
    return Promise.resolve(callback()).then(() => {
      return value;
    })
  }, error => {
    return Promise.resolve(callback()).then(() => {
      throwerror; })})}Copy the code

Chapter 14: The Question of Promise — All and Race to Fulfill Promises

Realize the Promise. All

For the all method, the following core functions need to be done:

  1. Resolve is passed as an empty iterable
  2. If one of the arguments fails, the promise object returned by promise.all fails.
  3. In any case, promise.all returns an array as the result of the completion state of the Promise

The concrete implementation is as follows:

Promise.all = function(promises) {
  return new Promise((resolve, reject) => {
    let result = [];
    let index = 0;
    let len = promises.length;
    if(len === 0) {
      resolve(result);
      return;
    }
   
    for(let i = 0; i < len; i++) {
      // Why not just promise[I]. Then, because promise[I] may not be a promise
      Promise.resolve(promise[i]).then(data => {
        result[i] = data;
        index++;
        if(index === len) resolve(result);
      }).catch(err => { reject(err); })}})}Copy the code

Realize the Promise. Race

The implementation of race is relatively simple: once a promise is completed, resolve and stop execution

Promise.race = function(promises) {
  return new Promise((resolve, reject) = > {
    let len = promises.length;
    if(len === 0) return;
    for(let i = 0; i < len; i++) {
      Promise.resolve(promise[i]).then(data= > {
        resolve(data);
        return;
      }).catch(err= > {
        reject(err);
        return; })}})}Copy the code

At this point, we’ve fulfilled a full Promise. From the principle to the details, we step by step dismantling and implementation, I hope you know the Promise design after a few bright spots, can also manually realize a Promise, let their thinking level and hands-on ability to the next level!

Chapter 15: Tell us your understanding of generators and coroutines

Generators are a new syntax in ES6 that is a bit harder to get started with than the asynchronous syntax. So let’s take a good look at Generator syntax.

Generator execution flow

That’s the generator function, right? A generator is an asterisked “function “(note: it is not really a function) that can be paused and resumed using the yield keyword as an example:

function* gen() {
  console.log("enter");
  let a = yield 1;
  let b = yield (function () {return 2}) ();return 3;
}
var g = gen() // Block, no statement will be executed
console.log(typeof g)  // object Not "function"
console.log(g.next())  
console.log(g.next())  
console.log(g.next())  
console.log(g.next()) 
// enter
// { value: 1, done: false }
// { value: 2, done: false }
// { value: 3, done: true }
// { value: undefined, done: true }
Copy the code

As you can see, there are several key points to the execution of a generator:

  1. After calling gen(), the program blocks and does not execute any statements.
  2. After calling g.ext (), the program continues until it encounters yield program pause.
  3. The next method returns an object with two properties:value 和 done. The value ofThe result after the current yieldSaid and doneWhether the execution is complete, metreturnLater,donebyfalseintotrue

yield*

When one generator calls another, it becomes convenient to use yield*. Take the following example:

function* gen1() {
    yield 1;
    yield 4;
}

function* gen2() {
    yield 2;
    yield 3;
}
Copy the code

We want to do it in 1234 order. How do we do it? In GEN1, the modification is as follows:

function* gen1() {
    yield 1;
    yield* gen2();
    yield 4;
}
Copy the code

After making this change, call next in turn

Generator implementation mechanism – coroutine

You might be wondering, how exactly does a generator pause a function and how does it resume? Now we’re going to coroutine the execution mechanism what is coroutine? Coroutines are more lightweight than threads. In the environment of threads, a thread can have more than one coroutine, which can be understood as a task in a thread. Unlike processes and threads, coroutines are not managed by the operating system, but are controlled by specific application code

How coroutines work so you might say, well, isn’t JS executed in a single thread, can you execute all these coroutines together? The answer is: no. A thread can execute only one coroutine at a time. For example, A coroutine is currently executing A coroutine, and there is another B coroutine. If you want to execute B’s task, you must transfer the control of JS thread to B coroutine in A coroutine, so now B is executing, A is equivalent to being suspended. Here’s a concrete example:

function* A() {
  console.log("I am A");
  yield B(); // A stops and transfers thread execution to B
  console.log("It's over.");
}

function B() {
  console.log("I am B");
  return 100;  // returns, and returns thread execution to A
}
let gen = A();
console.log(gen.next().value); //{ value: 100, done: false }

gen.next();
/ / I am A
/ / I'm B
/ / over
Copy the code

In this process, A gives execution to B, that is, A initiates B, and we also call A the parent coroutine of B. So the last return 100 in B is actually passing 100 to the parent coroutine. It’s important to note that for coroutines, it’s not controlled by the operating system, it’s completely user defined, so there’s no overhead of process/thread context switching, which is a very important reason for high performance. OK, that’s the principle. You might also wonder: doesn’t this generator just pause-resume, pause-resume? What does it have to do with asynchrony? Also, if you call next every time you execute, can you do it all at once? In the next video, we’ll take a closer look at these problems

Chapter 16: How do I get Generator asynchronous code to complete in sequence?

There are really two problems:

  1. How does Generator relate to asynchrony?
  2. How do I complete Generator execution in sequence?

Thunk function

To understand the relationship between Generator and asynchronism, let’s take a look at the concept of thunk functions, although this is only one way to implement the relationship (Promise is the other way, as we’ll see later). For example, we now need to determine data types. The following logic can be written:

let isString = (obj) = > {
  return Object.prototype.toString.call(obj) === '[object String]';
};
let isFunction = (obj) = > {
  return Object.prototype.toString.call(obj) === '[object Function]';
};
let isArray = (obj) = > {
  return Object.prototype.toString.call(obj) === '[object Array]';
};
let isSet = (obj) = > {
  return Object.prototype.toString.call(obj) === '[object Set]';
};
Copy the code

As you can see, there is a lot of repetitive logic. Let’s encapsulate them:

let isType = (type) = > {
  return (obj) = > {
    return Object.prototype.toString.call(obj) === `[object ${type}] `; }}Copy the code

Now let’s do this:

let isString = isType('String');
let isFunction = isType('Function');
Copy the code

The corresponding isString and isFunction are isType functions, but they can still tell if the argument isString (Function), and the code is much cleaner

isString("123");
isFunction(val= > val);
Copy the code

Functions such as isType are called thunk functions. Its core logic is to take certain parameters, produce a custom function, and then use the custom function to accomplish the function. The thunk function implementation is a little more complicated than a single judgment function, but just that little complexity makes it much easier to do

The Generator and asynchronous

Thunk version

Taking file operations as an example, let’s see how asynchronous operations apply to generators

var fs = require('fs')

const readFileThunk = (filename) = > {
  return (callback) = >{ fs.readFile(filename, callback); }}Copy the code

ReadFileThunk is a thunk function. Part of the core of asynchronous operations is binding callback functions, and the thunk function does that for us. The file name is passed in first, and a custom function is generated for a file. This function passes in a callback that becomes the callback for an asynchronous operation. This associates the Generator with asynchrony and then we do the following:

const gen = function* () {
  const data1 = yield readFileThunk('001.txt')
  console.log(data1.toString())
  
  const data2 = yield readFileThunk('002.txt')
  console.log(data2.toString)
}
Copy the code

Then we let it finish:

var fs = require('fs')

const readFileThunk = (filename) = > {
  return (callback) = > {
    fs.readFile(filename, callback);
  };
};

const gen = function* () {
  const data1 = yield readFileThunk("001.txt");
  console.log(data1.toString());

  const data2 = yield readFileThunk("002.txt");
  console.log(data2.toString());
};

let g = gen();
// Step 1: Since approach is paused, we call next to start execution.
// Next returns a value, which is the result of yield. Thunk is the custom function that needs to be passed a callback
g.next().value((err, data1) = > {
  // Step 2: Get the last result, call next, pass in the result as an argument, and continue executing.
  // Similarly, value is passed in as a callback
  g.next(data1).value((err, data2) = > {
    g.next(data2);
  });
});

/* the contents of 001. TXT */
Copy the code

If there are many tasks, there will be many layers of nesting, which is not strong operability. It is necessary to encapsulate the code executed:

function run(gen){
  const next = (err, data) = > {
    let res = gen.next(data);
    if(res.done) return;
    res.value(next);
  }
  next();
}
run(g);
Copy the code

Ok, execute again, still printing the correct result. The code is only a few lines long, but it contains recursion, so feel free to use thunk to do an asynchronous operation

Promise version

Use the above example to make it easier to use Promise:

var fs = require("fs");

const readFilePromise = (filename) = > {
  return new Promise((resolve, reject) = > {
    fs.readFile(filename, (err, data) = > {
      if (err) {
        reject(err);
      } else{ resolve(data); }}); }).then((res) = > res);
};
const gen = function* () {
  const data1 = yield readFilePromise("001.txt");
  console.log(data1.toString());
  const data2 = yield readFilePromise("002.txt");
  console.log(data2.toString());
};

let g = gen();

g.next().value
  .then((data1) = > {
    return g.next(data1).value;
  })
  .then((data2) = > {
    return g.next(data2).value;
  });
  
/* The following output is displayed: 001. TXT contents 002. TXT contents */

Copy the code

Similarly, we can encapsulate code that executes Generator:

var fs = require("fs");

const readFilePromise = (filename) = > {
  return new Promise((resolve, reject) = > {
    fs.readFile(filename, (err, data) = > {
      if (err) {
        reject(err);
      } else{ resolve(data); }}); }).then((res) = > res);
};
const gen = function* () {
  const data1 = yield readFilePromise("001.txt");
  console.log(data1.toString());
  const data2 = yield readFilePromise("002.txt");
  console.log(data2.toString());
};

let g = gen();

function run(g) {
  const next = (data) = > {
    let res = g.next(data);
    if(res.done) return;
    res.value.then(data= > {
       next(data);
    })
  }
  next();
}

run(g);
  
/* The following output is displayed: 001. TXT contents 002. TXT contents */
Copy the code

It also outputs the correct results. The code is very concise, and I hope to see the recursive call process in detail by referring to the chain call example

Using co library

We have encapsulated the one-time execution of Generator asynchronous operations thunk and Promise, but mature toolkits already exist. If the well-known CO library, in fact, the core principle is that we have already written (just encapsulated in the Promise case of the implementation of the code), but the source code will handle various boundary cases. It’s very simple to use:

var fs = require('fs')
const co = require("co");

const readFilePromise = (filename) = > {
  return new Promise((resolve, reject) = > {
    fs.readFile(filename, (err, data) = > {
      if(err) {
        reject(err);
      }else {
        resolve(data);
      }
    })
  }).then(res= > res);
}
const gen = function* () {
  const data1 = yield readFilePromise('001.txt')
  console.log(data1.toString())
  const data2 = yield readFilePromise('002.txt')
  console.log(data2.toString())
}

let g = gen();
co(g).then(res= >{
  console.log(res);
})

/* The following output is displayed: 001. TXT contents 002. TXT contents */
Copy the code

A few lines of code do all the work for the Generator, so co and Generator are a perfect match.

Chapter 17: Explain how async/await works

Async /await is called the ultimate asynchronous solution in JS. It can write asynchronous code synchronously, like CO + Generator, with low-level syntax support, without the need for third-party libraries. Next, let’s take a look at what’s going on behind this grammatical sugar from a principle point of view

async

MDN definition: Async is a function that executes asynchronously and implicitly returns a Promise as a result.

async function func() {
  return 100;
}
console.log(func());
// Promise {<resolved>: 100}
Copy the code

This is the effect of implicitly returning a Promise

await

Let’s see what the await does, taking a piece of code as an example:

async function test() {
  console.log(100)
  let x = await 200
  console.log(x)
  console.log(200)}console.log(0)
test()
console.log(300)
Copy the code

Let’s analyze this program. Test = 0; test = 0; test = 0;

await 100;
Copy the code

Converted by the JS engine into a Promise:

let promise = new Promise((resolve,reject) = > {
   resolve(100);
})
Copy the code

Resolve is called, and the resolve task enters the microtask queue and then the JS engine will pause the current coroutine, give execution of the thread to the parent coroutine and return it to the parent coroutine, The first thing the parent coroutine does is call then on the Promise returned from the await to listen for state changes to the Promise

promise.then(value= > {
  // The related logic is called after the resolve execution
})
Copy the code

According to the EventLoop mechanism, the macro task of the current main thread is complete, and now the microtask queue is checked. There is still a Promise to execute resolve, and now the parent coroutine executes the callback passed in then. So what does this callback do

promise.then(value= > {
  // 1. Delegate thread execution to the test coroutine
  // 2. Pass value to test coroutine
})
Copy the code

Test receives 200 from the parent coroutine, assigns the value to A, and then executes the following statement, printing 200, 200. The final output is:

0
100
300
200
200
Copy the code

To sum up, async/await uses coroutines and promises to achieve synchronous writing of asynchronous code, in which Generator is an implementation of coroutines. Although the syntax is simple, the engine does a lot of work behind the coroutines, and we have also disassembled these works one by one. The code written with async/await is also more elegant and beautiful. Compared with the previous way of calling THEN continuously, the semantic is more obvious. Compared with CO + Generator, the performance is higher and the cost of getting started is lower, which is worthy of being the ultimate JS asynchronous solution!

Chapter 18: What is the problem with “await” in forEach? How to solve this problem?

Problem: For asynchronous code, forEach does not guarantee sequential execution

async function test() {
	let arr = [4.2.1]
	arr.forEach(async item => {
		const res = await handle(item)
		console.log(res)
	})
	console.log('the end')}function handle(x) {
	return new Promise((resolve, reject) = > {
		setTimeout(() = > {
			resolve(x)
		}, 1000 * x)
	})
}

test()
Copy the code
The expected results are:4 
2 
1End but it actually prints: end1
2
4
Copy the code

Question why

Why is that? I think it’s worth looking at the underlying implementation of forEach, right

// Core logic
for (var i = 0; i < length; i++) {
  if (i in array) {
    varelement = array[i]; callback(element, i, array); }}Copy the code

As you can see, forEach takes it and executes it directly, which makes it unable to guarantee the order in which the asynchronous tasks are executed. For example, the later task is short, and it may be executed before the earlier task

The solution

How to solve this problem? We use for… Of can be easily solved

async function test() {
  let arr = [4.2.1];
  for (const item of arr) {
    const res = await handle(item);
    console.log(res);
  }
  console.log("The end");
}

function handle(x) {
  return new Promise((resolve, reject) = > {
    setTimeout(() = > {
      resolve(x);
    }, 1000 * x);
  });
}

test();

End 4 2 1 */
Copy the code
async function test() {
  let arr = [4.2.1];
  async function asyncMethod(params) {
    for (const item in arr) {
      const res = await handle(arr[item]);
      console.log(res); }}await asyncMethod();
  console.log("The end");
}

function handle(x) {
  return new Promise((resolve, reject) = > {
    setTimeout(() = > {
      resolve(x);
    }, 1000 * x);
  });
}

test();

/* 4 2 1 End */
Copy the code

Solution Principle – Iterator

Okay, this seems like a simple problem to solve, but have you ever wondered why this works? In fact, for… Instead of iterating over execution in a crude way like forEach, of uses iterators to iterate over execution. First of all, for arrays, it’s an iterable data type. So what are iterable data types? The data type native to the [symbol. iterator] property is an iterable data type. Such as

  • An array of
  • Class arrays (e.g. Arguments, NodeList)
  • Set
  • Map

Iterables can be traversed by iterators

let arr = [4.2.1];

// function* arr(params) {
// yield* [4, 2, 1];
// }
// let iterator = arr(); 
let iterator = arr[Symbol.iterator](); // This is the iterator

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());

/*
{value: 4, done: false}
{value: 2, done: false}
{value: 1, done: false}
{value: undefined, done: true}
*/
Copy the code

Therefore, our code can be organized like this:

async function test() {
  let arr = [4.2.1];
  let iterator = arr[Symbol.iterator]();
  
  let res = iterator.next();

  while(! res.done) {let value = res.value;
    console.log(value);
    res = iterator.next();
  }
  console.log("The end");
}

test();

/* 
4
2
1
结束
*/
Copy the code

Multiple tasks successfully executed in sequence! Actually, for… The of loop code is the syntactic sugar of this code

Relearning generators

Go back to the code for iterator over the array [4,2,1]

let arr = [4.2.1];

/ / the iterator
let iterator = arr[Symbol.iterator]();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());

// {value: 4, done: false}
// {value: 2, done: false}
// {value: 1, done: false}
// {value: undefined, done: true}
Copy the code

Yi? The return value has value and done attributes, and the generator can also call next, which returns the same data structure. Yes, the generator itself is an iterator. Since it is an iterator, it can use for… Did you go through “of”? Of course, write a simple Fibonacci sequence (up to 144) :

function* fibonacci() {
  let [prev, cur] = [0.1]
  while (true) {
    [prev, cur] = [cur, prev + cur]
    yield cur
  }
}

for (let item of fibonacci()) {
  if (item > 144) break;
  console.log(item)
}

/* 1 1 2 3 5 8 13 21 34 55 89 144 */
Copy the code

Isn’t that cool? That’s the beauty of iterators, and it gives us a deeper understanding of generators than we thought our old acquaintance, Generator, could have