This is the 15th day of my participation in the August More Text Challenge.

preface

Chapter 4 workflow control of Async. Js

Problem solving: Need to perform a set of I/O operations (parallel or serial)

This problem is common in Nodejs and has a proper name, workflow control

The most popular is Async.js by Caolan McMahon

This chapter discusses what async.js can do in a NodeJS environment

The sequencing of asynchronous workflows

Example: Imagine reading all the files in your recipes directory alphabetically, concatenating them into a string and displaying them

Var fs = require('fs') process.chdir('recipes') var concatenation = 'fs.readdirsync ('.') .filter(function(filename){// skip directories that are not files return fs.statsync (filename).isfile ()}).foreach (function(filename){// Concatenation += fs.readfilesync (filename, 'utf8')}) console.log(concatenation)Copy the code

New requirement: Address the inefficiency of I/O blocking, where the application can do something else at the same time

Trouble: Can’t simply convert this code concatenation += fs.readFileSync(filename, ‘utf8’) into asynchronous code

fs.readFile(filename, 'utf8',function(err, contents){
    if(err) throw err
    concatenation += content
})
Copy the code

If this is the case, the length of time readFile takes to read the file is out of control, so that the file read first may not be read first. The order will be out of order.

Asynchronous data collection methods

The simplest idea is that the callback to the previous readFile runs the next readFile.

Var fs = require('fs') process.chdir('recipes') var concatenation = 'fs.readdir('.',function(err,filenames){ if(err) throw err function readFileAt(i){ var filename = filenames[i] fs.stat(filename,function(err,stats){ if(err) throw err if(! stats.isFile()) return readFileAt(i+1) fs.readFile(filename,'utf8',function(err,text){ if(err) throw err concatenation +  =text if(i+1===filenames.length){ return console.log(concatenation) } readFileAt(i+1) }) }) } readFileAt(0) })Copy the code

Question: This does not seem to solve the PROBLEM of I/O blocking, is my understanding of I/O blocking error code

The problem is that I am not familiar with Nodejs FS-related apis.

Fs. readdir = filenames; fs.readdir = filenames; fs.readdir = filenames;

2. After traversal is complete, run the readdir callback function

3. Declare the function readFileAt to read the array of Filenames files starting at subscript 0

4. Query the current reading status of filename instead of the file being read and operating on the next file

5. Confirm that the file is a file, read and operate the next file

All of these callbacks are designed to address I/O inefficiencies asynchronously

Here really make oneself feel doubt is not a second task, that is to say here only solve the asynchronous implementation, and does not give a more complex real scenarios, such as, users click on the button to generate the file name string, then the user during this period, a bit the other buttons, if not the asynchronous here, ordered the operation of the other buttons will be blocked, Until all strings are generated, you can continue with other operations.

Async. Js in functional form

Referring to the API of synchronous implementation, Async also provides asynchronous methods corresponding to filter and forEach

1. Async. filter and async.forEach are processed in parallel

2. Async. filterSeries and async.forEachSeries are processed sequentially

So what do we want to do in parallel and sequentially? That’s why you still need sequential processing.

1. The asynchronous order is unpredictable, although it is possible to generate an array after reading all the folders as before. And then joining. But it’s not the easiest way.

2. Nodejs has a limit on the number of files that can be read at the same time.

Let’s look at an example of sequential processing

Var async = require('async') var fs = require('fs') process.chdir('recipes') var concatenation = '' var dirContents = fs.readdirSync('.') async.filter(dirContents,isFilename,function(filenames){ async.forEachSeries(filenames,readAndConcat,onComplete) }) funciton isFilename(filename,callback){ fs.stat(filename,function(err,stats){ if(err)throw err callbacks(stats.isFile())  }) } funciton readAndConcat(filename,callback){ fs.readFile(filename,'utf8',funciton(err, fileContents){ if(err) return callback(err) concatenation += fileContents callback() }) } function onComplete(err){ if(err) throw err console.log(concatenation) }Copy the code

Async. Js task organization technology

The execution of asynchronous function sequences

Async.series ([],function(err,results){console.log('Completed in '+ (new date-start) + 'ms')}) //[] Funciton (callback){setTimeout(callback,100)} funciton (callback){ setTimeout(callback,200)}, funciton (callback){ setTimeout(callback,300)}, ]Copy the code

Parallel running of asynchronous functions

Async.parallel ([],function(err,results){console.log('Completed in '+ (new date-start) + 'ms')}) //[] We want the functions in this array to execute in parallel [] could be [funciton (callback){setTimeout(callback,100)}, funciton (callback){ setTimeout(callback,200)}, funciton (callback){ setTimeout(callback,300)}, ]Copy the code

Dynamic queuing technology for asynchronous workflows

Async. Series and Async. Parallel solve problems related to static task lists.

Async. queue resolves dynamic task lists, access task lists, limited concurrency (not all in sequence or all in parallel)

Deeper understanding of queues

Scenario: DMV dynamic management view, analogous to a bank call number, queue, separate from employees. Maintain a queue. Line up when you get there and get the number. When any clerk is free, call the next line number.

Async. queue takes two arguments, the worker function and concurrency. The former resolves what to process and the latter resolves how many people are allowed to process at the same time.

var async = require('async') function worker(data,callback){ console.log(data) callback() } var concurrency = 2 var Queue = async.queue(worker,concurrency) queue.push(1) callback) queue.push(2) queue.push(3)Copy the code

Queue of tasks

Push method of queue

Queue. Push () [1, 2, 3] / / equivalent to the queue. Push (1) queue. Push (2) queue. Push (3) / /!!!!!! Queue. Push ([1,2,3],function(err,result){console.log('Task complete! ')}) // The callback function here is the callback function for the worker functionCopy the code

Callbacks are important because there is no other way to get the internal state of the task being executed.

Handling of completion events

var async = require('async')
function worker(data, callback){
    setTimeout(callback,data)
}
var concurrency = 2
var queue = async.queue(worker, concurrency)
var start = new Date
queue.drain = function(){
    console.log('Completed in ' + (new Date - start) + 'ms')
}
queue.push([100,200,300])
Copy the code

The drain property fires when the task list is empty

Advanced callback methods for queues

Other event handlers besides drain

1. Call empty when the queue task starts running

2. Call the saturated method when the concurrency limit is reached

3. If the second argument to the push method is a function, it will be executed in subsequent executions (refer to the previous event model, which is not overridden here).

How Step works

Async’s biggest competitor is TimCasWell’s Step

Lightweight JavaScript library

(A little understanding, there are still a lot of obvious two points, or analysis)

Step(Task1, Task2,task3) For each task, there are three ways to control workflow

1. Call this and let Step execute the next function

2. The callback generated by calling this.parallel or this.group n times tells Step that the next function in the list should be run n times

3. Return a value that calls the next function in the Step run list (simplifies synchronous and asynchronous mixing)

/ / Step case https://github.com/creationix/step official Step (function readSelf () {fs. ReadFile (__filename, this); }, function capitalize(err, text) { if (err) throw err; return text.toUpperCase(); }, function showIt(err, newText) { if (err) throw err; console.log(newText); }); // Execute the second function directly when executing thisCopy the code

Practical instance

Var Arr=[1,2,3,4,5]; async.map(Arr,function(item,callback){ var _setValue = parseInt(item)+1; callback(null,_setValue); },function(err,results){ console.log("map:"+results); }); Var Step = require(' Step ') function stepMap(arr,iterator,callback){Step(function(){var group = This.group () // Run this.group() to reserve a slot for the callback function; // This.group will merge the result into an array for(var I =0; i<arr.length; I ++){iterator(arr[I],group()) // group() generate a single callback and increment the internal counter}},callback)} stepMap(arr,function(item,callback){var _setValue = parseInt(item)+1; callback(null,_setValue); },function(err,results){ console.log("map:"+results); })Copy the code

Analysis:

Async. Map takes an array and executes iterators asynchronously.

process

The puzzle is what this.group() does and what group() does.

This.group () saves a parameter slot for the next call

Group () generates a single callback and increments the internal counter

I still feel like I can’t convince myself

From github.com/creationix/…

Group () is really where result[index] = arguments[1] is called

Arguments [1] from iterator(arr[I],group()). Group (null,setValue) is actually performed here.

Therefore, the real reason for my confusion does not lie in the group. I’m not familiar with async. Map.

I’m trying to be quick, but a lot of questions are confused. 😔

PS: This book, almost finished. The biggest takeaway today should be an understanding of this task queue. Until now, the essence of this book is to add those boring concepts to one scene after another, because the ES specification has been turned upside down, we usually read the essence summed up by others but feel nothing. But this book was part of the ES6 specification at the time. A lot of things are really written well, will be accepted by everyone. If you want to understand JavaScript in depth, this may seem like a detour, but you can actually learn more about it. I’m already dizzy, so I’ll be the one to comfort myself.

Today is also the first time to read a small frame source, read to this point in time. There seems to be some truth to shorter hairlines being stronger. There’s still a long way to go. Take your time.

I was going to sort out the next test and asynchrony this week. But I lost 21 games in a row. Remind yourself to play in moderation, overkill.