directory

  • The basic use
    • A readable stream
    • Streams can be written
    • Both readable and writable
  • Attached to the source

A stream is an ordered set of byte data transfers with a starting point and an ending point, and with good efficiency. With event and non-blocking I/O libraries, the flow module allows it to be processed dynamically when it is available and released when it is not needed. A stream is an abstract interface that handles streaming data in Node.js. The STREAM module provides the following basic apis for building implemented stream interface objects. Streams can be readable, writable, or read-write.

The basic use

A readable stream

In NodeJS, we rely on the core module FS, which integrates the createReadStream readable stream.

The fs.createreadstream (path, options) parameter is as follows:

  • Path: indicates the path of the read file

  • Options: string (the specified character encoding) | Object (detailed below)

    • flags: Identifier bit, default ‘r’
    • encoding: Character encoding. The default value is null. The value read is Buffer
    • fd: file descriptor, null by default;
    • mode: Permission bit, default is 0O666.
    • autoClose: Indicates whether to automatically close after reading data. The default value is true
    • start: Start reading position, default 0
    • end: End position, default end of article
    • highWaterMark:// The highest water mark, how many each time read. Default value: 64 x 1024 bytes

  • The return value is fs.readstream

Create a readable stream

// Introduce dependencieslet fs = require('fs')
let rs = fs.createReadStream('./readStream.txt', {// start: 0, // start: 0, // start: 0, // end: 3, // start: 0, // end: 3, // highWaterMark: 3, // maximum read, default: 64*1024 bytes per read})Copy the code

Event mechanisms in readable flows

In the following example, the./ readstream. TXT file is 1234567890

1. open

The open event is used to listen for the file to open, and the callback function is executed after the file is opened.

// Introduce dependencieslet fs = require('fs')
let rs = fs.createReadStream('./readStream.txt'{start: 0, end: 3, highWaterMark: 3}) {start: 0, end: 3, highWaterMark: 3}'open', () => {
  console.log('File open'}) // Result: the file is openCopy the code
2. data

The data event is raised each time a byte of highWaterMark is read until the read is complete, and the result of each read is returned in the callback function. If encoding is not set, Buffer is returned

// Introduce dependencieslet fs = require('fs')
let rs = fs.createReadStream('./readStream.txt', {
  // encoding: 'utf8'Start: 0, end: 3, highWaterMark: 3})'open', () => {
  console.log('File open')
})
rs.on('data', (data) => {console.log(data)}) // If encoding is not set, the result returned is /** the file is enabled *<Buffer 31 32 33> *<Buffer 34> */ / If encoding:'utf8'The result is /** the file is open * 123 * 4 */Copy the code
3. end

The end event is emitted when the file is finished reading and the callback is executed.

Read the situation as follows:

  1. endSet the value (end: 3) is triggered after reading the length of the setting
  2. endNo set value, default read all content trigger.

To make it easier to see the results, we implement both cases in the following example:

// Introduce dependencieslet fs = require('fs')
let rs = fs.createReadStream('./readStream.txt', {start: 0, // end: 3, highWaterMark: 3})'open', () => {
  console.log('File open')
})
rs.on('data', (data) => {
  console.log(data)
})
rs.on('end', () => {
  console.log('It's over'}) // set end: 3 run result /** file enabled * <Buffer 31 32 33> * <Buffer 34> * End */ // run result without end parameter /** file enabled * <Buffer 31 32 33> * <Buffer 34 35 36> * <Buffer 37 38 39> * <Buffer 30> * End */Copy the code
4. error

An error event listens for error messages, triggers a callback when a file is read, and returns the error message

// Error event automatically calls rs.on('error', (err) => {
  console.log(err)
})
Copy the code
5. close

The close event is used to listen for file closure, and the callback is executed after the file is closed. When creating a readable stream, autoClose: true (the default is true) automatically closes the file and triggers a close event to perform a callback.

rs.on('close', () => {
  console.log('close')})Copy the code

Pause and Resume

A readable stream has two states: a running state or a paused state.

A readable stream pauses when it starts and can be switched to flow by:

  • Listening to thedataThe event
  • callfs.resume()methods

A readable stream can be switched back to the paused state by:

  • callrs.pause()methods

When do we use pause and resume first? We all know that reading a file takes up content space, so if we have a really big file read, if we read it all, it takes up a lot of content space, and that’s not what we want to see, so we think about reading a piece of data, processing it, and then reading it again.

A simple example

// Introduce dependencieslet fs = require('fs')
let rs = fs.createReadStream('./readStream.txt', {
  encoding: 'utf8', // character encoding, default null highWaterMark: 2})let i = 0

rs.on('data', (data) => {I ++ console.log(' first${i}`, new Date ()); Console. log(data) rs.pause() // PausesetThe Timeout (() = > {rs. Resume () / / restore}, 1000)}) rs. On ('end', () => {
  console.log('It's over'}) // The first time 2018-09-16T10:11:00.993z // 12 // The second time 2018-09-16T10:11:01.996z // 34 // the third time 2018-09-16T10:11:02.997z // 56 // The 4th time 2018-09-16T10:11:03.997z // 78 // the 5th time 2018-09-16T10:11:04.998z // 90 // the endCopy the code

Streams can be written

In NodeJS, we rely on the core module FS, which integrates the createWriteStream readable stream

The fs.createWriteStream(path, options) parameter is as follows:

  • Path: indicates the path of the read file

  • Options: string (the specified character encoding) | Object

    • flags: Indicates the identifier bit. The default value is'w'
    • encoding: Character encoding. Default valueutf8
    • fd: file descriptor, null by default;
    • mode: Permission bit, default is 0O666.
    • autoClose: Indicates whether to automatically disable the function. The default value istrue
    • start: Start reading position, default0
    • highWaterMark: Number of writes

  • The return value is fs.writestream

Create writable streams

Writestream. TXT file, indicating the file to be written

// Introduce dependencieslet fs = require('fs')
let ws = fs.createWriteStream('./writeStream.txt', {start: 0, // start reading position, default 0})Copy the code

Write content and end

Write Writes content to the corresponding file. End indicates the end of writing. If the following parameter has content, the file can also be entered.

// Introduce dependencieslet fs = require('fs')
let ws = fs.createWriteStream('./writeStream.txt', {start: 0, // Start writing position, default 0}) ws.write('1'// Write to ws.end()'Done') // End // result: 1 finishedCopy the code

Event mechanisms in writable flows

The events open and close in writable flows are less detailed than the simple ones. Focus on the Finish and drain events.

1. The finish

The ‘Finish’ event is emitted when the ws-end () method is called and the buffered data has been passed to the underlying system.

The test code is as follows:

// Introduce dependencieslet fs = require('fs')
let ws = fs.createWriteStream('./writeStream.txt', {start: 0, // start writing position, default 0}) ws.on('open', () => {
  console.log('open');
});
ws.write('1')
ws.end('Done')
ws.on('finish', () => {
  console.log('All writes are complete. ');
});
ws.on('close', () => {
  console.error('close'); }); // Open // All writes are complete. // closeCopy the code
2. Drain the event tapping

If the ws-write (chunk) method is called, it returns false, which is the size of the written content to reach the cache, and the drain event is emitted

let fs = require('fs')
let ws = fs.createdWriteStream('writeStream.txt', {start: 0, highWaterMark: 5}) // Write 15 numbers, five at a time, only want to occupy five bytes of memorylet i = 0
function write() {
    let flag = true
    while(i < 15 && flag) {
        flag = ws.write(i++' '.'utf8'}} write() // If the write reaches the size of the cache, an event is raised when the write is complete.'drain', () => {
    console.log('fill') write() // clear the cache and continue writing}) // full // fullCopy the code

Both readable and writable

3. To read from one file to another through pipe. Often used.

// const fs = require("fs"); // Create readable and writable streamslet rs = fs.createReadStream("./ReadStream/readStream.txt", {
    highWaterMark: 3
});
let ws = fs.createWriteStream("./writeStream.txt", { highWaterMark: 2 }); / / will bereadThe contents of stream.txt are written to rs.pipe(ws) in writestream.txt;Copy the code

Attached to the source

Important:In order to facilitate you to understand, view, debugging code, the complete source code seegitHub

conclusion

This article is a basic introduction to streams in NodeJs. Hope to play a certain role in understanding the flow.

Part 2: Implementation Principle and Simple Implementation of NodeJS Stream — “The Origin of Flow, the Principle of Source”

Author: Xiang Xiang

In the future, you will be grateful for your hard work now!

Node Basics series

  • Based on CommonJS specification, simple implementation of NodeJs modularization
  • The basic use of Stream