This is the seventh day of my participation in the August More text Challenge. For details, see:August is more challenging

Sometimes we need to do a post mortem on the log to extract specific data for processing. For example: 1. The middleware module sends JSON data to the background, and each time the sent JSON will be converted into a string and saved to the log file. However, the sending fails due to network reasons, so it needs to be supplemented afterwards. 2. The log file contains information about different devices. You need to extract the information about the specified device number and analyze it. And so on.

Note that sometimes it is possible to process the data too fast and cause exceptions, such as sending JSON to the background frequently, but the background processing is not enough, and the data will be lost, so this article adds delayed processing.

This article simplifies the application scenario by assuming that the JSON data string in the log file is stored as a single line of data, and other data is mixed with it, and each log line is preceded by a timestamp. Use the program to read out JSON data for processing.

1, implementation,

  • To verify the delay, add the timestamp printing function getTimestamp.
  • Use the readline library to read each line of data in the file. After reading, judge. Json data begins with “{“. If you want to extract other data, according to the feature field.
  • Use json.parse to convert strings into JSON structures.
  • Use setTimeout to set the timing and call it again in an anonymous function.
  • Mybuffer push saves data, shift retrieves data. To be processed in a timed function.

3, the source code

The full source code is as follows, assuming the source file name is test.js:

/** Read a row of data from the file, convert, and reprocess. */ var fs = require('fs') */ var fs = require('fs') */ var fs = require('fs'); var readline = require('readline'); Function getTimestamp(){var now=new Date(); var y=now.getFullYear(); var m=now.getMonth()+1; var d=now.getDate(); var hour=now.getHours(); var min=now.getMinutes(); var sec=now.getSeconds(); var msec=now.getMilliseconds(); return"["+y+"-"+(m<10?"0"+m:m)+"-"+(d<10?"0"+d:d)+" "+(hour<10?"0"+hour:hour)+":"+(min<10?"0"+min:min)+":"+(sec<10?"0"+sec:sec)+"."+(msec<100?(msec<10?"00"+msec:'0'+msec):m sec)+"] "} function print(fmt, ... extras) { console.log(getTimestamp(), fmt, ... extras); } function handle_it(data) {// print(data); } var g_cnt = 0; // The buffer is a set of data, which can be understood as a queue. The element can be any value. Function start_timer(buffer, timeout) {function timer_task() {var item = buffer.shift(); if(item ! == undefined) { g_cnt++; console.log(`# ${g_cnt} will deal data`); handle_it(item); // Continue setTimeout(timer_task, timeout); } } setTimeout(timer_task, timeout); } function main(filename) { var fRead = fs.createReadStream(filename); var objReadline = readline.createInterface({ input:fRead }); var mybuffer = []; start_timer(mybuffer, 3000); Objreadline. on('line',function(line) {//console.log(line); if (line.indexOf('{') >= 0) { var org_msg = line.substr(line.indexOf("{")); var json = JSON.parse(org_msg); //console.log(json); mybuffer.push(json); }}); objReadline.on('close',function() { }); } main("foo.txt");Copy the code

Foo.txt file contents:

[the 2021-08-03 13:06:14. 497] got data {" op ":" got ", "devid" : "12345678", "time" : "20210803110614", "ID" : ""," status ": "100", "result": "0"} fdaf3adg [the 2021-08-03 13:06:15. 497] got data {" op ":" set ", "devid" : "222222222", "time" : "20210803110614", "ID" : "", "status": "101", "result": "0"} dffdaf [the 2021-08-03 13:06:16. 497] got data {" op ":" got ", "devid" : "3333333", "time" : "20210803110614", "ID" : "", "status": "102", "result": "1"} 3654 esxgf [the 2021-08-03 13:06:17. 497] got data {" op ":" set ", "devid" : "44444444", "time" : "20210803110614", "ID" : "", "status": "103", "result": "0"}Copy the code

4, test,

Simply run node test.js.

# 1 will deal data [2021-08-03 13:44:17.630] {op: 'got', devid: '12345678', time: '20210803110614', status: ' } # 2 will deal data [2021-08-03 13:44:20.650] {op: 'set', devid: '222222222', time: '20210803110614', ID: ' ', status: '101', result: '0' }Copy the code

5,

Nodejs operates asynchronously, and the objReadLine. on function in the previous source code is the response function to the line and close events. Asynchronous mechanisms come into play in many ways, especially in the response of web pages. However, there are some situations where synchronization, or timing, is needed. For example, when the fs.writeFile function is used to save data to a file, the data saved first is later in the file. This is inconvenient for filtering logs, and in the case of not finding a good way to do it and not wanting to use C, there are a few tricks to solve this problem. Use nodejs’s console.log function to print the output data. 2, under Windows using CMD terminal execution node test. The js | tee. The output TXT, so can the console. The log output information to the output. The TXT file. Note that you cannot use a terminal such as Git bash, as stdout is not a tty. 3. Untested on Linux.