I. Introduction to development

Project source code: Github

1. Development background

As the company’s V2 project needs to be upgraded by components, it is difficult to fully understand the scope of influence of the components to be adjusted due to the heavy historical burden of the V2 project, a large number of codes and files, and many nested components, so it is necessary to develop such a tool to achieve the following functions:

  • Need to be able to support custom keyword search, easy to search by different existing component names;
  • Need to support to retrieve the impact of the component file range, as well as page name routing, so as to facilitate the test according to the page fast test;
  • Need to be able to support data visualization, easy to judge the weight of all influence scope;
  • Routing files and necessary data that can be exported within the influence range are required.

Based on the above requirements, I have a general idea to use Nodejs and Python for requirements development for several reasons:

  • Requirements to operate files, including reading and writing;
  • More data processing operations are required, including filtering and assembling data formats;
  • Requirements Requirements for data visualization;

At first I was going to just use Nodejs to complete this requirement, but when I was halfway through development, I couldn’t find a satisfactory visualization plug-in for data visualization, so I decided to use Matplotlib, a 2D drawing library for Python, because it was very convenient to use.

This is also my first work with Nodejs. There is still a lot of room for improvement.

2. Tool documents

  • Nodejs- Official documentation
  • Python- Official documentation
  • Python- Chinese development Manual
  • Matplotlib- Development tutorial

Ii. Development environment construction

1.Nodejs environment construction

Nodejs environment setup, I believe for our front-end development, should be very simple, but here considering the possibility of native students are not clear, here I briefly introduce:

  • Download and InstallNodejs

Go to Nodejs official website, select the corresponding system environment to download, and then directly open the installation.

  • testNodejsThe environment

Open the command line tool and run the node -v command to check whether the corresponding Nodejs version number is displayed.

v10.8.0
Copy the code

Add global directory to nodejs installation directory:

Node: The "node" item cannot be recognized as the name of a cmdlet, function, script file, or runnable program. Check the spelling of the name, and if paths are included, make sure the path is correct, then try again.Copy the code
  • The installation is complete

2. Build the Python environment

  • Download and InstallPython

From the Python website, select the 3.x version to download (python2. x is no longer in maintenance and will soon be phased out) and install directly.

  • testPythonThe environment

After the installation is complete, open the command line tool, execute Python, and see if the output is the version number and command line interaction mode.

PS C: \ Users \ mi > python 3.6.3 | Anaconda, Inc. | (default, Oct 15, 2017, 03:27:45) [MSC v.1900 64 bit (AMD64)] on win32 Type"help"."copyright"."credits" or "license" for more information.
>>
Copy the code
  • Install the drawing libraryMatplotlib

NPM install packageName = NPM install packageName = NPM install packageName = NPM install packageName

pip install Matplotlib
Copy the code
  • The installation is complete

Iii. Development process

First of all, introduce the development idea:

1. The end result

Js and search_current_file_python.py, and get the corresponding data files by executing two commands:

  • Obtain the files and tables of data statistics such as the path of all files containing keywords, the number of files in the folder and the route/parameter/title of the corresponding page of all files.
node search_current_file.js
Copy the code

  • Gets pie chart results of the number of files in all folders as a percentage of total files.
python search_current_file_python.py
Copy the code

You need to enter the data of the specified folder to be generated. If you do not enter this parameter by default, the data of all folders will be generated.

2.Nodejs development

First define a few variables that are mainly used below, other variables and functions that are not written here, you can view the source code.

var Excel = require('exceljs');
var XLSX  = require('xlsx'); 

var filterFile = ['.html']; // The type of file to retrieve
var filterDir  = ['lib'];   // The folder to be excluded
var classArray = [          // An array of class names to retrieve
    'search-holder'.'exe-bar-search'.'Enter search content'.'<exe-search'.'learn-search'.'ion-android-search'
];  
var resultArray    = [];    // Final result
var resultAlassify = {};    // Final result classification
var excelFileArr   = [];    // Excel file content array
Copy the code

2.1 Obtaining Search Results

Purpose: Search all HTML files that contain keywords and save this data.

  • Core methodgetCurrenAllFile()

We use the fs.readdir method to get the names of all files and folders in the path as a collection; The collection is then iterated over, with stat.isDirectory() true indicating that the result is a folder, and false continuing with getCurrenAllFile() to read the file information at the next level.

/** * Get all HTML files for the current project * @param {string} Paths file */
var getCurrenAllFile = function (paths){
    / /... Omit the part
    var fileArr = [];// Initializes the final result classification object
    fs.readdir(paths, function(err, files){
        _.forEach(files, function(item, index){
            var c_path = path.join(paths, item);
            var stat = fs.lstatSync(c_path);
            / / TODO the key
            if(stat && stat.isDirectory()){
                / /.. Omit the operation of filtering foldersgetCurrenAllFile(c_path); }}else{
                / /.. Omit the operation of filtering foldersgetCurrentFile(c_path, item); }}); });return fileArr;
}
Copy the code
  • Core methodgetCurrentFile()

Read the contents of each file and then use the searchCurrentFile() method to retrieve the keyword we want to search for.

@param {string} paths file path @param {string} filename filename */
var getCurrentFile = function(paths, filename){
    fs.readFile(paths, 'utf8'.function(err, data){
        / /... Omit the part
        if (err) console.log(err);
        searchCurrentFile(data, paths);
    });
};
Copy the code
  • Core methodsearchCurrentFile()

So this is going to go through the classArray array that we’ve defined, which contains all the keywords that we need to retrieve, and if the retrieval result is true then it’s going to be stored in the resultArray array and the resultAlassify array.

@param {object} Data file * @param {string} Paths file */
var searchCurrentFile = function(data, paths){
    _.forEach(classArray, function(val){
        / /... Omit the part
        if(data.indexOf(val) >= 0){
            resultArray.push(paths);
            resultAlassify[val].push(paths); // Save the final result (the object under the current keyword)}}};Copy the code

2.2 Processing search Results

Purpose: To remove, format and save the obtained data into JSON as a visual data source. There are two simple methods: unique() for data deduplication, and setEachDirFileNum() for file count.

Here we use saveDataToJson() to organize the data into JSON format, and use the setJSONFile() method to save the JSON data as a JSON file for visual manipulation.

  • Core methodsaveDataToJson()

This step mainly uses loadsh’s grouping function _. Ground to process JSON data. The format we need is:

result = {
    template: [
        home:[ {}, {} ],
        my: [{}, {}]// ...].view: [
        // ...]}Copy the code

You then need to process it into the format you need to save Excel and save the JSON file using the setJSONFile() method.

@param {*} data the data that needs to be processed */
var saveDataToJson = function (data){
    var result = {};
    // Layer 1 group the outer folder
    result = _.groupBy(data, function(item){
        item = item.replace(filePath+'\ \'.' ');
        var list = item.split('\ \');
        return list[0];
    });
    // Layer 2 group inner folder
    for(var k in result){
        result[k] = _.groupBy(result[k], function(i){
            i = i.replace(filePath+'\ \'.' ');
            var r = i.split('\ \');
            return r[1];
        });
    }
    for(var i in result){
        for(var m in result[i]){
            for(var n in result[i][m]){
                var currentPath = result[i][m][n].replace(filePath+'\ \'.' ');
                currentPath = currentPath.replace(/\\/g.'/');
                var current = excelFileObj[currentPath];
                result[i][m][n] = {
                    title : current ? current['Route Name'] : 'This file is a module'.path  : current ? current['File path'] : currentPath,
                    url   : current ? current['url'] : 'This file is a module'.params: current ? current['Route Parameters'] : 'This file is a module'.ctrl  : current ? current['Controller name'] : 'This file is a module'.urls  : current ? current['url'] : 'This file is a module'}; } } } setJSONFile(result);// Save the JSON file
};
Copy the code

2.3 Adding Data such as file title and route

Objective: To parse the external routing Excel table and merge it into the original data

  • Core methodgetExcelFile()

    Read Excel data and passresolveTo return.
/** * Read Excel data */
var getExcelFile = function(){
    return new Promise(function(resolve, reject){
        var excelPath = path.join(__dirname, excelReadName);
        fs.exists(excelPath, function(exists){
            if(exists){
                var workbook = XLSX.readFile(excelPath, {type: 'base64'});// Get all table names in Excel
                var sheetNames = workbook.SheetNames;
                resolve({workbook: workbook, sheetNames: sheetNames});
            }else{
                reject({message:'Error: Please get the routing list file first! (execute node get_router.js) '}); }}); })};Copy the code
  • Core methodgetEachSheet()

Here we need to save the data of each table in Excel to excelFileObj, and note that our project loDash cannot use API versions higher than 4.0.0.

@param {object} workbook Excel workbook * @param {object} sheetNames Excel workbook data */
var getEachSheet = function(workbook, sheetNames){
    _.forEach(sheetNames,function(item,index){
        var sheet = workbook.Sheets[sheetNames[index]];
        var json = XLSX.utils.sheet_to_json(sheet);  // Return serialized JSON data for a single table
        excelFileArr = excelFileArr.concat(json);    // cannot use _. Concat of Lodash because the lodash version is too low
    })
    _.forEach(excelFileArr, function(val, key){
        excelFileObj[val['File path']] = val;
    });
}
Copy the code

2.4 Generating a Result File

Purpose: Generate the result in Excel/JSON/TXT file: This section does not describe how to generate JSON/TXT files, but uses fs.write, the built-in file storage method of Nodejs

  • Core methodsetExcelFile()

Mainly organize data into Excel data format.

/** * Save Excel data * @param {object} data To process data * return excelfilename.xlsx */
var setExcelFile = function(data){
    var workbook = new Excel.Workbook();
    workbook.creator = 'EXE';
    workbook.lastModifiedBy = 'Leo';
    workbook.created     = new Date(a); workbook.modified =new Date(a); workbook.lastPrinted =new Date(a);for(var item in data){    // Layer 1 loops the templates views folder
        for(var list in data[item]){
            var worksheet = workbook.addWorksheet(list.toUpperCase()),
                rowData   = data[item][list];
            worksheet.columns = [
                { header: 'Page title'  , key: 'title' , width: 40 },
                { header: 'File path'  , key: 'path'  , width: 60 },
                { header: 'Routing address'  , key: 'url'   , width: 40 },
                { header: 'Route Parameters'  , key: 'params'.width: 40 },
                { header: 'Controller name'.key: 'ctrl'  , width: 40 },
                { header: 'url'      , key: 'urls'  , width: 40},];for(var row in rowData){
                worksheet.addRow({
                    title : rowData[row].title,
                    path  : rowData[row].path,
                    url   : rowData[row].url,
                    params: rowData[row].params,
                    ctrl  : rowData[row].ctrl,
                    urls  : rowData[row].urls,
                }) 
            }
        }
    }
    workbook.xlsx.writeFile(path.join(__dirname, excelFileName)).then(function() {
        / /... Omit the part
    });
};
Copy the code

At this point we are done developing the Nodejs program, and we will end up with a file search_current_file_json.json as the data source for the Python part.

3.Python development

The Python part is relatively simple, just loading data, simple data processing and visualization.

Also at the beginning, write down some important definitions:

#... Omit some
import matplotlib.pyplot as plt
keyName    = []  # Category chart to display (by outer folder)
selectName = ' '  The name of the folder selected by the user
Copy the code

2.1 Reading data Sources

We read the file using Python’s built-in open method and import the built-in JSON method to read the search_current_file_json.json file generated in the previous Nodejs section.

file = open('./search_current_file_json.json'.'r', encoding='utf-8')
file = json.load(file)
Copy the code

2.2 Setting cli Input Items

You can enter the name of the folder to display the pie chart of the folder. By default, pie charts of all folders are displayed.

Before setting this, we need to get the names of all the layer 1 folders using the getKeyName() method:

def getKeyName(a): 
    for name in file: 
        keyName.append(name)
Copy the code

Then you can set the command line input:

getKeyName()
select     = ', '.join(keyName)
selectName = input('The retrieved folder is: [' + select + ', please enter the name of the folder you want to view (all by default) : ')
Copy the code

2.3 Draw a single pie chart

Then draw a single pie chart, which is mainly to set the parameters of the pie chart:

  • Core methoddrawOneChart()
def drawOneChart(name, label, data):
    plt_title = name
    plt.figure(figsize=(6.9)) # Resize the graph
    labels = label   # define tag
    sizes  = data    # each value
    colors = [       # Block color definitions are omitted here
        #...
    ]
    explode = []    The larger the value is, the larger the gap is
    max_data = max(sizes)
    for i in sizes: # Initialize the spacing between blocks and split the maximum
        if i == max_data:
            explode.append(0.2)
        else:
            explode.append(0)

    patches,text1,text2 = plt.pie(
        sizes, explode = explode, labels = labels, colors = colors,
        autopct = lambda pct: pctName(pct, data),  # Keep fixed decimal values
        frame   = 1.# Whether to display the pie chart box, set display here
        shadow  = True.# No shadow setting
        labeldistance = 1.1.Times the radius from the center of the circle
        counterclock  = False.# Whether to render the pie chart counterclockwise;
        startangle    = 90.# Counterclockwise starting Angle setting
        pctdistance   = 0.6      The value is a multiple of the radius from the center
    )       
    plt.xticks(())
    plt.yticks(())
    plt.axis('equal')
    plt.legend()
    plt.title(plt_title+'File Distribution under folder (clockwise)', bbox={'facecolor':'0.8'.'pad':5})
    plt.savefig(plt_title+'_'+saveImgName) # must be placed before plt.show()
    plt.show()
Copy the code

2.4 Draw multiple pie charts

Finally, all pie charts are generated by a loop call to drawOneChart() :

  • Core methoddrawAllChart()

This method reprocesses the previous JSON data, using the number of files in each folder as pie chart data, which is the value of values here.

def drawAllChart(openName):
    for name in keyName:
        labels = []
        values = []
        for view_name in file[name]:
            labels.append(view_name)
            values.append(len(file[name][view_name]))
        if openName == ' ' or openName == name:  
            drawOneChart(name, labels, values)
        else:
            print('Input error')

Copy the code

Four,

1. Nodejs knowledge

This section is used more frequently in Nodejs:

  • File read/write operations
  • Regular matching operation
  • Data format processing operations

Therefore, in order to develop similar or other types of tools in the future, it is still necessary to strengthen the knowledge of these three aspects. The code of this part may not be concise or beautiful, but after all, as my experience accumulation, I will have a clearer idea of the development of such tools.

2. The Python

What I used most in this part is actually some basic syntax in Python. This part of code is actually to deepen my use and understanding of basic Syntax in Python and practice operation.

3. Develop

Next, I will find time to optimize the project code, and then transform the project. I will develop a separate set using Nodejs and Python, and compare the difference between the two (execution efficiency/code volume). Nodejs also includes node-echarts and d3-node.