The foregoing

Opening a new pit today, talking about the front end: Istanbul.

When it comes to front end coverage, as early as 18 years of contact, but for a variety of reasons, and did not go to the specific landing. The only thing I know is what it is and what it is for!

So what do I do with this? How to land? I didn’t really look into it. Think of a little regret.

But this time! This article will describe how to use Istanbul to collect front-end coverage, parse source code, modify source code to fit business understanding, and report coverage with one click.

Writing in the front

Before I write, here are four articles I’ve looked at that, in my opinion, have written well about front-end coverage:

Building front-end JS coverage platform based on Istanbul gracefully

Front End Code Coverage

React Native code coverage acquisition exploration

Real-time coverage statistics tool

And related GitHub open source:

babel-plugin-istanbul

istanbul-middleware

nyc

code-coverage

Ok, move the small bench, let’s start ~

The body of the

Front-end Web coverage statistics

First of all, the front-end coverage, in the current business scenario, includes web and mobile, so in many cases, if the mobile is not written in native, mostly in the form of embedded H5 pages. If you are involved in business, most of it is using webView.

Just go straight to the web. Of course, if the situation is different, there is also a design link for RN in the link above, which can also be referenced

Of course, I believe that the front end is mostly vue or React, if you still use jQuery to do, it must be too backward. If your front-end projects are built using Vue or React, then the following article will help you a little bit.

On the other hand, I personally recommend not taking the time to watch it, because it might not land well. But if you’re interested enough to see it through, you’ll see something valuable. Strictly speaking, learn from each other.

Plug-in interpretation and selection

According to some predecessors, I cannot skip the following section. If you’re hearing about front-end coverage for the first time, you’ll be wondering what tools are available and which ones fit better. I can’t tell you what to use right away, okay? This will make you swallow the date, the cloud in the fog.

If so, I have not done the real purpose of writing this article, is to let the people can read with thinking and understand.

The figure above is a summary, comparison, and detail of the tools that can be used for coverage.

After reading this, I believe you can know, they are involved in the business, need to use what kind of tools to achieve coverage collection work.

Of course, just looking at this, you still don’t know how to use Istanbul, even if you choose it. What’s in it? How do I drive the stake? How do you pile in the case of different processing between server rendering and client rendering? And so on.

Obviously, it’s ready for you:

Just to explain, NYC is an advanced version of Istanbul. It solves many problems, and there are many good people working on it. Of course, this article doesn’t cover that.

After reading these two pictures, I believe you have some idea about the front-end coverage, why to choose Istanbul, and how to selectively pile front-end projects.

I’m going to have to make a copy of this teacher’s stuff for you to understand. Copy the relevant original text in the article listed above. Forgive me for being so shameless… They melody)

Drive pile before operation

nyc instrument

For the compiled JS file, manually insert the pile to form a new JS file after the pile

babel-plugin-istanbul

The Babel plugin, provided by Istanbul, can be used for front-end engineering using Babel, both react and Vue

Pile insertion during operation

**im.hookLoader **

For example, when a node application is started, hooks will be added to the require entry, so that the application will be loaded with the code after the plug

im.createClientHandler

If the react and Vue JS files specify a root path, they will intercept all JS file requests from that path, and return the embedded code. That means browsers requesting static resources will act like Babel-plugin-istanbul. The difference is that this method returns the stub code only when the browser requests JS, which is a dynamic process

babel-plugin-istanbul

At the top, I mentioned that if the project is using Vue or React, you can directly use the plugin Babel-plugin-Istanbul. I will briefly explain what this thing does. It is actually a finished NPM package, which is used to plug your project. When did it drive the stakes?

⭐ is when you run a project

When you run your front-end project after NPM install babel-plugin-istanbul, the compile time will be slightly longer than the original run time. That little bit of time, in fact, it is working, and then give your project processing, the design of your project files, piling processing.

The question is, is there any impact on the project? Does this affect front-end projects? Emmm, no.

It will only generate the corresponding coverage files in your project (there is a mapping in the process of calling, which will be discussed later).

Take a look at the specific NPM dependencies:

It is important to note that you try to install this dependency in the Dev environment.

Configuration file

If install is complete, there is one more thing you need to do:

babel.config.js

module.exports = {
  presets: [
    '@vue/app'
  ],
  plugins: [
    ['babel-plugin-istanbul', {
      extension: ['.js', '.vue']
    }],
    '@babel/plugin-transform-modules-commonjs'
  ],

  env: {
    test: {
      plugins: [
        ["istanbul", {
          useInlineSourceMaps: false
        }]
      ]
    }
  }
}
Copy the code

.babelrc

   {
  "presets":["@babel/preset-env"],
  "plugins": [
    "@babel/plugin-transform-modules-commonjs",
    ["babel-plugin-istanbul", {
      "extension": [".js", ".vue"]
    }]
  ],
  "env": {
    "test": {
      "plugins": [
        ["istanbul", {
          "exclude": [
            "**/*.spec.js"
          ],
          "useInlineSourceMaps": false
        }]
      ]
    }
  }
}
Copy the code

These are different configurations for two different files, so I’ll post them both for your convenience. What do these configurations do? Why use?

In the source code source-maps this section, you need to do their own processing, but according to the above method to configure, is completely OK.

Then, if you’ve noticed something here, it’s ‘@babel/plugin-transform-modules-commonjs’. Why use this? Everything else is normal configuration.

In the comment section reply-169807 on the top of my post, I will simply say that the use of Babel-plugin-istanbul is incompatible with the babel-plugin-import plugin and therefore failed to be built.

npm run

If you’ve done all of the above, try running your service.

Ok, assuming you’ve run it, open your project and open the console. Execution window. Coverage.

If you see the following, then congratulations, you have succeeded in the first step

And at this point, you’ll be glad to see what Babel-plugin-Istanbul does, which pins all the files involved in the project back to you with coverage information.

If you want to be specific, just click on any detail and it will tell you exactly what lines, statements, methods, etc are covered.

Introduction of the underlying

Having said so much, we have forgotten the basics of coverage. Careless, did not flash.

Coverage dimension

• Statements: statement coverage, rate of execution of all Statements;
• Branches: Branch coverage, execution rate of all code Branches such as if and triplet operations;
• Functions: Function coverage, the rate at which all Functions are called;
• Lines: Line coverage, the rate at which all valid Lines of code are executed, similar to statements, but calculated slightly differently

Instrumentation,

Instrumentation principle

The idea of a “babel-plugin-istanbul” would create a “poser” file. In fact, these files stored in your project, will not affect your project, the most is to take up the project capacity.

This diagram also makes it clear that the principle of reading coverage data is to get a one-to-one mapping based on the page you are currently visiting to find the file behind the pile. When I saw this principle, I capitalized a word, fu!

File comparison before and after instrumentation

Before the instrumentation:

After the instrumentation:

It should be obvious to see some counters. So these are the numbers for the statistics.

Istanbul middleware

Window.coverage is a very important aspect of your coverage, but you have to collect it.

The information collected by Window. Coverage is processed using another great open source tool, the one at the top: Istanbul Middleware, or IM for short. What is this mainly for?

Specific details on the point of this link to see, do not repeat. Basically, this middleware provides the following related functions.

These are the four interfaces given in the IM project, and when you see this, you should see the light. The middleware IM provides four interfaces for processing data collected by window.coverage.

So let’s look one by one, what are the four interfaces?

1. Request full amount /;
2. Reset reset;
3. Download the download;
4. Submitted to the client.

There’s actually a show interface for that.

Ok, so with that in mind, it’s relatively perfect, so we’re going to use this client to submit the data that we’ve collected. As for how to use these four interfaces, it is actually very simple, it is to start a local node service, these interfaces are written under the service, directly according to the IP call can be.

In this open source project, one of the demos is an app in the test directory. This is the author to a demo demo. You’re smart enough to see it.

Source code parsing -test/app/demo

So just to make it easier for you to get started, I’m going to give you the whole demo. Of course, I’ve already made changes to the source code here, but it doesn’t have much impact.

After entering the app path, the outermost index.js file is the entry file for the node service. You can see the script for starting the node service in the readMe. Nodeindex.js –coverage # start the app with coverage. He actually gave us four script statements, but we only use this one.

After this is done, the service is started. This time, enter https://localhost:8988/xxx. You can see the data collected by your coverage.

Ok, let’s say one more thing, which is index.js under server. In fact, in the outermost index.js file, you can see a reference. : require(‘./server’).start(port, coverageRequired); . So, basically, you know that some of the configuration items in this entry are read from the server.

const { json } = require('body-parser'); /*jslint nomen: true */ var // nopt = require('nopt'), // config = nopt({ coverage: Boolean }), istanbulMiddleware = require('istanbul-middleware'), coverageRequired = true, port = 8988; if (coverageRequired) { console.log('server start with coverage ! '); istanbulMiddleware.hookLoader(__dirname, { verbose: true }); } // console.log('Starting server at: http://localhost:' + port); // if (! coverageRequired) { // console.log('Coverage NOT turned on, run with --coverage to turn it on'); // } require('./server').start(port, coverageRequired);Copy the code

When we look at the index.js file in the server, we can see the port number, coverage parameters, and express operations.

So, seeing this, you can change things to your own liking.

For example, I don’t like the fact that every time I start a node service, it’s node index.js –coverage. For example, I don’t like the fact that every time I start a node service, it’s node index.js –coverage. Just set coverageRequired to true.

For example, in the case of new start, the service is started, and if it needs to be tuned by other ends, it involves cross-domain processing, so it needs to do cross-domain processing. For example, the coverage reporting plug-in, which will be discussed later, can only recognize HTTPS, your local services are accessed using HTTP, so you need to make changes here, and so on.

Here I give, I in the server index file to do processing, you can refer to:

/*jslint nomen: true */ var path = require('path'), express = require('express'), url = require('url'), publicDir = path.resolve(__dirname, '.. ', 'public'), coverage = require('istanbul-middleware'), bodyParser = require('body-parser'); function matcher(req) { var parsed = url.parse(req.url); return parsed.pathname && parsed.pathname.match(/\.js$/) && ! parsed.pathname.match(/jquery/); } module.exports = { start: function (port, needCover) { var app = express(); var http = require('http'); var https = require('https'); var fs = require('fs'); All ('*', function (req, res, next) {// console.log('req: ') ', req) res.header("Access-Control-Allow-Credentials", "true"); Res. header(" access-control-allow-origin ", req.headers. Origin); / res/Allow Access domain. The header (" Access - Control - Allow - Headers ", "the content-type, XFILENAME XFILECATEGORY, XFILESIZE"); / / Access head res. The header (" Access - Control - Allow - the Methods ", "PUT, POST, GET, DELETE, the OPTIONS"); // Access method res.header(" content-security-policy ", "upgrade-insecure requests"); Res. The header (" X - Powered By ", '3.2.1); if (req.method == 'OPTIONS') { res.header("Access-Control-Max-Age", 86400); res.sendStatus(204); } else {next(); }}); if (needCover) { console.log('Turn on coverage reporting at' + '/bilibili/webcoverage'); app.use('/bilibili/webcoverage', coverage.createHandler({ verbose: true, resetOnGet: true })); app.use(coverage.createClientHandler(publicDir, { matcher: matcher })); } app.use('/', express.static(__dirname + '')); app.use(bodyParser.urlencoded({ extended: true })); app.use(bodyParser.json()); app.set('view engine', 'hbs'); app.engine('hbs', require('hbs').__express); app.use(express['static'](publicDir)); // app.listen(port); var httpServer = http.createServer(app); var httpsServer = https.createServer({ key: fs.readFileSync('E:/coveragewithweb/app/cert/privatekey.pem', 'utf8'), //app\cert\privatekey.pem app\server\index.js cert: fs.readFileSync('E:/coveragewithweb/app/cert/certificate.crt', 'utf8') }, app); // httpServer.listen(port, function () { // console.log('HTTP Server is running on: http://localhost:%s', port); / /}); httpsServer.listen(port, function () { console.log('HTTPS Server is running on: https://localhost:%s', port); }); }};Copy the code

Source code parsing – Commit client

The request format is “Content-type “, “application/json”, and the request format is” content-type “, “application/json”.

That’s easy. Let’s just go ahead and start the service.

It’s perfect. It’s been submitted. Well, since it’s submitted, let’s see what happens.

Wow. What did you see? It was amazing.

And of course, the urls that YOU see here are the ones that I did, and I’ll talk about them in a second, but don’t worry.

Now that you know that the client is a submission interface, do you want to look at the source code?

Locate the source code. It’s a handler.js file, and you can obviously see the four interfaces that we just saw, and they’re all under an object called a creatHandler. In fact, see this, I can understand the original look at other teachers said, according to their own business to change creatHandler source code, in fact, that is the side.

Here we talk about client that only look at this one, you can go to the specific source code

//merge client coverage posted from browser app.post('/client', function (req, res) { // console.log('req: ', req) var body = req.body; if (! (body && typeof body === 'object')) { //probably needs to be more robust return res.send(400, 'Please post an object with content-type: application/json'); } core.mergeClientCoverage(body); Res. send({success: 'reported successfully'}); });Copy the code

As you can see from the client, the bottom layer is calling the core.mergeclientCoverage () method and passing in the coverage information we passed through the client.

So, let’s go straight to mergeClientCoverage(), which is in core.js on the same level as handlers. Let’s post the source code

Function mergeClientCoverage(obj) {// resolve coverage datas var mergeClientCoverage = getCoverageObject(); Object.keys(obj).foreach (function (filePath) {var original = coverage[filePath], / / filepath = key file path that | | what is already existing data added = obj [filepath], / / added the latest coverage information data result; If (original) {result = utils.mergeFilecoverage (original, added); } else { result = added; } coverage[filePath] = result; // The final coverage information is stored in temporary memory! getCoverageObject() } }); // Coverage is the final coverage information, stored each time, associated with the key value, key pointing to the business version}Copy the code

GetCoverageObject () is called to pull the initial coverage information to give an initial definition of coverage.

Object.keys(obj).foreach (function (filePath) {}

Original defines the data that the coverage service already has.

Added is to process newly added coverage data.

Finally, define a temporary memory store result.

In the later convenience process, there will be a merge process, which is to determine whether the original exists. If the original exists, the new data and the original data need to be merged, that is: result = utils.mergeFileCoverage(original, added); If not, add = add;

Finally, all coverage data are given to the result object. Finally, the final result is directly put into the coverage defined first, and the operation of coverage merge is completed.

In fact, you can take a deeper look at what this utils. MergeFileCoverage (original, added) does. I’m not going to go through it here.

In fact, there are several processes that need to be done in this area. One is, as I mentioned in the original article to the teachers, that is, to design their own coverage data collection according to their own business logic.

Here I have to face again, I’m going to copy.

That is, if you have several businesses, you can’t pass in all versions, branches, and project names at once. You need to do your own processing with your project’s branch, version, business name, etc., and then pass it in for collection.

The other thing is that the node services that you start up here are all temporary memory. We need to collect the coverage data collected and then drop it into the library. Store it, or the service goes down and everything’s gone. This is a terrible thing.

As a final rule, you need to handle file statistics on different machines here.

What does that mean? You might be curious. Actually, this is the pit I stepped in. For example, A project is released to one machine, called A, and the node service that your coverage counts are deployed to another machine, called B.

In this case, if you get the coverage data of A, it cannot be parsed by B, which means that there is no file on B server to match the coverage data sent back, because B server has not published your project. Coverage information on your A service can be reported back through window.Coverage. But I don’t have a local file that matches you.

If you do not write a node service on B, you may fail to find it.

So, you need to synchronize the items on service A to service B here, and you need to change the key in the coverage information collected, and the path of each subobject to the key and path of service B.

Emmm does not know if you can understand, if not, contact me to talk more about it.

Source code parsing – Submit show

Okay, so we’ve spent a lot of time talking about clients. That’s almost done.

Let’s watch show again. Again, look at the source first:

//merge client coverage posted from browser //show page for specific file/ dir for /show? file=/path/to/file app.get('/show', function (req, res) { var origUrl = url.parse(req.originalUrl).pathname u = url.parse(req.url).pathname, pos = origUrl.indexOf(u), // file = req.query.p; If (pos >= 0) {origUrl = origurl. substring(0, pos); } if (! file) { res.setHeader('Content-type', 'text/plain'); return res.end('[p] parameter must be specified'); } // console.log('res: ', res) core.render(file, res, origUrl); });Copy the code

Render (file, res, origUrl); But before we call it, we pass him the argument after show, return origUrl.

Without further comment, go straight to render().

function render(filePath, res, prefix) { var collector = new istanbul.Collector(), treeSummary, pathMap, linkMapper, outputNode, report, coverage, fileCoverage; // coverage = getCoverageObject(); Webvideopakageistanbul -1.0 try {coverage = getCoverageObject() if (! (coverage && Object.keys(coverage).length > 0)) { res.setHeader('Content-type', 'text/plain'); return res.end('No coverage information has been collected'); //TODO: make this a fancy HTML report } prefix = prefix || ''; if (prefix.charAt(prefix.length - 1) ! == '/') {prefix += '/'; } utils.removeDerivedInfo(coverage); collector.add(coverage); TreeSummary = getTreeSummary(collector); // Process coverage tree information, filepath, filename, fileInfo... pathMap = getPathMap(treeSummary); / / each cover under different road information processing Collect all the classification, deposited in the array filePath = filePath | | treeSummary. Root. The fullPath (); OutputNode = pathMap[filePath]; // outputNode = pathMap[filePath]; // If there are specific search parameters, return the node information if (! OutputNode) {// No related information exists in the query path res.statusCode = 404; return res.end('No coverage for file path [' + filePath + ']'); } linkMapper = {hrefFor: function () {return 'https://10.23.176.55:8988' + prefix + 'show? p=' + node.fullPath(); }, fromParent: function (node) { return this.hrefFor(node); }, ancestor: function (node, num) { var i; for (i = 0; i < num; i += 1) { node = node.parent; } return this.hrefFor(node); }, asset: Function (node, name) {/ / resource file processing resource files return 'https://10.23.176.55:8988' + + 'asset/prefix + name; }}; report = Report.create('html', { linkMapper: linkMapper }); Res.setheader (' content-type ', 'text/ HTML '); if (outputNode.kind === 'dir') { report.writeIndexPage(res, outputNode); } else { fileCoverage = coverage[outputNode.fullPath()]; utils.addDerivedInfoForFile(fileCoverage); report.writeDetailPage(res, outputNode, fileCoverage); } return res.end(); } catch (e) {res.send({' failed to find, error details: ': e}); }}Copy the code

Var collector = new Istanbul.Collector() Var collector = new Istanbul.Collector() Var collector = new Istanbul.

Coverage = getCoverageObject(); coverage = getCoverageObject(); Or the data your service has collected so far. You can’t use it all the time.

Later you will need to modify the coverage data with your own library. Replace coverage = XXXX with your own.

Emmm. I don’t know if you can read it, but if you can’t, ask me later.

treeSummary = getTreeSummary(collector); This is actually a tree that deals with coverage. You can see that we collect moral data, which is a JSON tree structure. This is a data processing of those data.

pathMap = getPathMap(treeSummary); // Process the coverage information of each different path to classify all, collect and store the array

filePath = filePath || treeSummary.root.fullPath(); // If no path is specified to find the relevant coverage information, the full data is displayed

LinkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper: linkMapper Built-in properties.

utils.addDerivedInfoForFile(fileCoverage); report.writeDetailPage(res, outputNode, fileCoverage);

These two are to determine the existence of the accessed data, file consolidation, is the above treeSummary pathMap filePath processing after the full data throughput. A report is a text processing of the data, and the final presentation of HTML.

What is the significance of analyzing this fast source code as well?

The processing of this piece is relatively simple. As mentioned above by client, different projects, branches and so on need to be dropped off to the library according to different businesses. In fact, this is the one-to-one processing when you get the data stored before the p parameter value at the end of “show” for display.

You can actually go down here, and say, what does getTreeSummary() do, and so on, and so on, and if you want to get to the bottom of it, you can go down here. I won’t go into the details here.

Plug-in report

The source code related to Istanbul is interpreted above, I believe everyone can understand, even can see deeper than ME. Let’s talk about reporting the data.

For data reporting, as you can see in other teachers’ articles, there are two methods: the Chrome plugin and Fiddler. Actually, another way is the sidebar, container sidecar mode. This is also I consulted thought big guy. The solution given. But I didn’t get it right. If you know, you can tell me. I’d like to ask you something.

I’m just going to talk about plugin reporting. The Chrome plugin. First of all, we need to say that our coverage data exists in the current page. If there is a coverage collection under the window object of the current page, we can obtain it through window.coverage, and then call the client to report it.

If you use the Chrome plugin, you need the Chrome plugin to be able to read the window object of your current page. Otherwise, the following coverage collection will not be obtained.

If you’ve written a Chrome plugin, you know that content_script.js can interact with the dom of the current page, but there’s a problem: you can’t get the window object of the current page.

So what I did here was to write a separate file to the node service that I covered, and write it to the DOM of the current page via content_script.js. The code is as follows:

test.js setTimeout(() => { if (window.__coverage__) { localStorage.setItem('coveragecollect', JSON.stringify(window.__coverage__)) } }, 3000) content_script.js setTimeout(function () { var ss = document.createElement('script') ss.src = "Https://10.23.176.55:8988/test.js" document. The body. The appendChild (ss)}, 3000).Copy the code

For convenience, write to local first, then use the plugin to execute an executeScript from the js on the previous page, and insert another script that interacts with content_script.js. This solves the problem of unavailability:

popup.js let changeColor = document.getElementById('changeColor'); changeColor.onclick = function (element) { let color = element.target.value; chrome.tabs.query({ active: true, currentWindow: true }, function (tabs) { chrome.tabs.executeScript( tabs[0].id, { file: 'js/test.js' }); }); }; Mysql.js setTimeout(function () {var aa = localstorage.getitem ('coveragecollect') if (aa) {console.log(' exists ') var data = aa; var xhr = new XMLHttpRequest(); xhr.withCredentials = true; xhr.addEventListener("readystatechange", function () { if (this.readyState === 4) { alert(this.responseText); }}); XHR. Open (" POST ", "https://10.23.176.55:8988/bilibili/webcoverage/client"); xhr.setRequestHeader("Content-Type", "application/json"); xhr.send(data); } else {console.log(' does not exist ')}}, 3000);Copy the code

See Coverage – Chrome for details.

For details on how to use the plug-in, see my previous article on Chrome Plug-ins

At the end

I’ll continue to explore the automated integration of front-end coverage, referring to Code-Coverage, another great open source.

Ok, that’s all for this article, if you have a higher point of view, welcome to communicate with me, we learn from each other.