I met a very interesting question in the school recruitment interview before:

“Let’s say you have a very, very, very large Web project where the JS source code is compressed to over 10MB. =), requiring the user to reload JS after every update is unacceptable, so how to solve this problem from an engineering perspective?”

At the beginning, I immediately thought of several solutions, such as:

  1. Pull out basic infrequently updated modules for long-term caching;
  2. If you use a framework like React or Vue2.0 that supports server-side rendering, use server-side rendering and then progressively block loading of JS.
  3. For Hybrid development, you can consider using local resources to load, similar to the idea of “offline packages” (which I encountered every day during my internship at Tencent).

Later, under the guidance of the interviewer, I came up with a solution of “incremental update”. Simply speaking, during the version update, there is no need to reload the resource, just need to load a small piece of diff information and merge it into the current resource, similar to the effect of Git merge.

1. The client uses LocalStorage or other storage schemes to store a copy of the original code + time stamp:

{
    timeStamp: "20161026xxxxxx",
    data: "aaabbbccc"
}
Copy the code

2. Send this timestamp to the server every time a resource is loaded.

3, The server identifies the client version from the received timestamp, does a diff with the latest version, and returns the diFF information for both:

diff("aaabbbccc", "aaagggccc"); // [3, "-3", "+ GGG ", 3]Copy the code

4. After receiving the diff information, the client updates the local resource and timestamp to the latest, implementing an incremental update:

mergeDiff("aaabbbccc", [3, "-3", "+ggg", 3]);
//=> "aaagggccc"
Copy the code


Second, the practice

The following is the core idea of this scheme to achieve again, in a nutshell, is to achieve diff and mergeDiff two functions.

A good diff algorithm was found today:

GitHub – kpdecker/jsdiff: A javascript text differencing implementation.

We just need to call its diffChars method to compare the difference between two strings:

var oldStr = 'aaabbbccc';
var newStr = 'aaagggccc';
JsDiff.diffChars(oldStr, newStr);
//=>
//[ { count: 3, value: 'aaa' },
//  { count: 3, added: undefined, removed: true, value: 'bbb' },
//  { count: 3, added: true, removed: undefined, value: 'ggg' },
//  { count: 3, value: 'ccc' } ]
Copy the code

The above diff information is a bit redundant, we can customize a more concise representation to speed up the transmission speed:

The integer represents the number of unchanged characters, the string starting with “-” represents the number of removed characters, and the string starting with “+” represents the newly added characters. So we can write a minimizeDiffInfo function:

function minimizeDiffInfo(originalInfo){
    var result = originalInfo.map(info => {
        if(info.added){
            return '+' + info.value;
        }
        if(info.removed){
            return '-' + info.count;
        }
        return info.count;
    });
    return JSON.stringify(result);
}

var diffInfo = [ 
    { count: 3, value: 'aaa' },
    { count: 3, added: undefined, removed: true, value: 'bbb' },
    { count: 3, added: true, removed: undefined, value: 'ggg' },
    { count: 3, value: 'ccc' } 
];
minimizeDiffInfo(diffInfo);
//=> '[3, "-3", "+ggg", 3]'
Copy the code

The client receives the reduced diff information and generates the latest resource:

mergeDiff('aaabbbccc', '[3, "-3", "+ggg", 3]');
//=> 'aaagggccc'

function mergeDiff(oldString, diffInfo){
    var newString = '';
    var diffInfo = JSON.parse(diffInfo);
    var p = 0;
    for(var i = 0; i < diffInfo.length; i++){
        var info = diffInfo[i];
        if(typeof(info) == 'number'){
            newString += oldString.slice(p, p + info);
            p += info;
            continue;
        }
        if(typeof(info) == 'string'){
            if(info[0] === '+'){
                var addedString = info.slice(1, info.length);
                newString += addedString;
            }
            if(info[0] === '-'){
                var removedCount = parseInt(info.slice(1, info.length));
                p += removedCount;
            }
        }
    }
    return newString;
}
Copy the code


Three, the actual effect

You can run this directly if you are interested:

GitHub – starkwang/Incremental

Use create-react-app to create a react project, change two lines of code, and compare the old and new versions of the react project:

var JsDiff = require('diff');
var fs = require('fs');

var newFile = fs.readFileSync('a.js', 'utf-8');
var oldFile = fs.readFileSync('b.js', 'utf-8');
console.log('New File Length: ', newFile.length);
console.log('Old File Length: ', oldFile.length);

var diffInfo = getDiffInfo(JsDiff.diffChars(oldFile, newFile));
console.log('diffInfo Length: ', diffInfo.length);
console.log(diffInfo);

var result = mergeDiff(oldFile, diffInfo);
console.log(result === newFile);
Copy the code

Here are the results:

As you can see, the code after build is 21W + characters (212KB), while the diff message is quite small at 151 characters, which is more than a thousand times smaller than when reloading the new version (of course I only changed two or three lines of code here, small is natural).

Four, some problems not covered

The above is just to realize the core idea again, there are more things to consider in the actual project:

The server cannot recalculate the DIFF for every request, so it must cache the DIFF information.

2. Implementation scheme of client-side persistent storage, such as favorite LocalStorage, Indexed DB, Web SQL, or interface provided by Native APP;

3. Fault tolerance, consistency check between client and server, and implementation of forced refresh.