The words written in the front
After seeing the comments, I suddenly realized that I did not explain it in advance. This article can be said to be a research study article, which is a feasible plan for me. I will read some similar open source code base later to make up for some details I have missed, so you can use it as a study article and be cautious in production environment.
Record the wrong scene
If your application has access to a Web APM system, you probably know that the APM system can help you catch uncaught errors that occur on a page, give you an error stack, and help you locate bugs. However, sometimes, when you do not know the user’s specific operation, there is no way to reproduce the error, at this time, if there is an operation screen recording, you can clearly understand the user’s operation path, so as to reproduce the BUG and fix.
Implementation approach
Idea 1: Canvas screenshots
This idea is relatively simple, which is to draw web content with canvas. The well-known library is HTML2Canvas. The simple principle of this library is as follows:
- Collect all DOM and store it in a queue;
- According to zIndex, the DOM passes certain rules one by one, and the DOM and its CSS style are drawn on Canvas.
The implementation is a bit complicated, but we can use it directly, so we can get the screenshots we want.
In order to make the generated video smooth, we need to generate about 25 frames per second, that is to say, 25 screenshots. The flow chart is as follows:
However, this idea has a fatal flaw: in order for the video to flow smoothly, we need 25 images per second, and each image is 300KB. When we need 30 seconds of video, the total size of the image is 220M, which is obviously not enough network overhead.
Idea 2: Record all operations repeated
To reduce the network overhead, let’s change the way we think about it. We take steps from the original page and apply them in sequence when we need to “play”, so we can see the changes on the page. This idea separates mouse operations from DOM changes:
Mouse changes:
- Listen for mouseover events and record the mouse’s clientX and clientY.
- During playback, use JS to draw a fake mouse and change the position of the “mouse” according to the coordinate record.
DOM changes:
- Take a full snapshot of the page DOM. This includes collection of styles, JS script removal, and marking each current DOM element with an ID based on certain rules.
- Listen for all events that might affect the interface, such as various mouse events, input events, scroll events, zoom events, and so on. Each event records parameters and a target element, which can be the ID just recorded, so that each change event can be recorded as an incremental snapshot.
- Send a certain amount of snapshot to the back end.
- Plays in the background according to the snapshot and operation chain.
Of course, this description is relatively brief, the mouse record is relatively simple, we will not expand on, mainly explain the IMPLEMENTATION of DOM monitoring ideas.
First full page snapshot
The first thing you might think is that to take a full snapshot of your page, you can use outerHTML directly
const content = document.documentElement.outerHTML;
Copy the codeThis simply keeps track of all the DOM of the page, you just need to add the tag ID to the DOM first, then get outerHTML, then remove the JS script.
But there’s a problem. The DOM logging using outerHTML will merge two neighboring TextNodes into one node. We’ll use MutationObserver to monitor DOM changes later, and you’ll need a lot of processing to accommodate this merging. Otherwise you will not be able to locate the target node of the operation during the restore operation.
So, is there a way to preserve the original structure of the page DOM?
The answer is yes, here we use the Virtual DOM to record the DOM structure, turn the documentElement into the Virtual DOM, record it, and then regenerate the DOM when restoring later.
The DOM is converted to a Virtual DOM
There are only two Node types of concern here: node.text_node and node.element_node. SVG and SVG child elements are created using the createElementNS API, so when we record the Virtual DOM, we need to pay attention to namespace records.
const SVG_NAMESPACE = 'http://www.w3.org/2000/svg';
const XML_NAMESPACES = ['xmlns'.'xmlns:svg'.'xmlns:xlink'];
function createVirtualDom(element, isSVG = false) {
switch (element.nodeType) {
case Node.TEXT_NODE:
return createVirtualText(element);
case Node.ELEMENT_NODE:
return createVirtualElement(element, isSVG || element.tagName.toLowerCase() === 'svg');
default:
return null; }}function createVirtualText(element) {
const vText = {
text: element.nodeValue,
type: 'VirtualText'};if (typeofelement.__flow ! = ='undefined') {
vText.__flow = element.__flow;
}
return vText;
}
function createVirtualElement(element, isSVG = false) {
const tagName = element.tagName.toLowerCase();
const children = getNodeChildren(element, isSVG);
const { attr, namespace } = getNodeAttributes(element, isSVG);
const vElement = {
tagName, type: 'VirtualElement', children, attributes: attr, namespace,
};
if (typeofelement.__flow ! = ='undefined') {
vElement.__flow = element.__flow;
}
return vElement;
}
function getNodeChildren(element, isSVG = false) {
const childNodes = element.childNodes ? [...element.childNodes] : [];
const children = [];
childNodes.forEach((cnode) = > {
children.push(createVirtualDom(cnode, isSVG));
});
return children.filter(c= >!!!!! c); }function getNodeAttributes(element, isSVG = false) {
const attributes = element.attributes ? [...element.attributes] : [];
const attr = {};
let namespace;
attributes.forEach(({ nodeName, nodeValue }) = > {
attr[nodeName] = nodeValue;
if (XML_NAMESPACES.includes(nodeName)) {
namespace = nodeValue;
} else if(isSVG) { namespace = SVG_NAMESPACE; }});return { attr, namespace };
}
Copy the codeWith the above code, we can convert the entire documentElement into a Virtual DOM, where __flow records parameters such as tag IDS, and Virtual Node records: Type, Attributes, children, namespace.
Restore the Virtual DOM to DOM
Restoring the Virtual DOM to the DOM is easier, just creating the DOM recursively, where nodeFilter is used to filter script elements, because we don’t need JS script execution.
function createElement(vdom, nodeFilter = () = >true) {
let node;
if (vdom.type === 'VirtualText') {
node = document.createTextNode(vdom.text);
} else {
node = typeof vdom.namespace === 'undefined'
? document.createElement(vdom.tagName)
: document.createElementNS(vdom.namespace, vdom.tagName);
for (let name in vdom.attributes) {
node.setAttribute(name, vdom.attributes[name]);
}
vdom.children.forEach((cnode) = > {
const childNode = createElement(cnode, nodeFilter);
if(childNode && nodeFilter(childNode)) { node.appendChild(childNode); }}); }if (vdom.__flow) {
node.__flow = vdom.__flow;
}
return node;
}
Copy the codeMonitor DOM structure changes
In this case, we use the API MutationObserver, which, more happily, is compatible with all browsers, so we can use it boldly.
Using MutationObserver:
const options = {
childList: true.// Whether to observe the child node changes
subtree: true.// Whether to observe all descendant node changes
attributes: true.// Whether to observe property changes
attributeOldValue: true.// Whether to observe the old value of the property change
characterData: true.// Whether the node content or node text changes
characterDataOldValue: true.// Whether the node content or node text changes the old value
// attributeFilter: ['class', 'SRC '] is ignored if an attribute change is not in this array
};
const observer = new MutationObserver((mutationList) = > {
// mutationList: array of mutation
});
observer.observe(document.documentElement, options);
Copy the codeIt is easy to use. You only need to specify a root node and some options to monitor. When the DOM changes, the callback will have a mutationList, which is a list of DOM changes.
{
type: 'childList'.// Or characterData, attributes
target: <DOM>,
// other params
}
Copy the codeWe use an array to store the mutation, and the specific callback is:
const onMutationChange = (mutationsList) = > {
const getFlowId = (node) = > {
if (node) {
// The newly inserted DOM has no markup, so compatibility is required here
if(! node.__flow) node.__flow = {id: uuid() };
returnnode.__flow.id; }}; mutationsList.forEach((mutation) = > {
const { target, type, attributeName } = mutation;
const record = {
type,
target: getFlowId(target),
};
switch (type) {
case 'characterData':
record.value = target.nodeValue;
break;
case 'attributes':
record.attributeName = attributeName;
record.attributeValue = target.getAttribute(attributeName);
break;
case 'childList':
record.removedNodes = [...mutation.removedNodes].map(n= > getFlowId(n));
record.addedNodes = [...mutation.addedNodes].map((n) = > {
const snapshot = this.takeSnapshot(n);
return {
...snapshot,
nextSibling: getFlowId(n.nextSibling),
previousSibling: getFlowId(n.previousSibling)
};
});
break;
}
this.records.push(record);
});
}
function takeSnapshot(node, options = {}) {
this.markNodes(node);
const snapshot = {
vdom: createVirtualDom(node),
};
if (options.doctype === true) {
snapshot.doctype = document.doctype.name;
snapshot.clientWidth = document.body.clientWidth;
snapshot.clientHeight = document.body.clientHeight;
}
return snapshot;
}
Copy the codeThe only thing to note here is that when you add a DOM, you need an incremental snapshot, which is still recorded using the Virtual DOM. The DOM is still generated during the playback and inserted into the parent element, so you need to refer to the DOM, which is the sibling node.
Form element monitoring
The above MutationObserver does not monitor changes in the values of elements such as input, so we need to do special processing for the values of form elements.
Oninput event listener
MDN documentation: developer.mozilla.org/en-US/docs/…
Event objects: SELECT, Input, textarea
window.addEventListener('input'.this.onFormInput, true);
onFormInput = (event) = > {
const target = event.target;
if (
target &&
target.__flow &&
['select'.'textarea'.'input'].includes(target.tagName.toLowerCase())
) {
this.records.push({
type: 'input'.target: target.__flow.id,
value: target.value, }); }}Copy the codeWe use capture to capture events on Windows, and we do this later. The reason we do this is that it is possible and often possible to prevent bubbling during the bubble phase to achieve some functionality, so using capture can reduce event loss. In addition, events like Scroll do not bubble and must be captured.
Onchange event listener
MDN documentation: developer.mozilla.org/en-US/docs/…
Input events cannot meet the requirements of checkbox and radio monitoring, so the onchange event is required to monitor
window.addEventListener('change'.this.onFormChange, true);
onFormChange = (event) = > {
const target = event.target;
if (target && target.__flow) {
if (
target.tagName.toLowerCase() === 'input' &&
['checkbox'.'radio'].includes(target.getAttribute('type'))) {this.records.push({
type: 'checked'.target: target.__flow.id,
checked: target.checked, }); }}}Copy the codeOnfocus event listener
MDN documentation: developer.mozilla.org/en-US/docs/…
window.addEventListener('focus'.this.onFormFocus, true);
onFormFocus = (event) = > {
const target = event.target;
if (target && target.__flow) {
this.records.push({
type: 'focus'.target: target.__flow.id, }); }}Copy the codeOnblur event listener
MDN documentation: developer.mozilla.org/en-US/docs/…
window.addEventListener('blur'.this.onFormBlur, true);
onFormBlur = (event) = > {
const target = event.target;
if (target && target.__flow) {
this.records.push({
type: 'blur'.target: target.__flow.id, }); }}Copy the codeMedia element changes monitor
This refers to audio and video, similar to the form elements above, which can listen for onPlay, onPause, timeUpdate, volumechange, etc., and store them in Records
Canvas Canvas change monitor
Canvas content changes do not throw events, so we can:
- Collect canvas elements and periodically update live content
- Hack some drawing API to throw events
The research on Canvas monitor is not very in-depth and needs further research
play
The idea is to get some information from the back end:
- Full snapshot Virtual DOM
- Chain operation records
- Screen resolution
- doctype
Using this information, you can first generate the page DOM, including filtering script tags, and then create an iframe and append it to a container that uses a map to store the DOM
function play(options = {}) {
const { container, records = [], snapshot ={} } = options;
const { vdom, doctype, clientHeight, clientWidth } = snapshot;
this.nodeCache = {};
this.records = records;
this.container = container;
this.snapshot = snapshot;
this.iframe = document.createElement('iframe');
const documentElement = createElement(vdom, (node) => {
/ / cache the DOM
const flowId = node.__flow && node.__flow.id;
if (flowId) {
this.nodeCache[flowId] = node;
}
/ / filter script
return! (node.nodeType === Node.ELEMENT_NODE && node.tagName.toLowerCase() ==='script');
});
this.iframe.style.width = `${clientWidth}px`;
this.iframe.style.height = `${clientHeight}px`;
container.appendChild(iframe);
const doc = iframe.contentDocument;
this.iframeDocument = doc;
doc.open();
doc.write(` <! doctype${doctype}><html><head></head><body></body></html>`);
doc.close();
doc.replaceChild(documentElement, doc.documentElement);
this.execRecords();
}
Copy the codefunction execRecords(preDuration = 0) {
const record = this.records.shift();
let node;
if (record) {
setTimeout((a)= > {
switch (record.type) {
// 'childList', 'characterData',
// 'attributes', 'input', 'checked',
// Handle events such as 'focus', 'blur', 'play', and 'pause'
}
this.execRecords(record.duration);
}, record.duration - preDuration)
}
}
Copy the codeDuration is omitted in the previous article. You can optimize the smoothness of playback according to your own optimization, whether multiple records are presented as a frame or as the original frame.