【 crawler knowledge 】 browser developer tools use skills summary

The overview

Browser developer tools are often used for simple packet capture analysis and JS reverse debugging in crawlers.

F12.
Shortcut Ctrl+Shift+I;
Right-click to check or review elements;
In the upper right corner of the browser – > More Tools – > Developer Tools

Common means to disable the developer tools: blog.csdn.net/cplvfx/arti…

The official document: developer.chrome.com/docs/devtoo…

Elements: Use the Elements panel to re-create the layout and design of your site by manipulating the DOM and CSS freely.
Console: During development, you can use the Console panel to record diagnostic information or to use it as a shell to interact with JavaScript on the page.
Sources (Source panel) : Set breakpoints in the source panel to debug JavaScript, or use the live editor of developer tools via Workspaces to connect to local files.
Network (Network panel) : Information about each requested resource (including status, resource type, size, and elapsed time) obtained after a web page Request Request is initiated. Network performance can be optimized based on this information.
Performance: Use the Timeline panel to improve page runtime Performance by recording and viewing various events that occur during the life of your site.
Memory panel: Analyzes execution time and Memory usage of web applications or pages.
Application: Records all resource information loaded by the website, including stored data (Local Storage, Session Storage, IndexedDB, Web SQL, Cookies), cached data, fonts, images, scripts, style sheets, etc.
Security panel: Use the Security panel to debug mixed content issues, certificate issues, etc.
Lighthouse (Diagnostic Panel) : It diagnoses the current web page in terms of network utilization and web performance, and provides recommendations for improving it.

– (element selection) : you can directly click on the page element, will automatically jump to the corresponding source code.

(Terminal simulation) : Simulates various terminal devices and supports custom terminals.
(Settings) : Developer tool Settings, including some appearance, shortcuts, terminal devices, location Settings, etc.
(Custom) : Customize and control developer tools, including adjusting tool location, global search, running commands, and other tools.

Terminal emulation

Click to simulate all kinds of terminal devices, suitable for viewing the data of mobile phone page, click on [More tools] — > [Sensors] can simulate the geographic location of terminal, terminal orientation, etc. Toolbar can be selected to simulate the terminal model, Responsive is adaptive.

The Network panel

Controls the controller

Preserve log: Whether to clear the request list after the page is reloaded.
Disable cache: Indicates whether cache is enabled.
: Indicates whether to enable packet capture.
: Clear the request.
: Whether to hide the Filter pane.
: search.
Allows you to test your website in a variety of Network environments, including 3G, offline, etc. You can also customize maximum download and upload traffic.
: Import/Export HAR file to Import or Export captured packet data.

The Filter Filter

Hide Data URLs: Data URLs are small files embedded in documents. Data: files that start with data: in the request table are, for example, the more common SVG files. To Hide such files, check the Hide Data URLs check box.
All: Displays All requests.
XHR: XMLHttpRequest, is a JavaScript API for creating AJAX requests. XHR is usually an option for fetching AJAX requests.
WS: WebSocket, is a protocol for full duplex communication over a single TCP connection provided by HTML5.
Manifest Android development file name, belongs to the Androidmanifest.xml file, in the simple Android system application presents important information code.
Has blocked cookies: Only requests that have blocked response cookies are displayed.
Blocked Requests: Only Blocked Requests are shown.

Breakpoint debugging

Regular breakpoint debugging

Good for analyzing key function code logic

Ctrl+Shift+F or three clicks in the upper right corner to open global search and search for keywords.
Locate the suspicious code and click the line number to bury the breakpoint.
Debug code, analyze logic, where the console template can write JS code directly for debugging.

Each option function:

: Executes to the next breakpoint.
: Performs the next step without going inside the called function.
: goes inside the function being called.
: Out of the function.
: Step by step to execute the code, when there is a function call, then enter the function.
: Disables breakpoints.
: Do not pause when exceptions occur.
Breakpoints: You can see Breakpoints that have been buried.
Scope: The value of the current local or global variable can be seen and modified.
Call Stack: Displays the Stack of current code calls in bottom-up order.

XHR breakpoint

If the keyword in the URL is matched, the parameter is generated. This method is applicable to encryption parameters in the URL that cannot be found in the global search.

Conduct a breakpoint

An Event Listener breakpoint is a breakpoint that can be triggered when a Mouse click, movement, keyboard keystroke or other events occur, such as Mouse – > click, which can quickly locate the JS executed after the button is clicked.

Insert the JS

You can create a new JS script under Sources — > Snippets.

Print the value of a Windows object

Enter the following code in console, such as printing only variable values starting with _$:

for (var p in window) {
    if (p.substr(0.2)! = ="_ $") 
        continue;
    console.log(p + " >>> " + eval(p))
}
Copy the code

Unlimited debugger anti-debugging

Some pages will display an infinite debugger when debugger is opened:

Middleman intercepts replace infinite debug functions

To view the call stack, click the second line to jump to the original function:

_0x2ba9bc[_0x20B2 (‘0x79’)] = _0x2ba9bc[_0x20B2 (‘0x7a’)] = _0x2ba9bc[_0x20B2 (‘0x7a’)]

Use ReRes to write the rule, replace the JS with our local modified JS, and the infinite debugger no longer exists:

Methods empty

It is also possible to override the infinite Debugger function directly on the Console and empty it. The drawback is that the infinite Debugger fails after refreshing.

Disarming timer

Debug for timer class triggering:

for (var i = 1; i < 99999; i++)window.clearInterval(i);
Copy the code

Hook Hook

In Windows, everything is a message, you press the keyboard, it is a message, the meaning of the message before the past, do not let the execution of the message, and then their own priority. That is, this technique provides an entry point to be able to perform my actions for different messages or apis before executing them. My operation is the hook function. In developer tools, insert a breakpoint where the keyword matches as a Chrome plugin.

Create a folder in which you will create the hook function file inject.js and the plug-in’s configuration file manifest.json:

Open the Chrome extensions, open developer mode, load the unzipped extensions, and select the folder you created:

Configuration file manifest.json

For example, a header hook has the following configuration file:

{
   "name": "Injection"."version": "1.0"."description": "RequestHeader hooks"."manifest_version": 1."content_scripts": [{"matches": [
                "<all_urls>"]."js": [
                "inject.js"]."all_frames": true."permissions": [
                "tabs"]."run_at": "document_start"}}]Copy the code

The header hook

The header hook is used to locate where key parameters in the header are generated, and the following code demonstrates inserting a breakpoint when Authorization is included in the header

var code = function(){
var org = window.XMLHttpRequest.prototype.setRequestHeader;
window.XMLHttpRequest.prototype.setRequestHeader = function(key,value){
    if(key=='Authorization') {debugger;
    }
    return org.apply(this.arguments); }}var script = document.createElement('script');
script.textContent = '(' + code + ') () ';
(document.head||document.documentElement).appendChild(script);
script.parentNode.removeChild(script);
Copy the code

Cookie hooks

The cookie hook is used to locate key arguments in the cookie. The breakpoint is inserted when abcdefghijk is matched in the cookie:

var code = function(){
    var org = document.cookie.__lookupSetter__('cookie');
    document.__defineSetter__("cookie".function(cookie){
        if(cookie.indexOf('abcdefghijk') > -1) {debugger;
        }
        org = cookie;
    });
    document.__defineGetter__("cookie".function(){returnorg; }); }var script = document.createElement('script');
script.textContent = '(' + code + ') () ';
(document.head||document.documentElement).appendChild(script);
script.parentNode.removeChild(script);
Copy the code

Request a hook

The request hook is used to locate key parameters in the request. The following code shows how to insert a breakpoint when the request URL contains AbCdE:

var code = function(){
var open = window.XMLHttpRequest.prototype.open;
window.XMLHttpRequest.prototype.open = function (method, url, async){
    if (url.indexOf("AbCdE") > -1) {debugger;
    }
    return open.apply(this.arguments);
};
}
var script = document.createElement('script');
script.textContent = '(' + code + ') () ';
(document.head||document.documentElement).appendChild(script);
script.parentNode.removeChild(script);
Copy the code