Comprehensive understanding of mobile DNS domain name hijacking and other miscellaneous diseases: principles, root causes, HttpDNS solutions, etc

This article quoted Tencent engineer Liao Weijian published in “the” public number “” [the” network “] global precise traffic scheduling new ideas -HttpDNS service details “article part of the content, thank the original author of the share.

1, the introduction

2. Related articles

Quick Understanding of Network Communication Protocols (Part 1)

A Quick Understanding of Network Protocols (Part 2)

Network programming lazy entry (six) : the history of the most popular hub, switch, router function principle entry

Network programming lazy introduction (7) : simple, comprehensive understanding of HTTP protocol

Network programming lazy introduction (nine) : popular explanation, IP address, why use MAC address?

Technology Literacy: A new generation of UDP-based low latency Network Transport layer protocol – QUIC in detail

Summary of optimization means of modern mobile terminal network short connection: request speed, weak network adaptation, security guarantee

Mobile IM Developers must read (1) : Easy to understand the “weak” and “slow” mobile web

Mobile IM Developers must read (ii) : Summary of the most comprehensive mobile weak Network optimization methods ever

Introduction to brain-damaged Network programming (5) : Ping command used every day, what is it?

Introduction to Network Programming (6) : What are public IP and internal IP? What the hell is NAT?

3. Overview of the text

4. First, what is DNS?

4.1 Working Principles of DNS

The basic principle of DNS is shown in the following figure:

4.2DNS Domain Name System Structure

As shown in the figure above, the structure of a typical DNS domain name system is as follows:

1) Root domain name: when DNS domain name is used, it is specified by the end of the period to specify that the name is in the Root or higher domain hierarchy;
2) Top Level domain: A type name used to indicate the name used by a country, region, or organization. Such as.net.
3) Second Level domain name: a registered name used by a person or organization on the Internet. Such as 52 im.net;
4) Third Level domain name: a domain derived from a registered second Level domain name. Such as docs.52im.net.

4.3DNS Resolution Process

As shown above, this is a typical domain name resolution process:

1) Enter www.52im.net in the browser to send a resolution request.
2) The local domain name resolver program queries whether the mapping between the local cache and the host file is domain name. If so, it calls the IP address mapping and completes the resolution.
3) If the hosts does not map to the Local parser cache, the Local parser will initiate a recursive query request to the preferred DNS server (called the Local DNS server) set in the TCP/IP parameter.
4) When the server receives the query, if the local server is responsible for resolving the domain name to be queried, the server returns the resolution result to the client to complete domain name resolution, which is authoritative. If the domain name to be queried is not resolved by the Local DNS server, but the Local DNS server has cached the IP address mapping, invoke the IP address mapping to resolve the domain name. The resolution is not authoritative.
5) If the Local DNS server fails to resolve the Local zone file and cache, the Local DNS server performs the query based on the Local DNS server Settings (whether recursive). If the enabled mode is not enabled, the Local DNS sends the request to the 13 Root DNS servers. In recursive mode, the DNS server forwards the request to the upper-level DNS server for resolution. If the upper-level SERVER fails to resolve the request, the DNS server searches for the root DNS server or forwards the request to the upper-level DNS server.
6) The Root DNS server receives the request and determines who is authorized to manage the domain name and returns an IP address that is responsible for the top-level domain name server.
7) When the Local DNS server receives the IP information, it will contact the server responsible for the.NET domain.
8) If the server in charge of the.COM domain cannot resolve the request, it will find a DNS server address of the next level of the MANAGEMENT of the.NET domain to the local DNS server.
9) When the Local DNS server receives the IP address, it searches for the domain server 52IM.net. 10 and 11 repeat the preceding steps to query the IP address.
10) finally www.52im.net returns the IP address of the domain name to the Local DNS server;
11) The Local DNS server caches the result (as well as the result returned by 6, 8, and 10);
12) The Local DNS server returns the result to the Local domain name resolver.
13) Local cache parsing results;
14) The native parser returns the results to the browser;
15) The browser initiates the request from the returned IP address.

4.4 Recursive and iterative DNS Query

Recursive query:

Iterative query:

5. Various DNS syndromes faced by domestic mobile terminal networks

To sum up, there are three main problems with DNS:

1) LocalDNS hijacking;
2) Average access latency decreases;
3) The user connection failure rate decreases.

LocalDNS hijacked:

Average access latency decreased:

User connection failure rate decreased:

6. What is the root cause of the DNS problem in China?

These problems caused by LocalDNS can be attributed to the following three reasons:

Domain name caching;
Parsing forward;
LocalDNS Recursive egress NAT.

6.1 Domain Name Caching

The schematic diagram is as follows:

Why does LocalDNS cache domain name resolution results? There are several reasons for this:

1) Ensure that users’ access traffic is digested in the network: there are great differences in bandwidth resources, inter-network settlement fees, IDC room distribution and ICP resource distribution within the network among domestic Internet access operators. In order to ensure the access quality of users on the network and reduce cross-network settlement, carriers set up content cache servers on the network. By forcibly pointing the domain name to the IP address of the content cache server, they realize the purpose of keeping the local traffic completely on the local network.
2) Push advertisement: some LocalDNS will replace the content cache pointed to by some domain name resolution results with advertisements of the third-party advertising alliance.

The above type of behavior is commonly referred to as domain name caching. Domain name caching can cause users to experience the following access exceptions:

A. Only the HTTP service of port 80 is cached. If the domain name is provided through HTTPS or other ports, users will fail to access the domain name. For example, payment services, games through the designated port connect server service;
B. The operation and maintenance level of the cache server is uneven, and the cache server failure sometimes leads to abnormal user access.

6.2 Resolving Forwarding

The normal LocalDNS recursive resolution process looks like this:

In order to save resources, some small operators directly forward the resolution request to the recursive LocalDNS of other operators:

6.3LocalDNS Recursive egress NAT

As shown below:

7. These problems must be addressed, but the traditional solutions are too problematic

So how did we deal with these domain name resolution exceptions in the past?

1) Real-time monitoring + business promotion:

2) Bypass automatic DNS assignment and use 114DNS or Google Public DNS:

A. How to construct a domain name request on the user side: It is not difficult for the PC client to construct a standard DNS request package. However, it is technically possible to send a standard DNS request package to a specified LocalDNS on the mobile end, and it is compatible with various iOS and Android versions, but the cost of compatibility will be high.
B. Push users to modify the configuration extremely high: If you want to push users to manually modify the DNS configuration of the PC, it is barely feasible under the WiFI of the PC and mobile client. However, it is very difficult for users to modify the DNS configuration in the mobile Internet environment.

3) Completely abandon the domain name and build connectCenter for traffic scheduling:

8, the current mainstream solution: HttpDNS appears

8.1 What HttpDNS?

HTTPDNS works like this:

8.2 Main problems solved by HttpDns

Local DNS hijacking:

Average access latency decreased:

User connection failure rate decreased:

8.3 Tencent’s Idea of HttpDNS

Tencent’s HttpDNS fundamentals:

The HttpDNS principle is very simple and has two main steps:

A. The client directly accesses the HttpDNS interface to obtain the IP address with the optimal access delay configured in the domain name management system. (For disaster recovery purposes, retain the carrier’s LocalDNS for domain name resolution.)
B. The client sends a service protocol request to the obtained IP address. Using Http requests as an example, you can send a standard Http request to the IP returned by HttpDNS by specifying the host field in the header.

Advantages of HttpDNS:

But there are countless benefits to this small shift:

A. Radical solution of domain name resolution exception: because the LocalDNS of the carrier is bypassed, the user’s domain name resolution request is directly transmitted to the IP address of Tencent’s HttpDNS server through Http protocol. The user’s domain name resolution request on the client will not suffer from domain name resolution exception.
B, accurate scheduling: HttpDNS can directly obtain the user IP, through the combination of Tencent’s own patented technology generated IP address library and speed measurement system, can ensure that users will guide access to the fastest IDC node;
C, low implementation cost: access to HttpDNS services only need to do a small amount of client access layer transformation, without user phone root or jailbreak; And because Http protocol request construction is very simple, compatible with all versions of mobile operating system is not a problem; In addition, the backend configuration of HttpDNS completely reuse the existing authoritative DNS configuration, and the management cost is very low. In a word, it can solve the problem of abnormal domain name resolution and meet the demand of accurate traffic scheduling with the minimum transformation cost.
D, strong scalability: HttpDNS provides reliable domain name resolution services, business can combine its own scheduling logic and HttpDNS return results, to achieve more refined traffic scheduling. For example, the IP address of the client connection request of the specified version or the IP address of the user connection of the specified network type.

Of course you may ask:

Access effect and future outlook:

9. As a startup team, how to transform APP and support HttpDNS?

9.1 Use the HttpDNS interface provided by the third-party cloud service provider

Ali cloud HttpDNS
Tencent cloud HttpDNS
Huawei cloud HttpDNS

http://203.107.1.1/d?host=www.52im.net

{" host ":" www.linkedkeeper.com ", "ips" : [" 115.238.23.241 ", "115.238.23.251"], "TTL" : 57}Copy the code

9.2 Open source HttpDNS library shared by Sina Weibo team: HTTPDNSLib

Github address for HTTPDNSLib:
Github.com/CNSRE/HTTPD…

HTTPDNSLib implementation of HttpDNS interaction flow principle:

HttpDns Lib library interaction

Here is a screenshot of the test Demo:

App domain name hijacking DNS high availability – open source VERSION of HttpDNS solution

Appendix: more excellent articles on network communication

TCP/IP, rounding
Chapter 11 ·UDP: User datagram protocol

TCP/IP, rounding
Chapter 17: TCP: Transmission control Protocol

TCP/IP, rounding
Chapter 18: Establishment and termination of TCP connections

TCP/IP, rounding
Chapter 21. TCP Timeouts and retransmission

Once upon a time in technology: TCP/IP changed the world

Easy to Understand – In-depth understanding of TCP (part 1) : Theoretical basis

Easy to Understand – In-depth understanding of TCP (part 2) : RTT, sliding window, and congestion handling

Theory classic: TCP protocol three handshake and four wave process in detail

Connection between theory and practice: Wireshark captures packets to analyze TCP three-way handshake and four-way wave

Protocol Diagram of Computer Network Communication (Chinese Edition)

What is the maximum size of a packet in UDP?

P2P technology details (a) : NAT details – detailed principle, P2P introduction

(2) : P2P NAT traversal (hole) solution

P2P technology details (three) : P2P technology STUN, TURN, ICE details

Easy to Understand: Quickly understand the PRINCIPLE of NAT penetration in P2P technology

High performance network programming (a) : the number of concurrent TCP connections can be a single server

High performance Network programming (II) : The last 10 years, the famous C10K concurrent connection problem

High performance Network Programming (III) : In the next 10 years, it is time to consider C10M concurrency

High performance Network programming (IV) : Theoretical exploration of high performance network applications from C10K to C10M

High performance network programming (5) : read the I/O model of high performance network programming

High performance network programming (6) : understand the threading model of high performance network programming

Unknown Network Programming (I) : A Brief Analysis of the DIFFICULT Problems in TCP Protocol (Part I)

Unknown Network Programming (II) : A Brief Analysis of the DIFFICULT Problems in TCP Protocol (Part II)

Why TIME_WAIT and CLOSE_WAIT when closing TCP connections

Unknown network programming (iv) : In-depth analysis of TCP abnormal shutdown

Network programming unknown (5) : UDP connectivity and load balancing

Unknown network programming (6) : Understand UDP in depth and use it well

Unknown network programming (7) : How to make unreliable UDP reliable?

Quick Understanding of Network Communication Protocols (Part 1)

A Quick Understanding of Network Protocols (Part 2)

Network programming lazy introduction (three) : a quick understanding of the TCP protocol is enough

Network programming lazy starter (4) : Quickly understand the difference between TCP and UDP

Why UDP sometimes has an advantage over TCP

Network programming lazy entry (six) : the history of the most popular hub, switch, router function principle entry

Network programming lazy introduction (7) : simple, comprehensive understanding of HTTP protocol

Network programming lazy entry (eight) : teach you to write based on TCP Socket long connection

Network programming lazy introduction (nine) : popular explanation, IP address, why use MAC address?

Technology Literacy: A new generation of UDP-based low latency Network Transport layer protocol – QUIC in detail

Making the Internet faster: A new generation of QUIC protocols shared in Tencent’s technology practices

Summary of optimization means of modern mobile terminal network short connection: request speed, weak network adaptation, security guarantee

Let’s talk about long connections in iOS network programming

Mobile IM Developers must read (1) : Easy to understand the “weak” and “slow” mobile web

Mobile IM Developers must read (ii) : Summary of the most comprehensive mobile weak Network optimization methods ever

IPv6 Technology details: Basic Concepts, Application Status, Technical Practice (Part I)

IPv6 Technology details: Basic Concepts, Application status, Technical Practice (Part 2)

From HTTP/0.9 to HTTP/2: Understand the history and design of the HTTP protocol

Brain-disabled network programming introduction (a) : follow the animation to learn TCP three handshakes and four waves

What are we reading and writing when we read and write sockets?

Brain-disabled network programming introduction (3) : HTTP protocol must know must know some knowledge

Quick Understanding of HTTP/2 Server Push

Introduction to brain-damaged Network programming (5) : Ping command used every day, what is it?

Introduction to Network Programming (6) : What are public IP and internal IP? What the hell is NAT?

Take the network access layer design of online game server as an example to understand the technical challenges of real-time communication

To the next level: The network basics that a good Android programmer must know

Comprehensive understanding of mobile DNS domain name hijacking and other miscellaneous problems: technical principles, root causes, solutions, etc

More of the same…