DNS optimization practice for APP network optimization

Word-wrap: break-word! Important; “> < p style =” max-width: 100%

One, foreword

Mobile APP network optimization is a relatively important client technology to optimize the direction of one of the direction, the vast majority of APP requires network request this step, most of the APP before initiating the first step to do is the DNS domain name resolution, only the domain name resolution into the right IP, to subsequent HTTP or HTTPS requests, Therefore, DNS optimization is the first step in mobile APP network optimization.

Second, the background

With the increasing number of APP users and the increasing coverage of users in different regions and operators, some users have reported that the APP is unavailable in some areas. After a period of positioning and investigation, it is determined that the FAULT is caused by DNS hijacking and DNS fault of operators. Therefore, DNS optimization cannot wait. The following figure shows an instance of carrier DNS hijacking and failure.

Third, HTTPDNS

To solve the problem of DNS hijacking and DNS faults, you need to start from the root cause of DNS resolution. Since the probability and risk of hijacking and fault exist in the carrier’s LocalDNS, you can use HTTPDNS for DNS resolution to bypass the carrier’s LocalDNS resolution, reduce the rate of domain name hijacking and improve the efficiency of domain name resolution.

As for the implementation scheme of HTTPDNS, we have a complete scheme before: APP domain name Dr Scheme. We can choose self-built or third-party SDK scheme according to the situation of our team. Based on the current severity of DNS hijacking and failure, and the cost comparison of implementation solutions. We choose to use Tencent Cloud HTTPDNS SDK for integration at this stage, the overall diagram after integration is as follows:

Best practices for HTTPDNS

Since our current choice is to use the SDK of Tencent Cloud HTTPDNS, we will introduce more best practices of HTTPDNS on the end below.

Best practices for HTTPDNS on Android

The current network layer on the Android side is encapsulated based on OkHttp, which provides the DNS interface for injecting DNS implementations into OkHttp. Thanks to the good design of OkHttp, the implementation of DNS interface can access HTTPDNS for DNS resolution, in more complex scenarios (HTTPS + SNI) do not need to do additional processing, minimal intrusion, so I will not do too much here, see: Tencent Cloud HTTPDNS access document on Android side.

Best practices for HTTPDNS on iOS

The network layer on the iOS side is encapsulated based on AFNetworking. The network framework NSURLSession on the iOS side does not provide interfaces related to DNS resolution for users to customize and modify DNS resolution results. Therefore, there are several general problems to be solved when accessing HTTPDNS on the iOS side. For example, the domain name of the REQUESTED URL is replaced with an IP address, the original HOST is set in the request header, SSL certificate verification, Cookie problem handling, redirection, SNI problem handling, data codecs and link reuse in the SNI scenario, and so on. All these problems require a unified solution.

Therefore, on top of the Tencent Cloud HTTPDNS SDK as the basic capability to provide HTTPDNS, we separately encapsulate the iOS HTTPDNS access layer SDK, which is mainly used to implement some customized policies and solve the above problems, and also facilitate the subsequent replacement of SDK or access to self-deployed HTTPDNS solutions. Upper-layer services are unaware of the existence of low-level HTTPDNS services, reducing service intrusion.

The SDK architecture of the iOS access layer is shown in the following figure:

The interface layer

The interface layer provides simple interfaces, reduces user access costs, and improves development efficiency. For example, the following interfaces are provided at the interface layer:

/// Enable HTTPDNS service - (void)startHTTPDNS; HTTPDNS @property (nonatomic, copy) NSArray<NSString *> *whiteDomainList; @property (nonatomic, copy) NSArray<NSString *> *blackDomainList; /// Whether to allow the caching of IP. If the caching is allowed, if the IP cannot be obtained from a third-party service, the last resolved IP is allowed to be used for the request. By default, YES @property (nonatomic, assign) BOOL is allowedenableCachedIP;
Copy the code

Strategy layer

The policy layer mainly provides different policy combinations and configurations to enable the SDK to provide external HTTPDNS services stably. The following is a brief introduction of each policy:

• Dr Policy: The SDK preferentially uses the HTTPDNS service. When the HTTPDNS service is unavailable, that is, a valid IP address cannot be obtained, the service is automatically degraded to the carrier’s LocalDNS service. This ensures that network requests cannot be sent due to system faults when the HTTPDNS service is unavailable. Note: There is no built-in IP policy at this stage, it will be considered later.
• Blacklist and whitelist policy: APP in the network request domain name, not all network requests to HTTPDNS service, set the whitelist or blacklist, according to the domain name in the blacklist and whitelist to execute HTTPDNS, if set the whitelist, only whitelist domain name HTTPDNS service; If the blacklist is configured, the domain names in the blacklist do not use the HTTPDNS service, and the blacklist has a higher priority than the whitelist.
• Cache policy: Caching strategy In addition to the ttL-based caching strategy provided by Tencent Cloud HTTPDNS SDK in the basic service layer, there is also a memory cache and localized persistent cache in our own encapsulated access layer SDK. The persistent cache is mainly used to solve the problem that the IP in HTTPDNS cannot be obtained when the APP is started. The in-memory cache mainly provides services for querying policies. When a request fails due to an IP address based on HTTPDNS, the cache data of the current domain name and IP address is cleared. The use of caching is also externally controlled.
• Query policy: Query strategy is mainly to solve, in a short period of time based service layer multiple calls the same domain name query service, when the state is being queried, latecomers will no longer call query service, directly read from the memory cache of the cache strategy available IP, if the cache is also use IP, LocalDNS queries directly downgraded to operators. The query policy can effectively reduce the number of interactions with the HTTPDNS server while ensuring service availability.

Injection layer

The injection layer on the iOS side relies on NSURLProtocol to intercept network requests. The usage of NSURLProtocol is not described here. Intercepting network requests based on NSURLProtocol, we have respectively implemented two sets of schemes, in the case of no NEED to deal with SNI scenarios, based on NSURLSession implementation; Server Name Indication (single IP address, multiple HTTPS certificates) scenario needs to be processed based on CFNetwork. Let’s look at two scenarios:

Implementation scheme based on NSURLSession in non-SNI scenarios:

The implementation based on NSURLSession is relatively simple. After intercepting the Request through NSURLProtocol, it only needs to replace the domain name in the Request with IP, set the original Host field and Cookie field in the Request header, rebuild the dataTask task, and initiate the Request. The simple example code is as follows:

NSMutableURLRequest *ipRequest = [originRequest mutableCopy]; // Process the URL and host dnsResultURL to replace the URL after the IP; ipRequest.URL = [NSURL URLWithString:dnsResultURL]; [ipRequestsetValue:url.host forHTTPHeaderField:@"Host"]; // Handle the cookie, because the URL changed, System does not carry the original domain cookie nsstrings * cookieString = [[MFSNICookieManager sharedManager] requestCookieHeaderForURL: url]; [ipRequestsetValue:cookieString forHTTPHeaderField:@"Cookie"];
            
self.ipRequest = ipRequest;
self.clientThread = [NSThread currentThread];
self.ipTask = [[[self class] sharedDemux] dataTaskWithRequest:ipRequest delegate:self modes:self.modes];
                        
if(self.ipTask){
    [self.ipTask resume];
}Copy the code

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didReceiveChallenge:(NSURLAuthenticationChallenge *)challenge completionHandler:(void (^)(NSURLSessionAuthChallengeDisposition, NSURLCredential *_Nullable))completionHandler {
    if(! challenge) {return; } NSURLSessionAuthChallengeDisposition disposition = NSURLSessionAuthChallengePerformDefaultHandling; NSURLCredential *credential = nil; NSString *host = [[self.originRequest allHTTPHeaderFields] objectForKey:@"Host"];
    if(! host) { host = self.originRequest.URL.host; }if ([challenge.protectionSpace.authenticationMethod isEqualToString:NSURLAuthenticationMethodServerTrust]) {
        if ([self evaluateServerTrust:challenge.protectionSpace.serverTrust forDomain:host]) {
            disposition = NSURLSessionAuthChallengeUseCredential;
            credential = [NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust];
        } else{ disposition = NSURLSessionAuthChallengePerformDefaultHandling; }}else{ disposition = NSURLSessionAuthChallengePerformDefaultHandling; } // Use the default validation scheme completionHandler(disposition, credential) directly for the other challenges; } - (BOOL)evaluateServerTrust:(SecTrustRef)serverTrustforDomain:(NSString *) Domain {// create certificate policies NSMutableArray *policies = [NSMutableArray array];if (domain) {
        [policies addObject:(__bridge_transfer id) SecPolicyCreateSSL(true, (__bridge CFStringRef) domain)];
    } else{ [policies addObject:(__bridge_transfer id) SecPolicyCreateBasicX509()]; } SecTrustSetPolicies(serverTrust, (__bridge CFArrayRef) Policies); / * * will assess whether or not the current serverTrust trusted, * official suggested that in the case of the result = kSecTrustResultUnspecified or kSecTrustResultProceed * serverTrust can be verified through, https://developer.apple.com/library/ios/technotes/tn2232/_index.html * about detailed information, please refer to the SecTrust SecTrustResultType h * / SecTrustResultType result;#pragma clang diagnostic push
    #pragma clang diagnostic ignored "-Wdeprecated-declarations"
    SecTrustEvaluate(serverTrust, &result);
    #pragma clang diagnostic pop
    return (result == kSecTrustResultUnspecified || result == kSecTrustResultProceed);
}Copy the code

/ / set the SNI host information nsstrings * host = [self. SniRequest. AllHTTPHeaderFields objectForKey: @"Host"];
if(! host) { host = self.originalRequest.URL.host; } [self.inputStreamsetProperty:NSStreamSocketSecurityLevelNegotiatedSSL forKey:NSStreamSocketSecurityLevelKey];
NSDictionary *sslProperties = @{ (__bridge id) kCFStreamSSLPeerName : host };
[self.inputStream setProperty:sslProperties forKey:(__bridge_transfer NSString *) kCFStreamPropertySSLSettings];Copy the code

In addition to setting SNI information, the implementation scheme based on CFNetwork also needs to consider the problem of data codec, which is rarely mentioned in many open source codes and articles we see. Therefore, we need to add code similar to the following to decode the response data when processing the response data:

// Check content-encoding to see if the returned data needs to be decoded. // Only GZIP is decoded here. If other encoding formats exist in the service scenario, you need to expand them by yourself. NSString *contentEncoding = [self.response.headerFields objectForKey:@"Content-Encoding"];
if (contentEncoding && [contentEncoding isEqualToString:@"gzip"]) {
    [self.delegate task:self didReceiveData:[self ungzipData:self.resultData]];
} else {
    [self.delegate task:self didReceiveData:self.resultData];
}Copy the code

In addition, it is very important to consider the problem of connection reuse based on CFNetwork implementation. It cannot be recreated on every request, and the cost of reconnection is very high. This is also an area that we will never mention in the open source code or in this article, and if left untreated, the performance cost can be severe.

Especially since most of our requests are already HTTP2.0, the performance comparison will be even more obvious. However, since apple’s CFNetwork framework does not support HTTP2.0, it is difficult to implement the related features of HTTP2.0 based on CFNetwork. We are currently implementing the HTTP1.1 protocol connection reuse part of the function, do not need to re-establish a connection every request.

The basic principle is that when the request of the same host, port and scheme is initiated, if there is an available connection that can be reused, there is no need to re-establish the connection, and the connection can be reused directly. If the connection expires locally, or the server initiatively closes the connection through the response header, the connection is not reused and the connection is closed. To determine whether the Connection of the server is multiplexed, determine whether the Connection of the response header is keep-alive or close.

Basic service layer

Basic service layer at the present stage mainly relies on Tencent Cloud HTTPDNS SDK to provide basic query services, mainly provides ttL-based cache storage and expiration processing logic, at the same time, this layer also provides SDK internal cache storage and log and basic verification and other functions.

Summary of best practices

Therefore, if you have high performance requirements and need to deal with SNI scenarios, I recommend not using HTTPDNS directly and actively, but using CFNetwork-based network requests directly at the bottom if the IP request obtained by the carrier’s LocalDNS fails. In this way, you can strike a balance between DNS hijacking and performance. You can ensure that you can use HTTPDNS when the carrier’s LocalDNS resolution fails, ensuring success and availability. At the same time, you can use nsurLSession-based requests when the carrier’s LocalDNS is available, and enjoy the performance improvement brought by the system’s implementation of HTTP2.0 features.

If you don’t need to deal with SNI, you can use an NSURLSession-based implementation.

Five, the revenue

Since the launch of DNS optimization, obvious optimization effects have been achieved, and the overall error rate of the interface has decreased by more than 20%. The whole site unknown host error decreased by 80% (the whole site many domain names, currently only the core domain name switched HTTPDNS, so the optimization effect is far greater than 80%), at the same time in the case of simulating DNS hijacking, the APP core functions can be used normally.

Six, the concluding

DNS optimization is a continuous thing, based on the present situation and the problems we have adopted the optimization scheme, this scheme is not necessarily a perfect solution at present, could still exist certain problems, in the design for the expansion of the subsequent iteration has kept good scalability, we will in the present scheme based on constant optimization and evolution.

Finally, thank you for your hard reading, I hope I can help you a little, thank you very much.

Vii. Reference materials

• Tencent Cloud HTTPDNS SDK documentation

The copyright belongs to the author. Commercial reprint, please contact this account for authorization, non-commercial reprint, please indicate the daily Excellent fresh big front end team and the original address.