DNS optimization practice for APP network optimization

One, foreword

Mobile APP network optimization is a relatively important client technology to optimize the direction of one of the direction, the vast majority of APP requires network request this step, most of the APP before initiating the first step to do is the DNS domain name resolution, only the domain name resolution into the right IP, to subsequent HTTP or HTTPS requests, Therefore, DNS optimization is the first step in mobile APP network optimization.

Second, the background

As the number of APP users keeps increasing and the coverage of users in different regions and operators keeps increasing, some users have reported that the network is unavailable in some regions of APP. After a period of positioning and investigation, it is determined to be caused by carrier DNS hijacking and carrier DNS failure. Therefore, DNS optimization is urgent. The following figure shows an example of carrier DNS hijacking and failure.

Third, HTTPDNS

In order to solve the problem of DNS hijacking and DNS failure, we need to start from the root of DNS resolution, since the carrier’s LocalDNS hijacking and failure probability and risk, so we use HTTPDNS to resolve the DNS to bypass the carrier’s LocalDNS resolution, so as to reduce the domain name hijacking rate, improve the efficiency of domain name resolution.

As for the implementation of HTTPDNS, the TO B team has a complete solution: APP domain name Dr Solution, depending on the situation of each team can choose TO build their own or third-party SDK solution. According to the current severity of DNS hijacking and failure, as well as the cost comparison of implementation solutions. We choose to use Tencent Cloud HTTPDNS SDK for integration at this stage, the overall diagram after integration is as follows:

Best practices for HTTPDNS

Because we choose to use Tencent cloud HTTPDNS SDK at this stage, so let’s introduce more HTTPDNS best practices on the end.

HTTPDNS best practices on Android

The current network layer on Android is encapsulated based on OkHttp, which provides an interface for DNS to inject DNS implementations into OkHttp. Thanks to OkHttp’s good design, the implementation of DNS interface can access HTTPDNS for DNS resolution. In more complex scenarios (HTTPS + SNI), there is no need to do additional processing, so the intrusion is minimal, so I will not do too much introduction here, see: Tencent Cloud HTTPDNS access document on the Android terminal.

Best practices for HTTPDNS on iOS

The network layer on iOS is encapsulated based on AFNetworking. The network framework NSURLSession on iOS does not provide DNS resolution interfaces for users to customize DNS resolution results. Therefore, there are several common problems to be solved when accessing HTTPDNS on iOS. For example, replace the domain name of the REQUESTED URL with an IP address, set the original HOST in the request header, verify the SSL certificate, handle cookies, redirection, handle problems in the SNI scenario, and codec data and link reuse in the CORRESPONDING SNI scenario. All these problems require a unified solution.

Therefore, on the basis of Tencent cloud HTTPDNS SDK as a basic capability to provide HTTPDNS, we separately encapsulated the iOS side HTTPDNS access layer SDK, mainly used to achieve some customized strategies and solve the above problems, but also convenient for the subsequent replacement of SDK or access self-deployed HTTPDNS solution. So that the upper-layer business parties can not perceive the existence of the underlying HTTPDNS service, reducing service intrusion.

The architecture diagram of iOS access layer SDK is as follows:

The interface layer

The interface layer mainly provides simple interfaces to reduce user access costs and improve development efficiency. For example, some interfaces provided by the interface layer are as follows:

// enable the HTTPDNS service
- (void)startHTTPDNS;

/// Whitelist list. If a whitelist is set, only whitelisted domain names go to the HTTPDNS service
@property (nonatomic.copy) NSArray<NSString *> *whiteDomainList;

/// Blacklist list. If a blacklist is set, the domain names in the blacklist do not go through HTTPDNS. The blacklist has the highest priority
@property (nonatomic.copy) NSArray<NSString *> *blackDomainList;

/// Whether to allow the CACHE of IP addresses. If the cache is allowed, the request can be made using the last successful IP address when the third-party service fails to obtain the IP address. The default value is YES
@property (nonatomic.assign) BOOL enableCachedIP;
Copy the code

Strategy layer

The policy layer provides different policy combinations and configurations to enable the SDK to stably provide HTTPDNS services externally. The following describes the contents of each policy:

• Dr Policy: The SDK uses the HTTPDNS service preferentially. When the HTTPDNS service is unavailable, that is, no valid IP address can be obtained, the service is automatically degraded to the carrier’s LocalDNS service. This ensures that the system cannot send network requests when the HTTPDNS service is unavailable. Note: There is no built-in IP policy at this stage. We will consider it later.
• Not all network requests go to the HTTPDNS service. After a whitelist or blacklist is set, HTTPDNS is executed based on the domain names in the whitelist. If a whitelist is set, only the domain names in the whitelist go to the HTTPDNS service. If a blacklist is configured, domain names in the blacklist do not use the HTTPDNS service. The priority of the blacklist is higher than that of the whitelist.
• Cache policy: Cache strategy In addition to the basic service layer in Tencent cloud HTTPDNS SDK provided based on TTL cache strategy, our own encapsulation of the access layer SDK also exists a memory cache and localized persistent cache, persistent cache is mainly used to solve the problem of starting APP can not get the IP in HTTPDNS. The memory cache primarily serves query policies. If a request fails due to an HTTPDNs-based IP address, the cache data of the current domain name and IP address is cleared. At the same time, external control can be used cache.
• Query policy: Query strategy is mainly to solve, in a short period of time based service layer multiple calls the same domain name query service, when the state is being queried, latecomers will no longer call query service, directly read from the memory cache of the cache strategy available IP, if the cache is also use IP, LocalDNS queries directly downgraded to operators. Query policies can effectively reduce the number of interactions with the HTTPDNS server while ensuring service availability.

Injection layer

The injection layer relies on NSURLProtocol to intercept network requests on the iOS side. The usage of NSURLProtocol will not be specifically introduced here. Based on NSURLProtocol to intercept network requests, we implemented two schemes respectively. In the case of no need to deal with SNI scenarios, we implemented them based on NSURLSession. In the case of Server Name Indication (SNI), it is implemented based on CFNetwork. Let’s take a look at the two options:

Implementation scheme based on NSURLSession in non-SNI scenario:

The implementation based on NSURLSession is relatively simple. After intercepting the Request through NSURLProtocol, it only needs to replace the domain name in the Request with IP, set the original Host field and Cookie field in the Request header, rebuild the dataTask task, and initiate the Request. Simple example code is as follows:

// Handle url and host dnsResultURL as the URL after the replacement IP address
NSMutableURLRequest *ipRequest = [originRequest mutableCopy];
ipRequest.URL = [NSURL URLWithString:dnsResultURL];
[ipRequest setValue:url.host forHTTPHeaderField:@"Host"];
            
// Handle cookies. Since the URL is changed, the system does not carry cookies under the original domain name
NSString *cookieString = [[MFSNICookieManager sharedManager] requestCookieHeaderForURL:url];
[ipRequest setValue:cookieString forHTTPHeaderField:@"Cookie"];
            
self.ipRequest = ipRequest;
self.clientThread = [NSThread currentThread];
self.ipTask = [[[self class] sharedDemux] dataTaskWithRequest:ipRequest delegate:self modes:self.modes];
                        
if(self.ipTask){
    [self.ipTask resume];
}Copy the code

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didReceiveChallenge:(NSURLAuthenticationChallenge *)challenge completionHandler:(void(^) (NSURLSessionAuthChallengeDisposition.NSURLCredential *_Nullable))completionHandler {
    if(! challenge) {return;
    }
    NSURLSessionAuthChallengeDisposition disposition = NSURLSessionAuthChallengePerformDefaultHandling;
    NSURLCredential *credential = nil;
    
    // Get the original domain name, host, with the original request
    NSString *host = [[self.originRequest allHTTPHeaderFields] objectForKey:@"Host"];
    if(! host) { host =self.originRequest.URL.host;
    }
    if ([challenge.protectionSpace.authenticationMethod isEqualToString:NSURLAuthenticationMethodServerTrust]) {
        if ([self evaluateServerTrust:challenge.protectionSpace.serverTrust forDomain:host]) {
            disposition = NSURLSessionAuthChallengeUseCredential;
            credential = [NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust];
        } else {
            disposition = NSURLSessionAuthChallengePerformDefaultHandling; }}else {
        disposition = NSURLSessionAuthChallengePerformDefaultHandling;
    }
    // Use the default validation scheme directly for other challenges
    completionHandler(disposition, credential);
}


- (BOOL)evaluateServerTrust:(SecTrustRef)serverTrust forDomain:(NSString *)domain {
    // Create a certificate policy
    NSMutableArray *policies = [NSMutableArray array];
    if (domain) {
        [policies addObject:(__bridge_transfer id) SecPolicyCreateSSL(true, (__bridge CFStringRef) domain)];
    } else {
        [policies addObject:(__bridge_transfer id) SecPolicyCreateBasicX509()];
    }
    // Bind the verification policy to the server certificate
    SecTrustSetPolicies(serverTrust, (__bridge CFArrayRef) policies);
    / * * will assess whether or not the current serverTrust trusted, * official suggested that in the case of the result = kSecTrustResultUnspecified or kSecTrustResultProceed * serverTrust can be verified through, https://developer.apple.com/library/ios/technotes/tn2232/_index.html * about detailed information, please refer to the SecTrust SecTrustResultType h * /
    SecTrustResultType result;
    #pragma clang diagnostic push
    #pragma clang diagnostic ignored "-Wdeprecated-declarations"
    SecTrustEvaluate(serverTrust, &result);
    #pragma clang diagnostic pop
    return (result == kSecTrustResultUnspecified || result == kSecTrustResultProceed);
}Copy the code

 // Set SNI host information
NSString *host = [self.sniRequest.allHTTPHeaderFields objectForKey:@"Host"];
if(! host) { host =self.originalRequest.URL.host;
}
[self.inputStream setProperty:NSStreamSocketSecurityLevelNegotiatedSSL forKey:NSStreamSocketSecurityLevelKey];
NSDictionary *sslProperties = @{ (__bridge id) kCFStreamSSLPeerName : host };
[self.inputStream setProperty:sslProperties forKey:(__bridge_transfer NSString *) kCFStreamPropertySSLSettings];Copy the code

In addition to setting SNI information, the implementation scheme based on CFNetwork also needs to consider the problem of data codec, which is rarely mentioned in numerous open source codes and articles we see. Therefore, we need to add codes similar to the following to decode response data when processing response data:

// Check 'content-encoding' to see if the data needs to be decoded;
// Only gZIP decoding is performed here. If the service scenario has other encoding formats, you need to expand them by yourself.
NSString *contentEncoding = [self.response.headerFields objectForKey:@"Content-Encoding"];
if (contentEncoding && [contentEncoding isEqualToString:@"gzip"{[])self.delegate task:self didReceiveData:[self ungzipData:self.resultData]];
} else{[self.delegate task:self didReceiveData:self.resultData];
}Copy the code

In addition, it is also very important that the implementation scheme based on CFNetwork needs to consider the problem of connection reuse, which cannot be recreated every time the request is made, and the cost of reconnection is very high. This is also something that we never talk about when we look at the open source code and articles, and if it’s not addressed here, the performance drain is very serious.

Especially since most of our requests are now HTTP2.0, the performance comparison is even more obvious. However, since Apple’s CFNetwork framework does not support HTTP2.0, it is difficult to implement the relevant features of HTTP2.0 based on CFNetwork. We currently implement the HTTP1.1 protocol connection reuse this part of the function, do not need to re-establish the connection with each request.

For requests with the same host, port, and scheme, if there are available connections that can be reused when the request is initiated, you do not need to re-establish the connection, but directly reuse the connection. If the connection expires locally or the server closes the connection through the response header, the connection will not be reused and the connection will be closed. You can determine whether the server Connection is multiplexed by checking whether the Connection of the response header is keep-alive or close.

Basic service layer

At this stage, the basic service layer mainly relies on Tencent Cloud HTTPDNS SDK to provide basic query services, mainly providing ttL-based cache storage and expiration processing logic. At the same time, this layer also provides SDK’s internal cache storage and log and basic verification functions.

Summary of best practices

Therefore, if you have high performance requirements and need to handle SNI scenarios, I recommend not using HTTPDNS directly and actively. Instead, in the event that the IP request obtained by carrier LocalDNS fails, you can use cfNetwork-based network request directly and retry underneath. In this way, a balance can be achieved between request DNS hijacking and performance, which can guarantee the success rate and availability of HTTPDNS in the case of LocalDNS resolution problems of the carrier. At the same time, it can use nSURlSession-based requests when the carrier’s LocalDNS is available, and enjoy the performance improvement brought by the HTTP2.0 feature implemented by the system.

If you don’t need to deal with SNI, use an nSURLSession-based implementation.

Five, the revenue

Since its launch, DNS optimization has achieved obvious optimization effect, and the overall interface error rate has decreased by more than 20%. At the same time, in the case of simulated DNS hijacking, the core functions of APP can be used normally.

Six, the concluding

DNS optimization is a continuous thing, based on the present situation and the problems we have adopted the optimization scheme, this scheme is not necessarily a perfect solution at present, could still exist certain problems, in the design for the expansion of the subsequent iteration has kept good scalability, we will in the present scheme based on constant optimization and evolution.

Finally, thank you for your hard reading. I hope I can help you a little bit. Thank you very much.

Vii. Reference materials

• Tencent Cloud HTTPDNS SDK documentation