A simple study of NSURLProtocol in iOS

In order to replace the domain name with IP after completing the HTTPDNS service, the way to use is to transform the network library, that is, directly transform the content of NSURLRequest in the network library layer, and send THE HTTP request through IP direct connection.

Specific problems to be solved are as follows:

  1. The domain in the URL changes IP addresses
  2. In NSURLRequesthttp headerTo bind the original domain name to the Host field of
  3. In NSURLRequesthttp headerAdd the Cookie field in the URL host field changed to IP, system layer will not help you inhttp request headerCookie information is added to the
  4. In NSURLSessionAuthManager ChallengeCallback forchallenge.protectionSpace.serverTrustYes For HTTPS certificate authentication,challenge.protectionSpace.hostYou need to obtain the original domain name to perform security authentication with the HTTPS certificate.

Such direct transformation at the network layer will have the following problems:

  1. Use Apple’s NSURLSession when the use of domain name request optimization, specific optimization methods can refer tolibcurlandhappy eyeballAlgorithm.
  2. The intrusion to the existing network library related code is obvious.

Note:

1: The underlying network request of NSURLSession is wrapped with libcurl

The OkHttp network library can be used in Android to replace the LocalDNS service directly at the bottom level, and the service can support a number of lists of IP addresses, and OkHttp is used for policy polling retry on multiple IP addresses!!

IP direct connection scheme of user-defined NSURLProtocol

In iOS, NSURLProtocol plays an important role in URL Loading System. It acts as a middleman. For the service layer network library or NSURLSession, this NSURLProtocol is a server. The NSURLProtocol calls back the information of the key process in the network request to the NSURLSession in the business layer through the client property.

I am studying the general usage of NSURLProtocol in some open source libraries on the current website, which can be roughly divided into three categories:

  1. Cache UIWebView or WKWebview
  2. Network monitoring and network Mock, such as DebugTool, Netfox, CocoaDebug, OHHTTPStubs, DoraemonKit, etc
  3. Underlying IP direct connection services such as Tencent Cloud and Ali Cloud about HTTPDNS best practices, as well as open source library KIDDNS

In the above content, the basic is the reference Apple official demo-CustomHttpProtocol implementation, which is the most key to the implementation of several key points as follows:

  1. NSURLProtocol’s self. Client API must be invoked in the client Thread, so the Apple Demo will cache the client Thread and runloopMode in startLoading. In the underlying real network request, through the following methods for data exchange with the business layer, and the lack of progress related API NSURLSessionDownloadDelegate callback, so using NSURLProtocol intercepting network request, When forwarding data by itself, the progress update of upstream and downstream data cannot be completed through the official API. In addition, for the following client callback methods, Apple officially divides them into three categories :pre-response, Response, and post-response, and explains the timing of each method invocation. Please refer to the ReadMe of the project

    1. -URLProtocol:wasRedirectedToRequest:redirectResponse:
    2. -URLProtocol:didReceiveResponse:cacheStoragePolicy:
    3. -URLProtocol:didLoadData:
    4. -URLProtocolDidFinishLoading:
    5. -URLProtocol:didFailWithError:
    6. -URLProtocol:didReceiveAuthenticationChallenge:
    7. -URLProtocol:didCancelAuthenticationChallenge:
  1. Demo builds a singleton patternQNSURLSessionDemuxCome forward as a point in the construction of the Request and then unified processing key NSURLSessionDelegate and NSURLSessionDataDelegate forwarding the Request. But we can see that some of the actual methods are not implemented. Apple’s implementation approach is recommended here.
  2. Demo built a NSURLProtocol Delegate, let the Delegate to implement the AuthManger Challenge logic, also worth learning.

In the implementation of some open source network monitoring modules, each NSURLProtocol creates an associated NSURLSession, and then uses this NSURLSession to forward requests. For specific reasons, please refer to the implementation of NSURLSession in Swift-Foundation. The bottom layer is multiHandle in curl, and the NSURLSession is strongly referenced with the delegate. If multiple requests are intercepted at the same time, the memory usage will be high.

Refer to this Demo, it is relatively easy to implement the interception of Request Request in startLoading, change it into IP, and then carry out IP direct connection service, but there will be some problems as follows:

1. The intercepting NSURLRequest HTTPBody data is too large and lost

To convert an HTTPBody to an HTTPStream, do the following:

+ (NSURLRequest *)canonicalRequestForRequest:(NSURLRequest *)request { NSURLRequest *newRequest = [self handleMutablePostRequestIncludeBody:[request mutableCopy]]; return newRequest; } + (NSMutableURLRequest *)handleMutablePostRequestIncludeBody:(NSMutableURLRequest *)req { if ([req.HTTPMethod isEqualToString:@"POST"]) { if (! req.HTTPBody) { NSInteger maxLength = 1024; uint8_t d[maxLength]; NSInputStream *stream = req.HTTPBodyStream; NSMutableData *data = [[NSMutableData alloc] init]; [stream open]; BOOL endOfStreamReached = NO; // Do not use [stream hasBytesAvailable]). When processing an image file, [stream hasBytesAvailable] will always return YES, resulting in an infinite loop in while. while (! endOfStreamReached) { NSInteger bytesRead = [stream read:d maxLength:maxLength]; If (bytesRead == 0) {// endOfStreamReached = YES; } else if (bytesRead == -1) {endOfStreamReached = YES; } else if (stream.streamError == nil) { [data appendBytes:(void *)d length:bytesRead]; } } req.HTTPBody = [data copy]; [stream close]; } } return req; }Copy the code

2. The intercepted request requires the Pin certificate during the TLS handshake

At present, there are two solutions. For the first solution, you can refer to the implementation of Apple Demo to configure a Delegate to complete certificate verification. However, note that the logic related to certificate verification needs to be implemented in the Delegate method. Not the upper-level AFNetworking callback method!!

At present, there is another scheme that can refer to KIDDNS, which transfers the specific certificate verification method to the original originalSession delegate, similar to the proxy method. The specific steps are as follows:

  1. HOOK NSURLSession to create a SessionTask in order to get the task created and the session that created the task

  2. Maintain a global URLSessionMap to cache tasks and sessions created in hooks

  3. When the NSURLProtocol is created, an originalTask is cached so that the session can be obtained from the WBURLSessionMap

  4. When demux receives the Auth challenge, it obtains the originalSession of the originalTask directly from the WBURLSessionMap. And then directly in originalSession. Call originalSession in delegateQueue. Delegate callback methods

    @interface WBHTTPURLProtocol()<NSURLSessionTaskDelegate, NSURLSessionDataDelegate, NSURLSessionDownloadDelegate> ... @property (nonatomic, strong) NSURLSessionTask *originalTask; // Cache a copy of the Task @end used by the business layer // When the upper layer initializes the Task, Cache an originalTask - (instancetype)initWithTask:(NSURLSessionTask *)task cachedResponse:(NSCachedURLResponse) *)cachedResponse client:(id<NSURLProtocolClient>)client { self.originalTask = task; if (self = [super initWithTask:task cachedResponse:cachedResponse client:client]) { } return self; } // QNSURLSessionDemux callback -(void)URLSession (NSURLSession *)session Task (NSURLSessionTask *)task didReceiveChallenge:(NSURLAuthenticationChallenge *)challenge completionHandler:(void (^)(NSURLSessionAuthChallengeDisposition, NSURLCredential * _Nullable))completionHandler{ if (task == self.originalTask) { return; } NSURLSession *originalSession = [WBURLSessionMap fetchSessionOfTask:self.originalTask]; if (originalSession.delegate && [originalSession.delegate respondsToSelector:@selector(URLSession:task:didReceiveChallenge:completionHandler:)]) { [originalSession.delegateQueue addOperationWithBlock:^{ [(id<NSURLSessionTaskDelegate>)originalSession.delegate URLSession:originalSession task:task didReceiveChallenge:challenge completionHandler:completionHandler]; }]; }}Copy the code
    @implementation NSURLSession (WBURLProtocol) + (void)load { [self swizzleMethods]; } + (void)swizzleMethods { NSArray<NSString *> *selectors = @[@"dataTaskWithRequest:", @"dataTaskWithURL:",@"uploadTaskWithRequest:fromFile:",@"uploadTaskWithRequest:fromData:",@"uploadTaskWithStreamedReques t:",@"downloadTaskWithRequest:",@"downloadTaskWithURL:",@"downloadTaskWithResumeData:"]; [selectors enumerateObjectsUsingBlock:^(NSString * _Nonnull obj, NSUInteger idx, BOOL * _Nonnull stop) { Method originalMethod = class_getInstanceMethod([NSURLSession class],NSSelectorFromString(obj));  NSString *fakeSelector = [NSString stringWithFormat:@"fake_%@", obj]; Method fakeMethod = class_getInstanceMethod([NSURLSession class], NSSelectorFromString(fakeSelector)); method_exchangeImplementations(originalMethod, fakeMethod); }]; } - (NSURLSessionDataTask *)fake_dataTaskWithRequest:(NSURLRequest *)request { NSURLSessionDataTask *task = [self fake_dataTaskWithRequest:request]; [WBURLSessionMapURLSessionMap recordSessionTask:task ofSession:self]; return task; }... Other task creation methods @endCopy the code
    @interface WBURLSessionMap()
@property (nonatomic, strong) NSMapTable *map;
@property (nonatomic, strong) dispatch_queue_t queue;
​
@end
​
@implementation WBURLSessionMap
+ (instancetype)sharedInstance {
    static WBURLSessionMap *instance = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        instance = [WBURLSessionMap new];
    });
    return instance;
}
​
- (instancetype)init {
    if (self = [super init]) {
        _map = [NSMapTable weakToWeakObjectsMapTable];
        _queue = dispatch_queue_create("com.xxx.urlprotocol.sessionmap", DISPATCH_QUEUE_CONCURRENT);
    }
    return self;
}
​
- (NSURLSession *)fetchSessionOfTask:(NSURLSessionTask *)task {
    __block NSURLSession *session = nil;
    dispatch_sync(_queue, ^{
        session = [self->_map objectForKey:task];
    });
    return session;
}
​
- (void)recordSessionTask:(NSURLSessionTask *)task ofSession:(NSURLSession *)session {
    dispatch_barrier_async(_queue, ^{
        [self->_map setObject:session forKey:task];
    });
}
​
+ (NSURLSession *)fetchSessionOfTask:(NSURLSessionTask *)task {
    return [[self sharedInstance] fetchSessionOfTask:task];
}
​
+ (void)recordSessionTask:(NSURLSessionTask *)task ofSession:(NSURLSession *)session {
    [[self sharedInstance] recordSessionTask:task ofSession:session];
}
@end
Copy the code

    But!!!!!! This implementation is dangerous and requires strict attention to the implementation of the upper layer code, and according to the official apple demo, the completionHandler in the didReceiveChallenge callback needs to be called in the client thread, which the implementation does not comply with. Can make a package here!!

    Therefore, it is recommended to implement the LOGIC such as the Pin certificate in the delegate, using the official Apple Demo method.

    This list is just for the sake of this idea. In the future, it can be expanded to explore the idea of instrumentation such as full proxy method hook NSURLSession and all methods of Task

    3. The progress of upload and download is abnormal

    It is not possible to monitor the progress of upload and download using NSURLProtocol.

    Similarly, there is no way for your NSURLProtocol subclass to call the NSURLConnection delegate's -connection:needNewBodyStream: or -connection:didSendBodyData:totalBytesWritten:totalBytesExpectedToWrite: methods (<rdar://problem/9226155> and <rdar://problem/9226157>). The latter is not a serious concern--it just means that  your clients don't get upload progress--but the former is a real issue.Copy the code

    4. The timing of NSURLProtocol activation is incorrect

    Because using the current basic NSULRSession, there are many online direct Hook system NSURLSessionConfiguration method, invasive, for ordinary NSURLSession or AFNetworking in class, We can use the following way to activate our custom NSURLProtocol, note that because NSURLSessionConfiguration default will separate protocols suggest some system don’t replace directly, but use the following way, So if our custom WBHTTPURLProtocol does not take effect, NSURLSession will use the system’s default URLProtocol to maintain compatibility:

    NSURLSessionConfiguration *config = [NSURLSessionConfiguration defaultSessionConfiguration];
Class urlprotocol = NSClassFromString(@"WBHTTPURLProtocol");
if (urlprotocol) {
  NSMutableArray *protocols = [config.protocolClasses mutableCopy];
  [protocols insertObject:urlprotocol atIndex:0];
  config.protocolClasses = protocols;
}
AFURLSessionManager *sessionManager = [[AFURLSessionManager alloc] initWithSessionConfiguration:config];
Copy the code

    System default implementation of URLProtocol _NativeProtocol, HTTPURLProtocol and so on. See Swift-core-foundation

    Use libcurl to forward the Request in NSURLProtocol

    The above detailed reference is made to URLSession forwarding Request in Apple Demo, but it does not solve the multi-IP problem similar to Android OkHttp. Someone on the Internet has used libcurl to wrap YMHTTP, an AFNetworking class, in the copybook of Swift-core-Foundation. If there is a need to replace the underlying use of NSURLSession, the IP direct connection service can not directly replace the DNS module.

    SNI problem in HTTPDNS

    If you have SNI issues in your business, use libcurl instead. Some people on the Internet use CFNetwork implementation, inefficient not recommended.