One of the biggest headaches in iOS development is debugging network requests. Whether it’s a back-end interface problem or a parameter structure problem, you always need a network debugging tool to simplify the debugging process.

The status quo

Outside the App debugging

A lot of network debugging in the early days was done by debugging outside the App, which has the advantage of completely not affecting any logic inside the App, and also not considering the possible impact on the network layer.

  • CharlesIt is really the first choice for network debugging, it supports emulators, real machine debugging, and attachedmap remoteandmap local“, can be said to be the mainstream debugging tool in iOS development, but the disadvantages are also obvious, iPhone and Mac must be under the same Wi-Fi when using, and need to set the Proxy corresponding to Wi-Fi when using, and once the Charles on the computer is turned off, the mobile phone will not be connected to the network. It’s great in the office, but once you leave the office, you can’t use it.
  • SurgeIt is also a good network debugging tool in recent years. After setting the certificate in iOS version, you can directly see the requests of all apps, while the Remote Dashboard provided by Mac version can increase the efficiency of viewing network requests. The new TF version has also been addedrewriteAs well asscript“, which basically meets most of Charles’ common needs, and can be done independently of the Mac. However, this method also has some problems, that is, every time to check the network request, you need to switch App, and the request is issued by all applications, and it is difficult to only look at the request of one application (in fact, it is also caused by the Filter is not detailed enough).

App in debug

There are already a number of network debugging frameworks on GitHub that provide simple in-app collection of network requests.

  • GodEye provides a complete set of network request monitoring features, but has not been updated since, and affects in-app requests (more on this later). It is only for debugging, not for online debugging.
  • BagelThe implementation of this has little effect on in-app requests, but it must be used in Mac applications, and for implementation reasons, if the in-app custom is usedURLProtocol, will make the network request fetching repeat. The above two kinds of debugging methods have their own advantages and disadvantages. Out-of-app debugging is often not targeted at a particular application, leading to a very general query experience. Most network debugging frameworks on Github are basically similar to these two principles, and the implementation of these debugging tools is mainly used in the Debug environment. For many network monitoring requirements are very low, for exampleGodEyeThis will significantly affect the existing network request, although the effect is very small, in the debugging environment also can accept, basic can accomplish the purpose, but once we hope in the online debugging (including testflight) environment, also can let a all the requests have affected the risk of (specific risk will talk about later).

The principle of network debugging

In order to solve the above problems, we decided to start with the existing in-app debugging scheme and start to optimize some details to achieve the goal of online debugging without affecting network requests. Below I first introduce the principle of several current mainstream network debugging schemes.

URL Loading System URL Protocol

When entering iOS, many people will send network requests through third-party network request libraries such as Alamofire. However, most network request libraries are based on the encapsulation of URLConnection or URLSession in the standard library, among which URLConnection is the old encapsulation. However, URLSession is a relatively new and recommended encapsulation. It processes a series of events such as URL loading and response, including the modification of the so-called transport protocol. The standard library provides basic URL transport protocols, including HTTP, HTTPS, FTP, etc. Of course, If we have our own protocol to deal with, the library provides the means to do so.

In the standard library, there is a URLProtocol class, from the name we know that it handles the protocol in the URL loading, so define the corresponding class, also have a way to let the standard library to use the custom protocol, we can change an array of URLProtocol to achieve this goal.

  • inURLConnectionIn, there’s going to be oneURLProtocolThe class variable represents thisURLProtocolThe array that we can pass throughregisterClassTo insert our own protocol into the array
  • inURLSessionWe can directly modify the array in the Configuration to insert our own protocol in the standard library. Whenever there is a network request, the system will from the corresponding arrayIn order to askeachURLProtocolClass can handle the current request
open class func canInit(with request: URLRequest) - >Bool
Copy the code

When encountering a class that can return true, the system calls the initialization method of the corresponding class to initialize an instance of the current class, and the new instance handles the sending, receiving, and callback of the request. The basic protocols provided by the system, such as HTTP and HTTPS, Are implemented by classes that exist by default in the URLProtocol array, so if we want to handle it ourselves, we need to insert our own protocol at the front of the array to ensure that we are asked whether we can handle the network request first.

Therefore, we can inherit URLProtocol and implement relevant methods to deal with various events after network sending and receiving as the middle layer. URLProtocol has the ability to change every link in the URL loading process, but it also needs to call the original response method. In this way, the protocol processing does not affect the way of the network call and the network response, and the network request sender does the intermediate processing without being aware of it.

It is this “stealth” feature that makes URLProtocol the preferred option for many network debugging frameworks, such as hookURLSession or URLSessionConfiguration initialization methods, Insert a custom network debugging Protocol into the Configuration of the URLSession. Then all network requests will be sent through this Protocol. In this Protocol, the request will be sent through the normal URLSession. After receiving the callback from the network request and calling back to the delegate of the original network request, you can get all the callback from the request and record it without affecting the original request.

GodEye starts with this method, but it sends requests internally using the old URLConnection instead of URLSession. However, this does not matter. The implementation of this method is almost the same

  1. Hook using Objc’s runtimeURLSession.init(configuration:delegate:delegateQueue:)Method, and then before calling the original initialization method, inURLSessionConfigurationInsert our customURLProtocol, while callingURLProtocolClass method underregisterClassTo register custom classes.
  2. In customURLProtocolImplementation in subclasses
    • canInit(with:)Method, in which to determine whether the network request needs to be monitored, if not, can directly permit
    • canonicalRequest(for:)Method, we usually do some processing on the original request, such as adding a flag to indicate that the request has already been processed
    • startLoading()Method, we need to send out the corresponding request, usually with a new oneURLSessionSend the request again, and set the new delegate to itself, so that the callback for the new request is replaced by the current oneURLProtocolTo deal with
    • stopLoadingMethod, we are responsible for stopping the request
  3. Also, in customURLProtocolThe callback that implements the new request mentioned above is passed in the callbackself.client.urlProtocolReturns the callback back to the original delegate
  4. So far, we finished the sending, receiving and a series of operations, and perfect will back turned back to the original agent, the rest is our gathering network request all kinds of information in the callback This method looks very perfect, through the chart to show the following (the above is the original process, the following is the new process)

This is where network monitoring stops for many apps. However, these apps usually debug only in debug mode because it doesn’t cause a lot of problems. However, we can’t require all back-end development to have a so-called debug version installed. A few minor problems with the scheme would be significant

  • First of all, normally there might only be one or two appsURLSessionIs now sending a request and getting a new oneURLSessionThis has some potential performance risks of its own, but it’s not because people don’t want to reuse the so-calledURLSessionInstead, as we explained above, the system initializes one for each requestURLProtocolEach instance handles its own callback, and in theURLProtocolCan’t get the originalURLSessionSo people don’t want to spend time onURLSessionAfter all, many apps may only enable this feature during debugging
  • Secondly, in theURLProtocolIn, we initialize the new one each timeURLSessionAll use the default configuration, including timeout, cache and other Settings are the same as the originalURLSessionDifferent, this can cause some performance not to meet expectations

Both of these are unacceptable for the online environment, so this scheme basically does not meet our requirements.

To solve the above problem, we need to introduce a method of URLSession reuse, that is, there needs to be a manager, to manage all URlSessions, and distribute their respective network request callback, call back the corresponding URLProtocol instance. In a reading of apple’s official URLProtocol example, I found some design ideas in this example to help us solve this problem, including the concept of Demux.

As we said before, we create a new URLSession instance every time we send a request, because if we only use URLProtocol, it’s very difficult to get the URLSession from the context, and we don’t do any reuse, because the original method, We make the DELEGATE of the URLSession be the current URLProtocol, and you can’t change the delegate of the session, so we do this for convenience, and Demux actually does a lot of complicated things, save the so-called URLSession and reuse it, So now that you’re reusing the delegate, the other thing Demux does is to aggregate the delegate and forward it out.

Demux will generate a new URLSession for each different original URLSession. Demux will record the ID of the current request and process the callback uniformly. During the callback, Demux will use this ID to find the corresponding URLProtocol and execute the callback. This perfectly solves the first problem above, and the diagram below shows how Demux works and flows.

In terms of implementation, when we introduced Demux, we did not have the problem of multiple URlsessions. However, in terms of implementation, it seems not so easy to get the configuration of the original URLSession. First of all, The URLProtocol itself can’t get the original URLSession, because from the design of the interface, it can only get the corresponding URLRequest to process the original request, and it can’t do anything more. Me through the apple the swift in the standard library open source of URLProtocol reading, found in the request, in fact, the standard library calls initWithTask: cachedResponse: client: will the corresponding URLSessionTask preach in the past, only is private property, We couldn’t access it, but it still inspired me. Our final solution was to write our own BaseLoggerurlProtocol by inheriting URLProtocol, then override the initialization method and save the incoming task. In this way, we can get the task corresponding to the request in the URLProtocol, and then get the original URLSession through the task. In this way, we can perfectly initialize the new URLSession through the original configuration, and solve the above two problems. This is the network monitoring method currently used in Jike. The following are some core functions to achieve the code.

#pragma mark - Base Url Protocol
@interface BaseLoggerURLProtocol : NSURLProtocol
@property (atomic, copy.readwrite) NSURLSessionTask * originTask;
@end

@implementation BaseLoggerURLProtocol : NSURLProtocol
- (instancetype)initWithTask:(NSURLSessionTask *)task cachedResponse:(NSCachedURLResponse *)cachedResponse client:(id<NSURLProtocolClient>)client {
    self.originTask = task;
    self = [super initWithRequest:task.originalRequest cachedResponse:cachedResponse client:client];
    return self;
}
@end
Copy the code
// MARK: - Logger Demux class LoggerURLSessionDemux: NSObject { public private(set) var configuration: URLSessionConfiguration! public private(set) var session: URLSession! private var taskInfoByTaskId: [Int: TaskInfo] = [:] private var sessionDelegateQueue: OperationQueue = OperationQueue() public init(configuration: URLSessionConfiguration) { super.init() self.configuration = (configuration.copy() as! URLSessionConfiguration) sessionDelegateQueue. MaxConcurrentOperationCount = 1 sessionDelegateQueue. Name = "com. Jike..." self.session = URLSession(configuration: self.configuration, delegate: self, delegateQueue: self.sessionDelegateQueue) self.session.sessionDescription = self.identifier } }Copy the code
// MARK: - Demux Manager
class LoggerURLDemuxManager {
    static let shared = LoggerURLDemuxManager(a)private var demuxBySessionHashValue: [Int: LoggerURLSessionDemux] = [:]

    func demux(for session: URLSession) -> LoggerURLSessionDemux {

        objc_sync_enter(self)
        let demux = demuxBySessionHashValue[session.hashValue]
        objc_sync_exit(self)

        if let demux = demux {
            return demux
        }

        let newDemux = LoggerURLSessionDemux(configuration: session.configuration)
        objc_sync_enter(self)
        demuxBySessionHashValue[session.hashValue] = newDemux
        objc_sync_exit(self)
        return newDemux
    }
}
Copy the code
// MARK: - Url Protocol Start Loading
public class LoggerURLProtocol: BaseLoggerURLProtocol {
override open func startLoading(a) {
        guard let originTask = originTask,
            letThe session = originTask. Value (forKey: "session")as? URLSession else {
            // We must get the session for using demux.client? .urlProtocol(self, didFailWithError: LoggerError.cantGetSessionFromTask)
            // Release the task
            self.originTask = nil
            return
        }
        // Release the task
        self.originTask = nil

        let demux = LoggerURLDemuxManager.shared.demux(for: session)

        var runLoopModes: [RunLoop.Mode] = [RunLoop.Mode.default]
        if let currentMode = RunLoop.current.currentMode, currentMode ! =RunLoop.Mode.default {
            runLoopModes.append(currentMode)
        }

        self.thread = Thread.current
        self.modes = runLoopModes.map{$0.rawValue }

        let recursiveRequest = (self.request as NSURLRequest).mutableCopy() as! NSMutableURLRequest
        LoggerURLProtocol.setProperty(true, forKey: LoggerURLProtocol.kOurRecursiveRequestFlagProperty, in: recursiveRequest)

        self.customTask = demux.dataTask(with: recursiveRequest as URLRequest, delegate: self, modes: runLoopModes)

        self.customTask? .resume()let networkLog = NetworkLog(request: request)
        self.networkLog = networkLog

        RGLogger.networkLogCreationSubject.onNext(networkLog)
    }
}
Copy the code

The new plan

The solutions mentioned above solved most of the problems of the traditional solutions and were also used in our app development stage, but we encountered new problems

Problem with the scheme

We mentioned above, according to the traditional scheme, made some improvements, avoids the problem of most of the traditional scheme, but there is one point is we still cannot avoid, then we still request sent to a network, rather than directly to the monitoring of the original network request, how to send the original request, We have to send it intact, otherwise if we send the wrong network request, we will receive the wrong response or even not receive the response, directly resulting in the application function damage, which is the problem that this scheme will have from the beginning.

Just because of this problem, we also encountered the biggest challenge of network monitoring this time, that is, unusual requests. Since Alamofire is used in our APP to make network requests, and it is uploading MultipartFormData, if the amount of data is too large, Then there will be a mechanism to put data in a temporary directory and Upload data through the Upload File. The specific mechanism can be seen in the logic of Alamofire source code.

However, our customized URLProtocol can only get the corresponding URLRequest directly. However, when uploading the Upload File, Through which we can’t simply get to upload data, so we pass this URLRequest request, only an empty body, and do not upload the real data, cause the failure of image upload, this also directly affects the function of the app, and then we can only through the way of monitoring request to upload pictures to bypass this problem.

Tackle the problem at its root

From this point of view, both the traditional scheme and our improved scheme will definitely re-send the network request. As long as we can’t perfectly send the original request, the scheme is not perfect, that is to say, the road of URLProtocol can’t go on.

This also tells us that we need to find a way to get all the network request callbacks without affecting the original network requests. In working with RxSwift, I came across an interesting concept called DelegateProxy, which can generate a proxy, set the proxy to the original delegate, and then forward all of the called methods to the original delegate, In this way, you can get all the callbacks as an intermediate layer without affecting the original processing, and in RxCocoa under RxSwift, this set of techniques has been used on various UI components that we normally call

tableView.rx.contentOffset.subscribe(on: { event in })
Copy the code

It’s the simplest example of how you can get a callback without affecting the tableView’s delegate.

With that in mind, I’m going to implement a set of DelegateProxy of URLSessionDelegate that doesn’t affect the original network request and still gets all the callbacks back to the original delegate. So I implemented a basic Delegate Proxy

public final class URLSessionDelegateProxy: NSObject {
    private var networkLogs: [Int: JKLogger.NetworkLog] = [:]
    var _forwardTo: URLSessionDelegate?

    // MARK: - Initialize
    @objc public init(forwardToDelegate delegate: URLSessionDelegate) {
        self._forwardTo = delegate
        super.init()}// MARK: - Responder
    override public func responds(to aSelector: Selector!) -> Bool {
        return_forwardTo? .responds(to: aSelector) ??false}}Copy the code

And then implement the corresponding URLSessionDelegate method, and call the corresponding method of _forwardTo, pass the callback back to the original callback, and then what we’re going to do is, Is the initialization method to hook off URLSession sessionWithConfiguration: delegate: delegateQueue:, then use our own DelegateProxy incoming delegate is initialized, Then set the new delegate back, as follows

// MARK: - URLSessionDataDelegate
extension JKLogger.URLSessionDelegateProxy: URLSessionDataDelegate {
    var _forwardToDataDelegate: URLSessionDataDelegate? { return _forwardTo as? URLSessionDataDelegate }

    public func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive response: URLResponse, completionHandler: @escaping (URLSession.ResponseDisposition) -> Void) { _forwardToDataDelegate? .urlSession? (session, dataTask: dataTask, didReceive: response, completionHandler: completionHandler) } }Copy the code

This gives us the desired effect and perfectly avoids the problem of resending the request in the previous method.

An interlude

After using the latest solution above for a period of time, there was basically no problem. However, when we used ReactNative, we encountered a problem. This solution would cause the APP to fail to connect to RN and load corresponding pages. In a class of RN RCTMultipartDataTask, it illustrates their follow NSURLSessionDataDelegate agreement in a statement, but realized NSURLSessionStreamDelegate method in the implementation, therefore, We use callback in our own DelegateProxy

_forwardTo as? URLSessionStreamDelegate // always failed
Copy the code

There is no real value for callbacks in the standard library, but it still depends on the runtime value of objC. Selector), so the library can call the corresponding method in RCTMultipartDataTask, but we can’t call that method directly in swift code, which causes RCTMultipartDataTask to receive one less callback, Not being able to work is normal. Although the ReactNative method is confusing and is not recommended, since we want to make a perfect network monitoring solution, we should keep the standard library approach and do callbacks through objC rather than through simple SWIFT AS conversion.

This sounds pretty simple. After all, for an ObjC with a powerful runtime, it’s pretty simple to call a method dynamically. The first thing that comes to mind is performSelector. After comparing NSInvocation solutions, we chose objc_msgSend, which is safe to use as long as we make a good judgment

H # import "_JKSessionDelegateProxy."
#import <objc/runtime.h>
#import <objc/message.h>
#define JKMakeSureRespodsTo(object, sel) if (! [object respondsToSelector:sel]) { return ; }

@interface _JKSessionDelegateProxy()"NSURLSessionDelegate.NSURLSessionTaskDelegate.NSURLSessionDataDelegate.NSURLSessionStreamDelegate._JKNetworkLogUpdateDelegate>
@end
@implementation _JKSessionDelegateProxy
- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didSendBodyData:(int64_t)bytesSent totalBytesSent:(int64_t)totalBytesSent totalBytesExpectedToSend:(int64_t)totalBytesExpectedToSend {
    JKMakeSureRespodsTo(self.forwardTo, _cmd);
    ((void(*) (id, SEL, NSURLSession*, NSURLSessionTask*, int64_t, int64_t, int64_t))objc_msgSend)(self.forwardTo, _cmd, session, task, bytesSent, totalBytesSent, totalBytesExpectedToSend);
}
@end
Copy the code

The code above also shows one of many callbacks, so you just need to complete all the callbacks in the corresponding way.

Above is my after multiple framework of contrast, and repeated practice to get the best solution, it can solve the traditional scheme of network need to send the request of the Achilles’ heel, also can not affect any network request, under the condition of monitoring to all network request within the app, basic reached us for debugging or online environment, Can be perfect for network debugging tool requirements.

After completing the debugging mentioned above, we only need to provide the displayed UI in the app, and it can be displayed as shown in the picture below, debugging in the app.

Jike App can now be updated and downloaded in all major App markets. Welcome home! Thank you for your patience. I hope you can spread the good news to your friends and let more people come back to Jike town as soon as possible. Click on the download