Since the rider cannot be in WIFI state at any time, traffic has become a very sensitive issue. In order to accurately reach the traffic of each API and carry out targeted optimization, we began to add traffic monitoring function in our APM.

This article will document your own summary of traffic monitoring. It included a lot of experience in trampling pits and defect analysis of some existing schemes, which was a very meaningful process for me.

Dry cargo alert 🤣 Be prepared to read a lot of code 😂😂😂

I. Data collection

At present, each big factory basically has its own APM (our company also had one before, but due to the different needs of each business division, it cannot fully meet the needs of the logistics platform), but there are not many open source APM projects of each big factory. Of course, it may also be due to different business scenarios and different subsequent processing of data.

So this time in the information stage, there is not too much source optional reference, but there are many articles.

Here are some of the articles and open source libraries referenced during the development process:

  1. iOS-Monitor-Platform
  2. GodEye
  3. NetworkEye
  4. Mobile terminal performance monitoring scheme Hertz
  5. Some issues to note with NSURLProtocol
  6. NSURLProtocol is used in iOS development to intercept HTTP requests
  7. Get the HTTPVersion of NSURLResponse

However, all the above materials have shortcomings for our needs:

1. Record Request and Response in the same record

In actual network requests, Request and Response may not be paired. If the network is disconnected or the process is suddenly shut down, mismatch will occur. If the Request and Response are recorded in the same data, statistical deviation will be caused

2. The upstream traffic is not accurately recorded

The main reasons fall into three categories:

  1. The Header and Line parts are ignored
  2. Ignore the Cookie part, in fact, bloated cookies are also part of the traffic consumption
  3. The body part’s byte size calculation is used directlyHTTPBody.lengthNot very accurate

3. Downstream traffic is inaccurately recorded

The main reasons are:

  1. The Header and status-line parts are ignored
  2. The body part’s byte size calculation is used directlyexpectedContentLengthNot very accurate
  3. Gzip compression is ignored. In actual network programming, Gzip is often used for data compression, and some monitoring methods provided by the system return NSData that is actually decompressed. If the number of bytes is directly counted, a lot of errors will be caused

More on this later.

Second, the demand for

Let’s start with a brief list of our requirements:

  1. Request Records basic information
  2. The upward flow
  3. Reponse Records basic information
  4. Downward flow
  5. Data classification: According to host and path, one record records the number of Request records, number of Response records, total Reqeust traffic (upstream traffic), and total Reponse traffic (downstream traffic).

We focus on traffic statistics to facilitate the analysis of which apis consume more traffic in APP use. Therefore, the upstream and downstream traffic needs to be recorded as accurately as possible.

The final database table display:

The type field indicates whether the record is Request or Response. Several lengths record various details of traffic, including total bytes, Line bytes, Header bytes, and Body bytes.

The final interface looks something like:

Analysis of existing data

Now analyze the shortcomings of the data collected above.

GodEye | NetworkEye:

NetworkEye is a part of GodEye, which can be used separately from the network monitoring library.

Access to the source code of both found, NetworkEye

  1. Only Reponse traffic is recorded
  2. The record is not accurate with the expectedContentLength (which will be explained later)
  3. It only records the sum, which is meaningless to us, and cannot analyze which API traffic is used more

Mobile terminal performance Monitoring scheme Hertz:

Meituan’s article shows a few code snippets:

- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
    [self.client URLProtocolDidFinishLoading:self];

    self.data = nil;
    if (connection.originalRequest) {
        WMNetworkUsageDataInfo *info = [[WMNetworkUsageDataInfo alloc] init];
        self.connectionEndTime = [[NSDate date] timeIntervalSince1970];
        info.responseSize = self.responseDataLength;
        info.requestSize = connection.originalRequest.HTTPBody.length;
        info.contentType = [WMNetworkUsageURLProtocol getContentTypeByURL:connection.originalRequest.URL andMIMEType:self.MIMEType];
    [[WMNetworkMeter sharedInstance] setLastDataInfo:info];
    [[WMNetworkUsageManager sharedManager] recordNetworkUsageDataInfo:info];
}
Copy the code

Record the whole of the network in the connectionDidFinishLoading request of the end of the time, the response data size, and the size of the request data, and some other data.

In general is more detailed, but it does not give self. ResponseDataLength specific logic, the other connection. OriginalRequest. HTTPBody. Length is only Request the size of the body.

IOS – Monitor – Platform:

This article introduced the whole PROCESS of APM production in detail, posted a lot of code segments, should be said to be very detailed and very reference value.

In the flow section, upstream flow and downstream flow are also distinguished respectively.

  1. Gzip compression is not handled
  2. The Header is resized from Dictionary to NSData, but the Header is not actually in Json format.

Four, do it yourself

The HTTP message

To better understand some key information about HTTP traffic calculation, you must first understand the composition of HTTP packets.

Let’s grab a bag and see:

Network monitoring in iOS

In this part, I adopted the well-known NSURLProtocol, which can intercept all the network requests except those issued by CFNetwork.

NSURLProtocol is described and used in great detail in the Apple documentation

An abstract class that handles the loading of protocol-specific URL data.

If you want to learn more about NSURLProtocol, you can also read this article by daeso

At the beginning of each HTTP request, the URL loading system creates an appropriate NSURLProtocol object to handle the corresponding URL request. All we need to do is write a class that inherits from NSURLProtocol and pass -registerClass: Method registers our protocol class, and the URL loading system uses the protocol object we created to process the request when it is issued.

NSURLProtocol is an abstract class, and the first step is to integrate it to complete our custom Settings.

Create your own DMURLProtocol, add a few attributes to it and implement the associated interface:

@interface DMURLProtocol()"NSURLConnectionDelegate.NSURLConnectionDataDelegate>

@property (nonatomic.strong) NSURLConnection *connection;
@property (nonatomic.strong) NSURLRequest *dm_request;
@property (nonatomic.strong) NSURLResponse *dm_response;
@property (nonatomic.strong) NSMutableData *dm_data;

@end
Copy the code

CanInitWithRequest & canonicalRequestForRequest:

static NSString *const DMHTTP = @"LPDHTTP";
Copy the code
+ (BOOL)canInitWithRequest:(NSURLRequest *)request {
    if(! [request.URL.scheme isEqualToString:@"http"]) {
        return NO;
    }
    // Intercepts are no longer intercepted
    if ([NSURLProtocol propertyForKey:LPDHTTP inRequest:request] ) {
        return NO;
    }
    return YES;
}
Copy the code
+ (NSURLRequest *)canonicalRequestForRequest:(NSURLRequest *)request {
    NSMutableURLRequest *mutableReqeust = [request mutableCopy];
    [NSURLProtocol setProperty:@YES
                        forKey:DMHTTP
                     inRequest:mutableReqeust];
    return [mutableReqeust copy];
}
Copy the code

StartLoading:

- (void)startLoading {
    NSURLRequest *request = [[self class] canonicalRequestForRequest:self.request];
    self.connection = [[NSURLConnection alloc] initWithRequest:request delegate:self startImmediately:YES];
    self.dm_request = self.request;
}
Copy the code

DidReceiveResponse:

- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response {
    [[self client] URLProtocol:self didReceiveResponse:response cacheStoragePolicy:NSURLCacheStorageAllowed];
    self.dm_response = response;
}
Copy the code

DidReceiveData:

- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data {
    [self.client URLProtocol:self didLoadData:data];
    [self.dm_data appendData:data];
}
Copy the code

The above section is intended to record each required attribute in a single HTTP request.

Recording Response Information

In the stopLoading method, dM_Response and DM_data objects can be analyzed to obtain downstream traffic and other related information.

It should be noted that if you need to obtain very accurate traffic, generally speaking, only through the Socket layer is the most accurate, because you can obtain data size including handshake, wave. Of course, our purpose is to analyze the App’s traffic consuming API, so analyzing only from the application layer basically meets our needs.

As mentioned above, the composition of the message is obtained according to the content required by the message.

Status Line

Unfortunately, NSURLResponse does not have any interface that can directly obtain the Status Line in the message, even HTTP Version and other interfaces that constitute the Status Line content.

Finally, I got the Status Line data by converting to the CFNetwork related class, which may involve reading the private API

Here I add an extension to NSURLResponse: NSURLResponse+DoggerMonitor, and add the statusLineFromCF method to it

typedef CFHTTPMessageRef (*DMURLResponseGetHTTPResponse)(CFURLRef response);

- (NSString *)statusLineFromCF {
    NSURLResponse *response = self;
    NSString *statusLine = @ "";
    / / get CFURLResponseGetHTTPResponse function implementation
    NSString *funName = @"CFURLResponseGetHTTPResponse";
    DMURLResponseGetHTTPResponse originURLResponseGetHTTPResponse =
    dlsym(RTLD_DEFAULT, [funName UTF8String]);

    SEL theSelector = NSSelectorFromString(@"_CFURLResponse");
    if ([response respondsToSelector:theSelector] &&
        NULL! = originURLResponseGetHTTPResponse) {// Get _CFURLResponse of NSURLResponse
        CFTypeRef cfResponse = CFBridgingRetain([response performSelector:theSelector]);
        if (NULL! = cfResponse) {// Convert CFURLResponseRef to CFHTTPMessageRef
            CFHTTPMessageRef messageRef = originURLResponseGetHTTPResponse(cfResponse);
            statusLine = (__bridge_transfer NSString *)CFHTTPMessageCopyResponseStatusLine(messageRef);
            CFRelease(cfResponse); }}return statusLine;
}
Copy the code

CFTypeRef is obtained by calling the private API _CFURLResponse and then converted to CFHTTPMessageRef to obtain the Status Line.

Then convert it to NSData to calculate the size of bytes:

- (NSUInteger)dm_getLineLength {
    NSString *lineStr = @ "";
    if ([self isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)self;
        lineStr = [self statusLineFromCF];
    }
    NSData *lineData = [lineStr dataUsingEncoding:NSUTF8StringEncoding];
    return lineData.length;
}
Copy the code

Header

Through httpResponse. AllHeaderFields get the Header of a dictionary, and then joining together into a key message: the value format, converted to NSData computing size:

- (NSUInteger)dm_getHeadersLength {
    NSUInteger headersLength = 0;
    if ([self isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)self;
        NSDictionary<NSString *, NSString *> *headerFields = httpResponse.allHeaderFields;
        NSString *headerStr = @ "";
        for (NSString *key in headerFields.allKeys) {
            headerStr = [headerStr stringByAppendingString:key];
            headerStr = [headerStr stringByAppendingString:@": "];
            if ([headerFields objectForKey:key]) {
                headerStr = [headerStr stringByAppendingString:headerFields[key]];
            }
            headerStr = [headerStr stringByAppendingString:@"\n"];
        }
        NSData *headerData = [headerStr dataUsingEncoding:NSUTF8StringEncoding];
        headersLength = headerData.length;
    }
    return headersLength;
}
Copy the code

Body

For the Body calculation, the expectedContentLength and The Content-Length value obtained from the allHeaderFields of the NSURLResponse object are not accurate enough.

First of all, the API documentation for the expectedContentLength is not accurate:

Secondly, the HTTP 1.1 standard also introduces that the content-Length field is not necessarily included in every Response. Most importantly, the content-Length field only represents the size of the Body part.

The way I did it, as I said in the previous code, is I assigned dM_data in didReceiveData

DidReceiveData:

- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data {
    [self.client URLProtocol:self didLoadData:data];
    [self.dm_data appendData:data];
}
Copy the code

So in the stopLoading method, you can get the data received for this network request.

However, it should be noted that the gZIP case is differentiated. We know that in HTTP requests, the client sends the request with accept-Encoding, and the value of this field tells the server what compression algorithm the client understands. When the server responds, content-Encoding is added to Response to inform the client of the selected compression algorithm.

So, we get the content-encoding in stopLoading and, if gzip is used, simulate a Gzip compression and calculate the byte size:

- (void)stopLoading {
    [self.connection cancel];

    DMNetworkTrafficLog *model = [[DMNetworkTrafficLog alloc] init];
    model.path = self.request.URL.path;
    model.host = self.request.URL.host;
    model.type = DMNetworkTrafficDataTypeResponse;
    model.lineLength = [self.dm_response dm_getLineLength];
    model.headerLength = [self.dm_response dm_getHeadersLength];
    if ([self.dm_response isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)self.dm_response;
        NSData *data = self.dm_data;
        if ([[httpResponse.allHeaderFields objectForKey:@"Content-Encoding"] isEqualToString:@"gzip"]) {
            // Simulate compression
            data = [self.dm_data gzippedData];
        }
        model.bodyLength = data.length;
    }
    model.length = model.lineLength + model.headerLength + model.bodyLength;
    [model settingOccurTime];
    [[DMDataManager defaultDB] addNetworkTrafficLog:model];
}
Copy the code

Here gzippedData refers to the contents of this library

[[DMDataManager defaultDB] addNetworkTrafficLog:model]; The code that calls the persistence layer drops the data into the repository.

Record Resquest information

Line

Unfortunately, for NSURLRequest, I didn’t have the luck of finding a private interface to convert it to CFNetwork data like NSURLReponse did, but we know what the Line part of the HTTP request message is, so we can add a method, Get an experience Line.

Also add an extension to NSURLReques: NSURLRequest+DoggerMonitor

- (NSUInteger)dgm_getLineLength {
    NSString *lineStr = [NSString stringWithFormat:@"%@ %@ %@\n".self.HTTPMethod, self.URL.path, HTTP / 1.1 "@"];
    NSData *lineData = [lineStr dataUsingEncoding:NSUTF8StringEncoding];
    return lineData.length;
}
Copy the code

Header

Header has a very big hole here.

Request. AllHTTPHeaderFields get the head of the data is there are a lot of missing, the exchanges with the industry friends, find a lot of people don’t pay attention to the problem.

The missing piece is not just the Cookie mentioned in the previous article.

If the packet is captured by Charles, it can be seen that the following fields including but not limited to are missing:

  1. Accept
  2. Connection
  3. Host

This problem is very confusing, and because it cannot be converted to the CFNetwork layer, the accurate Header data cannot be obtained all the time.

Finally, I also found two related problems on SO for your reference

NSUrlRequest: where an app can find the default headers for HTTP request?

NSMutableURLRequest, cant access all request headers sent out from within my iPhone program

The answers to the two questions basically indicate that you can only get the complete Header data if you make the request through CFNetwork.

Therefore, this piece can only get most of the headers, but basically the missing fields are fixed, which does not have a great impact on the accuracy of our traffic statistics.

Then the cookie part is mainly completed:

- (NSUInteger)dgm_getHeadersLengthWithCookie {
    NSUInteger headersLength = 0;

    NSDictionary<NSString *, NSString *> *headerFields = self.allHTTPHeaderFields;
    NSDictionary<NSString *, NSString *> *cookiesHeader = [self dgm_getCookies];

    // Add cookie information
    if (cookiesHeader.count) {
        NSMutableDictionary *headerFieldsWithCookies = [NSMutableDictionary dictionaryWithDictionary:headerFields];
        [headerFieldsWithCookies addEntriesFromDictionary:cookiesHeader];
        headerFields = [headerFieldsWithCookies copy];
    }
    NSLog(@ "% @", headerFields);
    NSString *headerStr = @ "";

    for (NSString *key in headerFields.allKeys) {
        headerStr = [headerStr stringByAppendingString:key];
        headerStr = [headerStr stringByAppendingString:@": "];
        if ([headerFields objectForKey:key]) {
            headerStr = [headerStr stringByAppendingString:headerFields[key]];
        }
        headerStr = [headerStr stringByAppendingString:@"\n"];
    }
    NSData *headerData = [headerStr dataUsingEncoding:NSUTF8StringEncoding];
    headersLength = headerData.length;
    return headersLength;
}
Copy the code
- (NSDictionary<NSString *, NSString *> *)dgm_getCookies {
    NSDictionary<NSString *, NSString *> *cookiesHeader;
    NSHTTPCookieStorage *cookieStorage = [NSHTTPCookieStorage sharedHTTPCookieStorage];
    NSArray<NSHTTPCookie *> *cookies = [cookieStorage cookiesForURL:self.URL];
    if (cookies.count) {
        cookiesHeader = [NSHTTPCookie requestHeaderFieldsWithCookies:cookies];
    }
    return cookiesHeader;
}
Copy the code

body

And then finally the body part, which also has a pit here. throughNSURLConnectionNetwork request maderesquest.HTTPBodyYou get nil.

Instead, you need to read the stream via HTTPBodyStream to get the Body size of the request.

- (NSUInteger)dgm_getBodyLength {
    NSDictionary<NSString *, NSString *> *headerFields = self.allHTTPHeaderFields;
    NSUInteger bodyLength = [self.HTTPBody length];

    if ([headerFields objectForKey:@"Content-Encoding"]) {
        NSData *bodyData;
        if (self.HTTPBody == nil) {
            uint8_t d[1024] = {0};
            NSInputStream *stream = self.HTTPBodyStream;
            NSMutableData *data = [[NSMutableData alloc] init];
            [stream open];
            while ([stream hasBytesAvailable]) {
                NSInteger len = [stream read:d maxLength:1024];
                if (len > 0 && stream.streamError == nil) {
                    [data appendBytes:(void *)d length:len];
                }
            }
            bodyData = [data copy];
            [stream close];
        } else {
            bodyData = self.HTTPBody;
        }
        bodyLength = [[bodyData gzippedData] length];
    }

    return bodyLength;
}
Copy the code

Fall library

Finally in DMURLProtocol – (nullable NSURLRequest *)connection:(NSURLConnection *)connection willSendRequest:(NSURLRequest *)request redirectResponse:(nullable NSURLResponse *)response; Method to call each part of resquest message size method fallback library:

- (NSURLRequest *)connection:(NSURLConnection *)connection willSendRequest:(NSURLRequest *)request redirectResponse:(NSURLResponse *)response {
    if(response ! =nil) {
        self.dm_response = response;
        [self.client URLProtocol:self wasRedirectedToRequest:request redirectResponse:response];
    }

    DMNetworkTrafficLog *model = [[DMNetworkTrafficLog alloc] init];
    model.path = request.URL.path;
    model.host = request.URL.host;
    model.type = DMNetworkTrafficDataTypeRequest;
    model.lineLength = [connection.currentRequest dgm_getLineLength];
    model.headerLength = [connection.currentRequest dgm_getHeadersLengthWithCookie];
    model.bodyLength = [connection.currentRequest dgm_getBodyLength];
    model.length = model.lineLength + model.headerLength + model.bodyLength;
    [model settingOccurTime];
    [[DMDataManager defaultDB] addNetworkTrafficLog:model];
    return request;
}
Copy the code

Handling of NSURLSession

Using DMURLProtocol and registerClass directly cannot completely intercept all network requests, because requests made by NSURLSession’s sharedSession cannot be broked by NSURLProtocol.

We need to make [NSURLSessionConfiguration defaultSessionConfiguration]. Also set up our DMURLProtocol protocolClasses properties, through the swizzle here, Replace the get method for protocalClasses:

Write a DMURLSessionConfiguration

#import <Foundation/Foundation.h>

@interface DMURLSessionConfiguration : NSObject

@property (nonatomic.assign) BOOL isSwizzle;
+ (DMURLSessionConfiguration *)defaultConfiguration;
- (void)load;
- (void)unload;

@end
Copy the code
#import "DMURLSessionConfiguration.h"
#import <objc/runtime.h>
#import "DMURLProtocol.h"
#import "DMNetworkTrafficManager.h"

@implementation DMURLSessionConfiguration

+ (DMURLSessionConfiguration *)defaultConfiguration {
    static DMURLSessionConfiguration *staticConfiguration;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        staticConfiguration=[[DMURLSessionConfiguration alloc] init];
    });
    return staticConfiguration;
    
}

- (instancetype)init {
    self = [super init];
    if (self) {
        self.isSwizzle = NO;
    }
    return self;
}

- (void)load {
    self.isSwizzle = YES;
    Class cls = NSClassFromString(@"__NSCFURLSessionConfiguration") ?: NSClassFromString(@"NSURLSessionConfiguration");
    [self swizzleSelector:@selector(protocolClasses) fromClass:cls toClass:[self class]];
    
}

- (void)unload {
    self.isSwizzle=NO;
    Class cls = NSClassFromString(@"__NSCFURLSessionConfiguration") ?: NSClassFromString(@"NSURLSessionConfiguration");
    [self swizzleSelector:@selector(protocolClasses) fromClass:cls toClass:[self class]];
}

- (void)swizzleSelector:(SEL)selector fromClass:(Class)original toClass:(Class)stub {
    Method originalMethod = class_getInstanceMethod(original, selector);
    Method stubMethod = class_getInstanceMethod(stub, selector);
    if(! originalMethod || ! stubMethod) { [NSException raise:NSInternalInconsistencyException format:@"Couldn't load NEURLSessionConfiguration."];
    }
    method_exchangeImplementations(originalMethod, stubMethod);
}

- (NSArray *)protocolClasses {
    // a custom protocolClasses can be set for users
    return [DMNetworkTrafficManager manager].protocolClasses;
}

@end
Copy the code

So, we’ve written the method substitution, and after executing the load method of that class singleton, [NSURLSessionConfiguration defaultSessionConfiguration] protocolClasses get would be we set good protocolClasses.

So, we add the start and stop methods to DMURLProtocol to start and stop network monitoring:

+ (void)start {
    DMURLSessionConfiguration *sessionConfiguration = [DMURLSessionConfiguration defaultConfiguration];
    for (id protocolClass in [DMNetworkTrafficManager manager].protocolClasses) {
        [NSURLProtocol registerClass:protocolClass];
    }
    if(! [sessionConfiguration isSwizzle]) {// Set the switch[sessionConfiguration load]; }} + (void)end {
    DMURLSessionConfiguration *sessionConfiguration = [DMURLSessionConfiguration defaultConfiguration];
    [NSURLProtocol unregisterClass:[DMURLProtocol class]].if ([sessionConfiguration isSwizzle]) {
        // Cancel the swap[sessionConfiguration unload]; }}Copy the code

At this point, the whole network traffic monitoring is basically completed.

Provide a Manger for users to call:

#import <Foundation/Foundation.h>

@class DMNetworkLog;
@interface DMNetworkTrafficManager : NSObject

/** All NSURLProtocol external set interface, can prevent other external monitoring NSURLProtocol */
@property (nonatomic.strong) NSArray *protocolClasses;


Singleton / * * * /
+ (DMNetworkTrafficManager *)manager;

/** Start the traffic monitoring module */ through protocolClasses
+ (void)startWithProtocolClasses:(NSArray *)protocolClasses;
/** Start the traffic monitoring module only with DMURLProtocol */
+ (void)start;
/** Stop traffic monitoring */
+ (void)end;

@end
Copy the code
#import "DMNetworkTrafficManager.h"
#import "DMURLProtocol.h"

@interface DMNetworkTrafficManager(a)

@end

@implementation DMNetworkTrafficManager

#pragma mark - Public

+ (DMNetworkTrafficManager *)manager {
    static DMNetworkTrafficManager *manager;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        manager=[[DMNetworkTrafficManager alloc] init];
    });
    return manager;
}

+ (void)startWithProtocolClasses:(NSArray *)protocolClasses {
    [self manager].protocolClasses = protocolClasses;
    [DMURLProtocol start];
}

+ (void)start {
    [self manager].protocolClasses = @[[DMURLProtocol class]];
    [DMURLProtocol start];
}

+ (void)end {
    [DMURLProtocol end];
}

@end
Copy the code

Five, the code

More code is posted in this article, so you can read it here for your overall viewing.

Because it contains some data manipulation content that does not need to be concerned, so I directly omit, although there is no Demo, BUT I believe that everyone can understand the whole monitoring structure.

Sixth, Other

If your APP is supported from iOS 9, you can use NetworkExtension, which can take over the entire network request via VPN, eliminating all of the above worries.