APM Monitoring System: Part 2 of Networking

2.5 iOS Traffic Monitoring

2.5.1 HTTP request and response data structure

HTTP request packet structure

Response packet structure

An HTTP packet is a formatted data block. Each packet consists of three parts: a starting line that describes the packet, a header block that contains attributes, and an optional body part that contains data.
The starting line and hand are ASCII text delimited by line delimiters, each line ending with a 2-character line termination sequence (including a carriage return and a newline).
The body of the entity or message is an optional data block. Unlike the start line and header, the body can contain text or binary data, or it can be empty.
The HTTP header (also known as Headers) should always end with an empty line, even if there is no physical part. The browser sends a blank line to inform the server that it has finished sending the header.

Format of the request packet

<method> <request-URI> <version>
<headers>

<entity-body>
Copy the code

Format of the response packet

<version> <status> <reason-phrase>
<headers>

<entity-body>
Copy the code

The following image shows a request for viewing extreme class times in Chrome. Including the response line, response header, response body and other information.

Using curl to view a complete request and response data

As we all know, in HTTP communication, response data will be compressed by GZIP or other compression methods, monitored by NSURLProtocol and other schemes, and analyzed by NSData type to calculate and analyze traffic, resulting in inaccurate data. Because the content of an HTTP response body is normally compressed using Gzip or other compression methods, using NSData tends to be too large.

2.5.2 problems

Request and Response don’t necessarily come in pairs

For example, the network is disconnected and the App crashes suddenly. Therefore, the monitoring of Request and Response should not be recorded in the same record
The request traffic calculation method is not accurate

The main reasons are:
- The monitoring technical solution ignores the data size of the header and row parts of the request
- The monitoring technical solution ignores the size of the Cookie part of the data
- Monitoring solutions are used directly when calculating the request body sizeHTTPBody.length, resulting in inaccuracy
The calculation method of response traffic is not accurate

The main reasons are:
- The monitoring technical solution ignores the data size of the response header and response row portions
- Monitor the technical solution in the body part of the byte size calculation, as usedexceptedContentLengthLead to inaccuracy
- The monitoring technical solution ignores the use of GZIP compression for the response body. In real network communication, the client is in the request header that initiates the requestAccept-EncodingThe field represents the data compression method supported by the client (indicating the compression method supported by the client when the data can be used normally). Similarly, the server processes the data according to the compression method desired by the client and the compression method currently supported by the server, in the response headerContent-EncodingThe field indicates what compression method is currently used by the server.

2.5.3 Technical implementation

The fifth part describes the various principles and technical schemes of network interception. Here, NSURLProtocol is used to realize traffic monitoring (Hook method). Knowing what we need from above, let’s implement it step by step.

2.5.3.1 Request part

First, NSURLProtocol is used to manage various network requests of App by network monitoring scheme

NSURLProtocol does not analyze the size and time consumed of the request, such as handshake, wave, etc., but it is sufficient for normal case interface traffic analysis. Socket layer is required at the bottom level.

@property(nonatomic, strong) NSURLConnection *internalConnection;
@property(nonatomic, strong) NSURLResponse *internalResponse;
@property(nonatomic, strong) NSMutableData *responseData;
@property (nonatomic, strong) NSURLRequest *internalRequest;
Copy the code

- (void)startLoading
{
    NSMutableURLRequest *mutableRequest = [[self request] mutableCopy];
    self.internalConnection = [[NSURLConnection alloc] initWithRequest:mutableRequest delegate:self];
    self.internalRequest = self.request;
}

- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
{
    [self.client URLProtocol:self didReceiveResponse:response cacheStoragePolicy:NSURLCacheStorageNotAllowed];
    self.internalResponse = response;
}

- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data 
{
    [self.responseData appendData:data];
    [self.client URLProtocol:self didLoadData:data];
}
Copy the code

The Status Line section

NSURLResponse does not have a Status Line or interface, nor does HTTP Version information, so if you want to get a Status Line, try converting it to CFNetwork layer. Found a private API to implement.

NSURLResponse is passed_CFURLResponseconvertCFTypeRefAnd then theCFTypeRefconvertCFHTTPMessageRefAnd then through theCFHTTPMessageCopyResponseStatusLineTo obtainCFHTTPMessageRefThe Status Line information of the

Add a classification of NSURLResponse to the ability to read the Status Line.

// NSURLResponse+apm_FetchStatusLineFromCFNetwork.h #import <Foundation/Foundation.h> NS_ASSUME_NONNULL_BEGIN @interface  NSURLResponse (apm_FetchStatusLineFromCFNetwork) - (NSString *)apm_fetchStatusLineFromCFNetwork; @end NS_ASSUME_NONNULL_END // NSURLResponse+apm_FetchStatusLineFromCFNetwork.m #import "NSURLResponse+apm_FetchStatusLineFromCFNetwork.h" #import <dlfcn.h> #define SuppressPerformSelectorLeakWarning(Stuff) \  do { \ _Pragma("clang diagnostic push") \ _Pragma("clang diagnostic ignored \"-Warc-performSelector-leaks\"") \ Stuff; \ _Pragma("clang diagnostic pop") \ } while (0) typedef CFHTTPMessageRef (*APMURLResponseFetchHTTPResponse)(CFURLRef response); @implementation NSURLResponse (apm_FetchStatusLineFromCFNetwork) - (NSString *)apm_fetchStatusLineFromCFNetwork { NSString *statusLine = @""; NSString *funcName = @"CFURLResponseGetHTTPResponse"; APMURLResponseFetchHTTPResponse originalURLResponseFetchHTTPResponse = dlsym(RTLD_DEFAULT, [funcName UTF8String]); SEL getSelector = NSSelectorFromString(@"_CFURLResponse"); if ([self respondsToSelector:getSelector] && NULL ! = originalURLResponseFetchHTTPResponse) { CFTypeRef cfResponse; SuppressPerformSelectorLeakWarning( cfResponse = CFBridgingRetain([self performSelector:getSelector]); ) ; if (NULL ! = cfResponse) { CFHTTPMessageRef messageRef = originalURLResponseFetchHTTPResponse(cfResponse); statusLine = (__bridge_transfer NSString *)CFHTTPMessageCopyResponseStatusLine(messageRef); CFRelease(cfResponse); } } return statusLine; } @endCopy the code

Convert the obtained Status Line to NSData, and then calculate the size

- (NSUInteger)apm_getLineLength {
    NSString *statusLineString = @"";
    if ([self isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)self;
        statusLineString = [self apm_fetchStatusLineFromCFNetwork];
    }
    NSData *lineData = [statusLineString dataUsingEncoding:NSUTF8StringEncoding];
    return lineData.length;
}
Copy the code

The Header section

AllHeaderFields retrieves the NSDictionary, concatenates it into a string based on key: value, and converts it to NSData to calculate the size

Curl or chrome Network

- (NSUInteger)apm_getHeadersLength
{
    NSUInteger headersLength = 0;
    if ([self isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)self;
        NSDictionary *headerFields = httpResponse.allHeaderFields;
        NSString *headerString = @"";
        for (NSString *key in headerFields.allKeys) {
            headerString = [headerStr stringByAppendingString:key];
            headheaderStringerStr = [headerString stringByAppendingString:@": "];
            if ([headerFields objectForKey:key]) {
                headerString = [headerString stringByAppendingString:headerFields[key]];
            }
            headerString = [headerString stringByAppendingString:@"\n"];
        }
        NSData *headerData = [headerString dataUsingEncoding:NSUTF8StringEncoding];
        headersLength = headerData.length;
    }
    return headersLength;
}
Copy the code

The Body part

The Body size cannot be directly calculated using excepectedContentLength. The official document states that it is not accurate and can only be used as a reference. Or the Content-Length value in allHeaderFields.

/ *!

@abstract Returns the expected content length of the receiver.

@discussion Some protocol implementations report a content length

as part of delivering load metadata, but not all protocols

guarantee the amount of data that will be delivered in actuality.

Hence, this method returns an expected amount. Clients should use

this value as an advisory, and should be prepared to deal with

either more or less data.

@result The expected content length of the receiver, or -1 if

there is no expectation that can be arrived at regarding expected

content length.

* /

@property (readonly) long long expectedContentLength;
- HTTP 1.1 version specifies if presentTransfer-Encoding: chunked, cannot be in the headerContent-LengthIf it does, it will be ignored.
- In HTTP 1.0 and earlier,content-lengthFields are optional
- In HTTP 1.1 and later. If it iskeep alive,Content-Length 和 chunkedIt has to be one or the other. If you arekeep aliveIs the same as HTTP 1.0.Content-LengthDispensable.
What is transfer-encoding: chunked

The data is sent as a series of chunks and the Content-Length header is not sent in this case. At the beginning of each partition the length of the current partition is added, in hexadecimal form, followed by \r\n, followed by the partition itself, and also \r\n. The terminating block is a regular partition, the difference being that its length is 0.

We recorded the data with NSMutableData earlier, so we can calculate the Body size in the stopLoading method. The steps are as follows:
- Keep adding data to didReceiveData
```
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
    [self.responseData appendData:data];
    [self.client URLProtocol:self didLoadData:data];
}
Copy the code
```
- Get the content-Encoding key value from the allHeaderFields dictionary in the stopLoading method. If it is gzip, process NSData as gzip compressed data in the stopLoading method. Let’s calculate the magnitude. (Use this tool for gZIP related functions)
  
  An additional blank line length needs to be computed
```
- (void)stopLoadi
{
    [self.internalConnection cancel];

    HCTNetworkTrafficModel *model = [[HCTNetworkTrafficModel alloc] init];
    model.path = self.request.URL.path;
    model.host = self.request.URL.host;
    model.type = DMNetworkTrafficDataTypeResponse;
    model.lineLength = [self.internalResponse apm_getStatusLineLength];
    model.headerLength = [self.internalResponse apm_getHeadersLength];
    model.emptyLineLength = [self.internalResponse apm_getEmptyLineLength];
    if ([self.dm_response isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)self.dm_response;
        NSData *data = self.dm_data;
        if ([[httpResponse.allHeaderFields objectForKey:@"Content-Encoding"] isEqualToString:@"gzip"]) {
            data = [self.dm_data gzippedData];
        }
        model.bodyLength = data.length;
    }
    model.length = model.lineLength + model.headerLength + model.bodyLength + model.emptyLineLength;
    NSDictionary *networkTrafficDictionary = [model convertToDictionary];
    [[HermesClient sharedInstance] sendWithType:APMMonitorNetworkTrafficType meta:networkTrafficDictionary payload:nil];
}
Copy the code
```

2.5.3.2 Resquest part

First, NSURLProtocol is used to manage various network requests of App by network monitoring scheme

@property(nonatomic, strong) NSURLConnection *internalConnection;
@property(nonatomic, strong) NSURLResponse *internalResponse;
@property(nonatomic, strong) NSMutableData *responseData;
@property (nonatomic, strong) NSURLRequest *internalRequest;
Copy the code

- (void)startLoading
{
    NSMutableURLRequest *mutableRequest = [[self request] mutableCopy];
    self.internalConnection = [[NSURLConnection alloc] initWithRequest:mutableRequest delegate:self];
    self.internalRequest = self.request;
}

- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
{
    [self.client URLProtocol:self didReceiveResponse:response cacheStoragePolicy:NSURLCacheStorageNotAllowed];
    self.internalResponse = response;
}

- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data 
{
    [self.responseData appendData:data];
    [self.client URLProtocol:self didLoadData:data];
}
Copy the code

The Status Line section

There is no way to find the StatusLine for NSURLRequest like there is for NSURLResponse. Therefore, the bottom pocket solution is to manually construct one according to the structure of the Status Line. Format: Protocol version number + space + status code + space + status text + newline

Add a class for NSURLRequest that gets the Status Line.
```
// NSURLResquest+apm_FetchStatusLineFromCFNetwork.m - (NSUInteger)apm_fetchStatusLineLength { NSString *statusLineString = [NSString stringWithFormat:@"%@ %@ %@ %@\n", self.HTTPMethod, self.url.path, @"HTTP/1.1"]; NSData *statusLineData = [statusLineString dataUsingEncoding:NSUTF8StringEncoding]; return statusLineData.length; }Copy the code
```

The Header section

An HTTP request builds the server IP address that determines whether there is a cache and then performs DNS domain name resolution to get the requested domain name. If the request protocol is HTTPS, you also need to establish a TLS connection. The next step is to establish a TCP connection with the server using the IP address. After the connection is established, the browser side will construct the request line, request information, etc., and attach the data related to the domain name, such as cookies, to the request header, and then send the constructed request information to the server.

So a network monitoring does not consider cookie 😂, borrow a word from Wang Duoyu “that is not over the calf”.

Read some articles about NSURLRequest not being able to get a complete request header. In fact, the problem is not big, a few information is not completely available. Measuring the monitoring scheme itself is to see whether the data consumption of the interface is abnormal in different versions or in some cases, whether the WebView resource request is too large, similar to the idea of the control variable method.

Therefore, after obtaining the allHeaderFields of NSURLRequest, add the cookie information to calculate the complete Header size

// NSURLResquest+apm_FetchHeaderWithCookies.m - (NSUInteger)apm_fetchHeaderLengthWithCookie { NSDictionary *headerFields  = self.allHTTPHeaderFields; NSDictionary *cookiesHeader = [self apm_fetchCookies]; if (cookiesHeader.count) { NSMutableDictionary *headerDictionaryWithCookies = [NSMutableDictionary dictionaryWithDictionary:headerFields]; [headerDictionaryWithCookies addEntriesFromDictionary:cookiesHeader]; headerFields = [headerDictionaryWithCookies copy]; } NSString *headerString = @""; for (NSString *key in headerFields.allKeys) { headerString = [headerString stringByAppendingString:key]; headerString = [headerString stringByAppendingString:@": "]; if ([headerFields objectForKey:key]) { headerString = [headerString stringByAppendingString:headerFields[key]]; } headerString = [headerString stringByAppendingString:@"\n"]; } NSData *headerData = [headerString dataUsingEncoding:NSUTF8StringEncoding]; headersLength = headerData.length; return headerString; } - (NSDictionary *)apm_fetchCookies { NSDictionary *cookiesHeaderDictionary; NSHTTPCookieStorage *cookieStorage = [NSHTTPCookieStorage sharedHTTPCookieStorage]; NSArray<NSHTTPCookie *> *cookies = [cookieStorage cookiesForURL:self.URL]; if (cookies.count) { cookiesHeaderDictionary = [NSHTTPCookie requestHeaderFieldsWithCookies:cookies]; } return cookiesHeaderDictionary; }Copy the code

The Body part

The HTTPBody of the NSURLConnection may not be available, similar to the problem with Ajax on a WebView. So you can calculate the body size by reading the stream at HTTPBodyStream.

- (NSUInteger)apm_fetchRequestBody
{
    NSDictionary *headerFields = self.allHTTPHeaderFields;
    NSUInteger bodyLength = [self.HTTPBody length];

    if ([headerFields objectForKey:@"Content-Encoding"]) {
        NSData *bodyData;
        if (self.HTTPBody == nil) {
            uint8_t d[1024] = {0};
            NSInputStream *stream = self.HTTPBodyStream;
            NSMutableData *data = [[NSMutableData alloc] init];
            [stream open];
            while ([stream hasBytesAvailable]) {
                NSInteger len = [stream read:d maxLength:1024];
                if (len > 0 && stream.streamError == nil) {
                    [data appendBytes:(void *)d length:len];
                }
            }
            bodyData = [data copy];
            [stream close];
        } else {
            bodyData = self.HTTPBody;
        }
        bodyLength = [[bodyData gzippedData] length];
    }
    return bodyLength;
}
Copy the code

In – (NSURLRequest *)connection:(NSURLConnection *)connection willSendRequest:(NSURLRequest *)request RedirectResponse :(NSURLResponse *) reporting data in response is part of building powerful, flexible and configurable data reporting components

-(NSURLRequest *)connection:(NSURLConnection *)connection willSendRequest:(NSURLRequest *)request redirectResponse:(NSURLResponse *)response { if (response ! = nil) { self.internalResponse = response; [self.client URLProtocol:self wasRedirectedToRequest:request redirectResponse:response]; } HCTNetworkTrafficModel *model = [[HCTNetworkTrafficModel alloc] init]; model.path = request.URL.path; model.host = request.URL.host; model.type = DMNetworkTrafficDataTypeRequest; model.lineLength = [connection.currentRequest dgm_getLineLength]; model.headerLength = [connection.currentRequest dgm_getHeadersLengthWithCookie]; model.bodyLength = [connection.currentRequest dgm_getBodyLength]; model.emptyLineLength = [self.internalResponse apm_getEmptyLineLength]; model.length = model.lineLength + model.headerLength + model.bodyLength + model.emptyLineLength; NSDictionary *networkTrafficDictionary = [model convertToDictionary]; [[HermesClient sharedInstance] sendWithType:APMMonitorNetworkTrafficType meta:networkTrafficDictionary payload:nil]; return request; }Copy the code

APM Monitoring System: Part 2 of Networking

2.5 iOS Traffic Monitoring

2.5.1 HTTP request and response data structure

2.5.2 problems

2.5.3 Technical implementation

2.5.3.1 Request part

2.5.3.2 Resquest part

Related Posts

In-depth understanding of Autolayout with list performance – back to the Cassowary and lazy CPU

IOS Swift5 from 0 to 1 Series (4) : Learn UINavigationController (2) : Bottom TabBar show and hide

How to write aN iOS app without Xcode