Author: Ao Zhimin
This article is original, please indicate the author and source
Domestic mobile network environment is very complex, WIFI, 4G, 3G, 2.5g (Edge), 2G and other mobile networks coexist, users may switch between WIFI/4G/3G/ 2.5g /2G network, which is a big difference between mobile network and traditional network. This is called a Connection Migration problem. In addition, there are some problems such as slow DNS resolution, high failure rate and DNS hijacking. There are also domestic operators interconnection and overseas access to domestic bandwidth low transmission and other problems. These network problems are a real headache. Due to the current situation of mobile network, users often encounter various network problems in the process of using the App. Network problems will directly lead to users unable to operate in the App. When some key business interfaces are wrong, users will even lose a large number of users directly. Network problems not only bring great challenges to mobile development, but also bring new opportunities to network monitoring. In the past, to solve these problems, we can only rely on experience and guess, but if we can monitor the network from the perspective of App, we can have a more targeted understanding of the root causes of the problems.
Network monitoring is generally implemented by NSURLProtocol and Code injection (Hook). As the upper interface, NSURLProtocol is more convenient to use, so it is a natural choice for network monitoring scheme. However, NSURLProtocol belongs to the URL Loading System and has limited protocol support at the application layer. It only supports several application layer protocols, such as FTP, HTTP and HTTPS, and is helpless for traffic using other protocols, so it has certain limitations. Monitoring the underlying network library CFNetwork does not have this limitation.
The following are the key performance indicators of network collection:
-
TCP connection establishment time
-
DNS time
-
SSL time
-
The first package time
-
The response time
-
HTTP error rate
-
Network error rate
NSURLProtocol
/ / in order to avoid canInitWithRequest and canonicalRequestForRequest appear dead circulation
static NSString * const HJHTTPHandledIdentifier = @"hujiang_http_handled";
@interface HJURLProtocol () <NSURLSessionTaskDelegate, NSURLSessionDataDelegate>
@property (nonatomic, strong) NSURLSessionDataTask *dataTask;
@property (nonatomic, strong) NSOperationQueue *sessionDelegateQueue;
@property (nonatomic, strong) NSURLResponse *response;
@property (nonatomic, strong) NSMutableData *data;
@property (nonatomic, strong) NSDate *startDate;
@property (nonatomic, strong) HJHTTPModel *httpModel;
@end
+ (BOOL)canInitWithRequest:(NSURLRequest *)request {
if (! [request.URL.scheme isEqualToString:@"http"] &&
! [request.URL.scheme isEqualToString:@"https"]) {
return NO;
}
if ([NSURLProtocol propertyForKey:HJHTTPHandledIdentifier inRequest:request] ) {
return NO;
}
return YES;
}
+ (NSURLRequest *)canonicalRequestForRequest:(NSURLRequest *)request {
NSMutableURLRequest *mutableReqeust = [request mutableCopy];
[NSURLProtocol setProperty:@YES
forKey:HJHTTPHandledIdentifier
inRequest:mutableReqeust];
return [mutableReqeust copy];
}
- (void)startLoading {
self.startDate = [NSDate date];
self.data = [NSMutableData data];
NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
self.sessionDelegateQueue = [[NSOperationQueue alloc] init];
self.sessionDelegateQueue.maxConcurrentOperationCount = 1;
self.sessionDelegateQueue.name = @"com.hujiang.wedjat.session.queue";
NSURLSession *session = [NSURLSession sessionWithConfiguration:configuration delegate:self delegateQueue:self.sessionDelegateQueue];
self.dataTask = [session dataTaskWithRequest:self.request];
[self.dataTask resume];
httpModel = [[NEHTTPModel alloc] init];
httpModel.request = self.request;
httpModel.startDateString = [self stringWithDate:[NSDate date]];
NSTimeInterval myID = [[NSDate date] timeIntervalSince1970];
double randomNum = ((double)(arc4random() % 100))/10000;
httpModel.myID = myID+randomNum;
}
- (void)stopLoading {
[self.dataTask cancel];
self.dataTask = nil;
httpModel.response = (NSHTTPURLResponse *)self.response;
httpModel.endDateString = [self stringWithDate:[NSDate date]];
NSString *mimeType = self.response.MIMEType;
// Parse response, traffic statistics, etc
}
#pragma mark - NSURLSessionTaskDelegate
- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didCompleteWithError:(NSError *)error {
if (! error) {
[self.client URLProtocolDidFinishLoading:self];
} else if ([error.domain isEqualToString:NSURLErrorDomain] && error.code == NSURLErrorCancelled) {
} else {
[self.client URLProtocol:self didFailWithError:error];
}
self.dataTask = nil;
}
#pragma mark - NSURLSessionDataDelegate
- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask
didReceiveData:(NSData *)data {
[self.client URLProtocol:self didLoadData:data];
}
- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveResponse:(NSURLResponse *)response completionHandler:(void (^)(NSURLSessionResponseDisposition))completionHandler {
[[self client] URLProtocol:self didReceiveResponse:response cacheStoragePolicy:NSURLCacheStorageAllowed];
completionHandler(NSURLSessionResponseAllow);
self.response = response;
}
- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task willPerformHTTPRedirection:(NSHTTPURLResponse *)response newRequest:(NSURLRequest *)request completionHandler:(void (^)(NSURLRequest * _Nullable))completionHandler {
if (response ! = nil){
self.response = response;
[[self client] URLProtocol:self wasRedirectedToRequest:request redirectResponse:response];
}
}Hertz uses NSURLProtocol, which inherits NSURLProtocol and implements NSURLConnectionDelegate to achieve interception behavior.
HOOK
If we use manual burying to monitor the network, it will invade the business code and be very expensive to maintain. The above problems can be avoided by automatically injecting the code of network performance Monitoring through Hook. Real User experience Monitoring (RUM: Real User Monitoring) can be achieved to monitor the performance of applications in the Real network environment.
AOP(Aspect Oriented Programming) is a technology that dynamically adds functions to programs without modifying source code through pre-compilation and runtime dynamic proxy implementation. Its core idea is to separate business logic (core concerns, the main functions of the system) from common functions (crosscutting concerns, such as logs, things, etc.), reduce complexity, and improve software system modularity, maintainability, and reusability. The core concerns are coded in OOP mode, and the crosscutting concerns are coded in AOP mode. Finally, the two codes are combined to form a system. AOP is widely used in logging, performance statistics, security control, transaction processing, exception handling and other fields.
The implementation of AOP in iOS is based on the Objective-C Runtime mechanism, and there are three ways to implement Hook: Method Swizzling, NSProxy and Fishhook. The first two work with objective-C implemented libraries such as NSURLConnection and NSURLSession, while Fishhook works with C implemented libraries such as CFNetwork.
The following figure is the hook method for three types of network interfaces given by Alibachuan code force monitoring:
The following three implementations are discussed separately:
Method Swizzling
Method Swizzling is a technique that uses objective-C Runtime features to replace one Method implementation with another. Each Class structure has a member variable of the Dispatch Table, which establishes the mapping between each SEL (method name) and corresponding IMP (method implementation, pointer to C function). Method Swizzling is to break the original SEL and IMP mapping relationship and establish a new correlation to achieve the purpose of Method replacement.
Therefore, Method Swizzling can be used to replace the original implementation, add network performance burying behavior to the replacement implementation, and then call the original implementation.
NSProxy
NSProxy is an abstract superclass defining an API for objects that act as stand-ins for other objects or for objects 1. a message to a proxy is forwarded to the real object or causes the proxy to load (or transform itself into) the real object. Subclasses of NSProxy can be used to implement transparent distributed messaging (for example, NSDistantObject) or for lazy instantiation of objects that are expensive to create.
NSProxy, like NSObject, is the root class. NSProxy is an abstract class that you can inherit and override. -forwardInvocation: And – methodSignatureForSelector: method to achieve forward messages to another instance. In summary, the purpose of NSProxy is to forward messages to the proxy class of the real Target.
The Method swizzling replacement Method needs to specify the class name, but the NSURLConnectionDelegate and NSURLSessionDelegate are specified by the business side, and are generally uncertain. Therefore, Method Swizzling is not suitable for this scenario. NSProxy can be used to solve the above problems. Specific implementation: Proxy replaces the delegate of NSURLConnection and NSURLSession. When the Proxy delegate receives a callback, it calls the proxy implementation if it wants to hook the method. The proxy implementation ends up calling the original delegate; If it is not a hook method, the message is forwarded to the original delegate through a message forwarding mechanism. The following figure illustrates the process.
Fishhook
Fishhook is a third-party framework developed by Facebook to dynamically modify the implementation of C functions. Fishhook can be used to replace the implementation of C functions in the dynamic link library. Specifically, to replace the related functions in CFNetwork and CoreFoundation. The monitoring CFNetwork will be described in detail later, which will not be repeated here.
After explaining the implementation technology of hook on iOS, we will discuss how to apply the above three technologies into practice in NSURLConnection, NSURLSession and CFNetwork.
NSURLConnection
NSURLSession
CFNetwork
An overview of the
NeteaseAPM is used as an example to explain how to implement network monitoring through CFNetwork. It is implemented by using the proxy mode. Specifically, It is to implement a Proxy Stream in CFStream of CoreFoundation Framework for the purpose of interception, record the length of network data read through CFStream, and then forward it to Original Stream. The flow chart is as follows:
A detailed description
Since CFNetwork is implemented by C functions, to Hook C functions need to use the Dynamic Loader Hook library function – fishhook.
Dynamic Loader (DYLD) binds symbols by updating Pointers saved in the Mach-O file. It can be used to modify the pointer to a C function call at Runtime. Fishhook is implemented by: Iterate through the symbols in the __nl_symbol_ptr and __la_symbol_ptr sections of the __DATA segment. Through the coordination of Indirect Symbol Table, Symbol Table and String Table, I can find my own functions to be replaced to achieve the purpose of hook.
CFNetwork uses CFReadStreamRef for data passing and callback functions to receive server responses. When the callback function receives notification that there is data in the stream, it saves the data to the client’s memory. It is not appropriate to modify the string table in order to read the stream. If you do so, you will hook the read function that is also used by the system. The read function is not only called by the stream requested by the network, but also all files.
The disadvantage of using the above method is that it is not possible to selectively monitor CFreadstreams associated with HTTP without involving CFreadstreams from files and memory. NeteaseAPM’s solution is to allow the system to construct HTTP streams at the same time. Bridge an NSInputStream subclass ProxyStream as CFReadStream back to the user to monitor the HTTP Stream separately.
The idea is to design a Proxy class that inherits from NSObject and holds an NSInputStream called OriginalStream. By forwarding all messages to the Proxy to OriginalStream processing, and then overwriting NSInputStream’s read:maxLength: method, we can get the stream size.
The code for the XXInputStreamProxy class is as follows:
- (instancetype)initWithStream:(id)stream {
if (self = [super init]) {
_stream = stream;
}
return self;
}
- (NSMethodSignature *)methodSignatureForSelector:(SEL)aSelector {
return [_stream methodSignatureForSelector:aSelector];
}
- (void)forwardInvocation:(NSInvocation *)anInvocation {
[anInvocation invokeWithTarget:_stream];
}Copy the codeInherit NSInputStream and rewrite read:maxLength:
- (NSInteger)read:(uint8_t *)buffer maxLength:(NSUInteger)len {
NSInteger readSize = [_stream read:buffer maxLength:len];
ReadSize / / record
return readSize;
}Copy the codeXX_CFReadStreamCreateForHTTPRequest CFReadStreamCreateForHTTPRequest method will be used to replace system:
tatic CFReadStreamRef (*original_CFReadStreamCreateForHTTPRequest)(CFAllocatorRef __nullable alloc,
CFHTTPMessageRef request);
/ * *
XXInputStreamProxy holds original CFReadStreamRef, forwards messages to original CFReadStreamRef,
The size of the retrieved data is recorded in the read method
* /
static CFReadStreamRef XX_CFReadStreamCreateForHTTPRequest(CFAllocatorRef alloc,
CFHTTPMessageRef request) {
// Use the system method function pointer to complete the system implementation
CFReadStreamRef originalCFStream = original_CFReadStreamCreateForHTTPRequest(alloc, request);
// Convert CFReadStreamRef to NSInputStream, save it in XXInputStreamProxy, and revert back to CFReadStreamRef on final return
NSInputStream *stream = (__bridge NSInputStream *)originalCFStream;
XXInputStreamProxy *outStream = [[XXInputStreamProxy alloc] initWithClient:stream];
CFRelease(originalCFStream);
CFReadStreamRef result = (__bridge_retained CFReadStreamRef)outStream;
return result;
}Copy the codeReplace the function address with fishhook:
void save_original_symbols() {
original_CFReadStreamCreateForHTTPRequest = dlsym(RTLD_DEFAULT, "CFReadStreamCreateForHTTPRequest");
}Copy the coderebind_symbols((struct rebinding[1]){{"CFReadStreamCreateForHTTPRequest", XX_CFReadStreamCreateForHTTPRequest, (void *)& original_CFReadStreamCreateForHTTPRequest}}, 1);Copy the codeAccording to the invocation of CFNetwork API, the design model of fishhook and Proxy Stream to obtain C function is as follows:
NSURLSessionTaskMetrics/NSURLSessionTaskTransactionMetrics
Apple in the iOS 10 NSURLSessionTaskDelegate agent added – URLSession: task: didFinishCollectingMetrics: If the proxy method is implemented, the collected network indicators can be obtained through the NSURLSessionTaskMetrics type parameter of the callback, and the statistics of the time of DNS query, TCP connection establishment, TLS handshake, request response and other links in the network request can be realized.
/ *
* Sent when complete statistics information has been collected for the task.
* /
- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task DidFinishCollectingMetrics: (NSURLSessionTaskMetrics *) metrics API_AVAILABLE (macosx (10.12), the ios (10.0), watchos (3.0), Tvos (10.0));Copy the codeNSURLSessionTaskMetrics
The NSURLSessionTaskMetrics object encapsulates the session task metrics. Each NSURLSessionTaskMetrics object has taskInterval and redirectCount attributes. There are also metrics that are collected in each request/response transaction that occurs during the execution of a task.
-
TransactionMetrics: The transactionMetrics array contains metrics collected in each request/response transaction generated during the execution of the task.
-
/ *
* transactionMetrics array contains the metrics collected for every request/response transaction created during the task execution.
* /
@property (copy, readonly) NSArray<NSURLSessionTaskTransactionMetrics *> *transactionMetrics;Copy the code -
TaskInterval: The total time that a task takes from creation to completion. The creation time of a task is the time when the task is instantiated. Task completion time is the time at which the internal state of the task will become complete.
-
/ *
* Interval from the task creation time to the task completion time.
* Task creation time is the time when the task was instantiated.
* Task completion time is the time when the task is about to change its internal state to completed.
* /
@property (copy, readonly) NSDateInterval *taskInterval;Copy the code -
RedirectCount: Records the number of redirects.
-
/ *
* redirectCount is the number of redirects that were recorded.
* /
@property (assign, readonly) NSUInteger redirectCount;Copy the code
NSURLSessionTaskTransactionMetrics
NSURLSessionTaskTransactionMetrics object encapsulates the task execution time to collect performance indicators, including the request and the response properties, the corresponding HTTP request and response, include the beginning fetchStartDate, The indicator between the end of responseEndDate and, of course, the networkProtocolName and resourceFetchType properties.
-
Request: indicates the network request object.
-
/ *
* Represents the transaction request.
* /
@property (copy, readonly) NSURLRequest *request;Copy the code -
Response: represents the network response object. If the network fails or there is no response, Response is nil.
-
/ *
* Represents the transaction response. Can be nil if error occurred and no response was generated.
* /
@property (nullable, copy, readonly) NSURLResponse *response;Copy the code -
NetworkProtocolName: Network protocol used to obtain resources. Protocol identified by ALPN after negotiation, such as H2, HTTP /1.1, SPDY /3.1.
-
@property (nullable, copy, readonly) NSString *networkProtocolName;Copy the code -
IsProxyConnection: whether to use proxy for network connection.
-
/ *
* This property is set to YES if a proxy connection was used to fetch the resource.
* /
@property (assign, readonly, getter=isProxyConnection) BOOL proxyConnection;Copy the code -
IsReusedConnection: specifies whether to reuse an existing connection.
-
/ *
* This property is set to YES if a persistent connection was used to fetch the resource.
* /
@property (assign, readonly, getter=isReusedConnection) BOOL reusedConnection;Copy the code -
ResourceFetchType: NSURLSessionTaskMetricsResourceFetchType enumerated types, identify resources are loaded through the network, server push or local cache access.
-
/ *
* Indicates whether the resource was loaded, pushed or retrieved from the local cache.
* /
@property (assign, readonly) NSURLSessionTaskMetricsResourceFetchType resourceFetchType;Copy the code
For all of the following NSDate type indicators, all corresponding EndDate indicators will be nil if the task is not completed. For example, if DNS resolution times out, fails, or the client cancels before resolution succeeds, domainLookupStartDate has data, whereas domainLookupEndDate and all metrics after it are nil.
This diagram illustrates what each part of an HTTP request does
If you reuse an existing connection or fetch a resource from the local cache, the following metrics are assigned nil:
-
domainLookupStartDate
-
domainLookupEndDate
-
connectStartDate
-
connectEndDate
-
secureConnectionStartDate
-
secureConnectionEndDate
-
FetchStartDate: The time when the client starts the request, whether the resource is fetched from the server or from the local cache.
-
@property (nullable, copy, readonly) NSDate *fetchStartDate;Copy the code -
DomainLookupStartDate: DNS start time, Domain – > IP address.
-
/ *
* domainLookupStartDate returns the time immediately before the user agent started the name lookup for the resource.
* /Copy the code -
DomainLookupEndDate: indicates the time when DNS resolution is complete. The client has obtained the IP address corresponding to the domain name.
-
/ *
* domainLookupEndDate returns the time after the name lookup was completed.
* /Copy the code -
ConnectStartDate: indicates the time when the client starts to establish a TCP connection with the server.
-
/ *
* connectStartDate is the time immediately before the user agent started establishing the connection to the server.
*
* For example, this would correspond to the time immediately before the user agent started trying to establish the TCP connection.
* /
@property (nullable, copy, readonly) NSDate *connectStartDate;Copy the code -
SecureConnectionStartDate: HTTPS TLS handshake start time.
-
/ *
* If an encrypted connection was used, secureConnectionStartDate is the time immediately before the user agent started the security handshake to secure the current connection.
*
* For example, this would correspond to the time immediately before the user agent started the TLS handshake.
*
* If an encrypted connection was not used, this attribute is set to nil.
* /
@property (nullable, copy, readonly) NSDate *secureConnectionStartDate;Copy the code -
SecureConnectionEndDate: HTTPS TLS handshake over time.
-
/ *
* If an encrypted connection was used, secureConnectionEndDate is the time immediately after the security handshake completed.
*
* If an encrypted connection was not used, this attribute is set to nil.
* /
@property (nullable, copy, readonly) NSDate *secureConnectionEndDate;Copy the code -
ConnectEndDate: indicates the completion time of establishing the TCP connection between the client and the server, including the TLS handshake time.
-
/ *
* connectEndDate is the time immediately after the user agent finished establishing the connection to the server, including completion of security-related and other handshakes.
* /
@property (nullable, copy, readonly) NSDate *connectEndDate;Copy the code -
RequestStartDate: The time to start transmitting the first byte of the HTTP request’s header.
-
/ *
* requestStartDate is the time immediately before the user agent started requesting the source, regardless of whether the resource was retrieved from the server or local resources.
*
* For example, this would correspond to the time immediately before the user agent sent an HTTP GET request.
* /
@property (nullable, copy, readonly) NSDate *requestStartDate;Copy the code -
RequestEndDate: Specifies the time when the last byte of the HTTP request is transferred.
-
/ *
* requestEndDate is the time immediately after the user agent finished requesting the source, regardless of whether the resource was retrieved from the server or local resources.
*
* For example, this would correspond to the time immediately after the user agent finished sending the last byte of the request.
* /
@property (nullable, copy, readonly) NSDate *requestEndDate;Copy the code -
ResponseStartDate: The time at which the client receives the first byte of the response from the server.
-
/ *
* responseStartDate is the time immediately after the user agent received the first byte of the response from the server or from local resources.
*
* For example, this would correspond to the time immediately after the user agent received the first byte of an HTTP response.
* /
@property (nullable, copy, readonly) NSDate *responseStartDate;Copy the code -
ResponseEndDate: The time when the client received the last byte from the server.
-
/ *
* responseEndDate is the time immediately after the user agent received the last byte of the resource.
* /
@property (nullable, copy, readonly) NSDate *responseEndDate;Copy the code
Wrap up
IOS network monitoring can be implemented in two ways: NSURLProtocol and Code injection (Hook), this paper gives the specific implementation of monitoring through NSURLProtocol, Then it introduces how to use Method Swizzling, NSProxy and Fishhook to implement AOP Hook in iOS. The paper also presents three cases of AOP Hook technology in NSURLConnection, NSURLSession and CFNetwork. Finally introduced in iOS 10 new NSURLSessionTaskMetrics and NSURLSessionTaskTransactionMetrics class, they can be used to access the network related metadata, Such as the time spent on DNS queries, TLS handshakes, and request responses, these data can help developers better analyze network performance.
The resources
-
Mobile terminal performance monitoring scheme Hertz
-
NetworkEye
-
netfox
-
Share netease NeteaseAPM iOS SDK technology implementation
-
Mobile Application Monitor IOS component design technology sharing
-
A practical path to performance visualization
reading
IOS Performance Monitoring Solution Wedjat
Recommended reading
Construction of unified Error code management platform
Translation first! “HTTPS for Stack Overflow: The End of a Long Road”
“Front-end Developer Guide (2017)” shock!
