This paper is participating in theNetwork protocols must be known and must be known”Essay campaign

preface

HTTP is arguably the most common and frequently used network protocol in upper-layer application development. There are a lot of articles on the web, but most of them are about the content and use of HTTP, and very few people talk about how HTTP is implemented. Network protocol can involve a great breadth and depth, not an article can explain clearly, I here is more to provide a train of thought for readers to think. This article will be based on the iOS platform, but it does not mean that this article is only for iOS development, because the protocol is cross-platform and the programming ideas involved are as well. This paper will be explained in four parts:

  1. Part one: How data is transferred over the network. This part mainly makes you think aboutA network modelHave a basic understanding of the layer protocols. If you are familiar with this part, you can start with Part 2.
  2. Part II:HTTPHow is protocol data converted toTCPData sent and received.
  3. Part III:HTTPIn the agreementRequestandResponseAnalysis and related logic processing.
  4. Part FOUR: ModificationHTTPThe underlying implementation completes its own requirements.

How is data transferred over the network

Data is sent and received according to network protocol, and when you talk about network protocol, you can’t get around itThe OSI modelandTCP/IP reference modelThey have different levels,The OSI modelDivided into seven layers,TCP/IP reference modelDivided into 4 layers. There are many generals on the InternetTCP/IP reference modelMap toThe OSI modelAs a result ofTCP/IP reference modelandThe OSI modelCan’t match exactly, there is not a completely correct, or authoritative answer, the general thought of the corresponding relationship is shown as follows:HTTP protocol is at the top levelThe application layer.A network modelIs relatively abstract, in the actual coding, the upper application developers generally only touch the application layer, developers only need to put oneHTTP RequesIt’s thrown into the network frame, and one is returned when the request is completeHTTP ResponseBut how is the underlying class implemented? Let’s start with the following image:In the figure above, we can see how HTTP data interacts between the client and server. The network model is very complex, but in a way, it can be described as a “nesting doll”RFC 1122The encapsulation decrement for applying data along different layers described in:The top layer in the image aboveDataData represents application layer protocol dataHTTPIn the agreement,HTTPtherequestA message andresponseAll packets containheader.TCPandIPThere are alsoheaderThey are sent through layers of nesting dolls.

Now we have a little bit of an overall idea of how data travels over the network, but the details are unclear. As can be seen from the above content, HTTP data is converted to TCP data for transmission. The content of the transport layer and below is not explained here. Here mainly describes how HTTP data of the application layer is transmitted through the transport layer, and how to parse.

HTTPHow is protocol data converted toTCPData receiving

Generally speaking, each system provides users with HTTP network framework, such as NSURLSession of iOS. On the HTTP network framework of the system, the developer community develops various easy-to-use packages, such as AFNetworking. For the upper level developers, the use of HTTP protocol is generally a Request object wrapped by the framework, or even just a URL. After the completion of the Request using the framework, a Response object is returned, and its underlying implementation is hidden.

As we all know, the bottom of the computer is binary, data transmission is no exception. To transfer the Request object from the host to the server, it must be converted to binary, so how is it converted? How is it transmitted?

How is the HTTP protocol converted to binary?

The Request object of the network framework is encapsulated for ease of use. To transmit it, it must be converted into binary data: Request object -> Request string that complies with HTTP protocol -> binary data.

HTTP request packets:Response message:

Following the Request message format shown in the figure, we can convert the Request object into an HTTP compliant string and convert it into a byte stream. The sample code is as follows:

/ / / to create NSURLRequest
NSURL *url = [NSURL URLWithString:@"https://www.baidu.com"];
NSURLRequest *request = [[NSURLRequest alloc] initWithURL:url];

// Convert NSURLRequest to binary data (only POST and GET requests are described here)
+ (NSData *)httpRequestDataWithRequest:(NSURLRequest *)request {
    NSMutableString * requestStrFrmt = [NSMutableString string];
    NSURL * url = request.URL;
    NSString *requestURI = url.path;
    // Parse the request line
    if ([request.HTTPMethod isEqualToString:@"POST"]) {
        if(! url.path || url.path.length ==0) {
            requestURI = @ "/"; }}else if ([request.HTTPMethod isEqualToString:@"GET"]) {
        if (url.path.length > 0  && url.query.length > 0) {
            requestURI = [NSString stringWithFormat:@ % @ "? % @", url.path, url.query];
        } else if (url.path.length > 0) {
            requestURI = url.path;
        } else if (url.query.length > 0) {
            requestURI = url.query;
        } else {
            requestURI = @ "/";
        }
    }
    
    [requestStrFrmt appendFormat:@ % @ % @ "HTTP / 1.1 \ r \ n", request.HTTPMethod, requestURI];
    if ([request.allHTTPHeaderFields objectForKey:@"Host"] = =nil) {
        [requestStrFrmt appendFormat:@"Host: %@\r\n", url.host];
    }
    
    // Parse the request header
    for (NSString * key in request.allHTTPHeaderFields.allKeys) {
        [requestStrFrmt appendFormat:@"%@: %@\r\n", key, request.allHTTPHeaderFields[key]];
    }
    
    // Parse the request data (body)
    if ([request.HTTPMethod isEqualToString:@"POST"] && request.HTTPBody) {
        [requestStrFrmt appendFormat:@"Content-Length: %@\r\n", @(request.HTTPBody.length)];
        // The request header ends with two CRLFS
        [requestStrFrmt appendString:@"\r\n"];
        NSData *headerData = [requestStrFrmt dataUsingEncoding:NSUTF8StringEncoding];
        NSMutableData * requestData = [NSMutableData dataWithData:headerData];
        [requestData appendData:request.HTTPBody];
        return requestData;
    } else {
        // The request header ends with two CRLFS
        [requestStrFrmt appendString:@"\r\n"];
        return [requestStrFrmt dataUsingEncoding:NSUTF8StringEncoding]; }}// prints the text data of the NSURLRequest message
NSData *data = [NSData httpRequestDataWithRequest:request];
NSString *requestText = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
NSLog(@ "% @", requestText);
Copy the code

Call Through the above code, we can get the text data of the Request message. Sample:

GET/HTTP/1.1 Host: www.baidu.comCopy the code

The response message can be analyzed by referring to the Request message. To avoid being too long, no explanation is given here.

How is HTTP binary data transferred?

In each operating system, applications are usually provided with a set of application program interfaces, called socket apis, whose main function is to implement interprocess communication and network programming. In plain English: a socket is an application development library written in C, and it is a library.

Network sockets in sockets, including streaming sockets (SOCK-STREAM), it usesTCPProtocol to implement byte stream transmission. throughsocketThe framework will containHTTPThe data ofTCPThe byte stream is sent to the server, which passessocketFrame get containHTTPThe data ofTCPAfter byte stream, according toHTTPThe protocol is parsed and then returned by the HTTP network framework of the server, as shown below:The above illustration contains onlyrequestThe packet part does not containresponsePart of the packet due toresponseThe data transmission of a message is not very different from this, so I won’t elaborate on it here.

HTTPIn the agreementRequestandResponseAnalysis and logical processing of

At this point, we should have a clearer picture of the HTTP implementation, but there are a few details that need to be added. The students who have used Socket should know that it is based on TCP streaming transmission, there will be half packet and sticky packet problems. These problems are usually solved by adding headers to the data. As we know, HTTP data contains two parts, namely Header and Body. The HTTP protocol defines two CRLF between Header and Body, one CRLF is a carriage return and a newline: \r\n. With this identifier, we can separate the HEADERS of HTTP data from the TCP stream. Then parse the content-Length field of the Header, which is the Length of the body.

In HTTP/1.1, Body parsing is also related to the Transfer-Encoding field, which is not discussed here.

Of course, HTTP protocol contains not only the data parsing part, but also a lot of logical control parts. Its response header and request header have many control fields, such as cache related Etag, last-Modified, etc., and data compression related content-encoding, accept-encoding, etc. The network framework of the system implements the logic of these control fields so that users can use them out of the box.

Modify theHTTPThe underlying implementation completes its own requirements.

Changes to the upper layers of HTTP are common, such as YTKNewwork, which adds custom cache logic on top of THE HTTP protocol and controls the cache time through the cacheTimeInSeconds method. However, changes to the underlying LAYER of HTTP are less common, and my understanding of this part is based on a special requirement.

We all know that mobile phones can send and receive data over WiFi or cellular channels, but generally, when both are connected, routing will make the traffic go only over the WiFi channel. However, for some WiFi connection tools, data acquisition needs to be carried out under WiFi that cannot access the Internet to meet the requirements of WiFi authentication. In this case, the cellular network can access the network, so can HTTP requests not go through the default WiFi channel, and request data through the cellular network? The upper-layer HTTP network framework does not have this function, but the underlying socket framework does, which allows data to be sent and received from a specific interface regardless of the route. We can completely on socket, HTTP protocol in the request,response parsing and logical processing, to achieve the support of this function. Of course full HTTP support is unaffordable, but a simple implementation that meets your own needs is possible. I packaged this functionality into a framework: XXSocketReqeust, used as follows:

_manager = [[XXSocketRequestManager alloc] init];
NSURLRequest *request = [[NSURLRequest alloc] initWithURL:[NSURL URLWithString:@"https://www.baidu.com"]].Use XXNetworkInterfaceCellular / / /, the HTTP request will ignore routing, mandatory cellular network channels to request.
XXSocketDataTask *task = [_manager dataTaskWithRequest:request viaInterface:XXNetworkInterfaceCellular completionHandler:^(NSURLResponse * _Nonnull response, id  _Nullable responseObject, NSError * _Nullable error) {
    NSLog(@"error is :%@\n response is %@", error, response);
    NSLog(@"responseObject: %@"The [[NSString alloc] initWithData:responseObject encoding:NSUTF8StringEncoding]);
}];
[task start];
Copy the code

If you’re interested, you can download it.

Afterword.

A lot of times network protocols are cold, common network protocols are very complex in order to be common and meet various requirements. But it’s not as difficult as you might think to tailor protocols to your own business, or tweak them to suit your own needs.

Personal blog – Star sky

The resources

  • XXSocketReqeust
  • An article that gives you a thorough understanding of the structure of HTTP request and response messages
  • Reinforce your HTTP body of knowledge
  • Network socket
  • Internet Protocol Suite
  • OSI 7 layer model and TCP/IP 4 layer model