Image Source:
https://unsplash.com


The author:
Xie Fugui

background

WebView is ubiquitous in mobile applications and serves as an entry point for many core services in cloud music. In order to meet the increasingly complex business scenarios of cloud music, we have been continuously optimizing the performance of WebView. One of the technologies that can improve the loading speed of WebView in a short time is the offline package technology. This technology can save network loading time, especially for large pages. The most critical link in the offline package technology is to intercept the request sent by WebView and map resources to the local offline package. However, for the request intercept of WKWebView, the iOS system does not provide direct capability. Therefore, this paper will focus on the WKWebView request intercept for discussion.

research

We have studied the existing WKWebView request interception schemes in the industry, which are mainly divided into the following two types:

By default, NSURLPROTOCOL intercepts all requests that pass through the URL Loading System, so any request from a WKWebView that passes through the URL Loading System can be intercepted. After our attempt, we found that WKWebView runs independently of the application process, and the request sent by default will not go through the URL Loading System, so we need additional hooks to support it. For specific methods, we can refer to the processing of WKWebView by NSURLProtocol.

WkurlSchemeHandler WkurlSchemeHandler is a new feature introduced in iOS 11 that handles data management for custom requests. If you want to support the data management of Scheme for HTTP or HTTPS requests, you need the handlesUrlScheme: method of Hook WKWebView, and then return NO. After some attempts and analysis, we compare the two schemes from the following aspects:

  • Isolation:NSURLProtocolOnce registered, it is globally open. Generally we only block our own business pages, but use itNSURLProtocol“In a way that would result in the in-app collaboration of three party pages being blocked and contaminated.WKURLSchemeHandlerYou can isolate by page as a dimension, because you are followingWKWebViewConfigurationConfigure.
  • Stability:NSURLProtocolThe Body will be lost during interception,WKURLSchemeHandlerBefore iOS 11.3 (not included) the Body will also be lost, after iOS 11.3 WebKit optimization will only lose Blob type data.
  • Consistency:WKWebViewThe request made isNSURLProtocolAfter interception, the behavior may change. For example, if a video is loaded to cancel the video tag, it usually sets the resource address (SRC) to emptystopLoadingMethod is not called, by contrastWKURLSchemeHandlerAct normal.

The conclusion of the survey is:WKURLSchemeHandlerIt performs better in isolation, stability and consistencyNSURLProtocol, but to be used in a production environment you have to solve the missing Body problem.

Our solution

It can be seen from the above that only throughWKURLSchemeHandlerRequest interception does not cover all request scenarios because the Body is missing. Therefore, our research focus is to ensure how not to lose the Body data or get the Body data in advance and then assemble it into a complete request. Obviously, the former needs to change the WebKit source code, which costs too much, so we choose the latter. By modifying JavaScript nativeFetch / XMLHttpRequestWait for interface implementation to get the BODY data in advance. The scheme design is shown in the figure below:



The specific process is mainly as follows:

  • Inject a custom when loading an HTML documentFetch / XMLHttpRequestObject script
  • Before sending the request, collect the Body and other parameters throughWKScriptMessageHandlerPass it to a native application for storage
  • Notifications of calling agreed JavaScript functions when native application storage is completeWKWebViewSave your
  • Call nativeFetch / XMLHttpRequestWait for the interface to send the request
  • The request isWKURLSchemeHandlerManagement, take out the corresponding parameters such as Body to assemble and then send out

Script injection

Replace FETCH implementation

Script injection needs to modify the processing logic of the FETCH interface so that parameters such as the BODY can be collected and passed to the native application before the request is sent. The main problems solved are as follows:

  • Body loss before iOS 11.3
  • The Body after iOS 11.3BlobType data loss problem
  1. For the first point, you need to determine whether a request made by a device prior to iOS 11.3 contains a request body, and if so, call nativeFetchThe interface needs to collect the request body data and pass it to the native application.
  2. As for the second point, it is also important to determine whether a request made by a device after iOS 11.3 contains a request body and whether it is contained in the request bodyBlobType data, if satisfied, the same process as above.

Otherwise, you simply call the native FETCH interface directly, leaving the native logic intact.

var nativeFetch = window.fetch var interceptMethodList = ['POST', 'PUT', 'PATCH', 'DELETE']; Fetch = function(url, opts) {var hasBodyMethod = opts! = null && opts.method ! = null && (interceptMethodList.indexOf(opts.method.toUpperCase()) ! = = 1); Var shouldSaveParamstonative = islessThan11_3; if (hasBodyMethod) {// ShouldSaveParamstonative = islessThan11_3; if (! ShouldSaveParamstonative = opts!) {shouldSaveParamstonative = opts! = null ? isBlobBody(opts) : false; } if (shouldSaveParamstonative) {return saveParamstonative (url, shouldSaveParamstonative) {return saveParamstonative (url, shouldSaveParamstonative) {return saveParamstonative (url, shouldSaveParamstonative); Opts).then(function (newUrl) {return nativeSetch (newUrl, opts)}); }} // call the nativeFetch interface return nativeSetch (url, opts); }

Save the request body data to the native application

The WKScriptMessageHandler interface allows you to save the request body data to the native application, and you need to generate a unique identifier corresponding to the specific request body data for subsequent retrieval. The idea is to generate a standard UUID as an identifier and then pass it along with the request body data to the native application for storage, and then concatenate the UUID into the request link, After the request is managed by WkurlSchemeHandler, the identifier is used to retrieve the specific request body data and then assemble the request.

function saveParamsToNative(url, opts) { return new Promise(function (resolve, Var identifier = generateUUID(); reject) {var identifier = generateUUID(); var appendIdentifyUrl = urlByAppendIdentifier(url, "identifier", Identifier) // Parse the body data and save it to the original application if (opts && opts.body) {getBodyString(opts. Body, function(body) {// Set the save to the final callback, FinishSaveCallbacks [Identifier] = function() {resolve(AppendifyUrl)} // Notify the native application to save the request body data window.webkit.messageHandlers.saveBodyMessageHandler.postMessage({'body': body, 'identifier': identifier}}) }); }else { resolve(url); }}); }

Request body resolution

In the FETCH interface, you can get the request body parameter through the second OPTS parameter, namely, OPTS.BODY. Referred to the MDN FETCH BODY, it can be known that there are seven types of request body. After analysis, these seven data types can be divided into three types for parsing and coding processing. ArrayBuffer, ArrayBufferView, BLOB and File are classified as binary types, while String and UrlSearchParams are classified as string types. FormData is classified as a compound type and is eventually converted to a string type and returned to the native application.

function getBodyString(body, callback) { if (typeof body == 'string') { callback(body) }else if(typeof body == 'object') { if (body instanceof ArrayBuffer) body = new Blob([body]) if (body instanceof Blob) {// Base64 var reader = new FileReader() reader.addEventListener("loadend", function() { callback(reader.result.split(",")[1]) }) reader.readAsDataURL(body) } else if(body instanceof FormData) { generateMultipartFormData(body) .then(function(result) { callback(result) }); } else if(body instanceof urlSearchParams) {var resultArr = [] for (pair of body.entries())  { resultArr.push(pair[0] + '=' + pair[1]) } callback(resultArr.join('&')) } else { callback(body); } }else { callback(body); }}

Binary types are uniformly converted to Base64 encoding for transmission. The string type URLSearchParams is traversed to get the key-value pairs. The compound type storage structure is like a dictionary, and the values may be of type String or Blob, so you need to traverse and then concatenate them in the Multipart/form-data format.

other

The main content of the injected script is shown above. In the example, only the implementation of FETCH is replaced, and XMLHttpRequest is replaced in the same way. Cloud music due to the lowest version supports to the iOS 11.0, while the FormData. Prototype. Entries are only after the iOS 11.2 support, and for the previous version can modify the FormData. The prototype. The realization of the set method to save key-value pairs, I won’t go into details here. In addition, the request may be made from an embedded iframe, and calling FinishSaveCallBacks [Identifier]() directly is not valid because FinishSaveCallBacks is mounted on the Main Window. Consider using the window.postMessage method to communicate with child Windows.

WkurlSchemeHandler intercepts the request

The registration and use of WkurlSchemeHandler will not be described here. For details, please refer to the research section above and the Apple documentation. Here we will talk about the points to be aware of during the interception process

redirect

Some readers may have noticed that when we introduced WkurlSchemeHandler in the research section above, we defined its role as data management for custom requests. So why not custom request data interception? In theory, interception does not require the developer to care about the request logic, the developer can only handle the data in the process. For data management, the developer needs to focus on all the logic in the process, and then return the final data. With these two definitions in mind, let’s compare the WKURLSchemeTask and NSURLProtocol. We can see that the latter has more request processing logic, such as redirection and authentication, than the former.

API_AVAILABLE(macOS (10.13), iOS (11.0)) @protocol wkurlschemeTask <NSObject> @property (nonatomic, readonly, copy) NSURLRequest *request; - (void)didReceiveResponse:(NSURLResponse *)response; - (void)didReceiveData:(NSData *)data; - (void)didFinish; - (void)didFailWithError:(NSError *)error; @end
API_AVAILABLE (macos (10.2), the ios (2.0), watchos (2.0), TvOS (9.0)) @protocol NSURLProtocolClient <NSObject> - (void)URLProtocol:(NSURLProtocol *)protocol didReceiveResponse:(NSURLResponse *)response cacheStoragePolicy:(NSURLCacheStoragePolicy)policy; - (void)URLProtocol:(NSURLProtocol *)protocol didLoadData:(NSData *)data; - (void)URLProtocolDidFinishLoading:(NSURLProtocol *)protocol; - (void)URLProtocol:(NSURLProtocol *)protocol didFailWithError:(NSError *)error; - (void)URLProtocol:(NSURLProtocol *)protocol didReceiveAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge; - (void)URLProtocol:(NSURLProtocol *)protocol didCancelAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge; @end

So how do you handle the redirect response during interception? We try to call DidReceiveResponse every time we get a response: In this way, WKWebView will not be aware of the redirection, so it will not change the address and other relevant information. It may bring some unexpected effects to some pages with judged routing. At this point we get stuck again, and you can see that WkurlSchemeHandler does not support redirection when fetching data, because Apple originally designed it as pure data management. We can actually get the response every time, but we just can’t pass it to WKWebView in its entirety. After some consideration, we finally chose reload to solve the redirection problem of HTML document requests for three reasons.

  • All that can be modified at the momentFetch 和 XMLHttpRequestInterface implementation, for document requests and HTML tag requests are browser internal behavior, modify the source code is too costly.
  • Fetch 和 XMLHttpRequestBy default, only the final response is returned, so ensuring that the final data is correct at the server-side interface level does not affect the loss of the redirect response.
  • Images/videos/forms/stylesheets/scripts and other resources generally need only the final data to be correct.

Receiving the redirect response of the HTML document is returned directly to the WKWebView and the subsequent loading is cancelled. The redirection of other resources is discarded.

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task willPerformHTTPRedirection:(NSHTTPURLResponse *)response newRequest:(NSURLRequest *)request completionHandler:(void (^)(NSURLRequest * _Nullable))completionHandler { NSString *originUrl = task.originalRequest.URL.absoluteString; if ([originUrl isEqualToString:currentWebViewUrl]) { [urlSchemeTask didReceiveResponse:response]; [urlSchemeTask didFinish]; completionHandler(nil); }else { completionHandler(request); }}

WKWebView after receiving the response data is called a webView: decidePolicyForNavigationResponse: decisionHandler method to determine the final jump, In this method, you can get the redirected destination Location and reload it.

- (void)webView:(WKWebView *)webView decidePolicyForNavigationResponse:(WKNavigationResponse *)navigationResponse DecisionHandler (void (^) (WKNavigationResponsePolicy)) decisionHandler {/ / opens the intercept the if (enableNetworkIntercept) {if ([navigationResponse.response isKindOfClass:[NSHTTPURLResponse class]]) { NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)navigationResponse.response; NSInteger statusCode = httpResp.statusCode; NSString *redirectUrl = [httpResp.allHeaderFields stringForKey:@"Location"]; if (statusCode >= 300 && statusCode < 400 && redirectUrl) { decisionHandler(WKNavigationActionPolicyCancel); / / does not support 307, 308 post jump scene [webView loadHTMLWithUrl: redirectUrl]; return; } } } decisionHandler(WKNavigationResponsePolicyAllow); }

This is the end of the HTML document redirection problem for the most part. We haven’t found any boundary issues until this article was published, but feel free to talk about any other good ideas you have.

Cookies are synchronous

Since WKWebView and our application are not in the same process, WKWebView and NSHTTPCookieStorage are out of sync. This article will not cover the entire process of WKWebView Cookie synchronization, but will focus on the synchronization of cookies during intercepting. Since the request is ultimately made by the native application, the Cookie is read and stored using NSHTTPCookieStorage. It is worth noting that the response that WKSchemeHandler returns to WKWebView contains the Set-Cookie information, but WKWebView is not Set to the document. Cookie. It is also true that WkurlSchemeHandler is only responsible for data management, and the logic involved in the request needs to be handled by the developer. Cookie synchronization of WKWebView can be achieved through the WKHttpCookiestore object

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveResponse:(NSURLResponse *)response completionHandler:(void (^)(NSURLSessionResponseDisposition))completionHandler { if ([response isKindOfClass:[NSHTTPURLResponse class]]) { NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)response; NSArray <NSHTTPCookie *>*responseCookies = [NSHTTPCookie cookiesWithResponseHeaderFields:[httpResp allHeaderFields] forURL:response.URL]; if ([responseCookies isKindOfClass:[NSArray class]] && responseCookies.count > 0) { dispatch_async(dispatch_get_main_queue(), ^{ [responseCookies enumerateObjectsUsingBlock:^(NSHTTPCookie * _Nonnull cookie, NSUInteger idx, BOOL * _Nonnull stop) {// Synchronize to WKWebView [[WKWebSiteDataStore DefaultDatastore]. HttpCookiestore SetCookie: Cookie completionHandler:nil]; }]; }); } } completionHandler(NSURLSessionResponseAllow); }

In addition to synchronizing the Cookie of the native application to WKWebView during the interception process, the document. Cookie should also be synchronized to the native application when modifying the document. After trying to find that document.cookie on the real device will actively delay synchronization to NSHTTPCookieStorage after modification, but the simulator did not do any synchronization. For some requests that are sent immediately after the document. Cookie is modified, the changed cookie information may not be carried immediately, because the cookie will go to NSHTTPCookieStorage after the interception. Our solution is to modify the implementation of the document.cookie setter method and synchronize to the native application before the cookie setting is completed. Note that native applications need to do cross-domain verification at this time to prevent malicious pages from arbitrarily modifying cookies.

(function() { var cookieDescriptor = Object.getOwnPropertyDescriptor(Document.prototype, 'cookie') || Object.getOwnPropertyDescriptor(HTMLDocument.prototype, 'cookie'); if (cookieDescriptor && cookieDescriptor.configurable) { Object.defineProperty(document, 'cookie', { configurable: True, enumerable: true, set: function (val) {/ / set to come into force when first transmitted to native applications window. Its. MessageHandlers. Save. PostMessage (val). cookieDescriptor.set.call(document, val); }, get: function () { return cookieDescriptor.get.call(document); }}); }}) ()

Memory leak caused by NSURLSession

Through NSURLSession sessionWithConfiguration: delegate: delegateQueue constructor to create the object when the delegate is NSURLSession strong references, this is easy to ignore you. We will create an NSURLSession object for each WKurlSchemeHandler object and set the former to the delegate of the latter, resulting in a circular reference. It is recommended that the invalidateAndCancel method of NSURLSession be called at WKWebView destruction to remove the strong reference to the WKSchemeHandler object.

Stability improvement

It can be seen from the above that if we “work against” the system (WKWebView itself does not support HTTP/HTTPS request interception), there will be a lot of unexpected things happen, and there may be a lot of boundary areas to cover, so we have to have a set of perfect measures to improve the stability of the interception process.

Dynamic distributed

We can turn off some page blocking by dynamically Posting the blacklist. By default, cloud music will preload two empty WKWebView, one is a WKWebView registered with WKURLSchemeHandler to load the main site page, and supports blacklist shutdown. The other is the normal WKWebView to load some tripartite pages (because the logic of tripartite pages is more diverse and complex, and there is no need to intercept the request of tripartite pages). In addition, for some teams who just try to solve the loss of request body through script injection, they may not be able to cover all the scenarios. They can try to update the script by dynamically issuing, and they should also sign the script content to prevent malicious tampering by others.

monitoring

Log collection can help us better identify potential problems. All request logic during interception is consolidated in the WkurlSchemeHandler so that log collection can be performed on some critical links. For example, you can collect if the injected script executes an exception, if the Body received is missing, if the response status code returned is normal, and so on.

Full proxy request

In addition to the above measures, we can also delegate the network request such as the server API interface to the client. The front-end simply passes the corresponding parameters to the native application via JSBridge and then fetches the data through the native application’s network request channel. In addition to reducing potential problems during interception, this approach can also reuse native application network-related capabilities such as HTTP DNS, anti-cheating, etc. Moreover, it is worth noting that Apple has enabled ITP (Intelligent Tracking Prevention) Intelligent Tracking function in WKWebView by default in IOS 14, which is mainly affected by the use of cross-domain cookies and Storage. For example, some third party pages in our application need to be embedded in an iframe to achieve authorization. At this time, because the cross-domain default is not to obtain the Cookie under the domain name of our master site, similar problems can be solved if we use the proxy request of native application. Finally, if you use this method, remember to do a good authentication check to prevent malicious pages from using this ability. After all, native applications have no cross-domain restrictions on requests.

summary

By combining iOS native WkurlSchemeHandler with JavaScript script injection, this paper realizes the request interception capability required by WKWebView in offline packet loading, stream free and other services. It solves the possible problems of redirection, request body loss, Cookie asynchronization and so on in the process of interception and can intercept and isolate with page as dimension. In the process of exploration, we increasingly realize that technology has no boundaries. Sometimes, due to some limitations of the platform, a complete set of capabilities cannot be realized by one party alone. Only by combining the technical capabilities of relevant platforms can a reasonable technical scheme be formulated. Finally, this article is some of our explorations in the WKWebView request interception practice, if there are any errors welcome to correct and communicate.

This article is posted from
Netease cloud music big front end teamThis article is not allowed to be reproduced in any form without authorization. We recruit front-end, iOS, and Android all year round. If you’re ready for a career change and you happen to like cloud music, then join us. Grp.music-fe (at)corp.netease.com!