This article was written after I finished developing Tiercel 2.0, so it mentioned Tiercel 2.0. Now Tiercel has been developed to 3.0, the content in the article is still applicable, I will continue to update

Update 2020/1/22: iOS 13 resumeData structure, iOS 12.0-ios 12.2 resumeData Bug

The original

A long time ago, I discovered a problem I was going to face:

How can I download a bunch of files concurrently and then do anything else after all the files have been downloaded?

Of course, the problem is simple, and there are many solutions. But the first thing that came to my mind was, is there a task group download framework that is authoritative, popular, stable and reliable, written by Swift, and very popular on Github? If such a wheel exists, I intend to use it as a dedicated download module for the project. Unfortunately, there are many download frameworks, and there are many articles and demos in this field, but there are many famous authorities like AFNetworking and SDWebImage, there is really no star, and some of them are implemented by NSURLConnection. Even fewer were written in Swift, which gave me the idea of implementing one myself.

Ideal and reality

Since I want to lift wheels by myself, I can’t do it casually, and there is no authoritative and famous downloading framework. Therefore, AT the very beginning, I planned to do as much as possible to meet my own needs, and strive to use it for all the projects I am responsible for in the future. The first thing to meet is background download, as we all know, iOS App is suspended in the background, so to achieve background download, you need to follow Apple’s regulations, use URLSessionDownloadTask.

A web search turned up a ton of articles and demos, and I happily started coding. As a result, half of the implementation was found to be not as easy as the online article said. The test found that the open source wheel and Demo also had many bugs, incomplete, or did not fully implement the background download. So can only rely on their own to continue in-depth research, but at that time did not have this aspect of the study to more thorough article, and time is not allowed, must be as soon as possible off a wheel out of use. So finally I compromise, I used a relatively easy way to deal with, changed to use URLSessionDataTask implementation, although it is not native support background download, but I think there are always some dark ways can be implemented, finally I wrote Tiercel, a compromise to the reality of the download framework, but has met my needs.

Don’t forget the beginner’s mind

Since I didn’t really have a hard need for background downloads, I didn’t look for another way to do it, and I felt that if I did, I would have to use the URLSessionDownloadTask for native level background downloads. As time went by, I always felt it was a great pity that I did not complete my original idea, so I finally made up my mind to thoroughly study the background download of iOS.

Finally, Tiercel 2 was born with perfect support for native background downloads. Below I will explain in detail the implementation of background download and matters needing attention, hoping to help those in need.

The background to download

Regarding Background Downloading, We provide the documentation Downloading Files in the Background. However, we have to face many more problems in implementing this document.

URLSession

First, if you want to implement Background downloads, you must create Background Sessions

private lazy var urlSession: URLSession = {
    let config = URLSessionConfiguration.background(withIdentifier: "com.Daniels.Tiercel")
    config.isDiscretionary = true
    config.sessionSendsLaunchEvents = true
    return URLSession(configuration: config, delegate: self, delegateQueue: nil)
}()
Copy the code

The URLSession created this way is actually __NSURLBackgroundSession:

  • You must use thebackground(withIdentifier:)Method to createURLSessionConfigurationAnd this oneidentifierIt must be fixed and recommended to avoid conflicts with other appsidentifierWith the AppBundle IDrelated
  • createURLSessionMust be passed indelegate
  • It must be created when the App startsBackground SessionsIn other words, its life cycle is almost the same as that of App. For convenience, it is best to be used asAppDelegate, or global variables, for reasons explained later.

URLSessionDownloadTask

Only the URLSessionDownloadTask supports background downloads

let downloadTask = urlSession.downloadTask(with: url)
downloadTask.resume()
Copy the code

Through the Background Sessions created downloadTask, is actually __NSCFBackgroundDownloadTask

So far, tasks have been created and started to support background downloads, but now the real challenge begins

Breakpoint continuingly

Apple’s official documentation is —-Pausing and demodownloads

The resumable URLSessionDownloadTask relies on resumeData

// Save resumeData when canceling
downloadTask.cancel { resumeDataOrNil in
    guard let resumeData = resumeDataOrNil else { return }
    self.resumeData = resumeData
}

/ / or in the session the delegate urlSession (_ : task: didCompleteWithError:) method to get inside
func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
    if let error = error,
    	let resumeData = (error as NSError).userInfo[NSURLSessionDownloadTaskResumeData] as? Data {
        self.resumeData = resumeData
    } 
}

// Restore the download with resumeData
guard let resumeData = resumeData else {
    // inform the user the download can't be resumed
    return
}
let downloadTask = urlSession.downloadTask(withResumeData: resumeData)
downloadTask.resume()
Copy the code

Normally, this would have resumed the download task, but it didn’t, and resumeData had all sorts of problems.

ResumeData

In iOS, this resumeData is a weirdo, and if you’ve ever looked at it, it’s amazing, because it’s constantly changing and buggy, and it seems like Apple just doesn’t want us to work with it.

The structure of ResumeData

Before iOS 12, resumeData was saved locally as resumeData.plist, and you can see the structure inside.

  • On iOS 8, resumeData’s key:
// url
NSURLSessionDownloadURL
// The size of the received data
NSURLSessionResumeBytesReceived
// currentRequest
NSURLSessionResumeCurrentRequest
// Etag, the unique identifier of the downloaded file
NSURLSessionResumeEntityTag
// Path of cached files that have been downloaded
NSURLSessionResumeInfoLocalPath
/ / resumeData version
NSURLSessionResumeInfoVersion = 1
// originalRequest
NSURLSessionResumeOriginalRequest

NSURLSessionResumeServerDownloadDate
Copy the code
  • For iOS 9 to iOS 10, the changes are as follows:
    • NSURLSessionResumeInfoVersion = 2.resumeDataVersion update
    • NSURLSessionResumeInfoLocalPathtoNSURLSessionResumeInfoTempFileNameThe cache file path becomes the cache file name
  • In iOS 11, the changes are as follows:
    • NSURLSessionResumeInfoVersion = 4.resumeDataThe version is updated again, so it should have skipped 3 directly
    • Starting with iOS 11, downloadTask can be downloaded multiple timesCancel - RestoreOperation, generatedresumeDataI’m going to have one more keyNSURLSessionResumeByteRangeThe key/value pair
  • In the iOS 12,resumeDataChange the encoding mode, need to useNSKeyedUnarchiverTo decode it. The structure doesn’t change
  • In iOS, 13,NSURLSessionResumeInfoVersion = 5The structure has not changed

Understanding resumeData structure plays a key role in solving the bugs caused by it and realizing offline breakpoint continuation.

ResumeData的Bug

Not only does the structure of resumeData change all the time, it is also buggy all the time

  • On iOS 10.0 – iOS 10.1:
    • Bug: system generatedresumeDataThe download cannot be resumed directly becausecurrentRequestandoriginalRequest 的NSKeyArchivedCoding exception, fixed by iOS 10.2 and above.
    • Solution: ObtainresumeDataAfter, you need to modify it, and use the modifiedresumeDataCreate a downloadTask, and then downloadTaskcurrentRequestandoriginalRequestThe assignment,Stack OverflowThere are specific instructions.
  • On iOS 11.0 – iOS 11.2:
    • Bug: Due to multiple attempts on the downloadTaskCancel - RestoreOperation, generatedresumeDataI’m going to have one more keyNSURLSessionResumeByteRangeThis will result in a direct download success (it doesn’t) and the downloaded file size will be 0. IOS 11.3 and above will fix this.
    • Solution: change the key toNSURLSessionResumeByteRangeDelete key value pairs from.
  • On iOS 12.0 – iOS 12.2:
    • Bug: When a download task is started for the first time, it is used immediately without receiving any datacancel(byProducingResumeData:)Cancelling the task will result in an empty oneresumeData. Since no data has actually been received, it should not normally be generatedresumeDataIn other system versions it does notresumeData. If you use thisresumeDataAn error occurs when you resume the download
    • The solution: There are two ways:
      • Determine if there is a cache file, and since no data has actually been received, there is no cache file
      • Checks whether data has been received
  • On iOS 10.3 – Latest system version (iOS 13.3) :
    • Bug: Starting with iOS 10.3, as long as downloadTaskCancel - RestoreOperation, useresumeDataCreate a downloadTask, itsoriginalRequestNil, so far the latest system version (iOS 13.3) is still the same, although it does not affect the download of files, but it will affect the management of download tasks.
    • Solution: UsecurrentRequestThe matching task, which involves a redirection problem, will be explained later.

The above is the current summary of resumeData in different system versions of the changes and bugs, to solve the specific code can refer to Tiercel.

The specific performance

A downloadTask to support background downloads has been created, the resumeData issue has been resolved, and you can now happily start and resume downloads. The next step is to see how the downloadTask behaves, which is the most important part of implementing a download framework.

Downloading process

A great deal of time and effort has been spent testing the Performance of downloadTask in different situations, as follows:

operation create In the operation of the Suspend. Cancel (byProducingResumeData:) Cancel
An immediate effect Create the TMP file in the Caches folder of the App sandbox Write the downloaded data to TMP in the Caches folder TMP files in caches will not be moved The TMP file in the Caches folder is moved to the TMP folder, and didCompleteWithError is called The TMP file in the Caches folder is removed and didCompleteWithError is called
Into the background Automatic download Continue to download Nothing happened Nothing happened Nothing happened
Manually kill the App TMP file in caches folder will be removed when closed, session with same Identifier will be created when App is opened again, didCompleteWithError will be called (equal to cancel called) Download stops when closed, TMP file in caches folder will not be moved, re-open App to create session with same identifier, TMP file will be moved to TMP folder, It calls didCompleteWithError (equal to cancel(byProducingResumeData:)) The TMP file in the Caches folder will not be moved when closed, the same identifier session will be created when the App is opened again, and the TMP file will be moved to the TMP folder. It calls didCompleteWithError (equal to cancel(byProducingResumeData:)) Nothing happened Nothing happened
Crash or the system shuts down Download automatically opens, TMP file in caches folder will not be moved, download will continue after opening App regardless of whether there is a session to create the same identifier (remain downloaded) Continue downloading, TMP file in caches folder will not be moved, after opening App again, regardless of whether there is a session to create the same identifier, download will continue (remain downloaded) The TMP file in the Caches folder will not be moved, a session with the same identifier will not be created when the app is reopened, didCompleteWithError will not be called, and the task will still be stored in the session. The task is still paused and the download can be resumed Nothing happened Nothing happened

Support background download URLSessionDownloadTask, real type that is __NSCFBackgroundDownloadTask, embodied with ordinary there’s a big difference, and apple official document: according to the above form

  • When creating theBackground SessionsThe system will pick it upidentifierLog in as soon as the App is restarted after creating the correspondingBackground SessionsAnd its proxy methods will continue to be called
  • If the mission issessionManage, the TMP cache file in download will be in the caches folder of sandbox; If you don’t besessionThe cache files will be moved to the Tmp folder. If you don’t besessionManaged, and cannot be restored, the cache file will be deleted. That is:
    • DownloadTask runs and is invokedsuspendMethod, the cache file will be in the Caches folder of the sandbox
    • callcancel(byProducingResumeData:)Method, the cache files will be stored in the Tmp folder
    • callcancelMethod, the cache file is deleted
  • Manual Kill App will now be calledcancel(byProducingResumeData:)orcancelmethods
    • On iOS 8, the manual kill will be called immediatelycancel(byProducingResumeData:)orcancelMethod is then calledurlSession(_:task:didCompleteWithError:)Proxy method
    • On iOS 9 – iOS 12, manual kill will stop the download immediately and create the corresponding App after the App restartsBackground SessionsAfter that, the call will be madecancel(byProducingResumeData:)orcancelMethod is then calledurlSession(_:task:didCompleteWithError:)Proxy method
  • After entering the background, crashing or being shut down by the system, the system will have another process to manage the download task. The tasks that are not started will be automatically started, and those that have been started will remain in the original state (continue running or pause). When the App restarts, the corresponding process will be createdBackground Sessions, you can usesession.getTasksWithCompletionHandler(_:)Method, and session’s proxy method will continue to be called (if needed).
  • The most surprising thing is that, as long as there is no manual Kill App, even if the phone is restarted, the download task that was running will continue to download after the restart, which is awesome

Now that we have the pattern, it’s easy to deal with:

  • Created when the App startsBackground Sessions
  • usecancel(byProducingResumeData:)Method to suspend a task to ensure that the task can be resumed
    • You can actually use itsuspendIn iOS 10.0-ios 10.1, if a task is not resumed immediately after being paused, it will not be able to resume the task. This is another Bug, so it is not recommended
  • Manual Kill App will now be calledcancel(byProducingResumeData:)orcancel, and finally callsurlSession(_:task:didCompleteWithError:)Proxy methods can do centralized processing here, manage downloadTask, and putresumeDataSave up
  • Entering the background, crashing or being shut down by the system does not affect the status of the original task. When the App is restarted, create the correspondingBackground SessionsAfter usingsession.getTasksWithCompletionHandler(_:)To get the task

The download is complete

As background download is supported, the App may be in different states when the download task is completed, so it is necessary to know the corresponding performance:

  • In the foreground: Invoke the relevant Session proxy method as with a normal downloadTask
  • In the background: whenBackground SessionsIs called when all the tasks in it (note that all tasks, not just download tasks) are completeAppDelegatetheapplication(_:handleEventsForBackgroundURLSession:completionHandler:)Method, activate the App, then call the relevant Session proxy method as in the foreground, and finally call againurlSessionDidFinishEvents(forBackgroundURLSession:)methods
  • Crash or App is disabled by the system: WhenBackground SessionsAfter all tasks (note that all tasks, not just download tasks) are completed, the App will be automatically started and calledAppDelegatetheapplication(_:didFinishLaunchingWithOptions:)Method, and then callapplication(_:handleEventsForBackgroundURLSession:completionHandler:)Method when the correspondingBackground SessionsAfter that, the relevant session proxy methods are called and then called as in the foregroundurlSessionDidFinishEvents(forBackgroundURLSession:)methods
  • Crash or App is closed by the system, open the App to keep the foreground, and create the corresponding one after all tasks are completedBackground Sessions: is called only when no session is createdAppDelegatetheapplication(_:handleEventsForBackgroundURLSession:completionHandler:)Method when the correspondingBackground SessionsAfter that, the relevant session proxy methods are called and then called as in the foregroundurlSessionDidFinishEvents(forBackgroundURLSession:)methods
  • If crash or App is closed by the system, open the App and create the correspondingBackground SessionsThen all tasks are completed: just like at the front desk

Conclusion:

  • As long as it is not in the foreground, it will be called when all tasks are completedAppDelegatetheapplication(_:handleEventsForBackgroundURLSession:completionHandler:)methods
  • Only the corresponding is createdBackground SessionsThe corresponding session proxy method is called, or if it is not in the foregroundurlSessionDidFinishEvents(forBackgroundURLSession:)

Specific treatment methods:

The first is when Background Sessions are created, as described above:

The URLSession must be created when the App starts, meaning that its lifetime is almost the same as the App’s, preferably as an AppDelegate property or as a global variable for ease of use.

Reason: The download task may be completed when the App is in different states, so it is necessary to ensure that the Background Sessions have been created when the App starts, so that its proxy method can be called correctly and the subsequent operation is convenient.

Based on the performance of the download task at completion, combined with apple’s official documentation:

This method must be implemented in an AppDelegate
//
// -identifier: identifier corresponding to Background Sessions
// -CompletionHandler: Needs to be saved
func application(_ application: UIApplication,
                 handleEventsForBackgroundURLSession identifier: String,
                 completionHandler: @escaping (a) -> Void) {
    	if identifier == urlSession.configuration.identifier ?? "" {
            // Use the property as an AppDelegate to save the completionHandler
            backgroundCompletionHandler = completionHandler
	    }
}
Copy the code

And then want to be in the session proxy method calls completionHandler, it see: application (_ : handleEventsForBackgroundURLSession: completionHandler:)

// You must implement this method and call completionHandler on the main thread
func urlSessionDidFinishEvents(forBackgroundURLSession session: URLSession) {
    guard let appDelegate = UIApplication.shared.delegate as? AppDelegate.let backgroundCompletionHandler = appDelegate.backgroundCompletionHandler else { return }
        
    DispatchQueue.main.async {
        // The completionHandler saved above
        backgroundCompletionHandler()
    }
}
Copy the code

At this point, the download is complete

Download error

If the downloadTask that supports background downloads fails, In urlSession (_ : task: didCompleteWithError:) method (error as NSError.) inside the userInfo there may be a key For NSURLErrorBackgroundTaskCancelledReasonKey key-value pairs, which can obtain only have relevant Background download Task failure information, see: Background Task Cancellation

func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
    if let error = error {
        let backgroundTaskCancelledReason = (error as NSError).userInfo[NSURLErrorBackgroundTaskCancelledReasonKey] as? Int}}Copy the code

redirect

DownloadTask supports Background download. As the App may be in the Background, crash, or be closed by the system, it will be activated or started only when all Background Sessions tasks are completed, so it cannot handle redirection.

According to apple’s official documentation:

Redirects are always followed. As a result, even if you have implemented urlSession(_:task:willPerformHTTPRedirection:newRequest:completionHandler:), it is not called.

Mean still follow redirects, and will not call urlSession (_ : task: willPerformHTTPRedirection: newRequest: completionHandler:) method.

As mentioned above, originalRequest of downloadTask may be nil, and currentRequest can only be used to match the task for management, but currentRequest may also change due to redirection, and the proxy method of redirection will not be called. So you can only use KVO to observe currentRequest, so that you can get the latest currentRequest

Maximum concurrency

URLSessionConfiguration a httpMaximumConnectionsPerHost properties, its role is to control the amount of the same host connection at the same time, apple’s document, according to the default is 6 in the macOS, is 4 in the iOS. If set to N, at most N tasks can be downloaded concurrently from the same host, while other tasks are waiting, and tasks with different hosts are not affected by this value. But there’s actually a lot to be aware of.

  • There is no data to show what the maximum value of this URL is. After testing, it works fine if it is set to 1000000, but if it is set to int. Max, it will fail to download for most urls (should be related to the server of the target URL). If set to less than 1, it will not download for most urls
  • When usingURLSessionConfiguration.defaultTo create aURLSession, whether on a real machine or an emulator
    • httpMaximumConnectionsPerHostIf set to 10000, multiple tasks (over 180 tested) can be downloaded concurrently regardless of whether the host is the same
    • httpMaximumConnectionsPerHostIf this parameter is set to 1, only one task can be downloaded from a host at the same time. Multiple tasks can be downloaded concurrently from different hosts
  • When usingURLSessionConfiguration.background(withIdentifier:)To create one that supports background downloadsURLSession
    • On the simulator
      • httpMaximumConnectionsPerHostIf set to 10000, multiple tasks (over 180 tested) can be downloaded concurrently regardless of whether the host is the same
      • httpMaximumConnectionsPerHostIf this parameter is set to 1, only one task can be downloaded from a host at the same time. Multiple tasks can be downloaded concurrently from different hosts
    • On a real machine
      • httpMaximumConnectionsPerHostIf set to 10000, there is a limit on the number of concurrent download tasks regardless of whether the host is the same (currently up to 6)
      • httpMaximumConnectionsPerHostIf set to 1, only one task can be downloaded for the same host at the same time. There is a limit on the number of concurrent downloaded tasks for different hosts (currently, the maximum is 6).
      • Even if you use multipleURLSessionIf download is enabled, the number of tasks that can be concurrently downloaded does not increase
      • The following are the concurrency limits for some systems
        • 3 on iOS 9 and iPhone SE
        • 3 on iOS 10.3.3 and iPhone 5
        • 6 on iOS 11.2.5 and iPhone 7 Plus
        • 6 on iOS 12.1.2 and iPhone 6s
        • 6 on iOS 12.2 iPhone XS Max

It can be concluded from the above points that due to the URLSession feature that supports background downloads, the system will limit the number of concurrent tasks to reduce the cost of resources. At the same time for different host, even if httpMaximumConnectionsPerHost is set to 1, will have multiple concurrent download task, so you can’t use httpMaximumConnectionsPerHost to download task concurrency control. Tiercel 2 is concurrency control by determining the number of tasks being downloaded.

Front and background switching

When downloadTask is running, the App switches between front and back. Can cause urlSession (_ : downloadTask: didWriteData: totalBytesWritten: totalBytesExpectedToWrite:) method is not called

  • In iOS 12-ios 12.1, iPhone 8 or lower, the App enters the background and then returns to the foreground, and the proxy method of progress is not called. When entering the background again, the proxy method of progress will be called for a short period of time
  • In the simulator of iOS 12.1 and iPhone XS, the foreground and background switch was carried out for many times, and occasionally the proxy method of progress was not called, which could not be seen on the real computer
  • In iOS 11.2.2 and iPhone 6, if you switch between foreground and background, the proxy method of progress will not be called. If you switch multiple times, you will have a chance to recover

The above are the problems I found after testing some models. I did not cover all models. More cases can be tested by myself

Solution: use advice to monitor UIApplication didBecomeActiveNotification, 0.1 seconds delay call suspend method, method calls resume again

Matters needing attention

  • Sandbox path: Running and stopping the project with Xcode can achieve the effect of App crash. However, no matter in the real computer or the simulator, every time running with Xcode, the sandbox path will change, which will cause the system to fail to operate the files related to downloadTask. In some cases, the system records the sandbox path of the last project, resulting in the failure to open task download, unable to find folders and other errors. This is what happened to me in the beginning, and I didn’t know why, so I felt unpredictable and unsolvable. When you’re developing tests, be careful.
  • Real and emulator: Because there are so many features and considerations for iOS background downloads, and there are some differences between iOS versions, using emulators for development and testing is a convenient choice. However, some features will behave differently on the real machine and the emulator. For example, the number of concurrent tasks downloaded on the emulator is very large and the number of concurrent tasks downloaded on the real machine is very small (6 on iOS 12), so be sure to test or verify on the real machine.
  • Cached files: The recovery of downloads depends onresumeData, actually also need the corresponding cache file, inresumeDataYou can get the file name of the cache file in iOS 8 (you can get the cache file path in iOS 8), as previously recommendedcancel(byProducingResumeData:)Method to pause the task, the cache file is moved to the Tmp folder in the sandbox. This folder will be automatically cleaned up at some point, so it is best to save an extra copy just in case.

The last

If you’ve had the patience to read through the previous section, congratulations, you’ve seen all the features and considerations of iOS background downloads, and you’ve seen why there isn’t an open source framework to fully implement background downloads, because there are so many bugs and situations to deal with. This article is just my personal summary. There may be some problems or details that I haven’t found. If you have any new findings, please leave a message to me.

Tiercel 2 has been released at present, which perfectly supports background download and adds file verification and other functions. For more details, please refer to the code. Welcome to use, test, submit bugs and suggestions.