One, the introduction

The core of big data analysis is data. We not only need to collect data, but also need to upload the data to the specified server. Then, after storage, extraction, analysis and presentation by the server, the real value of data can be fully brought into play.

Shence analyzed iOS SDK for data transmission and designed and implemented a network transmission scheme suitable for data collection from the aspects of integrity, correctness and efficiency.

The following is the analysis of iOS SDK network module for Divine strategy, hoping to provide you with some reference.

Second, network request scheme

There are many ways to realize network request in iOS. For example, The official Network request API provided by Apple; Or some open source web framework. The advantages and disadvantages of the two schemes are described below.

2.1 Apple official API

Network requests based on apple’s official apis generally use NSURLSession or NSURLConnection. However, the interface in which NSURLConnection sends network requests has been marked expired as of iOS 9.0. Therefore, developers are generally in the NSURLSession camp. Let’s take a look at the pros and cons of NSURLSession.

The network request scheme based on NSURLSession has the following advantages [1] :

  1. Connection based authentication scheme;
  2. Support HTTP request configuration;
  3. Privatized stored objects;
  4. Support background upload and download.

However, there are also the following disadvantages:

  1. Some open source networking frameworks are not as easy to use, such as building complex network request and response processing;
  2. Some details are not encapsulated and need to be handled by themselves.

2.2 Open source Network framework

The network request scheme based on open source network framework has the following advantages:

  1. Mainstream open source network frameworks are rich in functions and relatively stable. For example, AFNetworking;
  2. It can realize the function of network request easily and quickly.

However, there are also the following disadvantages:

  1. More functions, complex code logic, high learning cost;
  2. Internal defects are difficult to repair and even need to rely on the author to update maintenance;
  3. Contains many features that may not be used;
  4. The introduction of open source network frameworks resulted in increased volume.

It has advantages and disadvantages to realize network request scheme based on open source network framework.

SDK network module

If the SDK network module is implemented based on the open source network framework, the customer integrates the SDK and uses the same open source network framework, causing conflicts. In addition, the maintenance of the open source network framework may lead to risks such as delayed version update and difficulty in troubleshooting.

Because of these disadvantages based on open source network framework, SDK network module is implemented by NSURLSession.

NSURLSession is the network access API provided by the system, which can not only meet the needs of SDK network requests, but also has stable functions and is easy to expand.

3.1 Implementation Principle

NSURLSession can create one or more instances, each of which coordinates a related set of data transfer tasks [1]. For example, when creating a Web browser, an App might create a session for each TAB or window. One session is for user interaction and the other is for background download. Within each session, the App adds a series of tasks, each representing a request for a specific URL.

3.2 Specific Implementation

3.2.1. Network configuration

The SDK can carry out a series of configurations for data sending, and developers can set corresponding configurations according to specific requirements to achieve efficient data sending effect.

The SDK configuration is completed during initialization. You can configure the following parameters:

  1. ServerURL: Data sending address. Collected data is sent to this address.
  2. FlushInterval: specifies the minimum interval (in milliseconds) between sending data. The default value is 15000 milliseconds.
  3. FlushBulkSize: the minimum number of cache entries that can be sent twice. When the number of local cache entries reaches flushBulkSize, data will be sent. The default value is 100.
  4. SecurityPolicy: configures SSL certificates and self-signed certificates.
  5. FlushNetworkPolicy: network policy for sending data.

3.2.2. Data sending thread encapsulation

The SDK data is sent in the child thread. When the collected data meets the sending policy, the asynchronous sending is triggered, and the upload task is completed in the SAHTTPSession class. When initializing the SDK, an instance of SAHTTPSession is created and NSOperationQueue is instantiated to send events through multiple threads. The specific implementation code is as follows:


- (void)requestWithRecords:(NSArray<SAEventRecord *> *)records completion:(void (^)(BOOL success))completion {
    [SAHTTPSession.sharedInstance.delegateQueue addOperationWithBlock:^{
            ......
        // Network request callback processing
        SAURLSessionTaskCompletionHandler handler = ^(NSData * _Nullable data, NSHTTPURLResponse * _Nullable response, NSError * _Nullable error) {
            ......
        };
 
        // Convert to the body of the sent HTTP
        NSData *HTTPBody = [self buildBodyWithJSONString:jsonString isEncrypted:isEncrypted];
        NSURLRequest *request = [self buildFlushRequestWithServerURL:self.serverURL HTTPBody:HTTPBody];
        NSURLSessionDataTask *task = [SAHTTPSession.sharedInstance dataTaskWithRequest:request completionHandler:handler];
        [task resume];
    }];
}
Copy the code

3.2.3. Data sending policy

The SDK will store the collected data in the local SQLite database and send it only when the following sending policies are met:

  • The number of data stored locally on the client exceeds a certain value (100 by default).

You can configure flushBulkSize to limit the number of flushBulkSize items during SDK initialization. The default value is 100. If the specified number is less than 50, the value is 50. The SDK collects a lot of data. If the number of limit items in the sending policy is too small, frequent network requests may occur, affecting performance. If the limit set by the user is too large, too much data will be sent at a time. This will not only prolong the upload time, but also increase the probability of upload failure.

  • Interval between two data transmission (15 seconds by default)

When the SDK is initialized, you can configure flushInterval to control the interval between sending data (15 seconds by default). If the interval set by the user is less than 5 seconds, the interval is 5 seconds. The SDK starts a timer that sends data every 15 seconds.

  • The App goes back to the background and sends the data

The SDK to monitor the UIApplicationDidEnterBackgroundNotification notice, in the App when sending data back to the background. The specific code is as follows:


- (void)applicationDidEnterBackground:(NSNotification *)notification {
    ......
    dispatch_async(self.serialQueue, ^{
        [self.eventTracker flushAllEventRecords];
        endBackgroundTask();
    });
}
Copy the code

In addition to the send policy mentioned above, data in the local database is sent when the following events are triggered:

  1. Trigger the activation event;
  2. Trigger the $SignUp event;
  3. Triggers the $AppRemoteConfigChanged event.
  4. When the SDK is disabled through remote configuration, all data in the local database is sent.

3.2.4. Data security

3.2.4.1. Data encryption

The data sent by the SDK involves user privacy, and it is the developer’s responsibility and obligation to protect user privacy. The SDK provides data encryption to prevent data leakage during transmission.

Currently, SDK uses a mixed encryption policy (RSA + AES) to encrypt data, which can be divided into two parts:

  1. The generated AES key is used to encrypt the collected data symmetrically.
  2. The AES key is encrypted using the RSA public key.

The encrypted data format is as follows:


[{
    "pkv": 1."ekey": "e7lE4W67gUoER1al86Fg8CsMVhVpIDQReuONmwVyiIfZQA+U0J5J67UBnABFc6YKIYpWYPgAyQ5U+wPal17zOtyA7EeO2H+bxui1ESfKrh0pOaViElbHS7W TD9fBcAiOacNxukGlpjK70KWtSEFt+35XejWRw09AUIn8KeYSwnV7wetu4Ba783VvHsOd0vyWace3+I+T3tr7hiAnAxaeKtaYdeoKWCAydj8AM1jK+3z+kIc 1aVTwDXKEw/Cw03EyO5wKF/0pHYBCkCnTRhXFIULVR6EDQWJh/fW0Bc5YpT2YDP9KRzXP6HfAML2/k7YkwuMRXhR4p12h0RPFxvmTRg=="."flush_time": 1542625604461."payloads": ["Tg7E5sMghLePA3yW/1X6xO+MAPnncKvn9wYGk/T912JMljW0bK0hxXL14ttPY26uc1bksBHAqFW5xRb3LUYX+kcuM/N7shaw1/4XJcghw2JhexICA3Lf3Vs v37UtS0o8hW0LNq7kkSZt9wOa1Xb3agwtL7vhENtreqFBM+k+5ZH7MjVK8GalQdDauR7cZ1dtcprkFJiXuKrotp0DSeTCCtYiABlS3mDdVc1/NjbvvavbV4p 4FC+R5VA8aDWszKC02gMyF4pohPKpYgJsFSwCKbNPoPY8TEop3HS6UnXDI6hlTPmzspaDfJHXThLhb83mnqWnrtQih2HllGUUthA2fKZj+QgqoGj7EHhjsGg aXxV0I7Op6NEe5EA1nOejOK8ibo7s0c67lkyb7AFcBfRNgJzhSkuZy3lqGXl61d1KIo3UT4+iBnRAgxZf"."Tg7E5sMghLePA3yW/1X6xO+MAPnncKvn9wYGk/T912JMljW0bK0hxXL14ttPY26uc1bksBHAqFW5xRb3LUYX+kcuM/N7shaw1/4XJcghw2JhexICA3Lf3Vs v37UtS0o8hW0LNq7kkSZt9wOa1Xb3agwtL7vhENtreqFBM+k+5ZH7MjVK8GalQdDauR7cZ1dtcprkFJiXuKrotp0DSeTCCtYiABlS3mDdVc1/NjbvvavbV4p 4FC+R5VA8aDWszKC02gMyF4pohPKpYgJsFSwCKbNPoPY8TEop3HS6UnXDI6hlTPmzspaDfJHXThLhb83mnqWnrtQih2HllGUUthA2fKZj+QgqoGj7EHhjsGg aXxV0I7Op6NEe5EA1nOejOK8ibo7s0c67lkyb7AFcBfRNgJzhSkuZy3lqGXl61d1KIo3UT4+iBnRAgxZf"."Tg7E5sMghLePA3yW/1X6xO+MAPnncKvn9wYGk/T912JMljW0bK0hxXL14ttPY26uc1bksBHAqFW5xRb3LUYX+kcuM/N7shaw1/4XJcghw2JhexICA3Lf3Vs v37UtS0o8hW0LNq7kkSZt9wOa1Xb3agwtL7vhENtreqFBM+k+5ZH7MjVK8GalQdDauR7cZ1dtcprkFJiXuKrotp0DSeTCCtYiABlS3mDdVc1/NjbvvavbV4p 4FC+R5VA8aDWszKC02gMyF4pohPKpYgJsFSwCKbNPoPY8TEop3HS6UnXDI6hlTPmzspaDfJHXThLhb83mnqWnrtQih2HllGUUthA2fKZj+QgqoGj7EHhjsGg aXxV0I7Op6NEe5EA1nOejOK8ibo7s0c67lkyb7AFcBfRNgJzhSkuZy3lqGXl61d1KIo3UT4+iBnRAgxZf"] {},"pkv": 1."ekey": "e7lE4W67gUoER1al86Fg8CsMVhVpIDQReuONmwVyiIfZQA+U0J5J67UBnABFc6YKIYpWYPgAyQ5U+wPal17zOtyA7EeO2H+bxui1ESfKrh0pOaViElbHS7W TD9fBcAiOacNxukGlpjK70KWtSEFt+35XejWRw09AUIn8KeYSwnV7wetu4Ba783VvHsOd0vyWace3+I+T3tr7hiAnAxaeKtaYdeoKWCAydj8AM1jK+3z+kIc 1aVTwDXKEw/Cw03EyO5wKF/0pHYBCkCnTRhXFIULVR6EDQWJh/fW0Bc5YpT2YDP9KRzXP6HfAML2/k7YkwuMRXhR4p12h0RPFxvmTRg=="."flush_time": 1542625604461."payloads": ["Tg7E5sMghLePA3yW/1X6xO+MAPnncKvn9wYGk/T912JMljW0bK0hxXL14ttPY26uc1bksBHAqFW5xRb3LUYX+kcuM/N7shaw1/4XJcghw2JhexICA3Lf3Vs v37UtS0o8hW0LNq7kkSZt9wOa1Xb3agwtL7vhENtreqFBM+k+5ZH7MjVK8GalQdDauR7cZ1dtcprkFJiXuKrotp0DSeTCCtYiABlS3mDdVc1/NjbvvavbV4p 4FC+R5VA8aDWszKC02gMyF4pohPKpYgJsFSwCKbNPoPY8TEop3HS6UnXDI6hlTPmzspaDfJHXThLhb83mnqWnrtQih2HllGUUthA2fKZj+QgqoGj7EHhjsGg aXxV0I7Op6NEe5EA1nOejOK8ibo7s0c67lkyb7AFcBfRNgJzhSkuZy3lqGXl61d1KIo3UT4+iBnRAgxZf"."Tg7E5sMghLePA3yW/1X6xO+MAPnncKvn9wYGk/T912JMljW0bK0hxXL14ttPY26uc1bksBHAqFW5xRb3LUYX+kcuM/N7shaw1/4XJcghw2JhexICA3Lf3Vs v37UtS0o8hW0LNq7kkSZt9wOa1Xb3agwtL7vhENtreqFBM+k+5ZH7MjVK8GalQdDauR7cZ1dtcprkFJiXuKrotp0DSeTCCtYiABlS3mDdVc1/NjbvvavbV4p 4FC+R5VA8aDWszKC02gMyF4pohPKpYgJsFSwCKbNPoPY8TEop3HS6UnXDI6hlTPmzspaDfJHXThLhb83mnqWnrtQih2HllGUUthA2fKZj+QgqoGj7EHhjsGg aXxV0I7Op6NEe5EA1nOejOK8ibo7s0c67lkyb7AFcBfRNgJzhSkuZy3lqGXl61d1KIo3UT4+iBnRAgxZf"."Tg7E5sMghLePA3yW/1X6xO+MAPnncKvn9wYGk/T912JMljW0bK0hxXL14ttPY26uc1bksBHAqFW5xRb3LUYX+kcuM/N7shaw1/4XJcghw2JhexICA3Lf3Vs v37UtS0o8hW0LNq7kkSZt9wOa1Xb3agwtL7vhENtreqFBM+k+5ZH7MjVK8GalQdDauR7cZ1dtcprkFJiXuKrotp0DSeTCCtYiABlS3mDdVc1/NjbvvavbV4p 4FC+R5VA8aDWszKC02gMyF4pohPKpYgJsFSwCKbNPoPY8TEop3HS6UnXDI6hlTPmzspaDfJHXThLhb83mnqWnrtQih2HllGUUthA2fKZj+QgqoGj7EHhjsGg aXxV0I7Op6NEe5EA1nOejOK8ibo7s0c67lkyb7AFcBfRNgJzhSkuZy3lqGXl61d1KIo3UT4+iBnRAgxZf"]}]Copy the code

In order to improve the efficiency of encryption and reduce the length of the key, Elliptic Curve Cryptography (ECC) is finally selected after investigation and a lot of tests.

Compared with RSA, ECC has the following advantages: under the same level of security, ECC has smaller key length and faster encryption speed [2].

Therefore, another hybrid encryption strategy provided by SDK is ECC + AES, which can also be divided into two parts:

  1. The generated AES key is used to encrypt the collected data symmetrically.
  2. The ECC public key is used to encrypt the AES key.

This feature is still under development and will be available soon.

3.2.4.2. Letter of self-certification

The SASecurityPolicy class is provided in THE SDK (refer to AFSecurityPolicy in AFNetworking, and the usage method is similar [3]), which is mainly used to verify whether the HTTPS certificate is correct.

Let’s take a look at the certificate verification policy:

  1. SASSLPinningModeNone: indicates that the client unconditionally trusts the certificate returned by the server.
  2. SASSLPinningModeCertificate: it means the client will return to the server certificate from the local save the certificate of “all content” check;
  3. SASSLPinningModePublicKey: it means the client will return to the server certificate from the local preservation in the certificate check “PublicKey part”.

After the SDK is initialized, a SASecurityPolicy object of SASSLPinningModeNone is created by default. This can be supported with the following code:


SASecurityPolicy *securityPolicy = [SensorsAnalyticsSDK sharedInstance].securityPolicy;
  
/* allowInvalidCertificates does not require validation from the certificate, default is NO */
securityPolicy.allowInvalidCertificates = YES;
  
/* validatesDomainName Specifies whether to validate the domain name. The default value is YES */
securityPolicy.validatesDomainName = NO;
  
/* pinnedCertificates stores local certificates */
securityPolicy.pinnedCertificates = [SASecurityPolicy certificatesInBundle
Copy the code

Note: iOS only supports DER certificates. OpenSSL can be used to convert other certificates.

3.2.5. Data sending process

During data collection, the attribute information needs to be encapsulated into the JSON format required by the gods, and then stored in the database.

When the SDK triggers a data, it checks whether the current state is in Debug mode or whether the maximum cache of the local database is exceeded. If the preceding conditions are met, the data is sent.

If the event triggered is $SignUp, the number of data items in the local cache is larger than flushBulkSize, and the time interval since the last sending event is larger than flushBulkSize. Data will be sent if any parameter is met. Figure 3-1 shows the specific process:

Figure 3-1 Flowchart for sending data

Data is not sent when:

  • No data is sent when the serverURL is empty;
  • No data will be sent when there is no network;
  • Data will not be sent if the network sending policy configured by the SDK is not met.

If the sending conditions are met, the SDK sends all local data. If a large amount of data is transferred at a time, the transmission failure probability increases and performance deteriorates. Therefore, THE SDK reads up to 50 pieces of data at a time. Gzip is used to compress the read original data, and Base64 is used to encode the compressed content to ensure efficient transmission.

After data is sent, mark locally that part of the data is being sent. Then, the SDK determines whether the network request is successfully sent based on the network request status code: When the status code is between 200 and 300, the SDK considers that the data is successfully sent, marks the status of the locally sent data as successfully sent, and deletes the local data. When sending fails, the local data is not deleted and is marked as unsent.

Data stored locally is read in a loop each time until all data is sent.

3.2.6. Data transmission verification

In the process of using SDK, developers need to verify whether the data collected by SDK can be successfully sent to the specified server. Therefore, the SDK provides ways to verify that data is successfully sent, such as viewing logs and using dynamic debug mode.

3.2.6.1. Console log verification

Check whether data is successfully sent on the Xcode console. Log printing must be enabled during initialization. The SDK will send data to the specified server if the sending policy is met.

Logs displayed on the console are classified as follows:

  • If the buried event is triggered successfully, the event data starting with “Track Event” is output.
  • When the buried event fails to trigger, the corresponding error cause is output.
  • If the event data is sent successfully, the event data starting with Valid Message is output.
  • When the event data fails to be sent, the event data starting with invalid Message is output and the error cause is displayed.

During development, you can determine whether data is successfully sent according to logs.

3.2.6.2. Dynamic debugging mode

The SDK provides the function of dynamic debugging and sending data, which is convenient for developers to verify data when using the SDK. In dynamic debugging mode, the data collected by the SDK will be sent to the specified server in real time. DEBUG_ONLY and DEBUG_AND_TRACK modes are provided:

  • DEBUG_ONLY: Sends data in real time but does not enter the database, avoiding the dirty data generated during the test.
  • DEBUG_AND_TRACK: Sends data in real time and also stores it in the database.

How does the analytics server know that the SDK is in dynamic debug mode? The answer is that we replace “sa” in the serverURL with “debug” when sending the request, so that the server knows that the data is dynamically debugged.

DEBUG_ONLY and DEBUG_AND_TRACK, the server is how to distinguish the two modes? The answer is that when we use DEBUG_ONLY we add the attribute “dry-run” in the request header to distinguish DEBUG_ONLY from DEBUG_AND_TRACK.

In dynamic debugging mode, Scheme is required. After Scheme is correctly configured, you can scan the QR code in real-time debugging data to pull up the App. The specific method of use is:

  1. First use the debugging device to scan the TWO-DIMENSIONAL code of the web page and enable the “Debugging mode” of the device.
  2. After clicking start refresh, the operation App triggers the event;
  3. If the event is uploaded successfully, the corresponding event is displayed in real-time debugging data viewing.

Four,

This paper mainly introduces the implementation of Shence analysis iOS SDK network module. SDK network requests are not implemented based on open source network frameworks, avoiding some potential risks. By encapsulating system class NSURLSession, adopting perfect sending strategy, and compress, encode and verify data, the data can be sent timely, accurately and efficiently.

Finally, I hope that through this article, we can have a systematic understanding of the shence analysis of the iOS SDK network module.

References:

[1] developer.apple.com/videos/play…

[2] wiki.openssl.org/index.php/E…

[3] github.com/AFNetworkin…