Explore ways to optimize the Android network

preface

1. Overview of network optimization

Android network optimization method is mainly about offline network testing tools, online network monitoring scheme, traffic optimization scheme and quality optimization scheme.

Do network optimization, to consider multiple dimensions, statistics of a variety of data, but also to establish a perfect monitoring system.

Common Network testing tools are Network Profiler, Charles, and Stetho.

Common online network monitoring schemes include OkHttp’s EventListener, NetworkStatsManager, and TrafficStats.

Common traffic optimization schemes include data caching, data compression, and image compression.

Common quality optimization schemes include HttpDns optimization, protocol version optimization and capital optimization.

Considering multiple dimensions means that network optimization should consider the traffic dimension and quality dimension, and then reduce traffic consumption and improve request speed according to the data of these two dimensions.

Traffic dimension refers to the accurate measurement of traffic consumption of App in a period of time. Traffic consumption can be divided into dimensions such as network type, foreground and background, request and response. The client records the data so that the server can send commands to the client to control the client to upload the data when required. The client can also report the data when it exceeds the threshold.

The quality dimension can be distinguished from data such as request duration, request success rate, and Top failed interfaces to quickly locate and resolve problems in the future.

Statistics of various data means that not only the online traffic consumption is counted, but also the time of App use and other data, so as to analyze whether the traffic consumption is reasonable by combining various data.

Establishing a sound monitoring system means that network optimization should monitor the success rate of network requests and network request anomalies, such as excessive traffic consumption, excessive request times, and excessive file download.

Network optimization requires offline testing and online monitoring. During offline testing, it is necessary to test the user experience of clients under different network types, such as switching Wifi and traffic, and different network states, such as weak network test and no network test. In offline testing, the test cycle should be extended, because some network requests on the client are not reported in real time, and the network test takes longer to detect problems. Another point is that some apps have high security requirements, which may lead to login failure when switching network. In this case, it is necessary to pay attention to whether the loading popover is closed when the network request is cancelled.

Network Profiler is a Network analysis tool provided by Android Studio. Network Profiler contains connection view, thread view, request overview, response information, request information, and call stack data. In the connection view, you can see information about the content type of the request, the status of the request, and how long the request took.

Charles is a packet capture tool. In addition to Charles, there are other packet capture tools such as Fiddler, Wireshark, and TcpDump. Charles supports breakpoint debugging, simulated data, and weak network simulation. Mock data can be used when the server interface is not fully developed or when we want to verify that different response data is being processed properly on the client side.

Stetho is a Facebook open source application bridge that allows you to view information about Android devices and applications in Chrome. Stetho includes web request information, view view, database view, and command line extensions. To use Stetho, add the Stetho interceptor to the OkHttp client and type Chrome://inspect into Chrome to see information about the requests the client made and the response data it received.

The server monitors the request duration, failure rate, and Top failed interfaces in different dimensions, such as region and time segment. The failure rate includes request failure and service failure.

The client monitors information about each step of the request, such as DNS resolution time, connection establishment time, request time, and network packet size. The client also monitors image requests, as a single image can consume more traffic than five or more interfaces. In addition, a network disaster recovery mechanism should be established. For example, when standby servers are shunted and multiple requests fail within a certain period of time, no more requests are sent and the retry time is extended to prevent the server from being able to withstand the surge of users.

In addition, both the client and the server should monitor network abnormalities. For example, the server should determine whether someone is using our server, and reject the service of these IP addresses if there is. The client should add the active warning function, such as the user downloads a file that exceeds the set value, after downloading, the result can be reported to the server, indicating that there is an exception and further confirmation is needed.

OkHttp’s EventListener captures the time and data for each step of the request, and provides callback methods for the start, end, and failure of EVENTS such as DNS resolution, establishing a connection, establishing an HTTPS connection, writing to the request header, and writing to the request body. And EventListener can be used with Glide to retrieve data for each step of the image request.

By default, a single host can execute a maximum of 5 requests concurrently. If we do convergence, we can set OkHttpClient’s maxRequestsPerHost value to a higher value.

NetworkStatsManager is a traffic statistics manager. It can obtain the traffic consumption of different network types in a certain period of time. However, users need to enable the usage check right, which results in poor user experience.

Traffic optimization should first set expectations and compare with competing products. For example, it is expected that a new function will consume 300KB traffic once it is used by users, but it will exceed 300KB after it is launched. At this time, it is necessary to find out and solve this problem. In addition, it should also be compared with competing products, such as the main process of Posting comments, the same comments and the same pictures, and how much traffic consumption is compared with competing products.

TrafficiStats is also a traffic statistics tool, with the drawback that it can only measure traffic consumption since a phone was last restarted.

After we recorded the log of network related, when reporting performance logs, roughly do have four parts, respectively is the App starts executing a background task, every once in a while (e.g., 30 seconds) will get a network data, add logo to the data, such as to distinguish the Taiwan before and after, at the right time to elevate the data to the APM background, Appropriate timing refers to situations where user feedback, thresholds are reached, or WiFi networks are available, so that the traffic consumed by performance logs does not have a significant impact on users.

Data caching can be divided into two schemes: OkHttp Cache and netless interceptor. In our project, there are generally some interfaces that do not require high real-time performance, such as the list of provinces and cities, so we can consider caching the data of these interfaces.

Cache is a class used by OkHttp’s Cache interceptor. If this is not set, OkHttp does not Cache response data by default. The Cache uses DiskLruCache. To create a Cache, you need to specify the Cache directory and Cache size, and then set it to OkHttpClient. When using Cache, note that the default Cache interceptor only caches responses to request methods such as GET and HEAD to retrieve resources.

A netless Interceptor is a custom class that implements the Interceptor interface and then passes the FORCE_CACHE policy to the Request Builder’s cacheControl() method when the network is unavailable. When caching data, the data is added with an expiration date or version number. Each time a request is made, the client determines whether the data has expired or whether the version number of the data has been updated.

The second traffic optimization scheme is data compression, which includes three schemes: opening Gzip compression, compression request header and merge request.

To enable Gzip compression, set the content-Encoding header to Gzip. If our network framework uses OkHttp, then the content-Encoding header is not set. OkHttp defaults us to set Content-encodingShe to gzip.

Compressing the request header allows the server to cache the request header while the request header remains intact. When some information about the request header is needed, it is fetched from the previous cache based on the MD5 of the previous headers.

Merge request refers to merging requests that can be merged because each network request header will have redundant information.

The third traffic optimization solution is image compression, which includes thumbnails, WebP and Luban.

Thumbnails refer to the use of thumbnails in a list first, since it doesn’t make sense to show the original image directly in the list.

WebP is an image format that provides lossy compression and lossless compression, reducing image size by 50% or more.

Luban is an image compression tool, which imitates the compression strategy of wechat moments, and can reduce the size of pictures by 95% or more. The disadvantage is losiness, just like the pictures in Wechat moments are somewhat blurred.

The two indexes of network request quality optimization are the success rate of network request and the speed of network request. The three common schemes of network request quality optimization are using HttpDNS, upgrading HTTP version and spending money.

Using HttpDNS can bypass the carrier domain name resolution process. HttpDNS uses HTTP rather than traditional DNS to prevent hijacking and improve access speed.

HTTP/2 is a binary protocol. The biggest improvement is that clients and servers can send multiple requests and responses at the same time, instead of requesting them sequentially like HTTP/1.1.

Throwing money at CDN means spending money on CDN acceleration, improved bandwidth and static resource separation.

1. Three key points of network optimization

The three key points of network optimization are multidimensional, precise and monitoring.

Multidimensional refers to the network optimization to consider multiple dimensions, when it comes to network optimization, most people first think of traffic consumption, but in fact, traffic consumption is only one of the dimensions of network optimization.

It is not enough to optimize only one dimension of traffic consumption. Some teams even fail to do a good job in traffic optimization, for example, the statistics of network traffic consumption are not comprehensive and accurate.

Precision means that the measurement of network traffic statistics should be accurate. If we only obtain the value of the specific consumption, it is not very helpful for us to locate and solve the problem, because this value can only show how much traffic users use. Otherwise, if online users report that the App consumes a lot of traffic, but we do not know how long the user has been using the App in total, it will be difficult to locate the problem. If the user has been using the App for a long time, it is probably normal to consume more traffic. Another example is the user feedback that the App consumes a lot of traffic in the background, but we only counted the overall value, so it is impossible to determine how much traffic the App consumes when running in the background.

Monitoring refers to the construction of a comprehensive and perfect network monitoring system, rather than monitoring only one index. If we only monitor the success rate of network requests, we can only know the general network usage of users. Such coarse-grained monitoring cannot help us find out and solve the root causes of problems. For example, online users use a certain function for 1000 times, and then an exception occurs, and the user clicks Retry to restore the normal. In this way, the success rate of network requests is relatively high only from the data, but it is impossible to know the reason for the occurrence of this exception only by the success rate value, and it is impossible to avoid the subsequent occurrence of such exceptions.

2. Two dimensions of network optimization

The two dimensions of network optimization are traffic dimension and quality dimension respectively.

Traffic dimension refers to the accurate measurement of App traffic consumption in a period of time. Large traffic consumption not only has an impact on users, but also has an impact on the company’s operating costs, such as bandwidth, number of servers, CDN and other expenses. In addition, network request density also has a certain impact on mobile phone power consumption. We need to distinguish types, monitor exceptions, and report logs.

• To distinguish the type of

  To distinguish types is to know the traffic consumption of users in different network types (traffic, WiFi), and to distinguish the traffic consumption of App in the foreground and background. Only by accumulating data of different dimensions can problems be quickly determined and solved.

• Monitor the abnormal

  Anomaly monitoring means that we need to know not only the average traffic consumption of users, but also the anomaly rate of traffic consumption of online users. The exceptions include excessive traffic consumption, too many requests, and too many files to download.

• Report to the log

  The ideal situation is that we have a complete local monitoring of all network requests, and all information related to each Request Request and Response is recorded.

  The server can send commands to control the client to upload the data. The client can also report the data when it exceeds the threshold.

2. Quality dimension

The quality of network request is also very important, which directly corresponds to the real experience of users. If the network request speed is slow or the success rate of the request is low, it will lead to bad user experience.

You can monitor network request quality by request duration, request success rate, request failure rate, and Top failed interfaces, so that problems can be quickly located and solved.

3. Two misunderstandings of network optimization

In network optimization, we should not only focus on traffic consumption but ignore other dimensions. In network monitoring, we should not only focus on average and overall data but ignore individual data.

For example, the success rate of the request mentioned above is very high overall, but this kind of data does not help us improve the single request.

1. Three offline testing tools

Network Profiler is a Network analysis tool provided by Android Studio. Network Profiler can display real-time Network activity. Such as sending network requests, receiving data, and connection counts. Charles is a packet capture tool, and Stetho is an open source network analysis tool from Facebook.

1.1 Precautions for offline Testing

Offline test is to expose the problems as much as possible before the online, in the offline test, we should pay attention to the network switch, weak network/no network test, whether the request is wrong and extend the test period.

In the offline test, we need to switch to different networks, such as Wifi and traffic, and do weak network test and hopeless test to ensure that our App has a good experience under different network types and different network states.

When a request fails, you must first check whether the interface request is incorrect, such as too many parameters or incorrect parameters.

In the offline traffic consumption test, the cycle should be prolonged, because the number of users of THE C-terminal App may reach tens of millions or even more when it reaches a stable period. At this time, App functions are generally very complex and include other functions, such as performance monitoring. Network requests for these functions are often not reported in real time, so when we do traffic consumption tests, the cycle will be extended, and a simple test of 10 minutes may not detect problems.

Some apps have high security requirements. If the network status is changed suddenly, the user’s login status will become invalid, that is, the user needs to log in again. At this time, we should pay attention to whether the App will return to the previous process after re-login, and do not interrupt the user’s ongoing process.

In the state of weak network or no network, it is necessary to test whether the Loading popup stops. If you are not careful, in the state of no network, the application request fails, but the Loading popup is not closed, which affects the user’s other operations. If you don’t pay enough attention to the netless state, you won’t be able to detect such bugs.

1.1 the Network Profiler

Network Profiler is a built-in Network analysis tool of aN AS. It displays real-time Network activities, such AS sent Network requests, received data, and connection numbers.

In the Profiler, there is a Connection View and a ThreadView. The Connection View contains Connection information, such as request time, response data, request data, and call stack. The ThreadView contains execution timelines of different network request threads.

Start the analysis by clicking the Profile ‘app’ button next to Run.

You then see the Connection View below the Profiler and, to the right, information about a Connection, including request time, response, request, and call stack.

Below is the thread view.

The Request panel contains header information such as User-Agent and Connection, as well as the Request body.

The Response panel contains the Response body and Response header data.

The Call Stack contains the Call Stack for network requests.

1.2 Charles

Common packet grabbers include Charles, Fiddler, Wireshark, and TcpDump. Charles is developed in Java and is widely used on Mac, while Fiddler is widely used on Windows. We will focus on the use of Charles, which supports breakpoints, simulated data and weak network simulation.

1.2.1 Basic Usage

First download Charles from the official website

2. Enable Charles proxy

3. Check the COMPUTER IP address

Vboxnet2 is the address of the Virtual Box of the VM.

4. Set the WiFi proxy address

5. Agreed to connect

   After setting the proxy, when we access the network on the device, Charles will pop up a pop-up window whether to Allow the connection. We can click Allow.

6. Install the certificate

   If you are fetching HTTPS, you need to install the certificate, otherwise the request and response information will be garbled.

Search Charles and select Always Trust.

7. Installing a Mobile Phone Certificate

Select Install certificate After mobile device, Charles will pop up a proxy prompt telling us to go to CHLS. Pro/SSL to download and install the certificate.

I have already downloaded it, so the prompt here is to download it again, and then click Install.

8. Add the host

   In SSL Proxying Settings, add the address of the host using the wildcard *.*.

9. Add network security configuration

Add network_security_config.xml under res/ XML

Then add the configuration to the Application tag of the Manifest.

Then you can see the result of the packet capture.

1.2.2 breakpoint

When we right-click on any of the requests, we can see the Breakpoints option.

To view a list of breakpoints that have been added, go to Proxy->Breakpoint Settings.

If you want to work only on Request or Response breakpoints, you can click on breakpoints and set them in the Edit Breakpoints popup.

Let’s take a look at the interface after entering the breakpoint. When we make the breakpoint request again, we can see the breakpoint interface.

This is where we can Edit the requested information after clicking the Edit Request TAB.

When I change version to 3, I can see that the parameters in the request change to 3.

1.2.3 Simulated data

Charles has a feature called Map Local, which is used when the server interface is not developed and only the protocol is defined.

We can use Map Local to simulate the fake data we want, and we can freely simulate all kinds of dirty data, so that even if the development progress of the server side is slow, it will not affect the development progress of the client side. Besides Map Local, we can also use MockK and other simulation frameworks for testing to achieve the function.

Let’s see how to use Map Local.

Then right click on the request, and at the bottom of the pop-up menu is Map Local.

After clicking Map Local, you can see the Edit Mapping popup.

The original response looks something like this.

Then create a maplocal file and change the group count to 20.

When the client makes the same request again, you can see that the group count value has changed to 20.

To disable Map Local, click Tools -> Map Local and deselect Enable Map Local.

1.2.4 Weak network simulation

Charles supports weak network simulation. Go to Proxy -> Start throtting.

After Enable Throttling is selected, the traffic limiting function is enabled. The traffic limiting function supports the following options:

• Throttle preset

• Bandwidth

• Utilisation

• Round-trip-latency

• MTU (Maximum size of TCP packets transmitted)

• Reliability (packet loss)

• Stability

• Unstable quality range

  Mainly for the range set in Statbility.

Here are some common presets.

For the following request, the total processing time before limiting traffic is 177 ms.

When the weaknet simulation was turned on, it was 9.3 seconds.

1.3 Stetho

Stetho is a powerful application connection bridge that can connect Android and Chrome. It has network monitoring, view view, database view, and command line extension functions.

First add the dependency, which is Stetho’s implementation of OkHttp.

 // Android application debugging tool
implementation 'com. Facebook. Stetho: stetho - okhttp3:1.5.1'
Copy the code

Then initialize and add interceptors.

Type Chrome://inspect into Chrome, and you’ll see the TAB that Chrome has opened on your device, just below the App that Stetho debugged.

Click inspect below the bean sprout to see DevTools, and click the Network TAB in DevTools to see data such as requests made and responses received by the App.

Stetho is usually not used as a packet capture tool because the packet capture function of Stetho is not that powerful.

2. Three key points of online monitoring

Before we look at capturing network traffic, let’s look at three key aspects of offline monitoring: server monitoring, client monitoring, and exception monitoring.

2.1 Server Monitoring

The server monitors the network request duration, failure rate, and Top failed interfaces of different dimensions, such as region, time segment, version, and model.

The failure rate can be divided into two dimensions: request failure and service failure. Request failure refers to network request failure. Service failure refers to a successful network request, but users do not get the data they need.

On the APM background, it is recommended to collect statistics of Top failed or abnormal interfaces in a period of time, for example, one day or one week. In this way, you can know which interfaces are unstable and optimize them accordingly.

2.2 Client Monitoring

The client monitoring is more critical than the server monitoring. The client monitoring is more comprehensive and can obtain more data. The client online monitors the information of each step of the request, the image request, and the establishment of the network disaster recovery mechanism. Although the image loading time and success rate can be measured on the server side, the data on the server side is incomplete because some requests fail without ever reaching the server. Moreover, the data sent by the server and the delay time of the network channel must be longer than the time calculated by the server, so we need to add the statistics on the client side.

The client can get information about each step of the request, including DNS resolution time, connection establishment time, request time, network packet size, etc. At the same time, we can record the user’s operation of each network request, such as the specific interface requested, whether the request is successful and the reason for the failure of the request. These information can be sent to the APM server as the basic information for monitoring. In the following part, we will introduce how to monitor each step of the network request.

The client also monitors image requests, and one image can consume more traffic than five or more interfaces.

The client should establish a network disaster recovery mechanism, if one day the number of our users suddenly skyrocketed, the server may not bear the pressure, for the server, can use the standby server diversion, to avoid the main server down. The client can also make a policy that if a network request fails several times in a certain period of time, it will not make any network request.

2.3 Abnormal Monitoring

The purpose of exception monitoring is to improve our awareness of exceptions, rather than passively waiting for user feedback.

1. Server brush prevention

On the server side, we need to determine whether someone is brushing our server, which is a malicious attack, and if we detect that someone is brushing the server, we can lock down the IP address and deny access to the IP address.

2. Abnormal bottom line

On the client side, we can add active warning capability. For example, if a user downloads a file that exceeds the set value, the client can report the result to the server after downloading, indicating that there is an exception and further confirmation is needed.

In some scenarios, the server may be overloaded with traffic. In this case, the client can implement a bottom-pocket policy. If the interface fails for five consecutive times within a certain period of time (for example, within 30 seconds), the client disallows continuous access and sets a longer retry time.

3. Single point tracing

If online users report that the App consumes too much traffic or consumes too much traffic in the background, we can analyze the network request logs of users in detail and issue commands to check the traffic consumption in a specific period.

3. Three online monitoring schemes

Some problems only occur online, but cannot be detected offline. For example, when a certain version is tested offline, the H5 package may not be the new version, but some users may be hit online, and then the package of the new version is delivered.

If these packages are not compressed, such exceptions cannot be found in the offline test environment, only through online monitoring.

Let’s take a look at the following three online monitoring solutions:

• OkHttp event listener
• NetworkStatsManager
• TrafficStats

Of the three, the OkHttp event listener captures the most detailed and useful data, while NetworkStatsManager and TrafficStats are primarily used to capture traffic consumption.

3.1 OkHttp event listener

3.1.1 Customizing event Listeners

Let’s take a look at how to get network request quality data in combination with OkHttp. OkHttp leaves us with an EventListener callback that we can implement ourselves to listen for each request.

Start by defining a Model to store request and response data. Only some of the data has been added here for demonstration purposes, so you can customize your Model according to your needs.

Create OkHttpEventListener and rewrite the method we want. We’ve only overwritten some methods here for demonstration purposes. OkHttpEventListener provides a lot of methods that you can rewrite as much as you want.

Then set up the OkHttpClient event listener factory.

After setting up the factory, we can get the time and relevant data of each step of each network request, including DNS resolution events, request events, response time, response bytes and other data, which are essential data support for us to do online monitoring.

3.1.2 Customizing GlideModule

GlideModue is designed to monitor image loading. Let’s see how GlideModue monitors image loading time and related data.

Start by adding plug-ins and dependencies.

/ / Kotlin plugin
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'
apply plugin: 'kotlin-kapt'

android {
  // ...
}

dependencies {
  // ...

  // Image loading
  kapt 'com. Making. Bumptech. Glide: the compiler: 4.11.0'
  implementation 'com. Making. Bumptech. Glide: glide: 4.11.0'
  implementation "Com. Making. Bumptech. Glide: okhttp3 - integration: 4.11.0"
}
Copy the code

Then define a GlideModue that replaces the ModelLoader factory.

Click Build -> Clean Project, and then click Build -> Make Project.

And then replace Glide with GlideApp.

Then we can listen to the time and size of the image loading process in our custom event listener.

3.1.3 Maximum number of OkHttp concurrent requests

As for the frequency of requests, we can look at OkHttp’s default request pool. In OkHttp’s Dispatcher Dispatcher, there is an executeService thread pool with a core pool size of 0, the maximum of which is an integer.

This does not mean that network requests sent through OkHttp can be sent in infinite numbers. We can send more for IO intensive tasks, which are not too CPU intensive, but we should note that there is also a limit to the number of threads that can be created by a single App.

Although there is no limit to the thread pool size, the promoteAndExecute() method determines whether the maximum number of requests has been exceeded before the request is executed. The criteria are maxRequests (maximum number of concurrent requests) and maxRequestsPerHost (maximum number of concurrent requests per host).

The maximum number of requests that can be concurrently executed by a host is set to prevent excessive requests from a domain name from being executed by other domain names.

If we already have domain name convergence in our project, we can increase the number of requests under a single domain name so that more network requests can be executed at the same time.

3.1.4 Distinguish front and back traffic

Separating Taiwan before and after the flow in the log, because many users to worry about the App has been consumed in the background traffic, if coarse-grained get traffic data, only is did not know how much these traffic is App runs in the background when consumed, so don’t say that the problem of user feedback, can’t even positioning positioning problem.

The request data is flagged by ProcessLifecycleOwner so that we can see the user’s traffic consumption in the foreground and in the background, and when the user reports back, we can look at the user’s traffic consumption statistics ourselves to determine if there is an exception.

The disadvantage of this scheme is that there will be some error when the user switches between the front and back state of the App within 30 seconds, but the error is within the acceptable range.

In combination with the APM background threshold, the client automatically reports traffic when the traffic consumption reaches the threshold, achieving more accurate traffic consumption monitoring.

Let’s see how to use ProcessLifecycleOwner.

1. Add dependencies

There are many Libraries on AndroidX that come with the process library. Check the External Libraries to see if there is one.

dependencies {

  AndroidX Lifecycle process
  implementation 'androidx. Lifecycle: lifecycle - process: 2.2.0'

}
Copy the code

2. Define application lifecycle observers

3. Add observers

4. ProcessLifecycleOwner principle

Before ProcessLifecycleOwner comes out, we can count ourselves to see if all activities are hidden, and if so, the application.

The ProcessLifecycleOwner just does this for us by calling attach in the ProcessLifecycleOwner initialization method ini, The Attach method implements the Activity lifecycle callback interface.

In the onStart example, ProcessLifecycleOwner has an mStartdCounter, which is incremented by one each time an Activity’s onStart method is called, If mStartCounter equals 1, the application is started, and ProcessLifecycleOnwer will call our custom onAppForeground method through mRegistry.

3.1.5 Uploading Data in four steps

The following are the four general steps for reporting performance logs.

1. Background tasks

   Perform a background task when the App starts;

2. Interval statistics

   This task retrieves network data at regular intervals (e.g., within 30 seconds);

3. Since the data set

   Maintain a data statistics, add labels to the data, remember users in the front and back of the total traffic consumption;

4. Reported data

   At the appropriate time (such as user feedback, threshold, WiFi network), the data will be reported to the APM background, so that the traffic of users will not be affected, and the data can also be used as the basis of traffic management;

3.2 NetworkStatsManager

If we use NetworkStatsManager instead of OkHttp EventListener, we can also send a command to the client in the background to upload traffic statistics for a specific period of time when the user reports that the traffic consumption is high. Then, it is determined whether the App traffic consumption is really abnormal based on the user’s usage time.

NetworkStatsManager is a traffic statistics manager after API 23. NetworkStatsManager can obtain traffic information of a certain period of time and traffic consumption of different network types. However, its biggest disadvantage is poor user experience, requiring users to enable the View Usage permission.

Here are some scenarios that use the network or consume a lot of traffic.

• API request

• Upgrade package

  Various upgraded resource packages, such as App upgrade package, H5 Zip package used by WebView, bundle package used by RN, etc.

• configuration

  After the App becomes large, it needs to use various configuration information, such as the configuration information used in A/B testing and the configuration information delivered by operational activities.

• The picture

  Image is a large user of traffic consumption, image download and upload are very consumption of traffic;

• monitoring

  After the App becomes larger, it will also do various monitoring functions. For example, all kinds of data monitored by APM can only be uploaded to the server through network.

3.2.1 Three key points of flow optimization

Before we talk about how to use NetworkStatsManager to get traffic consumption, we need to understand the three key points of traffic optimization.

1. You can’t just look at absolute values

Absolute value cannot be used as the only statistical standard for high traffic consumption. It cannot be said that App consumes 10M traffic, so it needs to be optimized immediately.

Absolute value comparison is meaningless. For example, if you use the App for 30 minutes and browse a lot of products or videos, then 10M is probably too little.

2. Compare competing products

It is better to compare the traffic consumption of competing products in the same scene. For example, we run a main process of Posting comments with competing products. Attention should be paid here to ensure that the comments published by the two apps are the same and the pictures are the same, so as to ensure that the variable is unique.

By contrast, if the gap between our flow consumption and that of competing products is relatively large, then we should speed up the pace of flow optimization, and the absolute value can be combined with the comparison.

3. Set expectations

We need to judge the traffic consumption of the new function. For example, we should expect that after users use the new function, the single traffic consumption should be about 300K, but when it exceeds 300K after the launch, we need to confirm whether the traffic consumption is too high.

3.2.2 NetworkStatsManager Basic Usage

1. readNetworkStats

2. querySummary

The following is an introduction to the parameters of the commonly used querySummary method.

The internal processing flow of querySummary is as follows.

In this process, the most important thing is to read the network statistics saved by the system from the mStats of NetworkStatsCollection.

3. Apply for authorization

Using NetworkStatsManager requires additional permission PACKAGE_USAGE_STATS, which is a system permission that requires active bootstrap-enabled applications.

The permissions are first declared in the manifest file.

It then actively directs the user to enable permissions in the code.

3.3 TrafficStats

TrafficStats is a traffic statistics scheme provided by API 8, which actually measures the traffic consumption of our phones since the last restart, i.e. traffic before the restart is not counted.

TrafficStats Common methods:

• getUidRxBytes(int uid)

  Get the received traffic of the specified Uid.

• getTotalTxBytes()

  Get the total App sent traffic;

Here’s how TrafficStats is used.

TrafficStats is not normally used because it has two problems:

• Unable to get traffic consumption for a specific application

  We can’t get the traffic consumption for a particular application, but we can only get a total traffic consumption value, which is not very helpful for us to solve the problem.

• The traffic consumption for a specific period of time cannot be obtained

  For example, an online user reported that the traffic consumption of the App was too much yesterday. At this time, we cannot know how much traffic the user consumed.

4. Three traffic optimization schemes

Let’s look at the next three flow optimization solutions:

• Data cache
• Data compression
• Image compression

4.1 Data Caching

Let’s look at two methods of data caching:

• OkHttp cache
• Expiration time and incremental updates

4.4.1 OkHttp cache

If we take a closer look at the interfaces in our own projects, we will find that many interfaces are not so demanding in real time. Using caching can not only save traffic, but also greatly improve the speed of data access.

Common web libraries, such as OkHttp and Volley, have good caching practices.

In addition, the lack of caching is also bad for user experience. Generally, an App will display an interface with no data after opening it. Compared with displaying the data last time, the user experience is actually poor.

1. Netless interceptors

Let’s focus on OkHttp’s caching practices, starting with defining a netless interceptor.

The next step is to add an interceptor to OkHttpClient.

After the addition of no network interceptor, when we open our App without network, we can also obtain the last data and use the App, so as to improve user experience.

2. OkHttp cache processing flow

OkHttp’s cache interceptor processes requests as follows.

4.1.2 Expiration time and incremental update

1. Expiration time

Add an expiration time to the data returned by the server, so that we can determine whether the request is expired or not. If it is not, we do not need to request again.

2. Incremental updates

The specific idea of incremental data update is to add the concept of a version to the data, and compare the versions of the received data each time, and only receive the changed data.

In this way, the amount of data transferred will be reduced a lot, such as provincial and configuration data is less updated, if every time to request provincial and urban data, this is a waste of traffic.

We only need to update the data that has changed, so we won’t give you an example here because it’s very server related.

4.2 Data Compression

1. Gzip

In the case of a Post request, the Body is Gzip compressed, meaning that the request is sent with a Gzip request header and the server returns with Gzip compression, so that the data stream is compressed.

2. Compress the request header

The request header also occupies a certain volume. In the case that the request header remains unchanged, we can only pass it once. In the future, we only need to pass the MD5 value of the previous request header.

3. Merge network requests

Each network request will have redundant information, such as request headers, and merging network requests can reduce the transmission of redundant information.

4.3 Image Compression

1. The thumbnails

The first way to compress images is to give priority to thumbnails in the list, because showing the original image increases memory consumption and traffic consumption, and it doesn’t make sense to show the original image directly in the list.

Below is a comparison of the size of the original and the thumbnail, which is 50% of the original and 10% of the original.

2. WebP

The second method of image compression is to use Webp format. Here is a comparison of the same image in PNG format and Webp format. Webp format is 51% of the size of PNG format.

3. Luban

For example, when we upload pictures, we should do a compression, such as a 2M picture locally. It is not meaningful to upload the whole picture, which will only increase our traffic consumption. It is best to compress the picture before uploading.

And in the picture compression to do better is Lu Ban, let’s look at the use of Lu Ban.

First add the dependency.

dependencies {
  // Image compression
  implementation 'top. Zibin: Luban: 1.1.8'
}
Copy the code

Then add to compress the image.

The image below was originally 1.6m in size, compressed to 213KB, or 13% of its original size.

5. Three quality optimization schemes

Previously, we learned network request traffic optimization, but in fact, the biggest damage to user experience is the poor quality of network requests. Generally, we test with WiFi in the company during the development or testing stage, and the network quality is relatively good. Assuming that our App has no problem in traffic consumption, However, users often feedback that the interface cannot be opened, opened slowly, and pictures cannot be loaded.

At this time, users are likely to uninstall our App and switch to competing products. Only when the network request quality is high and user experience is good and users continue to use our App, users are likely to encounter the problem of traffic consumption. Therefore, network quality optimization is more critical than traffic optimization.

Two indicators of network quality optimization:

1. Network Request Success rate
2. Network request speed

Both of these metrics affect the user experience. Before introducing network request quality optimization, let’s look at the process for the next Http request.

The client sends a request, which reaches the carrier’s DNS server and is resolved to the corresponding IP address.

When a connection is created, three TCP handshake attempts are made, and a request is sent to the corresponding server based on the IP address.

The server finds the resource and returns it to the client.

5.1 HttpDNS

First of all, let’s look at how to optimize the step of sending requests. The success rate and speed of network requests are affected by the DNS server. If our DNS resolution to IP address is hijacked or DNS resolution is slow, user experience will be seriously affected.

The result of DNS hijacking is that the data users get is not the data we really want to provide to the user. If DNS resolution is slow, the user will wait longer for the request.

Therefore, DNS optimization is the first step of network quality optimization. We use HttpDNS to bypass the carrier domain name resolution process. Instead of using the traditional DNS protocol, HttpDNS sends requests to port 53 of the DNS server, but uses Http protocol to send requests to port 80 of the server.

There are two advantages to this:

1. Anti hijacking

   Reduce Local DNS hijacking, bypassing the carrier domain name resolution process.

2. speed

   Reduced average access times because a parsing process is saved;

Tencent cloud and Ali cloud both provide HttpDNS service, the specific implementation may see their official documentation.

5.2 Http Version Optimization

As mentioned earlier, the second step of network request is to create a connection, which involves TCP three-way handshake. This process is relatively long, if each request requires three-way handshake, it will be inefficient.

As a result, different versions of Http optimize this a lot. Here are the main differences between different versions of THE Http protocol.

• The Http 1.0

  The biggest disadvantage of the older version is that the TCP connection is not reusable. Each TCP connection can only send 1 request. If you want to request other resources, you must re-establish a connection.

  TCP was very expensive to create a connection, required a three-way handshake, and was slow to send initially, which meant that Http 1.0 had very poor performance.

• The Http 1.1

  Http 1.1 came only six months after 1.0, and the biggest change was the introduction of persistent connections. From this version on, persistent connections are not closed by default and can be reused by multiple network requests, resulting in significant efficiency improvements.

  It has some drawbacks, though it allows TCP reuse, but all data traffic within the same TCP connection must be processed sequentially, that is, after one request is processed, the next request is responded to.

  If the first network request is slow, the next one will have to wait.

• The Http 2.0

  Http 2.0 is a binary protocol, its biggest improvement is that the client and server can send multiple requests and responses at the same time, do not need to request in order like Http 1.1, is a two-way real-time communication.

  This is the biggest improvement of Http 2.0 over Http 1.1. When you want to cooperate with the server in the future, if you have a choice, choose the higher version of Http protocol as possible.

5.3 Capital Optimization

The last optimization scheme is to throw money at it. Common means include CDN acceleration, improved bandwidth and static and dynamic resource separation.

We need to note that after using CDN, if a certain resource needs to be updated, the cache needs to be cleaned after the update. These optimizations do not involve clients, and at the same time, we should not forget to reduce the transmission volume and pay attention to the timing and frequency of requests, which is related to the traffic optimization mentioned earlier.

The resources

• Play with Android performance analysis and optimization
• MacOS Charles 4.x version installation and use
• Setting up Charles to Proxy your Android Device
• Network Security Configuration
• Charles Network Setup
• Stetho
• Android application traffic statistics – used by NetworkStatsManager
• AppOpsManager permission detection adaptation
• Android in the uid
• Glide 4.x Loads images using custom OKHTTP (HTTPS validation ignored)
• ProcessLifecycleOwner determines whether to switch between front and back Android applications
• NetworkStatsCollection