Shell commands used to test startup

This article is more about performance, startup testing, and the reason behind my startup testing. But if you just want a quick conclusion, you can refer directly to the following:

  1. Lock the CPU frequency whenever possible (see below);
  2. From the command line, run the following command (to make sure your device is connected).
$ for i in `seq 1 100` > do > adb shell am force-stop com.android.samples.mytest > sleep 1 > adb shell am start-activity  -W -n com.android.samples.mytest/.MainActivity | grep "TotalTime" | cut -d ' ' -f 2 > doneCopy the code

The command above loops 100 times: start the application, print out how long it takes to start the process, and then terminate the process to prepare for the next loop.

It’s not easy to “test” startup performance well

I recently needed to test the Startup performance of an app (and played around with the Startup library to see how it affected Startup performance, more on that in a future article). I found, as I always do with this kind of thing, that boot performance was not easily tested unambiguously.

If you’re testing a piece of runtime code, there are a number of solutions. From the trivial of “writing tight loops and counting time increments using System.CurrentTimemillis ()” to more complex and useful solutions such as those provided by the AndroidX Benchmark library.

But by definition, many of the actions at application startup run before the system calls your code. So how do you determine how long the entire startup process will take?

I browsed through some log messages, checked out some underlying apis, and asked some engineers on the platform team to come up with some useful information. Even better, I can now fully automate my tests and output information using adb shell tools, making it easy to import the results into a spreadsheet for analysis.

I’ll explain some of the code snippets used by the above commands in the text below, and show you one or two simple steps to start the test.

ActivityTaskManager startup logs

As I wrote in an earlier (unfortunately outdated and incorrect) blog post, a very handy log has been keeping track of system information since KitKat was released. Whenever an Activity starts, you can see the following output from the tool in the log:

ActivityTaskManager: Displayed com.android.samples.mytest/.MainActivity: +1s380ms
Copy the code

This duration (1,380ms in this case) represents the time it takes from the time the application is started to the time the system considers it “started”, including drawing the first frame (hence the “displayed” state).

The process of getting to the “Displayed” state does not have to include the time spent doing things before your application was ready. As long as your application has been finished loading and initialization, you can through a call to the Activity. The reportFullyDrawn () to the system to provide the additional information. When you call this optional method, another log is logged with a timestamp and duration:

The 2020-11-18 15:44:02. 171, 1279-1336 / system_process I/ActivityTaskManager: Fully drawn com.android.samples.mytest/.MainActivity: +2s384msCopy the code

I only want how long it takes until “displayed”, so the built-in log is good enough for me.

Automatic start

Performance testing should always run test cases multiple times to eliminate variables in the results. The more runs you run, the more reliable the average is. I try to run the test at least ten times, but doing more would be better. Depending on how the results vary and over time (since the presence of variables can have a greater impact on shorter tests), you may need to run more times.

Insanity is doing the same thing over and over again and expecting a different result.

— Albert Einstein

Performance test corollary:

“Crazy” is doing the same thing once and hoping for the best result.

Not Einstein

Launching an app multiple times in a row by clicking on an icon can be tedious. It’s also inconsistent and unpredictable because it’s easy to introduce variables — if you accidentally start another application by mistake, or cause the system to do extra work and not get timing results.

Therefore, what I really want is some way to launch the application from the command line. With it, I can run the command repeatedly to perform the same operation, avoiding the variability (and tedium) that comes with manually launching the application.

Adb (Android debug Bridge, which should be familiar to readers by now) provided what I needed. More specifically, ADB Shell provides a command-line interface for launching applications: ADB shell am start-activity. The command also blocks until the application is started, so we’ll use the -w argument (which is necessary for the next step). We will next use the following command to kill the started application. Here’s the full startup command:

$ adb shell am start-activity -W -n 
com.android.samples.mytest/.MainActivity
Copy the code

The last parameter is the application package name and component information. You can see that they are the same logs as the ActivityTaskManager output in the previous section.

Running this command launches the application (unless the application is already in the foreground, which is not ideal and we will deal with that in the next step) and prints the following:

Starting: Intent { cmp=com.android.samples.mytest/.MainActivity }
Status: ok
LaunchState: COLD
Activity: com.android.samples.mytest/.MainActivity
TotalTime: 1380
WaitTime: 1381
Complete
Copy the code

Check the TotalTime result: the result is exactly the same as the information we saw in the log:

ActivityTaskManager: Displayed 
com.android.samples.mytest/.MainActivity: +1s380ms
Copy the code

This means that we don’t need to look at Logcat, but can get the information directly from the console where the command is being run. Even better, we can strip out excess text and just keep the startup results, making it easier to extract this data for use elsewhere.

To convert the above output to boot duration, I output the content using grep and cut shell commands (there are several ways to do this, I just chose one at random):

adb shell am start-activity -W -n 
com.android.samples.mytest/.MainActivity | grep "TotalTime" | cut -d ' ' -f 2
Copy the code

Now, when I run this command, I get a simple number as I expected:

$ [start-activity command as above...]  1380Copy the code

A cold start is the best starting point for performance testing

Before you check startup performance, it’s a good idea to understand the difference between “cold start” and “hot start.”

A “cold boot” is the first time your application is started, restarted, or not in the background after installation.

“Hot start,” on the other hand, is when your application is already up and running (but suspended) in the background.

Both cases are worth testing and understanding. But in general, a cold boot is the best place to start your startup performance testing for two reasons:

  • Consistency: A cold start ensures that your application undergoes the same operation every time it starts. When an application is hot started, there is no way to know exactly what steps are being skipped and what steps are being executed, so there is no way to know exactly what you are timing (and no guarantee that what is being tested is consistent when repeated testing);
  • Worst case: By definition, a cold start is the worst case — this is the scenario where your users experience the boot process the longest. You need to focus on worst-case statistics, not best-case hot starts. If you ignore the worst-case scenario, many major problems will go unsolved.

To force a cold start on each run, you need to terminate the application between runs. Again, doing this on the screen (for example, sliding the application from the launcher’s “overview” list) is tedious and error-prone, and the ADB shell solves this problem.

There are several different shell commands that can be used to terminate an application. Adb shell am kill… . But in fact this order does not solve the problem. When you start an application, the application is in the foreground, and kill does not terminate the foreground application. Instead, you need to use the force-quit command:

adb shell am force-stop com.android.samples.mytest
Copy the code

You can use the application’s package name to tell it which application needs to be terminated.

I love loops. Let’s do loops

You now have a series of commands to start the application, output startup duration data, and exit the application so that it can be started again. You can type this over and over again in the console, but in the shell we can put these commands in a loop and run it repeatedly with a single command.

When you do this, you may want to delay the next startup after terminating the application to avoid side effects (for example, when the application is terminated, the system pulls the launcher to the foreground). To do this, I added a one-second sleep to insert a small buffer between operations.

Here is the final version of the command I used, which includes terminating the application, waiting for a second, and then restarting the application. I repeated this process 100 times to provide a reasonable sample size:

$ for i in `seq 1 100` > do > adb shell am force-stop com.android.samples.mytest > sleep 1 > adb shell am start-activity  -W -n com.android.samples.mytest/.MainActivity | grep "TotalTime" | cut -d ' ' -f 2 > doneCopy the code

When running this command, I get the boot duration output to the console every time the boot is complete, which is the data I want to track and analyze.

Note: There is a simpler way to do this. You can start the Activity in a loop using -s (to stop the Activity first) and -r COUNT (to execute the start-activity command COUNT times), so I can also do this with the following command:

$ adb shell am start-activity -S -W -R 100-n com.android.samples.mytest/.MainActivity | grep "TotalTime" | cut -d ' ' -f  2Copy the code

However, in order to allow buffer time between application termination and startup to ensure that it is inactive, I wanted to use the sleep 1 command, so I took a more verbose approach to the loop. Besides, the code for shell scripts is very elegant, isn’t it?

Lock the main frequency as much as you can

CPU architecture, especially CPU frequency, is an important factor affecting the performance of mobile devices. Specifically, one of the main ways mobile devices can reduce power consumption and avoid overheating is by limiting CPU speed.

Limiting the CPU is useful for saving power, but has a negative impact on performance testing, where consistency of results is critical.

Ideally, you should control the CPU frequency when running performance tests. However, whether or not you can perform this operation depends on the device you own — you need root access to the device to control the CPU governor and thus the CPU frequency, and different devices may perform this behavior differently.

What follows only applies if your device allows and you can get root access. On the device side, I know Pixel devices can get access, but that doesn’t mean other devices can.

In any case, it is recommended that you lock the CPU frequency if you can. For your particular test, there may not be a noticeable impact (in fact, the system usually starts the application with the CPU running at a high frequency, so it may already provide the required consistency). However, doing so can at least eliminate the variable CPU frequency.

Locking the CPU frequency manually can be tricky, but luckily AndroidX Benchmark makes it easy for you. In fact, you don’t even need to write code for the Benchmark API — you can use the library by using the lockClocks and unlockClocks tools provided.

First, add a benchmark dependency to the engineering-level build.gradle file:

No. / / to check the latest version of the Benchmark library / / https://developer.android.google.cn/jetpack/androidx/releases/benchmark def benchmark_version = "1.0.0" classpath "androidx. Benchmark: benchmark - gradle - plugin: $benchmark_version"Copy the code

Next, apply the Benchmark plugin to the build.gradle file at the application level:

apply plugin: androidx.benchmark
Copy the code

Now you can synchronize your project (Android Studio may already be forcing you to do this), and once that’s done you can use the lock task from Gradlew.

You can now lock the main frequency by running a command on the command line (I ran it through the “terminal” tool inside Android Studio, but you can also run it outside the IDE):

$ ./gradlew lockClocks
Copy the code

After I run the command, I see the following output on the command line:

Locked CPUs 4,5,6,7 to 1267200/2457600 KHz Disabled 0,1,2,3Copy the code

This output shows that Benchmark works fine on my Pixel 2. The even better news is that my startup tests now take much longer than they used to. You may wonder why the main frequency is slowing down.

The Benchmark tool locks the main frequency to a level that is easy to run continuously, not a high performance level. Better performance may be achieved if the main frequency is set as high as possible, but:

  • To make the test results realistic enough, you might even expect worse performance, as many users experience in the real world. You don’t want to see only best-case performance, because that’s not what people typically encounter in reality;

  • Running cpus for too long at high frequencies can cause them to overheat. I don’t know how the system will respond if it overheats (hopefully it will lower the frequency or shut down the system automatically before something serious happens), but I don’t want to know either.

Please note that after completing the test, you need to unlock the main frequency. The device will be unlocked on reboot, but you can also unlock the master by running the opposite Gradle task:

$ ./gradlew unlockClocks
Copy the code

This command simply restarts the device to perform the reset operation. For more information on Benchmark locking, check out the User guide.

And you’re done!

After locking the clock, I had everything in place: a system that reliably reproduced boot conditions, a simple command line that returned a stream of results when executed. I can copy and paste the results into a spreadsheet and analyze them (by comparing the average startup time to the various scenarios I want to try).

Ideally, I wouldn’t need to write an article explaining how to do all of this. To be honest, you don’t need all of the above. (But it’s always more fun to know how things work and why, isn’t it?) All you really need is the for() loop shell command and an optional way to lock the main frequency.

$ for i in `seq 1 100` > do > adb shell am force-stop com.android.samples.mytest > sleep 1 > adb shell am start-activity  -W -n com.android.samples.mytest/.MainActivity | grep "TotalTime" | cut -d ' ' -f 2 > doneCopy the code

To simplify performance testing and analysis, and to improve application performance in general, our team is working on ways to simplify this process, so stay tuned for more updates. In the meantime, I hope the above commands and information are helpful for you to start the performance test.