This article is also an extension of the ADAT project, aiming at the testing part of the development cycle. It mainly introduces Xcode testing concepts and automated testing practices. It does not discuss how to write high-quality test cases. This article covers many of the concepts of Xcode build, most of which have been covered in the previous two articles, so it is recommended to browse through them first.
An overview of the content
- Xcode tests related concepts
- Test the relevant usage in the xcodebuild command
- Xcodebuild test practice
- Existing problems
Xcode tests related concepts
Before you start testing, it’s important to briefly cover the test-related concepts in Xcode, which are important for writing automated test commands. Because these concepts will be repeated in the text, it is important to know how they relate to each other when you use them.
Test package
Test Bundle, a Target for testing. Xcode 13.1 Checking Include Tests when creating a project automatically creates unit test packages and UI test packages. Multiple test cases are organized within the test suite.
The test case
TestCase, the specialized Class for testing, inherits from XCTestCase. Import the headers to be tested within the test case, set up the initial environment, and write multiple test methods.
The test method
Test Method, a Method used exclusively for testing, must be an instance Method with a name that starts with Test and has no parameters or return values. Otherwise, Xcode will not recognize it as a Test Method and will not call it automatically.
Unit testing
Unit Test is a kind of testing dimension. It mainly tests Code logic, and relies on good architecture design and writes Testable Code. Call the code you want to test in the test method and use XCTAssert and related methods to determine the results.
UI test
UI Test, in a coarser dimension to Test, simulate user operations. Xcode uses THE UI Test Recorder to record the sequence of operations and automatically inserts the sequence into the Test method in the form of code. Running the Test method is to “replay” the sequence once to observe the performance of the same operation on different device environments. UI tests can take screenshots where needed and remain live.
The test plan
Test Plan, a JSON file with the.xctestPlan extension, combines the Test package and the Configuration. You can set the Test package, Test cases, and Test methods to determine whether to participate in the Test. The Configuration includes a default Configuration and multiple custom configurations. Most configuration items come from Scheme. User-defined Configuration Unspecified configuration items are determined by the default configuration items.
In Scheme Manager, you can convert Scheme into a Test Plan or create a new Test Plan by going to Xcode -> Product -> Test Plan. For large projects, it is recommended to convert Scheme into a test plan because a test plan contains more configuration items, such as enabling test timeout, repeated testing, and JSON files are easier to version than Scheme.
At least one custom configuration must be created for the test plan, or any tests within the test plan will not run. Xcode does this automatically for us when we turn Scheme into a test plan. You can also add additional custom configurations.
The test report
Test Reports, which lists the Test results of each Test case, including execution steps, time, screen shots, and logs.
Code coverage
Code Coverage, which shows how much Code the test case covers, Xcode provides a visual interface from which to refine the test case. After code coverage is enabled and tests are executed, the right edge of the code editor shows how many times a line has been tested.
Test the relevant usage in the xcodebuild command
The xcodeBuild test command can be used to analyze the xcodeBuild test command. The xcodeBuild test command can be used to analyze the xcodeBuild test command. Run man xcodeBuild in Terminal to extract test related options. The instructions for each option and operation are as follows:
Multi-environment parallel testing
-disable-concurrent-destination-testing Disables parallel testing in multiple environments. If multiple environments are specified, finish one before starting the next.
-maximum-concurrent-test-device-destinations NUMBER If more than one real environment is specified during the test, the test can be performed on a maximum of one real environment at a time. And -disable-concurrent-destination-testing are mutually exclusive. If NUMBER is not specified, there is no upper limit.
-maximum-concurrent-test-simulator-destinations NUMBER If multiple simulator environments are specified during the test, the maximum NUMBER of simulators can be executed at a time. And -disable-concurrent-destination-testing are mutually exclusive. If NUMBER is not specified, the default value is 4.
Parallel testing with multiple runners
– the parallel – testing – enabled YES | NO whether to enable the parallel test. Parallel testing distributes test cases (classes) to different runners or processes for execution to improve test efficiency. The runner of a unit test is usually an instance of an application, and the runner of a UI test is a custom application created by Xcode.
-parallel-testing-worker-count NUMBER Specifies the NUMBER of parallel tests to be performed by the runner. -maximum-parallel-testing-workers NUMBER specifies the NUMBER of parallel tests.
-maximum-parallel-testing-workers NUMBER Specifies the maximum NUMBER of parallel test runners.
– Parallelize-tests-among -destinations Distributes parallel tests to multiple environments for execution. If parallel testing is enabled and tests are specified in multiple environments, test cases are distributed for execution in multiple environments rather than a complete set of tests being executed in each environment.
The test plan
-showTestPlans Displays test plans associated with Scheme. It must be used with – Scheme.
-testPlan specifies a testPlan when a test is executed. Only the file name of the testPlan is passed. Xctestplan suffix is not passed. This parameter must be used with -scheme.
-only-test-configuration tests only the specified configuration. The parameter is a user-defined configuration name and case sensitive. Note that this is a test plan configuration, not a variant of Target’s BuildSettings.
-skip-test-configuration Skips the test of the specified user-defined configuration.
Test the timeout
– test – timeouts – enabled YES | NO whether to enable test timeout.
-default-test-execution-time-allowance SECONDS Specifies the default timeout period of a test method. The unit is rounded up every 60 SECONDS. If SECONDS is less than 60 SECONDS, 60 SECONDS is counted; if SECONDS is greater than 60 SECONDS but less than 120 SECONDS, 120 SECONDS is counted, and so on. For example, if SECONDS is 59, it is 60. SECONDS is equal to 61, which is 120. To enable test timeout, run the following command: -test-timeouts-enabled YES. This is equivalent to the executionTimeAllowance of the XCTestCase instance, and the default timeout can also be configured in the test plan. The priorities from highest to lowest are:
- The executionTimeAllowance property value for XCTestCase
- Xcodebuild -default-test-execution-time-allowance
- Value of the Default Test Execution Time Allowance (s) configuration item for the Test plan
- If this parameter is not specified, the default value is 600 seconds.
-maximum-test-execution-time-allowance SECONDS Indicates the maximum timeout period of a test method, in SECONDS, rounded up every 60 SECONDS. To enable test timeout, run the following command: -test-timeouts-enabled YES. XCTestCase instances do not provide properties or methods to set, and there is no default maximum timeout, so in descending order of priority:
- Xcodebuild -maximum-test-execution-time-allowance parameter value
- Value of the Maximum Test Execution Time Allowance (s) configuration item for the Test plan
The timeout for a test method is determined by the lesser of the default timeout and the maximum timeout. When the timeout period is reached, the test is judged to have failed and a Spindump file is generated:
A subsequent article will explain how to parse Spindump files. Follow me for the first time: Virusbee – author of this article
Repeat the test
-test-iterations
Repeats the test
times.
– Retry-tests-on-failure Tries again after a test fails until the test succeeds or the total number of tests reaches. The default number of times is three. If -test-iterations
is specified, the total number of times is
. Cannot be used with -run-tests-until-failure.
-run-tests-until-failure Runs again after the test succeeds until the test fails or the total number of tests reaches. The default number of times is 100. If -test-iterations
is specified, the total number of times is
. Cannot be used with -retry-tests-on-failure.
– test – repetition – relaunch – enabled YES | NO whether each test are carried out in the new process, if for NO, all the test will be performed in the same process. This value must be combined with -test-iterations
, -retry-tests-on-failure, or -run-tests-until-failure.
Localization test
-testLanguage specifies the testLanguage that complies with ISO 639-1, such as nl, hr, and ar.
-testRegion specifies the testRegion. The region complies with the ISO 3166-1 standard, such as GN, KE, and IT.
Code coverage
– enableCodeCoverage YES | NO enable code coverage
Filter test cases
-only-testing: test-identifier Tests only the tests specified by test-identifier. For details about test-identifier, see TN2339: Faqs on Building with the Xcode command line How to implement unit tests with the command line?
-skip-testing: test-identifier Skips the TEST specified by test-identifier.
The test operation
Test Performs a test.
Build-for-testing generates the.xctestrun file as a parameter to the test-without-building.
Test-without-building tests with the.xctestrun file generated by build-for-testing, specified with the -xctestrun option.
For details on how to use the test operation, see TN2339: FAQ on Building with the Xcode Command Line How to Use the Command Line to Implement unit Tests? And how to use the command line to implement Build For Testing and Test Without Building functions in Xcode?
Xcodebuild test practice
Test requirements
Project Overview An iOS application, which supports iOS 9.0 at least. The app was released to Hong Kong, Japan and South Korea; Support simplified Chinese, traditional Chinese, Japanese, Korean, English 5 languages; There are several unit and UI test cases, some of which have been integrated into test plans.
Targets: App (primary application), AppTests (unit test package), AppUITests (UI test package) Scheme: Supported languages: Simplified Chinese zh-Hans, Traditional Chinese zh-Hant, Japanese JA, Korean KO, English EN Use cases (classes) in the unit test package for Core.xctestPlan (only one default configuration and one custom configuration with default configuration items) : AccountTests, DataTests, ParseTests, HelperTests (testExample is not tested) Use cases (classes) in the UI test package: LoginTests, ChatTests, PaymentTests
Existing equipment
| type | The name or ID | OS |
|---|---|---|
| A: | ID: 4 cbbc9c59cd4c29dad494141b81b2f64ab45d643 | IOS 15.1 |
| A: | Df4313a ID: 00004032-102044371 | IOS 14.5.1 |
| The simulator | Name: iPhone 13 Pro | IOS 15.0 |
| The simulator | Name: iPhone SE | IOS 13.5 |
| The simulator | Name: iPad Pro | IOS 12.4 |
Test requirements The test plan needs to complete 10 tests on each device until they fail, each test is limited to 1 minute, and the simulator can run at most 2 at the same time; The rest of the test cases adopt parallel testing, test failure does not stop, test up to 10 times; Output code coverage, test reports, logs, and failure screenshots.
Requirements analysis and implementation
According to the test requirements, the test plan is executed separately from the rest of the test cases, and two commands are written.
For the testPlan core. xctestplan, specify -testplan Core; The test plan must be used with Scheme, and the test plan is configured under Scheme App, that is, -scheme App. Run 10 times, that is, -test-iterations 10. Until it fails, i.e. -run-tests-until-failure; To ensure that you test in a new process every time, add -test-repetition-relaunch-enabled YES; -tests-timeouts-enabled YES = -maximum-test-execution-time-allowance 60 = -maximum-test-execution-time-allowance 60 = -maximum-test-execution-time-allowance 60 = -maximum-test-execution-time-allowance 60; Each device listed in the above table will participate in the test, and the -destination parameter value will be constructed according to their type, name, ID, and version. The emulator can run up to 2 at a time, since the emulator defaults to run up to 4 at a time, set -maximum-concurrent-test-simulator-destinations 2; Output code coverage, that is, enableCodeCoverage -enablecodecoverage YES; To easily obtain code coverage, test reports, and so on, output the test results to the specified location result1, i.e. -resultBundlePath result1.
Complete command:
$ xcodebuild test \ -project App.xcodeproj \ -scheme App \ -testPlan Core \ -test-iterations 10 \ -run-tests-until-failure \ -test-repetition-relaunch-enabled YES \ -test-timeouts-enabled YES \ -maximum-test-execution-time-allowance 60 \ -destination "platform=iOS,id=4cbbc9c59cd4c29dad494141b81b2f64ab45d643" \ -destination "platform=iOS,id=00004032-102044371DF4313A" \ -destination "platform=iOS Simulator,name=iPhone 13 Pro,OS=15.0" \ -destination "platform=iOS Simulator,name=iPhone SE,OS=13.5" \ -destination "platform=iOS Simulator,name=iPad Pro,OS=12.4" \ - maximum-concurrent-test-simulation-destinations 2 \ -enablecodecoverage YES \ -quiet \ -resultBundlePath result1Copy the codeConsider, after this command is executed and all tests pass, how many times does a normal test method execute? The answer is 50 times. This is a parallel test, so the test plan is fully executed 10 times on each device, with 5 devices participating in the test, and the test plan has only one custom configuration: 10 times * 5 devices * 1 configuration = 50 times.
For the rest of the test cases, the unit test cases belong to the AppTests package and the UI test cases belong to the AppUITests package. Both packages are configured under the AppTests Scheme, so specify -scheme AppTests. AppTests package HelperTests class the testExample method without any test, namely the skip – testing: AppTests/HelperTests/the testExample; Perform a maximum of 10 iterations, that is, -test-iterations 10. -retry-tests-on-failure does not stop when a test fails. To use parallel testing, first enable parallel testing (-parallel testing-enabled YES) and then run test cases in -destination (-parallelize-tests-among-destinations). The rest of the configuration is consistent with the test plan.
Complete command:
$ xcodebuild test \
-project App.xcodeproj \
-scheme AppTests \
-skip-testing:AppTests/HelperTests/testExample \
-test-iterations 10 \
-retry-tests-on-failure \
-destination "platform=iOS,id=4cbbc9c59cd4c29dad494141b81b2f64ab45d643" \
-destination "platform=iOS,id=00004032-102044371DF4313A" \
-destination "platform=iOS Simulator,name=iPhone 13 Pro,OS=15.0" \
-destination "platform=iOS Simulator,name=iPhone SE,OS=13.5" \
-destination "platform=iOS Simulator,name=iPad Pro,OS=12.4" \
-parallel-testing-enabled YES \
-parallelize-tests-among-destinations \
-enableCodeCoverage YES \
-quiet \
-resultBundlePath result2
Copy the codeConsider, after this command is executed and all tests pass, how many times does a normal test method execute? The answer is once. This is because -retry-tests-on-failure determines only subsequent operations that fail the test, while -run-tests-until-failure determines only subsequent operations that succeed the test. It is not specified here, nor can it be specified, and the two are mutually exclusive. If the first execution is successful, resulting in all methods being tested only once, then the test is not sufficient, which is what the test requirements are, and it can be inferred that the requirements are not reasonable.
The test results
The test results are saved in the path specified by -resultBundlePath, which cannot be an existing path. The test results for the above test plan are result1.xCREsult, and the test results for the remaining test cases are result2.xCREsult.
Double-click result1.xcresult automatically open in Xcode to view test results:
Code coverage:
Existing problems
localization
The command above is not perfect enough to support testing requirements for multiple locales and languages, and in some cases to simulate geographical locations. How should this be handled?
Delegate testing to a third party
If the test task is heavy and there are not many test equipment available, it is often necessary to delegate the test task to a third party, but the source code is not available. How to deal with this situation?
Integrate with the continuous build system
Many organizations have their own continuous build systems that automatically email followers when a project is built. It is unwise to send the.xcresult file directly to the mail, so how do you parse the.xcresult file for metadata and generate custom test reports, code coverage interfaces?
Due to limited space, there is no explanation here. There will be a follow-up article to provide solutions. Please pay attention to the author for the first time: Virusbee – the author of this paper
Good forecast
There are a lot of concepts in this paper, and the practice part also needs to combine concepts to write code step by step verification. If it is not easy to understand, it is suggested to collect it and slowly digest it.
Here are the plans for subsequent articles (titles and order may change, depending on the actual release) :
- Explore Xcode command line usage three: XcodeBuild package
- Explore Xcode command line usage 4: CoDesign signatures
- Explore Xcode command line usage 5: upload and distribute
- Explore Xcode command line usage 6: Jenkins keeps building
- Explore Xcode command line usage 7: XcodeBuild test related problem solutions