Preface:

Due to the rapid development of mobile applications, many companies need to develop mobile applications to gain market competitiveness. When developing mobile applications, they are usually faced with the choice of various frameworks, especially for IOS and Android, such as Native development:

  • IOS App: Swift/Objective-C
  • Android App: Kotlin/Java

The development of the above two ends requires developers to learn completely different technologies. The DIFFERENCES in UI rendering mechanism and display of the same App at the two ends are very large, and the development and testing effort will increase. Therefore, cross-platform technology solutions beyond the original emerge: Using the same framework and language can be applied to different platforms. Flutter is a very popular cross-platform development framework. Because the implementation mechanism of Flutter is different from other frameworks, the test strategy for Flutter will be different from other frameworks from the perspective of testing. This chapter will start with the principle of Flutter. Learn more about Flutter testing from the analysis of Flutter testing strategy to the project practice.

1. What is Flutter?

Flutter is Google’s free, open source mobile UI framework for quickly building high-quality native user interfaces on iOS and Android. Optimised for current and future mobile devices, Focusing on low latency input and high frame rates on Android and iOS, Flutter has several significant advantages:

  • Fast development: It can realize hot loading. After the code is modified, the application interface will be updated immediately. Rapid debugging can be realized through hot loading for development and testing.
  • Expressive and flexible UI: Quickly deliver functionality focused on the native experience. The layered architecture allows for complete customization, enabling incredibly fast rendering and expressive, flexible design.
  • Native performance: Flutter contains many core widgets, such as scrolling, navigation, ICONS and fonts, that can perform as well as native apps on iOS and Android.

Flutter is being used by more and more developers and organizations around the world. Here is a chart of the trend of Flutter in the market over the past five years (source: Google Trends) :

We can see that the use of Flutter is beginning to lead and keep rising. Take a look at the apps developed by Flutter on the market:

In summary, Flutter is very competitive in the market and there is a high probability that beta mobile will touch Flutter. When QA touches a Flutter project, can we use a similar testing strategy to other beta mobile apps? How is Flutter different from other projects, and what should be noticed?

2. Flutter is different from other frames

To know if other mobile test strategies are applicable to the Flutter project, it is important to know what makes Flutter applications different from other applications.

2.1 Differences in rendering mechanics:

In addition to the Native mode mentioned above, the development of mobile terminals is mainly cross-platform framework, among which React Native and Flutter are the most popular ones in recent years:

Let’s take a look at some of the differences in front-end rendering principles:

  1. First of all, for the implementation of Native (the most middle figure, take Android as an example), it is the original Development framework of Android. Java calls Skia (2D vector graphics processing function library, currently the official image rendering engine of Android), which is regenerated into CPU/GPU instructions and finally rendered on the screen.
  2. For React Native, Java is called through JS, then Skia is called, which is regenerated into CPU/GPU instructions and finally rendered on the screen.
  3. For Flutter, developers write DART code (officially the only specified development language) that calls Skia directly, regenerates it into CPU/GPU instructions, and eventually renders it on screen.

From the above analysis, we can see:

  • If the pure Native implementation is adopted, there is a huge difference between Android and IOS, because their respective rendering mechanisms are different, and testers need to completely implement full tests for both ends, which will cost a lot of time and cost in development and testing.
  • React Native, though it only writes one set of code for development, still calls the Native rendering mechanism in principle. In terms of testing, it is not much different from the APP test implemented in the Native implementation.
  • However, Flutter bypasses the native rendering mechanism entirely and renders in a custom way, which gives a great advantage to Flutter App:
    1. Performance comparable to native: direct calls to the underlying rendering engine with no intermediate translation costs.
    2. Achieve consistency across platforms: bypass native rendering mechanisms and mask native differences.
    3. High degree of customization: a large number of rich auto-moving components.

2.2 The naming mechanism is different from typesetting

The properties of the front-end elements of a Flutter generally include Type, tooltip, keyValue,text, etc. Among these properties, only text can be found by using UiAutomator. However, many components of a Flutter have no text at all, such as icon, image, etc. Therefore, Traditional automated testing tools are not enough.

2.3 Flutter’s test ecosystem is different.

Flutter has a complete and detailed test stratification:

  • Unit tests: Tests a single function, method, or class.
  • The widget test(Called in other UI frameworksComponent test) test a single widget. Testing widgets involves multiple classes and requires a test environment that provides the appropriate widget lifecycle context.
  • Integration testing: Tests a complete application or a large portion of an application. Typically, integration tests can be run on a real device or on an OS Emulator, such as an iOS Simulator or Android Emulator. The goal of integration testing is to verify that the application as a whole works correctly and that all the widgets it is made of integrate with each other as expected. Integration testing can also be used to verify application performance.

The integration test mentioned above is the front-end E2E test. It can be seen that Flutter already has a very complete automatic test system, including App test support, CI/CD integration, auxiliary performance test, etc.

2.4 Different Build methods

Flutter supports building apps in one of three modes. It also supports testing in Headless mode.

  • The Debug mode.
  • Profile mode.
  • The Release pattern.

The difference between these three modes is very important for the selection of testing strategy, so let’s analyze the purpose and characteristics of the three modes in detail.

The Debug mode

To facilitate dev debugging during development, Debug packages are compiled by running Flutter Run or by running the IDE. Debug mode packages have the following features:

  1. Hot reload only works in Debug mode.
  2. Emulators and emulators can only run in Debug mode;
  3. In Debug mode, application performance may drop frames or lag. In Profile mode, application performance is closer to real performance.

The Release pattern

When publishing your application, you need to choose to use the Release build mode. The release mode can be compiled by running flutter run –release. The release package has the following characteristics:

  1. Debugging information is not visible.
  2. Debugging is disabled.
  3. Compilation is optimized for fast startup, fast execution, and the size of small packages.
  4. It can only run on a real machine.
  5. Generally after CI/CD integration, the package is released mode.

Profile mode

Typically used to test the performance of your app, in Profile mode some debugging capabilities are reserved – enough to analyze your app’s performance. On emulators and emulators, Profile patterns are not available because their behavior does not represent reality.

Based on these three compilation modes, we can analyze that the release package does not allow debugging and extension functions, so we cannot run E2E tests on it.

Suppose we had to run e2E test in release mode, run:

flutter drive --release --target=test_driver/e2e.dart
Copy the code

You can see the following errors:

Flutter Driver (non-web) does not support running in release mode.
Use --profile mode for testing application performance.
Use --debug (default) mode for testing correctness (with assertions)
Copy the code

2.5 summary –Analysis of Flutter test strategy

To sum up, due to the implementation principle, front-end layout, naming mechanism, test ecology, and packaging and compilation characteristics of Flutter, test strategies will be different, mainly reflected in the following aspects:

  1. Manual testing:
    1. Just test one end: the front end of each platform appears almost the same, masking native differences.
    2. Attention should be paid to the particularity of individual mobile phone manufacturers: For example, Samsung has customized functions that make some of its behaviors inconsistent with those of other mobile phones.
    3. Proxy packet capture is not supported: Generally, mobile testing must use a packet capture tool, such as Charles. However, Flutter does not allow packet capture. Therefore, the app needs to be implemented in some way to enable this function.
    4. Android supports the Android Debug Bridge (ADB) tool.
  2. Automated testing:

    1. The test tool uses the Automatic test tool, Flutter Driver, which is officially recommended by Flutter. See the detailed strategies below.

    2. If you run tests using an emulator, you need to run tests in Debug mode, which is also convenient to run tests in Headless mode with CI/CD integration.

    3. If you want to test front-end performance through e2E test, you can use the Profile mode. However, in this mode, the server can run only on the real server, and the CI/CD integration needs to be connected to the real server.

3. Flutter Driver

3.1 what isFlutter Driver?

If you are familiar with Selenium/WebDriver (Web), Espresso(Android), or UI Automation(iOS), then The Flutter Driver is the Flutter equivalent of these integration test tools.

The Driver of Flutter is:

  • A command line toolflutter drive
  • A packagepackage:flutter_driver(API)

That is: The E2E test script (Made by DART) uses the API provided by the Flutter_Driver to locate the elements in the APP and define a series of actions. The Flutter_Driver drives the APP to perform operations in the automatic test script on real machines or virtual machines.

3.2 Install the Flutter Driver

Required before installing the Flutter Driver

  • The Flutter environment and dependencies have been installed and configured. For details, see the tutorial on the website.
  • There is already a code base for the project.

Note:

  1. The E2E test of Flutter is different from other types of Flutter. Normally, the E2E test code of Flutter is kept separate from the project codebase to facilitate QA maintenance and management. However, the E2E test of Flutter needs to be written in the project codebase.
  2. To demonstrate the process, we will use a demo: Counter App.

Next, prepare to install the Flutter Driver

3.2.1: Add the Flutter Driver dependencies

Assuming you have set up the flutter environment and dependencies and have the project’s codebar, go to the pubspec.yaml file ** (Flutter dependency Management TBC) ** and add the following dependencies:

dev_dependencies:
  flutter_driver:
    sdk: flutter
  test: any

Copy the code

3.2.2: Create a test file

Create a folder called test_driver and create two files in that folder: test_driver

  1. The first file only needs to be.dartThe ending will do, as in:e2e.dart. The main purpose of this file is to introduce the Flutter Driver extension and start the application.
  2. The second file needs a suffix after the name of the first file_test, such as:e2e_test.dart. This file is where the test scripts are actually written.

After adding, the project’s root directory will look something like:

project_name/
  lib/
    main.dart
  test_driver/
    app.dart
    app_test.dart
Copy the code

3.2.3: Create an instructed Flutter application

The application of a command is a Flutter application, it enable the Flutter Driver extensions, enabling extension please call enableFlutterDriverExtension (), the following code added to the newly created e2e. Dart files:

import 'package:flutter_driver/driver_extension.dart';
import 'package:project_name/main.dart' as app;

void main() {
  // This line enables the extension.
  enableFlutterDriverExtension();

  // Call the `main()` function of the app, or call `runApp` with
  // any widget you are interested in testing.
  app.main();
}
Copy the code

Important: This file first enables the Flutter Driver extension and then starts the entire application development. As you can see from the second line, when we run the test, all we need to start is to call the main.dart file on the front end, which is why we need to integrate the test script and project into a code base.

3.2.4: Writing the E2E Health Check test

Add the following files to the e2e_test.dart file with comments for each step:

// Imports the Flutter Driver API.
import 'package:flutter_driver/flutter_driver.dart';
import 'package:test/test.dart';

void main() {
  group('Counter App Test', () {
    FlutterDriver driver;

    // Connect to the Flutter driver before running any tests.
    setUpAll(() async {
      driver = await FlutterDriver.connect();
    });

    // Close the connection to the driver after the tests have completed.
    tearDownAll(() async {
      if(driver ! =null) { driver.close(); }}); }); }Copy the code

The script above establishes a connection with the Flutter Driver before the test starts and closes the connection after the test starts. Check whether the connection to the Flutter Driver has been successfully established by checkHealth() before writing the actual test script:

test('check flutter driver health', () async {
  Health health = await driver.checkHealth();
  print(health.status);
});
Copy the code

3.2.5: Run tests

To complete the above steps, connect to the virtual machine or real machine and run the following command:

flutter drive --target=test_driver/e2e.dart
Copy the code

This command will:

  • build-targetApply and install it on the device.
  • Start the application.
  • runtest_driver/Under thee2e.dart

The flutter drive command uses a convention to find test files with the same file name but the _test suffix in the same directory as the –target application.

The Flutter Driver is configured if the following return occurs:

00:02 +0: Counter App check flutter driver health

HealthStatus.ok

00:02 +1: Counter App (tearDownAll)

00:02 +1: All tests passed!

Stopping application instance.
Copy the code

You may have noticed that every time you run a test script, you need to start the application and run the test again, which takes a long time. The Flutter driver provides the Hot Reload function, but only applies to local environments. For details, see medium.com/flutter-com… .

3.3 Writing test scripts for E2E Projects

3.3.1: Write test scripts

Next, we need to write the actual test script. The demo we want to test is a Counter, as shown below:

Test process:

  1. Find The button that increases The count (The yellow button)
  2. Click on it
  3. The validation count changes from 0 to 1
  4. Click again on the
  5. The validation count changes from 0 to 2

First you need to find the button. You can use the following methods to find the element:

byTooltip(...)
byType(...)
byText(...)
byValueKey(...)
bySemanticsLabel(...)
Copy the code

Chapter 1.3 mentioned the naming mechanism of flutter, and the properties of the component are type, tooltip, keyValue,text, etc. Therefore, we need to find the flutter according to the properties of the reshuffle and the corresponding methods. For example, the implementation of this button is as follows:

floatingActionButton: FloatingActionButton(
        onPressed: _incrementCounter,
        tooltip: 'Increment',
        child: Icon(Icons.add),
      )
Copy the code

It has the attribute “Increment” for all tooltips, so it can be found by:

final buttonFinder = find.byTooltip('Increment');
Copy the code

To find the component, click:

await driver.tap(buttonFinder);
Copy the code

Finally, to verify whether the count is 1, we need to find the count component first, which is implemented as follows:

Text(
  '$_counter',
  style: Theme.of(context).textTheme.display4,
  key: ValueKey("counterText"),),Copy the code

You can see that the contents of its text property are not fixed and will change with the count, so use keyValue:

final counterTextFinder = find.byValueKey('counterText');
Copy the code

Finally, the verification results can be verified in the following ways:

//Get text from counterTextFinder, then compare result
expect(await driver.getText(counterTextFinder), "1");

//wait this text 
await driver.waitFor(find.text('1'));
Copy the code

Once again, the complete test is:

test('Increment the counter', () async {
  // First, find widget
  final counterTextFinder = find.byValueKey('counterText');
  final buttonFinder = find.byTooltip('Increment');
  
  // Then, tap on the button
  await driver.tap(buttonFinder);

  // Then, verify the counter text has been incremented by 1
  expect(await driver.getText(counterTextFinder), "1");

  // First, tap on the button
  await driver.tap(buttonFinder);

  // Then, verify the counter text has been incremented by 1
  expect(await driver.getText(counterTextFinder), "2");
});
Copy the code

In addition to the methods used above, commonly used are:

//enter text
final example = find.byValueKey('example');
await driver.tap(example);
await driver.enterText("text");

//scroll screen until the widget is completely visible 
final example = find.text('example');
await driver.scrollIntoView(example);

//to wait until the target specified in finder is visible
final example = find.text('example');
await driver.waitFor(example);

//to wait until the target specified in finder is no longer available
final example = find.text('example');
await driver.waitForAbsent(example);
Copy the code

For more information about how to do this, see the Flutter Driver API documentation.

The above is about functional testing methods. For visual testing, the Flutter Driver still provides great screenshots:

In the setUpAll() method in the e2e_test.dart file, add the following code:

newDirectory (" screenshots "). The create ();await Directory('e2e_test/screenshots').create();
Copy the code

Add the takeScreenshot() method to the same file:

Future<void> takeScreenshot(FlutterDriver driver, String path) async {
  var pixels = await driver.screenshot();
  var file = File('e2e_test/screenshots/' + path);
  await file.writeAsBytes(pixels);
}
Copy the code

Add the takeScreenshot() command to the test case:

await takeScreenshot(driver, 'screenshot_name.png');
Copy the code

Dart: Run flutter drive –target=test/e2e. Dart:


00:01 +0: Counter App check flutter driver health
HealthStatus.ok
00:01 +1: Counter App Increment the counter
00:04 +2: Counter App Test with alert window
00:05 +3: Counter App (tearDownAll)
00:05 +3: All tests passed!
Stopping application instance.
Copy the code

E2e_test /screenshots:

3.3.2: Find the location widget

For Native mobile automation, you can use uiAutomateViewer and Appium Inspector from the Android Sdk package, but these tools cannot be used by the Flutter app. The reason is the difference between the UI naming mechanism and the front-end layout mentioned above, so we need to use the Flutter plugin, The Flutter Inspector, and the Dart Devtools.

1. Flutter Inspector

First open IntellJ and run flutter Run. You can see the Flutter Inspector on the panel.

Click on the Flutter Inspector to see the following screen:

From the Flutter Inspector, we can see the widgets and Render Tree of our application. Click the button shown below to activate the Select Widget Mode:

Then click on the widget you want to locate on the emulator or real phone.

  • The Flutter Inspector automatically displays the widget’s structure and render details.
  • The widget’s properties defined in the source application are automatically displayed.

Now you can find the attributes of the element.

2. Dart DevTools

We can also locate widgets in another way by opening IntellJ, running Flutter Run, and clicking the DevTools button as shown below:

Wait a few seconds and a Web page will automatically open:

Dart DevTools will automatically display information about the structure and layout of the element (this process is similar to using the Flutter Inspector) by clicking on the Select Widget Mode button shown below and clicking on the element you want to locate on the emulator or real machine.

As you can see from the image above, the Dart DevTools allows you to view element properties using the Flutter Inspector, so the first method is sufficient to view elements only.

Looking at the Dart DevTools Web panel navigation, in addition to the Flutter Inspector, there are performance-related properties such as “Performance, CPU profile”, so, Dart DevTools can be used to aid in testing application performance.

3.3.3: The Flutter Driver extension

  1. Behavior-driven Development (BDD) tool: Flutter_gherkin

BDD describes and writes test cases from the perspective of functional users in the same way as user stories or use cases are written in natural language or quasi-natural language. The most significant advantages of BDD are:

  • Enhance intra-team collaboration and cross-functional team communication.
  • Complete processes that ensure that user stories are implemented from a business perspective, implemented from a technical perspective, and ultimately tested from a business perspective.
  • Ensure that automated test scripts are easy to write and read.

The Flutter Driver also provides a method to support BDD: flutter_gherkin. However, FLUTTER support for BDD is not as complete as it should be. Instead, flutter has a lot of restrictions and rules:

  1. Mapping is not supported: i.etestCase.featureThe natural language cannot be mapped directly tostep.
  2. The class file for each step in each feature needs to be split: if there are ten different steps, ten different Dart files need to be created.
  3. Define all the steps that need to be performed in a Config file. It is the steps in the config that actually perform the test steps.

If a project has a large number of automated test cases, using Flutter_GHerkin is difficult to maintain and write, so it is not recommended to practice BDD for FLUTTER testing.

  1. Visual Testing tool: Micoo.

Visual testing is an increasingly common strategy for E2E Tests, and Micoo is a pixel-based screen snapshot comparison tool for Visual Regression tests. Micoo is a Web application that sets up base images, You can compare the latest screenshots and output test results. The benefits of Flutter Driver in combination with Micoo include:

  1. Micoo makes it easy to compare screenshots from different versions and flexibly set the baseline.
  2. Micoo’s configuration and image upload process are straightforward.
  3. Functional testing and visual testing can be implemented at the same time, such as:
    1. Implement functional testing through code assertions.
    2. Visual tests are performed by comparing images.

4. Practice of Flutter Driver Project:

Test environment:

In order to test the real E2E, the environment chosen for the test was: QA environment (API automation testing is also available on the project).

Test type:

In order to achieve both functional and visual testing, two automated test accounts were adopted:

  1. Account 1 will not do any operation to cause changes in user data, so as to ensure the consistency of each screenshot to the maximum extent. Then, the screenshots generated by account 1 will be uploaded to Micoo for automatic picture comparison.
  2. Account 2 performs various actions to determine test results through code assertions.

Testing machine

To run automated tests on a pipeline, simulators are used to run the program.

CI/CD run policy

** Tools: **Codemagic, consistent with front-end support deployment tools.

Automated test process:

  1. Since only the Debug package can run automated tests, a separate package must be built to deploy the debug version of the pipeline.
  2. Start the VM and run the E2E test.
  3. The Pipeline outputs the results of functional tests and screenshots of visual tests and automatically uploads the screenshots to the Micoo server.
  4. Micoo automatically compares images and outputs the results.

5. Conclusion:

Generally speaking, the Flutter Driver and the FLUTTER project support each other very well. The operation of the Flutter driver is relatively fast after getting used to the writing method. However, since the Current flutter driver is not very mature and has limited functions, it is sufficient for most scenarios of the project. The combination of functional testing and visual testing is a good practice to not only realistically test the front and back ends, but also to capture details and changes that are easily overlooked. If you have any suggestions or comments, please feel free to contact us.