How do small and micro teams play App automated testing

preface

In this blog post, I will stand on the small micro team’s point of view to introduce some of my views on App automation testing. Convince you to start practicing App automated testing while helping you lower your expectations.

App automated testing has always been a field that small and micro teams rarely set foot in. In the big scene of rapid iteration of the Internet, with the development of business, the pressure of regression gradually increases. I’m sure I’m not the only one who hits the table with frustration every time a lack of regression coverage causes an online accident.

In general, the tests of small and micro teams, including regression tests, are carried out manually. Some key businesses deviating from the main process are often easily missed by manual regression tests. When you run out of people, the idea of automated testing pops into your head, and you do research. But that may be the end of the story. Not to talk about the automation framework, all kinds of subdivision of the boundary case, an inevitable very tedious thing to think about is even more tedious. If you are App development, the original business development is not very sufficient manpower, and then to develop automated scripts, have the heart to powerless! If you’re testing, business requirements testing for each iteration fills up your schedule, not to mention taking on more than App testing. As a small micro team, we would like to automate, but we don’t have the resources…

First of all, it’s important to understand that App automated testing isn’t as powerful as you might expect, but if you’re facing regression testing pain points like I am, it can definitely meet your needs. It’s not as powerful, so it’s not as complex as you might expect, and its participants should never be limited to Tester or Developer, so you can have more flexibility with your resources. Of course, automated testing is a long-term process, and its future is not just a regression…

Finally, I’ll give you a cool automated testing tool called Calabash. I also have the Calabash tutorial blog and a little bit of my experience with it. Calabash is introduced because Calabash is more suitable for small and micro teams with limited resources in my opinion.

App automated testing

Only represents the personal view, knowledge is shallow, welcome many dozen face.

The status quo

Let’s start with what we all know. Take Android as an example, since 2010, Android development environment and its rapid development trend to today, nearly mature, developers have long been not limited to this single development platform, began to seek the unification of Android and iOS development: ReactNactive,WEEX,H5…

The development environment is mature, but the mobile client testing environment is lagging behind. There are several mountains in between that many teams are facing and stopping:

1. Unlike server-side testing, App testing usually only needs to verify the data itself.App involves interface display and interaction, so it is difficult to automate identification. 2. Internet enterprises have always been pursuing fast iteration, and App directly connects with users, so the interface and logic changes of App are more common. The stability of writing automatic test scripts is very poor, and all the scripts linked to the page may be discarded once the interface is changed. 3.Android and iOS dual platform,web page. It’s almost impossible to automate testing on a single set of code on multiple platforms. And think about the need for constant change, you know. 4. Automated testing may require more scripting for various boundary cases than manual testing. Compared with manual test, it may only need to design a few scenarios in advance, and other exception boundary cases can be observed manually in the test. Automated testing costs double!

Therefore, a complete mobile automation test environment requires a large test team, but only a large company can afford a large test team. Mobile automated testing has been neglected by small and medium-sized startups. Small companies develop their own tests, and medium-sized companies rely on testers to verify them manually.

Is automated testing really closed to small teams?

Just do it

All of this is a conundrum, and perhaps it’s too early to talk about solving it. I think there’s something you’re not sure about, so why don’t you just follow my lead? And so on some of the following things to understand, maybe these problems can avoid also avoided, need hard, also have enough psychological preparation.

1. Set a trial and error schedule

In fact, one of the biggest problems for small teams facing automated testing is the input-output ratio! It is difficult to predict whether the test or developer will get bogged down in the process of implementing automated tests between frequent page changes and script development and maintenance.

But talking the talk is never going to work, and the lessons for other big companies aren’t very good either, because it involves teams, resource allocation, how often business changes, testing tools, how decoupled script development is, and so on. But automated testing is an inevitable trend, so action comes first!

You’ll never know if your team is a good fit until you try. Start with a few stages and use the first stage of trial and error:

Phase 1: Allocate the resources of one iteration per person to complete the automated test case of the main process business, run two or three iterations, and add the main process exception case during this period. Use these three iterations to evaluate the viability of future development! Stage 2- (Successfully passed Stage 1) : If you think the status of Stage 1 is good, then on the basis of maintaining the results of Stage 1, select a new business from the remaining business scenarios according to the degree of business criticality and change frequency, and write automated test cases at the pace of one person iterating one business! Stage 2- (Excessive failure of Stage 1) : Of course, after three iterations of evaluation, as the number of abnormal cases increases and synchronous maintenance becomes more difficult, you may think that the cost of implementing automated tests is too high, but don’t give up the results of these three iterations too easily. Please first use programming thinking to check all test scripts, whether there is the possibility of extracting similar codes, encapsulating specific View operations, and extracting instructions irrelevant to business. At the same time, consider using the whole group resources to maintain this main process automation case. Proceed to phase 3 or phase 1 again until future resources are sufficient or better alternatives are found. Phase 3: After 5 to 8 iterations, you have successfully made it through Phase 2, indicating that your automated test environment is on the right track. At this point, you can accelerate your automated case coverage depending on your team’s resources!

Trial and error with the resources of one iteration per person is a loss you can live with. Of course, don’t do it just yet. You may just decide to automate your tests, but make sure you have some planning and some thought and action!

2. Set clear and simple goals

With a big plan, you also need to identify specific requirements. The options are as follows:

1. Black box testing or white box testing although UI automation testing, but can also be divided into black box or white box, this depends on the test precision you want, that is, at this time, you can initially determine what automated testing tools to use. For example, we usually choose Appium, a cross-platform testing tool, in order to enable dual platforms. However, if you have high testing requirements, take Android as an example, we suggest you use Espresso, the official testing framework recommended by Android, and write Test directly in the Android project. This sounds obvious to everyone, but if you can’t identify your needs early on, it can cause a lot of trouble later on! Appium is selected because it is a black box and some special scenarios or requirements are encountered, leading to certain cases that cannot be tested. Or choose Espresso, but later found that there was not such a sophisticated testing requirement, so it could not be cross-platform later, or it was considered that Espresso was too expensive to write and could not be handed over to the testing team.

2. Front-end UI or overall business take the scene of order grabbing and delivery as an example. Find an order, click the grab order, pick up the food, and finally complete the delivery, which is a set of UI layer operations that can only be observed with the naked eye. We tentatively define this as the front-end UI logic. However, other data changes caused by order status flow behind this series of operations, such as wallet data changes, activity reward data changes, fraud order judgment and so on, belong to the overall business category. What we need to determine now is whether the testing expectation you want to achieve is UI testing or overall business testing. This directly determines the complexity of your test scripts. My advice is to test only the UI logic, which is one point I want you to lower your expectations.

Decide on a clear and simple goal, and then dive into it. If you think too much, it will be more difficult, and in the end it may just be thinking, which is the next point: Lower your expectations!

3. Lower your expectations

Under the environment of manual (point and point on the mobile phone) test, we usually carry out various case verification through the operation of App. As long as the operation of App verification passes, we can basically determine that there is no problem in the front and back end. But the premise is that manual verification is the human brain plus the naked eye, it will have more accurate subjective judgment. The interaction between the front and back ends will increase great complexity. Your test case will not be infinite in number and precision, but as a human brain, you can find more unreasonable bugs in the limited case execution: unreasonable text folding, inaccurate numerical display, wrong button color, incorrect Toast display number and so on. Automated tests will follow your script rigidly. Can you make sure your case covers everything?

This is also when it comes to automated testing, a lot of people will throw out a question, so many abnormal case how to write ah, think about tired, consider the input-output ratio or manual test.

First of all, it should be clear that the front and back end automated tests must be separated, because a set of problems cannot be solved, so many cases can be ignored in the front-end test. In addition, automated testing is a long-term process as well as the growth of the project. It still takes some time for automation to completely replace manual labor, so you don’t want to achieve the goal in one step. There are still a lot of cases to write, right? Resources we can run through main process first, run through the main means that scripts rely on (environment set up, view positioning) has been relatively mature, other abnormal case script to the main process changes, we slowly to, and this time you can open to other development or testing them with difficulty. This is what we called stage 1.

If you can’t figure out these questions, you may find that you can’t achieve the results you want under the limited conditions of App automated testing, and you may feel pressured to give up. Convince yourself to lower your expectations!

Lower expectations and aim for regression first
To lower expectations, both front and back end automated tests need to have a set of problems that won’t solve
To lower expectations, there are still many limitations to client automation testing, and manual testing validation cannot be completely abandoned
Low expectations, not a fat man, automated testing also needs to be slowly iterative improvement, first run through and then care about verifying abnormal cases. Iterate quickly and improve step by step

4. Specialize

Maybe someone on your team in development or testing just couldn’t take the pressure and started exploring App testing automation. The advantages of developing automated tests are that I have richer development experience and more skilled in setting up the test environment. Besides, because I wrote the code by myself, I can better understand where the risk points are and write targeted cases. The advantage of automated testing is to discover more boundary scenarios and write more comprehensive test cases.

Development and testing have different strengths for automated testing, but they are also weaknesses for each other.

The test case must be as comprehensive as possible, and the automated test itself is an engineering project, when writing scripts, if more consideration of some programming ideas, reasonable coupling and decoupling, will make the subsequent script writing more convenient. Therefore, it is highly unlikely that automated testing should be left entirely to testers or developers, preferably in a tightly collaborative relationship. It’s also a solution to a lack of resources. Although it is testing, the addition of development will make the test project develop in a better direction.

5. It takes two to tango

First, to restate my point: mobile client automation should focus on regression testing without sufficient resources.

Automated testing is very rigid compared to manual testing, so it needs very “rigid” data support. If it’s an App developer (like me) doing automated testing, do I mock all the data? Although the front and back end automation should be separate, it was stated earlier that you should only care about the front end logic. But always test in a real service environment (test environment, not production environment). If mock data is too rigid, many cases may not be verified. For example, in registration scenarios, it is difficult to verify the exception prompt when a registered account cannot be registered again. Mock data is obviously difficult to combine normal registration and abnormal registration. And it would be nice to test it in the real world in case a back-end bug is detected. What’s more, there seems to be no big problem with the direct testing of a completely 2C application like Mobile Taobao in the production environment, but for a semi-2C application like Hummingbird Delivery, it is quite awkward. It is obviously impossible to directly take online orders. Then who will issue and how to issue this order in the test environment?

So at this time, you can not do it alone, go to the test classmate, see if you can build a test environment with automatic testing, take snatches for example, I hope there will always be a lot of orders in this environment for me to grab. I’m the app that grabs orders. I don’t play without orders.

If the test students can not handle, then go to the back end of the students, I believe that in order to improve the project, they will not refuse you.

Automated testing, not only automated testing tools are enough, but also need the support of the testing environment.

summary

All this talk is about lowering your expectations and taking action. There are a lot of automation tools available to help you with your App’s automation recognition and multi-platform, and learning to lower expectations will also reduce the difficulty of script development and maintenance.

Are you still struggling with too many subdivided cases? What if we leave it for a second? Anyway, you’ve been naked for so long, so you need to wear pants first, pants slowly! Or are you still stuck with the input-output ratio? Don’t you also want to try on clothes before you know the code number?

OK, let’s talk about these shorts and how to wear them!

Testing tools

Do a good job of ideological work, specific implementation or need to choose a specific test tool, tool is not good or bad, only suitable. Calabash is a testing tool THAT I personally recommend to small teams because of its low barrier to entry, cross-platform, and easy script maintenance features, which small teams are most in need of. And while its BDD (Behavior Driven Development) philosophy is a bit utopian, from a long-term perspective, the possibility of implementing advanced project management where test scripts are written before application development is greater than any other tool, making testing more than just testing!

Below I will give a brief introduction to Calabash’s features, detailing its advantages and disadvantages. You can choose whether to use Calabash based on your team’s experience. If you like, I will attach a beginner’s tutorial for Calabash, plus more detailed features and my own suggestions for using it.

Calabash

Before we talk about Calabash, let’s be clear about what Calabash is.

At the heart of Calabash is Cucumber, a collaboration tool that can write instances in natural language. Its core idea is behavior Driven development (BDD), and regression testing is a natural by-product of that. Calabash-android or Calabash-ios is an extension to Cucumber to automate testing for Android or iOS.

To explain BDD, I took an intro from the web: You can write user scenarios in Cucumber that represent business rules rather than just UI functions. This allows you to involve business analysts in the process and write scenarios before the coding begins. Programmers are able to code according to this clear specification. This approach is called behavior-driven development (BDD).

OK, let’s stop here with BDD, it’s still in the future, let’s talk about something real.

Calabash is similar to Appium and Espresso. They operate the interface through scripts, such as clicking and sliding, and identify UI components on the interface. That’s how run works. Analog controls the phone. Let’s take a look at Calabash’s script, trust me, love it, and start with the script:

Feature: Login Feature Scenario: Login test case When I press view with ID"account_edit"
  	Then I enter "15104053650 traffic" into input field number 1

  	Then I press "Send verification code"
  	Then I wait for 5 seconds

  	When I press view with id "verifying_code_edit"
  	Then I enter "306423" into input field number 2

  	Then I wait for 5 seconds

  	Then I press view with id "login_login"

Copy the code

I don’t think I need to explain the meaning of this script anymore, it’s plain English!! Calabash’s greatest appeal is that it transforms an unintelligible programming language into a natural language that anyone can read and understand. Thus reducing the learning threshold and maintenance difficulty. Feature 1: High readability. The more readable the test script, the more meaningful it is. When written prior to or in sync with development, test scripts can also serve as design documents and reference manuals during development.

While it is very readable, it is also very easy to write. Feature 2: The grammar is easy to understand, anyone can write.

Thankfully, the code supports both Android and iOS, and there is some web support as well! Not to mention RN, running RN is native code! WEEX needs to be examined again. In the case of proper decoupling and coupling, Android and iOS only need to provide a set of View location packages to share a set of scenario case scripts. Feature 3: Support a variety of platforms, strong code reuse.

A script written in a natural language, of course, takes a lot of work at the bottom. Above we saw a test case written using Calabash’s predefined Steps.

The essence of these Steps is as follows:

Then /^I press "([^ \] *)" "$/ do |identifier|  # -- Method name
  tap_when_element_exists("* marked:'#{identifier}'")  # -- Method body
end
Copy the code

A silly explanation: you can easily think of the above paragraph as a function, a method. The first line is a natural language method name mixed with regular notation. The second line is the body of the method implemented in Ruby. Don’t be intimidated by this code, for the most part, you don’t need to touch the concrete implementation of these plain English words (Ruby), and just write small compositions in plain English. Predefined Steps are definitely not enough for our needs, so Calabash flexibly supports different levels of custom Steps, something like the one above. Calabash supports different levels of code-to-natural language conversion. Meet business requirements to the greatest extent. It’s this flexible transformation that allows us to imagine more:

Feature 4: The development of Steps is decoupled from the development of scripts, reducing the threshold for entry and allowing more script developers. Let a part of the first (the actual situation may be one) learn to write Steps in plain English. Let in another group of people, like other developers or testers, or even product managers! Write test cases in plain English, and then customize Steps if you have an idea. Even products don’t need to learn to customize Steps, which is a requirement for development. When you have a very rich Steps library, the endless boundary cases you are annoyed with are nothing more than a national topic

Steps gives you complete freedom to define method names, which improves readability while reducing memory and scripting speed. Depending on the specification you define, you can customize a set of syntactic sugar that most people are comfortable with, thus solidifying most of the instruction formats while maintaining readability and improving the memory and scripting speed of Steps.

And oh, if you’re constantly complaining about how dehumanizing a grammar is in a language you’re currently using, it sucks! Now, are you ready to develop your own language?

Feature 6: simple maintenance, I do not know the small composition maintenance up what difficult ~

Feature 7: Case also has a sense of design, which is not so boring, but at least I feel very excited about the feature. Considering reuse, the custom Steps should consider the decoupling problem with the business. Some need to be decoupled, but some need to be coupled. In addition, there will be more communication between Android and iOS development and more common design of test script writing in order to minimize code changes to ensure that Android iOS code is universal. There is a pleasure in design.

Feature 8: In addition to simulating events, Calabash also supports various hooks, such as App lifecycle hooks

Feature 9: Rich element positioning aid methods. A common pain point in App automation testing is element location. Calabash has a lot of element location assistance methods, all of which are relatively simple and brainless.

Finally, we’re looking at a Calabash script from the Internet:

Feature: login &emsp; &emsp; Scenario: Enter the correct user name and password to log in normally & EMsp; &emsp; When opens the login page &emsp; &emsp; And enter username XXX And password XXX &emsp; &emsp; And click login &emsp; &emsp; Then verify that the login succeededCopy the code

Is there any scripting that is completely pure Chinese? Although the blogger himself does not recommend such a pure Chinese definition of step, we will take a look, just know that Calabash is strong.

Finally, we summarize the advantages of Calabash’s general direction:

High readability
The grammar is easy to understand and the learning threshold is low
One language support multiple platforms, code reuse is strong
Steps development is decoupled from script development, lowering the barrier to entry and allowing for more development
Custom syntax sugar
Small composition maintenance simple
Case also has design sense
In addition to simulating events, Calabash also supports various hooks, such as hooks for the App life cycle
A lot of element positioning AIDS
Non-invasive code, pure black box testing.
It’s possible to build your own cloud platform with Calabash

These are just a few of the advantages of the general direction, and the introduction to Calabash will be provided at the end of this article, with a more detailed description of Calabash’s various features, which you will find to be too imaginative.

Nothing shortcomings

Said so many advantages, disadvantages of course also have ~ we ridicule

Data and Demo are scarce

Calabash has a history of more than five years, and the earliest first commit was in February 2012, which is quite old for a technical tool, but Calabash is still a niche product, which leads to the problem that there is very little data… Github has a very basic API and almost no Demo.

Less information also means that there are many pits to go through, but with the introduction of the end of the article, it should be no problem to get started quickly

I dare to doubt the reason for Calabash’s small market. Maybe big factories prefer the mainstream testing framework appium, after all, complex syntax means powerful functions, while small factories prefer manual testing to maintaining a set of automatic testing framework, so Calabash, You’re in a very awkward position. (This article is for you to crack the embarrassment!)

Cross-process is not supported

The bottom layer of Calabash is Robotium, and Robotium doesn’t support cross-process operations, so Calabash doesn’t support either. When you jump out of the current test App, Calabash can’t respond and do anything. In some apps that require a lot of cross-process calls, this point can be painful, and you can almost give up Calabash.

Taking hummingbird crowdsourcing as an example, cross-process services include sharing and taking pictures (taking pictures invokes the system camera). Sharing is a very boundary business for hummingbird crowdsourcing, which will not be discussed for the moment, but taking pictures affects the main process. However, if you modify the camera as a custom camera, but do not call the system camera, this problem will not exist, and can solve the problem of different phone system camera style, the script needs to be compatible. So even this level of change to the business code for testing is worth it!

In addition, through recent research, **Calabash support for cross-processes also seems promising! **Espresso implements cross-process operations by calling the UIAutomator API. As Calabash is an open source project, it is perfectly OK to modify its source code. Take Android for example, its internal core is implemented by Java calling The Robotium API, so we just need to modify its source code to call the UIAutomator API to implement cross-process. Well… The realization of the idea is like this, but I do not know why the official did not do so, the specific situation remains to be studied… (Before the end of 2015, it was easy to find the blogs related to Calabash to modify its source code for function extension. However, Calabash pulled the source code of Calabash out to another project at the end of 2015, so the solution mentioned in these blogs could not be used, and the specific implementation method of the follow-up is still to be studied. More on this in the tutorial at the end of this article)

Lax natural language

Most people are probably skeptical of natural languages like Calabash, and think Cucumber style languages like Calabash (see the test script above) aren’t rigorous enough. It can’t be as concise and strict as the code, and to some extent it also reduces the writing speed, and it is not strict enough and error-prone!

The first thing to clarify is that while subordinated programming languages like Java and Python, Cucumber style scripting tends to be a descriptive language. They are two different fields.

Therefore, the rigor of Cucumber style scripts should not be required from the perspective of programming language, but “rigor” is just a kind of habitual thinking of programmers.

In terms of memory and writing speed, remember the grammar candy we talked about above? An inappropriate definition of Steps can indeed lead to such problems. Let me give you an example

# Then I skip welcome page # Then I have finished the loginCopy the code

Above are two custom operations that abuse natural language features. If all of your test scripts are written this way, it will read fine, but write without documentation. Let’s look at the more recommended way:

# I pass "welcome" scenario # I pass "login" scenarioCopy the code

The actual meaning of the two Steps is just to go through the action directly. If the definition of the Steps is appropriate to our syntax, the very messy script will become very regular, ensuring readability while you only need to remember a few specific instructions, thus speeding up script writing.

Of course, it is Calabash’s overly loose grammar that causes this fatal flaw to be restricted by strict management, and the slightest mistake could be a grave cause. So that’s really one of Calabash’s downsides.

End

Finally, although I personally recommend using Calabash, for small micro teams, what is appropriate is best! If you want to implement automated testing under tight resources, my suggestion is to allocate more manpower to share the pressure on the basis of improving human efficiency. Then the requirements for low learning threshold and low promotion cost will be very high, so the learning threshold and promotion cost should be the most important point you need to consider when choosing automation tools.

It’s because Calabash fits these two things that I highly recommend it, and it’s also highly recommended if you want to try out the idea of BDD in the future, or if you want to get a product manager involved. Of course, if you have more resources or are looking for more engineered automated testing, you can also choose mainstream testing tools like Appium. Or the same words: the right is the best!

And finally, here’s Calabash’s primer blog:

Calabash Quest 1-Run Calabash

Calabash Exploration 2-Calabash Usage in Detail

Calabash Quest 3-Calabash Progression

Calabash Quest 4-Calabash Trample Summary