http://sauceio.com/index.php/2015/09/can-you-test-it-all-test-coverage-vs-resources/
On every project I’ve worked on recently, I’ve had this question: Can all aspects of the product be tested? Of course the answer is no. There’s either no time to test, or no one to test. So the question is: how do we ensure that we deliver high-quality products without compromising testing? Maybe we should have done it more accurately.
[Common Causes] In my years of experience, the rush to finish testing and launch a product is usually caused by one of the following reasons:
1. User stories are too big. When a story is too large, it becomes difficult to decompose tasks and determine acceptance criteria for all features. Not only is it difficult to plan for unforeseen circumstances, but it is also difficult to coordinate problems encountered by the project.
2. Product workflow is too complex. The workflow of the product may be very complicated due to its features, and it is difficult to determine whether it is the product that users actually need. At the same time, due to complex workflows, testers will face more challenges to tease out each test scenario and design end-to-end test cases for it. Even with more small stories, the overall workflow should still include all the stories, and if the workflow is too complex, it can also lead to missed testing.
Don’t use test-driven development. The abandonment of rules and QA for the sake of temporary convenience is a huge challenge for the future of the project, and problems can delay or even block the progress of testing.
4. Release deadline. Are you involved in a project where the project members clearly understand the overall plan? Is the delivery date clear? If the development schedule is behind, but still insist on delivery, how to resolve the conflict?
5. Too much demand. The project had so many requirements to fulfill that it was mind-boggling to see all of them. We need to consider multiple versions of the product, different browsers (or browser versions), multiple mobile devices, operating systems, etc. These can be challenging for anyone. If all of the requirements mentioned above were to be met and 100% test coverage was to be achieved, could this really be done?
What to do? The above points are not an objection to QA achieving adequate test coverage. But in reality, does testing really need to cover all the bases? We need to be more precise in our testing.
First, make the story smaller! If the story is small enough, it is easier to identify acceptance criteria and ensure coverage (at least for those isolated features), while developing a test strategy based on the classic test triangle (unit test, integration test, and UI test).
Capture the main workflow! Everyone’s usage habits are different, and we can’t predict how users will interact with the system, but we can know what most users will do, and we can communicate with designers or users to learn more about it. By combing through the common operational flows, we can drill down into these workflows to identify the parts of the workflow that really need to be covered by testing, and prioritize the automated testing of these parts of the workflow. Other less involved user scenarios can be explored.
The 80/20 rule. Which module is the riskiest? Ask yourself, how can we find as many bugs as possible with as little testing as possible? In layman’s terms, how do you find 80% of bugs with 20% of your tests? At this point, if we have accumulated enough historical data and found few problems with certain modules, do we need to invest a lot of testing resources? Should we focus our testing resources on modules where problems are frequently found?
Big data. I must admit, I resisted hearing the buzzword “big data” at first, but it turned out to work. We can learn about the client, browser version, and click flow from this analysis, which can help us develop a testing strategy. Then we can spend our time more wisely, focusing on testing progress and staffing.
Finally, I would like to say that quality assurance is the responsibility of the entire project team. It is true that we cannot achieve complete testing coverage, but we can make the whole testing process more precise through the process of testing strategy, test aggregation, and test execution. It is important to note that the level of test coverage is a decision of the entire project team, not just the testers. We can drive communication and make recommendations, but the project team ultimately decides when we finish testing and what points we should test.
Copyright belongs to the author. Commercial reprint please contact the author for authorization, non-commercial reprint please indicate the source.