Welcome to Tencent cloud community, get more Tencent mass technology practice dry goods oh ~

New team: Tencent Mobile Quality Center TMQ

An overview,

This paper records the whole process of TBS Studio development debugging tool testing in a realistic style. Including test manpower application, test strategy formulation, system testing and mass testing experience. For testing beginners can understand how the whole process is step by step down. Students with certain experience can appreciate the testing concept based on risk and cost in the process of testing strategy formulation.

Ii. Introduction of test project

TBS Studio is an overall solution for tBs-based Web developers and mobile application developers (including wechat, Mobile Q, third-party apps, etc.) to improve the development efficiency of the majority of developers in the real environment, and help developers analyze and optimize the design of Web pages. The main functions include Web Inspector debugging, Webpage performance analysis, etc.

Details: https://x5.tencent.com/tbs/guide/debug/season1.html.

Iii. Application for testing manpower

On May 23rd, Brian, a development student, came to me and told me that he wanted to apply for testing resources for a TBS Studio product. Through telephone communication, I learned that this belongs to Tencent browsing service (TBS) subsidiary products, to provide development to do web debugging. So I went to the leader of our test group and explained the situation. The Leader says Bonnie and Mekhi are familiar with web debugging and suggests that I bring them along to discuss testing requirements. It has been proven that it is a good idea to communicate testing requirements with several related colleagues.

The next day, Bonnie, Mekhi and I went to find developer Brian to talk to him about his needs. After half an hour of explanation, we have a clear understanding of the testing requirements. It was also made clear that the main job was to follow up on projects (which I was better at), rather than testing through technical means (which Bonnie and Mekhi were better at). The following is a transcript of the communication results :(from which you can see what the test requirements communication generally needs to know).

Background:

Develop debugging tools. It is mainly used to enhance the influence of TBS. It was a small launch before, but now we want to ensure quality through full testing. At present, daily activity xx (specific data is not convenient to disclose, the same below), the target of the first half of the year is daily activity XXX.

TBS Studio features overview and Test highlights:

It consists of two parts: ADB detection and Inspector module. The Inspector module is primarily guaranteed by development self-testing. Tests are responsible for ensuring that adb detects modules. The ADB detection module has four steps. Respectively is:

Step1: Please connect the mobile phone to allow USB debugging;

Step2: Confirm the App to be debugged and check whether the current App is connected to the X5 kernel;

Step3: Test whether TBS debugging is supported;

Step4: Set the TBS debugging state.

The Inspector module only tests the element change functionality this time.

TBS Studio Release Rhythm:

A small release every 3 weeks and a large release every 6 weeks (to keep pace with TBS kernel releases).

Minor releases follow the following process:

(1) Develop and use moCHR method for self-testing;

(2) Test verification modification point;

(3) Develop internal test;

(4) Pre-launch test.

Major releases follow the following process:

(1) Develop self-test (mainly to ensure compatibility between Inspector module and new VERSION TBS kernel);

(2) Core process use cases (smaller than pre-live use cases. Mainly to ensure that ADB test is normal).

The integration is expected to be tested next week, the release plan is not yet clear.

TBS Studio Roles:

Products: Brian

Front-end development: April

Terminal development: Josh

Test: eason

Test points:

(1) Function points: overwrite different branches of adb detection module step1-Step4 operations;

(2) Platform adaptation: Windows/MAC (mainly adb detection step 1);

(3) Host adaptation: own kernel/shared kernel /QB (mainly adapted to adb detection step 3);

(4) Model adaptation: all Android phones (mainly for ADB testing step 1 and Step 3)

(5) TBS version adaptation: all online versions (main inspector module adaptation);

(6) inspector module: the only need to test function change element (https://x5.tencent.com/tbs/guide/debug/season2.html).

Test method consideration:

(1) Mainly manual testing;

(2) Preliminary analysis is not suitable for automation, please consult Application Bao for details;

(3) We can consider mass testing to find some problems that we can’t consider;

(4) Due to the small number of current users, it is considered to evaluate product quality with minimum investment.

Follow-up Plan:

(1) Eason first evaluates the workload and whether to adopt automatic testing;

(2) Eason confirms outsourcing manpower;

(3) Eason writes test cases;

(4) Outsourcing test execution.

Four, test strategy formulation

After communicating the requirements with the developer, I had a basic understanding of the project. Next you need to assess the workload and develop a testing strategy.

Estimated working hours:

(1) Test strategy formulation (test method, test model, coverage, etc.) 2h full-time job;

(2) Test case writing (integration use case — currently there are 16 test points, pre-launch use cases, and core process use cases) 6h full-time job;

(3) Test environment preparation (Win8, Win10, MAC computer, mobile phone coordination) 4 hours;

(4) Integrated use case execution (16 single-machine full use cases + 6 platform adaptation cases + 4 host adaptation cases + 60 model adaptation cases + 6 other cases =92) outsourcing for 2-3 days;

(5) Use cases (10) were outsourced for 2 hours before going online;

(6) Core use cases (3) are outsourced for 1 hour.

This integration test requires 12h full-time labor and 3-4 days outsourced labor.

Test strategy formulation:

Test strategy formulation is mainly to answer “what to worry about”, “how much to measure”, “how to measure” these three questions, which will be combined with the risk and cost consideration.

What to worry about (risk) :

This test is to understand and improve the quality of the version, to prepare for the increase of product promotion. So the current concern is that there will be various problems when the version is sent to users, which will affect the reputation of the product. To address this concern, we need to understand user usage scenarios, and we can test to cover those scenarios. User scenarios can be obtained in two ways. One is to look at statistics, and the other is to find users (mass testing and experience).

Overall, the risk of this project is low. TBS Studio currently has only 89 daily users. If through system testing and mass testing, there are still individual problems leaked to real users is not a big problem. Collect questions through user feedback channels.

How much to measure (cost) :

As mentioned above, TBS Studio is less risky. At the same time, TBS Studio has no firm release plans, so there is no urgency. Therefore, system testing only needs to use primary outsourcing to cover the most important scenarios and ensure that more than 80% of users can use it smoothly. Scenarios that are not covered can be found by crowdtesting and experimentation, two cheaper methods. With these two passes, the quality is basically guaranteed.

How to measure:

Here is mainly automated testing, artificial system testing, mass testing this centralized way of choice. After talking to PC App Treasure, they also don’t use automated tests for features like ADB connectivity. So we gave up on automated testing. System testing is definitely a must. Crowdtesting and in-company experiences can complement points not covered by system testing, so they can be used.

Test strategy details:

Platform adaptation:

Through online information, we can see that the total market share of Win7, Win10, MAC10 of these three systems reached 78%. Therefore, these three systems are selected as platform adaptation systems.

https://www.idcps.com/news/20170420/94526.html.

The Windows system is divided into 32 bits and 64 bits. We’ve found that 64-bit systems have become mainstream. So we mainly used 64-bit systems in our tests.

http://digi.163.com/16/1103/08/C4UDM2QS001687H3.html

Screen resolution adaptation (for all interactive interfaces) :

Since TBS Studio is a desktop application, consider the resolution of the monitor. Ensure that the interface displays properly on different screens. Through online data, we can see that 1366768 (the mainstream resolution of laptop screen) and 19201080 (the mainstream resolution of desktop screen) account for more than 50% of the screen, other resolutions account for a lower proportion. So the screen adaptation chose 1366768 and 19201080 two screens.

The host adapter

Through the background statistics of TBS (Tencent Browsing service), we can know the number of users of different hosts (apps using Tencent browsing service). By selecting each type of top host, we can tell how many user scenarios the test case can cover.

The final test hosts are as follows:

Own kernel host: wechat/HandQ (covering 95% of user scenarios);

Shared kernel host: QQ Music/Tencent News/JINGdong/App Treasure/King of Glory/Vipshop (covering 55% of user scenes);

QB: Mobile QQ browser (covering 100% user scenarios).

Model adaptation:

For model coverage, we will consider top coverage, mainstream system version coverage and mainstream phone manufacturers coverage when selecting models. Because adb connections are vendor dependent. Therefore, we choose the test model through the manufacturer ranking first. Through TBS background monitoring data, it can be learned that Huawei, Vivo, OPpo, Samsung and Xiaomi are the top 5 manufacturers. Users account for 80% of the total number of TBS users.

Combined with the existing models in the test group, the model coverage list we selected is as follows:

oppo A33

vivo x7plus

Huawei p8

Millet 4

Samsung S6.

Test Plan:

(1) Use cases before running on-line (smoke test);

(2) All use cases of single mobile phone;

(3) one mobile phone for each of the top5 brands (one mobile phone is tested according to the top5 brands to decide whether to add model adaptation);

(4) Host adaptation;

(5) Run through the platform adaptation case (plus screen adaptation test);

(6) Release mass test;

(7) Release the rewarding experience for developers within the company.

5. System test

After the test strategy and plan are specified, start writing test cases.

1. Test case writing

First, to ensure that use cases cover every logical branch.

I’ll start with a mind map to tease out each logic:

However, since it is not convenient for the primary outsourcing to execute the mind map directly (many of our intermediate and above outsourcing directly watch the mind map to execute the use cases, or even write their own mind map use cases), they still need to convert the mind map into use cases. Considering that this use case is used very infrequently except for the pre-live use case, which is used once a month. So only explain what needs to be explained. Leave empty where there is no need to explain:

The most important information in the table is the function point and test point columns. The rest is only supplementary information where special reminders are required. This use case may seem messy, but it is fast and useful.

After writing the standalone use case, I added various adaptation use cases:

These use cases are based on stand-alone use cases. Modify some test conditions as needed. Like a different platform, or a different host.

Here’s a lesson to share: Select use cases based on the scope of the test condition, rather than testing all use cases when any condition changes.

For example, covering different platforms. We have run all the use cases on win7 computer in standalone test. Here needs to adapt win10 and MAC system, is it also necessary to run all use cases? The answer is no. Because when communicating with development, we have learned in advance that platform adaptation only affects Step1, and the logic of other steps has nothing to do with platform. So only four use cases related to Step1 have been used here. The number of use cases was reduced by 80%.

Once a use case is written, one of the most overlooked aspects is the use case review. Many people consider use-case reviews to be optional, or just online. However, based on my personal experience, I can responsibly tell you that offline use case reviews are valuable for small projects! Because it can not only find problems with use cases, but also find problems with requirements and code implementation through discussion, it is very cost-effective!

2. Stand-alone test

The process of executing the use case is familiar to all of us, so here we paste the test report directly.

TBS Studio tool single use case test results:

Test version: TBS Studio: V1.3.1

Platform: windows7 professional 64-bit

Mobile phone model: OPPO A33 (5.1.1 root)

Task links: omitted

Test results: a total of 22 use cases. Nineteen passed, two failed, and one failed to be implemented.

Use case list fails and cannot be executed:

The bug list:

3. Model adaptation

After passing a single use case, various adaptation tests will be arranged successively. I won’t expand it here, but share a little lesson:

“Comprehensive” is more important than “in-depth”!

We found two typical questions in our tests:

Win10 32-bit PC program flash back;

7.0 Enter the Inspector’s flash screen.

Both of these problems are obvious and can be picked up in a test. If we continue to test Windows 10 64-bit systems or 4.x-6.X phones, it will be hard to get the same amount of human input. That’s because the more we test the same conditions, the smaller the marginal benefit.

So generally speaking, “comprehensive” is more important than “in-depth”.

4. Host adaptation

A little.

5. Adapt to the platform

A little.

6. Bug regression and pre-launch testing

A little.

Six, mass testing and experience

After the pre-launch test passed, we started mass testing.

1. Release mass tests

The company has 2 public testing platform: penguin public testing and QQ public testing. Penguin mass testing tends to be divergent testing, while QQ mass testing tends to be use case testing. The purpose of the mass test was to find points that the system test could not cover, so we chose the penguin mass test.

First of all, I communicated with Vincent, the business interface person of Penguin Test, whether they could undertake the test of PC client like ours, and Vincent said yes.

Then I wrote the test requirements for Vincent. Here are two lessons to learn about the design of test requirements:

(1) The test requirements should be simple and clear enough. Because users are amateurs, it’s too complicated for them to understand. With TBS Studio, for example, we started to want users to cover a lot of third-party apps. But it’s hard to explain to users how to get to a web page in a third-party app. Therefore, we only chose Tencent News, a third-party APP, as the representative. Because it’s the easiest way to get into a web page.

(2) To have clear feedback requirements, do a good job is formatted feedback. For example, we need to know about the PC and the phone we use. If the user text feedback in the body, the collected results will be very messy, also difficult to statistics. Instead, we make this information into options or individual text boxes. Mobile phone to the information is more standardized and convenient statistics.

See the attachment for specific test requirements.

After 2 days of question collection. We got over 40 responses. I made a statistical analysis of the problem and output the mass test result report:

TBS Studio tool mass test results analysis

Test version: TBS Studio: v1.3.1 pre-launch test version.

Test duration: 2 days.

Test requirements: See attachment.

Data analysis:

(1) Public collection of 42 feedback (after the public test review), of which 30 were successful, there are 8 problem feedback. The proportion of users who encounter problems is not small.

(2) Among the 8 feedbacks, “Blank screen on inspector page” is the most common one with 5 feedbacks. In addition, there were 1 crash users, accounting for 1/18 of all win10 64 users (there were also many developers feedback this problem before).

(3) According to the PC type, a total of 32 PCS participated in the test (it is considered that one QQ user corresponds to one PC). Win10 has the largest proportion of 64-bit users. Received no feedback from MAC users.

(4) According to the statistics of mobile phone brands, a total of 8 brands are covered. Xiaomi has the most mobile phones participating in the mass test. No particular brand was found to have a particular concentration of problems.

(5) According to ROM versions of mobile phones, ROM versions of 4.x-7.x are covered. 6. X has the most mobile phones.

(6) According to the statistics of app use, most users use wechat and mobile Q test. The number of users using Tencent News test is 0.

(7) In the 6 feedback of QB browser, users who started debugging without opening the web account for 50%. Indicates that the program’s boot prompt is not sufficient.

2. Release rewarding experiences for developers within the company

After the completion of the public test, we released the company’s internal award-winning public test in the company BBS forum.

In the space of a week, we received six valid responses. Although the feedback is not much, but the feedback information is more detailed. Development colleague April’s evaluation of penguin mass test and internal experience is as follows:

“In fact, they are not bad, the coverage of the mass test is relatively wide, it will be easy to encounter strange problems. This time I also found a bug in Inspector’s blank screen.

Internal experience, communication is more convenient, and most of them are colleagues who need this tool, can give some suggestions and so on.”

7. Online data monitoring

With that, TBS Studio launched without a hitch. In terms of the background data, the daily active users increased quite a bit (the increase on June 16 was due to mass testing, the introduction of new users into the internal experience, and the release of V1.3.1).

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Old driver teach you how to gracefully complete a small project test

An overview,

Ii. Introduction of test project

Iii. Application for testing manpower

Four, test strategy formulation

5. System test

1. Test case writing

2. Stand-alone test

3. Model adaptation

4. Host adaptation

5. Adapt to the platform

6. Bug regression and pre-launch testing

Six, mass testing and experience

1. Release mass tests

2. Release rewarding experiences for developers within the company

7. Online data monitoring

Recommended reading

Old driver teach you how to gracefully complete a small project test

An overview,

Ii. Introduction of test project

Iii. Application for testing manpower

Four, test strategy formulation

5. System test

1. Test case writing

2. Stand-alone test

3. Model adaptation

4. Host adaptation

5. Adapt to the platform

6. Bug regression and pre-launch testing

Six, mass testing and experience

1. Release mass tests

2. Release rewarding experiences for developers within the company

7. Online data monitoring

Recommended reading

Related Posts

Strings and Characters Of Swift

Redis is probably the most detailed Redis memory model and application interpretation so far

Four design principles and 19 solutions for microservices