When you want to grayscale some new versions for some tests, you can choose Google Play scheme, but many of Google Play’s test schemes are black boxes for us, which need to be explored. After nearly a year’s experiment, we have gradually explored some experience of Google Play grayscale. I’ll share it with you here.
Recommended testing procedures
Suppose we have a major new release coming up, what process should we choose? Here is my suggested process for your reference. \
Internal testing
It also needs to be divided into internal team members and external groups of invited users for internal experience.
If it is your own internal team, you can use the internal APK directly install experience. However, if the user base is external users using the APP, then we need to apply it to The Alpha channel of Google Play. Also called internal testing on Google Play, you can refer to its procedures for specific methods.
After basically no problems in the internal test, we can release some parts to the external test experience, and then enter the external test process.
External test
External testing means that we need to provide this to real users to use, and then observe the data of real users to do some feedback testing. Here we divide into two cases, one is the user actively join the test plan, and the other is the user passively join the test plan.
We don’t want to affect the majority of users in the early stage, so we need to choose the first solution and let users actively join our test plan.
Beta testing plan
This feature is called open testing in Google Play. If you want to do this, you can send notifications within your own app to groups of users that you want to have open testing. The benefits of this are:
- Corresponding groups can be selected for testing. For example, anchors are the most affected by the revision. In order to avoid affecting the user experience of anchors, an open reminder is first carried out for the test plan.
- Avoid affecting a large number of users, and only test some users. Since you need to actively join the test plan, the test has little impact on the overall users.
- The beta can be closed at any time. If problems are found, the beta can be closed at any time and users can upgrade back to the latest official version;
Gray scale test in stages
Once all of the above processes have been tested and no version or data issues have been found, we can perform grayscale testing, which is called releasing app updates in phases on Google Play. This is often applied when we release new versions. We can observe the overall data in phases of 5%, 10% and 100%.
Frequently asked Questions
Grayscale test users, assuming that we are in multiple versions of grayscale 5%, then whether the 5% of users will be the same batch of user groups? It is the same batch of users. The gray scale mechanism of Google Play will cover the users with the gray scale of the previous version. Assuming that our current official version is 2.0.0, we need to test the situation of 3.0.0. This is great because 95% of users can download version 2.0.0 and 5% can download version 3.0.0. Then we found that 3.0.0 had a serious problem, or the data did not reach the normal standard, at this time we updated a grayscale version 3.0.1, so we closed 3.0.0 and continued with 5% of the grayscale 3.0.1, the 5% and the last batch of 3.0.0 version 5% are the same users. Let’s say we grayscale 3%, so that 3% is also a subset of 5%. If we add 3.0.2 and increase the gray scale by 10%, then 10% will contain 5% of the users and add 5% of the users.
If multiple locales are involved, what should I do if I only want to test in one location? Google Play’s grayscale mechanism supports grayscale testing by country, so it’s fine if you only want one region.
When there is irreversible gray scale, solve this kind of abnormal situation? Since this policy is usually adopted only when the files are of a large version, some problems may occur when the files are not deleted. That is, users cannot go back to the old version because the data is irreversible. At this time, suppose that user A uses the new version 3.0.2 in some cases, but the old version 2.0.0 is always displayed on Google Play, resulting in an exception when trying to use some functions when downloading the old version later, and prompting user A to upgrade. But once you go to Google Play, you can’t download the grayscale version. The solution to this situation is to create a separate beta, upgrade the beta to 3.0.2, and then inform this group of users that they can join the beta program to download the new version without affecting normal usage.
This is an important point to highlight, because we often get this kind of feedback in the gray-scale process, and the solution is to use Google Play’s open beta channel solution.
The business test is over, we found that we can fully measure when testing a certain country/region, but it is a gray scale region at this time, can we directly change it to full measure? It is possible, if you find that the version in a certain area gray no problem, then want to direct full, can be directly operated, do not need to be audited.
If multiple Google Play accounts are activated on one device, how should it be displayed? Imagine a scenario where I have logged in multiple accounts on Google Play. Due to the grayscale mechanism, I have one account in the grayscale and several others in the grayscale. Which version will be displayed on Google Play? At this point, Google Play will display the latest version, if the grayscale is the latest version, then the grayscale version will be displayed, if the beta is the latest version, then the official version will be displayed. And Google Play will have a reminder that XXX is already in the beta program or other test programs.
Is Google Play grayscale based on device dimension or account dimension? From the information provided by its official website, as well as the actual operation, are grayscale with the account dimension. However, there is a conflict point, which I mentioned above. If multiple accounts are logged in to the device, as long as one account is in the gray scale and the gray scale is the latest version, other accounts can also see the latest version, but this kind of situation is rare.
What is the reason for the fact that more users actually downloaded 3.0.2 than 2.0.0? Mainly because the gray scale of Google Play can only reflect the situation of Google Play store, there are also many users who transform APP crawling into APK and put it on other websites for download, including some domestic stores such as Oppo and Vivo will take the initiative to crawl the latest package. And that affects the overall molar ratio. What if you want to distinguish Google Play downloads from other sources? You can apply the Android installer to obtain the installation source to distinguish.
Why do you see a lot of old download data in the Google Play store, you should see the latest version, or only 3.0.2 and 2.0.0 have other version data? You can still see a lot of download data for older versions on the Google Play Console, but that data is entirely from Google Play, not non-Google Play. After seeing these exceptions, I go to check the cause of the problem, then find an article that show: android-developers.googleblog.com/2018/10/off… The main meaning is that offline sharing exists, and such offline sharing also belongs to Google Play. There are many applications of offline sharing on Google Play, and the old version can be shared with other users, such as SHAREIt, Files Go and Xender.