Author | GrowingIO growth team, engineering, product, market and analysis of multiple roles at a suit, is responsible for the new user is active and with data driven business growth.
Source | excerpts from GrowingIO the data analysis product manager’s handbook, click here to download immediately
There is a popular question on Zhihu: “What do you think about product managers talking about data? How do you do data analysis?” “Attracted 6,800 views and 110 responses. To some extent, the description of the subject reflects the embarrassment and helplessness of product managers in the process of data analysis.
“In fact, it’s a feature that users across the platform want to do, so there’s no need to evaluate it, just do it.”
In reality, isn’t it true that features that users want to do don’t need to be evaluated? Are user requirements met?
“A lot of data and estimates are necessary, but some of them are very formal. Does it make sense?”
How can data analysis be done without becoming “formalized”?
First, talk about product data analysis from a revision
By the end of 2015, our GrowingIO product had gone live in version 1.0 and served many customers. With the increasing number of new users, the product manager receives a lot of feedback from new users every day: they hope to see the product introduction video on the official website, so as to further understand the functions and features of our products.
Since users are mentioning this every day, it must be something important. As the zhihu netizen said above: “In fact, the whole platform of users want to do this function, there is no need to spend manpower evaluation, just do it.” So without thinking, with the efforts of the product manager and marketing students, we soon finished the introduction video and put it on the home page of the official website.
After adding the new introductory video, the product operation wanted to see how it affected the conversion rate of new users to sign up. After observing the data for a week, the Click Throught Rate (CTR) of new users did not rise, but fell by 20 percent from 40 percent.
For a fast-growing startup, a significant drop in conversion rates is certainly unacceptable; But just to be sure, we kept watching for another week. After two weeks of no improvement, the product manager removed the introductory video from the website. Miraculously, CTR began to return to its pre-revision level.
This product introduction video took a lot of manpower, energy and material resources from the product and marketing departments, and was specially made because of the needs of users. Why did the conversion effect decrease after it was made?
After reflecting on the whole process, we got some inspirations:
(1) The views of some users cannot represent the real experience of all users, and the product introduction video may be a fake demand;
(2) The subjective perception of product managers cannot represent the real experience of users;
(3) The newly added video introduction distractions the energy of new users, resulting in a significant decrease in the registered click rate of the home page.
Fortunately, we used our own products to do real-time monitoring of our official website and compared the CTR index before and after the revision, otherwise we are still in the pit of the official website revision. Therefore, data monitoring and data verification before and after revision is very important.
Product managers need data to speak for themselves
Believe the above GrowingIO case has been able to give you an intuitive feeling. Gone are the days of making decisions purely by head, by heart and by experience. Product managers must master data analysis skills and use data to speak for themselves!
(I) Iterating products through data analysis
In the process of user research, many products will put themselves in the user’s shoes. Good intentions, but it’s easy to get caught up in mistakes; Because any simulation is lame, the mindset dictates that the product manager cannot imagine 100% of the actual user behavior.
I’ll give you a classic example: Instagram. Instagram’s predecessor, Burbn (pictured left), was a location-based check-in loyalty app that had long been a tepid success. Until one day, they analyzed the data and found that Burbn’s users showed little interest in the check-in feature of the main product, preferring instead to use the photo-sharing feature embedded in the system. That’s when the product logic began to rework, and the focus was on photo-sharing, which led to Instagram (pictured right).
Users are so strange, never according to the routine card; You need to constantly observe their interests through data, rather than sitting in your office simulating or assuming.
(2) Insight into users through data analysis
Every iteration and upgrade of the product needs to be evaluated for future improvement. If it is only based on the visual observation and subjective feelings of the product manager, the next conclusion like “user feedback is good” and “users like this new feature” is very weak. More often, there will be a misjudgment based on subjective perception.
We had a client who had a period of increased conversion rates; The product manager was pleased, though it was not clear why. After a period of time, analysts found a BUG in the [retrieve password] function through a detailed check of user behavior trajectory; Many old users forget their passwords and have to register a new account.
This BUG has inadvertently increased the site’s conversion rate, which is not a good thing; If product managers don’t do data analysis, they can fall into the trap of “conversion improvement”!
(iii) Verify products through data analysis
How to verify whether a new feature is good or bad requires data to speak for itself. In addition to the product’s own indicators, product managers should also focus on the business goals of the product or enterprise, for which the product is ultimately responsible. Above, we shared the case of GrowingIO homepage revision, which met the needs of some users, but resulted in a significant decrease in the registration conversion rate. The following case is from Facebook. Qin Chao, an early employee of Facebook and a partner of Fengrui Capital, once shared it in our offline activities, hoping to inspire you.
The left side of the image below shows the Facebook homepage in 2009, when a product manager wanted to experiment with a waterfall, flat design to improve the user experience. After designing several versions, it was decided to go with the design on the right. It took the engineering team several months to build it, and when we tested it internally, everyone thought it was fine. So, the new home page began to face 2% of the user gray release, look at the effect.
As a result, this 2% average time spent online indicator began to decline, directly affecting the advertising exposure of this segment of users. We all know that advertising is an important source of revenue for Facebook, and that exposure and click-through declines are unacceptable.
The students of the project team thought that it might be the time for users to adapt to the new version, so they decided to open another 10% of users and observe the overall effect. As a result, overall activity and time spent online fell by about 20 per cent over the course of three months. In the end, Facebook didn’t get through the redesign, and the home page rolled back to its original look. With such a large team, the project lasted for more than half a year, but it was cancelled directly because the data verification failed. Companies as big as Facebook are embracing data-driven thinking, so why shouldn’t we?
Iii. How to avoid trampling pits in product data analysis
Data analysis is important for product managers, but how do you do it? How can data analytics avoid becoming mere formality, and how can they avoid stomping holes?
(I) Vocal users vs. silent users
The GrowingIO case at the beginning of the article is a good illustration. Vocal users put forward demands for us to provide product videos, but silent users may be directly lost. In fact, the essence of analysis does not lie in whether users speak or not. The essence lies in the feelings of the real target users.
Data from vocal users is easily accessible, making it easier to analyze; Many silent users are lost, and users choose to vote with their feet, which makes data acquisition and data analysis of this part more difficult. If product managers only use the data of users when making data analysis, the conclusions of analysis are often biased. There are special studies on sampling in statistics, which is the same thing.
(2) Preconceived versus objective
Most of the time, people prefer to see what they want to see, which is preconceived or preset. Product managers are also prone to this kind of prediction when doing data analysis, which often turns into finding evidence for their own opinions.
For example, when a new feature is launched, management hopes to increase user activity by 5%. At this time, the product manager will pay too much attention to the current 5% and ignore the growth opportunity of 50% or even other negative effects when making the final offer.
(iii) Report driven vs. business driven
The ultimate goal of product manager data analysis is to guide product and user growth and help enterprises increase profits. New product people tend to attribute the end point of data analysis to data report when doing data analysis, thinking that finishing a product data analysis report is equal to finishing a data analysis.
Data analytics driving product and user growth is not a one-shot deal, but an iterative process. Product managers should constantly exercise their data analysis ability in the cycle of “user data – data analysis – product optimization”, focusing on product optimization and user growth.
More and more enterprises in the recruitment of product managers, will write in the job description “master data analysis methods”, “good at product & user behavior data analysis” and other words, product managers to know data analysis has been the general trend. We can’t deny the significance of data analysis just because some people formalize data analysis. Only by facing up to data analysis and avoiding misunderstandings can data analysis play its maximum value.
This text is excerpted from GrowingIO data Analysis Manual for Product Managers