With the acceleration of enterprise digital transformation and the deepening of the concept of user behavior analysis, various data analysis models, such as event analysis, funnel analysis, retention analysis and so on, have become indispensable “little assistants” in the daily operation of enterprises.
As a big data analysis and marketing technology service provider, Shence Data has served 1500+ enterprises in more than 30 industries in the past 6 years. In this process, we gradually realize that data analysis is only one part of the process of enterprise digital transformation. In order to make data truly valuable to enterprises, All decisions, actions and feedbacks based on data flow should be added into the enterprise operation framework SDAF based on data flow, namely Sense, Decision, Action and Feedback, to form a complete closed loop of business value output.
And when we think about it further, why is SDAF a ring, and why is it a continuous ring? At this point, we will find that this is because there are two common pain points: one is the imprecise goal, which may lead to inaccurate action and ineffective output; Second, imprecise means may lead to less final output than input. This is not only because of our own limitations, but also because users are fickle.
Therefore, Shence Data launched A/B test, combined with user behavior analysis, to bring solutions for enterprises to deal with user changes, maximize value output and efficiency, continuous optimization, continuous iteration, and bring users better experience “love”.
So how do we apply behavior analysis and A/B testing to achieve good integration and maximize the efficiency of value-enhancing output? To put it in plain English: “Leverage like an investor, experiment like a scientist.” And to find leverage, you first have to understand what our goals are. If the goal is described from the perspective of user behavior, it can be divided into one-time behavior and periodic behavior.
One-time behavior: We want users to engage in a single targeted behavior as much as possible, and this kind of behavior is often critical and can lay a solid foundation for the subsequent use of the product, such as user registration, first order payment, real name authentication, etc. This kind of behavior corresponds to our business goal of “transformation,” the event of getting more users on the right path to reach their goals.
Cyclical behavior: We want users to perform an action as many times as possible, or at a deeper level, usually as an action that reflects the core value of the product, such as paying for orders, viewing content, etc. This kind of question usually corresponds to the business goal of “engagement” or “retention,” which is to get users to do something more deeply.
Next, we will introduce the application practice of combining A/B testing with user behavior in detail based on the SDAF operation framework proposed by Shence data.
Sense: Use behavior analysis model to Sense user behavior
Our business goal is to get more users to do something on the right path for a one-time user behavior. Observations can be made by the following model:
Funnel analysis: How is churn on a strictly defined path?
Path analysis: How do the actual behavior paths of user groups diverge?
Behavior sequence: How does the actual behavior path of the individual user jump?
With periodic behavior, our business goal is to get users to do something more and more consistently, and make it a habit. Observations can be made by the following model:
Retention Analysis: How persistent are user actions over time?
Distribution analysis: What is the frequency (intensity) of user behavior?
Decision: Make decisions based on observed data using the data correctly
Data alone is difficult to generate value, so after completing Sense, further interpretation and decision-making are needed. I summarize data application as the following three points:
1. Limited restore scenario. Restore the whole decision-making process of users from an abstract and framed perspective, such as funnel analysis and user path analysis, so as to describe the overall behavior of users.
2. As a diagnostic basis. Perceived user behavior, through horizontal and vertical comparison of time or classification dimension, can roughly determine whether there is a problem.
3. Prioritize leverage. Knowing what’s important right now and what’s not so important right now can be a lot of problems, but one thing at a time.
Beyond that, what’s more important than using data is knowing how to do things? To describe this well, I usually make two lists.
The first list is called: Why? If we have a more accurate understanding of the problem to be solved, the effect of the solution is of course better, so we can list the possible causes from the perspective of power, resistance, opportunity and so on, and do some subjective ranking.
The second chart is called: What to Do? For these specific reasons, we can list possible solutions or the direction of solutions. Finally, the ICE model (Impact range, Confidence level, Ease implementation) was used for subjective ranking to determine what experiments we should do in the recent period of time.
For example, we can sort out the core path of e-commerce and find the weak links or leverage indicators on the core path, and further locate our focus.
When a problem page or module is located, we further outline why and what to do. For example, in the data analysis, we found that there was a transformation problem in the rote chart module of an e-commerce company. The penetration rate of several advertisements at the back of the module dropped sharply, and we believed that the importance of the module was relatively high after evaluation, so we could further list the following:
Action: Implements A/B testing based on business decisions
When we have a more specific direction of the test, we can further design the details of the test itself to list several elements of the test implementation:
The hypothesis of the experiment: the causal relationship between what we think is the goal and the means. (What changes affect what? What’s the reason?
Test variables: specific modified elements, usually single elements, facilitate attribution.
Test indicators: indicators to evaluate the success or failure of the test and related indicators.
Audience of the experiment: the users who will perform the experiment.
Feedback: Analyze test data and obtain Feedback and cognition
Feedback, literally, refers to the direct conclusions we draw from the results of trials, whether the groups are different, and which strategies are better or worse. Cognition, on the other hand, is something deeper, something we’ve learned that can be precipitated.
For example, in an e-shopping mall of our client, the conversion rate of luxury goods with large picture versions of various related slogans is worse statistically, which is the “feedback”. However, the slogan added to the large picture of luxury products will cause attention disturbance for users and affect the aesthetic feeling of the product, which is “cognition”.
“Even if there is a significant difference in the data, we are likely to make a mistake whether we accept a new solution or not,” he says in statistics. That said, the feedback could be wrong. What’s more, even when we get the right feedback, we can make cognitive mistakes. For example, we mistakenly believe that it is better not to add slogans on the big picture of luxury goods, but in fact, we should not only add the slogan of preferential policies, because it will look copycat and affect customers’ trust. Appropriately adding slogans can also improve customers’ trust.
Based on feedback, we can determine whether the latest experiment is valid or not, and we can accumulate more experience based on cognition.
Just like the refined operation that enterprises have been pursuing, behind it is the cumulative effect of such accumulation, which can help us accumulate more and go further!