This article is participating in “Python Theme Month”. Check out the details: Python Writing Season, show your Python articles – $2000 limited prize waiting for you to win

A lifelong learner, practitioner and sharer committed to the path of technology, a busy and occasionally lazy original blogger, an occasionally boring and occasionally humorous teenager.

Welcome to dig friends wechat search “Jie Ge’s IT journey” attention, background reply: AB testing actual combat, get the complete code.

Data Analysis: AB Testing for data analysis

directory

Growth hackers

1.1 introduction

When it comes to AB testing, we have to talk about growth hacking. The term was introduced to China in 2015, but it became hot in 2018. Therefore, if Internet companies want to increase active users and increase revenue, they still use growth hacking for product operation at present, which is not something that product managers can think of at a moment’s notice or bosses can make decisions directly.

People are now “data-driven”, using data-driven methods to help run better products.

So what is “growth hacking”? The popular saying is “trees die, people live”. If an Internet company wants to grow and become a giant, it also needs to move and change, constantly changing its products and upgrading its products. Otherwise, it will be killed in such a jungle and fierce competition.

We sometimes feel that the Internet company is [financing, burning money, pull new, financing, burning money, pull new… listing (bankruptcy)] such a process, good luck on the listing, bad luck on the bankruptcy. But in fact, many Internet companies, even if it is burning money, the way of burning money is also a lot of exquisite, not the boss, product manager or a director clap head decision.

1.2 Use the analysis index framework to drive Internet products and operations

Specifically, how do growth hackers use the analytical index framework to drive Internet products and operations?

Now share a link for you to understand:

www.woshipm.com/data-analys…

The whole Internet, whether it is product, marketing, sales, etc., is now basically using “data driven” such a way to operate, which is why the “data analysis industry” in recent years so hot. “Growth hackers” a lot of people have summed it up with AARRR, as shown in the figure above. Let’s make a caption for the figure above.

The first choice is “Acquisition”, how to use a more efficient way (APP, website, Baidu or Taobao to buy some advertising,) to acquire users and increase the number of users. Next comes Activation, how to activate acquired users and make them active.

Then there is “Retention improvement”. I try to attract users into my product through various channels, how to make them become loyal users of our product. Then comes Revenue. If a company needs to survive, it must make money to obtain Revenue. Then how to obtain more Revenue? Do you subscribe to more VIPs, or do you sell them more products or ads to generate revenue?

The last one is “fission”. If we do the process well, will the users help us do the Referral and spread the word of mouth (wechat, moments, etc.) to promote us and expand more users?

In this way, the above mentioned parts form a good closed loop, which continuously degenerates benign development.

1.3 Growth hacking can be roughly divided into the following steps

The first is to “analyze the status quo”. What are the problems with current products? What can be improved?

And then “set goals.” What do you want to do? Like iQiyi, Tencent Video, is to increase the number of VIP; Like Taobao, how to increase more advertising revenue; For Douyin, how to increase daily activity and how to make users keep checking Douyin every day? In this way, I will have more advertisements, more active users and more income.

Next comes “propose improvements”. After presenting your proposal, is your proposal the best? It’s not. This is what we call “small-scale testing.”

And then the “start small test”, take a part of the test users, let them to see if they are satisfied, whether the feedback is good. How do you know if the feedback is good? This is the collection and analysis.

Then it is “collection and analysis”, data analysis of the data obtained by the test user, if the feedback effect is good, adjust the flow, and constantly expand the scale to test. If the feedback is not good, stop or modify the solution (go back to the previous step), which is an iterative process known as “AB testing”.

“AB Testing” is to help us optimize the growth hacker process through data analysis, and use data-driven methods to help analyze our products and our user feedback.

Two, introduction of AB testing

2.1 Schematic display of AB testing schemes

Figure 1: revision of two web pages on Tmall

Figure 2: Revision of two versions of wechat

The product manager changed a new version. Is it any good? Can’t we just slap our heads and use whatever we think is best? If the feedback isn’t good, if the number of active users goes down, if the revenue goes down, that’s a huge loss for the big companies, and nobody can afford it. Therefore, data analysis such as AB testing is needed to minimize this risk.

Another example is Facebook. They have a very senior executive who promotes their products and thinks that a product is so good that they need to use AB testing for the release and iteration of all Facebook products. In a small scale of user testing, if user feedback is not good, It becomes less active and feels very difficult to use, even if the senior executive pushes it, it will not work. Data must be used to speak. If the product is not good, it will give up this version and develop the next version or find another breakthrough.

Many of the things at home company basic it is such a play, for instance WeChat version 1.0 and above WeChat version 2.0, is not to say that just patted his head is used to, is also using the AB testing, a lot of time for everyone to see a different page, find the part of the small flow of users, to help us to test, Look at their feedback.

2.2 What is feedback?

We’ve mentioned “user feedback” many times above, but what is “feedback”?

In fact, it is the length of use of this part of the user ah, the revenue ah, such as Baidu advertising (whether he points ads), such as iQiyi paid website (whether he has changed from an ordinary user to a VIP user), these indicators can verify whether your new version is good.

2.3 How to select such a group of small traffic users?

The most important thing is randomness. We can’t just select some users somewhere in Shenzhen as test users. We also can’t only select a limited range of users, such as 25-30, as test users. Do a version test with a random sample of say 1% of your users as low traffic users and see their feedback.

If the feedback is good, we will consider increasing traffic by 2%, 5%, 10%, 20%, 50% or even 100%.

If the feedback is not good, we have a choice to terminate the experiment and find a new breakthrough, or choose to improve our version.

2.4 What is AB testing?

Simply put: Decide which of the two elements or versions (A and B) is better!

2.5 How to do AB testing?

Develop two (or more) solutions for the same goal prior to the formal iteration release of the product;

  • Propose ideas and set assumptions;
  • Estimate costs and set priorities;
  • Design scheme;

In daily life, we always say AB testing, which is the comparison of two versions. In fact, it can also be ABCD testing, which is the comparison of four versions, but in reality, we do more AB testing. Another, it is estimated costs, this is necessary, if you cut 50% of the traffic traffic (or more), will be a new version of the online you run for a week, if the situation is very bad, for a large company, possible loss of several million, or a few dozen million, so before experiment, must take the budget, How much risk you can afford to take is the last step in setting your plan.

The user traffic is divided into several groups. On the premise of ensuring the same characteristics of each group of users, users can see different scheme designs.

Note a few terms that will be useful later in the field code. When doing AB testing, it is generally divided into the control group and the treatment group, where the control group sees the old page and the treatment group sees the new page.

According to the real data feedback of several groups of users, scientific help product decision-making;

Analyze log data used by users to determine whether to expand the experiment or continue to modify the scheme and iterate again. If the experiment is judged to be successful, the experiment scope is expanded. If the experiment is judged to have failed, the experiment will be terminated and the scheme will continue to be modified.

Third, AB testing

3.1 Statistical basis of AB testing (independent two-sample hypothesis testing)

About hypothesis testing this knowledge point, I have been introduced in detail in the previous article, here is not one explanation, a few pictures for your reference.

If you are interested, you can take a look at this article: blog.csdn.net/weixin\_412…

3.2 Python code demonstrated by AB Testing

Before we get into the code, let’s make a statement about each field: user_id is the user ID; Timestamp is the time when the user visited the page; The new landing page is assigned to the Treatment group and the old landing page is assigned to the Control group. Landing_page indicates the landing page; For example, for a member of iQiyi, the average user converted to a VIP user. For example, for a member of iQiyi, 1 means converted and 0 means not converted.

Import pandas as pd -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # read data, View the first five lines of df = pd read_csv df (" ab_test. CSV "). The head () -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # data preview, . View the data line how many, how many columns df shape -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # to check the data values are free of df. Isnull (.) any (df), info () -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # to check errors in the data line of print ((True)! = (True)) print((True) ! = (False)) print((False) ! = (True)) print((False) ! = (False)) """ true ! = true fasle treatment new_page true ! = false true treatment old_page false ! = true true control new_page false ! = false false control old_page """ Group =treatment and landing_page=old_page and group=control and landing_page=new_page; num_error = df[((df.group == "treatment")!=(df.landing_page == "new_page"))].shape[0] num_error -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # after remove the wrong line, again to see if there are mistakes line print (" no delete error before line number of records: ", df.shape[0]) df2 = df[~((df.landing_page == "new_page")&(df.group == "control"))] df3 = df2[~((df2.landing_page == "Old_page ")&(df2.group == "treatment"))] print(" ", Df3. shape[0] print(" error line common ", STR (df.shape[0]-df3.shape[0])," record ") num_error2 = df3[(df3.group == "treatment")!=(df3.landing_page == "new_page"))].shape[0] num_error2 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # check whether there is a repeated lines of print (" data on the number of records for: ", df3.user_id. Shape [0]) print(", df3.user_id. Nunique ()) print(", df3.user_id. Next, we can find the duplicate record and erase it. "" "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # to check the duplicate rows df3 [df3. User_id. Duplicated (keep = False)] # remove duplicate rows df4 = df3.drop_duplicates(subset=["user_id"],keep="first") df4.shape[0] -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- control_converted = # we look at some of the control group conversion Treatment_line = on line with treatment_line = on line with treatment_line = on line with treatment_line = Df4.query ('group=="treatment"').mean() treatment_in "" According to the above results, the old page conversion rate is better than the new page conversion rate, does that mean the old page is better? "" "-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- # independently two samples of hypothesis test import statsmodels. Stats. The proportion as the SSP converted_old = df4[df4.landing_page == "old_page"].converted.sum() converted_new = df4[df4.landing_page == "new_page"].converted.sum() n_old = len(df4[df4.landing_page == "old_page"]) n_new = len(df4[df4.landing_page == "new_page"]) data = pd.DataFrame({"converted":[converted_old, converted_new], "total":[n_old ,n_new]}) display(data) z_score, p_value = ssp.proportions_ztest(count=data.converted, nobs=data.total, Alternative ="smaller") print("Z =", z_score) print("P =" ", p_value) ---------------------------------------------------------Copy the code

The results are as follows:

Results analysis:

According to the above results, the P value is 0.9, much higher than 0.05, that is to say, we have no reason to reject the null hypothesis, that is, we can only accept the null hypothesis, that is, there is no big difference between the new version and the old version. The next thing we need to do is stop the experiment and continue to refine our approach.

3.3 AB testing expansion

So far as AB testing is concerned, let’s stop here.

Here are a few links for your reference:

Reference

  • www.woshipm.com/data-analys…
  • Blog.csdn.net/weixin_4126…
  • M.blog.itpub.net/31555699/vi…
  • www.jianshu.com/p/61e6c34d0…

Recommended reading

Are you still manually manipulating files or folders?

The most detailed Python and RFM model implementation of user layered practice case!

Use Python to merge multiple sheets, multiple workbooks, and split a table by column

Original is not easy, if you think this article is useful to you, please kindly like, comment or forward this article, because this will be my power to output more high-quality articles, thank you!

By the way, please give me some free attention! In case you get lost and don’t find me next time.

See you next time!