This article has participated in the activity of “New person creation Ceremony”, and started the road of digging gold creation together.

This is A series of articles about experiment and decision making in Netfilx. It systematically introduces the importance of experiment to Netflix, the most important tool of experiment –A/B testing, and the importance of data science and culture in the experiment and decision making process. This is the first in a seven-part series. 原文 : Decision Making at Netflix[1]

  1. Netflix’s decision making 👈
  2. What is A/B testing?
  3. False positive and significance statistics of A/B test results
  4. False negatives and statistical efficacy of A/B test results
  5. Build confidence in your decision making
  6. Experimentation is the main focus of data science
  7. Culture of learning

This article is the first in A series on how Netflix makes its decisions based on A/B testing. Based on decisions made through A/B testing, Netflix can continue to improve the product, which will continue to increase member satisfaction and make our members’ lives happier. Subsequent articles will cover basic statistical concepts of A/B testing, the role of experimentation in Netflix, how Netflix invests in infrastructure to support and expand experimentation, and the importance of experimentation culture in Netflix.

From its inception, Netflix’s philosophy has been to put consumer choice and control at the center of the entertainment experience. As a company, we constantly improve our products to improve this value proposition. Netflix’s user interface, for example, has undergone a complete transformation over the past decade. Back in 2010, the user interface was static, with limited navigation options, and the presentation was inspired by the VCD rental store display. Today, user interfaces are immersive and video previews, navigation options are richer but less obtrusive, and box designs take full advantage of the digital experience.

Netflix made countless decisions to move from the user experience of 2010 to the user experience we have today. Display one video title or multiple video titles on a large display? Is video better than still images? How to provide a seamless video transfer experience on a restricted network? How do I choose which titles to display? Where does the navigation menu go, and what should it contain? The list goes on and on.

Making a decision is easy; making the right one is the hard part. How can we be confident that the decisions we make will provide a better product experience for existing members and help recruit new members? In the traditional way, here are some ways Netflix can improve its product in order to make it more fun for its members:

  • Let the leader make all the decisions.
  • Hire experts in design, product management, user experience, streaming delivery, and other areas, and adopt their best ideas.
  • Have an internal debate and let the views of our most charismatic colleagues prevail.
  • Copy your competitors.

Either way, decisions depend on the opinions of a few people. There are only a few people on the leading panel, the panel debate is small, and Netflix has only so many experts in each area where decisions need to be made. There are probably dozens of streaming or related services that we can use as inspiration. Furthermore, these models do not provide a systematic way to make decisions or resolve conflict perspectives.

At Netflix, we believe there is A better way to determine how to improve the experience we offer our customers: A/B testing. This is an experiment-based approach that, unlike a small group of executives or experts involved in decision-making, gives all of our members the opportunity to vote on how to continue to grow the enjoyable Netflix experience through their own actions.

More broadly, A/B testing, along with other methods of causal reasoning such as quasi-experimentation [2], is Netflix’s way of using scientific methods [3] to inform decision making. We make hypotheses, collect experimental data, provide evidence for or against our hypotheses, and then draw conclusions and generate new hypotheses. As my colleague Nirmal Govind explains [4], experiments play a key role in the iterative cycle that underpins the deduction of the scientific method (drawing specific conclusions from general principles) and induction (forming general principles from specific results and observations).

Subsequent articles will delve into the details of A/B testing and how Netflix uses testing to inform decisions.

References: [1] Decision Making at Netflix: netflixtechblog.com/decision-ma… [2] Quasi Experimentation at Netflix: netflixtechblog.com/quasi-exper… [3] Scientific Method: en.wikipedia.org/wiki/Scient… [4] A/B Testing and Beyond Improving the Netflix Streaming Experience with Experimentation and Data: Netflixtechblog.com/a-b-testing…

Hello, MY name is Yu Fan. I used to do R&D in Motorola, and now I am working in Mavenir for technical work. I have always been interested in communication, network, back-end architecture, cloud native, DevOps, CICD, block chain, AI and other technologies. The official wechat account is DeepNoMind