This article translated from netflix technology blog, the original: netflixtechblog.com/python-at-n…
Netflix (Netflix) is the world’s largest video streaming platform, in all countries and regions except the Chinese mainland to provide video on demand services. Netflix is a bright spot for companies operating online businesses. IT successfully combines traditional video rental business with modern marketing means and advanced IT network technology, thus creating a new situation of online video rental. Netflix has managed to change consumer habits and build its brand advantage by integrating its own marketing and more recently IT network technology.
Netflix’s technology essentially answers two questions:
1. How to improve users’ viewing experience?
2. How to make users like the content?
These two questions, with the help of Python, have been satisfactorily answered. Let’s take a look at how Python works on Netflix.
1. Speed up video distribution
To speed up the delivery of videos, Netflix has developed a Content Delivery Network (CDN) called Open Connect, as close to users as possible, to improve the streaming experience for customers and speed up the download of videos.
Many of the various software systems needed to design, build, and operate this CDN infrastructure are written in Python. It has a configuration information management system that tracks Netflix’s inventory of network equipment: which devices it has, which models, which hardware components it uses, and where they are located. The configuration of these devices is controlled by several other systems, including data sources, application and backup of device configurations. Device interaction for gathering health and other operational data is another Python application.
2. Quickly solve operational requirements
Netflix also uses Python to handle the Netflix cloud’s area failover, traffic management, and capacity operations management (limits on managing content that can be repaired), using Python modules that are roughly as follows:
NumPy and SciPy
NumPy and SciPy are libraries for scientific computing. Netflix uses these Python libraries to perform numerical analysis, allowing for administrative area failover.
Boto3
Boto3 is a software development kit (SDK) for Amazon Web Services (AWS) for Python. This helps Python developers integrate Python into AWS, allowing development within the infrastructure.
RQ (Redis queue)
This is a Python library that helps keep track of tasks that exist in queues and allows them to execute, thus allowing asynchronous workloads to be managed.
Flask
Finally, Netflix uses the Flask (Python Web Development Library) API to bundle all the previous functionality together.
Their tools are largely built in Python. Choreographed failover services use NUMpy and SCIpy to perform numerical analysis, BOto3 makes changes to the AWS infrastructure, and RQ is used to run asynchronous workloads packaged in a simple interface layer of the Flask API. The ability to drop in a BPython shell and improvise has saved the day more than once.
(BPython is a nice interface for the Python interpreter. The developers aim to provide users with all the built-in functionality, much like today’s IDES (integrated Development Environments), but packaged in a simple, lightweight package that can be run in a terminal window, making up for the shortcomings of the original interpreter.)
3. Alarm and statistics, automatic repair
They also use Python for alarm and statistical analysis work. When the alarm system shows problems, they use Python’s many statistical and mathematical libraries (such as Numpy, Scipy, Ruptures, Pandas) to help automatically analyze more than 1000 related signals. The Spectator Python, a time series related system for both inside and outside the team, and the Distributed work system were developed to process large amounts of analytical work in parallel to deliver results quickly.
In addition, they often use Python as a tool for task automation, data mining, and cleansing, as well as a handy tool for visualization.
4. Information security
Netflix’s information security team uses Python to achieve many high-level goals for Netflix: security automation, risk classification, automated fixes, vulnerability identification, and more. Use Python to protect SSH resources using Bless, use Python help to adjust IAM permissions using Repokid, use Python to help Lemur generate TLS certificates, and more.
5. Personalized recommendations
When it comes to personalization algorithms, of course, machine learning and deep learning are indispensable. They used Python to train machine learning models in key aspects of Netflix’s impact on the user experience, such as decision trees and XGBoost, and implemented an automated process along the way.
6. Optimize Python limits
To make things more productive, they also developed a Python framework called Metafolw using machine learning. This framework pushes the limits of Python and, through good design and some modifications to Python code, makes it possible to retrieve tens of gigabytes of data using Python, greatly increasing computing power.
7. Big data orchestration
To better orchestrate big data, they also built an event-driven platform written in Python, unifying many systems into the tool and scheduling them in the form of event streams, with template-based job types.
So they can decouple microservices all the time, and through this platform, they can understand how the data is flowing, and everything that’s going on. Of course, there are more technical factors that make Netflix’s system so stable than we can describe here. Have a look at the following book if you are interested.
Chaos Engineering: Netflix System Stability
Official addresses of some projects and libraries involved in the article:
Open the Connect: https://openconnect.netflix.com/en/
Bpython: https://bpython-interpreter.org/
Nteract: https://nteract.io/
The visualization tools: https://github.com/nteract/nteract/tree/master/packages/data-explorer
Spectator： https://github.com/Netflix/spectator-py
Security Monkey： https://github.com/Netflix/security_monkey
Bless： https://github.com/Netflix/bless
So that’s the end of our article, if you enjoyed our Python tutorial today, please keep checking us out, and if it helped, please give us a thumbs up/check it out below. If you have any questions, please leave them in the comments below, and we’ll be patient to answer them!
The Python Utility Guide is more than just a guide
How does Netflix use Python?
