Since last year, users’ products on Baidu MEG (Mobile Ecosystem Business Group) architecture platform have gradually undergone cloud native transformation, which has been basically completed. At present, more attention is paid to the construction of dynamic elastic capacity and dynamic management mechanism. In this issue of Geek Talk, we have invited Chuanyu, a teacher from the Recommendation Technology Architecture Department, to talk about the phased strategy of baidu’s cloud native transformation of search and recommendation engine, as well as thinking about its future development.
Guest profile: Pass jade
Since 2012, focus on search engine and recommendation engine direction; Since 2016, I have been responsible for the research and development of my own resource scheduling and container scheduling system. In 2019, I began to take charge of the research and development of some general basic components, and comprehensively promoted the transformation of cloud native architecture in MEG user products.
01 Focus on two “efficiencies” to achieve cost reduction and efficiency increase
“I agree that the goal of cloud native is to make the whole process of developing live applications easier. Baidu’s search and recommendation engines are huge and complex, and search, in particular, is more than 20 years old and cannot be designed entirely according to a new set of logic. So when we try cloud native, we have to combine our own genetics with our own reality.”
The main value of cloud native architecture lies in the improvement of efficiency, including resource utilization efficiency and R&D efficiency.
From the perspective of resource utilization efficiency, it is expected to realize full mixing of online services through containerization and dynamic resource scheduling, so as to make the overall resource utilization of the cluster more balanced and efficient, thus reducing the overall machine cost. In addition, cloud platform can realize efficient resource-based delivery instead of physical delivery, improve the efficiency of resource transfer, enable internal resources to be transferred to key businesses more quickly, and support rapid business development.
In terms of R&D efficiency, on the one hand, micro-service transformation can decouple the iterations of different business teams, reduce the interaction between teams, and improve the efficiency of business iteration. On the other hand, it is expected to sink the common infrastructure capabilities onto the cloud native infrastructure and raise the baseline level of new business architecture capabilities.
Some local fault tolerance ability, for example, in some business online similar architectural mechanisms already very mature, but for the new lines of business it is difficult to directly reuse, often need to pit, with reference to the mature business line experience, build, if we can get the ability of these common precipitation to the cloud in the form of standardization and normalization in the underlying infrastructure, Then the innovation line of business can be relatively easy to reuse, less detours, as little as possible owe technical debt.
In addition, the improvement in r&d efficiency of cloud native architecture is also reflected in the reduction of manpower and time in online problem handling and maintenance.
Generally, there is a saying in the industry: whether a storage system is good or not depends on its operation and maintenance level.
But in fact, for many innovation projects, not just storage systems, if too many people support the operation of online services and solve online problems, then relatively fewer people are invested in research and development, and the corresponding growth rate may be affected
Through some cloud-native infrastructure, we can standardize and automate all kinds of routine operation and maintenance operations, such as automatic maintenance of machine failures, automatic migration of service instances and automatic adjustment of service capacity. On the one hand, it can reduce the labor cost of operation and maintenance. On the other hand, in many cases, automated systems can do better than manual ones.
“Before, we also had automation mechanisms. But the benefit of applying cloud native architecture is that it allows us to do these automated mechanisms in a more disciplined, standard, and sustainable way. Is to free up a lot of manpower from the maintenance of this online service. “With the same team size, you have fewer maintenance people, so you have more people available to focus on r&d, and overall r&d efficiency increases.”
Overall, the greatest significance of cloud native is to improve efficiency and improve the baseline of overall R&D.
Especially when making new products, it can save the cost of purchasing resources and save too much human input in the basic stage to ensure the smooth launch of products. The lower the cost, the more innovation you can do. This allows each new product to avoid losing the starting line.
02 Standardize service design standards, set good rules for cloud original transformation
The MEG Architecture platform went fully cloud in 2019. However, the migration of most services is just a change from physical machine deployment to in-container deployment of PaaS platform, without the transformation and upgrade of cloud environment and cloud capabilities to achieve greater cost and efficiency benefits. Based on this problem, it is expected to further standardize the MEG architecture platform service design standards to realize the transformation from cloud architecture to cloud native architecture.
“We already have a certain foundation before we implement cloud biogenesis. First, the whole organization has the idea of micro-service; Secondly, a series of micro-service best practice standards were formulated from practice, and “MEG User Product Architecture Cloud Native Internal Specification” was established. Third, we already have a set of public infrastructure.”
In order to ensure the efficiency and effect of the implementation of cloud native architecture, Chuanyu standardized the service module design from the following three aspects by referring to the characteristics of cloud native application widely recognized in the industry and combining with the prior practice of Baidu:
1. Microservitization: Each service granularity should be within a limited range; 2. Containerization encapsulation: the deployment of a service should rely only on the infrastructure and components in the container, rather than on other business services; 3. Dynamic management: Each service should be able to adjust its deployment dynamically without affecting its external COMMITMENT to SLA.
* Evaluation methods for specifications:
* Overall business evaluation method:
** Services that are not connected to PaaS shall be calculated as not meeting the standards;
**2. ** evaluates whether a service meets the specification on a service-by-service basis. Only when a service simultaneously meets all the above criteria, a service is considered to meet the cloud native architecture specification;
**3. ** Each line of business calculates the cloud native specification index by a percentage (total quota/ total quota occupied by conforming service modules), using CPU quota/MEM quota, the lowest proportion;
**4, ** each individual score is only used as internal indicator reference for analyzing bottlenecks and promoting landing.
03 Key points, the phased realization path of cloud biogenesis
From cloud to cloud biogenesis, it is a very complex process. After the development of the specification of cloud native transformation, MEG successively went through 4 stages, which are: ** micro-service transformation, containerization transformation, dynamic management, and advanced cloud native. ** And MEG’s process of cloud primitive transformation did not stop, but continued to explore the fifth stage — declarative architecture.
The first stage: micro-service transformation
At the beginning, when Baidu MEG architecture platform realized full cloud, all architecture services and resources were hosted on the internal cloud platform, but there were still problems in resource utilization at that time. MEG architecture platform to implement cloud native first thing, is to require all services to do micro service transformation, eliminate giant services.
“These huge services will lead to fragmentation of the overall resource allocation. For example, a service occupies 80% of the resources of a machine, and the remaining 20% May not be separated, so it will be monopolized and cannot be mixed. Some services need to be modified before deployment.
So even though all of our resources were hosted in the cloud, our usage was still very similar to that of the machine. OP invested a lot and the overall online resource utilization, including the resource allocation rate, was relatively low.”
The split of microservices brings several changes: First, performance improvements.
Although there is some RPC overhead, after the split, each service can be optimized specifically, and all scaling operations can be performed only for the logic of this service. Therefore, the overall cost, delay and other aspects of performance to achieve a significant improvement.
The second is to improve the efficiency of r&d.
According to the original product and strategy iteration, a service often needs hundreds of people to jointly carry out, which takes a long time to launch. But after the split, although there are dozens of modules, but a module only need two or three people to iterate, can also be online anytime and anywhere. This will greatly improve the overall efficiency of r&d.
“Our Feed video recommendation service, for example, was a huge service with lots of iterations before it was split. Single instance 450 CPU, 100G memory, single module more than 40+ policy RD to participate in the development, online feature 10+ every day. Therefore, a large number of resource fragments, long online time and difficult memory iteration are generated in the operation process.
We did three things:
First, according to the recommendation business logic, split into aggregation and recall two layers;
Secondly, the recall layer is divided into several parallel recall services according to the recall algorithm, and the part of recall services can be discarded.
Third, the aggregation layer is split into machine learning prediction service and aggregation service, and machine learning service is accelerated using AVX instruction set.
The results of Feed video recommendation service transformation are as follows:
L A single large module is divided into 10+ small modules, and the largest module occupies less than 40G memory.
L Overall CPU usage is reduced by 23% and memory usage is reduced by 84%
L Delay reduced by 50+ms, stability improved from less than 99.9% to 99.97%
L The efficiency of iteration is greatly improved, and the situation of mutual block on the line of hitchhikers is completely eliminated.
The second stage: container transformation
The MEG architecture platform is containerized, which requires all services to put their dependencies into containers. Achieve one-click deployment of all services, that is, automated deployment.
Some emerging Internet enterprises may not have this problem, because many of their services are based on Docker. But baidu’s search system is 20 years old, so it has to take time to do this. In this process, a typical transformation is search BS module, its age is almost as old as Baidu search.
Twenty years ago, baidu architects designed BS with a view to maximizing the resources of the entire system.
“SSDS were very expensive at the time, so when YOU designed the BS, you wanted to use them to the full, and also, for convenience, you didn’t show them all, like you said you were using 10 gigabytes, but you actually used 60 gigabytes. This was fine at the time, because a machine had only one service, and it was no one else’s business to use resources either explicitly or implicitly. Today’s disk hardware is so different from what it was 20 years ago that the computing capacity of a single service is often not enough to cover the entire machine, so to avoid waste, other services need to be mixed on top of each other. Then there are problems and they have to be changed.”
First, each service explicitly declares the resources it needs to occupy, eliminating greedy preemption.
Put all resources in his own container. In other words, BS is changed from a machine-level service to a container-level service, with no out-of-container resources. This is where the scheduling capabilities of the container choreographer system come into play.
The second thing is to improve service deployment efficiency.
Some older services may need to pull a lot of extra data during deployment, or even require the OP to manually do some machine parameters and configuration adjustments during deployment. This can result in deployments that are either not automated or are deployed too slowly. To improve efficiency, services need to be completely dependent on the physical environment and only rely on the environment inside the container. In addition, we also need to optimize P2P download and some real-time data transformation to optimize the speed of service startup.
“We used a service to store data classes that was logically portable, but it actually took about eight hours to migrate an instance. This kind of portability doesn’t make much sense. Because the storage data service is limited by the number of copies/capacity/concurrency, for example, there are hundreds of instances in a cluster, but only a few instances can be migrated at most at a time, and then it takes 8 hours to migrate a round. Then the whole cluster migration time will be very long. Kernel upgrades, OS upgrades, bug fixes, etc. So we need to optimize for P2P distribution, data download and data distribution to speed up deployment.”
Stage 3: Dynamic management
Dynamic management, mainly by “dynamic”, such as whether online services can be moved at any time, at any time to expand and shrink.
It is divided into two levels:
On the one hand, from the perspective of business itself, dynamic management requires all services to have a certain degree of flexibility and fault tolerance.
Because instances online that involve scaling or migration can be restarted or unavailable for a short period of time. This first requires that all services on the line have a certain fault tolerance. In addition, the service needs to have automatic load balancing capability (the simplest way is using the Naming Service to access the service). In addition, the service needs to automatically disperse traffic when new instances are added, and timely shield instance failures or exits
On the other hand, from the perspective of infrastructure, all services can be migrated and scaled up at any time.
Then we can automate on-demand allocation of service capacity and traffic on demand.
“Once a business is running an operation, it needs to do a lot of capacity expansion on an AD hoc basis. This process is very troublesome in non-cloud native mode. We need to first find a batch of physical machines, and then deploy the service to these new machines to realize capacity expansion. After finishing the activities, we will remove the service, and then exit the physical machine. But after the dynamic transformation, the operation of finding the physical machine is gone. All of our services are in one big resource pool. Any business that needs additional resources for a short period of time can simply expand and shrink within it. Because the demand period of different businesses is also different, but also the use of off-peak.
“Then there is the flexibility of resource use. For example, for my recommendation system, if additional resources are available in the resource pool, my recommendation system can provide better user experience through more complex calculations. Therefore, when there are no operational activities, we use this part of idle resources to improve the recommendation and retrieval effect. When there is an operational activity, we spit out this resource to the operational activity. This allows us to balance resources across the board, and it should be an automated process with a very low cost.”
Stage 4: Advanced cloud native
In order to continue to reduce costs and improve efficiency, MEG architecture platform will be transformed into cloud native from 2021, and further operations such as Serverless and Function as a Service will be added on the basis of dynamic management.
Before the transformation, the capacity of the whole system is basically fixed, once there is a burst of traffic can only be degraded. The Serverless mechanism monitors traffic in real time and automatically expands capacity in a few minutes if exceptions are detected.
Function as a Service is a direction to improve the r&d efficiency to the extreme. It allows business students to focus only on the logic they want to implement. In terms of architecture, how to split microservices, how to control traffic, how to do capacity management, all the implementation of the underlying system.
Phase 5 declarative architecture
Chuanyu mentioned that ** actually did some things in the advanced cloud native stage, are close to the declarative architecture. ** For example, Function as a Service is a key part of declarative architecture, including the implementation of Serverless mechanism. The ultimate goal is to completely decouple policy from architecture.
“A lot of today’s architectures don’t have much of a problem when they’re first designed. However, with the development of services, the system needs to be reconstructed after running for a period of time. As services change, the system faces different service scenarios. However, the adjustment of the architecture is very complex, usually will break bones, and will involve a lot of business strategy logic migration. We want to separate business from architecture as much as possible, business logic from architectural design. This makes the business description logic as simple as possible, and our system can automatically split functions based on these descriptions, and then send those functions to the system for execution.”
If architecture and business can be separated, the MEG architecture platform becomes very flexible.
Including which nodes, which functions to perform, how to use these functions to do the connection, how to do the combination, all by the automatic deduction system to achieve. In this way, our system will become declarative, which means you declare your business logic on it, and it will automatically deduce what architecture is needed to implement it automatically.
“So of course this is the ultimate ideal state. We need to do many, many more things to get there.”
The above are some key paths for Baidu MEG architecture platform to complete cloud native transformation.
In the following sharing, Chuanyu will also focus on some problems and solutions in the process of service governance, automation, chaos engineering and other aspects.
The original link mp.weixin.qq.com/s/lVzHC5ISr…
Recommended reading
Billions of traffic baidu search in Taiwan, is how to do observable construction?
Billion level traffic search front end, is how to do architecture upgrade?
|Exploration and application of elastic near line computing in Baidu information flow and search business
———- END ———-
Baidu Geek Talk is now online
Interested students can also join “Baidu Geek talk wechat Communication wechat group” ** to continue communication within the group.
If you are interested in other positions in Baidu, please join baidu Geek official Recruitment group ** to keep up with the latest job trends. There is always a JD for you.
Technical dry goods, industry information, online salon, industry conference
Recruitment information · Internal push information · technical books · Baidu surrounding