Jim Zemlin, executive director of the Linux Foundation, said, “I’ve seen two big events in the open source community: the success of Linux and the explosion of Kubernetes and cloud native. Open source is one of the most successful ways to drive global innovation in history, Linux has grown to become the most important software platform in the world, and cloud native is exploding in a new era.”

In 2019, cloud +AI has become the mainstream of the era as the discussion on cloud native architecture gradually heats up. In the emerging cloud native trend, China has made a great contribution: Kubernetes contributors in China rank second in the world, with more than 10% of CNCF members coming from China and 26% of Kubernetes certified service providers coming from China. The concept of cloud native has been widely accepted by domestic developers and has entered the stage of enterprise practice.

Baidu intelligent cloud cloud native team in 2015 began to invest in the direction of Kubernetes, for Kubernetes community open source contribution, Baidu entered the global top 10 in 2019. However, within Baidu, the mainstream services are containerized deployment based on matrix, while Kubernetes + Docker has not been fully applied yet. In the field of enterprise services, the open source ecosystem based on Kubernetes + Docker has become the mainstream technical architecture. Aiphanan cooperated with the cloud native team of the Infrastructure Department to deeply promote the deployment of the company’s internal services based on Kubernetes + Docker, and switched all online traffic of Aiphanan to Kubernetes. After more than 2 weeks’ observation, the operation was stable.

The full text is 4834 words, and the expected reading time is 14 minutes

If you are a r&d student reading this article, you are also planning to upgrade the technology stack of products to the cloud native direction, then you may wonder why Aipan chose Kubernetes. Who will help me with the Kubernates cluster? Is migration a lot of work? What are the benefits? So, here’s an overview of what you might be concerned about.

I. Why does Aipanpan choose Kubernetes

In the direction of ToB, we always emphasize “customer-centered”, “technology serves products, products serve customers”, and Kubernetes is the cornerstone of technology to serve products.

With the deepening and acceleration of Aipanpan’s business, some problems exposed by the original technical framework are becoming more and more obvious. We will elaborate from three aspects:

Fast delivery:

  • How to arrange the online dependency of 200+ modules and complete the online within 1 hour?

  • How to achieve the goal of biweekly iteration and biweekly release?

Maintaining stability and reducing costs:

  • How to achieve the goal of maintaining stability above 4 9 in the long term?

  • How do I reduce pvlost during live?

  • How to achieve full observability and governance of 2000+ service instances?

  • The performance/stability/throughput of some self-developed microservices infrastructure is insufficient. Should we continue to invest more manpower to transform it or try to introduce open source solutions?

Out of the box:

  • How to realize the diversified sales demand of Aipanpan (toB products)?

  • How to scale, replicable deployment and operation, reduce environmental differences and mental costs?

Our answer is to adopt a cloud native architecture to build love on the cloud, reduce duplication of wheels and accelerate customer value point delivery.

Kubernetes + Docker technology has been very mature in the industry, CNCF has a very comprehensive cloud native technology to support our demands for containerization, CICD, application orchestration, monitoring analysis and other aspects, based on the existing open source components and combined with the company’s infrastructure, We can quickly improve the ability of basic technology with less r&d cost to achieve the goal of supporting rapid product iteration.

Ii. Cooperation mode between Aifanfan and Infrastructure Department

The cooperation model is for the Cloud native team of the Infrastructure Division (INF) to provide Kubernetes technology development and support, and for the upgrade and deployment of aiphanan services. Our cooperation can be roughly divided into two stages:

  • Stage 1: Kubernates technology r&d and cluster deployment. The infrastructure cloud native team completed Kubernetes technology r&d and cluster deployment to adapt to the internal network environment.

  • The second stage: Aiphanan service deployment and flow migration, aiphanan completed the deployment of 100+ modules Kubernetes environment; Carry out small traffic according to customer dimension, and gradually complete 100% formal customer traffic migration.

Third, Aipanpan migration Kubernetes scheme

Key issues and solutions for migrating Kubernetes.

3.1 How to deploy multiple types and services efficiently

Difficulties: Challenges of differentiated infrastructure to sustainable delivery

  • Infrastructure: How can infrastructure be operated and maintained to ensure high availability? How to manage a mirror warehouse?

  • Deployment pattern: How to containerize zero-intrusion services? How does the container manage and support different types of applications?

  • Application Management: How to maintain 10+ resources on K8S platform? How can an Application be rapidly deployed and rolled back?

  • CICD Flow: How does CICD flow connect to the new infrastructure? How can CICD stream further improve r&d performance?

Solution: Standardized, automated, cloud-native CICD processes

Noun explanation:

Agile: an enterprise-level continuous integration platform supporting Baidu’s agile development

Archer3: Uniform deployment protocol specification

Research and program design of Kubernetes + Docker technology adapted to the internal network environment, realize the simultaneous deployment of the original platform /Kubernetes, ensure the consistency of services and improve the efficiency of research and development.

In order to achieve the goal of fast delivery, it is necessary to complete the standardization of the whole r&d life cycle first, and develop unified standards and specifications to support multiple types of applications. In order to achieve standardization, Aipanpan Integrated the continuous integration platform and extended the completion of the three core links to achieve a fully automated CICD process.

  • Unified packaging: Create a standard application package based on the unified scaffolding, complete the application package standard through the unified packaging service into the archer3 deployment protocol of the company, can be seamlessly deployed on other platforms.

  • Unified containerization: Move containerized application rules up to avoid scattered containerized rules in various applications, facilitating management and maintenance. The rule file of Docker image build is completely decoupled from the business source code. In the process of defining to CI, different Dockerfile template files are selected to build images according to the application metadata information. The version number of Docker image Tag is based on the difference of pipeline. Identify git commit log and take the first 6 bits for mirroring and source traceability.

  • Unified Application package: Integrate containerized images and manage unified APPLICATION K8S resources based on standardized Helm Chart. Prepare unified Helm Chart Template with version number on CICD side. During application integration, associate the containerized name with image tag in the previous step. On the other hand, with the help of Helm Chart dynamic rendering templates and powerful configuration capabilities, the application can be deployed to different clusters in different forms.

In addition to focusing on the improvement of standardization and automation band efficiency, it also provides unified application package management, covering surrounding resources such as image, K8S resources and environment, to further enhance continuous integration capability.

Benefits: Zero service intrusion, 10.6 minutes on average for compilation and deployment

150+ applications can be upgraded based on pipeline at zero cost, and containerized and unified application package can be deployed. In terms of application coverage, it supports full coverage of application types and one-keysecond rollback strategy, which enables rapid expansion of clusters and environments at zero cost.

3.2 Smooth and stable K8S migration scheme

Difficulties: With the rapid growth of business, how to transfer production flow to K8S platform with low cost and gray scale?

Scheme: unified flow inlet, gray drainage, step – by – step full and monitoring back – cut pocket bottom

Noun explanation:

  • BFE: Baidu unified application layer traffic access and forwarding platform

  • BNS: Baidu name service, which is used for resource locating and IP whitelist maintenance in service interaction. It can also be used for machine list query, service locating, whitelist maintenance, and intelligent database authorization

  • Access Gateway: API gateway for k8S clusters built based on OpenResty

  • BOS: Baidu Object Storage (BOS) Provides stable, secure, efficient, and scalable Storage services

The unified flow entrance of K8S is constructed as access Gateway, which is used as the entrance of external traffic into K8S cluster. Service discovery is carried out in the cluster through K8S Service, and the gray scale proportion is gradually expanded according to the rhythm in a planned way until it turns to full.

Through the unified BFE, the company realizes the control and routing of small-granularity traffic and multi-version traffic, the minute-level traffic rollback scheme, and the gradual migration of official traffic.

This section describes the key steps for migrating traffic to k8S

  1. The K8S deployment pre-release environment is successful

  2. Application deployment k8S and the original platform isomorphic online environment

  3. The K8S Access Gateway is configured to forward all traffic to the original platform. The Access Gateway is authenticated

  4. If the verification fails, debug the system in the K8S environment until the verification passes without affecting the original platform traffic

  5. Adjust the sample configuration of the Access Gateway and deliver the customer ID traffic for the internal test to the K8S cluster

  6. If the verification fails, repeat Step 5 until the verification succeeds

  7. Add the Access Gateway BNS cluster to the gateway BFE

  8. Add forwarding rules and manually seed cookies. If the traffic matching the specified cookie rules is forwarded to the Access Gateway, the authentication succeeds

  9. The BFE of the gateway is switched to the Access Gateway of the K8S. The verification succeeds

  10. If the verification fails, cut the BFE back to the API gateway of the original platform for stop-loss analysis, and repeat Step 7 until the verification succeeds

  11. Observe the system stability until the system is stable. Ensure that the same user traffic is in the same cluster

  12. Adjust the sample configuration to implement small-granularity traffic for tenants

  13. Gradually expand the proportion of small flow until full

Earnings: Line last week stability remained above 99.99%

Complete 150+ applications, 2K+ service interface transformation and full online service transformation within four months.

The deployment architecture is further optimized to centrally converge traffic from multiple domain names and sources and manage traffic in a unified manner.

4. Kubernetes migration benefits

Through the container technology to solve the standardization of resources, for the following service, automation has laid a foundation, at the same time, the industry unified Kubernetes platform, accelerate the upgrade of basic design, specific benefits mainly include:

  • Improved iteration speed: achieved the goal of two iterations and two releases a week. Complete the operation and transformation of Aipanfan 100+ application and its dependent technical components in Kubernetes + Docker environment, and realize the whole process from development, testing, deployment, operation and observation.

  • Rapid construction of monitoring system: Rapid construction of unified product health monitoring/observation platform through Prometheus + Grafana, breaking the existing status of monitoring services based on 5-6 in-house products.

  • Low-cost infrastructure upgrade: Upgrade and implementation of cloud native micro-service infrastructure, including configuration center solution based on Config Map, log collection solution based on EFK/Skywalking, log warning solution based on ElasticAlert, unified traffic entrance based on Kubernetes, This improves system stability and troubleshooting efficiency.

5. Planning of Aipanfan in the direction of Yunyuansheng

  • Expand service types: promote the migration of more types of services to Kubernetes, complete the migration of online communication based on Golang and scheduled tasks based on scripting language to Kubernetes.

  • Release the dividend: High availability based on cloud native architecture, introducing Kubernetes native service registration, discovery mechanism, automatic scaling and self-healing capabilities.

  • High availability service: Multi-region and multi-cluster deployment provides better customer experience for customers in different regions and implements unitary service deployment to improve the disaster recovery capability of the system.

  • Efficient Service governance: The microservice architecture based on Service Mesh can realize the observation of the whole link of the system and provide stronger control ability in Service invocation, retry, fusing, current limiting, degradation and other aspects. Multiple service versions are supported to achieve multi-version release and grayscale release through deployment isolation and tenant isolation.

  • Automated continuous integration: CICD toolchain platform based on Kubernetes, realizing multi-environment isolation, rapid deployment, one-click release, visual operation and maintenance management, etc.

Vi. Planning of the Infrastructure Department in the direction of cloud origin

Build a competitive cloud native application platform product matrix to support more products access.

The stone of another mountain can attack jade. Aipanfan always adheres to the concept of “technology serves products”, embraces open source, and actively introduces mature and advanced technical solutions in the industry to support the development of products. In the direction of cloud native, Aipanfan has taken the first step. In the future, we will continue to make full use of the power of CNCF open source community to continuously improve product research and development efficiency and system stability. At the same time, we also hope to see more products benefit from cloud native technology.

Vii. Conclusion: Embrace cloud origin and accelerate value creation

Cloud native is the concept of cloud application design, which helps enterprises build flexible, reliable, loosely coupled, easily managed and observable application systems, improve delivery efficiency and reduce operation and maintenance complexity, mainly in the following three aspects:

  • The release of dividend of K8S platform gives full play to the advantages of the platform, and implements the capabilities of service autonomy, fault self-healing, rapid flexibility, automatic expansion and reduction into actual production practice, thus reducing the operation and maintenance complexity.

  • Introduce and integrate service Mesh capabilities, build a service governance platform, and provide rich and powerful traffic control and service routing capabilities.

  • Integrate various monitoring indicators, establish a unified monitoring system, integrate Metric, tracing and log to provide one-stop solutions and service capabilities.

Surrounding cloud native system, love will continue on the cloud native direction development, evolution of basic technology innovation new technology system, gradually grinding technology products and services for the team, continue to introduce cloud native technology to support the development of products, adhere to the “technology service” for the product idea to speed up customer value delivery.

About the author:

Chhoho, the head of basic technology of Baidu Aipanfan, has many years of EXPERIENCE in CRM industry architecture, and is good at distributed system, microservice architecture, CICD, Cloud Native and other technical fields.

Recommended reading:

Baidu Library a new generation of document reader! Core technology full analysis!

Explain the application of pretraining model in the first stage of information retrieval

Fast clip – help the strength of smart clip improve efficiency practice

———- END ———-

Baidu said Geek

Baidu official technology public number online!

Technical dry goods, industry information, online salon, industry conference

Recruitment information · Internal push information · technical books · Baidu surrounding

Welcome to your attention