In this paper, byNetease cloudRelease.

As a big winner in the competition for container cluster management technology, Kubernetes has been closely associated with microservices, and enterprises adopting Kubernetes have often started to explore microservices architecture. However, the problems faced by different enterprises in different stages of micro-service practice are very different, which is destined to produce bifurcation in the technology route. How to choose the right technology is the first problem that every team practicing microservices faces. Netease Cloud is the first batch of heavy users of Kubernetes, and has solved many challenges in different business scenarios. In this paper, Liu Chao, chief solution architect of netease Cloud, has sorted out the advanced road to build a micro-service system based on Kubernetes.

Rapid growth stage, the necessity of microservitization

The development of a product can usually be divided into cold start-up phase, high growth phase and mature phase.

In the cold launch phase of the product, the requirement is to validate the business with the simplest architecture. Taking netease Kaola Overseas Shopping (hereinafter referred to as “netease Kaola”) as an example, the initial architecture design goal is to quickly start up and verify the product direction. The architecture includes four aspects: online, cache, offline and management services, namely, the general e-commerce platform plus the necessary purchase-sale-inventory system for cross-border e-commerce. Virtual machines (VMS) managed by Oracle database and OpenStack are adopted without considerations such as high concurrency.

Product rapid growth stage, the business scale expanding gradually, as product complexity increases, the enterprise needs to solve problems such as rapid iteration, high reliability and high availability, is a natural choice as a service, a monomer architecture split into smaller modules, and following conway’s law, in 5 to 9 a small team to adapt to changes in the architecture. Take kaola as an example. In the rapid growth stage, Kaola gradually evolved various new modules, such as independent payment module, warehouse module, third-party merchant module, push module, etc., and built service discovery function based on the Dubbo framework to support the mutual invocation of each module.

Servitization mainly solves the problem of change. In the process of evolution of the whole structure and various modules are under explosive growth, such as overseas online shopping, proprietary and third-party merchants of supply chain, Web, APP, H5, timed purchase, seconds kill and pre-sales activities page, as well as warehouse and logistics systems, payment systems, such as docking, tight coupling is extremely important, bloated, influence on the speed of iteration, Separate online is more conducive to meet the needs of business development. In the rapid growth stage, Netease Kaola first split vertically in terms of homepage, activity page, coupon, payment and other dimensions, and then gradually evolved into more than 100 interrelated modules. The change frequency increased from 2 times a day to more than 1000 times a day, and the product quality improved by 52%.

Advantages and challenges of containerization

When broken down into a large number of small modules, there are many new challenges in working with the virtual machine and the service-oriented architecture, hence the need for containerization.

Liu Chao explained that before splitting, we should first solve the problem of “integration”, that is, we need to ensure that the function is the original function, the code quality is the original code quality, and no new bugs will be introduced. Microservitization requires continuous integration from the very beginning, and containers are great tools for continuous integration, from code submission to automated testing and release. Containerization changes the development process, moving the environmental delivery process from operations to developers.

In the case of complex architectures, such as more than 100 modules, plus various replicas, all environments are handled by an operations team, which is not only heavy workload, but also error-prone, but this is the mode of using virtual machines. And if you write a Dockerflie into the code warehouse, by the developer to consider the deployment of the application configuration environment, permissions and other problems after the completion of development, including the deployment of the test environment, the deployment of the joint adjustment environment, the deployment of the production environment, the problem is well solved. This is the process change that containerization brings.

However, this shift involves the willingness of developers to learn container technology. Liu chao recommended solution, is to use a mirror in the form of a hierarchical, namely the internal environment consists of the operating system and system tools of mirror is done by the operations staff, middle tier environment image of work by the core developers, ordinary developers only need to throw the jar or war to the corresponding path, this will greatly reduce enterprise organization container obstacles.

Scenario 1: Kubernetes + Docker + VM + Host Network

The first scenario is to use Kubernetes to manage virtual machines. The network and storage of containers will face various types of models. If an enterprise is not familiar with the network and storage of containers, it can use containers as a continuous integration tool. Embedding a container into a VIRTUAL machine is equivalent to using container images instead of script deployment. In this way, two problems need to be solved: one is the PROBLEM of IP retention and the other is the problem of disk retention. The original design of virtual machines was based on stateful design, and IP and Volume were considered to remain unchanged. When containers are only used as continuous integration tools, teams may not be able to break this habit.

One solution is to implement a stateful container to maintain IP. When a node hangs up, the restarted virtual machine and container can still use the original IP. The other is to Mount the image of the Docker container layer by layer to the outside Volume. All Docker images and volumes are actually mounted on Ceph, so data is not lost. This is very similar to using VM, which can be docker-enabled and microservified without changing user habits. Teams with relatively high pressure using Kubernetes can be approached in this way.

Scenario 2: Kubernetes + Docker + PM + Bridge Network

In the second scenario, the enterprise does not use virtual machines (VMS), but has some applications deployed on the PM (note: IN this article, THE PM refers to physical machines) and wants to migrate some applications to containers. At this point, no matter whether the physical machine is nested with virtual machines or not, a Bridge Network is directly created and the physical Network card is also inserted. When the Docker Network card is connected to the Bridge, the whole Network is flat and the container and the physical machine next to the container all use the same specified Network segment. After the Network flattens out, teams using Dubbo can also smoothly migrate some applications deployed on physical machines to containers, which would be awkward for load balancing (LB) or NAT without the Bridge Network. Because Kubernetes layer maintainers often have a hard time persuading Dubbo layer developers to change the way applications are developed.

Using Bridge Network, Kubernetes Network configuration is very simple, using CNI mode. If there is customization to meet the requirements of the application layer, you can refer to the manual configuration of Docker Run to develop your own CNI plug-in. The general process is to create a network bridge (if it does not exist), obtain a Namespace, configure the Veth pair and add it to the Namespace, and then obtain the IP address, network, and route.

Scenario 3: Kubernetes + Docker + PM + SR-IOv Network

Bridge can meet the requirements of common Java application deployment. However, sr-IOV can be used instead of Bridge to solve problems in applications requiring higher performance, such as caching in the case of e-commerce promotion. Sr-iov can solve the problem of large bandwidth but no PPS (large packets or a large number of small packets). However, sr-IOV nics need to be purchased, which costs a lot.

High availability design essentials

Stateless. The first thing you could do with continuous integration was to split your application into Stateful and Stateless parts, and make most of your applications Stateless to better accommodate elastic scaling. Even though Kubernetes already supports the deployment of stateful applications, Liu chao recommends making the application layer as stateless as possible, so that stateful applications are clustered in a small number of clusters. The most important things to be stateful are databases and caches. Typically, in-memory data is stored in caches and persistent data is stored in databases.

Distributed database. Database high availability, netease cloud is DDB (distributed database) scheme, many sets of main equipment and load balancing based on MySQL do depots table, netease cloud RDS based on their own MySQL kernel optimization, to achieve the main switch for not losing data, is able to support the container, the stateful container after hanging up, restart a container, As long as you do a good preordering reset and idempotence, there will be no business problems. Therefore, the active/standby switchover of netease Cloud RDS is also transitioned from VMS to containers.

The cache. High-concurrency applications require caches at each level, keeping customer needs as far ahead of the curve as possible, and throughput is much higher. However, the cache does not have the same persistence mechanism as the database, and its high availability and cross-room need to do double write, because the cache is kept in memory, there is no hanging, difficult to repair. Other components, such as ZooKeeper, Kafka, message queue, and HBase, have their own high availability mechanisms. Therefore, a growing application should be split into two distinct parts, one stateless, and the other stateful ones placed in components with their own highly available mechanisms.

Mature stage architecture

In the mature stage of products, the main problems to be solved are how to improve reliability and availability through service governance and system operation and maintenance automation, how to efficiently complete complex collaboration of large projects, and how to sort out functions and deepen user experience. Taking Netease Kaola, which is undergoing full service, as an example, the number of projects increased by more than 20 times and the scale of application and storage cluster expanded by 5 times during The Double 11 of 2017. Needless to say, the challenges are enormous. Liu Chao emphasized two points for the mature stage architecture design:

Immutable infrastructure: Using Kubernetes container technology can not follow the operation mode of the virtual machine era, but should adopt immutable infrastructure, that is, all changes should be reflected in the change of Git, modify the environment is to modify the Dockerfile, modify the configuration file is also the change of code level, the deployment of the whole environment, When code is merged, scripts that are automatically deployed by the container are triggered, which is a good way to keep the environment consistent. If large-scale nodes are deployed manually, errors are easy to occur, but troubleshooting is difficult. So immutable infrastructure is very important.

IaC (Infrastructure as Code) Deployment and Expansion: Any changes, such as upgrading 5 applications, the version numbers of these 5 applications are configured in this repository. After the code is committed, automatic deployment will be triggered. If problems are found, it is easy to roll back, just revert the code back. Subsequent processes are automatically triggered. If you rely on writing Yaml files, frequent upgrades and poor version control make it easy to roll back errors.

Scenario 4: Kubernetes+Docker+PM+Overlay Network

In the mature stage, the mode of Kubernetes+Docker+PM+Overlay Network is usually used. Once an enterprise begins to use Overlay Network, it will basically use physical machines; otherwise, two layers of Overlay will appear when virtual machines are used. Flannel, Calico, Romana and Weave are all available. Flannel performance is getting better and better.

Scenario 5: Deep integration between Kubernetes and IaaS layers

The way of netease Cloud is the deep integration of Kubernetes and IaaS to realize dynamic resource expansion. At present, the cluster scheduling scale supports 30,000 + nodes. Under this scale, the first thing to be solved is dynamic resource creation optimization, so as to meet the design of fine resource utilization and cost optimization. At the same time, whether the creation of a virtual machine or container, the creation of the application is transparent, that is to say, the application only needs to clear a module to be three or five nodes, don’t need to pipe Docker is to become the number of nodes, the nodes should be placed where, virtual machine and the physical machine whether there are issues such as resources, follow-up actions are linked.

To realize dynamic Resource creation, netease Cloud transformed the Kubernetes creation process, mainly listening to Pod creation events. Resource Controller determines whether there are enough Volume resources and Network resources, Schedule determines whether there are enough Node resources. If yes, the application is directly bound. If no, the application is dynamically bound and delivered by Kubernetes. Add resources, only the application layer and machine room two layers, machine room only need to add physical machine to the IaaS layer, do not need to tube whether there are Kubernetes above, virtual machine creation is all dynamic, application layer only application layer things, the middle is transparent.

Secondly, network optimization. Most containers run on VMS. In addition, the netease Cloud also provides bare – machine containers that use SR-IOV network adapters for caching and distributed databases that require higher performance. Most applications can scale horizontally, again in IaaS. However, netease Cloud hopes that the network card in the container can also be seen by the OVS on the outermost VIRTUAL machine, that is, there is only one Overlay. There is a Bridge in the VIRTUAL machine, but if it is not needed, it can be directly connected to the OVS on the outside. In addition, there is a management network, and the same Kubernetes manages the cross-tenant. Only one Overlay means that there is no secondary virtualization. Meanwhile, applications originally deployed in virtual machines are migrated to containers. The networks of virtual machines and containers are managed through OVS, and Dubbo is used for service discovery. In fact, OpenStack has a CNI plug-in that adopts a similar approach. It works with Neutron to connect VIF to external OVS.

summary

Internal and external customer Kubernetes netease cloud service, this paper practical experience, summed up the product rapid growth and mature period using Kubernetes container technology to realize the five kinds of micro service architecture application scenarios, have been put forward for the different challenges, easy to implement solutions, and introduces the netease cloud use the special optimization method, hope to inspire the reader.


Understand netease Cloud:

The official website of netease Cloud is www.163yun.com/

New user package: www.163yun.com/gift

Netease Cloud community: sq.163yun.com/