1. Project migration background
1.1 Why do we need to ground on “Tai Sui”?
At present, k8S has been used for maintenance and management of the company’s test environment, UAT environment and production environment. Most of the projects have been containerized and have been running smoothly online for a long time. After containerizing large and small projects, we gradually realized unified management of testing, UAT, release tools in production environment and CICD process. We also developed an internal release audit platform based on K8S and connected Jira and other project management tools.
In the research platform Release, automatically associated development progress and Release version of the project, the most important thing is that it can control the Release authority, unified hair version of the tool and the hair version of the model, and support the one-click Release multiple modules of the project, and also includes the application failed to Release the unification of the rollback and the rollback of a single application.
Because the project from the beginning has been using GitRunner version, and based on the virtual machine deployment, so has been no integration and arrival version verification platform, but because the project is important, and involves more service and machine, so have to put this project on the container and unified release tools to better adapt to the environment of the company, And better prepare for the next generation of cloud computing.
1.2 Why abandon Git Runner?
First of all, let’s take a look at the Git Runner release page. Although it looks very simple and clean, it will inevitably encounter some problems.
1.2.1 Problem of multi-branch parallel development
When multiple branches are developed in parallel or when there are many branches that can be shipped to production, it is easy to misspot or serialize during the manual deployment phase, although this is unlikely.
If Git Runner is created on a virtual machine, it is likely to cause a build queue. If Git Runner is created on a virtual machine, it is likely to cause a build queue. Of course, this queuing problem can be solved.
1.2.2 Configuration and Maintenance of Multiple Micro Services
Secondly, if a project is slightly larger, it is not very convenient to maintain. For example, the project to be migrated includes a front-end and more than 20 business applications, plus nearly 30 services including Zuul, ConfigServer and Eureka. Each service corresponds to a Git repository, and each service has many branches under development at the same time. If a machine that wants to upgrade GitLab CI scripts or microservices wants to add nodes, this can be a tedious task.
1.2.3 Security Issues
Finally, there is a security problem. GitLab’S CI scripts are generally built into the code repository, which means that anyone with Push or Merge permission can modify CI scripts at will, which will lead to unexpected results and threaten server and business security. For release, It’s possible that any developer can click the release button, which is always a security risk.
But that doesn’t mean Git Runner isn’t a recommended tool. The new version of GitLab has built-in Auto DevOps and integrated Kubernetes. But maybe for us, there are not many projects using Git Runner for release, so we want to unify the release tool and the MANAGEMENT of CI scripts, so maybe other CI tools are more suitable.
1.3 Why containerization?
1.3.1 Port Conflict
Container before this project using the virtual machine deployment, each virtual machine cross launched two or three services, it will encounter a problem, is the question about which port conflicts, in the project to join a new application, need to consider the server port conflict between, also consider each micro service port, not because the use of virtual machine deployment applications, Some machine nodes may fail and applications may need to be migrated manually. If some microservice ports are the same, the migration process may be blocked.
In addition, ports can be easy to maintain when a project has only a few applications, like this one, which involves more than 30 microservices, which can be a pain. With container deployment, each container is isolated from the other, and all applications can use the same port, so you don’t need to worry about ports.
1.3.2 Program health issues
Most of the people who have used Java programs have encountered the process of fake death, such as port is clearly through, but the request is not processed, this is a process of fake death phenomenon. And when we are in the use of virtual machine deployment, often cannot do health check is very good, perhaps in a virtual machine didn’t do the interface level above the health check, it will cause program suspended animation automatic processing problems, and on the virtual machine to do some interface level health inspection and processing operations is not a simple thing, is also a boring thing, This is especially painful when a project has too many microservices and inconsistent health check interfaces.
On K8S, however, the built-in Read and Live probes are extremely simple to handle the above problems. As shown in the figure, we can see that there are currently three kinds of health checks supported:
- TcpSocket: port health check
- Exec: Based on the return value of the specified command
- HttpGet: interface-level health check
The httpGet interface health check also supports custom Host name, request header, check path, HTTP or HTTPS configuration. As you can see, using the k8S built-in health check saves us a lot of work and a lot of annoying scripts to maintain.
1.3.3 Fault Recovery
When applications are deployed on virtual machines (VVMS), single-node applications may be unavailable due to host faults, or multi-node applications may experience service delays due to heavy pressure due to unavailable copies. However, the host machine can’t recover quickly and may need to manually add nodes or new servers to solve this problem, which can be a long and painful process. Because you need to prepare the dependency environment before you can deploy your application, and sometimes you may need to change the CI script…
With K8S, we don’t have to worry about this kind of problem, all the fault recovery, disaster recovery mechanism is taken care of by the powerful K8S, you can go to have a cup of coffee, or you just turn on the computer to deal with the problem, everything is back to normal.
1.3.4 Other minor problems
Of course, K8S brings us convenience and solves far more problems than the above mentioned, container image helps us solve the problem of environment dependence, service choreography helps us solve the problem of fault disaster recovery, we can use K8S package management tool to create a set of new environment with one click, We can use k8s service discovery let developers don’t need to pay attention to the development of the network part, we can use k8s access control for operations staff without having to manage the permissions for each server, and we can use k8s powerful applications release strategy let we don’t need too much to consider how to achieve zero downtime rollback, published applications and application, etc. The convenience of all this is quietly changing our behavior.
2. Migration plan
2.1 Blue-green migration
First, take a look at the architecture before the migration
As with most SpringCloud architectures, NodeJS is used as the front end, Eureka for service discovery, Zuul for route distribution, and ConfigServer as the configuration center. This architecture is also the most common architecture of SpringCloud in the enterprise. No additional components are used, so we did not think too much about the first migration. We still followed the solution used in other projects, that is, we created a new environment on K8S (middleware is not involved in this migration), that is, a container environment. Configure the same domain name and add hosts for parsing to test. If there is no problem, switch the domain name to complete the migration. This is the simplest and most common way, which is similar to the blue-green deployment of program release. At this time, the corresponding architecture diagram of a new environment in K8S is as follows:
When testing, the project in parallel with two sets of environment, a virtual machine environment, the environment of a container, the container traffic environment only receive testers, two sets of connected to the same set of middleware service environment, because most of the other project is according to this way of migration, and the project also had the same process in the test environment, did not appear what problem, Therefore, it is also believed that there will be no problems in this project. However, the reality is always different from the expectation. In the testing process, the coexistence of two sets of environments led to some problems in production data. As the container environment did not pass the integrity test, and there was no forced domain name switch, all the container problems were recovered after the emergency shutdown. Due to time constraints, we did not carefully investigate the problem and only fixed some data. Later, we thought that it might be caused by inconsistency between the master branch of some microservices and the production code in the migration process, but of course, it may not be so simple. To prevent such problems from happening again, you have to modify the migration scheme.
2.2 Gray migration
Due to some problems with the above migration scheme, we decided a new scheme, which is a little more troublesome than the last one, and adopted the migration of microservices one by one to K8S, similar to the grayscale distribution of application version.
When migrating an application, ensure that the code of the container environment is the same as that of the virtual machine environment. During the migration, the micro-service uses domain name registration. In other words, each micro-service is configured with an internal domain name and registered to Eureka through the domain name instead of using the IP and port of the container (because the INTERNAL IP of K8S is not connected to the VIRTUAL machine). At this time, the environment is as shown below:
K8s points to ServiceC. When ServiceC registers with Eureka, it changes its address to this domain name (host IP+ port by default). Then other applications use this address to invoke ServiceC. After the ServiceC test is complete, the ServiceC in the VM is offline and the final architecture is shown in the figure below:
With the exception of Zuul, front-end UI and Eureka, the other services are migrated to K8S in a grayscale manner, which is more complex than the blue-green form, requiring the creation of separate services and domain names for each microservice, which need to be deleted after the migration. At this point, all services except Eureka have been deployed on K8S, and the migration of Eureka is more detailed.
2.3 Eureka migration
At this point, there are no other problems with service access, and all services other than Eureka are already deployed on K8S, and Eureka’s transitional migration design may have more problems. Because we cannot directly deploy a set of highly available Eureka cluster on K8S, and then directly change the micro-service registration address in the ConfigServer to the Eureka address in K8S, because the two Eureka clusters are independent zones and registration information will not be shared. This will cause registration information to be lost during configuration changes, in which case the schema diagram may have the following situation:
In other words, in the process of configuration replacement, a ServiceA may register with the previous Eureka, and a ServiceB may register with the Eureka in K8S. As a result, the ServiceA cannot find the ServiceB, and vice versa.
Therefore, after k8S establishes Eureka cluster, it is necessary to configure a temporary domain name for each Eureka instance, and then change the zone configuration of the previous Eureka cluster and the Eureka cluster in K8S, so that the Eureka in K8S and the Eureka in the virtual machine form a new cluster. In this way, the registration information will be synchronized, no matter the service is registered in Eureka, the architecture diagram is as follows (at this time, all services are registered in the original Eureka cluster) :
The next thing you need to do is change the configuration of the microservice in three places:
- The address of micro-service registered to Eureka is more container IP and port, no longer using domain name registration, because at this time the micro-service has been in K8S, directly through the internal Pod IP can be connected;
- Change the registered Eureka address to the service address of k8S Eureka. Eureka is deployed using StatefulSet and can be connected directly through eureka-0/1/2. Eureka-headless-svc.
- After all microservices are migrated, change the zone of the Eureka cluster in k8S to eureka-0/1/2. Eureka-headless-svc, and delete the Service and domain name of other microservices.
The final architecture diagram is shown below:
3. Summary
In order to ensure the availability of services, we have no choice but to use gray scale for migration, which is much more troublesome than the blue-green way, and there are also a lot of problems to consider. In the program without any problems under the premise, or suggest the use of blue and green way to migrate, not only encountered fewer problems, migration is more convenient and fast. Of course, the grayscale approach may be more secure for large projects or projects that cannot be interrupted, because switching all at once may leave out areas that need to be tested. Of course, either way, the containerization of the application and migration to Kubernetes are more important, after all, cloud computing is the future, and Kubernetes is the future of cloud computing.
The original link: www.cnblogs.com/dukuan/p/13…