Wang Wei senior Architect of Tencent Cloud CODING DevOps, Head of Nocalhost Open Source Project, AMBASSADOR of CNCF
This article is based on a speech delivered by Wang Wei, senior architect of Tencent Cloud CODING DevOps, on “Nocalhost – five-star Cloud native Development Experience” in the offline activity “Tencent Cloud Native Technology Open Day” held in Shenzhen on April 10, 2021.
Follow the “Tencent Cloud Native” public account and reply to “Cloud native Technology” in the background, you can get all the lecturers’ PPT of the Open Day of Cloud Native on April 10.
Hello everyone, today’s topic to share with you is cloud native development environment Nocalhost. Previously shared K8s or containerization in the application has brought us some operational or publishing benefits, such as rolling, grayscale, A/B Test, some operational capabilities. In fact, we will find that after containerizing K8s and embracing microservices in our business applications, we will encounter a very difficult problem: how do we develop it locally?
I believe that most of the front-line development students should be relatively deep experience, so today we share the topic is closely related to cloud native development.
This sharing will mainly introduce you to these modules: The first one is why Nocalhost is born and why we do this project. The second is the current mainstream cloud native background development mode. The third is that we do open source products, of course, we need to know what we ultimately want it to be, for example, K8s is currently a cloud native standard. The fourth is the implementation principle of Nocalhost. The fifth is component composition, which components Nocalhost consists of. Sixth, we will show you how to use Nocalhost to develop Istio’s official demo Bookinfo.
Why Nocalhost was born
The first is why Nocalhost was born, and why is cloud native, microservice, distributed application development difficult? First look at the figure on the left, the figure on the left from the basic monomer architecture, local development experience is very good, for example, like the popular PHP, we need to modify the code locally when developing PHP, refresh the page can immediately see the effect, this is the previous traditional development mode or monomer development mode. It is less components, personnel or scale is relatively small, environment construction or operation and maintenance is relatively simple.
As the scale or application scenario becomes more complex, we embrace microservices and K8s containerization, so at this point we break down our single application into a huge microservice application, which may be made up of dozens or hundreds of microservice components. From the figure on the right, we can see that the calls between microservices and microservices are very complex, meaning how can a particular microservice component be developed locally? A common pain point is that I need to run all my microservices locally. If the microservices are very large, I may need to prepare a very strong local development machine, maybe 64GB of ram, I7 or I9, etc., to get the business system development done.
Perhaps the biggest problem with embracing microservices is the difficulty of setting up the development environment and the overall development and debugging.
From the development of the monomer architecture self-test, every time we write one line of code or a few lines of code, a dozen lines of code, we want to see the effect, under the architecture of monomer is a simple, rapid, but in the container, the K8s this environment, actually the whole development of self-test cycle is very slow, some common development way we may be after I write a line of code, Then I will run the Docker build and rebuild an image. I may need docker push to push the image to my warehouse, modify the load version of cluster work, and wait for pod restart to see the latest effect. The entire development self-test cycle is very slow, perhaps five, ten or more minutes.
In the development environment management, there will also be some problems, that is, in the single architecture of the development environment is very simple, prepare a configuration does not need so high development machine, I can do business development. However, in the context of microservices, some teams may adopt a combination of local and remote, and may encounter the problem of confusing the environment management of different versions, or even waiting for other development to run out of the environment before I can use it next.
As mentioned earlier, we have compiled a simple graph based on our current mainstream development approach, focusing on the development approach on the far left, where all services run locally. It is found to have the best coding and self-testing feedback loop experience. Development test environment, maintenance cost, horizontal capacity expansion, these are actually very weak. And, of course, we use K8s and container, most the right side of all services are running in the cloud, besides the bad experience coding self-test returns, feedback loops, the ability to bring us or other income is very good, especially the we mentioned at the operational level of the basic can achieve service this self-healing and automatic enlarge shrinks capacity and so on.
We got to the right-hand side of the graph, and for the Nocalhost project, we wanted to see if it was possible to find a way to combine all the advantages of the right-hand side of the development approach, and then put the five star advantage of the coding self-test feedback loop experience of all the left-hand side services running locally in the right-hand side, This is the problem that Nocalhost wants to solve or wants to solve.
Nocalhost vision
Nocalhost’s vision, our product’s vision is to bring cloud native development back to raw and simple. In other words, it’s as simple as developing a standalone application locally, and you don’t need to think about the background and the background of the application.
As we said, my traditional cloud native development might take five or even ten minutes of experience, but for Nocalhost, we can do the whole process in seconds. From this chaotic state of the development environment, Nocalhost now provides an orderly environment management. The development environment provided by Nocalhost does not provide computing resources, that is to say, Nocalhost does not provide K8s cluster. Instead, it uses existing cluster resources and allocates them to each developer through Namespace for development. If combined with some flexible capabilities of cloud computing, It can even be distributed on demand and destroyed.
Nocalhost puts the business container into a development mode
Fourth, I will share with you the principle of Nocalhost, why it can achieve such efficient development in containerized development scenarios, its main core point is that Nocalhost can put business containers into a development mode.
Nocalhost has several core concepts. The first is an application, which may consist of many microservices, such as Bookinfo, which is a Kubernetes application. Our company’s services are made up of more than 100 or nearly 200 microservices, so this is also a Kubernetes application. In fact, there is no application of this concept in Kubernetes system, so now application areas, such as the well-known OAM standard and Kubevela, have the concept of repackaging an application, which is complementary to K8s capabilities, because we all know that applications are composed of several workloads. In fact, this is only known to us, K8s cluster is not known, so this is the application. Of course, there are many types of applications, including Manifest applications, Helm applications, and Kustomize applications, which are supported in Nocalhost.
The second is clustering, where computing resources need to be provided by users. In fact, this provision can be said to only provide a cluster of resources. As for assignment to each developer, it is namespace-based and is used to deploy each developer’s development environment and development time.
The third one is developers. Developers also have a concept in Nocalhost. There may be many developers under my enterprise.
The last one is development space, which has already been covered.
Before moving on to the next topic, let’s initialize the Nocalhost environment with one click. Nocalhost provides a CLI tool for quick initialization of the entire development environment, with built-in applications such as BookInfo Quick Experience. This one command will do.
Through the plug-in side, developers can click to pull up their own development environment, that is, click on the right of the small rocket can be developed, you can pull up a complete set of development environment.
The implementation principle is to put a business container into development mode, or even different developers working together in the same development environment, for example, my business code needs to rely on your new interface, so there will be a collaborative development process. For Nocalhost, it is possible to have two developers sharing developers in the same Namespace and doing collaborative development work.
What happens when you go into development mode? First, we will replace the image of the entire business container. For example, we will disable things related to production, operation and maintenance, HPA, Liveness&Readiness, and make its instance copy count to 1. Most importantly, we will also add a Sidecar to the business container during development. Thus, the entire POD would have two containers, the first one providing Runtime capability support for business code, and the second one for file synchronization, which would share storage by sharing volumes.
With this in place, our native code can actually be synchronized to a POD in a remote development environment via Sidecar’s ability. Combined with Nocalhost providing a very convenient mechanism to obtain the Terminal of the remote container, it is possible to run the business code directly in the container.
Why can business code run in a container? Since this container is a true business container, it retains all the enVs you assigned to it, whether you mount them via configMap or declare them in Deployment. There may be businesses that do magic in InitContainer, and these things are always retained. This means that it is perfectly OK to RUN business code in the container, across the entire cluster network, at the network level, and at the traffic level.
In addition to providing direct access to the business container, Nocalhost will also do a dependency processing logic. For example, in the Kubernetes environment, if you deploy an application, the application may be composed of two hundred microservices, you find that many applications have dependencies in the business logic, but K8s does not provide us with this dependency handling, many pods will restart after deployment. Because some businesses may depend on DB, MQ, and base components. If it can’t find these services, it will hang up, which means that when a huge microservice is actually pulled up, the POD will be restarted N times. Because of the time and frequency that K8s restarts these pods, it is a slower and slower algorithm, so it will take you a long time to pull up the entire development environment. For example, our internal system has about 200 micro services. If we only deploy the whole business in a very rough way for about 30 minutes, we will find that many pods have been restarted dozens of times.
This is another capability that Nocalhost provides, namely that business parties can provide some dependent declarative configuration to Nocalhost, and Nocalhost will have an access controller at deployment time. We intercept all the workloads coming into the cluster and inject InitContainer into those services based on the service dependencies you declared. InitContainer is just waiting for the dependent services to start and then I start myself. It’s a waiting process. This is intercepted by the Nocalhost Agent and injected automatically, so this is the second great capability that Nocalhost provides.
One of the components of Nocalhost, which I’ve talked about in general, is the CLI tool, which is a tool for developers to install on their local computer. Of course, there are many commands, and you can use CLI tools to develop your workload, but the commands are very complex.
There’s also a Server, which has a couple of big functions. One is that I can create my apps, developers, users, all of my developers.
It is troublesome to use CLI tools, so we provide a Plugin, that is, IDE plug-ins, now can be used directly, so you only need to install, for developers only need to install CLI tools and IDE plug-ins can achieve a very simple cloud native development experience. The advantage of plug-ins is that because the IDE is close to the developer, it is easiest to navigate the development environment in plug-ins.
The last one is the Agent, which is also deployed in a cluster in the form of Deployment, all automatically of course.
In the previous tutorial, we realized that the development experience under Nocalhost is a collection of all the applications and all the development environment runs on the remote side, and the coding and self-testing feedback loop is a very good experience.
All of our CODING DevOps have been used to achieve the effect that new employees can independently submit business codes on the day they enter the company. This might have been difficult in the past, meaning that it would take a new employee a week or so to understand what the development environment was like, what the business systems were like, and how I was going to fit into the development environment. In the real building, it may start from the second week. In the process, we will encounter various problems, and we may have our own brothers or mentors to help us solve them. Under the original business development mode, it is not efficient in terms of efficiency. Within CODING, all applications are divided according to business lines, such as CI operating its own minimal CI applications, integrated applications such as microservices, and minimal applications. The CD also runs its own minimal microservice development environment.
For example, if I receive a request that I need to develop a component for login or the entire message notification, the only thing the new employee needs to know is which component I need to develop. When you hit Develop, it will automatically clone the code for you, open it automatically, and even configure how your application works in Nocalhost. That is, it doesn’t even need to know what your application’s running command is. Internally, Nocalhost currently supports the daily development work of our nearly 100 developers, so we use a centralized development environment management. In other words, a single operations team manages all of these development resources, and the cluster is a huge pool of resources.
Currently Nocalhost is open source and uses the Open source protocol of Apache. You can find our project on GitHub.
The overall RodaMap, current version 0.1.0, is a very good development experience in K8s. Of course, there is no rule out supporting other environments in the future.
Dependency analysis, on-demand startup, is also a very big pain point for development right now. The current disadvantage of Nocalhost is that each developer has to run a full development environment, which means that it contains all the microservice components, but in practice you may not need to start a full development environment to develop a microservice component. So we’re going to do dependency analysis and start up the components that your microservice component depends on as needed. Finally, we will implement a shared development environment. What does that mean? This sharing means that I may share some of the base components. For example, each of our development students has pulled up all of the microservice components, and some of the base components are usually not updated or are basically stable. The next goal of Nocalhost is to share these components together, to do some traffic forwarding with the Service Mesh technology and even to realize that the basic components are shared, which can greatly reduce some resources needed by the whole development environment.
Link of QA
Q: I would like to ask how to promote this development mode from scratch in a r&d team of over 100 people, because some of them are not familiar with Docker and K8s knowledge?
Wang Wei: The core problem Nocalhost needs to solve is actually one of them. You can see that when I was developing the Bookinfo microservice component, I didn’t run any kubectl or Docker-related commands, which means that knowledge is already hidden from developers. For example, if you need to see a workload, you can use kubectl get deploment to do so. If you are in the traditional development mode, you may need to use Kubectl Edit to modify the mirrored version of your workload. For traditional development, you need to be at least entry-level or know how to use kubectl and Docker commands. If you develop with Nocalhost, you will find that there is very little knowledge involved, all you need to do is click on the hammer and develop it.
Q: In the basic environment, such as K8s and Helm writing, are the operations team responsible for these?
Wang Wei: Actually, it’s not an operation and maintenance team. We have an infrastructure business team to maintain the basic environment. Of course, when it comes to each microservice you have to update, that’s when you update the baseline environment, and that’s the responsibility of each business team. In fact, we have encountered this problem before. It is very difficult to maintain a uniform set of Helm or Manifest. For example, my micro service may be updated by any department, so that when I try to pull it up, I find all kinds of errors, it does not pull up at all, or it is an old version. The core problem here is that the set of things that you maintain is not used very often, so even if it is broken, it is very difficult to be found or it takes a long time to be found, and then it is not maintained. This problem can be solved if Nocalhost is used for centralized application management. This is because pulling up the deployment environment and entering development is very frequent. As a team, we found that with instrumental help, you can find a virtuous circle within the team. That is to say, when I use Nocalhost to develop, I will find a component RUN does not work. This component may be in my charge or I maintain, I will update it, or there are some new features, I will update it, because I also rely on this set of things to develop.
Question: Suppose we set up such an environment now, new people come in, Nocalhost service is down, will it cause the whole company to not use? Is there a fault tolerance scheme?
Wang Wei: That’s right. First of all, the answer is yes, just like the company adopted K8s, the cluster is suspended, it will cause your business can not RUN, will stop the state. In fact, the entire Nocalhost is deployed in a K8s environment, so you can configure some HPA or something for Nocalhost. Of course, Nocalhost itself is highly available, but depends on the K8s cluster stability you provide. If you build your own cluster, the stability is probably not as high as TKE in terms of probability.
Question: If I have ABC service and need to develop two services, and AB two services depend on C service, will I have these problems online?
Wang Wei: You just mentioned these problems. The development mode is partly run locally and partly run remotely. However, Nocalhost’s development model is to deploy a complete set of your business applications and microservices components in your own development space, so this is not possible because you are using a separate development environment or completely incompatible with others.
Question: Hello, I want to ask A specific question. Suppose A service AB is iterating and B is iterating, can cross-namespace intermodulation be implemented? If we can achieve what we said in the future, one is that sharing resources can avoid the problem of multiple copies. Will your Namespace expand? About 100 or 200 employees, the cluster is very large, if it is a single cluster may have the problem of expansion.
Wang Wei: The first question is divided into two questions. First, you just asked whether cross-namespace is possible. This has been implemented and is being done. No matter how many namespaces you have, you can develop them under Nocalhost. This student is very perceptibly aware of one of the drawbacks of Nocalhost right now, for example, if I want to develop a Prometheus component, which is a component under a fixed Namespace, it’s ok to develop with Nocalhost, You need to use Nocalhost’s CLI tool, but it’s too high a barrier to use, so we’ll support this cluster-level at the plug-in level next. No matter what permissions you provide, either administrator or multiple Namespace permissions, you can develop under different namespaces. That’s the first question.
Question: Can we solve the first problem thoroughly through the future service grid, so that everyone will be a K8s cluster like you said, and the problem of parallel iterative development can be solved.
Wang Wei: Yes, it is possible to use the service grid solution, and it can be completely solved. However, considering the technical difficulty of the current implementation, Nocalhost may need to make an independent control surface by itself, so it is very difficult to implement it. Basically, it may have to do secondary development based on Istio. We have a long-term plan for this, but at the moment, maybe our team thinks cloud resources will be cheaper and cheaper in the future. In terms of environmental resources, it is not necessary to invest too much energy in saving resources to provide such programs or meet development conditions.
Container Service (TKE) is a one-stop cloud native PaaS service platform based on Kubernetes provided by Tencent Cloud. It provides users with enterprise-level services that integrate container cluster scheduling, Helm application orchestration, Docker image management, Istio service governance, automatic DevOps and a full set of monitoring operation and maintenance systems.