You’ve probably heard of “GitOps” before, but you don’t really know what it is. Besides GitOps, you’ve probably heard of DevOps, or AIOps, GOps, etc. Yes, this is the age of “Ops.”

GitOps is a model for continuous delivery. Its core idea is to store the declarative infrastructure and applications of an application system in Git’s version control repository. Choerodon pigfish used GitOps as a reference when building continuous delivery lines. As the saying goes, “Theory comes first when soldiers are not moved”. In this paper, we will focus on the principles and models of GitOps workflow and some practical experiences in applying them to production and large-scale operation of Kubernetes. In the next article, we’ll look at how The Choerodon toothfish practiced and landed GitOps to build a repeatable and reliable delivery process.

GitOps, 90% of the best practices, 10% of the interesting new things we need to build. — GitOps – Operations by Pull Request

From: www.weave.works

This post is based on a translation of several articles by Weave Cloud about GitOps:

  • GitOps: Operations by Pull Request

  • The GitOps Pipeline – Part 2

  • GitOps – Part 3: Observability

  • GitOps – Part 4: Application Delivery Compliance and Secure CICD

The main content

  • What are GitOps?

  • The main advantages of GitOps

  • Application scenarios for GitOps — suitable for cloud native continuous delivery

  • Basic principles of GitOps

  • Best practices

  • Pull pipeline – Pull Request operations

  • GitOps workflow

  • visualization

  • CI/CD for application delivery compliance and security

  • The value GitOps bring

What are GitOps?

GitOps is a means of continuous delivery. Its core idea is to store the declarative infrastructure and applications of the application system in Git repository.

With Git at the heart of the delivery pipeline, every developer can submit Pull requests and use Git to speed and simplify Kubernetes’ application deployment and o&M tasks. By using a simple familiarity tool like Git, developers can more efficiently focus on creating new functionality rather than on operations related tasks (for example, application installation, configuration, migration, and so on).

GitOps: versioned CI/CD on top of declarative infrastructure. Stop scripting and start shipping. t.co/SgUlHgNrnY — Kelsey Hightower (@kelseyhightower) January 17, 2018

The main advantages of GitOps

With GitOps, when infrastructure code changes are committed using Git, the automated delivery pipeline applies those changes to the actual infrastructure of the application. But GitOps goes further than that — it also uses tools to compare the actual production status of the entire application to the infrastructure source code, and then it tells the cluster which infrastructure source code doesn’t match the actual environment.

By applying GitOps best practices, application infrastructure and application code are “real sourced” — essentially, both infrastructure and application code are stored on a version control system such as GitLab or Github. This enables development teams to speed up development and deployment and improve application reliability.

The application of GitOps theory and method to continuous delivery pipeline has many advantages and characteristics:

  • Secure cloud-native CI/CD pipeline model

  • Faster average deployment time and average recovery time

  • Stable and reproducible rollback (for example, according to Git restore/rollback/fork)

  • Combined with monitoring and visualization tools, it provides comprehensive monitoring of deployed applications

GitOps application scenario – Meet continuous delivery in cloud native environment

As a CI/CD pipeline solution, GitOps has been described as the “holy grail” of the software development process. Since no single tool can do all the work required in the pipeline, you are free to choose the best tool for different parts of the pipeline. You can choose a set of tools from an open source ecosystem, a set of tools from a closed source, or even combine them, depending on usage, but the hardest part of creating an pipelining is gluing all the pieces together.

No matter how to choose to construct their own delivery line, will be based on the Git (or other version control tool) GitOps best practices used in the delivery pipeline is a choice, this will make the building a continuous delivery pipeline, and subsequent promotion easier, it not only from a technical point of view and from a cultural perspective.

Of course, GitOps is not a panacea, and it has its own application scenarios.

Immutable infrastructure

Applications need to run on multiple machines, organized into different environments, such as development, test, production, and so on. You need to deploy the same application on different machines. The system administrator is usually required to ensure that all machines are in the same state. Then all changes, patches, and upgrades need to be made on all machines. As time goes on, it becomes harder to ensure that all machines are in the same state and more and more error-prone. This is a common problem with traditional mutable architectures. Here we have immutable architecture, which packages the entire machine environment into a single immutable unit instead of just packaging applications in the traditional way. This unit addresses the previous problem by containing the entire environment stack mentioned above and applying all the changes, patches, and upgrades. — From InfoQ’s “On Immutable Architecture and Why We Need It” by Zhan Hui Bai

The concept of immutable infrastructure is not new, nor does it necessarily require container technology. However, through the container, it became more understandable, more practical, and attracted widespread attention in the industry. “Immutable infrastructure” enables us to understand and face application systems in a new way, especially in terms of deployment and operation of distributed systems represented by microservices, which are less complex and more controllable.

How immutable infrastructure can be easily applied to the actual production process raises another question for the industry. GitOps emerged in the application practice of specific Kubernetes, and GitOps needs to rely on “immutable infrastructure” to play its role. In part, “immutable infrastructure” created the necessary conditions for the emergence of GitOps, which in turn uses Kubernetes’ container choreography capabilities to quickly build the required components of an application system using images.

Declarative container choreography

As a cloud-native tool, Kubermetes can think of its “declarative” as “code”, stating that the configuration consists of a set of facts rather than a set of instructions, for example, “There are ten Redis servers”, rather than “Start ten Redis servers and tell me if it works”.

With the declarative nature of Kubermetes, the entire configuration file set of an application system can be version-controlled in Git libraries. By using Git libraries, applications are easier to deploy into Kubernetes, as well as for version rollback. More importantly, when a disaster occurs, the cluster’s infrastructure can be reliably and quickly recovered from Git repositories.

The declarative nature of cloud native tools such as Kubernetes is that the configuration of instance, container, network, storage and CPU can be easily expressed through a set of codes. Cloud native tools such as Kubernetes can use these configuration codes to run a set of container-based application systems, such as YMAL,

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: nginx-deployment

spec:

replicas: 1

template:

metadata:

labels:

app: nginx

spec:

containers:

– name: nginx

Image: registry.choerodon.com.cn/operation-choerodon-dev/nginx-demo:1.13.5-alpine

ports:

– containerPort: 80

GitOps leverages immutable infrastructure and declarative container choreography to make it easy to manage multiple deployments. To minimize the risk of post-deployment change, whether intentional or accidental “configuration bias,” GitOps builds a repeatable and reliable deployment process that provides rapid and complete recovery in the event of an entire application down or damaged.

Basic principles of GitOps

Here are a few principles of GitOps in a cloud native environment:

● Anything that can be described must be stored in a Git repository

Using Git as a repository for storing declarative infrastructure and application code makes it easy to monitor clusters and check to see if the state of the actual environment is consistent with the state on the code base. So, our goal is to describe system related to all of the content: strategy, code, configuration, monitoring events and even version control, etc., and all the content stored in the repository, in the building of a system through the contents of the repository infrastructure or application, if there is no success, can quickly rolled back, and again.

● Kubectl should not be used directly

As a general rule, it is not recommended to use kubectl command operations directly from the command line to perform deployment infrastructure or applications into a cluster. Other developers use CI tools to drive application deployment, but doing so can introduce potentially unpredictable risks to the production environment.

● The interface or controller that calls Kubernetes’ API should follow Operator mode

The interface or controller that calls Kubernetes’ API should follow Operator mode. , the state of the cluster and the configuration files in the Git repository should be consistent, and the state differences between them should be analyzed.

Best practices

Use Git as the only true source of facts

Git is part of every developer’s toolkit. Learning feels natural and less intimidating, and the tools themselves are very simple. By using Git as the source of facts for your application, you can manipulate just about anything. For example, version control, history, review, and rollback are all done through Git without using tools like Kubectl.

Therefore, Git is the foundation of GitOps formation, as described in the first principle “Anything that can be described must be stored in Git repositories” : Using Git as a repository for storing declarative infrastructure and application code makes it easy to monitor clusters and check to see if the state of the actual environment is consistent with the state on the code base. So, our goal is to describe system related to all of the content: strategy, code, configuration, monitoring events and even version control, etc., and all the content stored in the repository, in the building of a system through the contents of the repository infrastructure or application, if there is no success, can quickly rolled back, and again.

Pull pipeline – Pull Request operations

▌ Push pipeline

Most CI/CD tools today use a push-based model. A push-based pipeline means that the code starts with a CI system, goes through a series of build tests, etc., and is eventually mirrored, and any changes are pushed manually to the Kubernetes cluster using “Kubectl”.

Many developers may be reluctant to start the CD deployment process in CI, or to use a command-line tool to start the CD deployment process, because doing so would expose the cluster’s users and passwords, etc. While measures can be taken to secure CI/CD scripts and command lines, these operations still work outside the cluster in an untrusted zone. Therefore, it is not advisable to do this, which may bring potential risks to system security.

A typical push pipeline with out-of-cluster read/write (R/W) permissions:

  • CI runs the tests, and the output is passed to the container image repository.

  • CD The system automatically deploys the container (or on request, manually).

▌ Pull assembly line

In GitOps, mirrors are pulled out and credentials remain in the cluster:

At the heart of the pull pipeline pattern is the Git library, which stores sets of applications and configuration files. Developers push updated code to the Git code base; The CI tool picks up the changes and eventually builds the Docker image. GitOps detects an image, extracts the new image from the repository, and updates its YAML in the Git configuration repository. GitOps then detects that the cluster has expired, extracts the changed list from the repository, and deploys the new image to the cluster.

The pipeline of GitOps

As described in the previous section, GitOps uses a pull pattern to build a delivery pipeline, this section details what to look for and what best practices are available when building a GitOps pipeline.

▌ GitOps line

This is a new diagram showing that everything upstream of a deployment works around a Git repository. As mentioned in “Pull Pipeline”, developers push updated code to Git code base, CI tool picks up the changes and eventually builds Docker image. The GitOps Config Update detects an image, extracts the new image from the repository, and then updates its YAML in the Git configuration repository. The Deploy Operator of GitOps then detects that the cluster has expired, extracts the changed manifest from the repository, and deploys the new image to the cluster.

With Deploy Operator inside the cluster, the cluster credentials are not exposed outside the production environment. Once the Deploy Operator is installed in the cluster and connected to the Git repository, any changes in the online environment are made through Git pull requests with full rollback and the convenient audit logs provided by Git.

▌ Automatic Git → Cluster synchronization

Because there is no single tool can complete all the necessary work in the assembly line, can choose a set of tools from open source ecosystems, can also choose a set of tools, from closed source or by usage, can even combine them together, in fact, create assembly line is the most difficult part of all the parts together. To implement GitOps, new components must be developed that glue these tools together to enable the pull delivery pipeline.

Deployment and release automation is the foundation for implementing GitOps and enabling the delivery pipeline. **GitOps not only ensures that when a developer updates a configuration set with Git, the GitOps pipeline automatically updates the online environment with the latest configuration file state, but also that GitOps can match the latest configuration set state in the Git repository with the latest online environment state in real time.

In the previous section, we mentioned two terms: Config Update and Deploy Operator. According to GitOps practice, Config Update and Deploy Operator need to be designed and developed. They are the key components necessary to implement the GitOps pipeline. GitOps gives them their magic, as they are both tools for automating container upgrades and publishing to an online environment, and may also be responsible for tasks such as servicing, deployment, network policy, and even routing rules. Therefore, Config Update and Deploy Operator are the “glue” that maps all the relationships between the code, the service and the running cluster.

Of course, you can give a variety of other capabilities depending on your design, but automatic synchronization is definitely needed to ensure that if any changes are made to the repository, those changes are automatically deployed to the online environment.

▌ Only containers and configurations are deployed

Instead of deploying applications directly to an online environment, GitOps recommends packaging applications and associated configurations into images, storing them in a mirror library, and finally, mirroring containers for deployment to an online environment.

Why are containers so important? In the GitOps model, we use immutable infrastructure patterns. Once the code is committed in Git, GitOps doesn’t want anything else to change, which minimizes the potential risk of system uncertainty and inconsistency. For example, the same application needs to be deployed on different machines. The system administrator is usually required to ensure that all machines are in the same state. Then all changes, patches, and upgrades need to be made on all machines. As time goes on, it becomes harder to ensure that all machines are in the same state and more and more error-prone. However, containers are the perfect solution to this problem. Of course, using virtual machines is possible, which is obviously more convenient.

Observability of GitOps

“Observable is like drive testing in production. If you don’t know how to make sure it works, don’t accept a pull request. @mipsytipsy “- Adriano Bastos

In GitOps, Git libraries are used to store the configuration sets and applications of the application system. It ensures that developers will version control all configuration and application additions and modifications through Git libraries, making Git the only true source of configuration and applications. The observability of GitOps is to ensure that the true state of the online environment is consistent with that in the Git repository. This section introduces you to the observability of GitOps.

Observability is another source of truth

In GitOps, we use Git as the true source of the required state for the system. For example, if the application system goes down, GitOps can be rolled back to the previous correct state. Observability is the true source of the actual operating state of the system, which can be monitored by system developers or o&M personnel. This is a picture of the process.

Finding the answer to a question through observation

If you use Kubernetes as a cloud native environment and container choreographer tool, I believe you will have such feelings, although Kubernetes is a very good choreographer container platform, but the ensuing lack of friendly visual management interface to the developer or operation and maintenance personnel to bring a lot of inconvenience. Such as:

  • Was my deployment successful? My system is working now. Can I go home now?

  • How is my system different from before? Can I use Git or our system history to check?

  • Do my changes improve the overall user experience? (as opposed to system correctness)

  • I can’t find my new service (e.g. RED metrics) in the information center

  • Is this failure related to my last service update event, or is it related to another operation?

You might think of monitoring the server’s CPU, memory, network, etc., as well as the application’s logs, or even the call chain of microservices to solve the problem. Yes, there is nothing wrong with that, and you can get some feedback, but developers or operations people who have used similar monitoring will also feel that these monitoring dashboards give us a lot of information that needs to be carefully screened, and a lot of information is not available in these dashboards. This means that new dashboards need to be created to monitor new metrics and content.

▌ Observability of GitOps

Observability can be considered one of the main drivers of Kubernetes’ continuous delivery cycle, as it describes the actual operating state of the system at any given time. Observe the operating system to understand and control it. New features and fixes are pushed to Git and trigger the deployment pipeline, allowing you to see the running cluster in real time when it’s ready for release. At this point, the developer can use this feedback to go back to the beginning of the pipeline or deploy the image and release it to the production cluster.

Here GitOps introduces a new tool, Diffs, to monitor and compare system state. That is:

  • Verify that the current state of the online system matches the state described in the Git repository; for example, did my last release meet expectations?

  • Alert developers to the inconsistent status and the corresponding details.

As mentioned earlier in this article, what is stored in Git libraries is actually “declarative infrastructure”, such as Kubernetes’ YAML files, which are used to build the various components, domain names, networks, and other configuration information required for application systems. Diffs needs to read the configuration information in the Git library and the corresponding cluster information through the API for comparison.

For example, Kubernetes cluster: The required Kubernetes state might be “there are 4 Redis servers”. Diffs periodically checks the cluster and alerts when the number changes from 4. In general, Diffs converts YAML files into health queries.

GitOps is a publish-oriented operational model, as shown in the figure below. The speed of delivery depends on how quickly the team can bypass the phases of the cycle.

Application delivery compliance and security

Compliance and auditing are trivial because changes are tracked and recorded in a secure manner. Comparison tools such as Diffs also make it possible to compare the state of a cluster defined in a Git repository with an actual cluster running to ensure that the changes match the actual situation.

Recording all operation logs in Git

As described in the above article, developers or operations personnel can configure the Git operating system and create and update applications. All actions made through Git client Git Commit/Git merge are recorded in Git repositories, and auditors can look at Git to see who made any changes, when, why, and how they affected the running system deployment. Of course, you can customize different delivery compliance to your needs. Git records every step of the operation, as opposed to going directly to the server or operating the cluster through Kubctl, which provides a complete operation log for compliance and auditing.

▌ Role and permission control

Almost all Git libraries provide role and permission control, and people who are not involved in development and operations do not have permission to operate Git libraries. Rather than directly distributing the operation rights of the server or cluster, this is particularly vulnerable to security breaches.

Benefits of GitOps

Faster development

With the best practices of GitOps, developers can use familiar Git tools to continuously deploy applications and their corresponding configuration file sets to the cloud native environment such as Kubernetes, improve business agility, quickly respond to the needs of users, and help increase the competitiveness of the enterprise market.

Improving operation and maintenance

With GitOps, a complete end-to-end delivery pipeline can be implemented. Not only can pull continuous integration pipeline and continuous deployment pipeline be realized, but also the operation and maintenance of the system can be completed through Git.

More powerful security guarantee

Almost all Git libraries provide role and permission control, and people who are not involved in development and operations do not have permission to operate Git libraries. Rather than directly distributing the operation rights of the server or cluster, this is particularly vulnerable to security breaches.

Easier compliance auditing

Compliance and auditing are trivial because changes are tracked and recorded in a secure manner. Comparison tools such as Diffs also allow you to compare a trusted definition of cluster state with an actual running cluster to ensure that tracked and auditable changes match the actual situation.