Private Cloud Environments in laptops: Continuous Integration (PART 1)

In this article, the fifth in a series, we continue to talk about how to deploy a simplified private cloud environment in a notebook for a low-cost, low-power, low-latency lab environment. If you have a lightweight cloud server sitting idle, try it out.

Writing in the front

As the first article in the continuous Integration chapter, we’ll talk about the use of CI on a standalone server.

As for the foundation construction, it has been mentioned many times in previous articles, so I will not repeat it. This article will focus on introducing some details in the process. If you are interested in Gitea, Drone or GitLab, you can read the previous content:

• Lightweight Warehouse and CI Use scheme in container Mode: Gitea + Drone Foundation
• Compiling Drone CI with Containers
• Lightweight And Secure Deployment Solution
• Lightweight Code Repository with Docker and Traefik V2 (Gitea)
• Something about GitLab

In order to reduce the maintenance cost and extend the use of multiple machines, all the programs in this paper are used in the container environment.

Single-machine CI design

Before we get into the practical details, let’s talk about design.

Architecture design

In the CI process, there are mainly the following types of participants: users, Git services, CI services, and CI actuators.

A “user” can be a flesh-and-blood person, or an automated script or BOT, the creator of all kinds of data. Git service is used to store code data, provide basic permissions and interface management programs; “CI services”, programs that provide scheduling and management of continuous integration tasks; “CI executor”, a program used to perform specific CI tasks.

Considering that in addition to Git service and CI service, the standalone server will also run the programs that we need to update and deploy. In order to make the resource use efficiency better, maintain the cost lower, and avoid us to configure HTTPS certificate for every Web program, we can add an application gateway that supports service discovery.

Even if it is a single server, we still need to pay attention to the use of SSH security. In a multi-machine environment, we will use jumpers and cloud server security policies for centralized security management. In a single-machine scenario, I use SSH service switch to complete simple security protection (directly disable when not in use, And save power for the sniffer robots on the Internet).

If the above “actor” is represented by a legend, a basic single-machine CI usage pattern would look like this:

I marked the numerical numbers of data interaction between different roles in the figure, and briefly explained the specific content represented by these numbers:

• “1” indicates that users use specific domain names to access our Git service and CI service for warehouse management or CI configuration tasks. This type of interaction uses HTTP, such as access in a browserhttps://gitea.lab.com,https://gitlab.lab.com,https://drone.lab.com.
• 2 indicates that users or clients use SSH to access the Git repository, which requires an RSA Key.
• “3” and “4” represent the way Traefik uses service discovery, aggregating Git services and CI services to provide users with domain name access, again using HTTP proxy mode.
• 5 indicates the data exchange between the SSH switch and Git SSH service. The exchange mode is TCP.
• “6” and “7” represent the data interaction between THE CI service, Git service and CI executor, respectively, to obtain changes in the repository from Git, and then create CI tasks, and then continuously push the execution status of CI tasks to Git service. The interaction mode is not limited, but HTTP API can be used. Various TCP-based RPC approaches can also be used.
• “8” indicates how the CI executor obtains code from the Git server code repository, or updates some data back to the Git server. In general, it uses HTTP. I recommend Git Over SSH for interaction.

Deployment patterns

In standalone full container mode, there are two ways to deploy.

One is based on file mounting methods, such as connecting the file system in the CI JOB container with the host machine in the CI process, and then synchronizing the build product to the host machine, similar variations and using various network file protocols for file system mounting; The other is to use SSH, SCP, Rsync and other methods to access the host in the container to complete data exchange or service initialization or start and stop operations.

In addition, if we use software warehouse, container warehouse, we can also do pure container delivery, making the interaction more pure and “clean”. This is a topic we will explore in a future article.

Single-node CI configuration practices

Next, the application “SSH switch” in the above article takes continuous integration and deployment practice in Gitea and Drone environment as an example to talk about how to use CI in standalone mode.

Because this project type is a continuously running network application that does not support hot loading, the update of the application requires restarting the service. So we can just update the file using the mount file in “Deployment mode” and use SSH to stop and restart the service. (For static resource class project deployment, you only need to complete the resource replacement update.)

Define a CI profile

The first step is to upload the project that needs to integrate CI to a repository in Gitea, such as the Git SSH switch mentioned above. Create a CI configuration file called.drone.yml in your project.

A relatively generic CI configuration can be expressed as follows:

---
kind: pipeline
name: default

steps:

- name: clone

- name: stop-previous-services
  depends_on: [ clone ]

- name: update-services
  depends_on: [ stop-previous-services ]

- name: start-new-services
  depends_on: [ update-services ]
Copy the code

The above configuration includes four processes: downloading the repository code, stopping the original service, updating the service program code, and restarting the service. In actual production, depending on the type of business, our order of execution may change or even stop being “serial” as above.

After CI is configured according to the above configuration, when we push the code to the code repository and trigger the CI task. In the graphical interface, we should see a result similar to the one above.

Download the code using the SSH protocol

Regardless of which CI tool you are using, I recommend that you use Git Over SSH to get your code instead of using Git tokens or passwords to interact. This will make your program less dependent on a CI or Git repository, making it easier to switch to a more appropriate tool at the right point in time at a low cost.

In Drone CI, if you want to use SSH mode to download code, you can use the following configuration :(same in GitLab Runner)

---
kind: pipeline
name: default

clone:
  disable: true

steps:

- name: clone
  image: alpine/git
  pull: if-not-exists
  environment:
    KEY:
      from_secret: ssh_key
  commands:
    - GIT_HOST=$(echo $DRONE_GIT_SSH_URL | sed 's/git@/\1/' | sed 's/:.*/\1/') && mkdir "$HOME/.ssh" && echo "$KEY" > "$HOME/.ssh/id_rsa" && chmod 600 $HOME/.ssh/id_rsa && eval `ssh-agent -s` && ssh-add $HOME/.ssh/id_rsa && ssh-keyscan $GIT_HOST > ~/.ssh/known_hosts && chmod 400 "$HOME/.ssh/known_hosts";
    - git clone $DRONE_GIT_SSH_URL .
    - git -c advice.detachedHead=false checkout $DRONE_COMMIT
Copy the code

In the above code, in order to download the program code using SSH, the CI program does two things:

1. Read our pre-configured configuration from CI softwaressh_keyEnvironment variables, and then output the variables into a program that can be used directlyrsa_key, and set the right to usessh-agentLoader.
2. Use the default for the repositoryHTTPReplace protocol withGitProtocol for use by the program.

To download Git code using SSH, you need to configure the SSH Key in your Git account or repository.

Start and stop the service in SSH mode

In this application, we defined the container startup mode in docker-compose. Yml, so the service can be started and shut down using the familiar commands docker-compose up -d and docker-compose Down.

Since CI is executed in the container, we cannot operate the host directly, so we need to use SSH or Docker. sock in DIND mode to complete the change of service state.

This article begins by discussing how to use SSH to solve basic deployment operations:

- name: stop-or-start-services
  image: deploy-tool
  depends_on: [ clone ]
  pull: if-not-exists
  environment:
    KEY:
      from_secret: ssh_key
    Environment variables, in addition to the private definition of CI software environment variables, can also be explicitly declared in CI configuration
    TARGET_HOST: user@host
    TARGET_PORT: 22
  commands:
    - mkdir "$HOME/.ssh" && echo "$KEY" > "$HOME/.ssh/id_rsa" && chmod 600 $HOME/.ssh/id_rsa && eval `ssh-agent -s` && ssh-add $HOME/.ssh/id_rsa";
    # service shutdown
    - ssh -i "$HOME/.ssh/id_rsa" -p $TARGET_PORT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $TARGET_HOST "bash -c \"cd /app-path/ && docker-compose down\""
    # start service
    - ssh -i "$HOME/.ssh/id_rsa" -p $TARGET_PORT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $TARGET_HOST "bash -c \"cd /app-path/ && docker-compose up -d\""
Copy the code

Similar to the download code, we initialize the RSA key from the environment variable and load the private key in the Ssh-agent. Then use SSH client to connect to the host, switch the working directory, and run commands to start and stop the service.

Similarly, to use SSH to operate the server, you need to configure the corresponding public key in ~/.ssh/authorized_keys of the server user.

Update code using file mounts

There are two ways to update the code, one is to use SSH as mentioned above to remotely execute SCP, rsync and other commands to synchronize data, and the other is to use file mount. Since we were deployed on the same machine, file mounting was an efficient way to do it.

Take Drone CI configuration as an example to demonstrate how to mount the host directory to the container:

- name: update-services
  image: deploy-tool
  depends_on: [ stop-previous-services ]
  pull: if-not-exists
  commands:
    - rm -rf /deploy/*
    - cp -r /drone/src/* /deploy/
    - cp -r /drone/src/.env /deploy/
  volumes:
    - name: host-dir
      path: /deploy

volumes:
  - name: host-dir
    host:
      path: /app-path
Copy the code

The last

In the next “continuous Integration” article, I will discuss CI’s use in multiple machines and relatively complex scenarios, as well as the practical details of deployment in other scenarios.

–EOF

---

We have a little group of hundreds of people who like to do things.

In the case of no advertisement, we will talk about software and hardware, HomeLab and programming problems together, and also share some information of technical salon irregularly in the group.

Like to toss small partners welcome to scan code to add friends. (To add friends, please note your real name, source and purpose, otherwise it will not be approved)

All this stuff about getting into groups

---

If you think the content is still practical, welcome to share it with your friends. Thank you.

---

This article is published under a SIGNATURE 4.0 International (CC BY 4.0) license. Signature 4.0 International (CC BY 4.0)

Author: Su Yang

Creation time: January 02, 2022 statistical word count: 5438 words reading time: 11 minutes to read this article links: soulteary.com/2022/01/02/…