preface
Before, there was a special article about gitlab CI, in which gitlab-CI.yML was introduced. It may be questioned why we should write another article about YML?
The main reason is that I encountered many problems with Gitlab-CI before, and it was not easy to explore along the way. Although I learned from the previous experience, I still encountered indescribable problems, because I wanted to understand this part again as a note. On an article about gitlab ci introduction links: https://juejin.cn/post/6844903609864290311
Gitlab-ci operation introduction
It consists of the following two modules:
gitlab-ci server
gitlab-ci-runner
Copy the code
Among them, Gitlab-CI Server is responsible for scheduling, triggering runners, and getting returned results. The Gitlab-Ci-runner is responsible for running ci automation (testing, compiling, packaging, etc.).
The basic process is:
The user submits the code to check if there is a. Gitlab-ci. yml file. If so, runner is called to execute the script to get the result returned.Copy the code
.gitlab-ci.yml
Starting with version 7.12, GitLab CI uses YAML files (.gitlab-ci.yml) to manage project configuration. This file resides at the root of the project repository and defines how the project is built.
The YAML file defines a set of tasks with constraints before starting the build. These tasks start with the task name and must contain at least a script part:
job1:
script:
- echo "jb1"
job2:
script:
- echo "jb2"
Copy the code
The example above is a simple CI configuration with two separate tasks, each performing echo output.
Script can execute system commands directly (e.g../configure; make; Make install) or execute the script directly (test.sh)
The task is taken over by Runners and performed by the runner on the server. Each task is executed independently.
Example 1:
# define job job1: stage: test script: -echo "I am jb1" -echo "I am in test stage" - echo "I am jb2" - echo "I am in build stage"Copy the code
According to the definition in stages, the build stage should be run before the test stage. Therefore, the stage: Build jobs should be run first and the stage:test jobs should be run later.
Here’s another complicated example 2:
Image: Ruby :2.1 services: -postgres before_script: -bundle install after_script: -rm stages: - build - test - deploy job1: stage: build script: - execute-script-for-job1 only: - master tags: - docker5+4X5=25Copy the code
The reserved fields listed below cannot be defined as job names:
The keyword | Whether must | describe |
---|---|---|
image | no | For docker images, view docker documents |
services | no | For docker services, view docker documents |
stages | no | Define the Construction phase |
types | no | Alias of stages (deprecated) |
before_script | no | Defines the commands to run before each job |
after_script | no | Defines the commands to run after each job |
variable | no | Defining build variables |
cache | no | Defines a list of files that can be used in subsequent runs |
The keywords image and services allow the use of a custom Docker image and a set of services and can be used throughout the job cycle.
Before_script Before_script is used to define the commands to run before all jobs, including deploy(deploy) jobs, but after fixing artifacts. It can be an array or a multi-line string.
after_script
GitLab 8.7 was introduced and required GitLab Runner V1.2Copy the code
After_script is used to define the commands to run after all jobs. It must be an array or a multi-line string;
Stages Defines stages that can be called by a job.
The order of elements in stages determines the order in which jobs are executed:
1. Jobs of the same stage can be executed in parallel. 2. The job of the next stage is executed after the job of the previous stage succeeds.Copy the code
The job of the next stage will be executed after the job of the previous stage succeeds. Because of this, many interdependent steps can be set on the stage without judging whether the job is successful or not. If the job can be executed, it must be successful.
Look at example 1:
stages:
- build
- test
Copy the code
-
- First of all, all
build
All jobs are executed in parallel
- First of all, all
-
- all
build
After jobs successfully executed,test
The jobsWill only beStart parallel execution
- all
- 3. All
test
The commit is marked assuccess
- 4. If any of the pre-jobs fail, commit is marked as
failed
And the next jobs of stages will not be implemented.
Here are two special examples:
1. If stages are not defined in. Gitlab-ci. yml, jobs stages are defined as build, test, and deploy 2 by default. If a job does not specify a stage, the task is assigned to the Test stageCopy the code
types
Deprecated and will be removed in 10.0. "Stages" is used instead of "stages". It has the same meaning as "stages"Copy the code
Variable GItLab CI allows variables to be added to.gitlab-ci.yml files and to work in the job environment. Because these configurations are stored in git repositories, it is best to store non-sensitive configurations of the project, such as:
variables:
jb: "jb is here"
Copy the code
Note that the: is followed by a space, which would be a syntax error. These variables can be used by subsequent commands and scripts.
In addition to user-defined variables, Runner can also define its own variables. A good example is CI_COMMIT_REG_NAME, whose value represents the branch or tag name used to build the project. In addition to setting variables in.gitlab-ci.yml, you can also set private variables on the Gitlab interface.
Cache specifies the files or directories that need to be cached between jobs. Only paths within the project workspace can be used.
Caching in Charge and Job is enabled by default since GitLab 9.0
If the cache is defined outside of jobs’ scope, it is a global cache and all Jobs can use it.
Caches files in Git that are not tracked:
rspec:
script: test
cache:
untracked: true
Copy the code
Caches files in binaries that are not tracked by Git:
rspec:
script: test
cache:
untracked: true
paths:
- binaries/
Copy the code
The cache Key directive allows us to define the cache scope (affinity), which can be a single cache for all jobs, per job, per branch, or wherever you see fit.
Cache :key can use any of the predefined variables
Caches each job:
cache:
key: "$CI_JOB_NAME"
untracked: true
Copy the code
Cache each branch:
cache:
key: "$CI_COMMIT_REF_NAME"
untracked: true
Copy the code
Cache each job and each branch:
cache:
key: "$CI_JOB_NAME/$CI_COMMIT_REF_NAME"
untracked: true
Copy the code
Cache each branch and each stage:
cache:
key: "$CI_JOB_STAGE/$CI_COMMIT_REF_NAME"
untracked: true
Copy the code
If Windows Batch is used to run the script, use % instead of $
cache:
key: "%CI_JOB_STAGE%/%CI_COMMIT_REF_NAME%"
untracked: true
Copy the code
Jobs
.gitlab-ci.yml allows specifying an unlimited number of jobs. Each job must have a unique name, and it cannot be the keyword mentioned above. Job is a list of parameters that define the behavior of a job.
job_name:
script:
- rake spec
- coverage
stage: test
only:
- master
except:
- develop
tags:
- ruby
- postgres
allow_failure: true
Copy the code
keywords | If necessary | describe |
---|---|---|
script | yes | Runner A command or script executed |
image | no | Docker image used, see using docker image |
services | no | Docker service used, check using docker image |
stage | no | Define job Stage (default: test) |
type | no | Alias for stage (deprecated) |
variables | no | Define job-level variables |
only | no | Define a list of Git branches and create jobs for them |
except | no | Define a list of Git branches without creating a job |
tags | no | Defines a list of tags that specify which Runner to select (Runner also sets tags) |
allow_failure | no | Allow the job to fail. A failed job does not affect the COMMIT status |
when | no | Define when to start a job. It can be on_success, on_failure, always, or manual |
dependencies | no | Define job dependencies so that they can pass artifacts to each other |
cache | no | Defines a list of files that should be cached between subsequent runs |
before_script | no | Overrides a set of commands executed before a job |
after_script | no | Overrides a set of commands to execute after a job |
environment | no | Define the name of the environment in which this job is deployed |
coverage | no | Defines code coverage Settings for a given job |
Script Script is a YAML script executed by Runner.
job:
script: "bundle exec rspec"
Copy the code
This parameter can also contain multiple commands in an array:
job:
script:
- uname -a
- bundle exec rspec
Copy the code
Note here that if a script command contains a colon:, it needs to be wrapped in double quotes so that the YAML parser can correctly parse it as a string, not as a key:value pair. Use the following special characters with caution:
: {and}, [and], &, *, #,,,? , | -- - <, > =,! .Copy the code
Stage Stages allow a group of jobs to enter different stages. Jobs will parallel at the same stage; For more usage, click here;
Tags Tags can perform Jobs by selecting specific Runners from all Runners that are allowed to run this project. In the process of registering Runner, we can set the tag of Runner, such as Ruby, Postgres, development.
Tags can be used to run jobs by specifying special Runners via tags:
job:
tags:
- ruby
- postgres
Copy the code
In the example above, you need to ensure that the Runner building the job must define the tags Ruby and Postgres.
Allow_failure Allow_failure can be used when you want to set a job to fail without affecting subsequent CI components. Failed jobs do not affect the commit status.
In the following example, job1 and joB2 will run side by side. If job1 fails, it will not affect the next stage in progress because allow_failure: true is set.
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
job2:
stage: test
script:
- execute_script_that_will_succeed
job3:
stage: deploy
script:
- deploy_to_staging
Copy the code
when when is used to implement jobs that are run in case of failure or despite the failure. Used to implement a job that runs when it fails or fails.
You can set the following values:
On_success - This is performed only when all work on the previous stages is successful. This is the default value. On_failure - Executed when any jobs in current stages fail. Always - Executes regardless of the jobs state in the previous stages. 'manual' - Manual execution.Copy the code
Here’s an example:
stages:
- build
- cleanup_build
- test
- deploy
- cleanup
build_job:
stage: build
script:
- make build
cleanup_build_job:
stage: cleanup_build
script:
- cleanup build when failed
when: on_failure
test_job:
stage: test
script:
- make test
deploy_job:
stage: deploy
script:
- make deploy
when: manual
cleanup_job:
stage: cleanup
script:
- cleanup after jobs
when: always
Copy the code
Script description:
- ‘cleanup_build_job’ is executed only if build_job fails.
- ‘cleanup_job’ is executed whether the previous job failed or succeeded.
- Deploy_jobs can be executed manually from the GitLab interface.
Coverage allows you to configure that code coverage will extract output from the job. Here the regular expression is the only valid value. Therefore, a string must be preceded by a/to indicate a correct regular expression rule. Special strings need to be escaped.
Here’s an example:
job1:
coverage: '/Code coverage: \d+\.\d+/'
Copy the code
Git Strategy
Introduced as an experimental feature in GitLab 8.9. It may be changed or completely removed in future releases. GIT_STRATEGY requires GitLab Runner V1.7 +.Copy the code
You can set GIT_STRATEGY to get the latest code, either in the global Variables module or in the variables module of a single job. If not, the default value is used from the project.
The values you can set are: clone, fetch, and None.
Clone is the slowest option. It clones the entire repository from scratch, including every job, to ensure that the project workspace is original.
variables:
GIT_STRATEGY: clone
Copy the code
When it reuses the project workspace, the FETCH is faster (and returns a clone if it doesn’t exist). Git clean is used to undo any changes made to the previous job, and git fetch is used to get a commit from the previous job to now.
variables:
GIT_STRATEGY: fetch
Copy the code
None also reuses the project workspace, but it skips all Git operations (including pre-GitLab Runner clone scripts, if any). This is mainly used to manipulate job artifacts (for example, deploy).
variables:
GIT_STRATEGY: none
Copy the code
Git Checout When GIT_STRATEGY is set to clone or fetch, you can use the GIT_CHECKOUT variable to specify whether Git checkout should be run. If not specified, it defaults to true. Like GIT_STRATEGY, it can be set in the global variables or in the variables of a single job.
If set to false, Runner will:
- Fetch – Updates the repository and keeps a working copy in the current version,
- Clone – Clones the repository and keeps a working copy in the default branch.
Setting this to true will mean that both the Clone and FETCH policies will cause Runner to perform project workspace updates to the latest
variables:
GIT_STRATEGY: clone
GIT_CHECKOUT: false
script:
- git checkout master
- git merge $CI_BUILD_REF_NAME
Copy the code
Git Submodule Strategy The GIT_SUBMODULE_STRATEGY variable is used to determine whether or how Git submodules are introduced when code is pulled prior to build. Its available values are: None, normal, and Recursive: 1) None means that submodules will not be introduced when project code is pulled. This is the default. 2) Normal means that only top-level submodules are introduced. It is equivalent to
git submodule sync
git submodule update --init
Copy the code
3) Recursive means that all submodules (including submodules of submodules) are imported, which is equivalent to:
git submodule sync --recursive
git submodule update --init --recursive
Copy the code
Note: For this function to work properly, the submodule must be configured (in.gitModules) with one of the following:
Accessible common warehouse HTTP (s) address, a real address accessible to another warehouse on the same GitLab server.Copy the code
Job stages Attempts the following stages will be executed according to the number of attempts you set:
variable | describe |
---|---|
GET_SOURCES_ATTEMPTS | Number of attempts to obtain job sources |
ARTIFACT_DOWNLOAD_ATTEMPTS | The number of attempts to download Artifacts |
RESTORE_CACHE_ATTEMPTS | The number of attempts to rebuild the cache |
The default is one attempt.
Example:
variables:
GET_SOURCES_ATTEMPTS: 3
Copy the code
Pages Pages is a special job for uploading static content to GitLab that can be used to serve your web site. It has a special syntax, so it must meet the following two requirements:
Any static content must be placed in the public/ directory artifacts must be defined in the public/ directoryCopy the code
The following example moves all files from the project root directory to the public/ directory. .public workflow is cp, and it does not loop over public/ itself.
pages:
stage: deploy
script:
- mkdir .public
- cp -r * .public
- mv .public public
artifacts:
paths:
- public
only:
- master
Copy the code
Variables defined by Runner
In the introduction of variable above, it was mentioned that Runner also has variables defined by himself, as follows:
variable | Gitlab version | runner | describe |
---|---|---|---|
CI | all | 0.4 | Mark that the job is executed in a CI environment |
CI_COMMIT_REF_NAME | 9.0 | all | The branch or tag name for the build project |
CI_COMMIT_REF_SLUG | 9.0 | all | $CI_COMMIT_REF_NAME lowercase, except 0-9 and a-z, replaced with – |
CI_COMMIT_SHA | 9.0 | all | Build the commit SHA value of the project |
CI_COMMIT_TAG | 9.0 | 0.5 | The tag name of the commit that builds the project, which is only used when building tags. |
CI_COMMIT_MESSAGE | 10.8 | all | Complete commit message. |
CI_COMMIT_TITLE | 10.8 | all | Submitted title – The full first line of the message |
CI_COMMIT_DESCRIPTION | 10.8 | all | Description of the submission |
CI_CONFIG_PATH | 9.4 | 0.5 | CI configuration file path. The default value is.gitlab-ci.yml |
CI_DEBUG_TRACE | all | 1.7 | Whether to enable the debugging tracing function |
CI_DEPLOY_USER | 10.8 | all | The authentication user name for the GitLab deployment token |
CI_DEPLOY_PASSWORD | 10.8 | all | Authentication password for the GitLab deployment token |
CI_DISPOSABLE_ENVIRONMENT | all | 10.1 | Flags whether the job runs in a one-time environment |
CI_ENVIRONMENT_NAME | 8.15 | all | The name of the environment to execute the job, such as produciton,dev,pre-production, etc |
CI_ENVIRONMENT_SLUG | 8.15 | all | Simplified version of environment name for DNS, URL, Kubernetes tag, etc. |
CI_ENVIRONMENT_URL | 9.3 | all | URL of the environment where the job is executed |
CI_JOB_ID | 9.0 | all | Unique ID of the current job in the GitLab CI |
CI_JOB_MANUAL | 8.12 | all | Indicates that the job is to be started manually |
CI_JOB_NAME | 9.0 | 0.5 | The name of the job defined in.gitlab-ci.yml |
CI_JOB_STAGE | 9.0 | 0.5 | The stage name defined in.gitlab-ci.yml |
CI_JOB_TOKEN | 9.0 | 1.2 | The token used for authentication in the GitLab container registry |
CI_JOB_URL | 11.0 | 0.5 | The URL for the work |
CI_REPOSITORY_URL | 9.0 | all | Clone Git Repository URL |
CI_RUNNER_DESCRIPTION | 8.10 | 0.5 | Description of runner saved in GitLab |
CI_RUNNER_ID | 8.10 | 0.5 | Unique ID of the runner that is running |
CI_RUNNER_TAGS | 8.10 | 0.5 | Running Runner Tags |
CI_RUNNER_VERSION | all | 10.6 | GitLab Runner version of the current job being executed |
CI_RUNNER_REVISION | all | 10.6 | GitLab Runner revision of the current job being executed |
CI_RUNNER_EXECUTABLE_ARCH | all | 10.6 | GitLab GitLab Runner OS/ architecture for executable programs |
CI_PIPELINE_ID | 8.10 | 0.5 | Unique ID of current pipeline within GitLab CI |
CI_PIPELINE_TRIGGERED | all | all | Mark this job as triggered by a trigger |
CI_PIPELINE_SOURCE | 10.0 | all | Indicate how to trigger the pipeline. The options available are: Push, Web, Trigger, Schedule, API, and Pipeline |
CI_PROJECT_DIR | all | all | The full path to the cloned repository and the directory in which the job is running |
CI_PROJECT_ID | all | all | Unique ID of the current project within the GitLab CI |
CI_PROJECT_NAME | 8.10 | 0.5 | The name of the project currently being built (actually the project folder name) |
CI_PROJECT_NAMESPACE | 8.10 | 0.5 Project namespace currently being built (user name or groupname) | |
CI_PROJECT_PATH | 8.10 | 0.5 | The full project path, namely namespace + project name |
CI_PROJECT_PATH_SLUG | 9.3 | all | $CI_PROJECT_PATH is lowercase, except for 0-9 and a-z, replaced with – |
CI_PIPELINE_URL | 11.0 | 0.5 | Pipeline detailed URL |
CI_PROJECT_URL | 8.10 | 0.5 | Access the HTTP address of the project |
CI_PROJECT_VISIBILITY | 10.3 | all | Project visibility (internal, private, public) |
CI_REGISTRY | 8.10 | 0.5 | If the registry is enabled, it returns the address of GitLab’s registry |
CI_REGISTRY_IMAGE | 8.10 | 0.5 | If the registry is enabled for a project, it returns the registry address associated with that particular project |
CI_REGISTRY_PASSWORD | 9.0 | all | The password used to push containers to the GitLab container registry |
CI_REGISTRY_USER | 9.0 | all | The user name used to push containers to the GitLab container registry |
CI_SERVER | all | all | Mark execution in CI environment |
CI_SERVER_NAME | all | all | The name of the CI server used to coordinate the job |
CI_SERVER_REVISION | all | all | GitLab revisions for scheduling work |
CI_SERVER_VERSION | all | all | GitLab version for scheduling work |
CI_SHARED_ENVIRONMENT | all | 10.1 | The markup work is performed in a shared environment |
GET_SOURCES_ATTEMPTS | 8.15 | 1.9 | The number of times you tried to get the resource running the job |
GITLAB_CI | all | all | The job is performed in a GitLab CI environment |
GITLAB_USER_EMAIL | 8.12 | all | Email for the user who started work |
GITLAB_USER_ID | 8.12 | all | Id of the user that started work |
GITLAB_USER_LOGIN | 10.0 | all | The login username of the user who started the job |
GITLAB_USER_NAME | 10.0 | all | The real name of the user who started the job |
RESTORE_CACHE_ATTEMPTS | 8.15 | 1.9 | The number of times I tried to rest |
Because there is no localization version, so get online translation, just a general idea and detailed information, you can see the website: https://docs.gitlab.com/ce/ci/variables/README.html
However, from the above variables defined by runner himself, there are really a lot of very practical variables, before writing scripts are not variables, their own wheels ~
summary
This paper mainly introduces some common instructions of. Gitlab-ci. yML. The most common ones are the variables defined by Jobs, stages and Runner.
Introduced here, this paper mainly introduces the commonly used, the rest, can go to the website: https://docs.gitlab.com/ce/ci/yaml/README.html
In the next article, I will introduce how to get the information submitted on GitLab CI every time and send it to the nail notification. This is not difficult, and interested students can think about how to do it.
Thank you.