preface

Before, there was a special article about gitlab CI, in which gitlab-CI.yML was introduced. It may be questioned why we should write another article about YML?

The main reason is that I encountered many problems with Gitlab-CI before, and it was not easy to explore along the way. Although I learned from the previous experience, I still encountered indescribable problems, because I wanted to understand this part again as a note. On an article about gitlab ci introduction links: https://juejin.cn/post/6844903609864290311

Gitlab-ci operation introduction

It consists of the following two modules:

gitlab-ci server
gitlab-ci-runner
Copy the code

Among them, Gitlab-CI Server is responsible for scheduling, triggering runners, and getting returned results. The Gitlab-Ci-runner is responsible for running ci automation (testing, compiling, packaging, etc.).

The basic process is:

The user submits the code to check if there is a. Gitlab-ci. yml file. If so, runner is called to execute the script to get the result returned.Copy the code

.gitlab-ci.yml

Starting with version 7.12, GitLab CI uses YAML files (.gitlab-ci.yml) to manage project configuration. This file resides at the root of the project repository and defines how the project is built.

The YAML file defines a set of tasks with constraints before starting the build. These tasks start with the task name and must contain at least a script part:

job1:
  script: 
      - echo "jb1"
  
job2:
  script: 
      - echo "jb2"
Copy the code

The example above is a simple CI configuration with two separate tasks, each performing echo output.

Script can execute system commands directly (e.g../configure; make; Make install) or execute the script directly (test.sh)

The task is taken over by Runners and performed by the runner on the server. Each task is executed independently.

Example 1:

# define job job1: stage: test script: -echo "I am jb1" -echo "I am in test stage" - echo "I am jb2" - echo "I am in build stage"Copy the code

According to the definition in stages, the build stage should be run before the test stage. Therefore, the stage: Build jobs should be run first and the stage:test jobs should be run later.

Here’s another complicated example 2:

Image: Ruby :2.1 services: -postgres before_script: -bundle install after_script: -rm stages: - build - test - deploy job1: stage: build script: - execute-script-for-job1 only: - master tags: - docker5+4X5=25Copy the code

The reserved fields listed below cannot be defined as job names:

The keyword	Whether must	describe
image	no	For docker images, view docker documents
services	no	For docker services, view docker documents
stages	no	Define the Construction phase
types	no	Alias of stages (deprecated)
before_script	no	Defines the commands to run before each job
after_script	no	Defines the commands to run after each job
variable	no	Defining build variables
cache	no	Defines a list of files that can be used in subsequent runs

The keywords image and services allow the use of a custom Docker image and a set of services and can be used throughout the job cycle.

Before_script Before_script is used to define the commands to run before all jobs, including deploy(deploy) jobs, but after fixing artifacts. It can be an array or a multi-line string.

after_script

GitLab 8.7 was introduced and required GitLab Runner V1.2Copy the code

After_script is used to define the commands to run after all jobs. It must be an array or a multi-line string;

Stages Defines stages that can be called by a job.

The order of elements in stages determines the order in which jobs are executed:

1. Jobs of the same stage can be executed in parallel. 2. The job of the next stage is executed after the job of the previous stage succeeds.Copy the code

The job of the next stage will be executed after the job of the previous stage succeeds. Because of this, many interdependent steps can be set on the stage without judging whether the job is successful or not. If the job can be executed, it must be successful.

Look at example 1:

stages:
    - build
    - test
Copy the code

• 1. First of all, allbuildAll jobs are executed in parallel
• 2. allbuildAfter jobs successfully executed,testThe jobsWill only beStart parallel execution
• 3. AlltestThe commit is marked assuccess
• 4. If any of the pre-jobs fail, commit is marked asfailedAnd the next jobs of stages will not be implemented.

Here are two special examples:

1. If stages are not defined in. Gitlab-ci. yml, jobs stages are defined as build, test, and deploy 2 by default. If a job does not specify a stage, the task is assigned to the Test stageCopy the code

types

Deprecated and will be removed in 10.0. "Stages" is used instead of "stages". It has the same meaning as "stages"Copy the code

Variable GItLab CI allows variables to be added to.gitlab-ci.yml files and to work in the job environment. Because these configurations are stored in git repositories, it is best to store non-sensitive configurations of the project, such as:

variables:
  jb: "jb is here"
Copy the code

Note that the: is followed by a space, which would be a syntax error. These variables can be used by subsequent commands and scripts.

In addition to user-defined variables, Runner can also define its own variables. A good example is CI_COMMIT_REG_NAME, whose value represents the branch or tag name used to build the project. In addition to setting variables in.gitlab-ci.yml, you can also set private variables on the Gitlab interface.

Cache specifies the files or directories that need to be cached between jobs. Only paths within the project workspace can be used.

Caching in Charge and Job is enabled by default since GitLab 9.0

If the cache is defined outside of jobs’ scope, it is a global cache and all Jobs can use it.

Caches files in Git that are not tracked:

rspec:
  script: test
  cache:
    untracked: true
Copy the code

Caches files in binaries that are not tracked by Git:

rspec:
  script: test
  cache:
    untracked: true
    paths:
    - binaries/
Copy the code

The cache Key directive allows us to define the cache scope (affinity), which can be a single cache for all jobs, per job, per branch, or wherever you see fit.

Cache :key can use any of the predefined variables

Caches each job:

cache:
  key: "$CI_JOB_NAME"
  untracked: true
Copy the code

Cache each branch:

cache:
  key: "$CI_COMMIT_REF_NAME"
  untracked: true
Copy the code

Cache each job and each branch:

cache:
  key: "$CI_JOB_NAME/$CI_COMMIT_REF_NAME"
  untracked: true
Copy the code

Cache each branch and each stage:

cache:
  key: "$CI_JOB_STAGE/$CI_COMMIT_REF_NAME"
  untracked: true
Copy the code

If Windows Batch is used to run the script, use % instead of $

cache:
  key: "%CI_JOB_STAGE%/%CI_COMMIT_REF_NAME%"
  untracked: true
Copy the code

Jobs

.gitlab-ci.yml allows specifying an unlimited number of jobs. Each job must have a unique name, and it cannot be the keyword mentioned above. Job is a list of parameters that define the behavior of a job.

job_name:
  script:
    - rake spec
    - coverage
  stage: test
  only:
    - master
  except:
    - develop
  tags:
    - ruby
    - postgres
  allow_failure: true
Copy the code

keywords	If necessary	describe
script	yes	Runner A command or script executed
image	no	Docker image used, see using docker image
services	no	Docker service used, check using docker image
stage	no	Define job Stage (default: test)
type	no	Alias for stage (deprecated)
variables	no	Define job-level variables
only	no	Define a list of Git branches and create jobs for them
except	no	Define a list of Git branches without creating a job
tags	no	Defines a list of tags that specify which Runner to select (Runner also sets tags)
allow_failure	no	Allow the job to fail. A failed job does not affect the COMMIT status
when	no	Define when to start a job. It can be on_success, on_failure, always, or manual
dependencies	no	Define job dependencies so that they can pass artifacts to each other
cache	no	Defines a list of files that should be cached between subsequent runs
before_script	no	Overrides a set of commands executed before a job
after_script	no	Overrides a set of commands to execute after a job
environment	no	Define the name of the environment in which this job is deployed
coverage	no	Defines code coverage Settings for a given job

Script Script is a YAML script executed by Runner.

job:
  script: "bundle exec rspec"
Copy the code

This parameter can also contain multiple commands in an array:

job:
  script:
    - uname -a
    - bundle exec rspec
Copy the code

Note here that if a script command contains a colon:, it needs to be wrapped in double quotes so that the YAML parser can correctly parse it as a string, not as a key:value pair. Use the following special characters with caution:

: {and}, [and], &, *, #,,,? , | -- - <, > =,! .Copy the code

Stage Stages allow a group of jobs to enter different stages. Jobs will parallel at the same stage; For more usage, click here;

Tags Tags can perform Jobs by selecting specific Runners from all Runners that are allowed to run this project. In the process of registering Runner, we can set the tag of Runner, such as Ruby, Postgres, development.

Tags can be used to run jobs by specifying special Runners via tags:

job:
  tags:
    - ruby
    - postgres
Copy the code

In the example above, you need to ensure that the Runner building the job must define the tags Ruby and Postgres.

Allow_failure Allow_failure can be used when you want to set a job to fail without affecting subsequent CI components. Failed jobs do not affect the commit status.

In the following example, job1 and joB2 will run side by side. If job1 fails, it will not affect the next stage in progress because allow_failure: true is set.

job1:
  stage: test
  script:
  - execute_script_that_will_fail
  allow_failure: true

job2:
  stage: test
  script:
  - execute_script_that_will_succeed

job3:
  stage: deploy
  script:
  - deploy_to_staging
Copy the code

when when is used to implement jobs that are run in case of failure or despite the failure. Used to implement a job that runs when it fails or fails.

You can set the following values:

On_success - This is performed only when all work on the previous stages is successful. This is the default value. On_failure - Executed when any jobs in current stages fail. Always - Executes regardless of the jobs state in the previous stages. 'manual' - Manual execution.Copy the code

Here’s an example:

stages:
- build
- cleanup_build
- test
- deploy
- cleanup

build_job:
  stage: build
  script:
  - make build

cleanup_build_job:
  stage: cleanup_build
  script:
  - cleanup build when failed
  when: on_failure

test_job:
  stage: test
  script:
  - make test

deploy_job:
  stage: deploy
  script:
  - make deploy
  when: manual

cleanup_job:
  stage: cleanup
  script:
  - cleanup after jobs
  when: always
Copy the code

Script description:

• ‘cleanup_build_job’ is executed only if build_job fails.
• ‘cleanup_job’ is executed whether the previous job failed or succeeded.
• Deploy_jobs can be executed manually from the GitLab interface.

Coverage allows you to configure that code coverage will extract output from the job. Here the regular expression is the only valid value. Therefore, a string must be preceded by a/to indicate a correct regular expression rule. Special strings need to be escaped.

Here’s an example:

job1:
  coverage: '/Code coverage: \d+\.\d+/'
Copy the code

Git Strategy

Introduced as an experimental feature in GitLab 8.9. It may be changed or completely removed in future releases. GIT_STRATEGY requires GitLab Runner V1.7 +.Copy the code

You can set GIT_STRATEGY to get the latest code, either in the global Variables module or in the variables module of a single job. If not, the default value is used from the project.

The values you can set are: clone, fetch, and None.

Clone is the slowest option. It clones the entire repository from scratch, including every job, to ensure that the project workspace is original.

variables:
  GIT_STRATEGY: clone
Copy the code

When it reuses the project workspace, the FETCH is faster (and returns a clone if it doesn’t exist). Git clean is used to undo any changes made to the previous job, and git fetch is used to get a commit from the previous job to now.

variables:
  GIT_STRATEGY: fetch
Copy the code

None also reuses the project workspace, but it skips all Git operations (including pre-GitLab Runner clone scripts, if any). This is mainly used to manipulate job artifacts (for example, deploy).

variables:
  GIT_STRATEGY: none
Copy the code

Git Checout When GIT_STRATEGY is set to clone or fetch, you can use the GIT_CHECKOUT variable to specify whether Git checkout should be run. If not specified, it defaults to true. Like GIT_STRATEGY, it can be set in the global variables or in the variables of a single job.

If set to false, Runner will:

• Fetch – Updates the repository and keeps a working copy in the current version,
• Clone – Clones the repository and keeps a working copy in the default branch.

Setting this to true will mean that both the Clone and FETCH policies will cause Runner to perform project workspace updates to the latest

variables:
  GIT_STRATEGY: clone
  GIT_CHECKOUT: false
script:
  - git checkout master
  - git merge $CI_BUILD_REF_NAME
Copy the code

Git Submodule Strategy The GIT_SUBMODULE_STRATEGY variable is used to determine whether or how Git submodules are introduced when code is pulled prior to build. Its available values are: None, normal, and Recursive: 1) None means that submodules will not be introduced when project code is pulled. This is the default. 2) Normal means that only top-level submodules are introduced. It is equivalent to

git submodule sync
git submodule update --init
Copy the code

3) Recursive means that all submodules (including submodules of submodules) are imported, which is equivalent to:

git submodule sync --recursive
git submodule update --init --recursive
Copy the code

Note: For this function to work properly, the submodule must be configured (in.gitModules) with one of the following:

Accessible common warehouse HTTP (s) address, a real address accessible to another warehouse on the same GitLab server.Copy the code

Job stages Attempts the following stages will be executed according to the number of attempts you set:

variable	describe
GET_SOURCES_ATTEMPTS	Number of attempts to obtain job sources
ARTIFACT_DOWNLOAD_ATTEMPTS	The number of attempts to download Artifacts
RESTORE_CACHE_ATTEMPTS	The number of attempts to rebuild the cache

The default is one attempt.

Example:

variables:
  GET_SOURCES_ATTEMPTS: 3
Copy the code

Pages Pages is a special job for uploading static content to GitLab that can be used to serve your web site. It has a special syntax, so it must meet the following two requirements:

Any static content must be placed in the public/ directory artifacts must be defined in the public/ directoryCopy the code

The following example moves all files from the project root directory to the public/ directory. .public workflow is cp, and it does not loop over public/ itself.

pages:
  stage: deploy
  script:
  - mkdir .public
  - cp -r * .public
  - mv .public public
  artifacts:
    paths:
    - public
  only:
  - master
Copy the code

Variables defined by Runner

In the introduction of variable above, it was mentioned that Runner also has variables defined by himself, as follows:

variable	Gitlab version	runner	describe
CI	all	0.4	Mark that the job is executed in a CI environment
CI_COMMIT_REF_NAME	9.0	all	The branch or tag name for the build project
CI_COMMIT_REF_SLUG	9.0	all	$CI_COMMIT_REF_NAME lowercase, except 0-9 and a-z, replaced with –
CI_COMMIT_SHA	9.0	all	Build the commit SHA value of the project
CI_COMMIT_TAG	9.0	0.5	The tag name of the commit that builds the project, which is only used when building tags.
CI_COMMIT_MESSAGE	10.8	all	Complete commit message.
CI_COMMIT_TITLE	10.8	all	Submitted title – The full first line of the message
CI_COMMIT_DESCRIPTION	10.8	all	Description of the submission
CI_CONFIG_PATH	9.4	0.5	CI configuration file path. The default value is.gitlab-ci.yml
CI_DEBUG_TRACE	all	1.7	Whether to enable the debugging tracing function
CI_DEPLOY_USER	10.8	all	The authentication user name for the GitLab deployment token
CI_DEPLOY_PASSWORD	10.8	all	Authentication password for the GitLab deployment token
CI_DISPOSABLE_ENVIRONMENT	all	10.1	Flags whether the job runs in a one-time environment
CI_ENVIRONMENT_NAME	8.15	all	The name of the environment to execute the job, such as produciton,dev,pre-production, etc
CI_ENVIRONMENT_SLUG	8.15	all	Simplified version of environment name for DNS, URL, Kubernetes tag, etc.
CI_ENVIRONMENT_URL	9.3	all	URL of the environment where the job is executed
CI_JOB_ID	9.0	all	Unique ID of the current job in the GitLab CI
CI_JOB_MANUAL	8.12	all	Indicates that the job is to be started manually
CI_JOB_NAME	9.0	0.5	The name of the job defined in.gitlab-ci.yml
CI_JOB_STAGE	9.0	0.5	The stage name defined in.gitlab-ci.yml
CI_JOB_TOKEN	9.0	1.2	The token used for authentication in the GitLab container registry
CI_JOB_URL	11.0	0.5	The URL for the work
CI_REPOSITORY_URL	9.0	all	Clone Git Repository URL
CI_RUNNER_DESCRIPTION	8.10	0.5	Description of runner saved in GitLab
CI_RUNNER_ID	8.10	0.5	Unique ID of the runner that is running
CI_RUNNER_TAGS	8.10	0.5	Running Runner Tags
CI_RUNNER_VERSION	all	10.6	GitLab Runner version of the current job being executed
CI_RUNNER_REVISION	all	10.6	GitLab Runner revision of the current job being executed
CI_RUNNER_EXECUTABLE_ARCH	all	10.6	GitLab GitLab Runner OS/ architecture for executable programs
CI_PIPELINE_ID	8.10	0.5	Unique ID of current pipeline within GitLab CI
CI_PIPELINE_TRIGGERED	all	all	Mark this job as triggered by a trigger
CI_PIPELINE_SOURCE	10.0	all	Indicate how to trigger the pipeline. The options available are: Push, Web, Trigger, Schedule, API, and Pipeline
CI_PROJECT_DIR	all	all	The full path to the cloned repository and the directory in which the job is running
CI_PROJECT_ID	all	all	Unique ID of the current project within the GitLab CI
CI_PROJECT_NAME	8.10	0.5	The name of the project currently being built (actually the project folder name)
CI_PROJECT_NAMESPACE	8.10	0.5 Project namespace currently being built (user name or groupname)
CI_PROJECT_PATH	8.10	0.5	The full project path, namely namespace + project name
CI_PROJECT_PATH_SLUG	9.3	all	$CI_PROJECT_PATH is lowercase, except for 0-9 and a-z, replaced with –
CI_PIPELINE_URL	11.0	0.5	Pipeline detailed URL
CI_PROJECT_URL	8.10	0.5	Access the HTTP address of the project
CI_PROJECT_VISIBILITY	10.3	all	Project visibility (internal, private, public)
CI_REGISTRY	8.10	0.5	If the registry is enabled, it returns the address of GitLab’s registry
CI_REGISTRY_IMAGE	8.10	0.5	If the registry is enabled for a project, it returns the registry address associated with that particular project
CI_REGISTRY_PASSWORD	9.0	all	The password used to push containers to the GitLab container registry
CI_REGISTRY_USER	9.0	all	The user name used to push containers to the GitLab container registry
CI_SERVER	all	all	Mark execution in CI environment
CI_SERVER_NAME	all	all	The name of the CI server used to coordinate the job
CI_SERVER_REVISION	all	all	GitLab revisions for scheduling work
CI_SERVER_VERSION	all	all	GitLab version for scheduling work
CI_SHARED_ENVIRONMENT	all	10.1	The markup work is performed in a shared environment
GET_SOURCES_ATTEMPTS	8.15	1.9	The number of times you tried to get the resource running the job
GITLAB_CI	all	all	The job is performed in a GitLab CI environment
GITLAB_USER_EMAIL	8.12	all	Email for the user who started work
GITLAB_USER_ID	8.12	all	Id of the user that started work
GITLAB_USER_LOGIN	10.0	all	The login username of the user who started the job
GITLAB_USER_NAME	10.0	all	The real name of the user who started the job
RESTORE_CACHE_ATTEMPTS	8.15	1.9	The number of times I tried to rest

Because there is no localization version, so get online translation, just a general idea and detailed information, you can see the website: https://docs.gitlab.com/ce/ci/variables/README.html

However, from the above variables defined by runner himself, there are really a lot of very practical variables, before writing scripts are not variables, their own wheels ~

summary

This paper mainly introduces some common instructions of. Gitlab-ci. yML. The most common ones are the variables defined by Jobs, stages and Runner.

Introduced here, this paper mainly introduces the commonly used, the rest, can go to the website: https://docs.gitlab.com/ce/ci/yaml/README.html

In the next article, I will introduce how to get the information submitted on GitLab CI every time and send it to the nail notification. This is not difficult, and interested students can think about how to do it.

Thank you.