The author:Clfeng

Learners’

Through the first two chapters, we have our own CI/CD environment and have mastered the core concepts of CI/CD. In this chapter we will take a closer look at the various CI/CD keywords and their functions.

Note: this section looks more like a configuration lookup section, which is long and boring. The author wrote this chapter for the purpose of the key is also the integrity of the overall knowledge. At the same time, the author also believes that although the study of a knowledge point should be as complete as possible, as far as possible to know all its content, even after reading all the details can not remember. But knowing that there is one, we are more likely to analyze the cause of the problem and come up with a more reasonable solution when it arises

.gitlab-ci.yml reference

The default configuration

default

Define dummy configuration for all jobs in pipeline, and relevant configuration items that can be defined:

  • after_script
  • artifacts
  • before_script
  • cache
  • image
  • interruptible
  • retry
  • services
  • tags
  • timeout

Global configuration

stages

Define the names of pipeline stages and the order in which they are executed

workflow

Control the whole pipeline operation

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: never
    - if: '$CI_PIPELINE_SOURCE == "push"'
      when: never
    - when: always
Copy the code
variables:
  DEPLOY_VARIABLE: "default-deploy"

workflow:
  rules:
    - if: $CI_COMMIT_REF_NAME = = $CI_DEFAULT_BRANCH
      variables:
        DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE
    - if: $CI_COMMIT_REF_NAME = ~ /feature/
      variables:
        IS_A_FEATURE: "true" # Define a new variable.
    - when: always # Run the pipeline in other cases

job1:
  variables:
    DEPLOY_VARIABLE: "job1-default-deploy"
  rules:
    - if: $CI_COMMIT_REF_NAME = = $CI_DEFAULT_BRANCH
      variables: # Override DEPLOY_VARIABLE defined
        DEPLOY_VARIABLE: "job1-deploy-production" # at the job level.
    - when: on_success # Run the job in other cases
  script:
    - echo "Run script with $DEPLOY_VARIABLE as an argument"
    - echo "Run another script if $IS_A_FEATURE exists"

job2:
  script:
    - echo "Run script with $DEPLOY_VARIABLE as an argument"
    - echo "Run another script if $IS_A_FEATURE exists"
Copy the code

include

You can use the include keyword to split the contents in YML to improve the readability of the configuration file

Keyword Method
local Include a file from the local project repository.
file Include a file from a different project repository.
remote Include a file from a remote URL. Must be publicly accessible.
template Include templates that are provided by GitLab.
include:local
include:
  - local: '/templates/.gitlab-ci-template.yml'

# shorthand
include: '.gitlab-ci-production.yml'

Use wildcards
include: 'configs/*.yml'
Copy the code
include:file
include:
  - project: "my-group/my-project"
    ref: main
    file: "/templates/.gitlab-ci-template.yml"

  - project: "my-group/my-project"
    ref: v1.0.0
    file: "/templates/.gitlab-ci-template.yml"

  - project: "my-group/my-project"
    ref: 787123b47f14b552955ca2786bc9542ae66fee5b # Git SHA
    file:
      - "/templates/.builds.yml"
      - "/templates/.tests.yml"
Copy the code
include:remote
include:
  - remote: "https://gitlab.com/example-project/-/raw/main/.gitlab-ci.yml"
Copy the code
include:template
include:
  - template: Android-Fastlane.gitlab-ci.yml
  - template: Auto-DevOps.gitlab-ci.yml
Copy the code

Common configuration

image

Specifies the docker image running job

image

  • Name # Name of the mirror
    • Entrypoint # EntryPoint for docker images

services

Specify the Docker service image

services – name – alias – entrypoint – comand

before_script

Commands that are executed before job script commands are executed are generally used to install commands that depend on preset environment variables. It shares a shell with script

script

Defines the commands that need to be executed when the job is executed by runner

after_script

A command executed after a job is executed that uses a shell other than script

stage

Specify the phase of the job

extends

Used to reuse configuration fragments

.only-important:
  variables:
    URL: "http://my-url.internal"
    IMPORTANT_VAR: "the details"
  rules:
    - if: $CI_COMMIT_BRANCH = = $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_BRANCH = = "stable"
  tags:
    - production
  script:
    - echo "Hello world!"

.in-docker:
  variables:
    URL: "http://docker-url.internal"
  tags:
    - docker
  image: alpine

rspec:
  variables:
    GITLAB: "is-awesome"
  extends:
    - .only-important
    - .in-docker
  script:
    - rake rspec
Copy the code

The rspec job is evaluated as follows:

rspec:
  variables:
    URL: "http://docker-url.internal"
    IMPORTANT_VAR: "the details"
    GITLAB: "is-awesome"
  rules:
    - if: $CI_COMMIT_BRANCH = = $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_BRANCH = = "stable"
  tags:
    - docker
  image: alpine
  script:
    - rake rspec
Copy the code

incluce

Use include to include external YAML files in your CI/CD configuration. You can split a long Gitlab-ci.yml file into multiple files to improve readability, or to reduce duplication of the same configuration in multiple places.

# included.yml
.template:
  script:
    - echo Hello!
Copy the code
# .gitlab-ci.yml
include: included.yml

useTemplate:
  image: alpine
  extends: .template
Copy the code

rules

* Use rules to contain or eliminate jobs in charge.

Note: Cannot be used with only/except

rules – if – exists – allow_failure – variables – when

See related contentCI/CD Half bucket of water (2)The jobs section

only/except

You can use only and except to control when to add jobs to pipelines.

Note: Only /except is no longer focused on iterative development, rules are recommended

See related contentCI/CD Half bucket of water (2)The jobs section

tags

Use tags to select a particular runner from all available runners in the project

variables:
  KUBERNETES_RUNNER: kubernetes

  job:
    tags:
      - docker
      # Here is interesting, we can dynamically select runner by setting variables
      - $KUBERNETES_RUNNER
    script:
      - echo "Hello runner selector feature"
Copy the code

allow_failure

You can use allow_failure to allow a job to fail without affecting subsequent cis

allow_failure:exit_codes

Allow_failure :exit_codes allows you to configure which exit codes are not considered failures

test_job_1:
  script:
    - echo "Run a script that results in exit code 1. This job fails."
    - exit 1
  allow_failure:
    exit_codes: 137

test_job_2:
  script:
    - echo "Run a script that results in exit code 137. This job is allowed to fail."
    - exit 137
  allow_failure:
    exit_codes:
      - 137
      - 255
Copy the code

when

  • On_success: The current job is executed after all stages on the current stage are successful
  • On_failure: Execute the current job if at least one job in the current stage fails
  • Always: Perform jobs regardless of the state of the previous stages
  • “Manual” : indicates a direct manual job
  • Delayed: Job execution is delayed after the specified time
  • never:
    • With the rules keyword, job is not executed
    • Work with the Workflow: Rules keyword and do not execute pipeline
# when: Delayed example
timed rollout 10% :
  stage: deploy
  script: echo 'Rolling out 10% ... '
  when: delayed
  start_in: 30 minutes
# start_in Valid value
# '5'
# 5 seconds
# 30 minutes
# 1 day
# 1 week
Copy the code

environment

Environment-related attributes

  • Name: Sets the name of an environment

  • Url: specifies the URL of the deployment environment

  • On_stop: you can set a job for clearing the environment after executing the job of the deployment environment

  • Auto_stop_in: Specifies how long the environment expires

  • action

    • start
    • prepare
    • stop

  • Kubernetes: Configure to deploy the project to the Kubernetes cluster

default:
  image: centos:7
  tags:
    - clf-cicd-runner

stages:
  - deploy

deploy_prod:
  stage: deploy
  script:
    - echo "Deploy to production server"
  environment:
    name: develop
    url: https://dev.com
Copy the code

cache

Have learnedCI/CD Half a Bucket of Water (2)Can be skipped

Related keywords

  • Paths: Specifies cached files or directories
  • Key: unique identifier of the cache
    • Files: When a file in the specified file changes, a new key is generated
    • Prefix: Adds the prefix tocache:key:filesBefore calculating the hash
  • Untracked: Uses untracked: true to cache all untracked files in the Git repository
  • When: Define the cache under what conditions. Optional values: on_success, on_failure, always
  • Policy: defines the cache upload and download policies. The options are pull, push, and pull-push
prepare-dependencies-job:
  stage: build
  cache:
    key: gems
    paths:
      - vendor/bundle
    policy: push
  script:
    - echo "This job only downloads dependencies and builds the cache."
    - echo "Downloading dependencies..."

faster-test-job:
  stage: test
  cache:
    key: gems
    paths:
      - vendor/bundle
    policy: pull
  script:
    - echo "This job script uses the cache, but does not update it."
    - echo "Running tests..."
Copy the code

cache:key

Use the cache:key keyword to provide a unique identifying key for each cache. All jobs using the same cache key use the same cache, including in different pipelines.

Note: One key is corresponding to the cache file, which can be reused in different jobs of the same pipeline, or in different pipelines

artifacts

Have learnedCI/CD Half a Bucket of Water (2)Can be skipped

Artifacts Related keywords

  • Name: Define the name of the artifact
  • Paths: Artifacts Path to the cache file
  • Dependencies: Defines current job artifacts download the previous jobs of stages
  • Exclude: excludes the cached file
  • Expire_in: Use expire_in to specify how long job artifacts should be stored before expiration and deletion
  • Expose_as: Expose Job Artifacts in the merge request UI using the expose_AS keyword
  • Public: Sets whether artifacts are publicly available, which means that both anonymous users and visitors can download them in the public pipeline
  • Untracked: Adds any Git untracked files as artifacts (as well as paths defined in artifacts: Paths)
  • When: defines the conditions for uploading artifacts
  • reports
    • api_fuzzing
    • cobertura
    • . [Other Viewer]

Use artifacts to specify a list of files and directories that are attached to the job when it succeeds, fails, or always.

Use artifacts to specify a list of files and directories to attach to the job if it succeeds, fails, or always.

The artifacts are sent to GitLab after the job finishes. They are available for download in the GitLab UI if the size is not larger than the maximum artifact size.

After the job is completed, the artifacts are sent to GitLab. If the size is not greater than the maximum artifacts size, they can be downloaded in the GitLab UI.

By default, jobs in later stages automatically download all the artifacts created by jobs in earlier stages. You can control artifact download behavior in jobs with dependencies.

By default, the later Jobs automatically download all artifacts created by the earlier jobs. You can control the download behavior of artifacts in Jobs by using the dependencies keyword.

When using the needs keyword, jobs can only download artifacts from the jobs defined in the needs configuration.

When using the NEEDS keyword, Jobs can only download artifacts from Jobs defined in the requirements configuration.

Job artifacts are only collected for successful jobs by default, and artifacts are restored after caches.

By default, job Artifacts are only collected for successful jobs and restored after caches.

Key summary:

  • Store in GitLab
  • Artifacts created for the previous job are automatically downloaded for the later job and can be used directly
  • The download behavior of artifacts can be controlled using the dependencies keyword
  • If the job has the NEEDS keyword, only job artifacts specified by the NEEDS keyword are downloaded
artifacts:dependencies

By default, the job downloads all job artifacts from the previous stage. You can specify job artifacts to download by using dependencies.

build:osx:
  stage: build
  script: make build:osx
  artifacts:
    paths:
      - binaries/

build:linux:
  stage: build
  script: make build:linux
  artifacts:
    paths:
      - binaries/

test:osx:
  stage: test
  script: make test:osx
  dependencies: # Specify that only build: OSX-generated artifacts be downloaded
    - build:osx

test:linux:
  stage: test
  script: make test:linux
  dependencies: # specify that only build: Linux-generated artifacts be downloaded
    - build:linux

# deploy Job is not specified, so by default, go back and download all the previous artifacts generated by the job
deploy:
  stage: deploy
  script: make deploy
Copy the code
artifacts:exclude

Exclude prevents files from being added to artifacts

artifacts:
  paths:
    - binaries/
  exclude:
    - binaries/**/*.o
Copy the code
artifacts:expire_in

Use expire_in to specify how long job artifacts should be stored before expiration and deletion

job:
  artifacts:
    expire_in: 1 week

    Other types of values that can be specified
    # expire_in: '42'
    # expire_in: 42 seconds
    # expire_in: 3 mins 4 sec
    # expire_in: 2 hrs 20 min
    # expire_in: 2h20min
    # expire_in: 6 mos 1 day
    # expire_in: 47 yrs 6 mos and 4d
    # expire_in: 3 weeks and 2 days
    # expire_in: never
Copy the code
artifacts:expose_as
test:
  script: ["echo 'test' > file.txt"]
  artifacts:
    expose_as: "artifact 2"
    paths: ["file.txt"]
Copy the code

The effect is shown below:

artifacts:paths

Defines the path to a cache file or directory. The path is relative to the project directory and cannot be linked directly outside the project directory.

artifacts:untracked

Add any Git untracked files as artifacts (as well as the paths defined in artifacts: Paths) using artifacts: Untracked. Artifacts: Untracked Ignores the configuration in the.gitignore file for the repository.

artifacts:
  untracked: true
  paths:
    - binaries/
Copy the code
artifacts:when

Define the conditions for uploading artifacts

Optional value:

  • On_success (default): Upload artifacts if the job is successfully executed
  • On_failure: Upload artifacts when job execution fails
  • Always: Always upload artifacts

retry

You can use the retry command to configure the number of job retries after a job fails

Relevant properties

  • Max: indicates the maximum number of retries
  • When: failure to retry
test:
  script: rspec
  retry: 2
Copy the code
test:
  script: rspec
  retry:
    max: 2
    when: runner_system_failure
Copy the code

timeout

Use timeout to configure a timeout for a particular job

build:
  script: build.sh
  timeout: 3 hours 30 minutes

test:
  script: rspec
  timeout: 3h 30m
Copy the code

parallel

Parallel configures the number of job instances to run in parallel

dast_configuration

Use the dast_configuration keyword to specify the site profile and scanner profile to use in the CI/CD configuration.

coverage

How to extract coverage from job output logs

job1:
  script: rspec
  coverage: '/Code coverage: \d+\.\d+/'
Copy the code

trigger

Use trigger to define a downstream pipeline trigger. When GitLab starts a trigger job, a downstream pipeline is created. Use trigger to define downstream pipeline triggers. When GitLab starts the trigger Job, a downstream pipe is created.

See Pipeline’s Child/Parent architecture

# Basic trigger syntax for multi-project pipeline
rspec:
  stage: test
  script: bundle exec rspec

staging:
  stage: deploy
  trigger: my/deployment Point to the full path of other projects


  # trigger:
  # project: my/deployment
  # branch: stable can point to a specific branch
Copy the code
# trigger syntax for child pipeline
trigger_job:
  trigger:
    include: path/to/child-pipeline.yml
Copy the code
trigger_job:
  trigger:
    include:
      - local: path/to/child-pipeline.yml
    strategy: depend # Map the state of the sub-pipeline to the upstream pipeline via strategy: Depend
Copy the code
# dynamically generated configuration file:
generate-config:
  stage: build
  script: generate-ci-config > generated-config.yml
  artifacts:
    paths:
      - generated-config.yml

child-pipeline:
  stage: test
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config
Copy the code

interruptible

Use Interruptible to indicate that if running jobs become redundant due to the newer pipelines, the job should be eliminated.

Note: By default, if a new pipeline is running on the same branch, unrunning jobs in the old pipeline will be canceled, while running jobs will not be canceled. Interruptible Allows the ongoing job to be canceled

resource_group

Sometimes, running multiple jobs or pipelines in an environment at the same time can cause errors during deployment. To avoid these errors, use the resource_group attribute to ensure that the running program does not run some jobs at the same time.

When the resource_group keyword is defined for the job in the.gitlab-ci.yml file, job execution is mutually exclusive across different pipelines for the same project. If multiple jobs in the same resource group are added to the queue at the same time, the running program selects only one job. Other jobs wait until the resource_group is idle.

deploy-to-production:
  script: deploy
  resource_group: production
Copy the code

Note: When using resource_group with trigger, you need to configure trigger: Strategy: Depend to ensure that the lock is not released until the child pipeline finishes executing

inherit

Use inherit: Control the default values of global definitions and inheritance of variables.

default: The global default configuration is defined
  image: Ruby: "2.4"
  before_script:
    - echo Hello World

variables: # global variables defined
  DOMAIN: example.com
  WEBHOOK_URL: https://my-webhook.example.com

rubocop:
  inherit:
    default: false Do not inherit the default configuration
    variables: false Do not inherit global variables
  script: bundle exec rubocop

rspec:
  inherit:
    default: [image] Inherit the image field from the global default configuration
    variables: [WEBHOOK_URL] Inherit the WEBHOOK_URL variable from the global variable
  script: bundle exec rspec

capybara:
  inherit: Inherit the default configuration, but do not inherit global variables
    variables: false
  script: bundle exec capybara

karma:
  inherit:
    default: true
    variables: [DOMAIN]
  script: karma
Copy the code

YAML-specific features

Yaml features such as &, *, << can be used to simplify.gitlab-ci.yml code

# Hidden yaml configuration that defines an anchor named 'job_configuration'
How to interpret the above comment?
If you want to hide a job, you can comment out the configuration of the job by adding a. Then the job won't be executed
# Does this make sense why the above comment says Hidden Yaml Configuration? Further, it can be understood in conjunction with the next example
.job_template: &job_configuration
  image: Ruby: 2.6
  services:
    - postgres
    - redis

test1:
  < < : *job_configuration # Merge the contents of the 'job_configuration' alias
  script:
    - test1 project

test2:
  < < : *job_configuration # Merge the contents of the 'job_configuration' alias
  script:
    - test2 project
Copy the code
default:
  image: centos:7
  tags:
    - clf-cicd-runner

stages:
  - test

test_job1: &job_configuration
  stage: test
  before_script:
    - echo "run test job 1 before script"
  script:
    - echo "run test job 1"

test_job2:
  < < : *job_configuration
  stage: test
  script:
    - echo "run test job 2"
# test_job2 produces two sentences:
# run test job 1 before script
# run test job 2
Copy the code
.job_template: &job_configuration
  script:
    - test project
  tags:
    - dev

.postgres_services:
  services: &postgres_configuration
    - postgres
    - ruby

.mysql_services:
  services: &mysql_configuration
    - mysql
    - ruby

test:postgres:
  < < : *job_configuration
  services: *postgres_configuration
  tags:
    - postgres

test:mysql:
  < < : *job_configuration
  services: *mysql_configuration


# -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
# The expanded version is:
.job_template:
  script:
    - test project
  tags:
    - dev

.postgres_services:
  services:
    - postgres
    - ruby

.mysql_services:
  services:
    - mysql
    - ruby

test:postgres:
  script:
    - test project
  services:
    - postgres
    - ruby
  tags:
    - postgres

test:mysql:
  script:
    - test project
  services:
    - mysql
    - ruby
  tags:
    - dev
Copy the code

YAML anchor points for scripts

.some-script-before: &some-script-before
  - echo "Execute this script first"

.some-script: &some-script
  - echo "Execute this script second"
  - echo "Execute this script too"

.some-script-after: &some-script-after
  - echo "Execute this script last"

job1:
  before_script:
    - *some-script-before
  script:
    - *some-script
    - echo "Execute something, for this job only"
  after_script:
    - *some-script-after

job2:
  script:
    - *some-script-before
    - *some-script
    - echo "Execute something else, for this job only"
    - *some-script-after
Copy the code

YAML anchor points for variables

# global variables
variables: &global-variables
  SAMPLE_VARIABLE: sample_variable_value
  ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value

# a job that must set the GIT_STRATEGY variable, yet depend on global variables
job_no_git_strategy:
  stage: cleanup
  variables:
    < < : *global-variables
    GIT_STRATEGY: none
  script: echo $SAMPLE_VARIABLE
Copy the code

Hide the jobs

When the job name begins with a., the job is not processed by CICD, similar to commenting out code

.hidden_job:
  script:
    - run test
Copy the code

! reference tags

Use! Reference Custom YAML tags select the keyword configuration from other job sections and reuse it in the current section. Unlike YAML anchors, you can also use them! The Reference tag to reuse configurations from the contained configuration file.

# setup.yml
.setup:
  script:
    - echo creating environment
Copy the code
# .gitlab-ci.yml
include:
  - local: setup.yml

.teardown:
  after_script:
    - echo deleting environment

test:
  script:
    - ! reference [.setup.script]
    - echo running my own command
  after_script:
    - ! reference [.teardown.after_script]
Copy the code

** Summary: ** Through this chapter, we learned a lot about the special features of YAML, which make it easy to maintain our configuration files.

conclusion

After reading down is not feel this content is too much, also may not remember how many things. In fact, it doesn’t matter, it is good to know that there is such a thing, but when the problem occurs, you can naturally recall, whether there is such a thing can be used, but later in the back check is. By this time, we may think we can do anything, hahaha ~ but it is not enough!! In the next chapter, we’ll take a look at Webhooks and the GitLab API.

Refer to the link

Docs.gitlab.com/ee/ci/index…