After the company set up an internal GitLab platform, the front end activity projects were migrated from SVN to GitLab. This article describes how to implement an automated build and release based on GitLab CI/CD.

In the process of migrating from SVN to GitLab and accessing GitLab CI/CD, I would like to express my special thanks to the support of the publishing system and server side students.

I. Current construction and release process

In this section, we first describe the benefits of using GitLab CI/CD and then describe the build and release processes before and after using GitLab CI/CD.

1. Team revenue

  1. The average release time was reduced from 5 minutes to 1.5 minutes. From 5 minutes of manual work all the way to 1.5 minutes of merging branch code, automated build, and release.
  2. The front-end build is placed in CI/CD, which solves the problem that local builds can lead to inconsistent packaging of online code.

2. Build and release before using GitLab CI/CD

2.1 process

  1. After development, build locally and submit the packaged HTML file after build.
  2. Open the publishing system and publish the HTML file.

2.2 pain points

  1. The operation process of releasing code is long and time-consuming. After development, a lot of manual operation is needed to release the code, and the release system is slow. Each release takes about 5 minutes.
  2. Everyone’s local environment is inconsistent, and there is a risk of inconsistent distribution by people with different code online.

2.3 Time consuming points requiring manual operation

A total of at least eight manual operations are required

Local build:

  1. Run the build command and wait for the build to complete.
  2. When the build is complete, submit the packaged HTML file.

Release code:

  1. Open the publishing system
  2. Select the release project and environment
  3. Open the publishing page
  4. Select publish file
  5. Fill in the release information
  6. Click Ok to publish

3. Construction and release after using GitLab CI/CD

Release code in one step: just merge the development branch to the corresponding branch of the release environment, after submitting the branch, GitLab CI/CD will automatically build and publish.

What is GitLab CI/CD

In this section, we will briefly introduce GitLab CI/CD and then introduce how to build a GitLab CI/CD from scratch.

1. Brief introduction of GitLab CI/CD

As shown in the figure below, after the code is submitted to GitLab, the pipeline will be triggered for automatic construction and release after the specified conditions are met.

A pipeline can be understood as a build task and can contain multiple processes, such as downloading dependencies, running tests, compiling, and deploying. When a pipeline fires, is divided into several processes, and what each process does is defined in the project’s.gitlab-ci.yml file.

For our active project, pipeline mainly includes three processes: dependency download, build code, and release to the server. The follow-up practice will be introduced in detail.

2. GitLab CI/CD overall process

The specific pipeline process and operation of GitLab CI/CD are stated in.gitlab-ci.yml file. After pipeline is triggered, GitLab Runner will run according to.gitlab-ci.yml file. It will return to the GitLab system after running.

2.1. Gitlab – ci. Yml file

The.gitlab-ci.yml file is a declarative file that defines the stages of the Gitlab CI/CD process and what each stage does.

As for what to do and how to do, we mainly use the command line and script operations, which will be described in detail in the practice section later.

If there’s some logic involved, we’ll use scripts, and our project will use Shell scripts, Python scripts.

2.1 GitLab Runner

GitLab Runner is the CI execution environment, responsible for executing the Gitlab-ci.yML file and returning the results to the GitLab system. Runner can have various forms, such as Docker, virtual machine or shell. The way is selected when registering Runner.

3. Build a GitLab CI/CD from scratch

To understand the process, set up a project on GitLab and run through the CI/CD process.

3.1 Create a GitLab project

  1. Go to ITlab: gitlab.com/
  2. Create your own project

3.2 configuration Runner

GitLab provides some shared runners, so we don’t have to deal with runners.

3.3 Create the.gitlab-ci.yml file

  1. Pull items to local
  2. Create a.gitlab-ci.yml file in the project root directory
  3. Submit the.gitlab-ci.yml file
  4. In the CI/CD of the project, you can see the CI/CD in action

.gitlab-ci.yml file example

image: node
# define stages
stages:
  - build
  - test
# define job
 build Phase:
  stage: build
  script:
    - echo "build stage"
# define job
Publish to test environment:
  stage: test
  script:
    - echo "test stage"
Copy the code

3. New projects are connected to GitLab CI/CD process

The team needs to access GitLab CI/CD for a new project, first apply for the GitLab project, then ask the GitLab system maintainer to help configure Runner, write.gitlab-ci.yml file, and trigger the construction to see the SITUATION of CI/CD.

  1. New GitLab project
  2. Configuration GitLab Runner
  3. . Gitlab – ci. Yml file

At present, two new project routes have been connected to GitLab CI/CD, and the access condition is good. The operation process is relatively smooth according to the documents.

GitLab CI/CD practice

In the practice part, GitLab Runner and.gitlab-ci.yml files are mainly introduced here. The main processes and problems encountered and solutions are included in the introduction process of.gitlab-ci.yml files.

1. GitLab Runner

GitLab Runner is generally managed by GitLab system maintainer. After configuration, similar projects can be shared without modification. There is no specific introduction here, but mainly introduces the attention points in the process of use. For specific use, please refer to the GitLab Runner document.

1.1 GitLab Runner use process

  1. Download GitLab Runner
  2. Registered GitLab Runner
  3. Using GitLab Runner

1.2 GitLab Runner pay attention

In the process of using Runner, we encountered some problems. The problems and solutions are briefly introduced below without specific introduction.

1.2.1 After configuring Runner, push code starts pipeline, but it is always in Pending state

The error message is: This job is stuck, because you don’t have any active runners that can run This job

Runners can run untagged jobs from Settings→CI/CD→Runners Settings by default.

1.2.2 Types of GitLab Runner

There are three types of Runners. Shared Runners can be used in all projects in the whole system. After Group Runners are registered, different code bases under the same project are Shared. Using Specific Runners Consider whether Shared Runners and Group Runners need to be turned off.

  • Shared Runners
  • Specific Runners
  • Group Runners
1.2.3 Use Docker in GitLab CI

When deploying to Ali Cloud, you need to use Docker in GitLab CI/CD for mirroring and publishing. See Building Docker Images with GitLab CI/CD

1.2.4 Access Runner host directory in GitLab CI/CD

The Runner executor we used was Dokcer, and in Dokcer volumes we configured the directories we needed to access.

2.. gitlab – ci. Yml file

The activity project.gitlab-ci.yml file is as follows. The following is mainly through the.gitlab-ci.yml file of the activity project to introduce our practice process, detailed usage of.gitlab-ci.yml. Refer to the GitLab CI/CD Pipeline Configuration Reference document

2.1. Introduction to gitlab-ci. yML file

Image is the Docker base image that performs CI/CD dependencies. The image contains Node, Yarn, and Dalp (internal rsync tool).

Stages defines our pipeline processes as follows:

  1. Download the dependency phase pre_build
  2. Construction phase Build
  3. Release Phase Deploy

Stage States the current stage. It is used in stages

Variables are used to define variables

Before_script Operations performed before script

Script The operation that the current stage needs to perform

Changes specifies the stage trigger condition

Refs specifies the branches triggered by the stage

image: registry.huajiao.com/gitlab-ci/node-yarn:v1.4

variables:
  # $CI_PROJECT_PATH: the project ID that uniquely distinguishes this project from other projects
  # $CI_PROJECT_DIR: local project path
  # $PROCESS_PATH: temporary file directory (including logs and some temporary files)
  NODE_MODULES_PATH: /runner-cache/frontend/$CI_PROJECT_PATH/$CI_BUILD_REF_NAME/node_modules

stages:
  - pre_build
  - build
  - deploy

Download dependencies:
  before_script:
    If no node_modules file is available, create a node_modules file
    - /bin/bash ./ci/mkdir.sh $NODE_MODULES_PATH
    Softlink node_modules to the host
    - ln -s $NODE_MODULES_PATH .
    - cd webpack@next
  stage: pre_build
  script:
    - echo "yarn install"
    - yarn install  --network-timeout 60000
  only:
    changes:
      - webpack@next/package.json
    refs:
      - test
      - test-99
      - test-128
      - master
      - ci
      - feature/ci-test

Construction:
  stage: build
  variables:
    CI_COMMIT_BEFORE_SHA_PATH: /mnt/gv0/gitlab-runner-cache/$CI_PROJECT_PATH
    CI_COMMIT_BEFORE_SHA_FILE_NAME: $CI_BUILD_REF_NAME.sh
    CI_COMMIT_BEFORE_SHA_FILE: /mnt/gv0/gitlab-runner-cache/$CI_PROJECT_PATH/$CI_BUILD_REF_NAME.sh
  before_script:
    # save the files of this CI CI_COMMIT_SHA
    - /bin/bash ./ci/mkfile.sh $CI_COMMIT_BEFORE_SHA_PATH $CI_COMMIT_BEFORE_SHA_FILE_NAME
    Softlink node_modules to the host
    - ln -s $NODE_MODULES_PATH .
    - rm -rf php/share/*
    - cd webpack@next
  script:
    # cache the last CI
    - source $CI_COMMIT_BEFORE_SHA_FILE
    - echo "CI_COMMIT_BEFORE_SHA=$CI_COMMIT_SHA" > $CI_COMMIT_BEFORE_SHA_FILE - python3 .. /ci/build.py # build - /bin/bash.. /ci/commit  only:
    changes:
      - www_src/**/*
    refs:
      - test
      - test-99
      - test-128
      - master
      - ci


Test release:
  stage: deploy
  variables:
    PROCESS_PATH: /mnt/gv0/gitlab-runner-cache/deploy/process/$CI_JOB_ID  # do not change the directory, for the log server to get the log display
  script:
    - mkdir $PROCESS_PATH Create temporary path for release configuration file and result log
    - dplt $CI_PROJECT_DIR/.deploy_test.yml $CI_PROJECT_PATH $CI_PROJECT_DIR/php/ $PROCESS_PATH
    # DPLT publishes YML configuration
    - echo "Release is completed, the error log view" http://new.admin.wolffy.qihoo.net/log?path=$PROCESS_PATH
    - echo `ls $PROCESS_PATH/*.log`
  only:
    changes:
      - php/**/*
    refs:
      - test
Copy the code

2.2 Pre_Build Stage

When the package.json file changes, the pre_build stage is triggered and yarn install is executed. The downloaded node_modules is stored on the host machine, and dependencies are retrieved via the soft chain.

2.3 Build Stage

The construction stage is divided into three parts

  1. Diff file changes
  2. Front end of the build
  3. Result after commit build

2.3.1 Diff file changes

At each CI, the current CI commit SHA (CI_COMMIT_SHA variable) is stored in the file as CI_COMMIT_BEFORE_SHA variable. At diff, git diff changes the CURRENT CI from the last COMMIT SHA.

2.3.2 front-end build

According to the change of Git diff, determine the project to be packaged this time.

2.3.3 commit HTML file generated after packaging

When submitting code in GitLab CI/CD, commit packaged HTML files using Git credential stores. Git credential store details can be found in the credential store documentation

2.4 Deploy Stage

In the release phase, the packaged HTML files are deployed using the internal Rsync tool DPLT. DPLT allows you to configure the cluster and machine list.

V. Current problems and follow-up direction

  1. For a continuous integration, automated build and release is implemented, but critical testing is missing.
  2. With the help of GitLab CI/CD, we have achieved the consistency of the online environment, but the local development environment and the online environment are still inconsistent, and there may be no local problems, but online problems.
  3. After switching from SVN to Git development, branch management policy, branch naming specification, Commit Message specification, CodeReview, etc., need to be gradually standardized in the team.

Vi. Main reference materials

What is continuous integration?

What is CI/CD?

GitLab Docs

Introduction to CI/CD with GitLab

Continuous integration with GitLab CI

How to realize continuous integration and continuous deployment of front-end engineering?

Practice of FRONT-END engineering CI/CD based on GitLab CI