GITLAB CI/CD front-end cache optimization

What is the CI/CD

CI stands for Continuous Integration and CD stands for Continuous Deployment. With so many flowers blooming at the front end, how could you not automate this process? With CI/CD, The front end just needs to focus on developing the code, leaving lint, deployment, push to machine automation.

At the beginning, I felt it was quite new to access Gitlab CI/CD, because I finally freed my hands and the access method was also very simple, adding gitlab-ci.yml file in the root directory of the project, there are many uses in it, there are many excellent articles on the Internet, here I will briefly introduce it

The name of the use
image Runner mirror
stages Define each job
cache The cache
artifacts Define artifacts that are passed between jobs
script Scripts executed

The project configuration is as follows:

As the construction became slower and slower, I began to think about THE CI optimization of GitLab. With the increase of construction times, gitLab changed from more than three minutes at the beginning to more than seven minutes now. It was so slow that I was worried, so I had the following exploration.

Cache + artifacts

After checking the CI document of Gitlab, I found that cache can be optimized. I tried to do this. Since the project is multi-branch, I used the variable CI_COMMIT_REF_SLUG of GITlab to cache the project code of different branches, and the project was switched from NPM installation dependency to YARN. Yarn requires less than a second to repeatedly install dependencies, while NPM requires about 8s

Here is the yarn repeat build time. There are too many NPM tests, so I lost it. The value is about 20 seconds.

After optimization, the CI time can be found from 8 minutes

Five minutes now, and I’ve also broken down the Job, which pulls caches and artifacts by default, which is time consuming

Because I split the Job, I did the packaging on the first job, but each stage is independent, and things from the first stage cannot be shared with things from the second stage, so I’m going to use artifacts.

Later, I tested it for several times and found that the time was about 5 minutes. In fact, I found some points worth optimizing, such as

  1. Can not install dependencies, until package.json changes to reinstall
  2. Each job performs restore cache, save cache, and Upload artifacts operations. These operations are IO operations
  3. Can you disable cache and artifacts automatically

Manual cache node_modules + job merge

According to the documents on the Internet, the cache of Gitlab itself seems to be time-consuming, so can we do the cache by ourselves? My thoughts are as follows

  1. To do this, use gitlab’s only changes and reinstall the code when the package changes, but this will split the job
  2. Monitor package.json changes by yourself and package node_module into a compressed package. Note that gitLab runner needs to be mounted to the host host
# unique hash of required dependencies
PACKAGE_HASH=($(md5sum package.json))
# path to cache file
DEPS_CACHE=/cache/dependencies_${PACKAGE_HASH}.tar.gz
# Check if cache file exists and if not, create it
if [ -f $DEPS_CACHE ];
then
  echo 'Use cache'
  tar zxf $DEPS_CACHE;
else
  echo 'No caching'
  #find /tmp/dependencies_* -mtime +1 -type f -delete 
  #npm install --quiet;
  yarn install --pure-lockfile
  tar zcf - ./node_modules > $DEPS_CACHE;
fi

Copy the code

Md5sum can detect file changes. It has been tested to add, delete and change the version

Combine jobs into one, install dependencies, build all at the same time, and finally ci time

In terms of build, vue-CLI has done a lot for us by default. All I can do is extract large files. Because I need source-map, I did not remove source-map in production environment, otherwise it would be faster.

conclusion

Exploring CI optimization looks like only the above point, but in fact, it took almost a day, I hope I never stop exploring spirit, and I hope there is a better solution to point out, I don’t know if one day the front-end can be like PHP, Node, directly push code can be accessed