How to build an automated deployment management platform from scratch with Node without Jenkins

The background,

H5 page is an important channel and way to attract new information and promote publicity due to its outstanding characteristics of flexible release, cross-platform and easy dissemination, which is favored by various companies.

The daily work of xiaobian is to develop various H5 promotional activities for users. In the whole development cycle, I summarize the following features of the H5 campaign page based on the situation of our company:

For users, large traffic;
Different display schemes of each end need to be compatible with each end (such as activity rules and display modules, ios and Android are different);
Demand changes frequently;
There are many partners (need to cooperate and coordinate with various business lines);
A tight schedule;

So deployment efficiency is especially important during development testing.

Due to the CDN publishing platform of our company, it is necessary to manually create templates and paste codes, which leads to low deployment efficiency. In addition, the code of the active page is scattered, so it cannot be managed and engineered uniformly. Therefore, we decided to implement an automatic deployment system, which has been put into use for half a year now, greatly improving our work efficiency. I call this automated deployment system [H5 Activity Management Platform].

Ii. Automatic deployment scheme of H5 activity management platform

Before introducing the platform implementation scheme, first put a sketch, good to have an intuitive understanding.

The realization of the platform mainly depends on the automatic deployment of the local development project and GITLab through communication and interaction.

The result is that when the local development branch is merged into the devTest or Master branch, the platform will automatically pull the latest code, build the target file, and then deploy the target file to the corresponding server directory. In addition, the platform provides common functions such as online and offline, version rollback, and scheduled online and offline.

Overall architecture flow chart:

Some key technical points are described in detail below

1. Local development projects

Our local development project is a multi-page development project using node + Webpack + Babel and other related technologies, with different activities in different directories. Because we need to do automatic construction and deployment processing and interact with [H5 activity management platform], we need to pay attention to the following points (you can adjust the scheme freely according to your own project situation).

As the source of automated build deployment, the local development project needs to provide build command lines to build test files and online files for subsequent shell commands. As in thepackage.jsonTo add the following command:

"scripts": {
    "local": "cross-env NODE_ENV=local node build.js"// Local development command"build": "cross-env NODE_ENV=product node build.js"// Build the live file"test": "cross-env NODE_ENV=test node build.js"// Build test file}Copy the code

Provide the build configuration filedev-config.jsFor filteringwebpackBuild time entry directory, build and compile only active pages currently under development, improve build speed.

//dev-config.js
module.exports = {
    devPages: ['test'] // Compile all active pages}Copy the code

Provides active page directory information configurationconfig.json, this configuration information is used for the display of [H5 Activity Management platform], which is also the information source in the effect drawing.

// config.json
{
  "pages": [{"folder": "lion"."desc": "Front Name Lion"."author": "Tactic."."user": "juejiu"
        },
        {
            "folder": "test"."desc": "Active Test Page"."author": "Tactic."."user": "juejiu"}}]Copy the code

construction-generatedJS 和 HTMLFile, depositdistIn the corresponding active directory under directory. The directory structure generated by the build is as follows:

|--dist
   |-- lion
       |-- lion_app.js
       |-- index.html
   |--test
       |-- test_app.js
       |-- index.html

Copy the code

The development branch is merged into the devTest branch during test and the development branch is merged into the Master branch when online.

2. Gitlab server

As an enterprise code version management tool, Gitlab provides the function configuration of Webhook. Webhook, as the name implies, is actually a hook. When we do something specific on Gitlab, we can trigger hooks to execute scripts that we have programmed to perform certain functions (for example, automatic front-end project publishing).

You can actually think of it as a callback, or a delegate, or event notification, but ultimately it’s a message notification mechanism. When GitLab fires an event, it sends a Post request to your configured HTTP service.

Note:

The URL is the IP address of the server deployed by [H5 Activity Management Platform].
IP comes after IPmergeIs an interface provided by the platform to which the GitLab server sends a Post request after the hook is triggered.
Secret TokenEnter a token, which is used to perform security verification for merge interface requests. You can set the token as you like.

The specific configuration is shown as follows:

Merge Request Events: Merge Request Events: Merge Request Events: Merge Request Events

Request header:

X-Gitlab-Event: Merge Request Hook
Copy the code

Request body:

{
  "object_kind": "merge_request"."user": {
    "name": "Administrator"."username": "root"."avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon"
  },
  "object_attributes": {
    "id": 99,
    "target_branch": "master"."source_branch": "ms-viewport"."source_project_id": 14."author_id": 51."assignee_id": 6,
    "title": "MS-Viewport"."created_at": "2013-12-03T17:23:34Z"."updated_at": "2013-12-03T17:23:34Z"."st_commits": null,
    "st_diffs": null,
    "milestone_id": null,
    "state": "opened"."merge_status": "unchecked"."target_project_id": 14."iid": 1,
    "description": ""."source": {"name":"Awesome Project"."description":"Aut reprehenderit ut est."."web_url":"http://example.com/awesome_space/awesome_project"."avatar_url":null,
      "git_ssh_url":"[email protected]:awesome_space/awesome_project.git"."git_http_url":"http://example.com/awesome_space/awesome_project.git"."namespace":"Awesome Space"."visibility_level": 20."path_with_namespace":"awesome_space/awesome_project"."default_branch":"master"."homepage":"http://example.com/awesome_space/awesome_project"."url":"http://example.com/awesome_space/awesome_project.git"."ssh_url":"[email protected]:awesome_space/awesome_project.git"."http_url":"http://example.com/awesome_space/awesome_project.git"
    },
    "target": {
      "name":"Awesome Project"."description":"Aut reprehenderit ut est."."web_url":"http://example.com/awesome_space/awesome_project"."avatar_url":null,
      "git_ssh_url":"[email protected]:awesome_space/awesome_project.git"."git_http_url":"http://example.com/awesome_space/awesome_project.git"."namespace":"Awesome Space"."visibility_level": 20."path_with_namespace":"awesome_space/awesome_project"."default_branch":"master"."homepage":"http://example.com/awesome_space/awesome_project"."url":"http://example.com/awesome_space/awesome_project.git"."ssh_url":"[email protected]:awesome_space/awesome_project.git"."http_url":"http://example.com/awesome_space/awesome_project.git"
    },
    "last_commit": {
      "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7"."message": "fixed readme"."timestamp": "2012-01-03T23:36:29+02:00"."url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7"."author": {
        "name": "GitLab dev user"."email": "gitlabdev@dv6700.(none)"}},"work_in_progress": false."url": "http://example.com/diaspora/merge_requests/1"."action": "open"."assignee": {
      "name": "User1"."username": "user1"."avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon"}}}Copy the code

3. H5 activity management platform

When developers merge code to the GitLab server, the Merge event is triggered, and GitLab sends a POST request with data (data format) to the URL specified by Webhooks. The platform receives the URL request with the following key technical points:

1. According to the POST request header and the body data, we can get the following information:

Target branch of merge: req.body.object_attributes. Target_branch;

Verify token: req. Headers [‘x-gitlab-token’];

Gitlab engineering warehouse address: req.body.project.git_ssh_URL

The action type of the triggered hook is req.body.object_attributes. Action

// Gitlab triggers the merge request router.post('/merge'.function (req, res, next) {
    let git_ssh_url = req.body.project.git_ssh_url;
    letname = req.body.project.name; Merge branch masterif (req.headers['x-gitlab-token'] = ='mergeRequest' && req.body.object_attributes.target_branch == 'master' && req.body.object_attributes.action == 'merge') {
        if (config[name] && config[name].git_ssh_url == git_ssh_url) {
            mergeTaskQueue.addTask(function () {
                getCode.init(git_ssh_url, name, 'master').then(function (data) {
                    console.log(data);
                    mergeTaskQueue.run();
                }).catch(function (error) {
                    console.log(error);
                    mergeTaskQueue.run();
                })
            }.bind(null, git_ssh_url, name));
        }
        res.end('receive request'); // Test merge branch dev}else if (req.headers['x-gitlab-token'] = ='mergeRequest' && req.body.object_attributes.target_branch == config[name].testEnv.targetBranch && req.body.object_attributes.action == 'merge') {
        if (config[name] && config[name].git_ssh_url == git_ssh_url) {
            mergeTaskQueue.addTask(function () {
                getCode.init(git_ssh_url, name, req.body.object_attributes.target_branch).then(function (data) {
                    console.log(data);
                    mergeTaskQueue.run();
                }).catch(function (error) {
                    console.log(error);
                    mergeTaskQueue.run();
                })
            }.bind(null, git_ssh_url, name));
        }
        res.end('receive request');
    } else {
        return res.end('receive request'); }})Copy the code

2. Execute the script

The script section does not use shell scripts, but uses the Node version of shell.js library, which allows us to control the execution logic, more friendly handling of error messages, and help the platform to have more friendly information display.

Pull the latest code to build the target file, and the general logic is shown as follows:

function init(git_ssh_url, projectName, targetBranch) {
    deferred = Q.defer();
    if(! git_ssh_url || ! projectName) {return deferred.reject('Project address or project name is empty');
    }
    repository = git_ssh_url;
    repositoryName = projectName;
    clonePath = path.join(__dirname, '.. /projects/' + projectName);

    shell.exec('exit 0');
    if (shell.test('-e'.clonePath)) {
        shell.cd(clonePath);
        let currentBranch = shell.exec('git symbolic-ref --short -q HEAD', {async: false, silent: true}).stdout;
        if(currentBranch ! = targetBranch) {let outInfo = shell.exec('git branch', {async: false, silent: true}).stdout;
            let gitcmd = outInfo.indexOf(targetBranch) >= 0 ? ('git checkout ' + targetBranch) : ('git checkout -b ' + targetBranch + ' origin/' + targetBranch);
            shell.exec('git pull && ' + gitcmd, {async: false, silent: true});
        }
        shell.exec('git pull', {async: false, silent: true}, function (code, stdout, stderr) {
            if(code ! = 0) { console.log(stderr);return deferred.reject('git pull error');
            }
            console.log(stdout);
            console.log('git pull run success');
            returnbuildTest(projectName, targetBranch); })}else {
        if(! fs.existsSync(projects_path)) { fs.mkdirSync(projects_path); } shell.cd(projects_path); shell.exec('git clone ' + repository, function (code, stdout, stderr) {
            if(code ! = 0) { console.log(stderr);return deferred.reject('git clone error');
            }
            console.log('git clone success');
            shell.cd(clonePath);
            let outInfo = shell.exec('git branch', {async: false, silent: true}).stdout;
            let gitcmd = outInfo.indexOf(targetBranch) >= 0 ? ('git checkout ' + targetBranch) : ('git checkout -b ' + targetBranch + ' origin/' + targetBranch);
            shell.exec(gitcmd, {async: false, silent: true});
            returnbuildTest(projectName, targetBranch); })}returndeferred.promise; } // Build the projectfunction buildTest(projectName, targetBranch) {
    shell.cd(clonePath);
    shell.exec('npm config set registry https://registry.npm.taobao.org && npm install', {async: true, silent: true}, function (code, stdout, stderr) {
        if(code ! = 0) { console.log(stderr);return deferred.reject('npm install error');
        }
        console.log('npm install success');
        shell.rm('-rf', path.join(clonePath, 'dist'));
        let testCommand = config[repositoryName].commands.test || 'npm run test'; // Build test file command line shell.exec(testCommand, {async: true, silent: true}, function (code, stdout, stderr) {
            if(code ! = 0) { console.log(stderr);return deferred.reject('npm run test fail');
            }
            console.log('npm run test success');
            copyPage(repositoryName, 'test'); // Copy to the test directoryif(targetBranch ! ='master') {
                shell.exec('exit 0');
                deferred.resolve('build success and finish');
                return; // Build the final live file shell.rm()'-rf', path.join(clonePath, 'dist'));
            let buildCommand = config[repositoryName].commands.build || 'npm run build'; Shell. Exec (buildCommand, {async:true, silent: true}, function (code, stdout, stderr) {
                if(code ! = 0) { console.log(stderr);return deferred.reject('npm run build fail');
                }
                console.log('npm run build success');
                copyPage(repositoryName, 'online'); // After each master build, switch to the test branch, so that the platform can read the config.json information (the test branch is the latest) shell.exec('git checkout ' + config[projectName].testEnv.targetBranch, {async: false, silent: false}); 
                shell.exec('exit 0');
                deferred.resolve('build success and finish'); })})})}Copy the code

3. Dynamically expand projects

By modifying the project configuration file to access different projects, the configuration information includes the CDN path to be uploaded for each project, the construction command, and the project directory display information file path (config.json), as shown below:

Module.exports = {'h5-activity-cms': {
        git_ssh_url: '[email protected]:awesome_space/awesome_project.git',
        desc: 'Front End Famous Lion Project',
        tabContent: 'Front Name Lion'// Upload CDN parameters according to your own project set HTML: {domain:' ',
                path: ' '
            },
            js: {
                domain: ' ',
                path: ' '// Build the script command linetest: 'npm run test',
            build: 'npm run build'
        },
       
        configFile: 'config.json', // Active page list information}}Copy the code

4. Queue processing

In the process of building target files, many asynchronous operations such as file generation, compression, and copy are performed. Different merge requests may operate on the same file. Therefore, merge requests need to be queued.

class TaskQueue {
    constructor() {
        this.list = [];
        this.isRunning = false;
    }
    addTask(task) {
        this.list.push(task);
        if(this.isRunning) {
            return;
        }
        this.start();
    }
    shift() {
        return this.list.length > 0 ? this.list.shift() : null;
    }
    run() {
        let task = this.shift();
        if(! task) { this.isRunning =false;
            return;
        }
        task();
    }
    start() {
        this.isRunning = true;
        this.run();
    }
}
module.exports = TaskQueue;
Copy the code

5. Release the CDN

This requires the backend students to provide a service interface for pushing files to the CDN or server. We use a server interface, we upload to their server through Node, the interface side will periodically push files to the CDN, the specific situation of each person to deal with it.

Third, summary

The platform uses Node to realize a miniature deployment management platform similar to Jenkins, with the following outstanding advantages:

The platform can get through the deployment of local development environment and test environment, realize the automation of test deployment, save the manual uploading and pasting code time, greatly improve the work efficiency;
Category based on project engineering, easy for developers to find pages efficiently;
Support dynamic extension, you can access other GitLab projects by adding configuration files;
The platform operation page can be customized as required, which is more flexible and lighter than using Jenkins.

Scan my public number [front-end name lion], more exciting content to accompany you!

How to build an automated deployment management platform from scratch with Node without Jenkins

The background,

Ii. Automatic deployment scheme of H5 activity management platform

1. Local development projects

2. Gitlab server

3. H5 activity management platform

Third, summary

Related Posts

Conversion between various time formats

Passing by value VS passing by reference

Functional programming and its properties