preface
In my last blog, I have encapsulated the Gitlab Api through Egg. This article will introduce the project design around the DevOps process (somewhat behind the scenes), requiring readers to have some back-end knowledge.
This series is both a continuous delivery project tutorial and can be used as a Node development tutorial for a complete DevOps project from develop-test-build-deploy
It consists of the following two series: front and rear modules
The backend module
- DevOps – Gitlab Api usage (completed, click to jump)
- DevOps – Building a DevOps base (30% complete)
- DevOps – Gitlab CI pipeline construction
- Devops-jenkins pipelined construction
- Devops-docker use
- DevOps – Release task flow design
- DevOps – Code review points
- Devops-node monitors service quality
The front-end module
- DevOps – H5 base scaffolding
- Devops-react project development
The above series may be adjusted later based on the actual development progress of the DevOps project
Enterprise design
Briefly analyze the structure of the r&d process of this project, and then do the following steps (the script has been written, it depends on how to play).
Project requirements analysis (purpose and results of system development)
- From project development – test – build – deploy a complete process, simplify delivery costs
- The concept of energy efficiency (R&D time – test time – total delivery time -bug rate and repair time) is added into the R&D process as a reference standard for project efficiency improvement (there are too many influencing factors, just for reference).
- Reasonable pick and test points, reduce invalid pick and test, reduce the test burden, improve the closed-loop quality of the process
- Provide online monitoring, analyze the usage rate and error rate of each version, improve the quality of project development
- Fast rollback of a specified version ensures fast service recovery in case of a new version crash
This project is developed from zero. Before formal development, it is necessary to sort out the requirements in order to avoid serious design defects, which may cause difficulties in later development or expansion (the road can be taken slowly, but not deviated).
Process design
As shown in the figure above, further refinement of the previous release process can be divided into the following four categories:
- Single project release process (a requirement requires only one project to complete)
- Fast rollback function when the production environment fails
- Integrated project release process (a requirement may have multiple projects involved in development and release)
- Bug fix release process (no requirement, release process for quick fixes of known but not urgent bugs online)
The design of task flow is actually very complex. In order to speed up the delivery of the first version, the task flow is fixed into the above four categories to reduce the amount of development, and a process will be added or modified later
Database design
The use of sequelize
Sequelize provides the sequelize-CLI tool to implement Migrations, and sequelize-CLI can also be introduced in egg projects (see sequelize operations for details).
If you have set up the environment as described in the previous blog, you can use NPM install –save-dev sequelize-cli to install sequelize-cli, and then use the following configuration to generate the required tables.
use strict';
const path = require('path');
module.exports = {
config: path.join(__dirname, 'database/config.json'),
'migrations-path': path.join(__dirname, 'database/migrations'),
'seeders-path': path.join(__dirname, 'database/seeders'),
'models-path': path.join(__dirname, 'app/model'),
};
Copy the codeThe above is the.Sequelizerc configuration, please put it in the project root directory
npx sequelize init:config
npx sequelize init:migrations
Copy the codeJson file and database/migrations directory. Modify the contents of database/config.json to the database configuration used in the project:
{
"development": { // Local database, other environment database, according to the example of their own change
"username": "root".
"password": "123456".
"database": "devops_dev".
"host": "127.0.0.1".
"dialect": "mysql"
},
}
Copy the codeCreate a database table by sequelize migration:generate –name=init-users
module.exports = { // In order to reduce the workload, we use gitLab's permissions directly, so we only need to drop the following fields in the library
up: async (queryInterface, Sequelize) => {
const { INTEGER, DATE, STRING } = Sequelize;
await queryInterface.createTable('users', {
id: { type: INTEGER, primaryKey: true,},
name: STRING(30),
username: STRING(30),
email: STRING(100),
avatar_url: STRING(200),
web_url: STRING(200),
created_at: DATE,
updated_at: DATE,
});
},
down: async queryInterface => {
await queryInterface.dropTable('users');
},
};
Copy the codeFinally, execute Migrate to change the database
Update database
npx sequelize db:migrate
If you have a problem and need to roll back, you can pass`db:migrate:undo`Roll back a change
# npx sequelize db:migrate:undo
# Pass`db:migrate:undo:all`Rollback to the initial state
# npx sequelize db:migrate:undo:all
Copy the codeBasic Design table
Data commonly used by Gitlab Project and Branch are stored locally, and then fields are added according to project requirements. The approximate table structure is shown in the figure above
Combined with the above project flow design, explain the table structure relationship
- Project can manage multiple branches branch, you can query the status of all branches under the current project (whether it is being tested, whether it exists in the process)
- Create a process (equivalent to a requirement) that associates multiple Branch developments
- After the process is created, all steps must be completed until completion (development – test – pre-release – production)
- When a branch is associated with a process, it is locked and cannot be added to other processes (requirements lock isolation to ensure no interference with the development process)
- During the process of test, multiple test can be performed for different branches (complex requirements can be tested in batches to achieve the desired goal)
- After all branch states in the process have been tested, the process state moves to the next phase, otherwise it stays in the test phase
The test record table is not put up, for the time being, the above functions are developed and then modified in combination with the branch management
Enterprise development
Add interface global return parameter
import { Controller } from "egg";
export default class BaseController extends Controller {
get user() {
return this.ctx.user;
}
success(data) {
this.ctx.body = {
code: 0.
data,
};
}
error({ code, data, message }) {
// Return different error codes based on services for the front-end service judgment processing
this.ctx.body = {
code,
data,
message,
};
}
}
Copy the codeDefine the global return parameter base class. The business Controller inherits the base class. The front end can make business judgment according to the returned code value
JWT permission validation
The last article introduced the method of obtaining access_token from Gitlab to operate the Open API, but we still need to store the user information locally for our later use
For project authorization verification, simple JWT is adopted, user data and access_token are saved, and improvement will be made after the goal of the first stage is completed
Specific egg-JWT usage can be referred to (egg-JWT usage), here directly attached to the business side of the code for reference:
const excludeUrl = ["/user/getUserToken"]; // Request whitelist to filter request paths that do not require verification
export default() = > {
const jwtAuth = async (ctx, next) => {
if (excludeUrl.includes(ctx.request.url)) {
return await next();
}
const token = ctx.request.header.authorization;
if (token) {
try {
/ / decoding token
const deCode = ctx.app.jwt.verify(
token.replace("Bearer ".""), // When JWT middleware validates, Bearer needs to be removed
ctx.app.config.jwt.secret
);
ctx.user = deCode;
await next();
} catch (error) {
ctx.status = 401;
ctx.body = {
code: 401.
message: error.message,
};
}
return;
}
ctx.status = 401;
ctx.body = {
code: 401.
message: "Verification failed".
};
return;
};
return jwtAuth;
};
Copy the codeThe above is the global interception of JWT permission middleware. After verifying the permission, user data is stored in CTX for subsequent business invocation. For specific use of middleware, see Egg middleware
// Controller
import { Post, Prefix } from "egg-shell-decorators";
import BaseController from "./base";
@Prefix("user")
export default class UserController extends BaseController {
@Post("/getUserToken")
public async getUserToken({
request: {
body: { params },
},
{})
const { ctx, app } = this;
const { username, password } = params;
// Gitlab retrieves access_token
const userToken = await ctx.service.user.getUserToken({
username,
password,
});
// Gitlab obtains user information
const userInfo = await ctx.service.user.getUserInfo({
accessToken: userToken.access_token,
});
// User data is stored locally
ctx.service.user.saveUser({
userInfo,
});
// Register user information and token with JWT
const token = app.jwt.sign(
{
userToken,
userInfo,
},
app.config.jwt.secret
);
ctx.set({ authorization: token }); / / set the headers
this.success(userInfo);
}
}
// Service
import { Service } from "egg";
export default class User extends Service {
// Get the access_token using the Gitlab API
public async getUserToken({ username, password }) {
const { data: token } = await this.ctx.helper.utils.http.post(
"/oauth/token".
{
grant_type: "password".
username,
password,
}
);
if (token && token.access_token) {
return token;
}
return false;
}
// Use the GitLab API to get gitLab user information
public async getUserInfo({ accessToken }) {
const userInfo = await this.ctx.helper.api.gitlab.user.getUserInfo({
accessToken,
});
return userInfo;
}
// The user information falls into the database
public async saveUser({ userInfo }) {
const { ctx } = this;
const {
id,
name,
username,
email,
avatar_url: avatarUrl,
web_url: webUrl,
} = userInfo;
// Check whether the user has logged out of the database
const exist = await ctx.model.User.findOne({
where: {
id,
},
raw: true.
});
if (exist) return;
// Create user information
ctx.model.User.create({
id,
name,
username,
email,
avatarUrl,
webUrl,
});
}
}
Copy the codeThe above is an example of JWT on the server side, which can parse out the desired information during global middleware interception for subsequent use. The example on the client side is described separately in the React project.
The above is the example and introduction of database table building and user and authority operation. The next chapter of this series will be launched after the development of basic task flow, which is expected to take about 2 weeks
The end of the
This project is developed from zero, the subsequent series of blogs will be launched according to the actual development progress, after the completion of the project, part of the source code will be open for your reference.
If you have any questions or opinions about the content of the article, please add the wechat Cookieboty communication.
Also follow the public account Cookieboty1024, welcome to join the front end soldier growth camp
Manual doghead town building
