Original is not easy, I hope you can pay attention to us, and then easily point a praise ~~

GitLab Open API code volume statistics, let your efforts be seen by the boss

preface

Dunhuang system is the whole process management system of project development developed by the front end team of zhengcai Cloud. The goal is to manage all the processes of project development. From project creation, code initialization, to local development of code, test delivery, post-test release, version rollback, data statistics, etc. This paper is the realization principle of remote project creation and data statistics in this system. There will be subsequent technical articles on the rest of dunhuang system. You are welcome to pay attention to the wechat public number “Zhengcai Cloud front End Team”, or pay attention to the “Zhengcai Cloud front end Team” on digging Gold, so as to get the latest information in the first time.

Introduction to the

This article focuses on how to create a project, initialize code, and count team code through the GitLab Open API. In the process of front-end engineering construction, Git warehouse creation, project initialization and code volume statistics need to be carried out through the Node server. After a period of development and exploration, the initial realization of the demand. Here’s a summary of the record.

A, requirements,

  • Create the repository and initialize the code
    • Objective: To unify project new entrance, project development template and project development process. Save new member start cost.
    • Specific functions: Team members can directly create the GitLab warehouse by inputing the project name, GitLab group, project template and other fields, and initialize the project in the created GitLab warehouse according to the selected template and name and other information.
  • Code volume statistics
    • purpose
      • Quantify team code quality, count team workload, monitor business throughput changes, etc.
      • Implement the Commit specification across the team.
    • Specific function
      • Obtain Git Commit information of team members, store it in the database, and make statistical analysis based on Commit information data.
      • If the Commit information does not start with Fix, feat, Style, or refactor, a pie chart can be drawn for statistical comparison. This drives the implementation of the Commit specification.
      • If the number of single Commit rows exceeds a certain number, it will be prompted in the graph.

2. Create a project

Those who read the GitLab Open API documentation can easily find the create interface, but in addition to create we need to import the project template, modify the corresponding project name, description, author, etc. This involves combined calls to multiple interfaces.

1, API prefix https://GitLabHost/api/v4, all GitLab Open API as a prefix, for an example of creating the project interface: https://GitLabHost/api/v4/projects.

2. Each request requires the creator’s Private Token as a parameter. And the creator is required to have the corresponding permissions. I use the unified user Front as the creator. This eliminates the need to obtain each user’s Private Token to create a project. This has another advantage: when counting the code volume, the code volume of this part of the copied code will be counted on the virtual user Front, reducing the statistical error.

  • Create a project

    Name: project name (required if path is not passed) path: project path (required if name is not passed) namespace_id: Id of the Git group where the project is created Description: Project description import_url: code path for initializing the project, in the format of https://user:password@host/path.gitCopy the code

    You can create a project with an initialization template in the target Git group by calling the above interface.

    When creating a project, note the following:

    • Create Front only if you have Master permission on the target Git group. Add Front to the target group ahead of time and grant Master privileges. Take the front-test group as an example:

    • 2, the current user also needs to make the permission judgment, which requires developers to call the GET /groups/:id/members interface before creating groups and compare whether the current user has the permission to create. Such as:

      Access_level is the permission value, which corresponds to

      10 => Guest 20 => Reporter 30 => Developer 40 => Master 50 => OwnerCopy the code

  • Initialize the project information

    The name, desc, in the readme. md file has not been replaced. Now we need to replace some key information in the readme. md and package. json files.

    • Read the corresponding file, in this case, directly access the corresponding file in the browser and change the bloB in the path to RAW to directly read the corresponding file information.

      Here’s an example: https://GitLabHost/f2e-cube/template/leo-middle-react-pc-project/blob/master/README.md this link can access directly to the project file page, but it is not a simple file for information.

      Change the blob to raw or direct access to the https://GitLabHost/f2e-cube/template/leo-front-vue-pc-component/raw/master/README.md can direct access to the file content.

    • After reading the file information, use the Node template engine to inject the corresponding data into the obtained file information. The EggJs framework is used on the server. Ejs is used for the template engine.

      Key code:

      const result = await this.ctx.renderString(readMeStr, { author: userInfo.userName, name: gitProjectInfo.name, description: project.desc, ... others, });Copy the code

    • Submit the rendered string to the created project. Use the create Commit interface.

      POST/projects / : id/repository/commits (only list of key parameters in here, please see more parameters GitLab document) parameters: id: project id (just created good projects often return project information, contains the project id) branch: Commit_message: commit message Actions: [] Action: Change type Create, DELETE, Move, update (update in this case) File_path: indicates the path of the changed file. Content: Indicates the changed content, which is the result in the preceding codeCopy the code

      At this point, the project is created and initialized!

      You can see that name and desc have been replaced with the corresponding project name and Chinese description.

Three, code volume statistics

According to the code volume statistics, baidu and Google search can find a large number of codes, but basically all codes are pulled to the local, execute commands to obtain the code volume of the project or the code volume of contributors to the project code. A common solution is to add a Git Hook to the project. Call the request before the project is committed to pass the currently committed code volume to the backend for storage statistics. The disadvantages of this are

1. All projects need to add Git Hook. Not a small problem for teams with dozens or hundreds of historical projects.

2. Historical data statistics are not available.

There is an entity called Event in GitLab API, and each user operation will have corresponding Event generated and stored. Here we use events to count the code volume.

  • The basic information

    • 1. API prefixhttps://GitLabHost/api/v4
    • 2. Each request requires the query user’s Private Token as a parameter. The query user must have the corresponding permission. I used a unified user Front as the query user. All counted projects need to add Front, and grant Developer or above rights. (You can assign weights directly to groups)

  • Gets the user names of all users whose code counts are required

    First get the user names of all the team users through the pin interface (the team pin user name is the same as the Git user name). This step can be obtained manually for teams that are not too large.

  • Get the Git Id of the user (manually if the team is small)

    Through the GET/users? Username =: The username interface obtains the Git User Id corresponding to the User.

  • Example Query user events

    After obtaining all user ids, you can call the GET/Users /: Id /events interface to query all events of the current user. There will be Push events. The Event contains the corresponding Commit information and project information. However, the Commit information is incomplete and does not contain information about how many lines of code were added, how many lines were removed, and how many lines were total.

  • Get Commit details

    Commit information captured by the step on the Id and the project Id query again Commit details: GET/projects / : Id/repository/commits: sha. Here and here you can get the Stats information for the Commit. So you have additions, deletions, total.

  • Save database and set scheduled tasks

    The obtained data into the database can do the code volume statistics. Then set up a scheduled task, run once or twice a week for code volume statistics.

  • There are some things to watch out for

    • Some operations may cause the Commit to duplicate, so you need to redo the Commit for the same person.
    • GitLab Events only stores data for nearly a year. So only one year’s worth of code can be counted on the day it runs.
    • Make sure you split the Push time by person so that after the first run, you can then only get Commit after the last Push. The number of requests can be greatly reduced.
    • After executing the code, I found some problems, such as team members uploading the node_modules folder by mistake. This results in too many statistical lines of code, and the solution is to filter out commits greater than 10,000 lines (which you can optionally specify).

  • The problem

    • Due to the limitation of the interface, there are too many requests, so it will be a little slow to run the task for the first time. It will take about 20 or 30 minutes for more than 40 people in our team to run.

  • subsequent

    • The latest version of the GitLab Open API uses GraphQL technology. Can solve the above problems.

Write in the last

In fact, in the first article, we did not make it clear, because we did not notice that “code volume review” would bring some troubles to some readers. We may be in different companies, and some companies may calculate employees’ KPI simply and roughly based on the amount of code, which we also think is inappropriate. The purpose of the code and code quantity mentioned in this paper is to facilitate access to the code compliance testing (such as compatibility API testing and Lint testing) services of the team, and to facilitate quantification from the data dimension to verify the upgrade of architecture and the improvement of infrastructure. Data indicators can be used to explain the reduction of students’ workload. We hope that the upgrading of technology and architecture, as well as the improvement of infrastructure and tool link can help us achieve the goal of less Code, help our students become more “lazy”, and spare more time to focus on and study things that are more valuable to the future of business and individual. We also hope that you can join us and help us get closer and achieve this goal faster.

, recruiting

Recruitment, front-end, belonging to the ZooTeam, more than 50 partners are waiting for you to join the wave. If you want to change what’s been bothering you, you want to start bothering you. If you want to change, you’ve been told you need more ideas, but you don’t have a solution. If you want change, you have the power to make it happen, but you don’t need it. If you want to change what you want to accomplish, you need a team to support you, but you don’t have the position to lead people. If you want to change the pace, it will be “5 years and 3 years of experience”; If you want to change the original savvy is good, but there is always a layer of fuzzy window… If you believe in the power of believing, believing that ordinary people can achieve extraordinary things, believing that you can meet a better version of yourself. If you want to be a part of the growth of a front end team that has deep business understanding, technology systems, value creation, and impact spillover as the business takes off, I think we should talk. Any time, waiting for you to write something and send it to [email protected]

Recommended reading

After reading this, you can play React Hooks, too

Vue component data communication scheme summary

Automated Web performance optimization analysis solution