The background,

Some time ago, I was doing code audit and found that many projects have security risks, most of which are caused by unfiltered parameters. In order to solve this problem, I trained Web security development specification manual V1.0, but the effect was not very satisfactory. The reason was that after the training, developers mainly focused on the degree of function completion, and secure coding was not the core index for them.

In order to let developers always pay attention to safety problems, I put a hook on the gitlab server, this hook is to submit the testing code files, there may be a security risk problems to its output, so that developers can have deeper feelings on the content of the training, pay more attention to the safety of a coding time.

Two, operation steps

  1. Set up the environment
  2. Create a project
  3. Create a hook
  4. Hook test

Third, build the environment

3.1 installation gitlab

Before the formal deployment to the server, I need to build a GitLab service locally for the development and testing of hooks. Here, I use Docker to build faster, and the commands are as follows

docker run --detach  --publish 443:443 --publish 80:80  --name gitlab --restart always  gitlab/gitlab-ce

After the command is executed, the information returned is as follows

In the figure above, you can see that the container is running successfully, using the browser to access the GitLab address

http://127.0.0.1

After access, you need to set an administrator password, as shown in the figure below

After you fill in the password and confirm to change the password, you will be redirected to the main page of GitLab, as shown in the figure below

Create a project in this GitLab for hook testing, as shown in the figure below

After successfully creating the Project, pay attention to the Project ID:2 in the page, and record this 2, which will be used later. Next, I need to start the development and deployment of hooks, which can be developed in various languages. I am familiar with PHP here, so I adopt PHP development.

3.2 Installation Dependencies

The GitLab container does not support PHP by default, so you need to install PHP first. The installation commands are shown below

apt update -y && apt install php -y

After the command is executed, the information returned is as follows

In the image above, you can see that PHP has been installed successfully. To verify that the PHP command runs, I use the following command here

php -v

After the command is executed, the information returned is as follows

In the figure above, you can see that the PHP version is 7.4.3, indicating that PHP has been installed successfully.

3.3 installation semgrep

The hook program needs to call semgrep. This program is not installed in GitLab, so you need to install it. Here, you use PIP to install it, but you need to upgrade the version of PIP first

pip3 install --upgrade pip

After the command is executed, the information returned is as follows



In the figure above, you can see that the PIP version has been upgraded to 21.1.2, indicating that the upgrade was successful

Semgrep also relies on the setuptools module, which needs to be upgraded with PIP. The command to upgrade is shown below

pip3 install --upgrade setuptools

After the command is executed, the information returned is as follows



In the figure above, you can see that the setuptools module has been successfully upgraded

You are now ready to install semgrep, with the command shown below

cd /usr/local/bin/ && python3 -m pip install semgrep

After the command is executed, the information returned is as follows

In the image above, you can see that semgrep has been installed, so I need to verify this by using semgrep again, as shown below

semgrep --version

After the command is executed, the information returned is as follows



In the figure above, you can see that the version information of semgrep is 0.52.0, confirming that the installation was successful.

3.4 check the hash

Now we need to add the hook to the project we just created, and here we need to find the path where the project resides, in the project page

echo -n 2 | sha256sum

After the command is executed, the information returned is as follows

find / -iname d4

After the command is executed, the information returned is as follows



In the figure above, you can see where the project is stored. Two paths are returned. One of the paths is a soft connectioncdCommand Enter to enter the location of the project

cd /var/opt/gitlab/git-data/repositories/@hashed/d4

After the command is executed, execute it againlsCommand, the resulting information is shown below

In the figure above, you can see that there is a folder of 73, which is the naming convention of GitLab. Enter this folder and the command is as follows

cd 73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git/

After the command is executed, the information returned is as follows

You can see all the files for this project in the image above, where I need to develop the hook file

Create a hook

Custom hooks need to be stored in the custom_hooks directory. This folder is not available by default, so you need to create this folder, executing the command shown below

mkdir custom_hooks  && cd custom_hooks

5.1 New Hook

After creating the custom_hooks folder and entering it, create a hook file using Vim with the commands shown below

vim  pre-receive

After entering the Vim editor interface, add the following hook code, as shown below

#! /usr/bin/php <? php fwrite(STDOUT, 'please input:'); list($oldVer, $newVer, $ref_name) = explode(" ", fgets(STDIN)); //ob_start(); $cmd = "git diff --name-only {$oldVer}.. {$newVer}"; echo $cmd . PHP_EOL; exec($cmd, $result); $rand = date("Y-m-d-H-i-s"); $baseDir = "/tmp/11/$rand/"; $ruleFile = "/semgrep-rule.yaml"; foreach ($result as $value) { if (strstr($value, ".php") ! == false) { $randName = $baseDir . $value; if (! is_dir(dirname($randName))) { # if (file_exists($randName) == false) { mkdir(dirname($randName), 0777, true); } $cmd = "git show {$newVer}:$value > $randName"; # echo $cmd . PHP_EOL; exec($cmd, $result); } } $cmd = "/opt/gitlab/embedded/bin/semgrep -f '$ruleFile' $baseDir -o /tmp/11.txt"; exec($cmd, $result); //ob_clean(); $notice = file_get_contents("/tmp/11.txt"); echo $notice . PHP_EOL; file_put_contents("/tmp/11.txt", ""); exec("rm -rf $baseDir"); echo 0;

Save and launch the hook file, and then you need to set the permissions for the custom hook directory. Here I simply set the permissions to 777, and the command is as follows

chmod -R 777 .. /

Once the permissions are set, I also need to create a semgrep scan rule file to determine if the code is correct.

The command to execute is shown below

vim /semgrep-rule.yaml

Once in the Vim editor, you need to copy the following rule contents into it

rules: - id: assert-use patterns: - pattern: assert($ASSERT, ...) ; # - pattern-not: assert(<... $ASSERT ... >,...). ; - https://github.com/returntocorp/semgrep/issues/2035 - pattern-not: assert("..." ,...). ; Message: | using input from the user calls assert is equivalent to the eval '. metadata: references: - https://www.php.net/manual/en/function.assert - https://github.com/FloeDesignTechnologies/phpcs-security-audit/blob/master/Security/Sniffs/BadFunctions/AssertsSniff.php  languages: [ php ] severity: ERROR - id: backticks-use pattern: '`... `; 'the message: | use back ticks may lead to command injection vulnerabilities. metadata: references: - https://www.php.net/manual/en/language.operators.execution.php - https://github.com/FloeDesignTechnologies/phpcs-security-audit/blob/master/Security/Sniffs/BadFunctions/BackticksSniff.p hp languages: [ php ] severity: ERROR

After saving and pushing out the rule file, it is necessary to modify the permission of this rule file. Here I use 777 permission distance. The command is as follows

chmod 777 /semgrep-rule.yaml

After setting the rule file permissions, there are two more cache places to set the permissions, otherwise an error will be reported during the operation. The first is the cache file of semgrep. Set the permissions command as follows

mkdir -p /var/opt/gitlab/.cache  && chmod -R 777 /var/opt/gitlab/.cache

The other is the cache file of the hook itself, which also needs to set permissions. The command is shown below

echo '' > /tmp/11.txt  && chmod 777 /tmp/11.txt

5.2 Test Hook

Now you can formally test the availability of the hook by first pulling the project code you just created, as shown below

Git clone http://127.0.0.1/root/test.git

After executing the command, the information returned is as follows

In the image above you can see that the project has been pulled down. Next I need to edit a PHP file. The commands are as follows

vim index.php

After the command is executed, the test code is stored in it

<? php phpinfo(); $cmd = "ls {$_GET['x']}"; exec($cmd);

After saving and exiting, submit the code to GitLab with the command shown below

echo ' ' >> index.php && git add . && git commit . -m 'init' && git push

However, after Git is pushed to the GitLab server, GitLab calls the hook and outputs the information returned by the hook, as shown in the figure below

In the figure above, you can see that the hook indicates that line 8 of the index. PHP file is not safe, so the entire deployment is complete.

Author: Tang Qingsong

Date: 2021-06-03

WeChat: songboy8888