Introduction to the
Prow is the CI/CD system officially used by Kubernetes to manage the issue and PR of K8S. If you regularly check out PR on the Kubernetes community or submit some PR, you’ll often see a robot called K8S-CI-Bot respond to each PR and merge PR. What works behind the scenes in k8S-CI-bot is Prow. Prow was designed to fill in some of the functionality gaps on GitHub, and it’s also part of Jenkins-X, which has these features:
- Execute a variety of jobs, including testing, batching, and product publishing, and be able to configure when and what to execute based on Github Webhook.
- A pluggable robot function (Tide), can accept
/foo
This style of instruction. - Automatic merge Pr
- It comes with a web page, which can view the current task execution and Pr status, also includes some help information
- The OWNER of the module is configured in the same repo based on the OWNER file
- Capable of handling many PR for many Repos at once
- Prometheus indicators can be exported
Prow has its own CI/CD system, but it also works with our usual CI/CD, so if you’re used to Jenkins or Travis, you can use Prow.
The installation guide
The official Repo provides a quick installation guide based on GKE. This article will set up the Prow environment based on Qingyun Iaas. Don’t worry, most of these steps are platform independent and the entire installation process can be easily used on other platforms.
Prepare a Kubernetes cluster
There are several ways to prepare a Kubernetes cluster
- Self-built cluster using Kubeadm;
- Deploy KubeSphere cluster in QingCloud, or download and install KubeSphere directly;
- Copy the cluster’s Kubeconfig to the local, make sure it is running locally
kubectl cluster-info
Correct.
2. Prepare a GitHub bot account
If you don’t have a bot account, you can use your personal account. The robot account is easy to distinguish which Prow’s behavior, so the official use of the robot account should be used.
-
Set the bot account as administrator in the warehouse you want to manage with Prow.
-
Add a [Personal Access Token][1] to the account Settings. This token must have the following permissions:
- Must be:
public_repo
和repo:status
- optional:
repo
Suppose you need to use some private REPo - optional:
admin_org:hook
If you want to use it in an organization
- Must be:
-
Save this Token in a file, such as ${HOME}/secrets/oauth
-
Openssl rand -hex 20 generates a random string to verify webhook. Save this string locally, such as ${HOME}/secrets/h-mac
Note The tokens created in the last two steps must be saved. In addition to uploading the tokens to K8S, they are also used in subsequent configuration for bidirectional authentication
Configure the Kubernetes cluster
The default namespace used here is used to configure prow. If you need to configure prow in other namespaces, you need to configure the -n parameter in the related Kubectl command, and configure the namespace in the deployed YAML. It is recommended to clone the local repo, which comes with a number of gadgets to help you configure Prow.
- Save the token and HMAC created in the previous step in the K8S cluster
# openssl rand -hex 20 > ${HOME}/secrets/h-mac
kubectl create secret generic hmac-token --from-file=hmac=${HOME}/secrets/h-mac
kubectl create secret generic oauth-token --from-file=oauth=${HOME}/secrets/oauth
Copy the code
- Deploy the Prow. Since the GRC. IO image is used in the official YAML of Prow, which is not accessible in mainland China, we moved the corresponding repo to DockerHub and provided a yamL to replace the related image name. To deploy Prow (yamL modified with this repo), use the following command.
kubectl apply -f https://raw.githubusercontent.com/magicsong/prow-tutorial/master/prow.yaml
Copy the code
- use
kubectl get pod
If you see all pods running, the installation is complete. The diagram below:
- Configure Internet access
- If QKE is used, the cluster defaults to a LoadBalancer Controller. For a single cluster, just follow github.com/yunify/qing… Can be installed in, the installation is very convenient.
- Prow officially uses ingress to configure extranet access, so we need to configure
ingress-controller
. QKE has one by defaultingress-controller
In thekubesphere-control-system
In the. If the cluster does not already have oneingress-controller
One needs to be installed.The official documentationThere is no qingyun configuration guide yet, you need to install the following instructions to installingress-controller
: -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml kubectl apply -f https://raw.githubusercontent.com/magicsong/prow-tutorial/master/manifest/ingress-service.yaml # This command QKE needs to be executed Copy the code
After the preceding two commands are executed, press
kubectl get svc -n ingress-nginx
To obtain the public IP address, see the following figure. (If you need to specify the public IP address manually, see the LBThe configuration documentconfigurationingress-nginx
This service) :
Note that if you use QKE, you need to execute the second of the above two commands. The yamL corresponding to the second command is ingrsss, which needs to be changed to kubesphere-control-system before apply.
- The default Prow address is the public IP address + port 8080 of the ingress. You can view one of the addresses on the page
Echo Test
The task. The access effect is as follows:
Congratulations to you! Now that you have a Prow cluster and the cluster is ready to work, the next step is to do some configuration so that the ProW works as intended.
Configuration guide
The Prow configuration is quite complex, but here we will only show the minimum configuration to make our Pr robot work
Tools to prepare
- The installationbazel. Bazel is a tool that Google uses to build the K8S code, and prow is also built with Bazel. Subsequent configurations are configured using bazel’s dynamically generated tools (similar to
go run ./pkg/tool -a -b
). If you’re in a non-continental area, you can skip Bazel and just use itgo get
To get the static binay execution command. - If you want to use
bazel
After the installation is complete, the entire warehouse needs to be installedGithub.com/kubernetes/…Clone the entire warehouse down and use Bazel to run the command of the warehouse. Once clone is complete, the CD goes to the root directory of the repo
Configuration webhook
Prow works on a Webhook, where activity on Github is sent to processing
- Select a Github repository to configure Webhook. Execute the following command to add a repo, which needs to be replaced
hmac-path
andgithub-token-path
The address of the hook is the prow address above plus a “/hook” :
# Ideally use https://bazel.build, alternatively try:
# go get -u k8s.io/test-infra/experiment/add-hook && add-hook
bazel run //experiment/add-hook -- \
--hmac-path=/path/to/hook/secret \
--github-token-path=/path/to/oauth/secret \
--hook-url http://an.ip.addr.ess/hook \
--repo my-org/my-repo \
--repo my-whole-org \
--confirm=false # Remove =false to actually add hook
Copy the code
- If it works, you need to change the last line to
--confirm=true
. After running successfully, you should see a new Webhook in the Repo’s Webhooks configuration:
Configure the plug-in used by the cluster
Prow runs as a plug-in mechanism, similar to CoreDNS, which does nothing without plug-ins, but still works. We need to configure which plug-ins our Repo needs
- There are a number of plugins available, some of which can be seen in the Prow page above. Here’s how to use some of the built-in plug-ins. So let’s create a
plugins.yaml
The file is as follows:
plugins:
github.com/kubesphere-test/prow-tutorial:
- size
- cat
- dog
- pony
- yuks
- label
- trigger
- approve
- lgtm
Copy the code
- Create a blank
config.yaml
This file will be used in subsequent configuration tasks. Leave the plug-in configuration section blank. - If Bazel is installed, enter
test-infra
This directory, execute the following command (remember to replace the path of the relevant files in it), the command will checkconfig.yaml
andplugins.yaml
Is the configuration correct:
bazel run //prow/cmd/checkconfig -- --plugin-config=path/to/plugins.yaml --config-path=path/to/config.yaml
Copy the code
- Once you’ve checked, you can upload your plugins.yaml to the cluster (replace the path) :
kubectl create configmap plugins \
--from-file=plugins.yaml=${PWD}/samples/plugins.yaml --dry-run -o yaml \
| kubectl replace configmap plugins -f -
Copy the code
- such
size
The plug-in is done. A PR can be introduced, and the effect should look like this:
- There are a lot of fun plug-ins installed in the demo, so see the help page on the Prow page to learn how to use these commands.
Configure Tide robot
The main function of the Tide robot is to automatically Merge Pr. When a set goal is achieved, the Tide robot automatically merges the code into the main branch. The complete configuration of Tide is complex, but here we demonstrate a basic configuration that does not require much modification to run.
- Add the following fields to the above config.yaml (modify the corresponding repo and the corresponding tide page) :
tide:
merge_method:
kubesphere-test/prow-tutorial: squash
target_url: http://139.198.121.161:8080/tide
queries:
- repos:
- kubesphere-test/prow-tutorial
labels:
- lgtm
- approved
missingLabels:
- do-not-merge
- do-not-merge/hold
- do-not-merge/work-in-progress
- needs-ok-to-test
- needs-rebase
context_options:
# Use branch protection options to define required and optional contexts
from-branch-protection: true
# Treat unknown contexts as optional
skip-unknown-contexts: true
orgs:
org:
required-contexts:
- "check-required-for-all-repos"
repos:
repo:
required-contexts:
- "check-required-for-all-branches"
branches:
branch:
from-branch-protection: false
required-contexts:
- "required_test"
optional-contexts:
- "optional_test"
Copy the code
- Run the following command to push config.yaml to the K8S cluster (replace the corresponding config.yaml file location) :
kubectl create configmap config --from-file=config.yaml=${PWD}/samples/config.yaml --dry-run -o yaml | kubectl replace configmap config -f -
Copy the code
- Go to the PR just now, you should see the following check:
- Since this is the Pr I mentioned, it will be carried automatically
approved
Tag, now just add onelgtm
The label is ok. You need to get code Review people to look at the code and let them type it in/lgtm
“And Prow will type it automaticallylgtm
The label. Since no one else played/LGTM for this demo, and I can’t comment on it for myself/lgtm
, so this demo needs to manually select the LGTM label for this PR in Lables). The effect is as follows:
Advanced configuration
Prow is an efficient CI/CD system, but it is also a complex system. This article cannot explain all the advanced configurations. For further configuration, please refer to the official documentation. This Repo collates some commonly used scripts to facilitate subsequent configuration when using Prow. When using these scripts, take care to replace some data. For more information, see the OWNERS guide below.
OWNERS Guide
OWNERS is a configuration file that marks the owner of the folder to which the code belongs.
The working principle of
OWNERS
A file represents the owner of the directory where the file is located, including subdirectories. So in the root directoryOWNERS
The file has the highest permissions for the entire cluster, called “root owner”, and all Pr is automatically merged as long as root Owner agrees.- Each folder under repo can be set
OWNERS
, if a certain Pr changes the contents of the latter subfolders of this folder, the OWNER of this folder needs to approve. If multiple changes are made, more than one person needs to approve. You can also call the root owner - Can be found in
OWNERS
I’m going to set the Label in the file, so any Pr that changes to that folder will be labeled accordingly. Of course, this Label shouldn’t be in root OWNERS, so all Pr will be tagged.
The basic grammar
OWNERS is a YAML file whose basic form (and simplest configuration) is as follows:
approvers:
- alice
- bob # this is a comment
reviewers:
- alice
- carol # this is another comment
- sig-foo # this is an alias
lables:
- re/foo
- re/question
Copy the code
- approversCan be used
/approve
Command, which is the minimum condition that Pr can merge. You can notlgtm
But there has to beapproved
. soappovers
Has the Merge permission, similar to maintainer level. The other thing to note here is that the approvers proposed pr is satisfiedWorking principleThe second rule will be automatically typedapproved
(This can happen in two ways: 1) he is the root owner; 2) he is the root owner. He is the subdir owner, and his code is in this directory), so you need to add another label restriction (usually LGTM) to the PR merge to prevent the approvers code from being merged automatically. - reviewerTo be able to use
/lgtm
The command, “Looks good to me,” to review code. The regular REPo will make the LGTM command a necessary condition, and the code that anyone mentions will not be automatically typedlgtm
You must use the command manually. The main constraint is that the code must be reviewed. - If the approvers uses Github’s approve function, the approve label can also be labeled.
Advanced grammar
OWNERS also supports the Fliter parameter (which can’t be mixed with any of the above), which is primarily used to sort code files. See github.com/kubernetes/… Learn more about how to use it.