Introduction to the

Prow is the CI/CD system officially used by Kubernetes to manage the issue and PR of K8S. If you regularly check out PR on the Kubernetes community or submit some PR, you’ll often see a robot called K8S-CI-Bot respond to each PR and merge PR. What works behind the scenes in k8S-CI-bot is Prow. Prow was designed to fill in some of the functionality gaps on GitHub, and it’s also part of Jenkins-X, which has these features:

  1. Execute a variety of jobs, including testing, batching, and product publishing, and be able to configure when and what to execute based on Github Webhook.
  2. A pluggable robot function (Tide), can accept/fooThis style of instruction.
  3. Automatic merge Pr
  4. It comes with a web page, which can view the current task execution and Pr status, also includes some help information
  5. The OWNER of the module is configured in the same repo based on the OWNER file
  6. Capable of handling many PR for many Repos at once
  7. Prometheus indicators can be exported

Prow has its own CI/CD system, but it also works with our usual CI/CD, so if you’re used to Jenkins or Travis, you can use Prow.

The installation guide

The official Repo provides a quick installation guide based on GKE. This article will set up the Prow environment based on Qingyun Iaas. Don’t worry, most of these steps are platform independent and the entire installation process can be easily used on other platforms.

Prepare a Kubernetes cluster

There are several ways to prepare a Kubernetes cluster

  1. Self-built cluster using Kubeadm;
  2. Deploy KubeSphere cluster in QingCloud, or download and install KubeSphere directly;
  3. Copy the cluster’s Kubeconfig to the local, make sure it is running locallykubectl cluster-infoCorrect.

2. Prepare a GitHub bot account

If you don’t have a bot account, you can use your personal account. The robot account is easy to distinguish which Prow’s behavior, so the official use of the robot account should be used.

  1. Set the bot account as administrator in the warehouse you want to manage with Prow.

  2. Add a [Personal Access Token][1] to the account Settings. This token must have the following permissions:

    • Must be:public_repo 和 repo:status
    • optional:repoSuppose you need to use some private REPo
    • optional:admin_org:hookIf you want to use it in an organization

  3. Save this Token in a file, such as ${HOME}/secrets/oauth

  4. Openssl rand -hex 20 generates a random string to verify webhook. Save this string locally, such as ${HOME}/secrets/h-mac

Note The tokens created in the last two steps must be saved. In addition to uploading the tokens to K8S, they are also used in subsequent configuration for bidirectional authentication

Configure the Kubernetes cluster

The default namespace used here is used to configure prow. If you need to configure prow in other namespaces, you need to configure the -n parameter in the related Kubectl command, and configure the namespace in the deployed YAML. It is recommended to clone the local repo, which comes with a number of gadgets to help you configure Prow.

  1. Save the token and HMAC created in the previous step in the K8S cluster
# openssl rand -hex 20 > ${HOME}/secrets/h-mac
kubectl create secret generic hmac-token --from-file=hmac=${HOME}/secrets/h-mac
kubectl create secret generic oauth-token --from-file=oauth=${HOME}/secrets/oauth
Copy the code
  1. Deploy the Prow. Since the GRC. IO image is used in the official YAML of Prow, which is not accessible in mainland China, we moved the corresponding repo to DockerHub and provided a yamL to replace the related image name. To deploy Prow (yamL modified with this repo), use the following command.
kubectl apply -f https://raw.githubusercontent.com/magicsong/prow-tutorial/master/prow.yaml
Copy the code
  1. usekubectl get podIf you see all pods running, the installation is complete. The diagram below:

  1. Configure Internet access
  • If QKE is used, the cluster defaults to a LoadBalancer Controller. For a single cluster, just follow github.com/yunify/qing… Can be installed in, the installation is very convenient.
  • Prow officially uses ingress to configure extranet access, so we need to configureingress-controller. QKE has one by defaultingress-controllerIn thekubesphere-control-systemIn the. If the cluster does not already have oneingress-controllerOne needs to be installed.The official documentationThere is no qingyun configuration guide yet, you need to install the following instructions to installingress-controller:
  • kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/magicsong/prow-tutorial/master/manifest/ingress-service.yaml # This command QKE needs to be executed
Copy the code

    After the preceding two commands are executed, presskubectl get svc -n ingress-nginxTo obtain the public IP address, see the following figure. (If you need to specify the public IP address manually, see the LBThe configuration documentconfigurationingress-nginxThis service) :

Note that if you use QKE, you need to execute the second of the above two commands. The yamL corresponding to the second command is ingrsss, which needs to be changed to kubesphere-control-system before apply.

  1. The default Prow address is the public IP address + port 8080 of the ingress. You can view one of the addresses on the pageEcho TestThe task. The access effect is as follows:

Congratulations to you! Now that you have a Prow cluster and the cluster is ready to work, the next step is to do some configuration so that the ProW works as intended.

Configuration guide

The Prow configuration is quite complex, but here we will only show the minimum configuration to make our Pr robot work

Tools to prepare

  1. The installationbazel. Bazel is a tool that Google uses to build the K8S code, and prow is also built with Bazel. Subsequent configurations are configured using bazel’s dynamically generated tools (similar togo run ./pkg/tool -a -b). If you’re in a non-continental area, you can skip Bazel and just use itgo getTo get the static binay execution command.
  2. If you want to usebazelAfter the installation is complete, the entire warehouse needs to be installedGithub.com/kubernetes/…Clone the entire warehouse down and use Bazel to run the command of the warehouse. Once clone is complete, the CD goes to the root directory of the repo

Configuration webhook

Prow works on a Webhook, where activity on Github is sent to processing

  1. Select a Github repository to configure Webhook. Execute the following command to add a repo, which needs to be replacedhmac-pathandgithub-token-pathThe address of the hook is the prow address above plus a “/hook” :
# Ideally use https://bazel.build, alternatively try:
# go get -u k8s.io/test-infra/experiment/add-hook && add-hook
bazel run //experiment/add-hook -- \
  --hmac-path=/path/to/hook/secret \
  --github-token-path=/path/to/oauth/secret \
  --hook-url http://an.ip.addr.ess/hook \
  --repo my-org/my-repo \
  --repo my-whole-org \
  --confirm=false  # Remove =false to actually add hook
Copy the code
  1. If it works, you need to change the last line to--confirm=true. After running successfully, you should see a new Webhook in the Repo’s Webhooks configuration:

Configure the plug-in used by the cluster

Prow runs as a plug-in mechanism, similar to CoreDNS, which does nothing without plug-ins, but still works. We need to configure which plug-ins our Repo needs

  1. There are a number of plugins available, some of which can be seen in the Prow page above. Here’s how to use some of the built-in plug-ins. So let’s create aplugins.yamlThe file is as follows:
plugins:
  github.com/kubesphere-test/prow-tutorial:
  - size
  - cat 
  - dog
  - pony 
  - yuks 
  - label
  - trigger
  - approve
  - lgtm 
Copy the code
  1. Create a blankconfig.yamlThis file will be used in subsequent configuration tasks. Leave the plug-in configuration section blank.
  2. If Bazel is installed, entertest-infraThis directory, execute the following command (remember to replace the path of the relevant files in it), the command will checkconfig.yamlandplugins.yamlIs the configuration correct:
bazel run //prow/cmd/checkconfig -- --plugin-config=path/to/plugins.yaml --config-path=path/to/config.yaml
Copy the code
  1. Once you’ve checked, you can upload your plugins.yaml to the cluster (replace the path) :
kubectl create configmap plugins \
  --from-file=plugins.yaml=${PWD}/samples/plugins.yaml --dry-run -o yaml \
  | kubectl replace configmap plugins -f -
Copy the code
  1. suchsizeThe plug-in is done. A PR can be introduced, and the effect should look like this:

  1. There are a lot of fun plug-ins installed in the demo, so see the help page on the Prow page to learn how to use these commands.

Configure Tide robot

The main function of the Tide robot is to automatically Merge Pr. When a set goal is achieved, the Tide robot automatically merges the code into the main branch. The complete configuration of Tide is complex, but here we demonstrate a basic configuration that does not require much modification to run.

  1. Add the following fields to the above config.yaml (modify the corresponding repo and the corresponding tide page) :
tide:
  merge_method:
    kubesphere-test/prow-tutorial: squash

  target_url: http://139.198.121.161:8080/tide
  queries:
  - repos:
    - kubesphere-test/prow-tutorial
    labels:
    - lgtm
    - approved
    missingLabels:
    - do-not-merge
    - do-not-merge/hold
    - do-not-merge/work-in-progress
    - needs-ok-to-test
    - needs-rebase

  context_options:
    # Use branch protection options to define required and optional contexts
    from-branch-protection: true
    # Treat unknown contexts as optional
    skip-unknown-contexts: true
    orgs:
      org:
        required-contexts:
        - "check-required-for-all-repos"
        repos:
          repo:
            required-contexts:
             - "check-required-for-all-branches"
            branches:
              branch:
                from-branch-protection: false
                required-contexts:
                - "required_test"
                optional-contexts:
                - "optional_test"
Copy the code
  1. Run the following command to push config.yaml to the K8S cluster (replace the corresponding config.yaml file location) :
kubectl create configmap config --from-file=config.yaml=${PWD}/samples/config.yaml --dry-run -o yaml | kubectl replace configmap config -f -
Copy the code
  1. Go to the PR just now, you should see the following check:

  1. Since this is the Pr I mentioned, it will be carried automaticallyapprovedTag, now just add onelgtmThe label is ok. You need to get code Review people to look at the code and let them type it in/lgtm“And Prow will type it automaticallylgtmThe label. Since no one else played/LGTM for this demo, and I can’t comment on it for myself/lgtm, so this demo needs to manually select the LGTM label for this PR in Lables). The effect is as follows:

Advanced configuration

Prow is an efficient CI/CD system, but it is also a complex system. This article cannot explain all the advanced configurations. For further configuration, please refer to the official documentation. This Repo collates some commonly used scripts to facilitate subsequent configuration when using Prow. When using these scripts, take care to replace some data. For more information, see the OWNERS guide below.

OWNERS Guide

OWNERS is a configuration file that marks the owner of the folder to which the code belongs.

The working principle of

  1. OWNERSA file represents the owner of the directory where the file is located, including subdirectories. So in the root directoryOWNERSThe file has the highest permissions for the entire cluster, called “root owner”, and all Pr is automatically merged as long as root Owner agrees.
  2. Each folder under repo can be setOWNERS, if a certain Pr changes the contents of the latter subfolders of this folder, the OWNER of this folder needs to approve. If multiple changes are made, more than one person needs to approve. You can also call the root owner
  3. Can be found inOWNERSI’m going to set the Label in the file, so any Pr that changes to that folder will be labeled accordingly. Of course, this Label shouldn’t be in root OWNERS, so all Pr will be tagged.

The basic grammar

OWNERS is a YAML file whose basic form (and simplest configuration) is as follows:

approvers:
  - alice
  - bob     # this is a comment
reviewers:
  - alice
  - carol   # this is another comment
  - sig-foo # this is an alias
lables:
  - re/foo
  - re/question
Copy the code
  1. approversCan be used/approveCommand, which is the minimum condition that Pr can merge. You can notlgtmBut there has to beapproved. soappoversHas the Merge permission, similar to maintainer level. The other thing to note here is that the approvers proposed pr is satisfiedWorking principleThe second rule will be automatically typedapproved(This can happen in two ways: 1) he is the root owner; 2) he is the root owner. He is the subdir owner, and his code is in this directory), so you need to add another label restriction (usually LGTM) to the PR merge to prevent the approvers code from being merged automatically.
  2. reviewerTo be able to use/lgtmThe command, “Looks good to me,” to review code. The regular REPo will make the LGTM command a necessary condition, and the code that anyone mentions will not be automatically typedlgtmYou must use the command manually. The main constraint is that the code must be reviewed.
  3. If the approvers uses Github’s approve function, the approve label can also be labeled.

Advanced grammar

OWNERS also supports the Fliter parameter (which can’t be mixed with any of the above), which is primarily used to sort code files. See github.com/kubernetes/… Learn more about how to use it.