Author: Coding-Wei Wang

1. The background

If you are particularly concerned about Kubernetes cluster security, then we might want to implement these requirements:

  • How to implement Kubernetes cluster two-step authentication, in addition to cluster credentials, but also need to provide one-time Token verification?
  • How to verify whether the deployed image is safe and compliant, so that only Docker images in the company’s internal image warehouse can be deployed?
  • How to dynamically inject sidecar into each Deployment to meet specific security or business requirements?
  • How to implement cluster-level imagePullSecrets that automatically inject imagePullSecrets into a new namespace when it is created?

This paper takes Kubernetes two-step verification as an example, uses Kubernetes Admission dynamic Admission control, and implements a two-step verification Demo with the help of Serverless, so that readers can have a deeper understanding of dynamic Admission control and Serverless.

1.2 Effect

The Token two-step verification failed and cannot be deployed

Token two-step verification succeeds and deployment is allowed

2. What is Admission

Admission refers to the step of persisting resources to ETCD after users perform Kubectl certification. To decouple this part of logic, Kubernetes invokes Webhook to realize the supplement of user-defined business logic. All of the above is during the life cycle of the user executing kuberctl and waiting for the API Server to synchronize the results.

(1) and (2) shown in the figure above are the workflow of Admission. We can find the following features:

  1. Admission works after cluster certification
  2. There are two kinds of Admission:Mutating 和 Validating
  3. Both of these concrete implementations are based onWebhookImplementation of the
  4. The Admission operation objects can be currently deployed users, Yaml content, and so on

2.2 Admission Mutating

The Mutating controller can modify the resource file before persisting the resource to ETCD, such as dynamically adding Labels to a particular POD, dynamically injecting sidecar, etc. Careful readers will notice that Admission Mutating is used in many products, such as Istio, which uses it to dynamically inject each container with sidecar Envoy containers for traffic hijacking and management.

2.3 Admission Validating

Validating as you undergo Mutating, you can implement your own custom validation logic in this phase. This article uses it to implement a simple two-step verification mechanism.

3. What is Admission Webhook

As simulated by Mutating Controllers and Valcontrollers, that is to say, We need to provide an external Webhook Endpoint for the Kubernetes cluster. When the API Server executes the corresponding process, it will call our predefined Webhook to implement our predefined business logic. By returning the specified data structure, To implement changes or validation to Yaml files.

4. Get hands-on

4.1 Cluster Conditions

According to the official documentation, the prerequisites are as follows:

  • The Kubernetes cluster version must be at least V1.16
  • Enabled MutatingAdmissionWebhook and ValidatingAdmissionWebhook controllers

If you are not sure, run the following command:

kubectl get pods kube-apiserver -n kube-system -o yaml | grep MutatingAdmissionWebhook,ValidatingAdmissionWebhook
Copy the code

If you are using a managed cluster, use the following command to query:

kubectl api-versions | grep admission
Copy the code

If there is a admissionregistration. K8s. IO/v1beta1 illustrate clustering support, for the next step.

4.2 Other Conditions

  • Open CODING to the conversation
  • Clone the admission-webhook-example. Git repository and push it to your CODING git repository
  • Prepare a Tencent cloud account

4.3 Deploying the Tencent Serverless Service

  1. Log in to CODING, and configure Serverless identity authorization, record credential ID (similar: B68948CB-2AD9-4B67-8A49-AD7BA910ED92), and use it later

  2. Clone code repository admission-webhook-example

git clone https://e.coding.net/wangweicoding/admission-webhook-example.git
Copy the code
  1. Modify files in the root directory
  • Jenkinsfile in the root directory, replace the credential ID obtained in the previous step with the credential ID at the cursor
  • Modify the serverless /. The envVPC_ID 和 SUBNET_IDThese two items are available on Tencent cloud console.”Private network”Find; If no private network or subnet exists, you can create one by yourself. Select Guangzhou for region.
  • When you’re done, push the code to your ownCODING GitCode warehouse

  1. Create a build plan using the “Blank template”, select “Use Jenkinsfile from the repository”

  2. Run the build plan and deploy the Serverless service

    After the run is complete, click the “Output Endpoint” phase to see the output URL (similar to:Service-faeax9cy-1301578102.gz.apigw.tencentcs.com/release/ind…This URL is the URL for the Serverless service to provide services externally.Records are used for the next stage

So far, Tencent Cloud Serverless service has been deployed.

4.4 Kubernetes Cluster Deployment Is like Valberwebhook

Since Admission Webhook only allows HTTPS and requires certificate information, we need to generate it in advance. The code repository has provided scripts to configure cluster certificates.

$ ./deployment/webhook-create-signed-cert.sh

creating certs intmpdir /var/folders/mt/965plkfs62v6wqx2839qthz40000gq/T/tmp.i1imELSt Generating RSA private key, 2048 bit long modulus (2 primes) ................... + + + + +... +++++ e is 65537 (0x010001) certificatesigningrequest.certificates.k8s.io/admission-webhook-example-svc.default created NAME AGE REQUESTOR CONDITION admission-webhook-example-svc.default 1s admin Pending certificatesigningrequest.certificates.k8s.io/admission-webhook-example-svc.default approved secret/admission-webhook-example-certs configured (base)Copy the code

Modify the deployment/deployment. The yaml file, will serverlessURL replacing for the Endpoint of a phase record (similar to: Service-faeax9cy-1301578102.gz.apigw.tencentcs.com/release/ind…

Once the certificate is successfully created, deploy the Deployment and Services

$ kubectl create -f deployment/deployment.yaml
deployment.apps "admission-webhook-example-deployment" created

$ kubectl create -f deployment/service.yaml
service "admission-webhook-example-svc" created
Copy the code

So far we used to receive Validating requested service has been deployed, the final configuration ValidatingWebhookConfiguration, run the following command:

cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml
Copy the code

After executing, you can see that the caBundle field of ValidatingWebhook-ca-bundle. yaml has been replaced.

The script runs on jQ (Shell reads JSON), if you have not installed it, please go to www.ibm.com/developerwo…

Brew Install JQ can be used to install the Mac OS.

Next, we play tag for the default namespace, because our ValidatingWebhookConfiguration use namespaceSelector only to do two steps with a specific labels namespace validation.

$ kubectl label namespace default admission-webhook-example=enabled
namespace "default" labeled
Copy the code

Finally, create ValidatingWebhookConfiguration

$ kubectl create -f deployment/validatingwebhook-ca-bundle.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io "validation-webhook-example-cfg" created
Copy the code

This way, once the resource is created in the default namespace, our deployed service (Deployment) will intercept the request and verify it twice.

4.5 Try two-step verification

At this point, we have successfully deployed the two-step verification Demo, and the overall architecture diagram now looks like:

Now we can try to deploy

$ kubectl apply -fYaml Error from server (Token Error, deployment not allowed): Error when creating"deployment/sleep.yaml": admission webhook "required-labels.coding.net"Denied the request: The Token is incorrect and cannot be deployedCopy the code

Since we pre-configured four sets of tokens to the database when we created the Serverless service, they are: 1111, 2222, 3333, 4444, so we can modify the sleep, yaml, to annotate the metadata. The annotations. The token is changed to 1111, trying to deploy again

$ kubectl apply -f deployment/sleep.yaml
deployment.apps/sleep created
Copy the code

The deployment is successful. If the token is used repeatedly, the authentication fails. At this point, two-step Serverless validation is complete.

5. Source code analysis

5.1 What does the Deployment we deployed do

When kubectl apply is executed, API Server forwards the request to our deployed POD, with the core code in the project root, mainly main.go and webhook.go

Main. go basically starts an HTTP service and reads the certificate we created from the command line along with the Serverless Endpoint

// main.go

flag.IntVar(&parameters.port, "port", 443, "Webhook server port.")
flag.StringVar(&parameters.certFile, "tlsCertFile"."/etc/webhook/certs/cert.pem"."File containing the x509 Certificate for HTTPS.")
flag.StringVar(&parameters.keyFile, "tlsKeyFile"."/etc/webhook/certs/key.pem"."File containing the x509 private key to --tlsCertFile.")
flag.StringVar(&parameters.serverlessURL, "serverlessURL"."https://example.com"."serverless endpoint URL.")
Copy the code

Webhook. go mainly forwards requests sent by the API Server. We rewrite validate to forward all requests to Serverless Endpoint.

// webhook.go

glog.Infof("parameters.serverlessURL is %v", whsvr.parameters.serverlessURL) res, _ := Post(whsvr.parameters.serverlessURL, JsonData := make(map[string]string) Body _ = json.newdecoder (res.body).decode (&jsonData) glog.infof ("res is %v", jsonData)
allowed := false
reason := &metav1.Status{
    Reason: "Token error, deployment not allowed",}if jsonData["allow"] = ="true" {
    allowed = true
}
return &v1beta1.AdmissionResponse{
    Allowed: allowed,
    Result:  reason,
}
Copy the code

After POD forwards the request to our Serverless function, it does the business logic to decide whether to allow access. POD then reformats the Serverless results and returns them to the API Server.

5.2 What does Serverless do?

The Serverless service we deployed mainly consists of four parts:

  • API Gateway
  • Cloud function
  • Postgresql
  • VPC

We used CODING DevOps to deploy the above Serverless services in Tencent cloud, Jenkinsfile core code:

stage('Deploy Serverless service') {
  steps {
    withCredentials([string(credentialsId:"b68948cb-2ad9-4b67-8a49-ad7ba910ed92", variable:'tencent_serverless')]) {
      sh 'echo "${tencent_serverless}" > .tmp'
      sh ' '' SecretId=$(cat .tmp | jq -r .SecretId) SecretKey=$(cat .tmp | jq -r .SecretKey) token=$(cat .tmp | jq -r .token) AppId=$(cat .tmp | jq -r .AppId) echo "TENCENT_SECRET_ID=${SecretId}" >> ./serverless/.env echo "TENCENT_SECRET_KEY=${SecretKey}" >> ./serverless/.env echo "TENCENT_APP_ID=${AppId}" >> ./serverless/.env echo "TENCENT_TOKEN=${token}" >> ./serverless/.env '' '
      sh 'cd serverless && cat .env'
      sh 'cd serverless && npm run bootstrap && sls deploy --all | tee log.log'
      sh 'rm ./serverless/.env'
    }
    echo 'Deployment complete'
  }
}
stage('output Endpoint') {
  steps {
    sh 'cd serverless && cat log.log | grep apigw.tencentcs.com'}}Copy the code

The focus here is on using temporary credentials and deploying the predefined Serverless. Yml using the Serverless SDK.

The API Gateway provides external network access

#. / serverless/API/serverless. Yml API Gateway deployment file

events:
  - apigw:
      name: k8sAdmission
      parameters:
        protocols:
          - http
          - https
        serviceName:
        description: Based on Tencent Cloud Serverless, it provides dynamic access control for K8S
        environment: release
        endpoints:
          - path: /index
            method: POST

Copy the code

Postgresql is responsible for storing predefined tokens

#. / serverless/db/serverless. Yml database deployment files

org: k8sAdmission
app: k8sAdmission-db
stage: dev

component: postgresql
name: fullstackDB

inputs:
  region: ${env:REGION}
  zone: ${env:ZONE}
  dBInstanceName: ${name}
  vpcConfig:
    vpcId: ${output:${stage}:${app}:serverlessVpc.vpcId}
    subnetId: ${output:${stage}:${app}:serverlessVpc.subnetId}
  extranetAccess: false
Copy the code

The VPC communicates with the Postgresql network

#. / serverless/VPC serverless. Yml VPC deployment file

org: k8sAdmission
app: k8sAdmission-db
stage: dev

component: vpc # (required) name of the component. In that case, it's vpc.
name: serverlessVpc # (required) name of your vpc component instance.

inputs:
  region: ${env:REGION}
  zone: ${env:ZONE}
  vpcName: serverless
  subnetName: serverless
Copy the code

The cloud function is responsible for the access logic judgment, as you can seehandler: api_service.main_handler, that is, the entry function of the cloud function ismain_handlerWhen an external request comes in, it will be executedmain_handlerfunction

#. / serverless/API/serverless. Yml cloud function deployment file

org: k8sAdmission
component: scf # (required) references the name of Component, currently used in the 0700-SCf component
name: k8s # (Mandatory) Name of the instance created by this component
app: k8sAdmission-db # (Optional) Name of the SCF application
stage: dev # (Optional) Used to distinguish environment information. The default is dev

inputs:
  src: ./
  name: ${name}Description: K8S dynamic access control handler based on Tencent Cloud Serverless: API_service.main_handler# entry functionThe runtime: Python3.6# runtime environment for cloud functions. Except Nodejs10.15, the value can be Python2.7, Python3.6, Nodejs6.10, Nodejs8.9, PHP5, PHP7, Golang1, or Java8.
  region: ${env:REGION}
  vpcConfig:
    vpcId: ${output:${stage}:${app}:serverlessVpc.vpcId}
    subnetId: ${output:${stage}:${app}:serverlessVpc.subnetId}
  timeout: 10
  environment:
    variables:
      PG_CONNECT_STRING: ${output:${stage}:${app}:fullstackDB.private.connectionString}
      PG_DN_NAME: ${output:${stage}:${app}:fullstackDB.private.dbname}
  events:
    - apigw:
        name: k8sAdmission
        parameters:
          protocols:
            - http
            - https
          serviceName:
          description: Based on Tencent Cloud Serverless, it provides dynamic access control for K8S
          environment: release
          endpoints:
            - path: /index
              method: POST
Copy the code

Cloud function key code

We will create the TOKENS table on the first trigger (request) and insert the four predefined groups of TOKENS into the table. We checked whether tokens carried in annotations from kubectl Apply yamL file were valid and compared tokens stored in the Postgresql database.

#./serverless/ API /api_service.py Cloud function business logic

def main_handler(event,content):
    logger.info('start main_handler')
    logger.info('got event{}'.format(event))
    logger.info('got content{}'.format(content))
    Connect to database
    print('Start Serverlsess DB SDK function')

    conn = psycopg2.connect(DB_HOST)
    print("Opened database successfully")

    cur = conn.cursor()
    cur.execute(' ''CREATE TABLE IF NOT EXISTS TOKENS (ID INT PRIMARY KEY NOT NULL, tokens TEXT NOT NULL); '' ')
    conn.commit()

    cur.execute("select * from TOKENS")
    myresult = cur.fetchall()
    for row in myresult:
       print("ID = " + str(row[0]))
       print("tokens = " + row[1])

    if not bool(cur.rowcount):
        print("insert default tokens")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \ VALUES (1, '1111')")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \ VALUES (2, '2222')")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \ VALUES (3, '3333')")
        cur.execute("INSERT INTO TOKENS (ID,tokens) \ VALUES (4, '4444')")
        conn.commit()

    json_dict = json.loads(event["body"])
    if json_dict["object"] ["metadata"] ["annotations"] ["token"] = ="":
        return {"errorCode": 0."errorMsg":""."allow":"false"}
    cur.execute("SELECT * FROM TOKENS where tokens=%s",[json_dict["object"] ["metadata"] ["annotations"] ["token"]])
    myresult = cur.fetchall()
    allow = "false"
    if len(myresult) > 0:
        allow = "true"
        query_id = myresult[0][0]
        cur.execute("DELETE FROM TOKENS where ID=%s",[query_id])
        conn.commit()
    conn.close()
    return {"errorCode": 0."errorMsg":json_dict["object"] ["metadata"] ["annotations"] ["token"]."allow":allow}
Copy the code

If the token exists in the database, delete the token used this time from the database and return JSON to our POD deployed in the cluster

{"errorCode": 0."errorMsg":"tokens"."allow":"true"}
Copy the code

POD reassembles the information based on the result returned by Serverless and returns the following JSON to Kubernetes API Server

{
    "UID":"b24ab5f7-8b6b-4ea2-83ff-6f9834a9937e"."Allowed":false."Result": {"ListMeta": {"SelfLink":""."ResourceVersion":""."Continue":""
        },
        "Status":""."Message":""."Reason":"Token error, deployment not allowed"."Details":""."Code": 0}."Patch":""."PatchType":""
}
Copy the code

The Allowed field is the key to whether kubectl apply is Allowed, and the Reason information is displayed as the result.

Some students here may ask, why call the Serverless service through our deployed POD? Can’t we have the API Server request Serverless Endpoint directly? We need to create Kubernetes CA-signed TLS certificate to ensure the security of communication between Webhook and API Server. So we did it this way.

6. Conclusion

At this point, we have implemented a simple Kubernetes two-step validation. More logic, such as determining image compliance and rejecting deployment of images from non-internal repositories, can be implemented within the Serverless cloud function.

In production practice, token in this example is a dynamic YAML artifact type deployment, and we can combine CODING continuous deployment to provide dynamic parameter binding for artifact files.

If you want to dynamically inject a Sidecar into a Deployment, you can use Mutating Webhook to listen on the deployed Deployment and inject the sidecar dynamic Patch that needs to be injected.

If you want to create cluster-level imagePullSecrets, you can use Mutating Webhook to create namespaces. Automatically add existing imagePullSecrets Patch to new namespaces.

To implement Mutating Webhook, note that the mutate function in the webhook.go file in the project root directory is similar to Valwebhook except that it is mainly implemented through Patch.

Kubernetes Admission decouples the Kubectl process via Webhook so that our own business logic can be dynamically added to the process from kubectl execution to kubectl return. This two-step verification is just a simple Demo. For a more in-depth look, see the resources link.

7. Reference materials

  • In-depth introduction to Kubernetes admission webhooks
  • Expanded Admission is in Beta
  • Dynamic access control
  • Tencent Serverless
  • This project Fork source