The author

Ran Xin, TCM product manager of Tencent Cloud Service Grid, is now responsible for planning and designing product features such as cloud native traffic access gateway and application communication observability.

Xu Liu, senior engineer of Tencent Cloud, focuses on the native field of container cloud, has many years of large-scale Kubernetes cluster management and microservice governance experience, and is now responsible for the design and research and development of TCM data side product architecture of Tencent cloud service grid.

The introduction

Application traffic management has always been one of the focuses of development operation and maintenance. With the development and change of computing resources, network environment and application architecture of service deployment, the development of traffic management solutions at the access layer can be roughly divided into traditional architecture and cloud native container. In order to meet the efficiency and demands of application delivery, different access layer solutions have emerged in each stage, from the initial simple load balancing, to the later reverse proxy such as HAProxy and Nginx, and now all kinds of Kubernetes Ingress Controller in the container environment. What are the characteristics of each stage of development? What are the challenges? What are the solutions?

phase Application deployment resource granularity Application architecture Application access addressing
Traditional architecture Physical/VIRTUAL machine (Low resource usage) A single or simple split module Management based on fixed IP addresses
Cloud native containerization Container (high resource utilization) As a service The container IP changes dynamically and is updated through the dynamic service registry

In the traditional architecture stage, the business is single application and three-tier architecture. Deployed on a physical machine or VM. Based on IP address management, the network environment is relatively fixed and basically unchanged. The service update iteration speed is slow. The access layer requires load balancing at layer 4 and layer 7, which can be supported by traditional load balancers. With the evolution of application architecture (the application has made a certain module split) and the improvement of iteration efficiency, some more complex access layer demands appear: According to the traffic content characteristic routing, gray scale publishing, traffic limiting, authentication, etc., generally by adding a layer of network proxy (e.g. Nginx) after the load balancer, network proxy Nginx has more capability of traffic processing at layer 7. Advanced features such as content routing, grayscale publishing, and authentication flow can be extended through Lua of the OpenResty community.

The ideal state of cloud native containerization stage is that business developers only need to focus on the implementation of business logic, do not need to care about resource scheduling and operation and maintenance management, and can truly achieve on-demand use, billing by volume. Vm or dedicated machine resources have coarse granularity and low utilization efficiency. Therefore, computing, storage, and network resources must be planned in advance, which is far from ideal. Cloud native stage, the granularity of container resources is finer, high utilization rate, start/destroy speed reaches second level, flexible elastic scaling (Kubernetes has become the industry standard of container scheduling, the following container environment refers to Kubernetes cluster); The network management environment has also changed, and the concept of Service has emerged. A micro Service is usually carried by a group of flexible flexible and dynamic scheduling containers (PODS), and the IP address of pods changes dynamically. This group of PODS generally provides external access through Service. Traffic management is based on Service. It is easier to split business modules and build applications in servitization. Coupled with the flexible scalability of the container environment, The DevOps concept is well implemented and the pace of iteration of microservices is accelerated, often requiring rolling updates. At this time, incoming traffic management faces the following new challenges:

  1. Needs to be integrated with Kubernetes to support forwarding traffic to specified pods.
  2. The speed of update iteration is accelerated, and the demand for grayscale release of new version of service is stronger.
  3. The concept of cluster is introduced. Service discovery between clusters is isolated, and the access layer must support service discovery across clusters. That is, the access layer can choose BACKEND as a Pod of multiple clusters. This is different from the traditional physical machine/VM phase, without cluster isolation. As long as network connectivity is ensured, the IP address of the back end of the access layer can be configured as any corresponding service.
  4. During the migration from the traditional phase to the cloud native phase, VM and container environments are mixed.

Based on the above challenges, the following access layer traffic management solutions have emerged for container environments:

  1. Kubernetes defines the Ingress API: Established network agents (e.g. Nginx, HAProxy) or cloud vendors’ Load balancing products (e.g. AWS Elastic Load Balancer, Tencent cloud CLB) all implement their own Ingress Controller. As an entry traffic management solution for a single cluster. The capabilities of the Ingress Controller, such as grayscale publishing and permission stream authentication, can be extended through annotations. Some Ingress Controllers also design their own traffic management model and syntax.
  2. Service Mesh Ingress: The boundaries of service discovery and management in a service grid are larger than the latitude of a cluster. For example, Istio Ingress Gateway provides cross-cluster service discovery capabilities. Backend can provide services from different clusters and register services running on VMS in the grid. Istio has also designed its own management model and syntax that declaratively supports consistent configuration of North-South + east-west traffic management.
  3. Use the network agent deployed on the original VM to forward traffic to the VM service or the Kubernetes cluster service.

This article will introduce you to various solutions of cloud native access layer traffic management and their advantages and disadvantages from the application scenario of cloud native container environment.

Cloud native access layer traffic management scenarios and solutions

Scenario 1: Basic traffic management

The first use scenario for inbound traffic management is the need to expose the service externally for clients to invoke. A common way is to expose services by URL. For example, an e-commerce website needs to route the request of /login to the login service and the request of /product to the commodity service. In this scenario, the access layer is required to be capable of routing based on traffic content.

Solution: Load Balancer + NodePort

In the early stages of containerization, when applications are deployed on both virtual machines and Kubernetes clusters, many users will use the original load balancer (e.g. Nginx, Tencent cloud CLB) to forward requests to virtual machines and containers respectively. Meanwhile, due to the container network solution, the original load balancer cannot directly access Pod IP. Therefore, NodePort is required to expose services within the cluster.

However, the scheme has the following problems:

  1. NodePort Limited number of ports (default 3000-32767)
  2. As clusters grow, Nginx profiles become more complex and difficult to manage
  3. After publishing the application to the Kubernetes cluster, users need to modify the Nginx configuration separately. The experience is not cloud native

Solution: Kubernetes Ingress

Kubernetes provides the Ingress API [1] to expose HTTP services within a cluster. Ingress supports routing requests to different services based on Host and Path. In order for Ingress to work, the cluster must have a running Ingress Controller (e.g. Nginx Ingress Controller). The native Ingress syntax provides simple Host, Path based routing, and the ability to configure TLS.

1. Route based on Host

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-services
  namespace: default
spec:
  rules:
  - host: service1.tencent.com
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
        path: /
  - host: service2.tencent.com
    http:
      paths:
      - backend:
          serviceName: service2
          servicePort: 80
        path: /
Copy the code

2. Route based on Path

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: public-services
  namespace: default
spec:
  rules:
  - host: services.tencent.com
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
        path: /service1
      - backend:
          serviceName: service2
          servicePort: 80
        path: /service2
Copy the code

3. The TLS configuration

Ingress also provides TLS support to expose the HTTP service in the cluster to HTTPS. We need to save the SSL certificate in the cluster as Secret, and then use Ingress to reference the newly created Secret.

apiVersion: v1
kind: Secret
metadata:
  name: public-services-tls
  namespace: default
data:
  tls.crt: base64 encoded cert
  tls.key: base64 encoded key
type: kubernetes.io/tls
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: services-with-tls
  namespace: default
spec:
  tls:
  - hosts:
      - services.tencent.com
    secretName: public-services-tls
  rules:
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80
        path: /service1
      - backend:
          serviceName: service2
          servicePort: 80
        path: /service2
Copy the code

Kubernetes Ingress summary: For simple HTTP traffic routing, using Ingress configuration is very easy, which is why Ingress is currently popular (according to CNCF 2020 Cloud Native survey report [2], 50% of users are using or will use third-party proxy for application traffic forwarding, Nginx and Envoy [3] are the most popular Ingress providers).

On the other hand, the functions of the native Ingress are very limited and cannot meet the requirements of many complex scenarios. Many third-party Ingress controllers [4] extend the functionality of the native Ingress through annotations or new configuration models and syntax, but are still limited by the problem of service discovery isolation between clusters and can only serve as a single-cluster entry traffic management scheme.

Scenario 2: Grayscale Release

Once the service is exposed to external access, you need to think about how to release and iterate smoothly and without risk. Two common methods are to cut part of the flow to the new version to verify the stability according to the weight or flow content, and then gradually transition to the new version after no problem, that is, we know gray release and AB test.

Kubernetes Ingress API does not have the function of grayscale publishing. Nginx Ingress Controller extends the function of the native Ingress API through annotations to achieve grayscale publishing. In contrast, Istio CRD configuration is more flexible and easy to use. The following describes how to use Istio VirtualService to configure grayscale advertising routing rules.

1. Based on weights

Istio can configure weighted grayscale publishing using Virtual services. The following is the current version of configuring 95% routing of incoming traffic from {namespace}/{gateway} to {Service}. Example of routing to the Canary version:

apiVersion: ...
kind: VirtualService
metadata:
  name: canary-weight
spec:
  hosts:
    - '*'
  gateways:
    - {namespace}/{gateway}
  http:
    - route:
        - destination:
            host: {service}
            subset: current
          weight: 95
        - destination:
            host: {service}
            subset: canary
          weight: 5
Copy the code

2. Based on the request

VirtualService also supports the configuration of content-based grayscale publishing routing rules, Header cookie “version=stable” route to {service} current version Example of route to {service} canary version when “version=canary” :

apiVersion: ...
kind: VirtualService
metadata:
  name: canary-content
spec:
  hosts:
    - '*'
  gateways:
    - {namespace}/{gateway}
  http:
    - match:
        - headers:
            cookie:
              exact: version=stable
      route:
        - destination:
            host: {service}
            subset: current
    - match:
        - headers:
            cookie:
              exact: version=canary
      route:
        - destination:
            host: {service}
            subset: canary
Copy the code

Scenario 3: Application traffic authentication and traffic limiting

Authentication and traffic limiting are two important capabilities to ensure the safety and robustness of north-south traffic.

The access layer is a unified gateway to access back-end services. Ensuring the security of the access layer is an important scenario for traffic management at the access layer. Generally, authentication and authorization rules need to be configured at the entrance. Istio has provided AuthorizationPolicy and RequestAuthentication CRD [5] since 1.5, which can flexibly configure authentication and authorization rules at the entry layer.

1. Request authentication (JWT)

The entry authenticates the Json Web Token carried in the request, clears the request carrying a valid Token, and rejects the request carrying an invalid Token.

The following is an example of configuring the Ingress Gateway to carry legitimate JWT requests using Istio RequestAuthentication:

apiVersion: ..
kind: RequestAuthentication
metadata:
  name: jwt-example
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
  - issuer: {issuer that issued the JWT}
    jwksUri: {URL of the provider’s public key set to validate signature of the JWT}
Copy the code

Authorized by 2.

Configure an authorization policy at an entrance to allow or deny traffic access based on traffic characteristics. For example, configure a black or white IP address list at an entrance. If an external authentication service is available, the inbound component is expected to connect to the external authentication service and release or deny traffic based on the returned authentication result.

The following is an example of configuring an IP block whitelist for the Ingress Gateway using Istio AuthorizationPolicy:

apiVersion: ...
kind: AuthorizationPolicy
metadata:
  name: white-list
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        ipBlocks: {single IP or CIDR}
Copy the code

Istio 1.9 enhances the support for AuthorizationPolicy to interconnect with external authentication systems. You can configure the Ingress Gateway to release or deny traffic based on the results returned by external authentication systems.

apiVersion: ...
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: CUSTOM
  provider:
    name: "my-ext-authz-service"
  rules: ...
Copy the code

3. The current limit

If the service is large and the back-end service is provided to many tenants, the request rate must be controlled at the entry point. For example, each User ID can only request the /product interface 100 times per minute.

In order to use the traffic limiting function of the Istio Ingress Gateway, Ratelimit Service needs to be installed first. Ratelimit service can be implemented by itself or directly used by the community [6]. Then Envoyfilter is used to configure traffic limiting rules. For specific configuration methods, refer to the official document [7].

Scenario 4: Inbound traffic management in heterogeneous Multi-cluster Scenarios

With the increase of business scale, or the improvement of the requirements for disaster recovery, data compliance and business isolation, the business will consider and implement the deployment of multiple Kubernetes clusters, and even the heterogeneous mixing of containerized environment and non-containerized environment, bringing a series of new challenges to the management of incoming traffic.

Multi-kubernetes clusters are generally based on disaster recovery and service isolation:

(1) Dr. The Kubernetes cluster has the attribute of region. The same application may be distributed in different geographical regions according to the service access duration and disaster recovery requirements of application delivery. In the multi-cloud or hybrid cloud (IDC and public cloud) architecture, multiple clusters need to be deployed. Cross-region multi-cluster Dr And nearby access can be provided by DNS resolution. However, DNS has cache, and failover may take a long time to take effect. In addition, some traffic cannot be transferred to the backup region based on the health of the service.

Istio is based on the following capabilities: 1. Multi-cluster service discovery. 2. Region awareness, fault awareness, and Dr Traffic capacity planning: 1. When services in all clusters are healthy, routes are routed to the nearest service based on the source of the request. 2. If some services in a cluster are faulty, a certain proportion of the traffic is transferred to the backup services of other clusters based on the health of the services.

(2) Business isolation. According to CNCF 2020 Cloud native survey report [2], application isolation with multiple clusters is the second only method to use namespace isolation, with the usage rate rising from 47% in 2019 to 50% in 2020. If multiple services share the same traffic inlet, the access layer must be able to discover services in multiple clusters and route traffic to the services in the specified cluster based on the specified policy.

Solution: Service Mesh Ingress

One challenge with the Kubernetes Ingress Controller is that the Kubernetes cluster isolates service discovery between clusters, and the Ingress Controller can only act as an entry point for cluster-level traffic. By virtue of the Service discovery capability of the control plane, the Service Mesh technology can discover or register services of multiple clusters or even heterogeneous services, breaking down Service discovery barriers between clusters and providing consistent access traffic forwarding management capabilities, independent of application deployment platforms.

Istio is the most popular open source Service Mesh project. Its access layer Istio Ingress Gateway also provides support for the Ingress API. However, it is not recommended to configure the Ingress Gateway using Ingress. This has greatly reduced Istio’s capacity. Istio provides a higher degree of abstraction from the traffic management model. You can directly use the Istio API to achieve more flexible traffic management capabilities, and achieve advanced features such as grayscale publishing, cross-cluster routing, and region awareness.

Istio Ingress Gateway is implemented based on Envoy [3], originally created by Lyft, Envoy is a high-performance service agent designed for cloud native scenarios, which was later donated to CNCF community by Lyft and graduated from CNCF.

1. Multi-kubernetes cluster service management

Istiod can obtain endpoints information through the API Server of all clusters in the grid, aggregate the information of multiple clusters, and push the final configuration to the Ingress Gateway. The Ingress Gateway can forward requests on demand to all pods in the grid.

2. Region-aware load balancing

In the service grid, the geographic information of a Pod includes the following three parts [8] :

  • Region (Region): usually represents a large geographical area (LLDB Beijing and Shanghai). In Kubernetes, the location of a node is determined by the labeltopology.kubernetes.io/regiondecision
  • Available Zone: a region usually contains more than one zone (e.g. Beijing 1 zone, Beijing 2 zone). In Kubernetes, the node’s zone is labeledtopology.kubernetes.io/zonedecision
  • Sub-zone: Allows for further partitioning of available areas for more fine-grained control, such as by rack. In Kubernetes there is no concept of sub-zones, Istio uses nodestopology.istio.io/subzoneTag to define a sub-zone

If the Kubernetes service hosted by a cloud vendor is used, the Region and Zone labels of the nodes have been configured by the cloud vendor. For example, in a TKE cluster, the nodes in Shanghai Region 2 have the following labels:

  • topology.kubernetes.io/region: sh
  • topology.kubernetes.io/zone: "200002"

Clusters within the grid may be distributed in different geographies and availability zones, and in most cases we want to minimize cross-geographies/cross-availability requests because of increased request latency. Therefore, the access layer must be able to sense endpoints and configure load balancing and failover policies based on geographic information.

(1) Regional failover

When areal load balancing is enabled, Istio tells the Ingress Gateway to forward the request to the nearest location. When all instances are healthy, the request stays in the same place, and when the instance is abnormal, traffic is distributed to the instance in the next priority region.

For example, the Ingress Gateway at bj.bj-01 forwards requests with the following priorities:

priority The geographical position
1 bj.bj-01 Region Zone is fully matched
2 bj.bj-02 Region Match The Zone does not match
3 sh.sh-01/sh-02 Region Zone does not match

(2) Regionally weighted load balancing

Regionally weighted load balancing distributes a user-defined percentage of traffic to certain regions. For example, you can use the following configuration to distribute traffic:

global:
  localityLbSetting:
    enabled: true
    distribute:
    - from: bj/bj-01/*
      to:
        "bj/bj-01/*": 70
        "bj/bj-02/*": 20
        "sh/sh-01/*": 10
Copy the code

3. Manage inbound traffic of heterogeneous services

In addition to multiple clusters, users in the process of cloud native transformation, often face some services have been container transformation, running in Kubernetes cluster, some inconvenient services are still in the VIRTUAL machine. There may even be some use of cloud vendor serverless cloud function services (e.g. AWS lambda). The access layer must be able to register and discover heterogeneous services to manage northbound and southbound traffic of heterogeneous deployment services.

Services on virtual machines can be registered to the grid through WorkloadGroup and WorkloadEntry provided by Istio. The same service can run on both the Kubernetes cluster and virtual machines.

Istio Ingress Gateway Summary: Istio Ingress Gateway provides multi-cluster service discovery, region awareness, traffic capacity planning, and more powerful and flexible traffic management apis in the scenarios of entry gray scale publishing, security, and heterogeneous traffic management in multiple clusters.

But at the same time, users have to deal with the complexity of Istio. The cost of maintaining Istiod and Istio Ingress Gateway and integrating metric, Trace, log and other observability and certificate management systems is relatively high. You also need to correctly configure various CRDS, such as Gateway VirtualService DestinationRule.

Function comparison of access Layer solutions

The following is a comparison of common access layer solutions in the Tencent cloud container environment.

Multi-cluster grayscale publishing/Cross-cluster Dr Demo

The following will use Tencent cloud Service grid TCM console to demonstrate Service Mesh Ingress to do long Kubernetes cluster environment gray publishing and disaster recovery.

  1. Create a service grid, add two service discovery clusters for deploying services (basic monitoring indicators are automatically connected to cloud monitoring, which can be viewed on the console. Enable cloud native monitoring as required to meet custom monitoring requirements), and select Enable Ingress Gateway
  2. Use the Destination Rule to define the version of the Frontend service (frontend service has the same deployment in both clusters)
  3. Configure the ingress Gateway listening rule using the Gateway, enable HTTPS access through port 443, and use the Tencent cloud SSL platform server certificate
  4. Use VirtualService to configure routing rules. 50% of the traffic is routed to V1 and 50% to V2
  5. After receiving an access request, check the workload monitoring (frontend, frontend- Canary). The traffic of both versions is 1:1
  6. When the gray scale ends, change the weight, and 100% traffic is routed to V2. Check the workload monitoring data again, and find that all traffic is requested to Frontend – Canary
  7. In the following, we adjust the number of frontend service workload pods in one cluster to 0 to simulate the failure of frontend service in one cluster. After finding the failure of Frontend service in one cluster, the frontend service can still be accessed normally. Check the workload monitoring of the Frontend service in the other cluster. You can find that the inbound bandwidth is doubled, indicating that traffic Dr Is switched to the other cluster after the service in one cluster is faulty.
  8. If you need to extend east-west traffic management, you can inject An Envoy Sidecar into your business, which can use the same SET of Istio apis to implement consistent North-South east-west traffic management, out-of-the-box network topology, call tracking and other observable features.

TCM, Tencent Cloud Service Mesh product fully compatible with Istio, has implemented the hosting of control side components. Using TCM Ingress Gateway, you only need to deploy a set of data side Envoy Pods in the business cluster. All of the Istio Ingress Gateway’s entry traffic management capabilities are available out of the box. Meanwhile, TCM integrates Tencent cloud monitoring and certificate peripheral products to provide out-of-the-box observation capability and certificate configuration functions.

conclusion

This article is introduced from the two phases of business deployment evolution and introduces:

  1. Typical scenarios of traffic management at the access layer in a cloud native container environment.
  2. Inlet flow management solutions and pros and cons comparison.
  3. The TCM of Tencent cloud Service grid is taken as an example to demonstrate the grayscale publishing and cross-cluster Dr Capabilities of Service Mesh Ingress in a multi-cluster environment.

The main conclusions are:

  1. For simple HTTP traffic routing, using Kubernetes native Ingress configuration is very easy, Some Ingress controllers (e.g. Nginx, Traefik) also extend the functionality of the native Ingress through annotations or CRD, but remain traffic portals at the cluster level.
  2. Service Mesh-level access layer. With the Service discovery capability of the control plane, it can serve as a unified traffic entrance in a multi-cluster or heterogeneous environment, and has advanced features such as cross-cluster routing and region awareness. You can also smoothly extend the consistent syntax to manage east-west traffic.

This article is the first in a series on cloud native access layer traffic management, which will be followed by a series of articles that will detail best practices for inbound traffic management, security, observability, and multi-cluster heterogeneous inbound traffic management scenarios.

Reference

[1] kubernetes. IO/docs/concep…

[2] www.cncf.io/wp-content/…

[3] www.envoyproxy.io/docs/envoy/…

[4] kubernetes. IO/docs/concep…

[5] istio. IO/latest/docs…

[6] github.com/envoyproxy/…

[7] istio. IO/latest/docs…

[8] istio. IO/latest/docs…