Now that the frame is in place, it’s time to take a seat. In the process, we encountered some small problems that could lead to pit mining if we weren’t careful.

The gateway transmits the source IP address

The problem is twofold. On the one hand, of course, Kong passes the source IP, and on the other hand, Kubernetes retains the source IP when forwarding traffic to Pods.

So let’s assume that the link is,

Request --> load balancer --> Cluster node --> Kong Gateway instance --> PodCopy the code

Previously we said that our Kong is after the default nginx-ingress-controller in Aliyun, but we have directly removed it for simplicity.

Kubernetes’ policy of reserving client source IP addresses

Reference Documents:

K8s official document: Reserve the source IP address of the client.

Service. The spec. ExternalTrafficPolicy – said the service will hope the external flow routing to the end of the scope of local or cluster nodes. There are two options available: Cluster (default) and Local. The Cluster hides the client source IP, which may cause the second to jump to another node, but has a good overall load distribution. Local reserves the source IP address of the client and avoids the second hop of LoadBalancer and NodePort services. However, it may cause unbalanced traffic transmission.

Multiple SLBS in the container service load balancing and transparent source IP.

To sum up, if we want to keep the source IP, then we need to choose Local. However, it is important to note that external requests reach cluster nodes through load balancing of public IP addresses.

Note: How the load balancer loads requests to the nodes is determined by the cloud vendor. Taking Ali Cloud as an example, the back end supported by load balancing is a virtual server group, which contains all common nodes in the K8s cluster. If nodes in the cluster expand or shrink, the group can be synchronized in time.

As shown in the figure above, load balancing will evenly distribute traffic to each common node, and Kong Pod on a single node will evenly distribute node traffic. Overall, There will be uneven traffic received by Kong Pod. What is more serious is that there is no Kong Pod instance on the node on the far right of the figure above. Then the traffic on this node will be abnormal. This is the problem of uneven load mentioned earlier.

Kubernetes real solution to retain client source IP

Based on the previous analysis, we cannot temporarily require aliyun to load only the nodes where The Kong Pod instance is deployed. Therefore, we can only require each common node to have the Kong Pod instance, and in order to avoid uneven load, each node can only have one Kong Pod instance.

Summary: One Kong Pod instance is required on each normal node.

Option 1: Deploy the Kong gateway on DaemonSet. Scheme 2: Deploy The Kong gateway in Deployment mode, and the number of Pod and Pod is the same as that of nodes.

Here we chose option 2, mainly for compatibility with the original deployment, and option 1 is ok if there is no historical baggage.

Anti-affinity configuration

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values: ingress-kong
                  - 
              topologyKey: kubernetes.io/hostname
Copy the code

Kong Reserves the source IP address

At this point, we ensure that the request with source IP information reaches the Kong Pod instance, and then it is up to Kong Pod to forward the request to the back-end instance with source IP information.

Kong Official document: Preserving Client IP Address. Kong Nation How to Forward Client’s request IP

For example, I changed the configuration to:

        - name: KONG_TRUSTED_IPS
          value: 0.0.0.0/0,::/0
        - name: KONG_REAL_IP_HEADER
          value: X-Forwarded-For
Copy the code

Acceptance of the results

We deployed the Echo service in the K8s cluster and configured routing to expose it.

This header is forwarded to Forwarded-for, and the IP address of the client is forwarded to Forwarded-for by the x-Real-IP header.

conclusion

Requests can be forwarded multiple times in the gateway + K8s architecture. Therefore, ensure that the client IP information is not lost during each transfer. If you have a clear understanding of the overall architectural links, then implementing this functionality will come naturally.