Recently, I have been working to migrate a small Puppeteer-based data crawler from a stand-alone docker-compose deployed environment to a self-built mini-hybrid cloud (K3S cluster). The Istio service grid was deployed on the cluster for the purpose of lie learning. As a result, all the Ingress API interfaces were normal, except for the failure of the data crawl task, and the skull began to ache slightly.

The first stage of the circle

A log check found that the target site returned 502, and the return is 100%, not an occasional phenomenon

Since this interface is returned by the WAF of the target site, it can only be seen after it is considered as garbage traffic or the IP is banned due to incorrect configuration, so my first reaction is that it is banned. Using curl and Headless Chrome to request the same URL on the Node where the Pod is located.

Something magical happened and everything turned out fine.

So it is basically confirmed that there is a problem with the environment in K3S Pod. Considering that it was also run in Docker container before, it is the traditional Docker Network after all, and now the environment is Containerd + Flannel of K3S. I decided to add label IStio-injection =disabled to turn off Sidecar injection for the grid in bare Pod form.

Something even more amazing happened, and it turned out to be normal.

The problem is basically locked that after Istio’s Sidecar injection, those in the container cannot normally access the target site of the crawler. In order to verify this, we create two bare pods and run busybox

kubectl run curl-test-with-istio --image=radial/busyboxplus:curl -i --tty --rm --overrides='{"spec": { "nodeSelector": {"k3s.io/hostname": "homelab"}}}' Namespace default Infuses sidecar by default and runs on the host with hostname as homelab
kubectl run curl-test-without-istio --image=radial/busyboxplus:curl -i --tty --rm --overrides='{"metadata": {"label": {"istio-injection": "disabled"}}, "spec": { "nodeSelector": {"k3s.io/hostname": "homelab"}}}' Make sure sidecar injection is turned off and running on homelab with hostname
Copy the code

Running the curl target site in the two bare pods revealed that ISTIO was indeed causing the problem.

The second stage is root searching

From Istio official documentation (istio.io/latest/docs…) As we know, the Pod requests injected with Sidecar are forwarded by the Envoy and can be configured using the following command.

kubectl get istiooperator installed-state -n istio-system -o jsonpath='{.spec.meshConfig.outboundTrafficPolicy.mode}'
Copy the code

The obtained mode can be ALLOW_ANY or REGISTRY_ONLY. ALLOW_ANY (default) indicates direct forwarding without control, and REGISTRY_ONLY indicates that direct access to the Internet is prohibited. This mode can be configured via istioctl:

istioctl install -f config.yaml --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY
Copy the code

-f config.yaml is the parameter used for installation. If mode=REGISTRY_ONLY, ServiceEntry needs to be configured to allow external links as follows:

kubectl apply -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: httpbin-ext spec: hosts: - httpbin.org ports: - number: 80 name: http protocol: HTTP resolution: DNS location: MESH_EXTERNAL EOF
Copy the code

The ServiceEntry configuration allows access to httpbin.org. You can configure the port, Protocol, Resolution, and Location fields to allow each address.

Said back to the question, I installed the grid configuration file in meshConfig. OutboundTrafficPolicy. ALLOW_ANY mode, according to the truth will not obstruct the foreign connection, and the effect is connected to, added bootleg forwarding seems to lead to the request is unusual, So the above tolerance rules are necessary to control the flow outlet of the cluster container, but are not the cause of the problem.

Curl Naked Pod: httpbin.org

curl http://httpbin.org/headers
curl http://httpbin.org/user-agent
curl http://httpbin.org/ip
Copy the code

It can be seen that Istio’s Envoy does add personal effects to the request header, including traced headers and Envoy information, but we visited the target site with the same header directly on a bare machine and did not find 502, so this is not the main cause of the problem either.

Despairingly, he pulled out his bag:

sudo tcpdump tcp port 80 and host www.****.com -w dump.pcap
Copy the code

Two request packets were captured using isTIO and bare machine respectively to request the target site

Direct comparison did not find more information, until I tried to replicate the two requests into curl Request and finally found some clues

Bianry (data) and Content-Length (content-Length) headers (data) : error 502: error 502 (data) bianry (data) and Content-Length (content) headers (data) : error 502: Error 502 (data) And this request, retweeted by Envoy, did just that.

The third stage fills in the hole

Now that you know the problem, there is a solution:

  1. Add an Issue/PR to Istio community to increase outbound transparent forwarding capability (inbound transparent forwarding has been added istio. IO /latest/docs at the end of 2018 because it is used to get SourceIP)
  2. As described in the previous document, configure the includeIPRanges so that only outbound traffic within the cluster is forwarded and outbound traffic bypasses the proxy
  3. The Istio Egress Gateway attempts to forward requests in a centralized manner

Considering that this situation is relatively special, the community has not found similar problems and appeals (maybe I am too poor in English to find them), and I am still familiar with the code of Istio. The choice of 1 is a long-term project, so take your time.

My cluster node resources are relatively advanced and have multiple outlet appeals. The Gateway solution is expensive and doesn’t help much considering the underlying Envoy.

Bypass, istioctl can be used to complete global configuration:

istioctl install -f config.yaml --set values.global.proxy.includeIPRanges="10.42.0.0/24"
Copy the code

At present, however, only crawlers encounter this problem. This Bypass policy only takes effect for Pod in a single Deployment, which can be completed by configuring annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        traffic.sidecar.istio.io/includeOutboundIPRanges: "10.42.0.1/24" # bypass Istio Egress
    spec:
      .
Copy the code

At this point, the crawler access problem is solved. Outbound traffic management provided by Istio can effectively implement features such as link tracing, routing, and security control, but it also increases link length and risks of incompatibility. Therefore, it is advisable to use it as needed

The above is a small front-end cloud native practice notes, there may be improper, welcome to point out! Big guy pat (doge