Background:

Deploy the application in the Kubernetes cluster and stress test the application. Jmeter stress tests at around 300 requests per second (18,000 requests per minute for ElasticSearch). Nginx erro log:

But my CPU memory resources are not full either. Php-fpm is also a socket: ​

See also: www.bubuko.com/infodetail-…

Solve a problem:

Modify the.net. Core. Somaxconn

Enter your own nginx-php container to view:

Bash - 5.0 # cat/proc/sys/net/core/somaxconn 128Copy the code

Select somaxconn from a random work node:

root@ap-shanghai-k8s-node-1:~# cat /proc/sys/net/core/somaxconn
32768
Copy the code

Note: This is a TKE cluster. The parameters are default. Unmodified Modify the application configuration file as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: paper-miniprogram
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: paper-miniprogram
  template:
    metadata:
      labels:
        app: paper-miniprogram
    spec:
      containers:
        - name: paper-miniprogram
          image: ccr.ccs.tencentyun.com/xxxx/paper-miniprogram:{data}
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "1024M"
              cpu: "1000m"
            limits:
              memory: "1024M"
              cpu: "1000m" 
      imagePullSecrets:                                              
        - name: tencent
---

apiVersion: v1
kind: Service
metadata:
  name: paper-miniprogram
  labels:
    app: paper-miniprogram
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: paper-miniprogram

Copy the code

The modifications are as follows: Add the initContainers configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: paper-miniprogram
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: paper-miniprogram
  template:
    metadata:
      labels:
        app: paper-miniprogram
    spec:
      containers:
        - name: paper-miniprogram
          image: ccr.ccs.tencentyun.com/xxxx/paper-miniprogram:{data}
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "1024M"
              cpu: "1000m"
            limits:
              memory: "1024M"
              cpu: "1000m" 
      initContainers:
      - image: busybox
        command:
        - sh
        - -c
        - echo 1000 > /proc/sys/net/core/somaxconn
        imagePullPolicy: Always
        name: setsysctl
        securityContext:
          privileged: true
      imagePullSecrets:                                              
        - name: tencent
---

apiVersion: v1
kind: Service
metadata:
  name: paper-miniprogram
  labels:
    app: paper-miniprogram
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: paper-miniprogram

Copy the code

The php-fpm list. backlog parameter is modified

Take a look at the value of the system variable net.ipv4.tcp_max_syn_backlog

cat /proc/sys/net/core/netdev_max_backlog
#OR
sysctl -a|grep backlog
Copy the code

Then take a look at LISTEN in PHP. Backlog configuration:The value of net.ipv4.tcp_max_syn_backlog in the container must be modified in privileged mode.

Official about SYSCTL

Kubernetes officials have syscl instructions: kubernetes. IO/useful/docs/tas…

And then the aftereffects of doing so:

I personally feel that privileged mode will bring security issues, or do not like pod enabled privileged mode.

Personally, I think the best way is:

  1. The Grafana kanban shows that THE RESOURCE utilization of POD is still not that high. Adjust resource limits parameters properly.

  1. Enable automatic hPA horizontal scaling.
  2. I also want to keep the default net.core. Somaxconn =128. And rely on more copies to meet the high load. This is also consistent with the idea of using containers.
  3. The point is that many people are wrong to think that expanding resources can increase concurrent load. It’s better to tune the parameters.

About PHP-FPM Unix socket and TCP

See Also Zhihu:zhuanlan.zhihu.com/p/83958307 ​

Some configurations are available for reference:

Github.com/gaoxt/blog/…

Blog.csdn.net/pcyph/artic…