Teaches you to quickly deploy an ES cluster in Kubernetes

Abstract:ES cluster is a powerful tool for big data storage, analysis and fast retrieval. This paper briefly describes the architecture of ES cluster, and provides an example of rapid deployment of ES cluster in Kubernetes. This paper introduces the monitoring operation and maintenance tools of ES cluster, and provides some experience in problem location. Finally, it summarizes the common API call methods of ES cluster.

This article was shared from the Huawei Cloud Community “Deploying ES Cluster and Operation in Kubernetes” by Minucas.

ES cluster architecture:

ES cluster is divided into single point mode and cluster mode. The single point mode is generally not recommended in production environment, but the cluster mode is recommended for deployment. The cluster mode is further divided into the deployment mode in which the Master node and the Data node are undertaken by the same node, and the Master node and the Data node are undertaken by different nodes. Deployment of the Master node and the Data node separately is more reliable. The following diagram shows the deployment architecture of the ES cluster:

Adopt K8S for ES cluster deployment:

1. Deploy K8S StatefulSet to expand ES nodes quickly. This example uses 3 Master nodes + 12 Data nodes to deploy

kubectl -s http://ip:port create -f es-master.yaml
kubectl -s http://ip:port create -f es-data.yaml
kubectl -s http://ip:port create -f es-service.yaml

es-master.yaml:

apiVersion: apps/v1 kind: StatefulSet metadata: labels: addonmanager.kubernetes.io/mode: Reconcile k8s-app: Es kubernetes. IO /cluster-service: "true" version: v6.2.5 Name: es-master namespace: default spec: PodManagementPolicy: OrderedReady Replicas: 3 RevisionHistoryLimit: 10 Selector: MatchLabels: K8S-app: ES version: V6.2.5 ServiceName: Es template: metadata: Labels: k8s-app: camp-es kubernetes. IO /cluster-service: "true" version: V6.2.5 spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: ELASTICSEARCH_SERVICE_NAME value: es - name: NODE_MASTER value: "true" - name: NODE_DATA value: "false" - name: ES_HEAP_SIZE value: 4g - name: ES_JAVA_OPTS value: -Xmx4g -Xms4g - name: cluster.name value: es image: Elasticsearch :v6.2.5 ImagePullPolicy: Always name: es Ports: -ContainerPort: 9200 name: DB Protocol: TCP - containerPort: 9300 hostPort: 9300 name: transport protocol: TCP resources: limits: cpu: "6" memory: 12Gi requests: cpu: "4" memory: 8Gi securityContext: capabilities: add: - IPC_LOCK - SYS_RESOURCE volumeMounts: - mountPath: /data name: es - command: - /bin/elasticsearch_exporter - -es.uri=http://localhost:9200 - -es.all=true image: ElasticSearch_Exporter :1.0.2 ImagePullPolicy: ifNotPresent LivenessProbe: FailureThreshold: 3 HttpGet: Path: /health port: 9108 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 name: es-exporter ports: - containerPort: 9108 hostPort: 9108 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /health port: 9108 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 resources: limits: cpu: 100m memory: 128Mi requests: cpu: 25m memory: 64Mi securityContext: capabilities: drop: - SETPCAP - MKNOD - AUDIT_WRITE - CHOWN - NET_RAW - DAC_OVERRIDE - FOWNER - FSETID - KILL - SETGID - SETUID - NET_BIND_SERVICE - SYS_CHROOT - SETFCAP readOnlyRootFilesystem: true dnsPolicy: ClusterFirst initContainers: - command: - /sbin/sysctl -- w-vm.max_map_count = 262284 Image: alpine: 3.0imagePullPolicy: ifTextName: elasticsearch-logging-init resources: {} securityContext: privileged: true restartPolicy: Always schedulerName: default-scheduler securityContext: {} volumes: - hostPath: path: /Data/es type: DirectoryOrCreate name: es

es-data.yaml

apiVersion: apps/v1 kind: StatefulSet metadata: labels: addonmanager.kubernetes.io/mode: Reconcile k8s-app: Es kubernetes. IO /cluster-service: "true" version: v6.2.5 name: es-data namespace: default spec: PodManagementPolicy: OrderedReady Replicas: 12 RevisionHistoryLimit: 10 Selector: MatchLabels: K8S-app: ES version: V6.2.5 ServiceName: Es template: metadata: Labels: k8s-app: es kubernetes. IO /cluster-service: "true" version: V6.2.5 Spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: ELASTICSEARCH_SERVICE_NAME value: es - name: NODE_MASTER value: "false" - name: NODE_DATA value: "true" - name: ES_HEAP_SIZE value: 16g - name: ES_JAVA_OPTS value: -Xmx16g -Xms16g - name: cluster.name value: es image: Elasticsearch :v6.2.5 ImagePullPolicy: Always name: es Ports: -ContainerPort: 9200 name: DB Protocol: TCP - containerPort: 9300 hostPort: 9300 name: transport protocol: TCP resources: limits: cpu: "8" memory: 32Gi requests: cpu: "7" memory: 30Gi securityContext: capabilities: add: - IPC_LOCK - SYS_RESOURCE volumeMounts: - mountPath: /data name: es - command: - /bin/elasticsearch_exporter - -es.uri=http://localhost:9200 - -es.all=true image: ElasticSearch_Exporter :1.0.2 ImagePullPolicy: ifNotPresent LivenessProbe: FailureThreshold: 3 HttpGet: Path: /health port: 9108 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 name: es-exporter ports: - containerPort: 9108 hostPort: 9108 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /health port: 9108 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 resources: limits: cpu: 100m memory: 128Mi requests: cpu: 25m memory: 64Mi securityContext: capabilities: drop: - SETPCAP - MKNOD - AUDIT_WRITE - CHOWN - NET_RAW - DAC_OVERRIDE - FOWNER - FSETID - KILL - SETGID - SETUID - NET_BIND_SERVICE - SYS_CHROOT - SETFCAP readOnlyRootFilesystem: true dnsPolicy: ClusterFirst initContainers: - command: - /sbin/sysctl -- w-vm.max_map_count = 262284 Image: alpine: 3.0imagePullPolicy: ifTextName: elasticsearch-logging-init resources: {} securityContext: privileged: true restartPolicy: Always schedulerName: default-scheduler securityContext: {} volumes: - hostPath: path: /Data/es type: DirectoryOrCreate name: es

es-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: es
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Elasticsearch
  name: es
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: es
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: exporter
    port: 9108
    protocol: TCP
    targetPort: 9108
  selector:
    k8s-app: es
  sessionAffinity: None
  type: ClusterIP

ES cluster monitoring

Exporter, Esphead and KOPF are the three commonly used monitoring means for ES cluster monitoring. As ES cluster is deployed in K8S architecture, many features will be carried out in combination with K8S

Grafana monitoring

Exporting monitoring metrics through K8S deployment Es-Exporter, collecting monitoring data from Prometheus, and Grafana’s custom dashboard presentation

ES – head components

Making address:https://github.com/mobz/elast…

The ES-Head component can be installed by searching the Google browser app store, and the ES cluster can be viewed using the Chrome plugin

Cerebro (KOPF) components

Making address:https://github.com/lmenezes/c…

ES cluster problem handling

ES configuration

Resource allocation: Focus on the CPU, Memory, and Heap Size of ES, as well as the configuration of XMS XMX. If the machine has 8u32GB of Memory, it is recommended to set the Heap Memory and XMS XMX at 50%. The website recommends that a single node should have no more than 64GB of Memory

Index configuration: since ES searches through indexes to locate, ES will load the relevant index data into memory to speed up the retrieval. Therefore, reasonable index setting has a great impact on the performance of ES. At present, we create indexes by date (independent indexes with small individual data volume).

ES load

In this case, we can manually tell the node that the Load and CPU are unevenly distributed. In this case, we can tell the node that the shard is unevenly distributed

Shard configuration

The shard configuration should be an integer multiple of the number of data nodes, and the number of shards should not be as large as the number of shards. The shard configuration should be reasonably divided according to the data volume of the index to ensure that each shard does not exceed the heap memory size allocated by a single data node. For example, the index with the largest data volume is about 150G a day. It is divided into 24 shards, and the size of a single shard is about 6-7g

The number of replicas is suggested to be 1. If the number of replicas is too large, it is likely to cause frequent relocate of the data and increase the load of the cluster

Delete index from index

The curl -x DELETE "10.64. XXX. Xx: 9200 / SZV - prod - ingress - nginx - 2021.05.01"

Index names can be deleted in bulk using regular matching, such as -2021.05.*

Another reason for high node load

When locating the problem, it was found that the data shard of the node had been removed, but the load of the node could not be lowered. The top command of the login node was used to find that the CPU of the node, Kubelet, was very high, and the restart of Kubelet was also invalid. The load was relieved after the restart of the node

Experience summary of routine operation and maintenance of ES cluster (refer to the official website)

View cluster health status

The health status of ES cluster is divided into three types: Green, Yellow, and Red.

• Green: cluster health;
• Yellow(Yellow) : The cluster is not healthy, but it can automatically rebalance within the load allowable range;
• Red(Red) : The cluster has a problem, some data is not ready, and at least one of the primary shards has not been allocated successfully.

Cluster health status and unallocated shards can be queried via API:

GET _cluster/health
{
  "cluster_name": "camp-es",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 15,
  "number_of_data_nodes": 12,
  "active_primary_shards": 2176,
  "active_shards": 4347,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

View Pending Tasks:

GET /_cat/pending_tasks The priority field indicates the priority of the task

See why shards are not allocated

GET _cluster/allocation, explain the reason which causes field represents the shard of undistributed, detail said unallocated reason in detail

View all unallocated indexes and primary shards:

GET /_cat/indices? v&health=red

See which shards are out of order

curl -s http://ip:port/_cat/shards | grep UNASSIGNED

Reassign a primary shard:

POST _cluster/reroute? pretty" -d '{ "commands" : [ { "allocate_stale_primary" : { "index" : "xxx", "shard" : 1, "node" : "12345...", "accept_data_loss": true } } ] }

Curl ‘IP :port/_node/process?’ curl ‘IP :port/_node/process? Pretty ‘to query

Reduce the number of copies of the index

PUT /szv_ingress_*/settings
{
  "index": {
    "number_of_replicas": 1
  }
}

Click on the attention, the first time to understand Huawei cloud fresh technology ~