advertising

Want to work together to create a new serverless r&d system? Welcome to join alibaba CBU wireless server team. Now we are looking for senior R&D engineer/technical expert. Please send your resume to [email protected]

Today let’s try deploying a Redis cluster in K8S to learn more about the details and features of K8S.

Environment: Minikube V0.30 (Kubernetes 1.10)

Note: Background knowledge and details related to Redis-cluster are not described here, but can be reviewed by referring to previous articles

Problem analysis

In essence, deploying a Redis cluster on K8S is not much different from deploying a normal application, but there are a few things to be aware of:

  1. REDIS is a stateful application

    This is the problem we need to pay attention to when deploying redis cluster. When we deploy Redis in the form of POD in K8S, the data cached in each pod is different, and the IP of pod can change at any time. At this time, many problems will occur if common deployment and Service are used to deploy redis-cluster, so StatefulSet + Headless service is needed to solve the problem

  2. Data persistence

    Although Redis is based on memory, it still needs to rely on disk for persistent data, so that the cached data can be recovered when the service is restarted. In a cluster, we need to use shared file system + PV (persistent volume) to make all pods in the cluster share the same persistent store

The concept is introduced

Before we get started, let’s explain a few concepts and principles in detail

Headless Service

In k8S DNS mapping, the resolution result of a Headless Service is not a Cluster IP address, but a list of all the Pod IP addresses associated with it

StatefulSet

  • Reference to introduce

StatefulSet is a resource in K8S specifically designed for stateful application Deployment. In general, it can be considered a variant of Deployment/RC with the following features:

  1. Each Pod managed by StatefulSet has a unique document/network identity and is generated numerically, rather than having a random name and IP as in Deployment (e.g., The StatefulSet name is redis, So the pod name is redis-0, redis-1…

  2. StatefulSet ReplicaSet’s start and stop sequence is strictly controlled. Operation of the NTH POD must wait for the first N-1 to complete

  3. Pods in StatefulSet are stored in stable persistence and the corresponding PVS are not destroyed when pods are deleted

In addition, StatefulSet must work with Headless Service. It adds another layer to the DNS mapping provided by Headless Service, resulting in a per-POD domain name mapping in the following format:

$(podname).$(headless service name)
Copy the code

With this mapping, you can configure clusters using domain names instead of IP addresses to manage stateful application clusters

plan

With StatefulSet and Headless Services, the cluster deployment plan is designed as follows (figure from the reference article) :

The configuration steps are as follows:

  1. Configure the NFS share file system
  2. Create PV and PVC
  3. Create ConfigMap
  4. Create a Headless Service
  5. Create StatefulSet
  6. Initialize the Redis cluster

The actual operation

Since the single-node environment of Minikube is used, in order to simplify the complexity, PV and PVC are not configured this time, and the data is directly mounted in the way of ordinary Volume

Create ConfigMap

Start by creating the redis.conf configuration file

appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
Copy the code

Kubectl create configmap redis-conf –from-file=redis.conf to create configMap

Create HeadlessService

apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster
Copy the code

Create StatefulSet

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-app
spec:
  serviceName: "redis-service"
  replicas: 6
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: "registry.cn-qingdao.aliyuncs.com/gold-faas/gold-redis:1.0"
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
            - name: redis
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"
            mountPath: "/var/lib/redis"
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
          items:
            - key: "redis.conf"
              path: "redis.conf"
      - name: "redis-data"
        emptyDir: {} 
Copy the code

Initialize the Redis cluster

After StatefulSet is created, you can see that the six pods have started, but the entire Redis cluster has not been initialized yet and you need to use the official provided Redis-Trib tool.

We can certainly run the corresponding tools on any Redis node to initialize the entire cluster, but this is not appropriate. We want to keep the responsibility of each node as single as possible, so it is best to have a separate POD to run the management tools for the entire cluster.

Redis-trib is an official redDIS-cluster management tool, which can achieve the creation and updating of redIS clusters. In the early version of Redis, It works as a Ruby script in the source package redis-trib.rb (you can also pull python on PIP, but I failed to run it) and is now officially integrated into redis-CLI (I used 5.0.3).

To initialize the cluster, start by creating an Ubuntu pod on K8S that will serve as the management node:

kubectl run -i --tty redis-cluster-manager --image=ubuntu --restart=Never /bin/bash
Copy the code

Install some tools, including wget,dnsutils, and then download and install Redis:

Wget http://download.redis.io/releases/redis-5.0.3.tar.gz tar - XVZF redis - 5.0.3. Tar. GzcdRedis - 5.0.3. Tar. Gz && makeCopy the code

After compiling, redis-cli will be placed in the SRC directory. Put it in /usr/local/bin to facilitate subsequent operations

To obtain the host IP addresses of the six nodes, you can use nsLookup in connection with StatefulSet’s domain name rules. For example, to find the IP address of the pod redis-app-0, run the following command:

root@redis-cluster-manager:/# nslookup redis-app-0.redis-service
Server:		10.96.0.10
Address:	10.96.0.10# 53Name: redis - app - 0. Redis - service. Gold. SVC. Cluster. The local Address: 172.17.0.10Copy the code

172.17.0.10 is the corresponding IP address. For this deployment we use 0, 1, and 2 as Master nodes; 3, 4, and 5 as Slave nodes, run the following command to initialize the Master node of the cluster:

Redis -cli --cluster create 172.17.0.10:6379 172.17.0.11:6379 172.17.0.12:6379Copy the code

Then attach corresponding Slave nodes to each of them. The cluster-master-ID is given in the previous step:

Redis-cli --cluster add-node 172.17.0.13:6379 172.17.0.10:6379 --cluster-slave --cluster-master-id adf443a4d33c4db2c0d4669d61915ae6faa96b46Copy the code
Redis-cli --cluster add-node 172.17.0.14:6379 172.17.0.11:6379 --cluster-slave --cluster-master-id 6e5adcb56a871a3d78343a38fcdec67be7ae98f8Copy the code
Redis-cli --cluster add-node 172.17.0.16:6379 172.17.0.12:6379 --cluster-slave --cluster-master-id c061e37c5052c22f056fff2a014a9f63c3f47ca0Copy the code

After cluster initialization, enter a node at random to check the cluster information:

Redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli:

Create a Service

Now any node in the Redis cluster can be accessed directly, but in order to provide access to the other services in the cluster, we need to create a service for service discovery and load balancing (note that the service is not the same as the headless service we created earlier).

Yaml files are as follows:

apiVersion: v1
kind: Service
metadata:
  name: gold-redis
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
  selector:
    app: redis
    appCluster: redis-cluster
Copy the code

Do a test after deployment:

Nice. All the work is done here

Reference article:

  • Deploy the Redis cluster on K8S
  • kubernetes-redis-cluster