Kubernetes distributed application deployment and face recognition app instance

KUBERNETES DISTRIBUTED APPLICATION DEPLOYMENT WITH SAMPLE FACE RECOGNITION APP

Original author: Skarlso

The Nuggets translation Project

Permanent link to this article: github.com/xitu/gold-m…

Translator: maoqyhz

Proofreader: CF020031308, HCMY

All right, man, let’s just calm down. It’s going to be a long but hopeful and interesting journey.

I will deploy distributed applications using Kubernetes. I tried to create an app that looked like a real world app. Obviously, due to limited time and energy, I had to ignore some details.

My focus will be on Kubernetes and application deployment.

Are you ready to get down to business?

About the application

Abstract

The application consists of six parts. The repository can be found here: Kube Cluster Sample.

It’s a facial recognition service that recognizes images of people and compares them to known people. The recognition results are presented in a simple front end, in the form of a table, and you can see who the people in the image to be recognized are. The application works as follows: First send a request to the receiver that contains the path of the image. These images can be stored somewhere like NFS, while the receiver stores the image path in DB (MySQL). Finally, a processing request is sent to the queue containing the ID to save the image. NSQ is used as the queue.

The image processing service continuously monitors the queue on which the job is to be executed. The processing process consists of the following steps: Obtain the ID; Load image; Finally, gRPC sends the image to a face recognition backend program written in Python. If the recognition is successful, the back end returns the name corresponding to the person in the image. The image processor then updates the person ID field of the image record and marks the image as “Processed successfully.” If the identification fails, the image is retained as “pending”. If a failure occurs during identification, the image will be marked as “failed”.

Images that fail to process can be retried with cron jobs, for example:

So how does this work? Let’s take a look.

The receiver

The receiver service is the starting point for the whole process. The API receives requests in the following format:

curl -d '{"path":"/unknown_images/unknown0001.jpg"}' http://127.0.0.1:8000/image/post
Copy the code

In this example, the sink stores the image path through a shared database cluster. When the database stores the image path successfully, the sink instance can receive the image ID from the database service. This application is based on a model that provides a unique identity for entity objects at the persistence layer. Once the ID is generated, the receiver sends a message to the NSQ. At this point, the work of the receiver is done.

Image processor

Here’s where the excitement starts. When the image processor first runs, it creates two Go coroutines (Routine). They are:

Consume

This is an NSQ consumer. It has three necessary jobs. First, it can listen for messages in queues. Second, when it receives a message, it adds the received ID to the thread-safe ID slice processed by the second routine. Finally, it tells the second coroutine that it has work to do via sync.condition.

ProcessImages

This routine processes the ID slices until the slices are completely exhausted. Once slices are consumed, the routine pauses instead of waiting for a channel. Here are the steps to handle a single ID:

Establish gRPC connection with face recognition service (explained in face Recognition section below)
Retrieves image records from the database
Set up theThe circuit breakerTwo functions of
- Function 1: The main function that runs the RPC method call
- Function 2: Perform a health check on the Ping of the circuit breaker
Call function 1 to send the image path to the face recognition service. The service needs to be able to access this path. It is better to share files like NFS
If the call fails, update the status field of the image record to “FAILED PROCESSING”
If successful, the name of the person in the database associated with the image is returned. It performs an SQL join query to retrieve the associated person ID
Update the status field of the image records in the database to “PROCESSED” and the person field to the ID of the recognized person

This service can be replicated, in other words, multiple services can run simultaneously.

The circuit breaker

Although this is a system that can replicate resources with minimal effort, conditions can still exist, such as network failures and communication problems between services. So I implemented a little circuit breaker on the gRRC call for fun.

Here’s how it works:

As you can see, once there are five unsuccessful calls in the service, the circuit breaker will be activated and no calls will be allowed through. After a configured period of time, a Ping call is sent to the service and checked to see if the service returns a message. If an error persists, the timeout is increased, otherwise it is turned on to allow traffic through.

The front end

This is just a simple table view, using the HTML template that comes with Go to render the list of images.

Face recognition

Here’s where to identify the magic. In pursuit of flexibility, I decided to encapsulate face recognition as a GRPC-based service. I originally intended to write it in Go, but found it much clearer to use Python. In fact, the face recognition section takes about seven lines of Python code, in addition to the gPRC code. I’m using an excellent library that contains all of the OpenCV calls implemented in C. Face recognition. Signing the API agreement here means that I can change the implementation of the face recognition code at any time under the license of the agreement.

Note that there is a library for developing OpenCV in the Go language. I almost used it, but it doesn’t contain calls to OpenCV implemented by C. This library is called GoCV and you can check it out. There are some really cool things about them, like real-time camera feedback processing that only takes a few lines of code.

Python’s library is simple in nature. Now we have a set of known people images and put them in folders named Hannibal_1.jpg, hannibal_2.jpg, gergely_1.jpg, john_doe.jpg. There are two tables in the database, person and person_images. They look like this:

+----+----------+
| id | name     |
+----+----------+
|  1 | Gergely  |
|  2 | John Doe |
|  3 | Hannibal |
+----+----------+
+----+----------------+-----------+
| id | image_name     | person_id |
+----+----------------+-----------+
|  1 | hannibal_1.jpg |         3 |
|  2 | hannibal_2.jpg |         3 |
+----+----------------+-----------+
Copy the code

The face recognition library returns the name of an image from a known person, which matches the person in the unknown image. After that, a simple join query, like this, returns the identified person information.

select person.name, person.id from person inner join person_images as pi on person.id = pi.person_id where image_name = 'hannibal_2.jpg';
Copy the code

The gRPC call returns the ID of the person and is used to modify the value of the Person column in the image record to be recognized.

NSQ

NSQ is an excellent GO-based queue. It is scalable and has a minimal footprint on the system. It also has a lookup service that consumers use to receive messages, and a daemon that senders use to send messages.

The idea of NSQ is that the daemon should run with the sender application. This way, the sender will only send to the local host. But daemons connect to the lookup service, and that’s how they implement global queues.

This means how many senders there are, and how many NSQ daemons need to be deployed. Since the resource requirements of the daemon are small, they do not affect the requirements of the main application.

configuration

To be as flexible as possible, and to use Kubernetes’ ConfigSet, I used.env files in development to store configurations such as the location of the database service or the lookup address of NSQ. In production, this means that in the Kubernetes environment, I will use environment variables.

Summary of face recognition applications

This is the architecture of the application we will deploy. All of its components are mutable and can only be coupled through databases, queues, and gRPC. This is important when deploying distributed applications because the update mechanism works. I’ll cover this section in the “Deployment” section.

Deploy the application in Kubernetes

basis

What is Kubernetes?

I’ll cover the basics here, but I won’t go into too much detail. If you want to learn more, read the whole book: Kubernetes Up And Running. Alternatively, if you’re feeling bold, you can take a look at Kubernetes Documentation.

Kubernetes is a containerized service and application management platform. It is easily extensible, manages a large number of containers, and most importantly, is highly configurable with YAML based template files. People often compare Kubernetes to Docker clusters, but Kubernetes really goes beyond that! For example, it can manage different containers. You can use Kubernetes to manage and orchestrate LXC, and you can manage Docker in the same way. It provides a layer above managing clusters of deployed services and applications. How’s that？ Let’s take a quick look at the building blocks of Kubernetes.

In Kubernetes, you describe the expected state of your application, and Kubernetes does something to bring it to that state. States can be deployed, paused, repeated twice, and so on.

One of the basics of Kubernetes is that it uses labels and annotations for all components. Services, Deployments, ReplicaSets, DaemonSets, everything can be tagged. Consider the following. To determine which pod belongs to which application, we will use a tag called App: myApp. Assume that you have deployed two containers for this application; If you remove the label app from one of the containers, Kubernetes will only detect one label and therefore start a new instance of MyApp.

Kubernetes Cluster

For Kuberenetes to work, there is a need for clusters of Kubernetes. Configuring a cluster can be painful, but fortunately, help is at hand. Minikube configures a cluster with one node for us locally. AWS has a test service that runs as a Kubernetes cluster, where all you need to do is request nodes and define your deployment. Kubernetes Cluster Components are available here.

Nodes

A node is a working host. It can be anything, such as physical machines, virtual machines, and virtual resources provided by various cloud services.

Pods

Pods are logically grouped containers, meaning that a Pod can hold multiple containers. Once created, a Pod gets its own DNS and virtual IP address, so Kubernetes can balance traffic for it. You rarely need to deal with containers directly, and even when debugging (such as viewing logs) it is common to call kubectl logs Deployment/your-app -f rather than looking at a specific container. Although -c container_name might be called. The -f parameter continues to display the end of the log file.

Deployments

When any type of resource is created in Kubernetes, it uses Deployment in the background. A Deployment object describes the expected state of the current application. It can be used to change the status of a Pod or Service, update or launch a new version of the application. You do not directly control ReplicaSet (as described later), but you can control Deployment objects to create and manage ReplicaSet.

Services

By default, the Pod gets an IP address. However, because Pods are an unstable thing in Kubernetes, you’ll need something more durable. Queues, mysql, internal apis, front ends, these take a long time to run and need to be behind a static, unchanging IP or preferably DNS record.

For this purpose, Kubernetes provides Services that define accessible patterns. Load balancing, simple IP or internal DNS.

How does Kubernetes know if the service is running correctly? You can configure health checks and availability checks. The health check will check if the container is running, but it does not mean that your service is running. To do this, you need to perform a usability check on the available endpoints in your application.

Since Services are so important, I suggest you read about them here later: Services. Be forewarned, this section of documentation is quite extensive, with 24 A4 pages covering networking, services and discovery. But it is critical to whether you decide to use Kubernetes in your production environment.

DNS / Service Discovery

If you create a service in a cluster, the service gets DNS records in Kubernetes provided by special Kubernetes Deployments objects (called kube-proxy and kube-DNS). These two objects provide service discovery in the cluster. If you are running the mysql service and set up the clusterIP: none, so everyone of cluster can be solved through ping mysql. The default. SVC. Cluster. The local to access the service. Among them:

mysql– Service name
default– Namespace name
svc– The service itself
cluster.local– Domain name of the local cluster

The domain name can be changed by customization. To access services outside the cluster, you must have a DNS provider and use Nginx (for example) to bind IP addresses to records. You can run the following command to query the public IP address of the service:

NodePort –kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services mysql
LoadBalancer –kubectl get -o jsonpath="{.spec.ports[0].LoadBalancer}" services mysql

Template Files

Like Docker Compose, TerraForm, or other service management tools, Kubernetes provides an infrastructure for configuring templates. This means you rarely have to do anything by hand.

For example, look at the following template to configure nginx deployment using yamL files:

apiVersion: apps/v1
kind: Deployment # (1)
metadata: # (2)
    name: nginx-deployment
    labels: # (3)
    app: nginx
spec: # (4)
    replicas: 3 # (5)
    selector:
    matchLabels:
        app: nginx
    template:
    metadata:
        labels:
        app: nginx
    spec:
        containers: # (6)- name: nginx image: nginx:1.7.9 ports: -containerport: 80Copy the code

In this simple deployment, we did the following:

(1) is usedkindProperty defines the type of the template
(2) Add metadata that identifies this deployment and create each resource using label (3)
(4) Then describe the required state specifications.
(5) For nginx applications, there are threereplicas
(6) This is the template definition for the container. The Pod configured here contains a container with the name nginx. Here, using the 1.7.9 version of the Nginx image (Docker in this case), the exposed port number is: 80

ReplicaSet

ReplicaSet is a low-level replication manager. It ensures that the correct number of replicates are run for the application. However, when deployed at a higher level, ReplicaSets should always be managed. You rarely need to use ReplicaSets directly, unless you have a specific case where you need to control the details of replication.

DaemonSet

Remember I said how Kubernetes keeps using tags? DaemonSet is a controller that ensures that daemon applications always run on nodes with specific labels.

For example, you want all nodes labeled logger or mission_critical to run the logger/audit service daemon. Then you create a DaemonSet and give it a node selector named logger or mission_critical. Kubernetes will look for nodes with that label. Always ensure that it will have an instance of the daemon running on it. Therefore, every instance running on the node can access the daemon locally.

In my application, the NSQ daemon might be a DaemonSet. To ensure that it runs on a node with a receiver component, I mark a node with Receiver and specify a DaemonSet with the Receiver application selector.

DaemonSet has all the advantages of ReplicaSet. It is extensible and managed by Kubernetes. This means that all life cycle events are handled by Kube, ensuring that they never die and that once they occur, they are immediately replaced.

Scaling

Making extensions in Kubernetes is easy. ReplicaSets manages the number of Pod instances and, as seen in nginx deployments, uses the “Replicas: 3” setting. We should write our application in a way that allows Kubernetes to run multiple copies of it.

Of course these Settings are huge. You can specify which replicates must run on which nodes or wait for instances to appear at various wait times. You can read more about this topic here: Horizontal Scaling and here: [Interactive Scaling with Kubernetes](HTTPS: / / kubernetes. IO/docs/tutorials/kubernetes – basics/scale -interactive /), And of course there’s a ReplicaSet control and all of the scaling can be done in Kubernetes.

Kubernetes summary

This is a handy tool for handling container choreography. Its basic units are Pods with a layered architecture. The top layer is Deployments, through which all other resources are handled. It is highly configurable and provides an API for all calls, so instead of running Kubectl, you can write your own logic to send information to the Kubernetes API.

Kubernetes now supports all major cloud providers, it is completely open source, feel free to contribute! If you want to learn more about how it works, check out the code: Kubernetes on Github.

Minikube

I’m going to use Minikube. Minikube is a local Kubernetes cluster emulator. While it’s not great to emulate multiple nodes, it costs nothing and is great if you just start learning and playing around locally. Minikube is virtual machine based and can be fine-tuned using VirtualBox etc if needed.

All the kube template files I will be using can be found here: Kube Files.

** Note: ** If you want to use Scaling later but notice that the replication is always “Pending”, remember that minikube only uses a single node. It may not allow more than one copy on the same node, or it may simply run out of resources. You can use the following command to check the available resources:

kubectl get nodes -o yaml
Copy the code

Create a container

Kubernetes supports most containers. I’m going to use Docker. For all the services I build, there is a Dockerfile in the repository. I encourage you to study them. Most of them are very simple. For the Go service, I’m using the recently introduced multi-phase build. The Go service is based on Alpine Linux. The face recognition service is implemented in Python. NSQ and MySQL are using their own containers.

context

Kubernetes uses namespaces. If you do not specify any namespace, it will use the default namespace. I will permanently set a context to avoid contaminating the default namespace. Here’s what you can do:

❯ kubectl config set-context kube-face-cluster --namespace=face
Context "kube-face-cluster" created.
Copy the code

Once it is created, you must also start using the context, as follows:

❯ kubectl config use-context kube-face-cluster
Switched to context "kube-face-cluster".
Copy the code

After that, all kubectl commands will use the namespace face.

The deployment of application

Overview of Pods and Services:

MySQL

The first Service I will deploy is my database.

I am using the Kubernetes sample Kube MySQL located here, which meets my needs. Notice that the configuration file is using plaintext passwords. I will do some security measures according to Kubernetes Secrets described here.

As described in the documentation, I created a secret key file locally using confidential YAML.

apiVersion: v1
kind: Secret
metadata:
    name: kube-face-secret
type: Opaque
data:
    mysql_password: base64codehere
Copy the code

I created the Base64 code with the following command:

echo -n "ubersecurepassword" | base64
Copy the code

Here’s what you’ll see in my deployment YAML file:

. - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: kube-face-secret key: mysql_password ...Copy the code

It is also worth mentioning that it uses a volume to hold the database. Volume is defined as follows:

. volumeMounts: - name: mysql-persistent-storage mountPath: /var/lib/mysql ... volumes: - name: mysql-persistent-storage persistentVolumeClaim: claimName: mysql-pv-claim ...Copy the code

PresistentVolumeClain is the key here. This tells Kubernetes that the resource needs a persistent volume. How to provide it is abstracted from the user. You can be sure that Kubernetes will provide the volume. It’s similar to Pods. To read more, check out this document: Kubernetes Persistent Volumes.

Use the following command to complete the mysql service deployment:

kubectl apply -f mysql.yaml
Copy the code

Apply or create? In short, apply is considered a declarative object configuration command, while create is imperative. This means that now “create” is usually for one of the tasks, such as running something or creating Deployment. With Apply, the user does not define the action to take. This will be defined by Kubernetes in terms of the current state of the cluster. So, when there is no service named mysql, I call apply-f mysql.yaml, which creates the service. When it runs again, Kubernetes doesn’t do anything. However, if I run Create again, it will throw an error indicating that the service has been created.

For more information, see the following documentation: Kubernetes Object Management, [Imperative Configuration](HTTPS: // kubernetes.io/docs/concepts/runtime/runtime/runtime/index.html / The Declarative Configuration).

To view progress information, run:

# Describe the process
kubectl describe deployment mysql
# display pod only
kubectl get pods -l app=mysql
Copy the code

The output should look like this:

. Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: mysql-55cd6b9f47 (1/1 replicas created)
...
Copy the code

Or in the case of Get Pods:

NAME                     READY     STATUS    RESTARTS   AGE
mysql-78dbbd9c49-k6sdv   1/1       Running   0          18s
Copy the code

To test the example, run the following code snippet:

Kubectl run it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql-pyourpasswordhereCopy the code

** Something to know ** : If you change your password now, reapplying the YAML file to update the container is not enough. Since the database persists, the password will not change you must use kubectl delete -f mysql.yaml to delete the entire deployment.

You should see the following when you run Show Databases.

If you don't see a command prompt, try pressing enter. mysql> mysql> mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | kube | | mysql | | Performance_schema | + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + 4 rows in the set (0.00 SEC) mysql > exit ByeCopy the code

You’ll also notice that I’ve installed a file here: Database Setup SQL into the container. The MySQL container does this automatically. This file will initialize some data and the schema I’m going to use.

Volume is defined as follows:

    volumeMounts:
    - name: mysql-persistent-storage
    mountPath: /var/lib/mysql
    - name: bootstrap-script
    mountPath: /docker-entrypoint-initdb.d/database_setup.sql
volumes:
- name: mysql-persistent-storage
    persistentVolumeClaim:
    claimName: mysql-pv-claim
- name: bootstrap-script
    hostPath:
    path: /Users/hannibal/golang/src/github.com/Skarlso/kube-cluster-sample/database_setup.sql
    type: File
Copy the code

To check the success of the boot script, run the following command:

~ / golang/src/github.com/Skarlso/kube-cluster-sample/kube_files master * ❯ kubectl run - it - rm - image = mysql: 5.6 --restart=Never mysql-client -- mysql -h mysql -uroot -pyourpasswordhere kube If you don't see a command prompt, try pressing enter. mysql> show tables; +----------------+ | Tables_in_kube | +----------------+ | images | | person | | person_images | +----------------+ 3 Rows in set (0.00 SEC) mysql>Copy the code

This completes the database service setup. You can use the following command to view the logs of the service:

kubectl logs deployment/mysql -f
Copy the code

NSQ lookup

NSQ lookup will run as an internal service, which does not need to be accessed externally. So I set clusterIP: None, which tells Kubernetes that the service is headless. This means that it will not be load balanced and will not be a single IP service. DNS will be based on service selectors.

The NSQ Lookup selector we define is:

selector:
matchLabels:
    app: nsqlookup
Copy the code

Therefore, the internal DNS will be as follows: nsqlookup. Default. SVC. Cluster. The local.

The Headless Service is described in detail here: Headless Service.

It’s basically the same as MySQ L, with a few modifications. As mentioned earlier, I used NSQ’s own Docker image called NSqio/NSQ. All NSQ commands are there, so NSQD will also use this image, but the commands are different. For nsqlookupd, the command is:

command: ["/nsqlookupd"]
args: ["--broadcast-address=nsqlookup.default.svc.cluster.local"]
Copy the code

What is broadcast-address, you might ask? By default, nsQLookup will use hostname as the broadcast address. When the consumer runs the callback, it will try to connect to something like http://nsqlookup-234kf-asdf:4161/lookup? Switchable viewer = image url. Note that nsqlookup- 234kF-ASdf is the host name of the container. By putting a broadcast address is set to the internal DNS, the callback will be: http://nsqlookup.default.svc.cluster.local:4161/lookup?topic=images. This will work as expected.

NSQ lookups also require two ports for forwarding: one for broadcasting and one for NSQD callbacks. These are exposed in Dockerfile and then used in Kubernetes templates. Like this:

In the container template:

ports:
- containerPort: 4160
    hostPort: 4160
- containerPort: 4161
    hostPort: 4161
Copy the code

In the service template:

spec:
    ports:
    - name: tcp
    protocol: TCP
    port: 4160
    targetPort: 4160
    - name: http
    protocol: TCP
    port: 4161
    targetPort: 4161
Copy the code

Name is required by Kubernetes.

To create this service, I use the same command as before:

kubectl apply -f nsqlookup.yaml
Copy the code

At this point, I’m done with nsQlookupd.

The receiver

This is a more complicated question. The receiver does three things:

Create some deployments
Create the NSQ daemon
Providing services to the public

Deployments

The first Deployment object it creates is its own. The container for Receiver is Skarlso/kube-receiver-alpine.

Nsq daemon

Receiver Starts an NSQ daemon. As mentioned earlier, the receiver runs the NSQD itself. It does this by communicating locally rather than over the network. By having the receiver do this, they end up on the same node.

The NSQ daemon also needs some tweaks and parameters.

ports:
- containerPort: 4150
    hostPort: 4150
- containerPort: 4151
    hostPort: 4151
env:
- name: NSQLOOKUP_ADDRESS
    value: nsqlookup.default.svc.cluster.local
- name: NSQ_BROADCAST_ADDRESS
    value: nsqd.default.svc.cluster.local
command: ["/nsqd"]
args: ["--lookupd-tcp-address=$(NSQLOOKUP_ADDRESS): 4160"."--broadcast-address=$(NSQ_BROADCAST_ADDRESS)"]
Copy the code

You can see that the two parameters lookup-tcp-address and broadcast-address are set. The lookup TCP address is the DNS of the NSQlookupd service. The broadcast address is required, just like NSQlookupd, so the callback works fine.

Service for the masses

Now, this is the first time I’ve deployed a public service. There are two options here. I can use LoadBalancer because the API can take a lot of load. If this is going to be deployed in production, it should use this one.

I do a single node deployment locally, so “NodePort” is sufficient. A NodePort exposes services on each node IP on a static port. If not specified, it will assign a random port to the host between 30000 and 32767. But it can also be configured as a specific port, using nodePort in the template file. To use this service, use

:

. If multiple nodes are configured, LoadBalancer can reuse them to a single IP.

For more information, see this document: Publishing Service.

Together, we get a receive service with the following template:

apiVersion: v1
kind: Service
metadata:
    name: receiver-service
spec:
    ports:
    - protocol: TCP
    port: 8000
    targetPort: 8000
    selector:
    app: receiver
    type: NodePort
Copy the code

For fixed node ports on 8000, a nodePort definition must be provided:

apiVersion: v1
kind: Service
metadata:
    name: receiver-service
spec:
    ports:
    - protocol: TCP
    port: 8000
    targetPort: 8000
    selector:
    app: receiver
    type: NodePort
    nodePort: 8000
Copy the code

Image processor

The image processor is where I process passing images for recognition. It should have access to NSQlookupd, mysql and gRPC endpoints for face recognition services. It’s actually pretty boring service. In fact, it’s not even a service. It does not expose anything, so it is the first component deployed. For brevity, here is the entire template:

---
apiVersion: apps/v1
kind: Deployment
metadata:
    name: image-processor-deployment
spec:
    selector:
    matchLabels:
        app: image-processor
    replicas: 1
    template:
    metadata:
        labels:
        app: image-processor
    spec:
        containers:
        - name: image-processor
        image: skarlso/kube-processor-alpine:latest
        env:
        - name: MYSQL_CONNECTION
            value: "mysql.default.svc.cluster.local"
        - name: MYSQL_USERPASSWORD
            valueFrom:
            secretKeyRef:
                name: kube-face-secret
                key: mysql_userpassword
        - name: MYSQL_PORT
            # TIL: Kubectl will get an error if 3306 is not quoted here
            value: "3306"
        - name: MYSQL_DBNAME
            value: kube
        - name: NSQ_LOOKUP_ADDRESS
            value: "nsqlookup.default.svc.cluster.local:4161"
        - name: GRPC_ADDRESS
            value: "face-recog.default.svc.cluster.local:50051"
Copy the code

The only interesting things in this file are the large number of environment attributes used to configure the application. Note the NSqlookupd address and the GRPC address.

To create this deployment, run:

kubectl apply -f image_processor.yaml
Copy the code

Face recognition

Face recognition service is a simple service that only the image processor needs. Its template is as follows:

apiVersion: v1
kind: Service
metadata:
    name: face-recog
spec:
    ports:
    - protocol: TCP
    port: 50051
    targetPort: 50051
    selector:
    app: face-recog
    clusterIP: None
Copy the code

The more interesting part is that it requires two volumes. These two volumes are known_people and unknown_people. Can you guess what they will contain? Yes, images. The “known_people” volume contains all images associated with known people in the database. Unknown_people volume will contain all new images. This is the path that we need to use to send the image from the receiver; That’s where the mount point is, in my case/unknown_people. Basically, the path must be accessible to the face recognition service.

Volume is now easy to deploy through Kubernetes and Docker. It can be mounted S3 or some type of NFS, or it can be mounted locally from host to client. There are other possibilities. For simplicity, I’ll use a local installation.

Installing a volume is done in two parts. First, Dockerfile must specify volume:

VOLUME [ "/unknown_people"."/known_people" ]
Copy the code

Second, the Kubernetes template needs to add volumeMounts to the MySQL service. The difference is that hostPath is not the volume claimed:

volumeMounts:
- name: known-people-storage
    mountPath: /known_people
- name: unknown-people-storage
    mountPath: /unknown_people
volumes:
- name: known-people-storage
hostPath:
    path: /Users/hannibal/Temp/known_people
    type: Directory
- name: unknown-people-storage
hostPath:
    path: /Users/hannibal/Temp/
    type: Directory
Copy the code

We also need to set up the known_people folder configuration for the face recognition service. This is done with environment variables:

env:
- name: KNOWN_PEOPLE
    value: "/known_people"
Copy the code

The Python code will then look for the image, as follows:

known_people = os.getenv('KNOWN_PEOPLE'.'known_people')
print("Known people images location is: %s" % known_people)
images = self.image_files_in_folder(known_people)
Copy the code

Image_files_in_folder functions are as follows:

def image_files_in_folder(self, folder):
    return [os.path.join(folder, f) for f in os.listdir(folder) if re.match(r'.*\.(jpg|jpeg|png)', f, flags=re.I)]
Copy the code

Neat.

Now, if the receiver receives a request (and sends it further down the line), it is similar to the request below.

curl -d '{"path":"/unknown_people/unknown220.jpg"}' http://192.168.99.100:30251/image/post
Copy the code

It looks for an image named Unknown220.jpg under/Unknown_people, finds an image in the Unknown_Folder that corresponds to the person in the unknown image, and returns the name of the matching image.

Check the log and you’ll see something like this:

# Receiver
❯ curl -d '{"path":"/unknown_people/unknown219.jpg"}'http://192.168.99.100:30251/image/post got path: {path: / unknown_people unknown219. JPG} image saved with id: 4 image sent to nsq# Image Processor
2018/03/26 18:11:21 INF    1 [images/ch] querying nsqlookupd http://nsqlookup.default.svc.cluster.local:4161/lookup?topic=images
2018/03/26 18:11:59 Got a message: 4
2018/03/26 18:11:59 Processing image id:  4
2018/03/26 18:12:00 got person:  Hannibal
2018/03/26 18:12:00 updating record with person id
2018/03/26 18:12:00 done
Copy the code

At this point, all services are deployed.

The front end

Finally, there is a small Web application that makes it easy to present information in a database. This is also a public receiving service with the same parameters as the receiver.

It looks something like this:

conclusion

We are now in the phase of deploying a number of services. To review the commands I’ve used so far:

kubectl apply -f mysql.yaml
kubectl apply -f nsqlookup.yaml
kubectl apply -f receiver.yaml
kubectl apply -f image_processor.yaml
kubectl apply -f face_recognition.yaml
kubectl apply -f frontend.yaml
Copy the code

Since the application does not allocate connections at startup, they can be sorted in any order. (Except for NSQ consumers of Image_Processor.)

If there is no error, kubectl get Pods query for kube running pod should show the following:

❯ kubectl get pods
NAME                                          READY     STATUS    RESTARTS   AGE
face-recog-6bf449c6f-qg5tr                    1/1       Running   0          1m
image-processor-deployment-6467468c9d-cvx6m   1/1       Running   0          31s
mysql-7d667c75f4-bwghw                        1/1       Running   0          36s
nsqd-584954c44c-299dz                         1/1       Running   0          26s
nsqlookup-7f5bdfcb87-jkdl7                    1/1       Running   0          11s
receiver-deployment-5cb4797598-sf5ds          1/1       Running   0          26s
Copy the code

Minikube service list running:

❯ minikube service list | -- -- -- -- -- -- -- -- -- -- -- -- - | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - | | NAMESPACE | NAME | | the URL |-------------|----------------------|-----------------------------| | default | face-recog | No node port | | default |  kubernetes | No node port | | default | mysql | No node port | | default | nsqd | No node port | | default | nsqlookup | No node port | | default | receiver - service | http://192.168.99.100:30251 | | kube - system | kube - DNS | No node port | | kube - system | kubernetes - dashboard | | http://192.168.99.100:30000 |-------------|----------------------|-----------------------------|Copy the code

Scroll to update

What happens during a rolling update?

As happens in software development, some parts of the system need/need to change. So what happens to our cluster if I change one of the components without affecting the others, while maintaining backward compatibility without disrupting the user experience? Fortunately, Kubernetes is there to help.

My problem is that the API can only handle one image at a time. Unfortunately, there is no batch upload option.

code

Currently, we have the following code snippet that processes a single image:

// PostImage image-processing article. Save it to the database // and send it to NSQ for further processing. func PostImage(w http.ResponseWriter, r *http.Request) { ... } funcmain() {
    router := mux.NewRouter()
    router.HandleFunc("/image/post", PostImage).Methods("POST")
    log.Fatal(http.ListenAndServe(": 8000", router))
}
Copy the code

We have two options: add a new endpoint with/images/POST and let the client use it, or modify the existing endpoint.

The advantage of the new client code is that if the new endpoint is not available, it can fall back to the old way of committing. However, the old client code doesn’t have this advantage, so we can’t change the way our code works now. Consider this: You have 90 servers, and you do a slow rolling update, pulling out one server at a time. If the update lasts a minute or so, the whole process takes about an hour and a half to complete (not including any parallel updates).

In the meantime, some of your servers will be running new code, and some of them will be running old code. The calls are load-balanced, so you have no control over which servers are hit. If the client tries to invoke in a new way but touches the old server, the client will fail. The client can try and fall back, but since you removed the old version, it will not succeed unless it coincidentally hits the server running the new code with the new code (assuming no sticky sessions are set up).

In addition, once all servers are updated, old clients will no longer be able to use your service.

Now, you can argue that you don’t want to keep old versions of your code forever. This is true in a sense. That’s why we modify the old code, and just add a little bit to call the new code. This way, once all clients have been migrated, the code can simply be removed without any problems.

A new endpoint

Let’s add a new path method:

. router.HandleFunc("/images/post", PostImages).Methods("POST")...Copy the code

Update the old version to call the new version with the modified version, as follows:

// PostImage image-processing article. Save it to the database // and send it to NSQ for further processing. func PostImage(w http.ResponseWriter, r *http.Request) { var p Path err := json.NewDecoder(r.Body).Decode(&p)iferr ! = nil { fmt.Fprintf(w, "got error while decoding body: %s", err)
        return
    }
    fmt.Fprintf(w, "got path: %+v\n", p)
    var ps Paths
    paths := make([]Path, 0)
    paths = append(paths, p)
    ps.Paths = paths
    var pathsJSON bytes.Buffer
    err = json.NewEncoder(&pathsJSON).Encode(ps)
    iferr ! = nil { fmt.Fprintf(w, "failed to encode paths: %s", err)
        return
    }
    r.Body = ioutil.NopCloser(&pathsJSON)
    r.ContentLength = int64(pathsJSON.Len())
    PostImages(w, r)
}
Copy the code

Well, naming might be better, but you should get the basic idea. I’m modifying the single path passed in, wrapping it in a new format and sending it to the new endpoint handler. That’s it! There are also some modifications. To see them, check out this PR: Rolling Update Bulk Image Path PR.

Now, receivers can be invoked in two ways:

# single path:
curl -d '{"path":"unknown4456.jpg"}' http://127.0.0.1:8000/image/post

# multiple paths:
curl -d '{"paths":[{"path":"unknown4456.jpg"}]}' http://127.0.0.1:8000/images/post
Copy the code

In this case, the client is curl. Usually, if the client is a service, I’ll change it and try the old path again when the new one throws 404.

For brevity, I don’t modify NSQ and other operations used for batch image processing, they will still receive one by one. I’ll leave that to you as homework.

A new image

To perform a rolling update, I must first create a new image from the receiver service.

Docker build-t skarlso/kube-receiver-alpine:v1.1.Copy the code

Once that’s done, we can start rolling out the changes.

Scroll to update

In Kubernetes, you can configure rolling updates in a number of ways:

Manual update

If I use a container version called V1.0 in my configuration file, then the update simply calls:

Kubectl rolling - update the receiver - image: skarlso/kube - receiver - alpine: v1.1Copy the code

If something goes wrong during deployment, we can always roll back.

kubectl rolling-update receiver --rollback
Copy the code

It will restore the previous version. No fuss, no trouble.

Apply a new configuration file

The problem with manual updates is that they are not in source control.

Consider this: Due to manual “quick fixes,” some servers get updated, but no one sees it, and no records. Another person comes along and makes changes to the template and applies it to the cluster. All servers are updated, and then there’s a sudden outage.

To make a long story short, the updated server has been overwritten because the template does not reflect the work done manually.

The recommended approach is to change the template to use the new version and apply the template using the apply command.

Kubernetes recommends using ReplicaSets for deployment should handle distribution, which means rolling updates must have at least two copies. If fewer than two replicas exist, the update will not work (unless maxUnavailable is set to 1). I increased the number of copies of YAML. I also set up a new mirrored version for the receiver container.

    replicas: 2
...
    spec:
        containers:
        - name: receiver
        image: skarlso/kube-receiver-alpine:v1.1
...
Copy the code

Take a look at the handling, here’s what you should see:

❯ kubectl rollout status Deployment /receiver-deployment Waitingfor rollout to finish: 1 out of 2 new replicas have been updated...
Copy the code

You can add additional deployment configuration Settings by specifying the Strategy section of the template, as follows:

strategy:
type: RollingUpdate
rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0
Copy the code

For more information on rolling updates, see the following documentation: Updating a Deployment, Manage Deployments, Rolling Update using ReplicaController.

Note to users of MINIKUBE: Since we are doing this on a local machine with a node and a copy of the application, we must set maxUnavailable to 1; Otherwise Kubernetes will not allow updates to occur and the new version will remain Pending. This is because we don’t allow a service without a running container, which basically means a service interruption.

Scaling

Scaling is easier with Kubernetes. Because it is managing the entire cluster, you basically just need to put a number into the template for the copy you want.

It’s a great article so far, but it’s taking too long. I’m planning to write a follow-up where I’ll really expand AWS functionality with multiple nodes and replicas; This is coupled with the Deployment of Kubernetes clusters by Kops. Stay tuned!

Clean up the

kubectl delete deployments --all
kubectl delete services -all
Copy the code

Write in the last

Ladies and gentlemen. We wrote, deployed, updated, and extended (not yet real, of course) distributed applications in Kubernetes.

If you have any questions, feel free to discuss them in the comments below. I’d be happy to answer them.

I hope you enjoy reading it, although it’s quite long and I’m thinking of breaking it up into multiple blogs, but an overall one-page guide is useful and can be easily found, saved and printed.

Thank you for reading.

If you find any mistakes in your translation or other areas that need to be improved, you are welcome to the Nuggets Translation Program to revise and PR your translation, and you can also get the corresponding reward points. The permanent link to this article at the beginning of this article is the MarkDown link to this article on GitHub.

The Nuggets Translation Project is a community that translates quality Internet technical articles from English sharing articles on nuggets. The content covers Android, iOS, front-end, back-end, blockchain, products, design, artificial intelligence and other fields. If you want to see more high-quality translation, please continue to pay attention to the Translation plan of Digging Gold, the official Weibo, Zhihu column.

Kubernetes distributed application deployment and face recognition app instance

About the application

The receiver

Image processor

Consume

ProcessImages

The circuit breaker

The front end

Face recognition

NSQ

configuration

Summary of face recognition applications

Deploy the application in Kubernetes

basis

Kubernetes Cluster

Nodes

Pods

Deployments

Services

DNS / Service Discovery

Template Files

ReplicaSet

DaemonSet

Scaling

Kubernetes summary

Minikube

Create a container

context

The deployment of application

MySQL

NSQ lookup

The receiver

Deployments

Nsq daemon

Service for the masses

Image processor

Face recognition

The front end

conclusion

Scroll to update

code

A new endpoint

A new image

Scroll to update

Manual update

Apply a new configuration file

Scaling

Clean up the

Write in the last

Related Posts

Product manager said after the interface does not look for you to write! What conditions? I’m getting cold?

AI to help you write a Python, install just 5 step, also can let you setup | open source

Compare performance of RDS for SqlServer, Aurora for MySQL, and Auro for postgreSQL