Generally, Kubernetes’ Cluster Network is a private Network, and deployed applications can be accessed only from within the Cluster Network. So how to expose the Kubernetes cluster applications to external networks to provide services to external users? This article will talk about access from the external network in Kubernetes Cluster Pod and Serivce several common implementation methods.

Relationship between Pod and Service

Let’s start by looking at the concepts of Pod and Service in Kubernetes and how they relate.

Pod (container group), Pod means Pod in English. As the name implies, a Pod is a set of dependent containers. A Pod is the most basic resource object in a Kubernetes cluster. Each Pod consists of one or more business containers and a root container (Pause container).

Kubernetes assigns each Pod a unique IP (i.e., Pod IP) that is shared by multiple containers within the Pod. Containers in a Pod share the same network namespace, port, volume, and so on, in addition to IP, which means they can communicate with each other via Localhost. All containers contained in pods run on the same node. You can also start multiple pods at the same time for Failover or Load balance.

The Pod life cycle is short. Kubernetes will create and destroy the Pod according to the application configuration, and scale and expand the Pod according to the monitoring indicators. Kubernetes can create pods on any free node in the cluster, so the network address is not fixed. Due to this feature of Pod, it is generally not recommended to access applications directly through the Pod address.

To solve the problem of inconvenient direct access to Pod, Kubernetes uses Service to encapsulate Pod. A Service is an abstraction of a set of PODS that provide services to the back end, bound to a fixed virtual IP. This virtual IP is visible only in the Kubernetes Cluster, but it does not correspond to a virtual or physical device, but to the rules in IPtables, which route service requests to the Pod on the back end. In this way, service consumers can be assured of stable access to services provided by PODS, regardless of changes such as Pod creation, deletion, migration, and load balancing with a set of pods.

The key to Service implementation is kube-proxy in Kubernetes. Kube-proxy runs on each node, listens for changes of service objects in API Server, and then implements network forwarding by managing IPtables. Kube-proxy supports UserSpace, IPtables, and IPVS. Here are the similarities and differences:

  • UserSpace

UserSpace allows kube-proxy to listen on a port in UserSpace, to which all services are forwarded, and then kube-proxy forwards it in the internal application layer.

Kube-proxy randomly listens on a Proxy Port for each Service and adds an IPtables rule. Packets sent from the client to ClusterIP:Port are redirected to Proxy Port. After receiving the packets, Kube-proxy uses Round Robin or Session Affinity. That is, IP addresses of the same Client go through the same link to the same Pod service.

The biggest disadvantage of this approach is that UserSpace will cause all packets to go through the user mode, resulting in overall performance degradation. This method has not been used since Kubernetes 1.2.

  • IPtables

The IPtables mode is completely implemented by IPtables. In this mode, IPtables is directly used as the user mode entry, and the real service is provided by the kernel Netilter. Kube-proxy only acts as Controller, which is currently the default.

Kube-proxy IPtables also supports Round Robin and Session Affinity.

The kube-proxy listens for the Kubernetes Master to add and delete Service and Endpoint messages. For each Service, the Kube Proxy creates the corresponding IPtables rules and forwards the traffic to the Service Cluster IP to the corresponding port of the Service backend Pod.

Note: Although the back-end Pod Service can be accessed through the Cluster IP address and Service port of the Service, the Cluster IP address cannot be pinged. The reason is that Cluster IP is just a rule in IPtables and does not correspond to any network device. The Cluster IP address in IPVS mode can be pinged.

  • IPVS

Kubernetes has added IPVS support since 1.8, which makes IPVS more efficient than IPtables. To use IPVS mode, you need to install ipvSADm, the Ipset toolkit, and load the IP_VS kernel module.

Note: IPVS is part of the LVS project and is a layer 4 load balancer running in the Linux Kernel. More than 100,000 forward requests per second can be easily handled with a tuned kernel. IPVS is now widely used in medium to large Internet projects to receive traffic at the entrance to web sites.

After understanding the basic concepts of Pod and Service, we will specifically talk about several common implementations of Kubernetes Cluster Pod and Serivce access from external networks. Currently, it mainly includes the following types:

  • hostNetwork

  • hostPort

  • ClusterIP

  • NodePort

  • LoadBalancer

  • Ingress

Exposure through Pod

hostNetwork: true

This is a direct way to define a Pod network.

With the hostNetwork: True configuration in a Pod, applications running in this Pod can directly see the network interface that started the Pod host. The application is accessible on all network interfaces of the host. Here is an example definition of a Pod using the host network:

apiVersion: v1 kind: Pod metadata: name: nginx-hostnetwork spec: hostNetwork: true containers: - name: Nginx - hostnetwork image: nginx: 1.7.9Copy the code

To deploy the Pod:

$ kubectl create  -f nginx-hostnetwork.yml
pod "nginx-hostnetwork" createdCopy the code

To access port 80 of the host where the Pod is located:

$ curl http://$HOST_IP:80 ...... <title>Welcome to nginx! </title> <body> <h1>Welcome to nginx! </h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> ......Copy the code

You will see the default Nginx welcome page, indicating that it is accessible.

Note: Each time the Pod is started, it may be scheduled to a different node, so that all external access to the Pod node host IP is not fixed, and when scheduling Pod also need to consider whether the conflict with the host port. Therefore, hostNetwork: true is generally used only when you know that a particular application is required to occupy a particular port on a particular host.

One use of this Pod networking mode is that network plug-ins can be wrapped in a Pod and then deployed on each host so that the Pod can control all the networks on that host.

hostPort

This is also a direct way to define a Pod network.

HostPort directly routes container ports to ports on the scheduled nodes, so that users can access pods by adding “hostIP” to the hostIP address, for example, “hostIP” : “hostPort”.

apiVersion: v1
kind: Pod
metadata:
name: nginx-hostport
spec:
containers:
- name: nginx-hostport
image: nginx:1.7.9
ports:
- containerPort: 80
hostPort: 8088Copy the code

To deploy the Pod:

$ kubectl create  -f nginx-hostport.yml
pod "nginx-hostport" createdCopy the code

To access port 8088 of the host where this Pod is located:

$ curl http://$HOST_IP:8088 ...... <title>Welcome to nginx! </title> <body> <h1>Welcome to nginx! </h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> ......Copy the code

You will see the default Nginx welcome page, indicating that it is accessible.

This approach has the same disadvantage as hostNetwork: true. When a Pod is rescheduled, the host to which the Pod is scheduled may change. In this case, the

will change, and the user must maintain the mapping between a Pod and the host.

This network mode can be used as the Ingress Controller. External traffic needs to pass through ports 80 and 443 of the host of Kubenretes node.

Port Forward

This is a method of data forwarding by kubectl port-forward instruction. The kubectl port-forward command sets port forwarding for pods. By specifying listening ports on the host, requests to access these ports will be forwarded to the corresponding ports in the Pod container.

Kubernetes Port Forward (Kubernetes Port Forward)

After creating a Port Forward using Kubectl, Kubectl actively listens for the specified local Port.

$ kubectl port-forward pod-name local-port:container-portCopy the code

When a Port connection is established to the local-port and data is sent to the Port, the data flow goes through the following steps:

  • The data is sent to the local-port that Kubctl listens on.

  • Kubectl sends data to ApiServer over the SPDY protocol.

  • ApiServer connects with Kubelet of the target node and sends data to the port of the target Pod through SPDY protocol.

  • After receiving the data, the Kubelet of the target node communicates with Socat through PIPE (STDIN, STDOUT).

  • Socat sends STDIN data to the container port specified inside the Pod and writes the returned data to STDOUT.

  • The STDOUT data is received by Kubelet and sent back along the reverse path.

Note: The SPDY protocol may be replaced with HTTP/2 in the future.

Next, we use an example to demonstrate how to forward a local port to a port in a Pod, using a Pod running Nginx as an example:

$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
nginx                       1/1       Running   2          9dCopy the code

Verify the port on which the Nginx server listens:

$ kubectl get pods nginx --template='{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
80Copy the code

Forward port 8900 on the node to port 80 on the Nginx Pod:

Kubectl port-forward Kubectl port-forward Kubectl port-forward $kubectl port-forward nginx 8900:80 Forwarding from 127.0.0.1:8900 -> 80 Forwarding from [::1]:8900 -> 80Copy the code

Note: You need to install Socat on all Kubernetes nodes. For more details about Socat, please refer to “Socat Tutorial”.

Verify whether the forwarding is successful:

$curl http://127.0.0.1:8900... <title>Welcome to nginx! </title> <body> <h1>Welcome to nginx! </h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> ......Copy the code

Because this type of forwarding port is bound locally, this method is only applicable to debugging services.

Exposure via Service

The ServiceType of a Service determines how the Service provides services externally. Depending on the type, the service can be visible only in the Kubernetes Cluster, or it can be exposed outside the Cluster. There are three types of Services: ClusterIP, NodePort and LoadBalancer.

The meanings of the three common ports in Service are:

  • port

Service Exposes the port on the Cluster IP, that is, the port to which the virtual IP is bound. A port is an entry point for customers within a cluster to access services.

  • nodePort

NodePort is the gateway that Kubernetes provides to customers outside the cluster to access services.

  • targetPort

TargetPort is the port of the container in Pod. Data from port and nodePort will eventually flow through kube-proxy to the port of the container in the backend Pod. If targetPort is not explicitly declared, it is forwarded by default to the port at which the Service receives the request (the same value as port).

In general, both port and nodePort are ports of services, with the former exposed to clients within the cluster and the latter to clients outside the cluster. Data coming from these two ports needs to pass through the reverse Proxy Kube-Proxy to the port of the container in the back-end Pod and then to the container on the Pod.

ClusterIP

ClusterIP is the default type of Service. This type of Service automatically allocates a virtual IP address that can be accessed only within the cluster, namely, ClusterIP. ClusterIP provides you with a service that can be accessed by other applications within the cluster, but not externally. YAML for ClusterIP services looks something like this:

apiVersion: v1
kind: Service
metadata:
name: my-nginx
selector:
app: my-nginx
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCPCopy the code

Although we cannot access a ClusterIP service directly from outside the cluster, you can access it using the Kubernetes Proxy API.

The Kubernetes Proxy API is a special API where Kube-Apiserver simply proxies HTTP requests from such apis and forwards the requests to ports that the Kubelet process on a node listens to. The REST API on that port actually responds to the request.

Create Kubernetes API proxy service on Master node:

$ kubectl proxy --port=8080Copy the code

Kubectl proxy listens on 127.0.0.1 by default, so if you need to provide external access, you can use some basic security mechanisms.

$kubectl proxy --port=8080 --address=192.168.100.211 --accept-hosts='^192\.168\.100\.*'Copy the code

If you need more help with command usage, you can use kubectl Help Proxy.

You can now access it using the Kubernetes Proxy API. For example, to access a SERVICE, use/API /v1/namespaces/

/services/< service-name >/proxy/.

$kubectl get SVC NAME TYPE cluster-ip external-ip PORT(S) AGE Kubernetes ClusterIP 10.254.0.1 My-nginx ClusterIP 10.254.154.119 <none> 80/TCP 8D $curl http://192.168.100.211:8080/api/v1/namespaces/default/services/my-nginx/proxy/... <title>Welcome to nginx! </title> </head> <body> <h1>Welcome to nginx! </h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> ......Copy the code

If you need to access a Pod directly, use/API /v1/namespaces/

/ Pods /< pod-name >/proxy/.

RESTARTS for Kubernetes $1.10 $kubectl get pod NAME READY STATUS RESTARTS AGE my-nginx-86555897f9-5p9c2 1/1 Running 2 8D my-nginx-86555897f9-ws674 1/1 Running 6 8d $ curl http://192.168.100.211:8080/api/v1/namespaces/default/pods/my-nginx-86555897f9-ws674/proxy/... <title>Welcome to nginx! </title> <h1>Welcome to nginx! </h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> ......Copy the code

Since accessing the Kubernetes Proxy API requires you to run Kubectl as an authorized user, you should not use this method to expose your service to the public network or for production purposes. This method is mainly used for debugging services.

NodePort

NodePort is a widely used service exposure method in Kubenretes. Based on the function provided by ClusterIP, Service is bound to a port on each node of the Kubernetes cluster, which is called NodePort. Service can be accessed externally from the cluster based on any NodeIP:NodePort. Services are available on the NodePort port of each node.

The NodePort service differs from the default ClusterIP service in YAML definitions in two ways: First, Type is NodePort. A second parameter, called nodePort, specifies which port to open on the node. If you do not specify this port, it will select a random port.

apiVersion: v1
kind: Pod
metadata:
name: my-nginx
labels:
name: my-nginx
spec:
containers:
- name: my-nginx
image: nginx:1.7.9
ports:
- containerPort: 80Copy the code

The default value of nodePort is 30,000-32767, which can be customized in the API Server configuration file with –service-node-port-range.

kind: Service
apiVersion: v1
metadata:
name: my-nginx
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
nodePort: 31000
selector:
name: my-nginxCopy the code

The NodePort service exposes a single port on all Kubenretes nodes (nodes running Kube-proxy) to provide services externally, so that the IP address of any Kubernetes node and the specified port can be used outside the cluster. 31000) to access the service. Kube-proxy automatically forwards traffic to each Pod of the Service in a round-robin manner.

The NodePort service does not affect the original virtual IP address access mode. Internal nodes can still be accessed through VIP:Port. NodePort’s approach to service exposure also has some drawbacks:

  • Each port on a node can have only one service.

  • If the IP address of a node is changed, a mechanism is required to handle the problem.

For the above reasons, NodePort is suitable for demo applications or temporary applications. It is not recommended to expose services using this method in production environments.

LoadBalancer

LoadBalancer is based on the NodePort and external LoadBalancer provided by the cloud service provider, through which external requests are forwarded to each NodeIP:NodePort for external exposure service.

LoadBalancer can only be defined on a Service. A LoadBalancer is a LoadBalancer provided by a specific public cloud and supported by a specific cloud service provider. Such as AWS, Azure, OpenStack, and Google Container Engine (GCE).

kind: Service
apiVersion: v1
metadata:
name: my-nginx
spec:
type: LoadBalancer
ports:
- port: 80
selector:
name: my-nginxCopy the code

View services:

$kubectl get SVC my-nginx NAME cluster-ip external-ip PORT(S) AGE my-nginx 10.97.121.42 10.13.242.236 8086:30051/TCP 39sCopy the code

ClusterIP and port can be used to access services within a cluster, for example, 10.97.121.42:8086.

This service can be accessed externally in one of two ways:

  • Use the IP address of any node plus port 30051 to access the service.

  • Access using external-IP, which is a VIP that is the IP address of the load balancer provided by the cloud provider, for example, 10.13.242.236:8086.

The biggest drawback of this LoadBalancer approach is that each exposed service needs to use a LoadBalancer IP provided by the public cloud, which can be costly.

Based on the conclusions of the above types of services, the load balancing function provided by services has the following limitations:

  • Only layer 4 load balancing is supported, but layer 7 load balancing is not supported. For example, you cannot customize the forwarding request based on the required matching rule.

  • To use the NodePort Service, an external load balancer must be deployed outside the cluster.

  • To use a LoadBalancer Service, Kubernetes must run on a specific cloud Service.

Exposure through Ingress

Unlike Service, Ingress is not actually a Service. Instead, it sits in front of multiple services and acts as an intelligent router or entry point in a cluster.

Ingress is a resource type introduced since Kubernetes 1.1. Ingress allows services to be exposed outside the Kubernetes cluster and can also customize Service access policies. Ingress can configure a Service as a URL accessible from the Internet, and can also provide virtual hosting by domain name. For example, load balancers can be used to access different secondary domain names to different services.

In fact, Ingress is just a general term, which consists of Ingress and Ingress Controller. Ingress is used to abstract rules that originally need to be configured manually into an Ingress object, which is created and managed using YAML format files. The Ingress Controller is used to dynamically sense Ingress rule changes in the cluster by interacting with the Kubernetes API.

Before using Ingress, you must first deploy the Ingress Controller, which is provided as a plug-in. Ingress Controller is usually a Docker container deployed on top of Kubernetes, The Ingress Controller Docker image contains a load balancer like Nginx or HAProxy and an Ingress Controller. The Ingress Controller receives the required Ingress configuration from Kubernetes, dynamically generates an Nginx or HAProxy configuration file, and restarts the load balancer process for the changes to take effect. In other words, the Ingress Controller is a load balancer managed by Kubernetes.

Note: Ingress Controller is officially referred to as Ingress Controller regardless of the load balancing software used (e.g. Nginx, HAProxy, Traefik, etc.).

Kubernetes Ingress provides typical features of a load balancer: HTTP routing, sticky sessions, SSL termination, SSL passthrough, TCP and UDP load balancing, etc.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
spec:
backend:
serviceName: my-nginx-other
servicePort: 8080
rules:
- host: foo.mydomain.com
http:
paths:
- backend:
serviceName: my-nginx-foo
servicePort: 8080
- host: mydomain.com
http:
paths:
- path: /bar/*
backend:
serviceName: my-nginx-bar
servicePort: 8080Copy the code

The Ingress Controller then forwards the traffic directly to the backend Pod. No further forwarding through kube-proxy is more efficient than LoadBalancer.

In general, Ingress is a very flexible and increasingly vendor-supported way of exposing services, including Nginx, HAProxy, Traefik, and various Service meshes, while other ways of exposing services are more suitable for Service debugging and special application deployment.

Reference documentation

http://www.google.com

http://t.cn/RdskHHt

http://t.cn/RdskYv5

http://t.cn/Rg5BG4h

http://t.cn/R6CD4ak

http://t.cn/Rgcurto

http://t.cn/RgcdjDw

http://t.cn/RgMWBpo

http://t.cn/RgMWFM1

http://t.cn/RgC3uk8

Thanks to have you

I’m glad you can see me and that you have finished reading this article. Very good yo! If you think the content of the article has a little bit of use, let your face great joy. It is recommended that you press the “praise” button below to express your joy. Of course, if you feel that “praise” can not express your love, you can also give a thumbs up to the lovely rectangular matrix in the article. People do not say dark words, the development of public accounts need your support. Ha ha!

Today’s idea

What is wealth? No need to bow to others, is expensive; He is rich without stretching out his hand.

– shi yuzhu

Recommended reading

  • Diagram of Docker architecture

  • Illustrates the Kubernetes architecture

  • MySQL User and role management

  • How Docker commands work

  • Vegeta is the recommended high-performance HTTP load testing tool