This article explored the possibility of processing data directly in ETCD without using the Kubernetes API, and successfully tested all the steps on a real K8s cluster.

By Flant Staff Translated by Bach

Proofreading: Wen Zai, Bot under the starry sky, source: K8sMeetup community

Have you ever considered a “low-level” way to change etCD data in a Kubernetes cluster? This means changing etCD data without using any common Kubernetes tools (such as CLI programs or even apis).

It all starts here

More and more customers, mostly developers, are requesting access to the Kubernetes cluster to interact with internal services. They want to be able to connect directly to a database or service, connect local applications to other applications in the cluster, and so on.

For example, from the local computer connect to memcached. The staging. SVC. Cluster. The local service. We can achieve this through the intra-cluster VPN connected by the client. We expose the subnets associated with Pod and Service and push DNS for the cluster to the client. As a result, when the client tries to connect to memcached. The staging. SVC. Cluster. The local service, requests to cluster the DNS, it from the cluster service network or Pod address returns the address of the service.

We used Kubeadm to configure the K8s cluster. In this case, the default Service Subnet is 192.168.0.0/16 and Pod subnet is 10.244.0.0/16. This works fine most of the time, but there are a few caveats:

  • 192.168.*.* Subnets tend to be used in client offices or even developers’ home offices. Here’s the problem: home routers use the same address space, and VPNS push these subnets from the cluster to the client.

  • In the case of several clusters (production, Stage, multiple Dev clusters), they will all have the same subnet as Pod and service by default, making it very difficult to use services in multiple clusters at the same time.

We used different subnets in the same project to implement different services and pods in the same project. In this case, any cluster has its own network. When maintaining a large number of K8s clusters, we don’t want them to be redeployed from scratch because there will be many running services, stateful applications, and so on. The question arises: How do we change subnets in an existing cluster?

The solution

The most common approach is to recreate all ClusterIP type Services. But there are problems:

The following process is faulty: After configuring everything, Pod uses the old IP address as DNS nameserver in /etc/resolv.conf.

Because there was still no solution, the entire cluster had to be reset using kubeadm Reset to initialize it again.

This isn’t suitable for all scenarios, but here’s a more detailed look at the case:

  • Use Flannel.

  • There are both bare machine clusters and Kubernetes clusters.

  • You want to avoid redeploying all services in the cluster.

  • We want to make the transition as easy as possible.

  • The cluster is managed by Kubernetes V1.16.6 (but also available in other versions).

  • The goal is to replace the 192.168.0.0/16 service subnet with 172.24.0.0/16 in the deployed cluster using kubeadm.

In fact, many people have been wondering how Kubernetes stores its data in ETCD and what it can do with it. So why don’t we update the data subnet IP in etCD with the new IP by replacing the old one?

It would be nice to have an off-the-shelf tool to modify the data in ETCD, and while nothing currently fits our needs well, OpenShift’s ETCDHelper is a good place to start. This tool can connect to etCD using certificates and read ETCD data with ls, get, and dump commands.

Extension etcdhelper

Therefore, we are going to extend etCDHelper to write data to etCD. We created an updated version of etCDHelper with two new features: changeServiceCIDR and changePodCIDR. The source code is available here: github.com/flant/examp…

What does the new feature do? This is the changeServiceCIDR algorithm:

  • Create a deserializer.

  • Compile the regular expression to replace CIDR.

  • Iterate through the List of ClusterIP Services in the cluster and perform operations on each service.

Here is our operation:

  • Decode the ETCD value and place it in the Go object.

  • Replace the first two bytes of the address with a regular expression.

  • Assign an IP address to the Service from the address range of the new subnet.

  • Create a serializer that converts the Go object to a protobuf and writes the new data to etCD.

The changePodCIDR function is basically the same as changeServiceCIDR, the only difference is that when we edit the specification of the node, we replace the value of.spec.PodCIDR with a new subnet instead of service.

usage

Replace serviceCIDR

This is easy to do, but it will cause an outage and all the pods in the cluster will be recreated. Here we show you the main steps, then share how to minimize downtime.

Preparation Steps:

  • Install the necessary software and build the patch etcdHelper tool;

  • Backup etcd and /etc/kubernetes.

Here’s a summary of how to change the serviceCIDR:

  • Change the Apiserver and controller manager listings.

  • Re-issue the certificate.

  • Modify the ClusterIP specification for Services in ETCD.

  • Restart all pods in the cluster.

Here is a detailed description of these steps:

1. Install etcd-client to dump data.

2. Build etcdHelper:

  • The installationgolang

  • copyetcdhelper.go, download dependencies, build tools:

3. Backup etCD data:

4. In the Kubernetes control plane list, switch the Service Subnet. In the/etc/kubernetes manifests/kube – apiserver. Yaml and/etc/kubernetes manifests/kube – controller – manager. Replace yaml –service-cluster-ip-range specifies the value of the new subnet (172.24.0.0/16 replaces 192.168.0.0/16).

5. Since we are making changes to the Service Subnet that issues apiserver certificates (and other certificates) to Kubeadm, we need to reissue:

5.1. Check the domain and IP address from which the current certificate is issued:

5.2. Prepare kubeadm basic configuration:

5.3. Delete old CRT and key files (delete them to issue new certificates) :

5.4. Reissue the API server certificate:

5.5. Check whether a certificate has been issued for the new subnet:

5.6. After the API server certificate is reissued, restart its container:

5.7. Renew the certificate embedded in admin.conf:

5.8. Edit data in ETCD:

The DNS stops resolving domain names in the cluster. This is because the existing Pod still has the old CoreDNS (kube-DNS) address in /etc/resolv.conf, and kube-proxy has used the new subnet and changed the iptables rules.

5.9. Edit ConfigMap in kube-system namespace:

In this CM:

Replace the ClusterDNS with the new IP address of kube-DNS service: kubectl -n kube-system get SVC kube-dns

In this CM:

Data.ClusterConfiguration.net working. ServiceSubnet parameter switch to the new subnet.

5.10. Because the kube-DNS address has changed, the kubelet configuration needs to be updated on all nodes:

5.11. Now it’s time to restart all pods in the cluster:

Reduced downtime

Here are some ideas on how to minimize downtime:

  • After editing the control plane manifest, we can create a new kube-DNS service with a new name (for example, kube-dns-tmp) and a new address (172.24.0.10).

  • We insert an if condition in etcdHelper, which prevents modification of kube-DNS service.

  • Replace all old ClusterDNS addresses in Kubelet with new ones (the old service will continue to run concurrently with the new service).

  • And all the applications’ PODS are redeployed at the agreed time.

  • Delete kube-dns-tmp service and edit servicesubnetcidrKube-dns service.

This policy can reduce the downtime to about one minute: the time required to delete kube-dnS-tmp service and switch the subnet of kube-DNS service.

Modify podNetwork

In this process, we use etCDHelper to modify the podNetwork. This is the order of operations:

  • Edit configuration in the kube-system namespace.

  • Edit the kube-Controller-manager listing.

  • Edit podCIDR directly in etCD.

  • Restart all nodes in the cluster.

The following is a detailed description of the above operations:

1. Edit ConfigMap in kube-system namespace:

Replace data.ClusterConfiguration.net working. PodSubnet for the new subnet (10.55.0.0/16).

Specify the new data. The config. Conf. ClusterCIDR: 10.55.0.0/16.

2. Edit the Controller-Manager listing:

Specify –cluster-cidr=10.55.0.0/16.

3. Verify the current values of.spec.podCIDR,.spec.podCIDRs,.internalip, and.status.addresses of all cluster nodes:

4. Directly edit etCD to replace podCIDR:

5. Check whether the podCIDR has been changed:

6. Restart all nodes in the cluster at a time.

7. If there is at least one node with an old podCIDR, kube-Controller-Manager will not start and pods in the cluster will not be scheduled.

Original link:

Medium.com/flant-com/m…