In this interview, I will continue to explain in detail my good partner, Alibaba’s open source Kubernetes container network solution Hybridnet, and how I use it to build a hybrid cloud unified network plane.

Authors: Ruo He, Yu Sheng, Yu Jia

Reporter: * * ** Dear readers of Alibaba Cloud, welcome to explore the mystery of life again. Today, we invite your old friend, “Alibaba Cloud Container Service ACK Release”. In the last interview, it introduced Alibaba’s open source cluster mirroring technology Sealer. And how they work together to achieve fast and stable delivery of Aliyun ACK services. For those of you who missed it, don’t forget to check back. So who will it introduce to us this time?

ACK Distro: Hello, I see you again! I believe that after the last self-introduction, you have a general understanding of me, this time I will not repeat. In this interview, I will continue to explain in detail my good partner, Alibaba’s open source Kubernetes container network solution Hybridnet, and how I use it to build a unified network plane of hybrid cloud.

** Reporter: ** OK, so you talk a little bit about what hybridnet is and how the members of the project team came to it.

Hybridnet definition and design concept

**ACK Distro: ** Well, first hybridnet is an open source Kubernetes container network solution from Alibaba for hybrid cloud scenarios. It helps users build a unified network plane with underlay and overlay on the heterogeneous environment of physical machines and virtual machines, and provides rich operation and maintenance (O&M) capabilities. At the same time, a new solution is proposed for container network deployment, operation and maintenance problems in cluster deployment and application delivery in hybrid cloud scenarios.

Its basic design principles are:

  1. In order to determine the unified network model, reduce the cost of cognition and maintenance, and ensure stable long-term evolution
  2. Shielding underlying heterogeneous infrastructure improves the robustness of delivery landing and reduces production delivery and PoC costs
  3. Under the constraints of the unified model, it can not only provide underlay high-performance passthrough network solution to meet the dual requirements of network connectivity and performance, but also provide overlay virtual network solution in some performance insensitive scenarios
  4. Minimize the dependence on the external environment to ensure the simplicity of the data surface
  5. Deep integration with Kubernetes, providing dual-stack, IP retention, IP fixed and other high-level IPAM capabilities, to ensure that users’ usage habits remain unchanged after accessing the cloud

Different from container network solutions such as Terway and AWS-CNI, which are bound to the output of public cloud or private cloud base of a single IaaS vendor, project team members hope hybridnet can solve the problems of consistency and adaptability brought by heterogeneous base in a multi-cloud hybrid cloud scenario. It provides agile, universal and stable delivery capabilities on different basic network environments, and solves network planning, management, and operation and maintenance problems in complex scenarios through model constraints and operation and maintenance control from a unified perspective.

Hybridnet hybridnet seeks to achieve “underlay/overlay hybrid deployment” and “underlay/overlay unified management operation and maintenance”.

**ACK Distro: ** Yes, I do, and hybridnet does, too. I can further expand the description. In a Kubernetes cluster that uses hybridnet, there can be pods with underlay and overlay on the same node. All pods have the same access behavior within the cluster without any extra perception. In this way, users can freely choose and convert between “pure Overlay cluster”, “pure Underlay cluster”, and “Underlay hybrid cluster”, while enjoying the high performance and network passthrough capability brought by The Underlay network. And overlay network with unlimited resources and high adaptability. In addition, under the constraints of the unified model, underlay network and Overlay network also maintain the same concept in management, operation and maintenance.

** Reporter: ** In addition to its design concept, can you use a more intuitive way to make readers understand what Hybridnet can do?

* * ACK Distro: ** Of course, I will explain its functional attributes by introducing its core model ~ In order to make the users of Hybridnet flexibly describe the basic network environment by initializing different core models, so that the container network can run in different forms, the project team members abstract the concepts in the classical network. The following three core CRD models are introduced to abstract and manage network resources.

Hybridnet’s core model

Network

In hybridnet, each Network represents a “scheduling domain”, and a scheduling domain represents a group of nodes with the same Network properties. Network is the main entrance to the environment topology information. A particular IP address can migrate freely between nodes in the scheduling domain to which it belongs.

Networks are associated with nodes through nodeSelector. For some special networks, such as overlay networks, nodeSelector may be empty. The scheduling domain of such networks is all nodes in the cluster.

Subnet

In hybridnet, Subnet represents the IP resources that can be allocated in a scheduling domain. Subnet is the main input of IP resource planning information in the environment network. Each Subnet must belong to a Network. Subnet has flexible properties:

  • Cidr distributable address range selection is supported. Spec.range. start and spec.range.end can be used to accurately divide a small network segment from a CIDR
  • You can reserve discrete IP addresses. When some IP addresses are used in the network segment, you can enter the used IP addresses into the array field of spec.range.excludeIPs. Hybridnet will not allocate these IP addresses to Pod
  • Reserve IP addresses. If you need to reserve the IP addresses of certain PODS in a network segment, enter these IP addresses in the spec.range.reservedPs array field. Hybridnet will no longer use these IPS for assignment of non-specified IP addresses

IPInstance IPInstance is used only for monitoring. Each IPInstance represents an IP address that has been allocated to the container network. You can use kubectl get IPInstance to see the corresponding Pod, Subnet, Pod node, and other information.

Reporter: So how do hybridnet’s advantages reflect on you? In other words, how do you manage Hybridnet to achieve best practices?

How to manage Hybridnet in ACK Distro

**ACK Distro: ** I’m working through the ABOVE CRD model to give you an example of network management. Hybridnet is deployed as my only built-in network plug-in. (Of course, it is also possible to customize third-party web plugins with Sealer’s capabilities, as mentioned in the first article in this series.)

The default behavior

The default Network type is overlay. Run the following command to check the Network and Subnet information:

[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get network
NAME        NETID   SWITCHID
network-0   4       virtual-switch
[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get subnet
NAME                 VERSION   CIDR            START   END   GATEWAY      TOTAL   USED   AVAILABLE   NETID   NETWORK
subnet-0-network-0   4         100.64.0.0/16                 100.64.0.1   65533   2      65531               network-0
Copy the code

The CIDR of the container network segment is 100.64.0.0/16. The CIDR of the container network segment is 100.64.0.0/16. In this case, all new pods are pulled in an overlay mode.

Because my base component itself has no special network appeal, the biggest advantage of this is that the overlay network helps me to shield the underlying infrastructure. In other words, I can use hybridnet to pull up in any network environment with one click in the same configuration without affecting the possibility of subsequent network expansion.

According to the delivery experience of hybrid cloud environment, this method can postpone network planning (mainly underlay network) to the “operation and maintenance” stage, minimize the implementation cost of the “delivery” stage and improve deployment efficiency.

Add the Underlay network

If there are some underlay network requirements (such as “overlay performance bottleneck” or “Pod IP directly exposed to external capabilities”), the underlay Pod is relatively small, especially if you do not want to occupy IP resources in the underlying network environment. You can optionally add an underlay Network and a Subnet in addition to the overlay Network created by the default behavior. (Old and new overlay/underlay networks don’t have any order of dependencies on the model)

In the experimental environment of this example, the node network segment is 192.168.56.6/24 (all nodes are in a classical layer 2 network), because only 192.168.56.1, 192.168.56.2, 192.168.56.3, 192.168.56.4 are used for node IP. We consider reserving the unused 192.168.56.100 ~ 192.168.56.150 address range for the container to build the simplest underlay network. In this case we just need to apply yamL as follows:

--- apiVersion: networking.alibaba.com/v1 kind: Network metadata: name: underlay-network1 spec: netID: 0 nodeSelector: network: network1 type: Underlay --- apiVersion: networking.alibaba.com/v1 kind: Subnet metadata: name: Underlay-subnet1 spec: network: underlay-network1 netID: 0 range: version: "4" CIDR: "192.168.56.0/24" Gateway: Start: "192.168.56.100" end: "192.168.56.150"Copy the code

Since Network is associated with nodes through nodeSelector, we need to label the Node where we want to deploy Underlay Pod as Network nodeSelector. Here we only want pods of type underlay on node izf8ZdygpBO4hx57g2wah8z:

kubectl label node izf8zdygpbo4hx57g2wah8z network=network1
Copy the code

At this point the default network type is still an overlay network, create an underlay Pod simply by adding the networking.alibaba.com/network-type: to Pod underlay the annotation to specify. The effect is as follows:

[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get po -owide -n test NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES curl-deployment-1-5CFB5DCB8C-65FR7 1/1 Running 0 11m 100.64.0.29 IZF8zdygpBO4hx57g2waHBz < None > <none> Curl - deployments-1-5cfb5dcb8c-hp626 1/1 Running 0 11m 100.64.0.26 izf8zdygpBO4hx57g2wahbz <none> <none> Curl - deployments-1-5cfb5dcb8c-qbr6w 1/1 Running 0 11m 100.64.0.27izf8zdygpBO4hx57g2wah7z <none> <none> Curl - deployments-1-5cfb5dcb8c-zclv2 1/1 Running 0 11m 100.64.0.31 izf8zdygpBO4hx57g2wahbz <none> <none> Curl - deployments-1-5cfb5dCB8c-zfqKP 1/1 Running 0 11m 100.64.0.28izf8zdygpBO4hx57g2wah7z <none> <none> curl-ss-0 1/1 Running 0 6m24s 192.168.56.140 izf8zdygpBO4hx57g2wah8z <none> <none> curl-ss-1 1/1 Running 0 6m5s 192.168.56.141 Izf8zdygpbo4hx57g2wah8z <none> <none> curl-ss-2 1/1 Running 0 6m1s 192.168.56.142 izf8zdygpBO4hx57g2wah8z <none> <none>Copy the code

Change the default network type to underlay

If you want to create an overlay Pod using underlay’s network, you can also change the default network type to Underlay. Then the Pod will be created using underlay’s network by default. And networks with overlay can still be created by specifying pods by adding annotations to pods. Overlay pods already created will not be affected.

To modify the default network type, kubectl Edit deploy hybridnet-webhook-n kube-system and kubectl edit deploy hybridnet-manager-n respectively Kube-system, change the container startup DEFAULT_NETWORK_TYPE environment variable to Underlay:

spec:
  containers:
    - name: hybridnet-[manager|webhook]           
      command:
        - /hybridnet/hybridnet-[manager|webhook]
      env:
        - name: DEFAULT_NETWORK_TYPE
          # "Overlay" or "Underlay", 
          # default "Underlay" if environment variable not configured. 
          value: Underlay
Copy the code

By default, pods will be created in the underlay mode. The network connectivity between the new Underlay Pod and the existing overlay Pod will not be affected. The underlay Pod will have an overlay identity that communicates with other overlay pods.

Add or delete network resources

As shown in the example above, you only need to apply YAML corresponding to CR to add Network resources of Network/Subnet. Once Network/Subnet is applied, Hybridnet thinks that basic Network configuration has been completed in the environment. The corresponding CR is used to allocate network resources.

For security reasons, deleting a Network/Subnet has basic constraints. A Subnet can be deleted only if no IP address is in use. Similarly, a Network can be deleted only after all subnets in the Network are deleted.

ACK Distro: In a word, with the help of Hybridnet, I can build a unified network plane of underlay + overlay based on the heterogeneous environment for ALI Cloud container service ACK, improve the ability of control operation and maintenance, and bring better container service experience for the vast number of developers.

** Reporter: ** Well, thank you very much for your careful explanation this time, the second round of in-depth interview here again to say goodbye to everyone, looking forward to the next meeting with readers.

**ACK Distro: ** See you next time!

The original link

This article is the original content of Aliyun and shall not be reproduced without permission.