Author: Qingxi Ali Cloud technical expert, mainly responsible for ACK container network design and research and development, main maintainer of Ali Cloud open source CNI project Terway, principal of Cilium Alibaba IPAM

background

Recently, Cilium community released the official version 1.10 of Cilium, which officially supports ENI mode of Ali Cloud. Ali Cloud is the first cloud vendor in China to support Cilium.

Cilium is a high performance container network project based on eBPF, providing three aspects of network, observability and security solutions.

Cilium itself supports Overlay network mode for deployment on various cloud platforms or self-built clusters, but this non-cloud native network mode can cause significant performance loss. Alibaba Cloud Native Container Service team has contributed ali Cloud ENI mode to Cilium community, enabling Cilium to run in cloud native mode on Ali Cloud. Contributed by cloud Native Container Services team PR github.com/cilium/cili… Github.com/cilium/cili…

architecture

The AlibabaCloud Operator is a network resource controller in a cluster. It centrally manages and allocates network resources (ENI and ENIIP).

The Cilium Agent configures and manages the address resources allocated by the Operator through the list-watch mechanism and CNI request. This architecture centralizes all Ali Cloud OpenAPI calls into Operator, which can effectively manage API requests and avoid API flow control problems in large-scale clusters.

High-performance cloud native network based on Cilium 1.10 + ENI ali Cloud

Cilium uses EBPF kernel technology to optimize the traditional data link, bypassing the Conntrack module and greatly improving the network performance in container scenarios. There are several ways to use Cilium 1.10 + Ali Cloud ENI mode on Ali Cloud. Please read the Cilium community installation documentation [1]. In order to enable users on the cloud to enjoy better network performance, The open source CNI plug-in Terway [2] developed by Ali Cloud has realized a better combination with Cilium. Terway supports container networks using aliyun’s elastic network card. In this way, the container network and vm network are on the same network plane, and there is no packet loss during communication between different hosts. The cluster scale is not limited by route entries and does not depend on distributed routing. Currently, the Terway IPvlan mode is deeply integrated with Cilium.

Using Terway IPvlan

Using Terway mode is very simple. In ali Cloud Container Service console, select network plug-in Terway in cluster creation and select IPvlan to enable it.

IPvlan + eBPF performance comparison:

Test environment:

  • 2 nodes ECS.G5NE 4xlarge models
  • Contrast test

Terway exclusive ENI (ipvS) Terway shared ENI IPvlan (EBPF) Terway shared ENI veth (ipvS) Flanne L Vxlan (ipvS)

Netperf performance comparison TCP_CRR

Test scenario: Test Pod communication using Netperf

In tests, it can be seen that ipvLAN-based Pod networks have low latency, and in tests with TCP_CRR performance is comparable to that of the exclusive ENI mode.

WRK + Nginx performance comparison

Test scenario: WRK is used to pressure nginx’s Service, and 100-byte pages are used to simulate common micro-service communication in the cluster.

​

The lower the number, the better the performance

The higher the number, the better the performance

Terway IPvlan mode compared to traditional Terway VETH PBR in wrK-Nginx short connection testing:

  • ClusterIP throughput increased by 277% and latency decreased by 50%.

conclusion

As Kubernetes has become the de facto standard for container scheduling, the cloud of choice on the enterprise. Container network, as the basic resource of application, has attracted more and more attention. By default, we provide high-performance Terway network plug-in [3] on Ali Cloud to help users make full use of cloud native network resources. Cilium, as an emerging container network solution in the community, has many outstanding features in terms of observability and security. The ENI mode of Ali Cloud added this time can help Cilium users make full use of network resources on Ali Cloud. We will also continue to work with the community to scale up high-performance cloud native networks. Installation documentation: docs. Cilium. IO/en/v1.10 / ge…

Terway: www.alibabacloud.com/help/zh/doc… Terway Web Plugin :_ help.aliyun.com/document_de… _