CNI benchmark: Cilium network performance analysis

Cilium. IO /blog/ 2020/0…

Thomas Graf is the co-founder of Cilium and also the CTO and co-founder of Isovalent, Cilium’s parent company. Prior to that, Thomas spent 15 years developing the Linux kernel in the areas of networking, security and eBPF. Note: this article has obtained the author’s translation authorization

Everybody is good! 👋

As more and more critical loads are migrated to Kubernetes, network performance benchmarking is becoming an important reference in choosing a Kubernetes network solution. In this article, we will explore Cilium’s performance characteristics based on the results of a number of benchmarking tests conducted over the past few weeks. At the request of our users, we will also present Calico’s test results for direct comparison.

In addition to presenting the resulting data from the tests, we will take a closer look at the topic of container network benchmarking and explore the following aspects:

• Throughput benchmark
• Whether container networks add overhead
• Break the rules: eBPF Host Routing
• Measurement delay: requests per second
• CPU flame map comparison of Cilium eBPF and Calico eBPF
• New connection processing rate
• Comparison between WireGuard and IPsec encryption costs
• The test environment

Summary of test results

Before looking at the benchmark and its data in detail, let’s present the aggregated test results. You can skip this section if you want to dive right into the details of testing and draw your own conclusions.

• EBPF is decisive: Cilium outperforms Calico’s eBPF Data Path in some respects, such as the latency observed in the TCP_RR and TCP_CRR benchmarks. In addition, the more important conclusion is that eBPF is significantly superior to IPtables. Both Cilium and Calico perform significantly better in configurations that allow eBPF to bypass Iptables than when iptables is not circumvented.

  After looking into the details, we found that Cilium and Calico do not use eBPF in exactly the same way. While some of the concepts are similar (and not surprising given the nature of open source), the CPU flame chart shows that Cilium takes advantage of additional context-switching savings. This may explain the difference in the TCP_RR and TCP_CRR test results.

  Overall, eBPF is clearly the best technology to address the cloud native requirements challenge, based on benchmarking results.

• Observability, NetworkPolicy, and Service: For this benchmark, we focused on the commonality of the two, namely the network. This allows us to directly compare performance differences brought about by node networks. However, observability, NetworkPolicy, and Service are also required in practical applications, where Cilium and Calico eBPF data paths differ greatly. Cilium supports some features that Calico eBPF data paths do not, but even standard features such as Kubernetes NetworkPolicy are implemented differently from Calico. If we put more effort into testing eBPF in these more advanced use cases, we might see a significant difference in performance in these areas. However, due to lack of space, this paper will not discuss this further, and a more in-depth study will be left to the next article.

• Comparing WireGuard and IPsec: Somewhat surprisingly, although WireGuard achieved a higher maximum throughput in our tests, IPsec was more CPU efficient at the same throughput. This is most likely due to the AES-Ni CPU instruction set. This instruction set supports unencrypting IPsec, but WireGuard does not benefit from it. When the AES-Ni instruction set is unavailable, the result is significantly reversed.

  The good news is that as of Cilium 1.10, Cilium supports not only IPsec but also WireGuard. You can choose one of them to use.

Throughput benchmark

Disclaimer:

Benchmarking is difficult. Test results depend heavily on the hardware environment in which the tests are run. Unless the results are collected on the same system, comparisons should not be made directly in absolute terms.

Let’s start with the most common and obvious TCP throughput benchmark, measuring the maximum data transfer rate between containers running on different nodes.

The figure above shows the maximum throughput that can be achieved for a single TCP connection, with the optimal configurations performing just over 40 Gbit/s. These results are based on Netperf’s TCP_STREAM test, which uses 100 Gbit/s network ports to ensure network adapters do not become bottlenecks. Since a single Netperf process is running to transfer data over a single TCP connection, most of the network processing is done by a single CPU core. This means that the maximum throughput above is limited by the available CPU resources of a single core, so you can show how much throughput each configuration can achieve when the CPU becomes the bottleneck. Later in this article, the tests will be expanded to use more CPU cores to eliminate CPU resource limitations.

Throughput achieved using high-performance eBPF is even slightly higher than node-to-node throughput. This is very surprising. It is generally accepted that container networks incur additional overhead compared to node-to-node networks. Let’s put that puzzle on the back burner for the moment and analyze it after further study.

CPU resources required for the 100 Gbit/s transfer rate

The results of the TCP_STREAM benchmark have hinted at which configurations are most efficient for achieving high transfer rates, but let’s take a look at the overall CPU consumption of the system when running the benchmark.

The figure above shows the CPU utilization required for the entire system to reach 100 Gbit/s throughput. Note that this is different from the CPU consumption corresponding to throughput in the previous figure. In the figure above, all CPU usage has been converted to a stable transfer rate of 100 Gbit/s for direct comparison. The shorter the bar chart, the more efficient the configuration is at 100 Gbit/s.

Note: TCP stream performance is often limited by the receiver, because the sender can use TSO large packets at the same time. This can be observed in the increased CPU overhead on the server side in the above tests.

The significance of TCP throughput benchmarks

While most users are unlikely to encounter these throughput levels on a regular basis, such benchmarks are important for specific types of applications:

• AI/ML applications that require access to large amounts of data
• Data upload/download service (backup service, VM image, container image service, etc.)
• Streaming services, especially 4K+ resolution streaming

Later in this article, we will continue to delve into measurement latency: requests per second and new connection processing rates to better illustrate the performance characteristics of typical microservice workloads.

Whether container networks add overhead

In the analysis of the first benchmark, we mentioned that container networks incur some additional overhead compared to node networks. Why is that? Let’s compare these two network models from an architectural perspective.

The figure above shows that the container network also needs to perform all the node-to-node network processing, and that these processes occur in the container’s network namespace (dark blue).

Since node network processing also needs to take place inside the container network namespace, any work outside the container network namespace is essentially additional overhead. The figure above shows the network path for Linux routing when using Veth devices. If you use a Linux bridge or OVS, the network model may be slightly different, but the basic overhead points are the same.

Break the rules: eBPF host-routing

In the benchmark above, you may have wondered the difference between the Cilium eBPF and Cilium eBPF Legacy Host Routing configurations and why the native Cilium eBPF data path is so much faster than the Host route. The native Cilium eBPF data path is an optimized data path called eBPF host Routing, as shown below:

EBPF host routing allows bypassing all iptables and upper network stacks in the host namespace, as well as some context switching across Veth pairs to save resource overhead. Network packets are captured as soon as they reach the network interface device and are delivered directly to the network namespace of the Kubernetes Pod. At the traffic outlet side, the packets also pass through the Veth pair, are captured by eBPF and directly transmitted to the external network interface. EBPF queries the routing table directly, so this optimization is completely transparent and compatible with all the services that provide route allocation running on the system. For details on how to enable this feature, see eBPF host Routing in the Tuning guide.

Calico eBPF is using some similar circumvention for Iptables, but it’s not exactly the same as Cilium’s principle, as described later in this article. However, the test results proved that bypassing slow kernel subsystems, such as Iptables, can provide significant performance gains.

Line Rate approaching 100 Gbit/s

In the previous article, we analyzed the benchmark results involving only one CPU core. Next we will let go of single-core and parallelize TCP streams to run multiple Netperf processes.

Note: Since the hardware has 32 threads, we deliberately chose 32 processes to ensure that the load is evenly distributed on the system.

The figure above does not provide much information except to show that all of the tested configurations can achieve line rates close to 100 Gbit/s if sufficient CPU resources are devoted. However, we can still see differences in efficiency in terms of CPU resources.

Note that the CPU usage in the figure above covers all CPU consumption, including the consumption of running Netperf processes, as well as the CPU resources required by the workload to perform network I/O. However, it does not include the CPU consumption associated with any business logic that the application typically needs to perform.

Measurement delay: requests per second

Requests per second is almost the opposite of throughput metrics. It measures the transmission rate of sequential single-byte round-trips over a single TCP persistent connection. This benchmark demonstrates the efficiency of network packet processing. The lower the latency of individual network packets, the more requests can be processed per second. Co-optimization of throughput and latency is often a trade-off. A larger buffer is ideal for maximum throughput, but large buffers can lead to increased latency. This phenomenon is known as buffer bloat. Cilium provides a feature called Bandwidth Manager that automatically configures fair queues, enables Pod rate limits based on earliest dispatch times, and optimizes TCP stack Settings for server workloads to achieve an optimal balance between throughput and latency.

This benchmark is often overlooked, but it is often much more important to users than you might think, because it emulates a very common microservice usage pattern: sending requests and responses between services using persistent HTTP or gRPC connections.

The following figure shows the TCP_RR test performance of a single Netperf process with different configurations:

Configurations that performed better in this test also achieved lower average latency. However, this is not enough to draw conclusions about P95 or P99 delays. We will explore these issues in a future blog post.

We further tested running 32 parallel Netperf processes to take advantage of all available CPU cores. As you can see, performance has improved for all configurations. However, unlike the throughput test, putting more CPU resources into this test does not compensate for the lack of efficiency, because the maximum processing rate is limited by latency rather than available CPU resources. Even if network bandwidth became a bottleneck, we would see the same number of requests per second.

Overall, the results were very encouraging and Cilium was able to achieve a processing rate of nearly 1,000,000 requests per second via eBPF host routing on our test system.

CPU flame map comparison of Cilium eBPF and Calico eBPF

Overall, Cilium eBPF and Calico eBPF perform roughly the same. Is this because they use the same data path? And it isn’t. There is no predefined eBPF data path. EBPF is a programming language and runtime engine that allows you to build data path features and many other features. Cilium and Calico eBPF data paths are very different. In fact, Cilium offers many features that Calico eBPF does not support. But even when it comes to interacting with the Linux networking stack, there are significant differences. Further analysis can be made by looking at the CPU flame maps of both.

Cilium eBPF (Receiving path)

Cilium’s eBPF host routing provides a good context-free way to transfer data (from the network card to the application’s socket). This is why in the flame diagram above the entire receiver path fits nicely into a flame diagram. The flame chart also shows processing blocks for eBPF, TCP/IP, and sockets.

Calico eBPF (Receive path)

The Calico eBPF receiver looks different. Although the same eBPF processing block executes the eBPF program, the Calico eBPF receive path passes through an additional Veth that is not required at the Cilium eBPF data path receiver.

The processing in the figure above is still performed in the context of the host. The flame chart below shows the work in the Pod restored by process_backlog. While this works the same as in the Cilium scenario (both TCP/IP+ socket data transfer), additional context switching is required because it crosses Veth.

If you wish to do further research on your own, you can open the interactive flame map SVG file by clicking on the following link to see the details:

• Cilium eBPF SVG flame diagram – sending end
• Cilium eBPF SVG flame diagram – Receiver
• Calico eBPF SVG flame diagram – sender
• Calico eBPF SVG flame diagram – Receiver