The CNI profile

The configuration of a container network is a complex process. There are a variety of solutions to meet various needs. For example, flannel, Calico, Kube-OVN, Weave, etc. At the same time, container platforms/runtimes are also diverse, such as Kubernetes, Openshift, RKT, etc. If every container platform had to be aligned with every network solution, it would be a huge and repetitive task. Of course, smart programmers would never allow this to happen. To solve this problem, we need an abstract interface layer that decouples the container network configuration scheme from the container platform scheme.

The Container Network Interface (CNI) is one such Interface layer. It defines a set of Interface standards and provides specification documents and some standard implementations. The container platform using CNI specification to set up the container network does not need to pay attention to the details of the network setting, just need to call the CNI interface according to the CNI specification to realize the network setting.

CNI was originally created by CoreOS for the RKT container engine and has evolved to become a de facto standard. At present, most container platforms adopt CNI standard (RKT, Kubernetes, OpenShift, etc.). This article is based on the latest CNI release v0.4.0.

It is worth noting that Docker did not adopt THE CNI standard, but synchronously developed the CNM (Container Networking Model) standard at the beginning of THE establishment of CNI. However, due to technical and non-technical reasons, CNM model has not been widely used.

How does CNI work

CNI interface does not refer to HTTP, gRPC interface, CNI interface refers to the call to executable program (exec). These executable programs are called CNI plug-ins. Take K8S as an example. The default CNI plug-in path of K8S node is /opt/ CNI /bin.

$ ls /opt/cni/bin/ bandwidth bridge dhcp firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr  static tuning vlanCopy the code

The working process of CNI is roughly shown in the figure below:

CNI describes the network configuration through the CONFIGURATION file in JSON format. When the container network needs to be set, the container runtime executes the CNI plug-in, transmits the configuration file information through the standard input (STDIN) of the CNI plug-in, and receives the execution result of the plug-in through the standard output (STDOUT). Libcni in the figure is a GO package provided by CNI, which encapsulates some standard operations conforming to CNI specifications and facilitates the container runtime and network plug-in to connect to CNI standards.

As an intuitive example, if we were to call the Bridge plug-in to connect a container to a host bridge, the command would look something like this:

# CNI_COMMAND=ADD as the name implies.
# XXX=XXX Other parameters are defined below.
# < config.json indicates passing the configuration file from standard input
CNI_COMMAND=ADD XXX=XXX ./bridge < config.json
Copy the code

Plug-in into the reference

The container runtime passes parameters to the plug-in by setting environment variables and by passing configuration files from standard input.

The environment variable

  • CNI_COMMAND: Defines the desired action, which can be ADD, DEL, CHECK, or VERSION.
  • CNI_CONTAINERID: container ID, unique identifier of the container managed by the container runtime.
  • CNI_NETNS: Path to the container network namespace. (like/run/netns/[nsname]).
  • CNI_IFNAME: Indicates the name of the network interface to be created, for example, eth0.
  • CNI_ARGS: Additional arguments passed to the run-time call in the form of semicolon-separated key-value pairs, for exampleFOO=BAR; ABC=123 
  • CNI_PATH: path to the CNI plug-in executable, for example/opt/cni/bin.

The configuration file

Example file:

{
  "cniVersion": "0.4.0".// Represents the version of the CNI standard that you want your plug-in to follow.
  "name": "dbnet".// Indicates the network name. This name does not refer to the network interface name, but is a representation for CNI management. It should be globally unique on the current host (or other management domain).
  "type": "bridge".// Plug-in type
  "bridge": "cni0".// Bridge plugin parameters, specifying the bridge name.
  "ipam": { // IP Allocation Management: Allocates Management IP addresses.
    "type": "host-local".// Type of ipAM plug-in.
    // Parameters defined by ipam
    "subnet": "10.1.0.0/16"."gateway": "10.1.0.1"}}Copy the code

Common definition section

The configuration file is divided into a common section and a plug-in definition section. Common parts are defined in CNI projects using the structure NetworkConfig:

type NetworkConfig struct {
   Network *types.NetConf
   Bytes   []byte}...// NetConf describes a network.
type NetConf struct {
   CNIVersion string `json:"cniVersion,omitempty"`

   Name         string          `json:"name,omitempty"`
   Type         string          `json:"type,omitempty"`
   Capabilities map[string]bool `json:"capabilities,omitempty"`
   IPAM         IPAM            `json:"ipam,omitempty"`
   DNS          DNS             `json:"dns"`

   RawPrevResult map[string]interface{} `json:"prevResult,omitempty"`
   PrevResult    Result                 `json:"-"`
}
Copy the code
  • cniVersionRepresents the version of the CNI standard that you want your plug-in to follow.
  • nameIndicates the network name. This name does not refer to the network interface name, but is a representation for CNI management. It should be globally unique on the current host (or other management domain).
  • typeRepresents the name of the plug-in, which is the name of the plug-in’s corresponding executable file.
  • bridgeThis parameter belongs tobridgeParameter to the plug-in that specifies the name of the host bridge.
  • ipamRepresents the configuration of the IP address assignment plug-in,ipam.typeIs the plug-in type of IPAM.

For more information, please refer to the official documentation.

Plug-in definition section

As mentioned above, the configuration file is ultimately passed to the specific CNI plug-in, so the plug-in definition part is the “whole” of the configuration file. The common part definition is just a convenience for plug-ins to embed it in their own configuration file definition structure. For example, the Bridge plug-in:

type NetConf struct {
	types.NetConf // <-- embed the public part
        // The bottom is the plug-in definition section
	BrName       string `json:"bridge"`
	IsGW         bool   `json:"isGateway"`
	IsDefaultGW  bool   `json:"isDefaultGateway"`
	ForceAddress bool   `json:"forceAddress"`
	IPMasq       bool   `json:"ipMasq"`
	MTU          int    `json:"mtu"`
	HairpinMode  bool   `json:"hairpinMode"`
	PromiscMode  bool   `json:"promiscMode"`
	Vlan         int    `json:"vlan"`

	Args struct {
		Cni BridgeArgs `json:"cni,omitempty"`
	} `json:"args,omitempty"`
	RuntimeConfig struct {
		Mac string `json:"mac,omitempty"`
	} `json:"runtimeConfig,omitempty"`

	mac string
}
Copy the code

For the configuration files of each plug-in, see the official documents.

Plug-in action types

The CNI plug-in has only four operation types: ADD, DEL, CHECK, and VERSION. The plug-in caller uses the environment variable CNI_COMMAND to specify the operations that need to be performed.

ADD

The ADD operation is responsible for adding containers to the network or making changes to existing network Settings. Specifically, the ADD operation either:

  • Create a network interface for the network namespace of the container, or
  • Modify the specified network interface in the network namespace of the container

For example, use ADD to connect the container network interface to the host network bridge.

The network interface name is specified by CNI_IFNAME and the network namespace is specified by CNI_NETNS.

DEL

The DEL operation is responsible for removing the container from the network or cancelling the corresponding modification, which can be interpreted as the reverse operation of ADD. Specifically, the DEL operation either:

  • Delete a network interface for the network namespace of the container, or
  • undoADDOperation modification

For example, delete the container network interface from the host bridge using DEL.

The network interface name is specified by CNI_IFNAME and the network namespace is specified by CNI_NETNS.

CHECK

The CHECK operation is added to V0.4.0 and is used to CHECK whether the network Settings are as expected. When CHECK returns an error (a non-zero status code is returned), the container runtime can Kill the container and restart to get a correct network configuration.

VERSION

The VERSION operation is used to view the VERSION information supported by plug-ins.

$ CNI_COMMAND=VERSION /opt/cni/bin/bridge
{"cniVersion":"0.4.0"."supportedVersions": ["0.1.0 from"."0.2.0"."0.3.0"."0.3.1"."0.4.0"]}
Copy the code

Chain calls

For example, the BRIDGE plug-in is responsible for configuring the network bridge, the Firewall plug-in for configuring the firewall, and the PortMap plug-in for configuring the port mapping. Therefore, when the network setup is complex, it is often necessary to call multiple plug-ins. CNI supports chained invocation of plug-ins, where multiple plug-ins can be combined and invoked in sequence. For example, call the Bridge plug-in to set the container IP address, connect the container network adapter to the host network bridge, and then call the PortMap plug-in to map container ports. The container runtime can do this by setting the plugins array in the configuration file:

{
  "cniVersion": "0.4.0"."name": "dbnet"."plugins": [{"type": "bridge".// type (plugin) specific
      "bridge": "cni0"
      },
      "ipam": {
        "type": "host-local".// ipam specific
        "subnet": "10.1.0.0/16"."gateway": "10.1.0.1"}}, {"type": "tuning"."sysctl": {
        "net.core.somaxconn": "500"}}}]Copy the code

A careful reader will notice that the plugins field does not appear in the configuration file structure described above. Indeed, CNI uses another structure, NetworkConfigList, to hold the configuration of chained calls:

type NetworkConfigList struct {
   Name         string
   CNIVersion   string
   DisableCheck bool
   Plugins      []*NetworkConfig 
   Bytes        []byte
}
Copy the code

But the CNI plug-in is unaware of this configuration type. In fact, when the CNI plug-in is called, NetworkConfigList needs to be converted into the configuration file format of the corresponding plug-in, which is passed to the CNI plug-in through standard input (STDIN). For example, in the above example, the Bridge plug-in is actually called first with the following configuration file:

{
  "cniVersion": "0.4.0"."name": "dbnet"."type": "bridge"."bridge": "cni0"."ipam": {
    "type": "host-local"."subnet": "10.1.0.0/16"."gateway": "10.1.0.1"}}Copy the code

Then use the following configuration file to invoke the Tuning plug-in:

{
  "cniVersion": "0.4.0"."name": "dbnet"."type": "tuning"."sysctl": {
    "net.core.somaxconn": "500"
  },
  "prevResult": { // The result of calling the Bridge plug-in. }}Copy the code

Note that when a plug-in makes a chain call, not only does it need to format NetworkConfigList, but it also needs to add the result of the previous plug-in to the configuration file (via the prevResult field), which is a tedious and repetitive task. Fortunately, libcni is already wrapped up for us, so the container runtime doesn’t need to worry about converting the configuration file or filling in the results of the last plug-in, just calling libcni’s methods.

The sample

Next, I’ll demonstrate how to use the CNI plug-in to set up the network for the Docker container.

Download the CNI plug-in

For convenience, let’s download the executable directly:

Wget https://github.com/containernetworking/plugins/releases/download/v0.9.1/cni-plugins-linux-amd64-v0.9.1.tgz mkdir -p ~/ cnI /bin tar ZXVF cni-plugins-linux-amd64-v0.9.1. TGZ -c./ cnI /bin chmod +x ~/cni/bin/* ls ~/cni/bin/ bandwidth bridge dhcp firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr static tuning vlan vrfzCopy the code

If you are running on a K8S node, the CNI plugin is already installed on the node, so you do not need to download it, but be careful to change the subsequent CNI_PATH to /opt/ CNI /bin.

Example 1 — Invoke a single plug-in

In example 1, we call the CNI plug-in directly, set up the eth0 interface for the container, assign it an IP address, and connect to the host bridge mynet0.

It’s the same network mode that Docker uses by default, but we’ve changed Docker0 to mynet0.

Start the container

Although Docker does not use the CNI specification, it is possible to make Docker not set up container networks by specifying –net= None. Take nginx mirroring as an example:

contid=$(docker run -d --net=none --name nginx nginx) ID # container
pid=$(docker inspect -f '{{ .State.Pid }}' $contid) # container process ID
netnspath=/proc/$pid/ns/net # namespace path
Copy the code

When starting the container, we need to record the container ID and namespace path for subsequent passing to the CNI plug-in. After the container is started, you can see that the container has no other network Settings except the LO card:

nsenter -t $pid -n ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
Copy the code

Nsenter is short for Namespace Enter, which, as the name suggests, is a tool for executing commands in a namespace. -t indicates the process ID, and -n indicates the network namespace of the process.

Add container network interfaces and connect host Bridges

Next we use the Bridge plug-in to create the network interface for the container and connect to the host bridge. Create a bridge.json configuration file with the following contents:

{
    "cniVersion": "0.4.0"."name": "mynet"."type": "bridge"."bridge": "mynet0"."isDefaultGateway": true."forceAddress": false."ipMasq": true."hairpinMode": true."ipam": {
        "type": "host-local"."subnet": "10.10.0.0/16"}}Copy the code

To invoke the Bridge plugin ADD operation:

CNI_COMMAND=ADD CNI_CONTAINERID=$contid CNI_NETNS=$netnspath CNI_IFNAME=eth0 CNI_PATH=~/cni/bin ~/cni/bin/bridge < bridge.json
Copy the code

If the call succeeds, it returns a similar value:

{
    "cniVersion": "0.4.0"."interfaces": [...]. ."ips": [{"version": "4"."interface": 2."address": "10.10.0.2/16".// The IP address assigned to the container
            "gateway": "10.10.0.1"}]."routes": [...]. ."dns": {}}Copy the code

Look again at the container network Settings:

nsenter -t $pid -n ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host LO valid_lft forever preferred_lft forever 5: eth0@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether c2:8f:ea:1b:7f:85 brd Ff :ff:ff:ff:ff:ff :ff link- netnSID 0 inet 10.10.0.2/16 BRD 10.10.255.255 scope global eth0 valid_lft forever preferred_lft foreverCopy the code

You can see that the eth0 network interface has been added to the container and assigned an IP address under the subnet set by the IPAM plug-in. The host-local ipAM plug-in saves the assigned IP address to a file to avoid IP conflict. The default path is /var/lib/cni/network/$NETWORK_NAME.

The ls/var/lib/the cni/networks/mynet / 10.10.0.2 last_reserved_ip. 0 the lockCopy the code

Access authentication from the host

Since mynet0 is the bridge we added and no route has been set, we need to add a route for the network segment where the container resides before verification:

IP route add 10.10.0.0/16 dev mynet0 SRC 10.10.0.1# add routeThe curl -i 10.10.0.2Change IP to the actual IP address assigned to the container
HTTP/1.1 200 OK
....
Copy the code

Delete the container network interface

The deleted call input parameter is the same as the added one, except that CNI_COMMAND is replaced with DEL:

CNI_COMMAND=DEL CNI_CONTAINERID=$contid CNI_NETNS=$netnspath CNI_IFNAME=eth0 CNI_PATH=~/cni/bin ~/cni/bin/bridge < bridge.json
Copy the code

Note that the delete command above does not clean up the host’s Mynet0 bridge. To delete the host bridge, run the IP link delete mynet0 type bridge command.

Example 2 — Chain call

In example 2, we will build on example 1 by adding a portmap to the container using the portmap plug-in.

usecnitooltool

From the previous introduction, we learned that during a chain call, the caller needs to transform the configuration file and insert the results of the previous plug-in into the configuration file of the current plug-in. This can be tedious work, and libcni encapsulates these processes. In Example 2, we will use the command line tool Cnitool based on libcni to simplify these operations.

Example 2 will reuse containers from Example 1, so when you start example 2, make sure you remove the network interface from Example 1.

Install Cnitool by source compilation or by going install:

go install github.com/containernetworking/cni/cnitool@latest
Copy the code

The configuration file

Libcni will read the configuration file with the. Conflist suffix, and we will create portmap.conflist in the current directory:

{
  "cniVersion": "0.4.0"."name": "portmap"."plugins": [{"type": "bridge"."bridge": "mynet0"."isDefaultGateway": true."forceAddress": false."ipMasq": true."hairpinMode": true."ipam": {
        "type": "host-local"."subnet": "10.10.0.0/16"."gateway": "10.10.0.1"}}, {"type": "portmap"."runtimeConfig": {
        "portMappings": [{"hostPort": 8080."containerPort": 80."protocol": "tcp"}]}}Copy the code

Two CNI plug-ins, Bridge and PortMap, are defined from the above configuration file. According to the configuration file above, Cnitool will first add a network interface for the container and connect it to the host mynet0 bridge (as in example 1), and then call the PortMap plug-in to map port 80 of the container to port 8080 of the host, just like docker run -p 8080:80 XXX.

Setting a Container Network

Using Cnitool we also need to set two environment variables:

  • NETCONFPATH: Specify the configuration file (*.conflist). The default path is/etc/cni/net.d
  • CNI_PATH: Specifies the directory where the CNI plug-in is stored.

Use the cnitool add command to set up the network for the container:

CNI_PATH=~/cni/bin NETCONFPATH=.  cnitool add portmap $netnspath
Copy the code

After the configuration is successful, you can access the container’s nginx service by accessing port 8080 on the host.

Deleting Network Configurations

Delete the container network using the cnitool del command:

CNI_PATH=~/cni/bin NETCONFPATH=.  cnitool del portmap $netnspath
Copy the code

Note that the delete command above does not clean up the host’s Mynet0 bridge. To delete the host bridge, run the IP link delete mynet0 type bridge command.

conclusion

So far, the working principle of CNI has been basically clear to us. The working principle of CNI can be summarized as follows:

  • Define the network configuration through a JSON configuration file;
  • Perform configuration on the container network by calling the executable (CNI plug-in);
  • The combination of multiple plug-ins is supported through chain calls.

CNI not only defines the interface specification, but also provides some built-in standards implementations and “glue layers” such as libcni, which greatly reduce the barrier to access between container runtime and network plug-ins.

reference

  • CNI V0.4.0 specification document
  • CNI Master branch specification document
  • CNI built-in plug-in documentation
  • Cnitool document
  • Why does Kubernetes not use the CNM model
  • Introduction to CNI
  • CNI deep dive