How to debug container threads in containerization era

In containerization era, services run in dockers with low permissions. When the service is abnormal, it is difficult to execute debugging tools such as GDB and PERf to conduct runtime analysis in Docker. At this time, we need to log in to the host of the Node node where pod is located to conduct debugging with root permission.

Install nsenter

yum install util-linux
Copy the code

What is a nsenter

One of the most typical uses is to access the container’s network command space. Quite a few containers do not contain basic commands for lightweight purposes, such as IP address, ping, Telnet, SS, tcpdump, etc. This makes debugging the container network quite confusing: Only the docker inspect ContainerID command can be used to obtain the IP address of the container, and connectivity to other networks cannot be tested. You can then use the nsenter command to access only the network namespace of the container and debug the container network using the host’s commands. In addition, nsenter can also access the MNT, UTS, IPC, PID, user command space, as well as specify root and working directories.

The principle of

A namespace is the scope of the properties of some processes in Linux. Namespaces are used to isolate different processes. Linux continues to add namespaces, currently:

  • Mount: To mount a namespace so that a process has a separate mounted file system, starting from Linux 2.4.19
  • Ipc: The IPC namespace that enables a process to have a separate IPC, including message queues, shared memory, and semaphores, started in Linux 2.6.19
  • Uts: UTS namespace, which enables processes to have a separate hostname and domainname, started in Linux 2.6.19
  • Net: network command space, enabling processes to have a separate network stack, starting with Linux 2.6.24
  • Pid: The PID namespace, which enables processes to have a separate PID space
  • User: The user namespace, in which a process has a separate user space, starts with Linux 2.6.23 and ends with Linux 3.8
  • Cgroup: A cgroup namespace that allows processes to have a separate cgroup control group, starting with Linux 4.6

Each Linux process has a namespace, and you can see the file descriptor for the namespace in the /proc/pid/ns directory.

use

Nsenter [options] [program [arguments]] options: -t, --target PID: specifies the pid of the target process to be entered into the namespace -m, --mount[=file] : enters the mount command space. If file is specified, enter the command space of file -u, --uts[=file] : enter the uts command space. If file is specified, the command space of file is entered. -I, --ipc[=file] : the command space of ipc is entered. -n, --net[=file] : net command space. If file is specified, the command space of file -p, --pid[=file] : the command space of pid is entered. -u, --user[=file] : enters the user command space. -g, --setgid gid: sets the gid of the running program -s, --setuid uid: sets the UID of the running program -r, --root[=directory] : Set the root directory -w, --wd[=directory] : set the working directory if no program is given, $SHELL is executed by default.Copy the code

Find the PID of the container

# docker ps|grep application
7b5f22758bf7        registry.cn-ha./xxxx/
# docker inspect -f '{{.State.Pid}} {{.Id}}' $(docker ps -q)|grep 7b5f22758bf7
14734 7b5f22758bf7ecf002e8aa3a418aef0593a4048f07c4297fde08cd7004facaba
Copy the code

Access the namespace and use the debugging tool

sudo nsenter -t 14734 -m -p gdb -p 1
Copy the code