First published on official blog: nebula-graph.com.cn/posts/debug…

With the Nebula Graph process as an example, you can debug the Nebula Graph process as if it were native without destroying the contents of the original container and installing a toolkit in it

demand

Vesoft inc/ Nebula Docker-compose deployments to the REPO are often used during development or testing because, in order to minimize the size of the Docker images per Nebula Graph service, So none of the usual tools for development are installed, not even the VIM editor.

This makes it difficult to locate the problem inside the container because you have to install a few toolkits at a time to do the rest of the work. There is another way to debug a process inside a container, without breaking the contents of the original container or installing any toolkits in it.

This technique is already very common in K8S environment, namely sidecar mode. The simple principle is to create a new container and have it share the same PID /network namespace as the container you are debugging. The debug container has everything you need installed in it, and the stage is left up to you.

demo

Here’s how to do it:

Let’s start by deploying a Nebula Graph cluster locally using the docker-compose approach described above, as shown in the README in the REPO. The deployment results are as follows:

$ docker-compose up -d
Creating network "nebula-docker-compose_nebula-net" with the default driver
Creating nebula-docker-compose_metad1_1 ... done
Creating nebula-docker-compose_metad2_1 ... done
Creating nebula-docker-compose_metad0_1 ... done
Creating nebula-docker-compose_storaged2_1 ... done
Creating nebula-docker-compose_storaged1_1 ... done
Creating nebula-docker-compose_storaged0_1 ... done
Creating nebula-docker-compose_graphd_1    ... done$ docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------------------------------ ----------------------------------------------- nebula-docker-compose_graphd_1 ./bin/nebula-graphd --flag ... Up (health: Starting 0.0.0.0) : 32907 - > 13000 / TCP, 0.0.0.0:32906 - > 13002 / TCP, 0.0.0.0:3699->3699/ TCP nebula Docker-compose_metad0_1 /bin/nebula Metad --flagf... Up (health: Starting 0.0.0.0) : 32898 - > 11000 / TCP, 0.0.0.0:32896 - > 11002 / TCP, 45500 / TCP, 45501/tcp nebula-docker-compose_metad1_1 ./bin/nebula-metad --flagf ... Up (health: Starting 0.0.0.0) : 32895 - > 11000 / TCP, 0.0.0.0:32894 - > 11002 / TCP, 45500 / TCP, 45501/tcp nebula-docker-compose_metad2_1 ./bin/nebula-metad --flagf ... Up (health: Starting 0.0.0.0) : 32899 - > 11000 / TCP, 0.0.0.0:32897 - > 11002 / TCP, 45500 / TCP, 45501/tcp nebula-docker-compose_storaged0_1 ./bin/nebula-storaged --fl ... Up (health: Starting 0.0.0.0) : 32901 - > 12000 / TCP, 0.0.0.0:32900 - > 12002 / TCP, 44500 / TCP, 44501/tcp nebula-docker-compose_storaged1_1 ./bin/nebula-storaged --fl ... Up (health: Starting 0.0.0.0) : 32903 - > 12000 / TCP, 0.0.0.0:32902 - > 12002 / TCP, 44500 / TCP, 44501/tcp nebula-docker-compose_storaged2_1 ./bin/nebula-storaged --fl ... Up (health: starting) 0.0.0.0:32905->12000/ TCP, 0.0.0.0:32904->12002/ TCP, 44500/ TCP, 44501/ TCPCopy the code

At this time we are divided into two scenarios to demonstrate, one is process space, one is network space. Instead of building it ourselves, we can take a packed version of the Docker Hub and use it as a demo. If it’s not enough, we can maintain a copy of the Nebula – Debug image and install all the debugging tools we want. Here we borrow nicolaka/ Netshoot from the community. Let’s first pull the image locally:

$ docker pull nicolaka/netshoot
$ docker imagesREPOSITORY TAG IMAGE ID CREATED SIZE vesoft/nebula-graphd nightly c67fe54665b7 36 hours ago 282MB vesoft/nebula-storaged  nightly 5c77dbcdc507 36 hours ago 288MB vesoft/nebula-console nightly f3256c99eda1 36 hours ago 249MB vesoft/nebula-metad nightly 5a78d3e3008f 36 hours ago 288MB nicolaka/netshoot latest 6d7e8891c980 2 months ago 352MBCopy the code

Let’s first see what it looks like to execute the mirror directly:

$ docker run --rm -ti nicolaka/netshoot bash
bash-5.0# ps
PID   USER     TIME  COMMAND
    1 root      0:00 bash
    8 root      0:00 ps
bash-5.0#
Copy the code

This container can’t see anything of the Nebula Graph Server process, so let’s add a few parameters to it and see:

$docker run -- rm-ti -- PID container:nebula- Docker-compose_metad0_1 --cap-add sys_admin Nicolaka /netshoot bash bash-5.0# ps
PID   USER     TIME  COMMAND
    1 root      0:03 ./bin/nebula-metad --flagfile=./etc/nebula-metad.conf --daemonize=false- meta_server_addrs = 172.28.1.1:45500172.28. 1.2:45500172.28. 1.3:45500 - local_ip = 172.28.1.1 - ws_ip = 172.28.1.1 --port=45500 --data_path=/data/meta --log_dir=/logs --v=15 --minloglevel=0 452 root 0:00 bash 459 root 0:00 ps bash-5.0# ls -al /proc/1/net/total 0 dr-xr-xr-x 6 root root 0 Sep 18 07:17 . dr-xr-xr-x 9 root root 0 Sep 18 06:55 .. -r--r--r-- 1 root root 0 Sep 18 07:18 anycast6 -r--r--r-- 1 root root 0 Sep 18 07:18 arp dr-xr-xr-x 2 root root 0 Sep 18  07:18 bonding -r--r--r-- 1 root root 0 Sep 18 07:18 dev ... -r--r--r-- 1 root root 0 Sep 18 07:18 sockstat -r--r--r-- 1 root root 0 Sep 18 07:18 sockstat6 -r--r--r-- 1 root root 0 Sep 18 07:18 softnet_stat dr-xr-xr-x 2 root root 0 Sep 18 07:18stat
-r--r--r--    1 root     root             0 Sep 18 07:18 tcp
-r--r--r--    1 root     root             0 Sep 18 07:18 tcp6
-r--r--r--    1 root     root             0 Sep 18 07:18 udp
-r--r--r--    1 root     root             0 Sep 18 07:18 udp6
-r--r--r--    1 root     root             0 Sep 18 07:18 udplite
-r--r--r--    1 root     root             0 Sep 18 07:18 udplite6
-r--r--r--    1 root     root             0 Sep 18 07:18 unix
-r--r--r--    1 root     root             0 Sep 18 07:18 xfrm_stat
Copy the code

This time it is a little different, we see the metad0 process, and its PID is still 1. It’s easy to see what you can do with this process, such as attach it directly to GDB. Since you don’t have the nebula Binary image handy, I’ll leave it to you to explore privately.

We’ve seen the pid space by specifying – pid container: < container_name | id > can share, so we then look at the situation of the network, after all, sometimes need to grab a bag, execute the following command:

Bash - 5.0 -# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
Copy the code

Nothing, a little bit different than expected, we have metad0 this process is not possible without a connection. To see the network space inside the container, add a few more parameters, and start the debug container as follows:

$ docker run --rm -ti --pid container:nebula-docker-compose_metad0_1 --network container:nebula-docker-compose_metad0_1 --cap-add sys_admin Nicolaka /netshoot bash bash-5.0# netstat -tulpnActive Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 00 172.28.1.1:11000 0.0.0.0:* LISTen-TCP 00 172.28.1.1:11002 0.0.0.0:* LISTen-TCP 00 0.0.0.0:45500 0.0.0.0:* Listen-tcp 0 0 0.0.0.00:45501 0.0.0.0:* listen-TCP 0 0 127.0.0.11:33249 0.0.0.0:* listen-udp 0 0 127.0.0.11:51929 0.0.0.0: * -Copy the code

Network container:nebula- Docker-compose_metad0_1 network container: Nebula -docker-compose_metad0_1 network container: Nebula -docker-compose_metad0_1

conclusion

Running another container and having it share a PID/Network namespace with the container you want to debug is the key to being able to debug locally. The community has even developed some gadgets based on the above ideas to make it easier to use:

  • Docker-debug

Recommended reading

  • Build the Nebula Graph source code using The Docker