There is a memory leak in the online service. Record the process of checking the problem using jemalloc

  • Platforms: Linux
  • Languages: c + +

The principle of

Jemalloc is set to dump memory as it grows. Jemallc provides a tool to draw a diff graph of two files that have been dumped for a period of time

Install jemalloc

Wget https://github.com/jemalloc/jemalloc/archive/5.1.0.tar.gz tar ZXVF 5.1.0. Tar. Gz CD jemalloc - 5.1.0 /. / autogen. Sh /configure --prefix=/usr/local/jemalloc-5.1.0 --enable-prof make -j make install./configure --prefix=/usr/local/jemalloc-5.1.0 --enable-prof make -j make installCopy the code

Note that –enable-prof will do

Have the program link to the newly installed Jemalloc

Jemalloc. a static library file in the program directory, and then change the program CMakeList file

set(MODULE_SHARED_LIB
  jemalloc
  ...
)
target_link_libraries(
    ...
    -Wl,-Bdynamic
    ${MODULE_SHARED_LIB}
)
Copy the code

And then recompile

Run the file

Change the environment variable first,lg_prof_interval:26 indicates that dump is performed every increase of 2^26 bytes (64M)

export MALLOC_CONF="prof:true,lg_prof_interval:26"
Copy the code

Then run the bin file

LD_PRELOAD = / usr/local/jemalloc 5.1.0 / lib/libjemalloc. So. 2. / a.outCopy the code

When you have enough time, stop the program. After running for a while, there will be a lot of.heap files in the directory, running for 20 minutes, comparing the first two to the last two

Usr /local/jemalloc-5.1.0/bin/jeprof -- PDF a.out --base=jeprof.start.heap jeprof.end. Heap > a.pdfCopy the code

The following work is to open the PDF file and analyze the code. It is not convenient to put the company’s map. Here is a network map

thinking

  • The cause of this leak isErase () is used for the bare-pointer containerThis does not delete the data to which the pointer points
  • The graph drawn by Jemalloc can be used to determine where the leaked memory is allocated. It can be used to locate functions, but where exactly should be deleted is not, or do you have to find it yourself. Don’t look at the allocated function, but track the subsequent use of allocated data structures
  • Fix the memory leak as soon as possible. The time is too long to know which commit caused the memory leak

Afterword.

Today, I checked a memory leak, which was not actually a memory leak. There was a queue inside, which belonged to the consumption of multiple production orders. Jemalloc analysis found that it occupied a large amount of memory, and the reason was actually insufficient consumption capacity, resulting in queue accumulation.

  • Memory growth does not necessarily result from a memory leak caused by new and no delete in the code, but also from a lack of consumption power resulting in memory accumulation
  • If you find a data structure memory increase, print its size in the log