A: background

1. Tell a story

The day before yesterday, his brother came to see me again. I solved the problem of CPU exploding high last time. It seems that he trusted me very much.

In fact, the old brother technology is very good, since he can give me dump, it is really encountered a very difficult and difficult disease 😂, I have to do a good job of psychological preparation, communication down probably is the program’s memory will slowly expand, until self-destruction, the problem is such a problem, the next offer out of my watch tool windbg.

2. WINDBG analysis

1. Where is the leak?

As I’ve said in many previous articles, the first thing to do when you encounter a memory leak like this is to check whether it’s a managed heap or an unmanaged heap? If it’s the latter, most of the time you just throw up your hands and surrender because the water is so deep… Don’t look at the examples where AllocHGlobal allocates unmanaged memory and then uses! The reality is much more complicated than that…

Use it first! Address -summary looks at the commit memory of the current process.

0:00 0 >! address -summary --- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 345 7DFD 'CA3CA000 (125.991 TB) 98.43% <unknown> 37399 201' 54DBF000 (2.005 TB) 99.83% 1.57% HEAP 29887 0 'D179B000 (3.273) GB) 0.16%0.00% Image 1312 0 '0861B000 (134.105MB) 0.01%0.00% STACK 228 0' 06E40000 (110.250MB) 0.01%0.00% Other 10 0 '001D8000 (1.844 MB) 0.00% 0.00% TEB 76 0' 00098000 (608.000 KB) 0.00% 0.00% PEB 10 '00001000 (4.000 KB) 0.00% 0.00% --- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_MAPPED 352 200`00a40000 Mem_image1312 0 '0861b000 (242.00MB) Mem_image1313 0' 0861b000 (242.00MB) Mem_image1313 0 '0861b000 (242.00MB) 0.01% - 0.00% State Summary -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- RgnCount -- -- -- -- -- -- -- -- -- -- - the Total Size -- -- -- -- -- -- -- -- % % ofBusy ofTotal MEM_FREE 345 7DFD 'CA3CA000 (125.991 TB) 98.43% MEM_RESERVE 11805 200' 22AE8000 (2.001 TB) 99.60% 1.56% MEM_COMMIT 57108 2 '1313e000 (125.991 TB) 98.43% MEM_RESERVE 11805 200' 22AE8000 (2.001 TB) 99.60% 1.56% MEM_COMMIT 57108 2 '1313e000 ( 8.298 GB) 0.40% 0.01%

MEM_COMMIT = 8.2G. Then we look at the managed heap size, using! EEHEAP-GC command.

0:00 0 >! eeheap -gc Number of GC Heaps: 1 generation 0 starts at 0x0000027795928060 generation 1 starts at 0x000002779572F0D0 generation 2 starts at 0x000002763DCE1000 Total Size: Size: 0xcd28c510 (3442001168) bytes. ------------------------------ GC Heap Size: Size: 0xcd28c510 (3442001168) bytes.

The current GC heap Size= 3442001168/1024/1024/1024 =3.2G: 8.2G – 3.2G = 5G of memory is lost… Nima, a typical unmanaged memory leak, is really not open to mention which pot, this may really fail…

Look for unmanaged memory leaks

In addition to the GC heap, there is also a process called the loader heap, which contains a lot of things. There are high frequency heaps, low frequency heaps, Stub heaps, JIT heaps, and so on. It holds information about AppDomains, modules, method descriptors, method tables, eClass, etc. As a rule of thumb, This loader heap is a priority for unmanaged leaks, so if you want to see it, use it! EEHEAP-loader command.

0:00 0 >! eeheap -loader ... Module 00007ffe2b1b6ca8: Size: 0x0 (0) bytes. Module 00007ffe2b1b7e80: Size: 0x0 (0) bytes. Module 00007ffe2b1b9058: Size: 0x0 (0) bytes. Module 00007ffe2b1ba230: Size: 0x0 (0) bytes. Module 00007ffe2b1bb408: Size: 0x0 (0) bytes. Module 00007ffe2b1bc280: Size: 0x0 (0) bytes. Module 00007ffe2b1bd458: Size: 0x0 (0) bytes. Module 00007ffe2b1be630: Size: 0x0 (0) bytes. Module 00007ffe2b1bf808: Size: 0x0 (0) bytes. Module 00007ffe2b1f0a50: Size: 0x0 (0) bytes. Module 00007ffe2b1f1c28: Size: 0x0 (0) bytes. Module 00007ffe2b1f2aa0: Size: 0x0 (0) bytes. Total size: Size: 0x0 (0) bytes. -------------------------------------- Total LoaderHeap size: Size: 0xc0fb9000 (3237711872) bytes total, 0x5818000 (92372992) bytes wasted.

This command is not lost, it was a shock, the Windbg interface brush for several minutes before stopping… Two things can be learned from the output:

  • Total footprint of the loader heap:3237711872 /1024/1024/1024 = 3.01G
  • There are a lot of modules produced, tens of thousands by my estimate…

To satisfy my curiosity, I decided to write a little script to see how many modules there are.

There are 19W modules, no wonder it takes up more than 3 G. It feels not far from the truth. The next question is what are these modules and where do they come from?

3. Find the source of the module

Module nested: module-> assembly-> AppDomain, so looking up the AppDomain might give us more information, so we can use it! DumpDomain exports all the application domains of the current process. It’s a few more minutes and, hey… The screenshot is as follows:

You can see from the figure that there are a large number of assemblies of type Dynamic. What does that mean, you might be wondering? Yes, this is the assembly dynamically created by the code, which is up to 19W… The next question to be solved is: how are these assemblies created?

4. Export the module content

Regular readers will know how I exported the problem code from the module. That’s right, looking for the startAddress of the module. Here I picked one of the modules: 00007ffe2b1f2aa0.

2 > 2:2:15! dumpmodule 00007ffe2b1f2aa0 Name: Unknown Module Attributes: Reflection SupportsUpdateableMethods IsDynamic IsInMemory Assembly: 000002776c1d8470 BaseAddress: 0000000000000000 PEFile: 000002776C1D8BF0 ModuleId: 00007FFE2B1F2EB8 ModuleIndex: 00000000000177CF LoaderHeap: 0000000000000000 TypeDefToMethodTableMap: 00007FFE2B1EE8C0 TypeRefToMethodTableMap: 00007FFE2B1EE8E8 MethodDefToDescMap: 00007FFE2B1EE910 FieldDefToDescMap: 00007FFE2B1EE960 MemberRefToDescMap: 0000000000000000 FileReferencesMap: 00007FFE2B1EEA00 AssemblyReferencesMap: 00007FFE2B1EEA28

BaseAddress doesn’t have an address. That means you can’t export the module. It’s dynamically generated, so maybe the person writing the code doesn’t know what’s in the module. Is there really nothing that can be done? But as the saying goes well, the day has no road to the person 😅 at home, in! The big clue is that the dumpModule command has a mt (methodTable) parameter that shows what types are currently in the module.

| | 2:2:15 2 >! dumpmodule -mt 00007ffe2b1f2aa0 Name: Unknown Module Attributes: Reflection SupportsUpdateableMethods IsDynamic IsInMemory Assembly: 000002776c1d8470 Types defined in this module MT TypeDef Name ------------------------------------------------------------------------------ 00007ffe2b1f3168 0x02000002 <Unloaded Type> 00007ffe2b1f2f60 0x02000003 <Unloaded Type> Types referenced in this module MT TypeRef Name ------------------------------------------------------------------------------ 00007ffdb9f70af0 0x02000001 System.Object  00007ffdbaed3730 0x02000002 Castle.DynamicProxy.IProxyTargetAccessor 00007ffdbaec8f98 0x02000003 Castle.DynamicProxy.ProxyGenerationOptions 00007ffdbaec7fe8 0x02000004 Castle.DynamicProxy.IInterceptor

You can see that there are two types defined in the module, each with its method table address, and then exchange mt for md (method descriptor) to get the last module content.

This old man is the use of Castle to do an AOP function, should not be the correct use of AOP, resulting in the generation of 19W + dynamic assembly, no wonder the final memory to blow up… The root is finally found, how to modify next??

5. Modify Castle AOP problem code

After all, I really haven’t played Castle 😥, but as a matter of fact, if you look at Bing, there are some articles about Castle AOP causing memory leaks: Castle Windsor Interceptor Memory Leak, the solution is provided as follows:

Hurriedly throw this link to the old brother, I feel can only help him to here, the rest can only see the nature.

Three:

Is really the fate of the people, the elder brother with lightning on the potential to get it done, the same evening has completed the self-test line.



I quickly ask my brother is how to change 😁, I also put out the source code, as expected in accordance with the foreign advice to set ProxyGenerator static on the done… If you do not create an Assembly, look at the code before the change. The screenshot is as follows:

Having solved these two difficult problems, does it feel like I need to be given a small trophy? 😕 😕 😕

For more high-quality dry goods: see my GitHub:dotnetfly