A: background
1. Tell a story
The day before yesterday, his brother came to see me again. Last time, I solved the problem of CPU explosion, and it seems that I have a lot of trust in me. This time, another program encountered memory leakage, hoping that I could help diagnose it.
In fact, the old brother is still very good technology, since he can dump me, it is really encountered a very difficult difficult disease 😂😂😂, I have to prepare for 😬, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port, port.
Two: WinDBG analysis
1. Where is the leak?
As I’ve said in many previous articles, the first thing you need to know when you get a memory leak like this is whether it’s a managed heap or an unmanaged heap. If it’s the latter, most of the time you just throw up your hands because the water is too deep… Don’t look at those cases. Use AllocHGlobal to allocate unmanaged memory and then use! Heap went to the pediatrician, and the reality was more complicated than that…
Next first! Address-summary takes a look at the committed memory of the current process.
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 345 7dfd`ca3ca000 ( 125.991 TB) 98.43%
<unknown> 37399 201`54dbf000 ( 2.005 TB) 99.83% 1.57%
Heap 29887 0`d179b000 ( 3.273 GB) 0.16% 0.00%
Image 1312 0`0861b000 ( 134.105 MB) 0.01% 0.00%
Stack 228 0`06e40000 ( 110.250 MB) 0.01% 0.00%
Other 10 0`001d8000 ( 1.844 MB) 0.00% 0.00%
TEB 76 0`00098000 ( 608.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00% -- -- --Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 352 200`00a40000 ( 2.000 TBMEM_PRIVATE 67249 2 '2cbcb000 ( 8.699 GB) 0.42% 0.01% MEM_IMAGE 1312 0 '0861b000 ( 134.105 MB) 0.01% 0.00% - State Summary -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- RgnCount -- -- -- -- -- -- -- -- -- -- - the Total Size -- -- -- -- -- -- -- -- % % ofBusy ofTotal MEM_FREE 345 7dfd`ca3ca000 ( 125.991 TB) 98.43% MEM_RESERVE 11805 200 '22ae8000 ( 2.001 TBMEM_COMMIT 57108 2 '1313e000 ( 8.298 GB) 0.40% 0.01%
Copy the codeMEM_COMMIT = 8.2g, then we look at the managed heap size, use! Eeheap-gc command.
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000027795928060
generation 1 starts at 0x000002779572F0D0
generation 2 starts at 0x000002763DCE1000
Total Size: Size: 0xcd28c510 (3442001168) bytes.
------------------------------
GC Heap Size: Size: 0xcd28c510 (3442001168) bytes.
Copy the codeThe current GC heap Size= 3442001168/1024/1024/1024 = 3.2g. Typical unmanaged memory leak. It’s a real problem.
2. Look for unmanaged memory leaks
In addition to the GC heap, there is also a loader heap, which contains a lot of stuff. There is a high frequency heap, a low frequency heap, a Stub heap, a JIT heap, and so on. It contains information about appDomains, modules, method descriptors, method tables, and EEClass. This loader heap is a priority place to inspect unmanaged leaks, to view, use! Eeheap-loader command.
0:000> !eeheap -loader
...
Module 00007ffe2b1b6ca8: Size: 0x0 (0) bytes.
Module 00007ffe2b1b7e80: Size: 0x0 (0) bytes.
Module 00007ffe2b1b9058: Size: 0x0 (0) bytes.
Module 00007ffe2b1ba230: Size: 0x0 (0) bytes.
Module 00007ffe2b1bb408: Size: 0x0 (0) bytes.
Module 00007ffe2b1bc280: Size: 0x0 (0) bytes.
Module 00007ffe2b1bd458: Size: 0x0 (0) bytes.
Module 00007ffe2b1be630: Size: 0x0 (0) bytes.
Module 00007ffe2b1bf808: Size: 0x0 (0) bytes.
Module 00007ffe2b1f0a50: Size: 0x0 (0) bytes.
Module 00007ffe2b1f1c28: Size: 0x0 (0) bytes.
Module 00007ffe2b1f2aa0: Size: 0x0 (0) bytes.
Total size: Size: 0x0 (0) bytes.
--------------------------------------
Total LoaderHeap size: Size: 0xc0fb9000 (3237711872) bytes total, 0x5818000 (92372992) bytes wasted.
Copy the codeThis command does not lose good, a lose startled, WinDBG interface brush a few minutes to stop… You can get two things from the output:
-
Total loader heap usage: 3237711872/1024/1024/1024 = 3.01 GB
-
There are a lot of modules, tens of thousands, I guess…
To satisfy my curiosity, I decided to write a small script to see how many modules there are.
No wonder it takes up more than 3 G’s. It feels not far from the truth. The next question is what are these modules and where do they come from??
3. Find the source of the Module
Module -> Assembly -> Appdomain -> Appdomain -> Appdomain DumpDomain exports all application domains for the current process, another few minutes of brushing, alas… Screenshot below:
As you can see from the diagram, there are a large number of assemblies of type Dynamic. What does this mean, you must ask? Yes, this is the code dynamically created assembly, surprisingly high 19W… The next question to solve is: How are these assemblies created?
4. Export module contents
Regular readers should know how I exported the problem code from the Module. Yes, look for the StartAddress of the Module. Here I’ll pick one of the modules: 00007ffe2b1f2AA0.
2:2:152> !dumpmodule 00007ffe2b1f2aa0
Name: Unknown Module
Attributes: Reflection SupportsUpdateableMethods IsDynamic IsInMemory
Assembly: 000002776c1d8470
BaseAddress: 0000000000000000
PEFile: 000002776C1D8BF0
ModuleId: 00007FFE2B1F2EB8
ModuleIndex: 00000000000177CF
LoaderHeap: 0000000000000000
TypeDefToMethodTableMap: 00007FFE2B1EE8C0
TypeRefToMethodTableMap: 00007FFE2B1EE8E8
MethodDefToDescMap: 00007FFE2B1EE910
FieldDefToDescMap: 00007FFE2B1EE960
MemberRefToDescMap: 0000000000000000
FileReferencesMap: 00007FFE2B1EEA00
AssemblyReferencesMap: 00007FFE2B1EEA28
Copy the codeBaseAddress does not have an address. This means that you cannot export the module. After all, it is dynamically generated, and people who write code may not know what the module is. Is there really nothing that can be done? But as the saying goes, heaven never shuts one door but 😅😅😅, in! The dumpModule command has an MT (methodTable) parameter that shows what types are present in the current Module, which is a big clue.
||2:2:152> !dumpmodule -mt 00007ffe2b1f2aa0
Name: Unknown Module
Attributes: Reflection SupportsUpdateableMethods IsDynamic IsInMemory
Assembly: 000002776c1d8470
Types defined in this module
MT TypeDef Name
------------------------------------------------------------------------------
00007ffe2b1f3168 0x02000002 <Unloaded Type>
00007ffe2b1f2f60 0x02000003 <Unloaded Type>
Types referenced in this module
MT TypeRef Name
------------------------------------------------------------------------------
00007ffdb9f70af0 0x02000001 System.Object
00007ffdbaed3730 0x02000002 Castle.DynamicProxy.IProxyTargetAccessor
00007ffdbaec8f98 0x02000003 Castle.DynamicProxy.ProxyGenerationOptions
00007ffdbaec7fe8 0x02000004 Castle.DynamicProxy.IInterceptor
Copy the codeYou can see that two types are defined in the Module, each with its method table address. Then mt is exchanged for md (method descriptor) to get the final module contents.
AOP is not used correctly, resulting in the creation of 19W + dynamic assembly, which will eventually run out of memory. Finally found the root, how to modify next??
5. Modify Castle AOP problem code
I have never played Castle 😥😥😥, but as usual, if you go to Bing, you can find some Castle AOP that causes memory leaks. Castle Windsor Interceptor Memory Leak provides a solution to this problem.
Throw this link to brother quickly, I feel can only help him here, the rest can only see fate.
Three:
Really is the fate of people, the elder brother with the potential of lightning to get done, that night has completed the self-test line.
I hurriedly cross-examine brother is how change 😁 😁 😁, brother also released at the source, sure enough, in accordance with the foreigner suggestion will ProxyGenerator have to set the static… Otherwise, new one assembly at a time. Take a look at the code before the change, as shown below:
With these two tough questions out of the way, do I feel like I’m getting a little trophy? 😕 😕 😕
For more high-quality dry goods: See my GitHub:dotnetfly