A: background

1. Tell a story

Yesterday, I continued to pay the technical debt. After the optimization round of the program was pulled online, the memory continued to rise and fall. When it was low, it was 20G, when it was high, it was 30G, and after a while, it dropped several G.

Two: solution idea

Finding a problem from a snapshot is like seeing a doctor.

1. Find objects in the managed heap that are larger than 10 MB

It is usually easiest to start with a managed heap when dealing with large collections. To avoid typing all types out, you can set a filter here and kick off any type smaller than 10M. Dumpheap-stat-min 10240, desensitize the sensitive object.

0:00 0 >! dumpheap -stat -min 10240 Statistics: MT Count TotalSize Class Name 00007ffe094e6fc0 4 523776 System.Object[] 00007ffe094e6948 6 7179822 System.String 00007ffe0780da08 33 46514160 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.Collections.Generic.HashSet`1[[System.Int32, mscorlib]], System.Core]][] 00007ffe09f95f40 250 188739344 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.Int32, mscorlib]][] 00007ffe094ec988 18 540828823 System.Byte[] 00007ffe07802da8 1620 622578672 System.Linq.Set`1+Slot[[System.Int32, mscorlib]][] 000001bc0452e600 1389 1038494910 Free 00007ffe094baf50 68 1128274800 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.DateTime, mscorlib]][] 00007ffe094e9220 2224 1513951832 System.Int32[] 00007ffe07819df8 2232 1668042480 System.Collections.Generic.HashSet`1+Slot[[System.Int32, mscorlib]][] 00007ffe094c8510 226 1672164568 System.Int64[] 00007ffdab8676e8 1137 1901228880 System.Collections.Generic.HashSet`1+Slot[[System.Int64, mscorlib]][] 00007ffdab89b3b0 136 1986723840 System.Linq.Set`1+Slot[[System.Int64, mscorlib]][] Total 13321 objectsCopy the code

2. Identify suspicious objects in the heap

Dictionary

= 1128274800/1024/1024=1G (Dictionary

= 1128274800/1024/1024) The method table address 00007FFe094BAF50 is used to find the memory addresses of the 68 collections.
,datetime>
,datetime>

0:02 8 >! dumpheap -mt 00007ffe094baf50 Address MT Size 000001c2f262a708 00007ffe094baf50 69438000 000001c1bb8e1020 00007ffe094baf50 16147872 000001c1bce04760 00007ffe094baf50 33486336 000001c37e8f1020 00007ffe094baf50 143987328 000001c44e8f1020 00007ffe094baf50 287974800 000001c3c419b268 00007ffe094baf50 16147872 000001c3f6b9ac28 00007ffe094baf50  16147872 000001c467336fa0 00007ffe094baf50 33486336 000001c46f3fa760 00007ffe094baf50 69438000 000001c489df3668 00007ffe094baf50 16147872 000001c494166828 00007ffe094baf50 33486336 000001c4a68f1020 00007ffe094baf50 69438000 000001c4d4c5c290 00007ffe094baf50 16147872 000001c4da8f1058 00007ffe094baf50 33486336 000001c4de8f1020 00007ffe094baf50 69438000 000001c5028f1058 00007ffe094baf50 33486336 000001c5068f1020 00007ffe094baf50 33486336 ...Copy the code

Next, select a few large dictionaries, such as 000001C44e8F1020 00007FFe094baf50 287974800, and calculate size: 279M.

3. Find the code block where the collection resides

The dictionary is 279M, I know, but how do I know which code block the dictionary is in? It’s easy to find the answer. Pass! Gcroot finds its reference root and can find its code block by reference chain, which is not very practical, 😄😄😄.

0:00 0 >! gcroot 000001c4de8f1020 Thread 2da8: 00000017f4c7e5d0 00007ffdab758ca1 xxxx.xxxx.xxxx.GetFlowAwayCustomer(Int32, System.String, System.Collections.Generic.Dictionary`2<System.String,System.Collections.Generic.List`1<xxxx>>) rbp-238: 00000017f4c7e628 -> 000001c3d5c1bdf0 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]], mscorlib]] -> 000001c3d8de7d10 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]], mscorlib]][] -> 000001c3d8d58630 System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]] -> 000001c4de8f1020 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, mscorlib],[System.DateTime, mscorlib]][]Copy the code

Three points of information can be seen from the above reference chain:

<1> The current dictionary is on the 2da8 thread

<2> The dictionary in the GetFlowAwayCustomer method probably counts the lost customer.

<3> The top of the call chain is the largest collection Dictionary<string,Ditionary<int,DateTime>>, address: 000001C3D5C1bDF0

4. Find out more

<1> Dig the dictionary contents

With the largest Dictionary, let’s look at the memory size of Dictionary

>.
,ditionary

0:00 0 >! objsize 000001c3d5c1bdf0 sizeof(000001c3d5c1bdf0) = 340008256 (0x14441d40) bytes (System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]], mscorlib]])Copy the code

Sizeof (000001c3d5c1bdf0) = 340008256 (0x14441d40) bytes: 324M Do go inside and have a look.

0:00 0 >! da -length 1 -start 1 -details 000001c3d8de7d10 Name: System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]], mscorlib]][] MethodTable: 00007ffdab940650 EEClass: 00007ffdab9405b8 Size: 192(0xc0) bytes Array: Rank 1, Number of elements 7, Type VALUETYPE Element Methodtable: 00007ffdab940520 [1] 000001c3d8de7d38 Name: System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]], mscorlib]] MethodTable: 00007ffdab940520 EEClass: 00007ffe08e92920 Size: 40(0x28) bytes File: C:\Windows\Microsoft.Net \ assembly \ GAC_64 \ mscorlib \ v4.0 _4. 0.0.0 __b77a5c561934e089 \ mscorlib DLL Fields: MT Field Offset Type VT Attr Value Name 00007ffe094e9288 4003474 10 System.Int32 1 instance 58671583 hashCode 00007ffe094e9288 4003475 14 System.Int32 1 instance -1 next 00007ffe094ebf10 4003476 0 System.__Canon 0 instance 000001c2cec43610 key 00007ffe094ebf10 4003477 8 System.__Canon 0 instance 000001c3d7b45370 value 0:000> ! do 000001c3d7b45370 Name: System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]] MethodTable: 00007ffe094b9ec8 EEClass: 00007ffe08e9d528 Size: 80(0x50) bytes File: C:\Windows\Microsoft.Net \ assembly \ GAC_64 \ mscorlib \ v4.0 _4. 0.0.0 __b77a5c561934e089 \ mscorlib DLL Fields: MT Field Offset Type VT Attr Value Name 00007ffe094e9220 4001858 8 System.Int32[] 0 instance 000001c46e8f1020 buckets 00007ffe094baf50 4001859 10 ... ime, mscorlib]][] 0 instance 000001c46f3fa760 entries 00007ffe094e9288 400185a 38 System.Int32 1 instance 2512598 count 00007ffe094e9288 400185b 3c System.Int32 1 instance 3194430 version 00007ffe094e9288 400185c 40 System.Int32 1 instance -1 freeList 00007ffe094e9288 400185d 44 System.Int32 1 instance 0 freeCount 00007ffe094dabb8 400185e 18 ... Int32, mscorlib]] 0 instance 000001bc06272ab8 comparer 00007ffe0a0463e0 400185f 20 ... eTime, mscorlib]] 0 instance 0000000000000000 keys 00007ffe0a046258 4001860 28 ... eTime, mscorlib]] 0 instance 0000000000000000 values 00007ffe094e6f28 4001861 30 System.Object 0 instance 0000000000000000 _syncRootCopy the code

Count =251w; details = details =251w;

<2> Dig the thread stack

Now that we have the dictionary content, let’s see what this thread [2da8] is doing.

0:028> ~~[2da8]s ntdll! NtWaitForSingleObject+0x14: 00007ffe`28646124 c3 ret 0:028> ! clrstack OS Thread Id: 0x2da8 (28) Child SP IP Call Site 00000017f4c7e388 00007ffe28646124 [HelperMethodFrame: 00000017f4c7e388] 00000017f4c7e4f0 00007ffe09e48e52 System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]].Resize(Int32, Boolean) 00000017f4c7e560 00007ffe09316c65 System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.DateTime, mscorlib]].Insert(Int32, System.DateTime, Boolean)Copy the code

Holy shit, this collection is being expanded… You should know that with capacity expansion comes empty memory.

Three:

Now, I’m sure some of you are saying, well, what’s the solution? Because it was discovered yesterday, and I did not write the code, you ask me?? We are going to optimize ➕ custom collections (hashsets, dictionaries) from both sides of the business logic. After all, these two collections consume too much memory, and we will analyze their expansion mechanism in the next article.