A: background

1. Tell a story

A few days ago a friend wx for help, its program memory often soars, CPU occasionally soars, did not find the reason, hope to help see.

The dump is only 2GB. It’s a good idea to find a memory leak in the dump. 😂 is right, so I still hope his program memory increased to 5G+ to give me a look, since the memory can not see, then see this occasionally surging CPU is what? The old way, talk to Windbg.

2. WINDBG analysis

1. How much CPU is it

To see the CPU utilization of the machine at the time this snapshot was generated, use! Tp command.

0:03 3 >! tp CPU utilization: 93% Worker Thread: Total: 800 Running: 800 Idle: 0 MaxLimit: 800 MinLimit: 320 Work Request in Queue: 3203 Unknown Function: 000007fefb551500 Context: 000000002a198480 Unknown Function: 000007fefb551500 Context: 0000000028a70780 Unknown Function: 000007fefb551500 Context: 000000002a182610 Unknown Function: 000007fefb551500 Context: 00000000262a2700 ...

What was supposed to be a simple command turned out to be a clatter of… The current CPU utilization rate is 93%. There is nothing wrong with it. The CPU is really surging. But even more tragic is that there are 3203 pending tasks in the thread pool queue, you can guess that the program is not only high CPU, but also hang phenomenon…

The next question is: What happened to those 800 warriors? The program is now employing people, so to find out, I’d like to follow my usual thinking and look at the synchronized block table.

2. Thread synchronization block table

To view the synchronized block table, use! Synblk command.

0:03 3 >! syncblk Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner 188 0000000010defc28 1 1 000000001e8fb400 9f4 715 00000003ff1e3d80 System.Web.HttpApplicationStateLock 126159 000000001e424e28 1 1 0000000023425e00 1f14 695 0000000301210038 ASP.global_asax 126173 00000000281acaf8 1 1 0000000024b8ea70 24ec 785 00000000ff8c5e10 ASP.global_asax 126289 00000000247a4068 1 1 0000000027ee93c0 808 413 0000000306aca288 ASP.global_asax 126368 0000000027180dd8 1 1 0000000028005cb0 1e7c 650 00000002008d6280 ASP.global_asax 126489 0000000027211dd8 1 1 0000000026862420 ec4 220 000000030611a290 ASP.global_asax 126788 00000000247924b8 1 1 0000000021871ff0 2784 529 00000004039901a8 ASP.global_asax 126843 00000000285b8d28 1 1 000000001cbd6710 2170 456 00000004007ec748 ASP.global_asax 126934 0000000021b212b8 1 1 0000000026ca7590 16cc 472 000000030090e810 ASP.global_asax 127251 0000000024769188 1 1 000000002831eaf0 2b68 648 0000000207051038 ASP.global_asax ... ----------------------------- Total 141781 CCW 2 RCW 4 ComClassFactory 0 Free 140270

When I went, there was another pile of hurrying. From the hexagrams above, we could see two messages:

  • MonitorHeld: 1

Indicates that a thread is currently holding a lock.

  • ASP.global_asax , System.Web.HttpApplicationStateLock

Represents the object held by the current thread.

Together a little strange, though, in addition to the first thread holds HttpApplicationStateLock, behind all the threads in ASP. Global_asax object has a different memory address: 0000000301210038000000 0 ff8c5e10, The lock object is not static, it is more like an instance. It is interesting to take a look at the stack of two threads, such as 715,695.

3. Look at the thread stack

To view the thread stack, use! Clrstack command.



Looking at these two threads on the stack, Are stuck in XXX. MvcApplication. Session_Start method in the System. Threading. Monitor. Enter System. (Object) and System. Threading. Monitor. ObjWait, overall Session_Start method must be have a problem here, so have to think of some way to export a look at the source code.

4. Review the problem code

To export the Session_Start method, use the combined command! ip2md + ! Savemodule can.

| | 2:2:17 81 >! ip2md 000007fe99c6f0c5 MethodDesc: 000007fe990fe080 Method Name: xxx.xxx.xxx.MvcApplication.Session_Start(System.Object, System.EventArgs) Class: 000007fe991ae0c0 MethodTable: 000007fe990fe238 mdToken: 0000000006000119 Module: 000007fe990fd750 IsJitted: yes CodeAddr: 000007fe99c6e1f0 Transparency: Critical ||2:2:1781> ! savemodule 000007fe990fd750 E:\dumps\Session_Start.dll 3 sections in file section 0 - VA=2000, VASize=17538, FileAddr=200, FileSize=17600 section 1 - VA=1a000, VASize=3ac, FileAddr=17800, FileSize=400 section 2 - VA=1c000, VASize=c, FileAddr=17c00, FileSize=200

Then with the help of ILSpy decompilation tool to view, because more sensitive, I will be more fuzzy, please forgive me!

After reading the above code, I am a little puzzled. Since we are assigning to Application, why not extract to Application_Start? I guess it doesn’t matter to the developer, so I’ll take a look at the source code of Application.

public sealed class HttpApplicationState : NameObjectCollectionBase { private HttpApplicationStateLock _lock = new HttpApplicationStateLock(); public void Set(string name, object value) { _lock.AcquireWrite(); try { BaseSet(name, value); } finally { _lock.ReleaseWrite(); } } } internal class HttpApplicationStateLock : ReadWriteObjectLock { internal override void AcquireWrite() { int currentThreadId = SafeNativeMethods.GetCurrentThreadId(); if (_threadId == currentThreadId) { _recursionCount++; return; } base.AcquireWrite(); _threadId = currentThreadId; _recursionCount = 1; } internal override void ReleaseWrite() { int currentThreadId = SafeNativeMethods.GetCurrentThreadId(); if (_threadId == currentThreadId && --_recursionCount == 0) { _threadId = 0; base.ReleaseWrite(); } } } internal class ReadWriteObjectLock { internal virtual void AcquireWrite() { lock (this) { while (_lock ! = 0) { try { Monitor.Wait(this); } catch (ThreadInterruptedException) { } } _lock = -1; } } internal virtual void ReleaseWrite() { lock (this) { _lock = 0; Monitor.PulseAll(this); }}}

The code is a little long, but in general, the code here is not simple. Application encapsulates a read/write lock through the lock itself, which is not simple, but what is the problem here? Even if you write it in the wrong place, it doesn’t seem to cause the CPU to explode, right?

Actually here involves a concept: that is the lock convoys escort (lock)

5. Lock Convoys, France, the United States and the United States

I found a good article that explains what Lock Convoys mean, ‘Lock Escorts,’ and here’s a snapshot of the whole thing.

This is what lockless programming has been attacking.

Three:

I look at the Session_Start method, there are about 105 applications [XXX], which means there are 105 locks waiting for the current thread to go through… At this point, there are nearly 800 threads already in the method, and there are no less than 8W locks waiting for these threads to break through, and then they are forced to switch between massive slices of CPU time, wake up and sleep, sleep and wake up, and they are all staggered together to pick up the CPU.

The solution is simple. Try your best to reduce the number of these serial locks to one or none, even better 😄 nil.

  • Assignments to Application are all extracted into Application_Start, after all there is no one competing when the program is started.
  • As far as possible willSpecial assignmenttoBatch assignment.

For more high-quality dry goods: see my GitHub:dotnetfly