During the latest load test we observed high CPU in our front end SharePoint servers. The servers host custom pages and under increasing load we observed ever increasing CPU utilization. The correlation was quite obvious – the higher the load the higher CPU utilization. We couldn’t get to the target load since at some point we reached to 100% CPU utilization.
To analyze the problem and ultimately find the culprit we simulated the load and then used Process Explorer, freely available for download here, to look at the processes and their resource consumption.
Here are chronological order of events:
- Identified w3wp.exe that was consuming CPU the highest.
- This w3wp.exe periodically spawned VBC.exe, the VB.NET compiler. A good candidate for high CPU culprit as on the fly compilation is CPU hungry operation especially under load.
- Using Process Monitor, freely available for download here, we quickly identified the VB.NET files created and compiled by the application on the fly. Turned out to be code related to charting of the reporting services.
- We disabled the charting part and re-run the load test. It didn’t make much of the impact and CPU remained high. Charting was obviously not the culprit.
- We took memory dump using Process Explorer.
- We opened the memory dump in WinDBG, freely available for download here. To load the dump use shortcut Ctrl+D.
- While in the WinDBG and after opening the dump, execute these commands:
- Specify Microsoft symbols path, srv*c:\debug\symbols*http://msdl.microsoft.com/download/symbols.
- Specify your assemblies symbol path using “;”. Normally you point to where you PDB files are.
- .loadby sos clr (to load managed code extension).
- !runaway (to show threads and their times in CPU).
- Copy and paste the !runaway output to Notepad.
- Back in WinDBG, run ~*e!clrstack (to dump all available CLR Stacks on all threads)
- Back to Notepad, choose slowest thread number from notepad, Example: (19), copy it.
- Back to WinDBG, search (19) in WinDBG.
- Review the stack of the prolonged thread. It will show you your slow methods in your code including the code lines, assuming you have specified.
- Repeat for all those that are longer than, say 10 seconds.
- If your code shows as Unknown, it may be due to WinDBG cannot resolve PDB/symbols.
- To find out why symbols not resolving:
- !sym –noisy
- Review what PDB’s WinDBG looks for, find the PDB on the build server and specify it in the symbols path.