By February 24, 2014 2 Comments Read More →

Performance: High CPU–The Case of Lack Of Caching And Dynamically Generated JavaScript Files

During the latest load test we observed high CPU in our front end SharePoint servers. The servers host custom pages and under increasing load we observed ever increasing CPU utilization. The correlation was quite obvious – the higher the load the higher CPU utilization. We couldn’t get to the target load since at some point we reached to 100% CPU utilization.

Analysis

To analyze the problem and ultimately find the culprit we simulated the load and then used Process Explorer, freely available for download here, to look at the processes and their resource consumption.

Here are chronological order of events:

  • Identified w3wp.exe that was consuming CPU the highest.
  • This w3wp.exe periodically spawned VBC.exe, the VB.NET compiler. A good candidate for high CPU culprit as on the fly compilation is CPU hungry operation especially under load.
  • Using Process Monitor, freely available for download here, we quickly identified the VB.NET files created and compiled by the application on the fly. Turned out to be code related to charting of the reporting services.
  • We disabled the charting part and re-run the load test. It didn’t make much of the impact and CPU remained high. Charting was obviously not the culprit.
  • We took memory dump using Process Explorer.
  • We opened the memory dump in WinDBG, freely available for download here. To load the dump use shortcut Ctrl+D.
  • While in the WinDBG and after opening the dump, execute these commands:
  • Specify Microsoft symbols path, srv*c:\debug\symbols*http://msdl.microsoft.com/download/symbols.
  • Specify your assemblies symbol path using “;”. Normally you point to where you PDB files are.
  • .loadby sos clr (to load managed code extension).
  • !runaway (to show threads and their times in CPU).
  • Copy and paste the !runaway output to Notepad.
  • Back in WinDBG, run ~*e!clrstack (to dump all available CLR Stacks on all threads)
  • Back to Notepad, choose slowest thread number from notepad, Example: (19), copy it.
  • Back to WinDBG, search (19) in WinDBG.
  • Review the stack of the prolonged thread. It will show you your slow methods in your code including the code lines, assuming you have specified.
  • Repeat for all those that are longer than, say 10 seconds.
  • If your code shows as Unknown, it may be due to WinDBG cannot resolve PDB/symbols.
  • To find out why symbols not resolving:
  • !sym –noisy
  • .reload
  • Review what PDB’s WinDBG looks for, find the PDB on the build server and specify it in the symbols path.

Following the procedure outlined above we identified the root cause of high CPU utilization, and that was dynamic JavaScript generation without caching for further use. So on each request there was massive string manipulation – normally CPU hungry operation – and the more requests came in the higher was the CPU utilization. Engineering team was provided with exact root cause and was able to provide the fix in no time, by optimizing string manipulation and caching results for further uses.

Posted in: Uncategorized

About the Author:

This blog is dedicated to share simple practices I that get me results.

2 Comments on "Performance: High CPU–The Case of Lack Of Caching And Dynamically Generated JavaScript Files"

Trackback | Comments RSS Feed

  1. J.D. Meier says:

    I remember the first time I dumped the stack and could inspect what the threads were up to.

    It was like the needles in the haystack were suddenly front and center.

  2. Alik Levin says:

    Nicely put, JD. Exactly the same feeling – from ambiguity to crisp clarity.

Post a Comment