Application Monitors may hurt performance badly

I’ve just spent two days and a night between them on the client site, trying to solve a severe performance problem with the application, that Netwise (the company I work for) has developed. It is an ASP.NET web application, running on Windows 2008 R2 with IIS 7.5 on a 4-CPU/8G machine.

We’ve fine-tuned every possible IIS and ASP.NET setting, defined data caching, output caching, profiled almost every line of the code, however, the application still ran very badly, even when tested with 50 concurrent requests. The symptoms were:

• Slow response – a web service, getting some data from the cache, manipulating it and returning result could take 8 seconds to respond (3ms when profiled on development environment)
• Sometimes it ran fine for a minute, and then, peaked to 20 sec per request
• Very few request/second rate – 30-50 instead of 500-1000
• About 10-20 thrown exceptions /sec (windows performance monitor)
• IIS crashes on KERNELBASE.DLL module – no info whatsoever on the net about it
• Too much time spent in JIT – unexplained – no dynamic assemblies loaded

We’ve used one of the well-known Application Performance Monitors (no names), this client uses to monitor .NET apps in production time. It was already preinstalled and configured on this machine. Those tools are basically using ELT hooks to monitor the CLR, which supposed to introduce about 5% of performance penalty on .NET apps.

The tool has indicated a lot of time spent in functions performing very simple LINQ-to-objects queries, which was ridiculous.

After hours of investigations, kilos of junk food, and almost being despaired, I’ve disabled the tool, and was amazed to discover, that suddenly the application performed amazingly well! No crashes, less than 100ms response time and more than a 1000 request/second!

It appears that the dreaded tool was either in buggy version, or poorly configured, and caused throwing tens of exceptions per request causing the app to perform badly. Look for my previous post to see how exceptions may hurt web apps performance. The funny thing was that this tool has indicated that there were functions in our code that perform badly. The tool’s producer should test its own tool before testing others.

The conclusion is: be aware, that similar tools, either buggy or misconfigured may drastically hurt the performance of your application. They hook to the CLR on a very low level, and may cause very strange issues. I’m not saying do not use these tools. On the contrary, in other cases they’ve helped me a lot to discover application performance issues. Just before rewriting half of the application code, try disabling them to see what happens.

In addition, it was probably not a very wise thing to blame an app for such a bad performance, although that is what I usually tend to do – look for pitfalls in my own stuff before blaming the others.

Want to know more about IIS7? Come to see me ad Developers Academy 4.