March 2010 - Posts
I’ve spent some time last week trying to solve production-time IIS crashes, caused by one of our web apps. With the great help of Gadi Meir we’ve detected, that some naive code somehow causes an endless recursion, resulting StackOverflowException, causing IIS to crash, and recycle the app pool. With some effort, I’ve finally understood what has happened there, and I’m able to simulate the behavior.
Consider the following code (which was created entirely for demo purposes):
1: public static object GetItem(string key)
2: { 3: if (HttpRuntime.Cache[key] == null)
4: { 5: InsertToCache(key);
6: }
7: return HttpRuntime.Cache[key];
8: }
9:
10: private static void InsertToCache(string key) { 11:
12: HttpRuntime.Cache.Insert(key, "Cached Value", null,
13: DateTime.Now.AddSeconds(10),
14: Cache.NoSlidingExpiration,
15: CacheItemPriority.Normal,
16: RemovedCallback);
17: }
18:
19: private static void RemovedCallback(string key, object value,
20: CacheItemRemovedReason removedReason)
21: { 22: InsertToCache(key);
23: }
This is a simple caching pattern, by which, the entry in the cache is being refreshed every predefined period, without an explicit request. This is very useful when gaining the value from the persistent repository takes some time, and we don’t want any user to wait for response from the server.
So the GetItem function checks whether the entry exists in the cache, and calls the InsertToCache, which will actually get the entry from the repository (here, just inserts some string), and will pass the cache a RemovedCallback, which will be called whenever the entry is removed from the cache. So, when the defined caching period, expires, the RemoveCallback delegate will be called, which, in turn will call the InsertToCache to refill the cache. Pretty straight-forward I think. So what’s the problem?
What is not so frequently realized is that Cache.Insert() method first looks for the key within the existing entries, and if found, removes it. Of course, entry removal causes the invocation of RemovedCallback delegate. Do you see the problem yet? If the InsertToCache will be called with the key, that already exists in the cache, the Insert will cause the RemovedCallback, which in turn will call the InsertToCache which will again call the RemovedCallback…. creating an unintended recursion without any halt condition, causing StackOverflowException. As it turns out, stack overflow in CLR 2 causes IIS application pool to ungracefully crash. I’ve heard it’s not the case with CLR 4, but haven’t got a chance to confirm.
Now, why would we enter the InsertToCache for the second time? As you see, we are checking for existence of the key in the cache, so theoretically, this function should be called only once.
The thing is, in multithreaded app (and every web app is multithreaded by definition), a context switch can happen exactly on line 4, exactly the moment the cache entry was expired. So the thread#1 will indicate that the entry does not exist, and then, the context switch will occur to thread#2, which will also indicate that the entry does not exist. This way, both threads will race into the Cache.Insert(), while the second thread will cause the recursion and the stack overflow.
This is fairly easy to simulate. Run IISRESET and then point two browser windows to the application. It’ll take ASP.NET a couple of seconds to compile the page, so you’ve got 2-3 seconds to open the second browser window after starting the first. Both browsers will wait for the compilation to end, and then will race concurrently into the GetItem function. In rare cases it will work, but in most cases, the app will crash.
There are two ways to solve this particular issue. Either lock the whole GetItem function, but this will have affection on performance (threads will now wait for each other). The other, and much simple and efficient solution is to check the removedReason argument of the callback. When the entry is removed because of expiration the argument will hold an “Expire” reason. If the removedReason=Removed, don’t call InsertToCache, cause this is the method that have just called you.
The point of this post is not to teach you caching design patterns, but to make you aware of the multi-threaded nature of web apps. And as such, in some cases, concurrent calls may have a devastating effect on your app.
Enjoy!
The Developer Academy 4 is behind us, and after completing most of the urgent tasks that I have postponed while being busy preparing my IIS 7 presentation, I’ve finally found a couple of minutes to write this. post
First of all, I think Microsoft stuff has done a great job preparing this event. Discarding some minor issues, everything went extremely smooth.
Second, I’d like to thank to all the people, who have participated in my “The 5 coolest IIS 7 features” talk. I had a great time, hope you too. The recording of the session, done by Sela University, is available here.
I’d like to share with you some of the resources linked to those five IIS 7 features, I’ve talked about during the session:
Web Platform Installer
The Web PI page on the Microsoft Site
Nice blog post about the Web PI by Scott Guthrie
Microsoft Web Platform Installer on the Official Microsoft IIS Site
Web Deployment Tool
Web Deployment Made Awesome: If You're Using XCopy, You're Doing It Wrong – Great blog post and MIX10 session by Scott Hanselman
URL Rewrite Module
Failed Request Tracing
Failed Request Tracing on IIS7 – Chanel9 Video
SEO Toolkit
Search Engine Optimization Toolkit : The Official Microsoft IIS Site
IIS Search Engine Optimization Toolkit - ScottGu's Blog – Great Post
Enjoy!
I’ve just spent two days and a night between them on the client site, trying to solve a severe performance problem with the application, that Netwise (the company I work for) has developed. It is an ASP.NET web application, running on Windows 2008 R2 with IIS 7.5 on a 4-CPU/8G machine.
We’ve fine-tuned every possible IIS and ASP.NET setting, defined data caching, output caching, profiled almost every line of the code, however, the application still ran very badly, even when tested with 50 concurrent requests. The symptoms were:
- Slow response – a web service, getting some data from the cache, manipulating it and returning result could take 8 seconds to respond (3ms when profiled on development environment)
- Sometimes it ran fine for a minute, and then, peaked to 20 sec per request
- Very few request/second rate – 30-50 instead of 500-1000
- About 10-20 thrown exceptions /sec (windows performance monitor)
- IIS crashes on KERNELBASE.DLL module – no info whatsoever on the net about it
- Too much time spent in JIT – unexplained – no dynamic assemblies loaded
We’ve used one of the well-known Application Performance Monitors (no names), this client uses to monitor .NET apps in production time. It was already preinstalled and configured on this machine. Those tools are basically using ELT hooks to monitor the CLR, which supposed to introduce about 5% of performance penalty on .NET apps.
The tool has indicated a lot of time spent in functions performing very simple LINQ-to-objects queries, which was ridiculous.
After hours of investigations, kilos of junk food, and almost being despaired, I’ve disabled the tool, and was amazed to discover, that suddenly the application performed amazingly well! No crashes, less than 100ms response time and more than a 1000 request/second!
It appears that the dreaded tool was either in buggy version, or poorly configured, and caused throwing tens of exceptions per request causing the app to perform badly. Look for my previous post to see how exceptions may hurt web apps performance. The funny thing was that this tool has indicated that there were functions in our code that perform badly. The tool’s producer should test its own tool before testing others.
The conclusion is: be aware, that similar tools, either buggy or misconfigured may drastically hurt the performance of your application. They hook to the CLR on a very low level, and may cause very strange issues. I’m not saying do not use these tools. On the contrary, in other cases they’ve helped me a lot to discover application performance issues. Just before rewriting half of the application code, try disabling them to see what happens.
In addition, it was probably not a very wise thing to blame an app for such a bad performance, although that is what I usually tend to do – look for pitfalls in my own stuff before blaming the others.
Want to know more about IIS7? Come to see me ad Developers Academy 4.
(Update: Thanks to Avi Pinto for pointing out at a stupid mistake regarding catching exceptions in every method. What I meant was that it is a bad practice catching exceptions in every method. Don’t catch exceptions if you have nothing to do with them)
Yes, exceptions are one of most common sources of performance issues of any .NET application, and especially of ASP.NET Web Applications. Without going into details, every thrown exception, either handled or not, introduces a serious penalty on the code execution time.
Consider the following two simplistic functions:
private static int Divide(int a, int b)
{
if (b == 0) return int.MaxValue;
return a / b;
}
private static int DivideEx(int a, int b)
{
try
{
return a / b;
}
catch (DivideByZeroException ex)
{
return int.MaxValue;
}
}The first one checks the input, and returns the maximum value of integer in case of invalid denominator. The second one, relies on the user input, and handles the exception. Unfortunately, in my practice I’ve seen much more of the second type of function. Now let’s see what this does to the execution performance.
I’m gong to use the following method to benchmark the performance of the above functions:
private static long ProfileDivideFunc(Func<int, int, int> func)
{
Random rnd = new Random(DateTime.Now.Millisecond);
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 10000; i++)
{
func(rnd.Next(0, i), 0);
}
sw.Stop();
return sw.ElapsedMilliseconds;
}Nothing fancy here. The only “interesting” thing is that the method accepts a delegate and invokes it. Inside the method, I’m just running the tested function 10,000 times, passing a random integer as nominator, and zero as denominator. Practically, I’m measuring the performance penalty of throwing and catching 10,000 exceptions.
The results are pretty much as expected. On my D-Core/4G/Win7 x64 laptop, the first function test returns 0ms, whereas the second function (the one with exceptions), takes 422ms to complete.
It sounds like nothing, however, consider what happens, in ASP.NET app, when 500 or 1,000 concurrent requests are being handled. Also take into consideration the fact that usually we don’t catch exceptions in every method (catching exception in every method is a very bad practice), so it bubbles all the way to some top-level-exception-catching-mechanism. In addition, it has to be logged, which also takes some CPU cycles.
I’ve recently seen a web app, that thrown and logged 3 exceptions on each and every request. Needless to mention how the performance was improved after eliminating the sources for those exceptions.
So here are couple of recommendations:
- Never, and I repeat, never, throw exceptions. Use response error codes if you need to indicate and error.
- Verify user input. Don’t handle exceptions following bad input.
- Try eliminating usage of functions that throw exceptions in case of invalid input, like Parse(). Use TryParse() instead.
- Watch for exceptions, log them and eliminate their sources ASAP.
- There is nothing worst than unhandled exceptions in ASP.NET application. Catch everything in some top-level mechanism, like Application_Error of global.asax, or connect to AppDomain.CurrentDomain.UnhandledException event to catch and log them all.
The most important this is: remember, Exceptions are EXCEPTIONS from the normal, so they should be treated accordingly. They are not NORMAL at runtime.
Have fun.
Want to know why you should use IIS7 ? Come to see me at Developers Academy 4.