Beware of Multi-threaded nature of Web Apps

28 במרץ 2010

2 תגובות

I’ve spent some time last week trying to solve production-time IIS crashes, caused by one of our web apps. With the great help of Gadi Meir we’ve detected, that some naive code somehow causes an endless recursion, resulting StackOverflowException, causing IIS to crash, and recycle the app pool. With some effort, I’ve finally understood what has happened there, and I’m able to simulate the behavior.


Consider the following code (which was created entirely for demo purposes):



   1: public static object GetItem(string key)
   2: {
   3:     if (HttpRuntime.Cache[key] == null)
   4:     {
   5:         InsertToCache(key);
   6:     }
   7:     return HttpRuntime.Cache[key];
   8: }
   9:  
  10: private static void InsertToCache(string key) {
  11:  
  12:     HttpRuntime.Cache.Insert(key, "Cached Value", null,
  13:             DateTime.Now.AddSeconds(10),
  14:             Cache.NoSlidingExpiration,
  15:             CacheItemPriority.Normal, 
  16:             RemovedCallback);            
  17: }        
  18:         
  19: private static void RemovedCallback(string key, object value,
  20:                     CacheItemRemovedReason removedReason)
  21: {            
  22:     InsertToCache(key);            
  23: }

This is a simple caching pattern, by which, the entry in the cache is being refreshed every predefined period, without an explicit request. This is very useful when gaining the value from the persistent repository takes some time, and we don’t want any user to wait for response from the server.


So the GetItem function checks whether the entry exists in the cache, and calls the InsertToCache, which will actually get the entry from the repository (here, just inserts some string), and will pass the cache a RemovedCallback, which will be called whenever the entry is removed from the cache. So, when the defined caching period, expires, the RemoveCallback delegate will be called, which, in turn will call the InsertToCache to refill the cache. Pretty straight-forward I think. So what’s the problem?


What is not so frequently realized is that Cache.Insert() method first looks for the key within the existing entries, and if found, removes it. Of course, entry removal causes the invocation of RemovedCallback delegate. Do you see the problem yet? If the InsertToCache will be called with the key, that already exists in the cache, the Insert will cause the RemovedCallback, which in turn will call the InsertToCache which will again call the RemovedCallback…. creating an unintended recursion without any halt condition, causing StackOverflowException. As it turns out, stack overflow in CLR 2 causes IIS application pool to ungracefully crash. I’ve heard it’s not the case with CLR 4, but haven’t got a chance to confirm.


Now, why would we enter the InsertToCache for the second time? As you see, we are checking for existence of the key in the cache, so theoretically, this function should be called only once.


The thing is, in multithreaded app (and every web app is multithreaded by definition), a context switch can happen exactly on line 4, exactly the moment the cache entry was expired. So the thread#1 will indicate that the entry does not exist, and then, the context switch will occur to thread#2, which will also indicate that the entry does not exist. This way, both threads will race into the Cache.Insert(), while the second thread will cause the recursion and the stack overflow.


This is fairly easy to simulate. Run IISRESET and then point two browser windows to the application. It’ll take ASP.NET a couple of seconds to compile the page, so you’ve got 2-3 seconds to open the second browser window after starting the first. Both browsers will wait for the compilation to end, and then will race concurrently into the GetItem function. In rare cases it will work, but in most cases, the app will crash.


There are two ways to solve this particular issue. Either lock the whole GetItem function, but this will have affection on performance (threads will now wait for each other). The other, and much simple and efficient solution is to check the removedReason argument of the callback. When the entry is removed because of expiration the argument will hold an “Expire” reason. If the removedReason=Removed, don’t call InsertToCache, cause this is the method that have just called you.


The point of this post is not to teach you caching design patterns, but to make you aware of the multi-threaded nature of web apps. And as such, in some cases, concurrent calls may have a devastating effect on your app.


Enjoy!

הוסף תגובה
facebook linkedin twitter email

כתיבת תגובה

האימייל לא יוצג באתר. (*) שדות חובה מסומנים

2 תגובות

  1. Gatewood22 באפריל 2013 ב 7:43

    Wow, that's what I was exploring for, what a stuff! present here at this blog, thanks admin of this web site.

    להגיב
  2. Hale22 באפריל 2013 ב 11:29

    Hello friends, how is the whole thing, and what
    you would like to say regarding this post, in my view its truly amazing for me.

    להגיב