Hi all! The meeting will take place tomorrow, March 14, 2011.
This time, I’m going to dive deep into JavaScript concepts, such as context, closures, functions and so on. In my feeling, understanding those concepts is an absolute must for every web developer. Come and learn for free!
You can still register using this link: http://wdcil3.eventbrite.com.
See you there!
Update: the event date has changed due to the “Family Day”
The February Web Developers Community meeting will be held on February 3, 2011,February 6, 2011 as usually at Microsoft Israel Offices in Ra'anana. This is going to be a special meeting. First, we are saying good bye to Noam King, who has established this community 3 years ago. Vlad Azarkhin and Gal Kogman will take of this communty starting with this meeting. Second, we're launching the WDC Israel official Facebook page, and following this, the whole meeting will be focused on Facebook development.
See you there!
Please see the note at the bottom about the parking at Microsoft.
Contact us on: wdcil@live.com
Facebook for Developers
Abstract:
As users, we cannot remain indifferent to the “Facebook revolution”. As developers, not acknowledging it ,is almost a crime. In this session we will be talking about Facebook development features and capabilities, emphasizing on its practical aspects. We will begin with basic things, such as adding a "Like" button to the exisiting web page, and advance further to building a complete application using Facebook API and .NET Framework.
Agenda:
17:30-18:00 - Gathering and Networking
18:00-18:15 - Farewell to Noam King
18:15-19:15 - Part I: Integrating Facebook into exising web site
19:15-19:30 - Break
19:30-20:30 - Part II: Developing Facebook applications
About the presenters:
Kogman Gal is a senior consultant in E4d Solutions with extensive experience in architecture, design and hands on development of complex distributed applications. Specializing in the Microsoft .NET architecture, He has been working as a solutions architect and technical advisor for many customers in numerous projects. He has delivered hundreds of presentations about architecture and NET Technologies.
Vlad Azarkhin is a .NET technologist and architect with more than 13 years of practical experience in the field of software development. Currently working as a CTO at Netwise, his specialty surrounds everything related to web applications development, including server-side scalability, client-side manipulations, performance and security.
Important note regarding the parking arrangements. Please read carefully:
The parking is free at Microsoft's guests' parking area for registered participants only. Moreover, we have to pass invitees list to the parking staff 2 days prior the event. Therefore, you have to register till Feb, 1st, to appear on invitees list and to be allowed entering the parking. Unless your name appear on the list, you will be redirected to one of the paid parking, so please, register and secure the parking lot. Please make sure you enter you full name correctly during the registration process.
Registration:
You can register either by clicking this button:
Or right now:
Internet Explorer 8 has introduced developers tools for inspecting and debugging web pages. It allows, for example, selecting a DOM element and investigating it, debugging JavaScript by stopping on errors or breakpoints and some other useful stuff. However, one thing was missing there: network traffic capturing and analysis.
Traffic capturing is vital for analysis of your web site. Such tools usually show all the resources that are downloaded during a request, along with their response status, download time, size and much much more. Those tools can be used, to investigate performance issues, verify server settings and just to understand how much bandwidth your site uses. In addition, those tools are indispensable for debugging AJAX requests, as you’re able to see all the data that was send and received from the server.
Without any doubt, the most popular and robust network traffic capturing tool is Fiddler. It operates as a proxy between the network and your browser, so all the browsers’ traffic, first captured and logged by Fiddler, which turns it into a nice graphical representation. If you’re not familiar with Fiddler, I strongly suggest you to try it.
Another tool, worth mentioning is Firebug, which is the most essential development plug-in for Firefox case need to debug or test on that browser. Basically, Firebug is like the IE developer tools, but for Firefox.
The developer tools for the new Internet Explorer 9 include some new capabilities, of which the most interesting is the “Network Tab”, which is used for capturing and inspecting network traffic. Although it is far from being as versatile as Fiddler, it is the most welcomed addition to the IE developer tools, and will suite at least 80% of your needs when examining your site in IE.
To use the new feature, you need to open the IE9 Developer Tools. The simplest way to do it is by pressing F12 while you are in the Internet Explorer 9. Or you can also choose the “Developer Tools” from the “Tools” menu.
Now go to the “Network” tab in the developer tools window:
To inspect a web site, click the “Start Capturing” button and point your browser to a required URL. For example, this is what I got when opening the bing.com site:
So what do we learn from this? First of all, that my browser has totally send around 10Kb and received around 21Kb (some of the resources are already in the browser’s cache – we’ll talk about it later). The response included 13 resources too much for such a site IMO.
Now, for each of the resources (actually requests), we can see the request’s URL, the request method (GET, POST etc.), the result, which is the HTTP status, response content type, amount of data received, the total time it taken to receive the response, who has initiated the request and the nice timings chart. Double-clicking a row, or selecting a row and clicking the “Go To Detailed View” button will display more interesting details regarding the request and the following response, including cookies, initiator and detailed timings chart:
As you probably know, every HTTP message consists of a collection of headers and a body.
The Request Headers shows all the headers that were sent in a request, like the request method (GET, POST, etc.), browser’s user-agent, cookies and more. The Request Body, in turn, will show what was sent in a body, for example: form fields, SOAP message or JSON string.
The Response Headers displays headers that the web server has sent to the browser, for example, response status, body content-type, cookies to set and others. As you’ve probably guessed, the Response Body tab shows the actual response content. It can be HTML, XML, an image or whatever the server is sending.
The next two tabs will display a list of Cookies that were sent and received, and the information about the Initiator of the request, which can be a user, another page or something else.
The most interesting tab is Timings:
Each request has number of stages, from initiating a request till receiving a response. All are enlisted in this window. The Timings windows is the most useful when debugging performance issues.
In the above example, we can learn that the request didn’t wait in a sending queue (the Wait row). Every browser has a limited number of connections it is allowed to open towards one domain concurrently. Therefore, when there are numerous simultaneous requests, some will wait in a queue.
Start row shows the time from when the request was initially created to when it is sent.
Request means time to first byte. This is a compound time of sending the request and server processing until receiving the first byte of the response. Unfortunately it is not possible to know how much took request processing on a server, but this value gives a good clue about it, because usually requests are very small in size and therefore sent very quickly (unless you’re uploading a file or sending a large SOAP/XML message).
Response shows the time taken to receive the response data from the server. It depends on your bandwidth and the network latency. Basically, this is the last stage of HTTP request, and the end of it. However, the tool shows more useful information.
The Gap is the time gap between the completion of the request and the time the whole page has finished loading.
The green vertical line shows the point of time when the OnDomContentLoaded event occurred. This event indicates the readiness of the DOM – from this time is is possible to use JavaScript to manipulate it.
The red vertical line indicates the point of time when the page’s Load event occurred, which means the whole page, including it’s resources is ready to be shown in browser.
A little less detailed time-line is also shown in the Summary View, to which you can switch by pressing the “Back to Summary View” button. There are some interesting things you can learn about the Bing’s page traffic.
For example, some of the requests appear after the green and red lines, which probably means those are deferred AJAX calls.
Some requests return 304 status (Not modified), which means there is an up-to-date resource’s copy in the browser cache. This seems to be good, since the browser didn’t have to download the resource, however, this is very bad, because the browser still had to perform the request, which is an expensive network call, just to know that it has a required resource. The better strategy would be to add Expired Header to those resources’ responses, so the browser will use the cached resource instead of sending an unnecessary request.
Summarizing, the network traffic capturing tool is the most welcomed addition to IE developers tools. It might not be as robust or extensive as Fiddler, but it is certainly a very useful tool which will provide you with a decent service in most of cases.
Have fun,
Vlad
Thanks to all the attendants of the yesterday’s session. I had a lot of fun, hope you had too. Please feel free sending me questions and comments. You can find slides and demos from the session on my SkyDrive:
Have fun.
Yours,
Vlad
It is still not late to register and take part in the talk.
.jpg "title")
This talk is going to be one of the richest sources of information about AppFabric Caching available, so you will definitely benefit from it.
For details and registration:
https://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032460308&Culture=he-IL
Here is the abstract for the talk:
A vast majority of web applications use some kind of relational database to keep data. As it turns out, databases, while being robust, are the source of the most bottlenecks preventing our applications from scaling . This session is all about one of the most important patterns of building scalable web applications: caching. We're going to talk about scalability as a concept, types of data in our system, caching, caching patterns, distributed caching, and of course, about one of the nicest distributed cache products: Microsoft Windows AppFabric Caching Services (a.k.a. "Velocity").
See you there.
Yours,
Vlad
I was planning updating my previous post, further explaining the “Padding Oracle” vulnerability, but have decided to write new post instead, explaining how you can minimize the chance of hacking. I think it will be more practical.
A brief reminder: the vulnerability is based on the ability of the attacker to detect a decryption error when sending a modified request.
In this post, I’ll work on a simple but typical application. I’ve seen hundreds of applications like that. The application uses login mechanism similar to the Form Authentication, and simulates the most painful vulnerability: impersonation into administrative account.
The application description
It doesn’t matter what the application does, the important thing is that it has authentication. The application consists of a user name/password dialog, and after the login was confirmed(using database or whatever), the system generates a cookie containing user’s name and roles, encrypts it and sends to the browser. Each consequent request, the system gets the cookie, decrypts it’s content, pulls out the user name and roles, creates a security principal and assigns it to the current request. The user then shown a home page.
The attacker’s actions
A potential attacker will read the cookie and begin modifying it according to the algorithm, and sending the modified request to the server. From each server response the attacker will try to conclude whether the cyphertext was decrypted correctly or not. In order to prevent, or at least minimize hacking possibility, our mission is to confuse the attacker. But before that, let’s understand how the attacker will distinguish between valid and invalid state.
Is it right and wrong
When processing the request by our application, two states are possible:
- The cookie is successfully decrypted, the principal is set. The response will be a home page, maybe even with a logged in user name shown on it. In this case it is easy to deduce that the modified cookie is valid.
- Cookie decryption failure. When following according to the attacking algorithm, the only possible exception will be because of invalid padding. In this case, it is not important how you deal with the error: allow the YSOD to appear, redirect to a constant error page (the Microsoft’s workaround), or even show a welcome screen without the user name shown. It doesn’t matter, because the attacker only needs to know that there was some abnormal behavior, which implies invalid decryption.
Without taking a proactive action we will not be able to confuse the attacker. So let’s see what can we do.
Confusing the attacker
To confuse the attacker, we need to make him believe that the invalid request is valid or vice versa. However, I don’t see how we can perform the diversion only using the above two states. Therefore, we need to introduce a third state, which will be different, but will be seen by the attacker as one of the two.
The additional state I’m going to introduce will be “invalid user name”. To create this state, I’m going to confirm, that the user name in the cookie is a valid one. In this case, the decryption will be successful, however, we are going to send back to the browser a generic error page. It is important to show the same error page, even with a random delay, exactly according to the guidelines described in the Microsoft’s workaround. The random delay is used to confuse the attacker even more. One of the ways to detect abnormality is to compare response times. Because exception throwing takes more time than just a simple validation, an attacker can take advantage of it to distinguish between the states.
There are numerous ways to validate the user name. Here are the ones I think are the most appropriate:
- Validate against the users’ repository, like a database table or an Active Directory. The problem is that reading from repository may be is expensive performance-wise, and may hurt system’s scalability. To overcome this, you can cache user’s data in a memory and try validating against the in-memory cache.
- As an addition to the previous method, do not store human-understandable values in the cookie. For example, store user’s GUID instead of the name, and on each request pull out user’s data from the cache or a repository. This way, even if the encryption will be broken, it will be hard to guess GUIDs. Moreover, a GUID can represent a single-use token, pointing to a cache value. This is the most secure way!
- Add a checksum to the value before encrypting it. After decrypting, validate the checksum. Remember, the attacker will send random requests. His chances to guess the checksum is extremely low, so you’ll most certainly will be able to detect an attack.
Now, the attacker will not be able to distinguish between the decryption error and validation error, and won’t be able to take advantage of the vulnerability.
Summing up
The “Padding Oracle” vulnerability is not the simplest to prevent, however, the prevention is possible. Microsoft’s workaround is vital to apply, however, it is not enough in many situations. Preventing the exploit requires thinking and coding
Summing up, here are my recommendations:
- Apply the Microsoft’s workaround. Even though it is not enough, it is vital.
- Do not keep sensitive values on the client. Instead, keep references to values stored on server. For example in a cache or a session variable key.
- Always validate values received from the client. In case of invalid value, do not show explicit message – redirect to a generic error page and log the actual error.
- Ensure DDOS filter activated on your firewall. The attacker will run thousands of requests to the server in a short time. Firewalls are capable of detecting such an activity and blocking the attacker’s IP address, leaving him outside.
Good luck,
Vlad
Yesterday (Sept 18), Microsoft have released a Security Advisory for a newly discovered vulnerability in ASP.NET applications. Following the advisory, Scott Guthrie has published a blog post regarding this vulnerability, detailing an eligible workaround for preventing the exploit.
However, according to the investigation I’ve done during the past couple of days, unfortunately, this workaround is far from being enough for plugging this security hole. In this post I’ll try to briefly (without boring you too much) explain what is this vulnerability about, why the workaround doesn’t work and what is the best way to prevent hacking your app.
At the end of the post, I’m providing a list of links, from which you can learn much more about the issue.
WTF is this “Padding Oracle” vulnerability?
To make the long story short, since encryption algorithms work on blocks of data (usually 8 or 16 bytes per block), the remaining bytes are “padded”. For example, a 5-letter word “TABLE”, will be padded with three bytes to become 8-byte block. I’ll skip the explanation on how exactly it is done – you can read here about that.
“Oracle” is a mechanism inside a cipher, capable of providing Valid or Invalid answer for a given ciphertext. Therefore, “Padding Oracle” is a mechanism, capable to answer, whether the padding of the provided cyphertext is valid or not. There is no relation whatsoever to the Oracle database or the Oracle company.
Again, without going too much into details, this simple Valid/Invalid answer allowed security researchers to create an algorithm, decrypting almost any cypertext encrypted in CBC-mode with PKSC#5 padding (bla…bla…bla…) without knowing the encryption passphrase. It is somewhat similar to brute-force attack, but with much less required checks, taking minutes to complete. The attacking application changes one byte in a cyphertext at a time and sending it to the oracle, asking “is it valid?” till the byte is decrypted.
How is this related to ASP.NET?
Ciphers (encryption algorithms), built in Microsoft in .NET framework, throw a System.Security.Cryptography.CryptographicException with a message “Padding is invalid and cannot be removed” in case of invalid padding. So this is our Oracle for padding!
Now, think of an application saving some encrypted sensitive data in a cookie. The attacker can read this cookie, containing a cyphertext, and play with its bytes sending simulated requests to a server with a modified cookie. The attacker can then analyze the response and to deduce which response means Valid, and which one is Invalid. Hopefully, now you’re beginning to understand the potential of this exploit. Either way, I’ll elaborate on it later.
Does the vulnerability exists only in ASP.NET?
Not at all! As a matter of fact, the vulnerability has first been discovered in JSF (Java Server Faces) framework. It also exists in Java, which throws exceptions on invalid padding.
How does the vulnerability affect me?
Using the vulnerability, the attacker may decrypt all the sensitive data, sent by ASP.NET application to a client, i.e., cookies, ViewState, URL strings, hidden fields etc. Then, the attacker may find your encryption passphrase, change the encrypted data and send the modified content back to the server. For example, the attacker may impersonate himself as a system administrator.
Scott, in his post, also mentions ability to download web.config files from the web site. I actually have no idea how it is possible using this exploit. I personally think it’s a mistake in the article. There is another mechanism preventing downloading *.config files.
How about Microsoft’s workaround?
(This is the most critical section of the post!)
Well, while the workaround contains a really valuable information, relevant for every system (as for not disclosing the real error), and it will prevent the automated tool released by the researchers to hack your system, it will, by far, NOT protect you from a potential attack!
How so? The workaround assumes that the potential attacker will look for an HTTP error response status (500), or for an error page containing a specific exception message. However, it is enough for attacker to recognize an abnormal, or just different system behavior on certain requests.
Let’s get back to our ASP.NET system that stores an encrypted sensitive information in a cookie. Each request, the system will probably decrypt this information and use it. In case the ciphertext in a cookie is invalid, an exception will be thrown, and the system may act according to one of the following scenarios:
- Return a 500 error response - very user unfriendly!
- Return a default ASP.NET YSOD exception page - extremely bad in production environment!
- Return a page stating only the exception’s message - also very bad!
- Return a constant page, stating there was an error, without providing details– a good practice, this is actually the Microsoft’s workaround
- “Swallow” the exception, and behave like the cookie does not exist. The response may be a redirect to another pager, or just a a slightly changed HTML (instead of user’s name, a “login” link) – This is the way ASP.NET Forms Authentication works.
Note that every one of the possible responses is different from the normal one. Even the last scenario I’ve described above, as clean as it is, still returns a distinctively different response. Therefore, an attacker can take advantage of it, and write a simple script that infers this abnormal behavior to an Invalid Oracle’s answer. It is that simple!
So what do we do now?
Unfortunately, this vulnerability is very complex to deal with, because the problem actually lies in encryption algorithms allowing this simple hack. I’m pretty confident, Microsoft will release some kind of patch long before others will, however, I’m not sure how good it will be and how long will it take.
Meanwhile, the best advise I can give you is: do not store any sensitive data on a client (cookies, ViewState, hidden fields, etc), even encrypted. The most important thing is not to store a currently logged in user name and rely on it on further requests (the thing that ASP.NET Forms Authentication does). Hackers can decrypt your cookie, find the encryption passphrase and encrypt back some other value, like “Administrator” (scary, ha?). What you can do instead is to store some bogus value in a cookie, like a GUID of the user name, or even better, a GUID of a session variable that keeps the user name.
It is also possible to implement a double encryption. The attackers can only decrypt the first level of encryption – the second is still unreachable. In this case, the second (the inner) level of encryption should be implemented as clean as possible, without returning any errors.
Conclusion
The new vulnerability is a harsh one. It basically allows a hacker to decrypt you sensitive data without knowing the encryption passphrase. It is not easy to protect your application against the exploit, however it is possible. For more information and for much deeper understanding of the issue, look ad the list of links bellow.
Links
Practical Padding Oracle Attacks paper by Juliano Rizzo and Thai Duong
A great explanation about the Oracle Padding, including an implemented python script
Padding Oracle Exploit Tool (POET) – original tool used for JSF attacks
A video, demonstrating attacking a DotNetNuke site
Scott Hanselman has written a great tutorial on installing and configuring AppFabric Cache (formerly known as “Velocity”). Just follow it, and you’ll have the Cache up and running in no time. There is, however, one issue you’ll have to deal with, when trying to create your own application on a 64-bit operating system. The two assemblies, needed to be references are not accessible from the Visual Studio 2010 “Add Reference” dialog.
Well, if you’ve installed the AppFabric, don’t worry, the assemblies are there. They are indeed in the %windir%\System32\AppFabric folder:
However, from the VS 2010’s “Add Reference”, you won’t see the AppFabric folder:
The thing is, %windir%\System32 is reserved for 64-bit applications. When 32-bit application, such as Visual Studio 2010 is trying to access the directory, is is actually redirected to %windir%\SysWOW64. This is why you’re not seeing the AppFabric folder.
But don’t worry, getting access the real %windir%\System32 directory is simple. Just type “c:\windows\sysnative” in the “Select the file to reference” dialog, and here it is:
The above is true not just for referencing AppFabric Cache client assemblies, but for referencing any assembly in system32 directory. Just remember: Visual Studio 2010 is a 32-bit application, so it runs in WOW mode.
Most of us are local administrators on our machines. However, the UAC (User Access Control) feature of Windows 7 annoys us every time there is a need to run an application as an Administrator. You are probably familiar with the darkening screen, the beep and the “Do you want the following program to make a change….” message popping out of nowhere when choosing “Run as Administrator”. Don’t know how about you, but it just freaks me out.
Fortunately, there is an easy solution, that doesn’t require turning off the great UAC feature. All you need to do is to change the local policy.
Click the Start button, and start typing “Local Security Policy” at the command prompt:
Now, in the Local Security Policy Screen expand Local Policies and choose Security Options. In the list of the security options look for User Account Control: Run all administrators in Admin Approval Mode, which will be enabled by default.
Double click it, and change to Disabled. After a reboot, you are the real administrator – no more UAC annoyances.
This is the message I’ve got while troubleshooting Windows 2008 R2 Server wireless LAN: 
Isn’t Windows Server meant to be a server operating system aimed for IT professionals, that are supposed to know how the Ethernet cable looks like?
And BTW, the Israeli telephone connector is actually larger then the Ethernet connector.
IIS SEO Toolkit is a free and powerful module of the IIS7 Web Platform, helping developers and web site administrators to improve their site’s relevance for the search engines, such as Google or Bing. The module is, basically, crawls through any given web site, downloads all of its content to the local hard drive, analyzes the content for SEO issues and gives a detailed report, including many different details, such as words count, or URLs leading to a certain resource of the analyzed web site.
The toolkit includes many useful capabilities, which can be leveraged to do some interesting stuff, for example, to find easily the Easter Egg, hiding on one of the pages of the Typemock’s site, and maybe win a desired Typemock Isolator license.
Note: I’m publishing this AFTER the draw. Sorry folks, you’ll be able to use this technique only the next time someone announces something similar. The purpose of this article is not to show how to hack web sites, but to demonstrate the power of the IIS SEO Toolkit.
So, the first thing we need, is of course the IIS7, which comes with Windows Vista/2008 or IIS 7.5, which comes with Windows 7/2008 R2. Then, we’ll install the SEO Toolkit. The simplest way to install the both, is by using the Microsoft Web Platform Installer. This amazing free tiny utility from Microsoft installs the all components of the MS Web Platform in a snap. Just select the components and click “Install”:
After installing the SEO toolkit, you’ll find a “Search Engine Optimization” icon under the “Management” Section of the IIS Manager. Make sure you’ve selected the server name and not one of the sites, otherwise you’ll be able to only analyze the selected site.
Clicking the icon brings the SEO main page. Choosing the “Create a new analysis” link will bring the “New Analysis” screen. Type in anything you want for the analysis name, and the Typemock’s web site URL as a Start URL.
Now I click “OK”, and go grab a cup of coffee. Depending on the internet connection and the site load, the crawling may take between couple of minutes and half an hour. Anyway, no intervention is required or desired from my side now. It took about 5 minutes to finish.
By the end of the crawling and analysis, the analysis report dashboard will appear:
I’m not really interested in the site’s SEO violations. What really interests me in the content of the site. Now, my guess is, that we’re supposed to look some resource, containing the word “easter”. I’ll click on the “Query” button on the toolbar, and then choose the “New Query” option from the drop-down menu. In the Query widow, I will enter the query details and click the “Execute” button. My guess now is that I’m done.
I bet, the second result (the HTML page) will lead me directly to the web page, where I need to enter my details to participate in the draw. Right-clicking the row and choosing “View in Browser…” indeed leads me to the desired page.
But I’m curious where the Typemock folks have really hidden the egg. My guess is that the third resource is probably the Easter Egg image. I’m going to double-click the row to open the resource details window. Voila, there it is! The egg image appears in “Content” tab of the resource details window:
Now, there is one thing left – to discover where this egg is hiding. To do it, I’m going to open the “Links” tab, which with no doubt will provide me the URL that contains this image:
This time, I can open the web page selecting the row, and choosing “Show in Browser…” from the “Actions” menu on the top-right corner of the screen.
To summarize, in this particular case it is probably much easier to just surf the web site and find the egg. However, it is much cooler (or geekier) to do this automatically. Anyway, as I’ve written before, I have no intentions whatsoever to show you how to hack web sites. My only intention was to demonstrate the IIS SEO Toolkit, which is an extremely powerful and free tool by Microsoft. It can be used not only for improving your web site relevance to search engines, but also to eliminate different problems with your web site, such as broken links, or heavy pages.
This tool helped me a lot in numerous tasks, I needed to perform lately. I’m sure it will help you with yours.
I’ve spent some time last week trying to solve production-time IIS crashes, caused by one of our web apps. With the great help of Gadi Meir we’ve detected, that some naive code somehow causes an endless recursion, resulting StackOverflowException, causing IIS to crash, and recycle the app pool. With some effort, I’ve finally understood what has happened there, and I’m able to simulate the behavior.
Consider the following code (which was created entirely for demo purposes):
1: public static object GetItem(string key)
2: { 3: if (HttpRuntime.Cache[key] == null)
4: { 5: InsertToCache(key);
6: }
7: return HttpRuntime.Cache[key];
8: }
9:
10: private static void InsertToCache(string key) { 11:
12: HttpRuntime.Cache.Insert(key, "Cached Value", null,
13: DateTime.Now.AddSeconds(10),
14: Cache.NoSlidingExpiration,
15: CacheItemPriority.Normal,
16: RemovedCallback);
17: }
18:
19: private static void RemovedCallback(string key, object value,
20: CacheItemRemovedReason removedReason)
21: { 22: InsertToCache(key);
23: }
This is a simple caching pattern, by which, the entry in the cache is being refreshed every predefined period, without an explicit request. This is very useful when gaining the value from the persistent repository takes some time, and we don’t want any user to wait for response from the server.
So the GetItem function checks whether the entry exists in the cache, and calls the InsertToCache, which will actually get the entry from the repository (here, just inserts some string), and will pass the cache a RemovedCallback, which will be called whenever the entry is removed from the cache. So, when the defined caching period, expires, the RemoveCallback delegate will be called, which, in turn will call the InsertToCache to refill the cache. Pretty straight-forward I think. So what’s the problem?
What is not so frequently realized is that Cache.Insert() method first looks for the key within the existing entries, and if found, removes it. Of course, entry removal causes the invocation of RemovedCallback delegate. Do you see the problem yet? If the InsertToCache will be called with the key, that already exists in the cache, the Insert will cause the RemovedCallback, which in turn will call the InsertToCache which will again call the RemovedCallback…. creating an unintended recursion without any halt condition, causing StackOverflowException. As it turns out, stack overflow in CLR 2 causes IIS application pool to ungracefully crash. I’ve heard it’s not the case with CLR 4, but haven’t got a chance to confirm.
Now, why would we enter the InsertToCache for the second time? As you see, we are checking for existence of the key in the cache, so theoretically, this function should be called only once.
The thing is, in multithreaded app (and every web app is multithreaded by definition), a context switch can happen exactly on line 4, exactly the moment the cache entry was expired. So the thread#1 will indicate that the entry does not exist, and then, the context switch will occur to thread#2, which will also indicate that the entry does not exist. This way, both threads will race into the Cache.Insert(), while the second thread will cause the recursion and the stack overflow.
This is fairly easy to simulate. Run IISRESET and then point two browser windows to the application. It’ll take ASP.NET a couple of seconds to compile the page, so you’ve got 2-3 seconds to open the second browser window after starting the first. Both browsers will wait for the compilation to end, and then will race concurrently into the GetItem function. In rare cases it will work, but in most cases, the app will crash.
There are two ways to solve this particular issue. Either lock the whole GetItem function, but this will have affection on performance (threads will now wait for each other). The other, and much simple and efficient solution is to check the removedReason argument of the callback. When the entry is removed because of expiration the argument will hold an “Expire” reason. If the removedReason=Removed, don’t call InsertToCache, cause this is the method that have just called you.
The point of this post is not to teach you caching design patterns, but to make you aware of the multi-threaded nature of web apps. And as such, in some cases, concurrent calls may have a devastating effect on your app.
Enjoy!
The Developer Academy 4 is behind us, and after completing most of the urgent tasks that I have postponed while being busy preparing my IIS 7 presentation, I’ve finally found a couple of minutes to write this. post
First of all, I think Microsoft stuff has done a great job preparing this event. Discarding some minor issues, everything went extremely smooth.
Second, I’d like to thank to all the people, who have participated in my “The 5 coolest IIS 7 features” talk. I had a great time, hope you too. The recording of the session, done by Sela University, is available here.
I’d like to share with you some of the resources linked to those five IIS 7 features, I’ve talked about during the session:
Web Platform Installer
The Web PI page on the Microsoft Site
Nice blog post about the Web PI by Scott Guthrie
Microsoft Web Platform Installer on the Official Microsoft IIS Site
Web Deployment Tool
Web Deployment Made Awesome: If You're Using XCopy, You're Doing It Wrong – Great blog post and MIX10 session by Scott Hanselman
URL Rewrite Module
Failed Request Tracing
Failed Request Tracing on IIS7 – Chanel9 Video
SEO Toolkit
Search Engine Optimization Toolkit : The Official Microsoft IIS Site
IIS Search Engine Optimization Toolkit - ScottGu's Blog – Great Post
Enjoy!
I’ve just spent two days and a night between them on the client site, trying to solve a severe performance problem with the application, that Netwise (the company I work for) has developed. It is an ASP.NET web application, running on Windows 2008 R2 with IIS 7.5 on a 4-CPU/8G machine.
We’ve fine-tuned every possible IIS and ASP.NET setting, defined data caching, output caching, profiled almost every line of the code, however, the application still ran very badly, even when tested with 50 concurrent requests. The symptoms were:
- Slow response – a web service, getting some data from the cache, manipulating it and returning result could take 8 seconds to respond (3ms when profiled on development environment)
- Sometimes it ran fine for a minute, and then, peaked to 20 sec per request
- Very few request/second rate – 30-50 instead of 500-1000
- About 10-20 thrown exceptions /sec (windows performance monitor)
- IIS crashes on KERNELBASE.DLL module – no info whatsoever on the net about it
- Too much time spent in JIT – unexplained – no dynamic assemblies loaded
We’ve used one of the well-known Application Performance Monitors (no names), this client uses to monitor .NET apps in production time. It was already preinstalled and configured on this machine. Those tools are basically using ELT hooks to monitor the CLR, which supposed to introduce about 5% of performance penalty on .NET apps.
The tool has indicated a lot of time spent in functions performing very simple LINQ-to-objects queries, which was ridiculous.
After hours of investigations, kilos of junk food, and almost being despaired, I’ve disabled the tool, and was amazed to discover, that suddenly the application performed amazingly well! No crashes, less than 100ms response time and more than a 1000 request/second!
It appears that the dreaded tool was either in buggy version, or poorly configured, and caused throwing tens of exceptions per request causing the app to perform badly. Look for my previous post to see how exceptions may hurt web apps performance. The funny thing was that this tool has indicated that there were functions in our code that perform badly. The tool’s producer should test its own tool before testing others.
The conclusion is: be aware, that similar tools, either buggy or misconfigured may drastically hurt the performance of your application. They hook to the CLR on a very low level, and may cause very strange issues. I’m not saying do not use these tools. On the contrary, in other cases they’ve helped me a lot to discover application performance issues. Just before rewriting half of the application code, try disabling them to see what happens.
In addition, it was probably not a very wise thing to blame an app for such a bad performance, although that is what I usually tend to do – look for pitfalls in my own stuff before blaming the others.
Want to know more about IIS7? Come to see me ad Developers Academy 4.
(Update: Thanks to Avi Pinto for pointing out at a stupid mistake regarding catching exceptions in every method. What I meant was that it is a bad practice catching exceptions in every method. Don’t catch exceptions if you have nothing to do with them)
Yes, exceptions are one of most common sources of performance issues of any .NET application, and especially of ASP.NET Web Applications. Without going into details, every thrown exception, either handled or not, introduces a serious penalty on the code execution time.
Consider the following two simplistic functions:
private static int Divide(int a, int b)
{
if (b == 0) return int.MaxValue;
return a / b;
}
private static int DivideEx(int a, int b)
{
try
{
return a / b;
}
catch (DivideByZeroException ex)
{
return int.MaxValue;
}
}The first one checks the input, and returns the maximum value of integer in case of invalid denominator. The second one, relies on the user input, and handles the exception. Unfortunately, in my practice I’ve seen much more of the second type of function. Now let’s see what this does to the execution performance.
I’m gong to use the following method to benchmark the performance of the above functions:
private static long ProfileDivideFunc(Func<int, int, int> func)
{
Random rnd = new Random(DateTime.Now.Millisecond);
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 10000; i++)
{
func(rnd.Next(0, i), 0);
}
sw.Stop();
return sw.ElapsedMilliseconds;
}Nothing fancy here. The only “interesting” thing is that the method accepts a delegate and invokes it. Inside the method, I’m just running the tested function 10,000 times, passing a random integer as nominator, and zero as denominator. Practically, I’m measuring the performance penalty of throwing and catching 10,000 exceptions.
The results are pretty much as expected. On my D-Core/4G/Win7 x64 laptop, the first function test returns 0ms, whereas the second function (the one with exceptions), takes 422ms to complete.
It sounds like nothing, however, consider what happens, in ASP.NET app, when 500 or 1,000 concurrent requests are being handled. Also take into consideration the fact that usually we don’t catch exceptions in every method (catching exception in every method is a very bad practice), so it bubbles all the way to some top-level-exception-catching-mechanism. In addition, it has to be logged, which also takes some CPU cycles.
I’ve recently seen a web app, that thrown and logged 3 exceptions on each and every request. Needless to mention how the performance was improved after eliminating the sources for those exceptions.
So here are couple of recommendations:
- Never, and I repeat, never, throw exceptions. Use response error codes if you need to indicate and error.
- Verify user input. Don’t handle exceptions following bad input.
- Try eliminating usage of functions that throw exceptions in case of invalid input, like Parse(). Use TryParse() instead.
- Watch for exceptions, log them and eliminate their sources ASAP.
- There is nothing worst than unhandled exceptions in ASP.NET application. Catch everything in some top-level mechanism, like Application_Error of global.asax, or connect to AppDomain.CurrentDomain.UnhandledException event to catch and log them all.
The most important this is: remember, Exceptions are EXCEPTIONS from the normal, so they should be treated accordingly. They are not NORMAL at runtime.
Have fun.
Want to know why you should use IIS7 ? Come to see me at Developers Academy 4.
More Posts
Next page »