This is an interesting case that was solved thanks to the help of Nitsan Raz – an all mighty system administrator and SharePoint admin.
So I was asked to investigate a performance problem at one of my regular clients.
To demonstrate the performance problem, the client’s developers showed me a simple WinForm application (on the server), that created an instance of an SPSite object. the code had 1 line of interest, which is:
That line alone took 30 seconds to execute! What’s more interesting, was that subsequent calls was very fast to execute. It was just the first one that took so long.
First thing we though was to isolate the location of the hang. Was it on the application side, or DB side? We ran SQL profiler and saw that SQL queries took like a millisecond to execute. That test eliminated the option of DB problem.
Having figured that out, we decided to run some tests on the application server (SharePoint Web Front End). After hours of time measures and watching endless performance counters go up and down the screen, we decided to monitor network activity.
We found 2 interesting lines in the log. These were DNS queries for crl.microsoft.com. How is that has to do with our application?
It turns out, that SharePoint’s assemblies are signed using a Authenticode, and that every time they are loaded into an app domain, the system tries to verify their validity by asking Microsoft for Certificate Revocation List (CRL). This is a list that contains certificates that are no longer valid.
Our environment was disconnected from the internet, so obviously we can’t reach Microsoft’s servers to get the CRL. That process failed on the very beginning , when trying to lookup Microsoft.com on DNS. The DNS didn’t respond on time, so we were waiting for the lookup to time out… guessing what’s the time out for DNS lookups? 15 seconds. This explains why we waited 30 seconds (for 2 assemblies to load).
There are several ways we can deal with this: (You should choose only one of the following)
1. Add the following configuration to the configuration file for the application that is loading the files: (This could be MyApp.config, web.config, machine.config, etc… – depending on your application)
This solution requires a hotfix for .NET 2.0, and is already built into later versions. More info:
3. Add a fake DNS record that points to yourself (127.0.0.1), that will shorten the time to fail.
4. Pre-fetch CRL files, so that it will be pulled locally and not from the internet.
5. Allow internet access, obviously (if only to crl.microsoft.com)
For more info on assembly signing: