itaysk

I recently wrote a post regarding a certain performance issue with SharePoint that we solved. I since got some comments and questions about when is this problem relevant.

Well.. It’s practically everywhere ,in an isolated environment, and It’s very common for development machines (usually virtual) to be disconnected. The fixes suggested in that post can help you speed up your work in the following cases:

• A desktop application (WinForm\Console) that uses SharePoint APIs.
• STSADM.exe.
• PowerShell work with SharePoint.
• First hit of your SharePoint site.
• IIS reset\Recycle app pool when deploying to SharePoint.
• Anything else I didn’t think of, that uses SharePoint API’s in it’s own process (app domain to be precise).

This is an interesting case that was solved thanks to the help of Nitsan Raz – an all mighty system administrator and SharePoint admin.

Problem

So I was asked to investigate a performance problem at one of my regular clients.

To demonstrate the performance problem, the client’s developers showed me a simple WinForm application (on the server), that created an instance of an SPSite object. the code had 1 line of interest, which is:

SPSite site = new SPSite("http://mySite");

That line alone took 30 seconds to execute! What’s more interesting, was that subsequent calls was very fast to execute. It was just the first one that took so long.

Cause

First thing we though was to isolate the location of the hang. Was it on the application side, or DB side? We ran SQL profiler and saw that SQL queries took like a millisecond to execute. That test eliminated the option of DB problem.

Having figured that out, we decided to run some tests on the application server (SharePoint Web Front End). After hours of time measures and watching endless performance counters go up and down the screen, we decided to monitor network activity.

We found 2 interesting lines in the log. These were DNS queries for crl.microsoft.com. How is that has to do with our application?

It turns out, that SharePoint’s assemblies are signed using a Authenticode, and that every time they are loaded into an app domain, the system tries to verify their validity by asking Microsoft for Certificate Revocation List (CRL). This is a list that contains certificates that are no longer valid.
Our environment was disconnected from the internet, so obviously we can’t reach Microsoft’s servers to get the CRL. That process failed on the very beginning , when trying to lookup Microsoft.com on DNS. The DNS didn’t respond on time, so we were waiting for the lookup to time out… guessing what’s the  time out for DNS lookups? 15 seconds. This explains why we waited 30 seconds (for 2 assemblies to load).

Solution

There are several ways we can deal with this: (You should choose only one of the following)

1. Add the following configuration to the configuration file for the application that is loading the files: (This could be MyApp.config, web.config, machine.config, etc… – depending on your application)

<configuration> 
  <runtime> 
    <generatePublisherEvidence enabled="false"/> 
  </runtime> 
</configuration>

This solution requires a hotfix for .NET 2.0, and is already built into later versions. More info:
http://support.microsoft.com/kb/936707

2. Turn off CRL checks (the the user level). Internet Options->Advanced tab->Security section under the Settings area->Check for publisher’s certificate revocation->turn that off (uncheck). 

3. Add a fake DNS record that points to yourself (127.0.0.1), that will shorten the time to fail.

4. Pre-fetch CRL files, so that it will be pulled locally and not from the internet.

Download these:
http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl
http://crl.microsoft.com/pki/crl/products/CodeSignPCA2.crl
Add to certificate store:
certutil -addstore CA CodeSignPCA.crl 
certutil -addstore CA CodeSignPCA2.crl

5. Allow internet access, obviously (if only to crl.microsoft.com)

Credits

Dirk Van den Berghe
Nitsan Raz

For more info on assembly signing:
http://blogs.msdn.com/b/junfeng/archive/2006/03/11/549355.aspx