WCF scaling – check your counters

August 11, 2011

A few months ago, I posted about the issue of WCF scaling and how it is affected by the throttling behavior, by ASP.NET settings (when hosted in IIS), and how it uses the thread pool.

Today I don’t want to talk about the scaling problem, but rather give you a small tip about how to check it – use performance counters.

WCF was shipped with built-in performance counter that show you various counters regarding throttling, security, MSMQ , WS-Reliable messages, and transactions.

To View these counters do the following:

1. Open your service configuration and add the following section in the <system.serviceModel>

<diagnostics performanceCounters="All"/>

2. Start your WCF service

3. Open the performance monitor utility (perfmon.exe)

4. Add a new counter. In the counters list, look for the counters that begin with “ServiceModel”, you should see three types of counters: Endpoint, Operation, and Service

- Endpoint counters lets you monitor a counter for a specific endpoint of your service

- Operation counters lets you monitor a counter for a specific operation of your service

- Service counters lets you monitor a counter for an entire service

5. For now, select the ServiceModelService counter for your version of WCF (ServiceModelService 3.0.0.0 for WCF 3.5, and ServiceModelService 4.0.0.0 for WCF 4), select the service that you started from the instances list, and click Add.

image

6. You will now see the performance counters in your graph. WCF performance counters have mixed ranges, so I usually prefer looking at them using the report view. So click the graph type button and select Report. You should see something like this:

image

7. Start you client application, and check how the counters change. The significant calls and instances counters are:

- Calls. The number of requests handled by the service since it was first activated.

- Calls Duration. Average latency of the service.

- Calls Faulted. Number of faulted requests.

- Calls outstanding. The number of requests currently processed.

- Instances. The number of service instances that are currently “alive” (running and idle).

- Percent of max concurrent calls. The percent of the achieved throttling. When you hit 100%, you will not be able to handle any more calls. The percentage counters were added in WCF 4.

For example, I ran a load test application on my service and after a minute or two got this result in the report:

image

You can see that I reached 100% throttling (100.781 percent to be exact), and I have 128 calls outstanding (currently processed requests) – this is because I have an 8 core machine, so the default max calls throttle is 16*8=128.

You can also see these 128 requests are handled by 128 instances, which means I’m probably using a perCall instancing mode. If I had used a single instancing mode, I would have seen only one instance.

Note: I have 128 instances and according to the default throttling settings the max instances should be 928 (max calls + max sessions = 128 + 800 = 928), so I should see 13% instance throttling, however the percent of concurrent instances is 0.
The reason the counter is zero is that if you don’t add the serviceThrottling behavior to your service, the default value of the max instances is set to Int32.Max which is 2147483647, and 128 out of 2147483647 is zero percent!. One you add the serviceThrottling behavior, the max instances will be calculated as the sum of the max calls and max sessions, unless you specify it directly.

So run your own load test on your servers, and see when you are hitting the throttling limit. After you change the throttling settings, retest your service, and verify that your average call duration is not affected by the increase in running threads.

kick it on DotNetKicks.com Shout it
Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

6 comments

  1. MichelFebruary 24, 2012 ב 3:03 pm

    Great article! Just one question…what’s the impact of using performanceCounters on my service ? I mean, could this be a performace problem if I have a lot of requests on my service ?

    Reply
  2. Ido FlatowFebruary 24, 2012 ב 3:32 pm

    Michel,

    Performance counters hava a small, negligible, hit on the overall .net application performance.

    You can read more here:
    http://stackoverflow.com/questions/290634/what-is-the-performance-hit-of-performance-counters

    Reply
  3. SurendraMay 9, 2012 ב 7:46 am

    Great post!

    Percent of max concurrent calls is 100.781 in your case. So seems like your service got 129 concurrent calls(100*129/128).

    This makes me believe that this counter also gives me idea of my queue length(calls waiting to be executed) at any point of time. For ex. if this counter is 300%, and my max concurrent setting is 128, then 256 calls are waiting in the queue. Is that understanding correct?

    Reply
  4. TimJanuary 10, 2013 ב 2:12 pm

    Awesome article thanks simple but effective

    Reply
  5. http://www.interhero.cn/forum.php?mod=viewthread&tid=2323973October 16, 2013 ב 2:00 pm

    WCF scaling – check your counters – Ido Flatow’s Blog Veni Vidi Scripsi

    Reply