WCF scaling – check your counters
A few months ago, I posted about the issue of WCF scaling and how it is affected by the throttling behavior, by ASP.NET settings (when hosted in IIS), and how it uses the thread pool.
Today I don’t want to talk about the scaling problem, but rather give you a small tip about how to check it – use performance counters.
WCF was shipped with built-in performance counter that show you various counters regarding throttling, security, MSMQ , WS-Reliable messages, and transactions.
To View these counters do the following:
1. Open your service configuration and add the following section in the <system.serviceModel>
2. Start your WCF service
3. Open the performance monitor utility (perfmon.exe)
4. Add a new counter. In the counters list, look for the counters that begin with “ServiceModel”, you should see three types of counters: Endpoint, Operation, and Service
- Endpoint counters lets you monitor a counter for a specific endpoint of your service
- Operation counters lets you monitor a counter for a specific operation of your service
- Service counters lets you monitor a counter for an entire service
5. For now, select the ServiceModelService counter for your version of WCF (ServiceModelService 188.8.131.52 for WCF 3.5, and ServiceModelService 184.108.40.206 for WCF 4), select the service that you started from the instances list, and click Add.
6. You will now see the performance counters in your graph. WCF performance counters have mixed ranges, so I usually prefer looking at them using the report view. So click the graph type button and select Report. You should see something like this:
7. Start you client application, and check how the counters change. The significant calls and instances counters are:
- Calls. The number of requests handled by the service since it was first activated.
- Calls Duration. Average latency of the service.
- Calls Faulted. Number of faulted requests.
- Calls outstanding. The number of requests currently processed.
- Instances. The number of service instances that are currently “alive” (running and idle).
- Percent of max concurrent calls. The percent of the achieved throttling. When you hit 100%, you will not be able to handle any more calls. The percentage counters were added in WCF 4.
For example, I ran a load test application on my service and after a minute or two got this result in the report:
You can see that I reached 100% throttling (100.781 percent to be exact), and I have 128 calls outstanding (currently processed requests) – this is because I have an 8 core machine, so the default max calls throttle is 16*8=128.
You can also see these 128 requests are handled by 128 instances, which means I’m probably using a perCall instancing mode. If I had used a single instancing mode, I would have seen only one instance.
Note: I have 128 instances and according to the default throttling settings the max instances should be 928 (max calls + max sessions = 128 + 800 = 928), so I should see 13% instance throttling, however the percent of concurrent instances is 0.
The reason the counter is zero is that if you don’t add the serviceThrottling behavior to your service, the default value of the max instances is set to Int32.Max which is 2147483647, and 128 out of 2147483647 is zero percent!. One you add the serviceThrottling behavior, the max instances will be calculated as the sum of the max calls and max sessions, unless you specify it directly.
So run your own load test on your servers, and see when you are hitting the throttling limit. After you change the throttling settings, retest your service, and verify that your average call duration is not affected by the increase in running threads.