Understanding Load Test Summary Numbers

Surprise

I was a bit surprised recently analyzing load test results. We ran a simple load test, consisting of a single web test with 3 requests. Test ran for 10 minutes under constant load of 5 users. After test completion we tried to compare test results with actual work done by server. We found some inconsistency in the result numbers.

In summary you can see 160 Total Tests and 483 Total Request. This already looking odd. Just to remind you we have 3 requests per test, 160*3 = 480 (???).

Investigation

To investigate this issue little further, we add Request Plugin and Web Test Pluguin to our test.

In PreWebTest event we logged 164 tests, in PostWebTest, we logged 160 tests. It appears that Total Tests number in Summary take in account only completed test. There were actually 4 more tests started, but have not enough time to complete all requests.

RequestPlugins results were even more interesting. We logged 486 requests in PreRequest event and 486 responses in PostRequest event. The same number of requests (486) was logged in IIS log. How about 483 Requests reported in summary??? Apparently Total Requests number count all requests that were completed, pass validation (successfully or not) and logged in local test results store. 6 additional request were completed and passed validation, but were not logged in time.

Conclusion

Load test summary numbers looks inconsistent in my opinion. I think it caused by misleading definition of counters. There is no direct relation between Total Tests and Total Requests. Just to make it clear, there is no bug in counters values. It is form of presentation that makes it unclear.

If you need to count actual requests number (as we did), you can not rely on numbers in summary. To solve this you can use test plugins or perform iteration based load test.