I’m getting this question a lot, how to run multiple tests on different machines in parallel?
Every customer that I work with has over 100 tests and some has more than 1000+.
So how to improve test execution time? running 1000+ tests can take a while and we want to get the results as fast as possible.
So, for Unit Tests there is a nice trick (Wrote about it in 2010 –>Run Unit Tests in Parallel using Visual Studio 2010) that allow you to run those tests in parallel on the same machine.
But what about Coded UI Tests? you most know we can’t run more than one test at time (on the same machine). So how to run those tests on different machines in parallel?
First let’s start with adding Test Settings to our Test Project, as you know from the Visual Studio 2012 Tests Settings File isn’t created automatically for Coded UI Test Project (Just for Web and Load Test Project).
Create Test Settings File For Remote Execution
Create new Coded UI Test Project, right click on the solution file and choose Add New Item.
Choose the Test Settings section, give it a proper name and click add
Now, even if you added the Test Settings to your solution it’s still not active, you need to select that settings file in order to activate it.
From the Test menu, choose Test Settings and “Select Test Settings File”.
After you select the settings file you will see the settings file marked – this means the settings is active.
Now open the file and open the Roles section, change the execution method to Remote and specify the Controller.
Let’s assume your controller has two Agents connected to him, if you run 8 tests, what will happened?
Only one agent will run those tests….. but if you run 20 tests? then both Agents will run 10 tests each.
Why??? The reason is the Test Controller has a setting define the minimum tests per agent and his value is – 10.
In the next section we’ll learn how to customize it.
Setup Parallel Execution
To change the amount of tests each agent can execute we need to access the Controller machine and locate the “QTController.exe.config” file, located in :
C:\Program Files (x86)\Microsoft Visual Studio 1*.0\Common7\IDE
Open the QTController.exe.config, locate the appSettings section and add the following key:
<add key=”MinimumTestsPerAgent” value=”1″/>
So we said the default value is 10, changing it to 1 will better spread the tests across agents.
Now if you run 8 tests (and have two agents), what will happened? each agent will run 4 tests!
Important : If you run 400 tests, what will happened? each Agent will assign with a set of 100 tests, once the first Agent will finish his 100 tests the Controller will send additional 100 tests.
This is because the Buckets size default is 100.
Let’s assume the 100 tests assigned to Agent 1 will take 2 hours, while Agent 2 already finished running the rest 300 tests…. how can we change the Bucket size so each agent will assign with just one test at a time?
Change Buckets Size
Open Test Settings File in XML editor, just right click on the Test Settings File, Open with and choose XML Editor.
Locate Execution section, add new element:
Any amount of tests you run will be spread across agents one test at a time.