How To: Run Tests On Different Machines In Parallel

18/06/2014

I’m getting this question a lot, how to run multiple tests on different machines in parallel?

Every customer that I work with has over 100 tests and some has more than 1000+.
So how to improve test execution time? running 1000+ tests can take a while and we want to get the results as fast as possible.
So, for Unit Tests there is a nice trick (Wrote about it in 2010 –>Run Unit Tests in Parallel using Visual Studio 2010) that allow you to run those tests in parallel on the same machine.

But what about Coded UI Tests? you most know we can’t run more than one test at time (on the same machine). So how to run those tests on different machines in parallel?

First let’s start with adding Test Settings to our Test Project, as you know from the Visual Studio 2012 Tests Settings File isn’t created automatically for Coded UI Test Project (Just for Web and Load Test Project).

Create Test Settings File For Remote Execution

Create new Coded UI Test Project, right click on the solution file and choose Add New Item.

image

Choose the Test Settings section, give it a proper name and click add

SNAGHTMLefe7d77

Now, even if you added the Test Settings to your solution it’s still not active, you need to select that settings file in order to activate it.

From the Test menu, choose Test Settings and “Select Test Settings File”.

image

After you select the settings file you will see the settings file marked – this means the settings is active.

image

Now open the file and open the Roles section, change the execution method to Remote and specify the Controller.

image

Scenario

Let’s assume  your controller has two Agents connected to him, if you run 8 tests, what will happened?

Only one agent will run those tests….. but if you run 20 tests? then both Agents will run 10 tests each.

Why??? The reason is the Test Controller has a setting define the minimum tests per agent and his value is – 10.

In the next section we’ll learn how to customize it.

Setup Parallel Execution

To change the amount of tests each agent can execute we need to access the Controller machine and locate the “QTController.exe.config” file, located in :

C:\Program Files (x86)\Microsoft Visual Studio 1*.0\Common7\IDE

Open the QTController.exe.config, locate the appSettings section and add the following key:

<add key=”MinimumTestsPerAgent” value=”1″/>

image

So we said the default value is 10, changing it to 1 will better spread the tests across agents.

Scenario

Now if you run 8 tests (and have two agents), what will happened? each agent will run 4 tests!

Important : If you run 400 tests, what will happened? each Agent will assign with a set of 100 tests, once the first Agent will finish his 100 tests the Controller will send additional 100 tests.

This is because the Buckets size default is 100.

Let’s assume the 100 tests assigned to Agent 1 will take 2 hours, while Agent 2 already finished running the rest 300 tests…. how can we change the Bucket size so each agent will assign with just one test at a time?

Change Buckets Size

Open Test Settings File in XML editor, just right click on the Test Settings File, Open with and choose XML Editor.

image

Locate Execution section, add new element:

<Buckets size=”1″/>

image

Scenario

Any amount of tests you run will be spread across agents one test at a time.

Enjoy.

Add comment
facebook linkedin twitter email

Leave a Reply