Chess – Deterministic parallel testing

2010/03/28

no comments

Chess – Deterministic parallel testing

 Chess, MS research, Testing, .net, parallel

modern software development is drifting in the direction of Parallel Computing.

even those the tools and libraries for Parallel Computing is continuously improving, 

we still left with the old problem of how to test the non deterministic execution nature

of Parallel Computing programming.

 

this post will focus on the Chess testing framework, which is trying to solve the

non deterministic testing issue.

 

you can download the chess framework from here.

the code for this post is available here.

 

Why does load testing isn’t enough?

common practice for testing Parallel Computing is to use load test.

this practice is a good one and shouldn’t be abandoned,

but it has some profound weaknesses in terms of:

  • reproducing the bugs.
  • and the ability to guaranty the optimal coverage of thread switching scenarios,
    sometimes the rare switching timing will never happens during the load testing,
    but they
    are exactly the one that will happens on the customer production environment.

 

What does Chess bring to the table?

as analogy I can describe the load testing as statistic weapon,

while the Chess can be describe as smart accurate weapon.

Chess will run the test code in loop, each time it will control the

context switching (through custom scheduler) in a way that it will be fully deterministic

and therefore can be reproduce.

this way Chess will gain a better thread switching coverage which will lead for better

code quality (and it will required shorter execution time, because it doesn’t have to run for days to fill up the statistic).

 

How to define Chess test?

Chess test is define as regular MS Test with one additional attribute

Code Snippet
  1. [TestMethod()]
  2. [HostType("Chess")]
  3. public void ParallelTest()
  4. {
  5.     MyLogic target = new MyLogic();
  6.     decimal expected = target.Read();
  7.     var trd = new Thread (() => target.Deposit(500));
  8.     trd.Start();
  9.     target.Withdraw(500);
  10.     trd.Join();
  11.     decimal actual = target.Read();
  12.     Assert.AreEqual(expected,actual);
  13. }

line 2, decorate the test for running under Chess.

we already saw this technique with Moles.

 

running the test without the Chess hosting will result with success 99.9% of the times.

while adding Chess hosting, the test will consistently fail each time it run.

 

if you want to see what Chess is doing during the test execution

you can add the following attribute

[TestProperty("ChessDebug","true")] (as shown in line 3)

Code Snippet
  1. [TestMethod()]
  2. [HostType("Chess")]
  3. [TestProperty("ChessDebug","true")]
  4. public void ChessParallelTest()
  5. {
  6.     …
  7. }

 

How to reproduce failures?

when ever Chess test execution failed, the test failure page supply useful information

of how to create reproduced failure test (it is recommended to create new test for reproducing

because this test will not goes though all the steps the original Chess test goes, and we

don’t want to loose any of those steps).

image

the reproduced test decoration will look as follow:

Code Snippet
  1.         [TestMethod()]
  2.         [HostType("Chess")]
  3.         [TestProperty("ChessDebug","true")]
  4.         [TestProperty("ChessMode","Repro")]
  5.         [TestProperty("ChessBreak","BeforePreemption,AfterPreemption")]
  6.         #region ChessScheduleString (not human readable)
  7.         [TestProperty("ChessScheduleString",@"bpilaiaaaaaaaaaaaeaaonlnahgabmejjgcfcgcpgnmkhlhpekpfeknhoahekbaiiagabdcenijaeabaommbiimnogjcombngjeh
  8. cdcjklckibmkgffggffnggbgeammonjnlmphnohloplnphnohloplnphlkdljneochphnpppdpfmgggeabgmpgmoeknkmjjocbia
  9. kkmibpdphohmbpdpcchomnfpodnhpidfpappbpndjpppphkpcjdpmnoplpifpopoglpmfkpmdpplplkpkdlpodldfpnhplnnmaij
  10. lfpplfpepplpjopooeopcchnjjppjlohpppohpaannppfghgkfaaaaaa")]
  11.         #endregion
  12.         public void ReproduceChessParallelTest()
  13.         {
  14.             …
  15.         }

lines 6-11, are non user readable decoration, which is the Chess instruction for
reproducing the failure (setting the right context switching).

line 4, is setting the reproduce mode.

line 5, is not mandatory but it is one of the great Chess feature,
it is actually instructing Chess to place break points at the right spots, so you no longer
have to search in the darkness for the right place and time for your break point.
obviously this save tones of debugging hours.

 

Summary

Chess, MS research, Testing, .net, parallelas you do write parallel coding, it will be wise to add Chess into your

testing toolbox, for gaining better deterministic of the parallel test result.

Chess is a powerful tool which will lead your products to better quality and shorter

debug efforts.

one last issue is that currently there is no bits compatible with VS 2010 RC so we

will have to be patient till we got it.

 

Downloads and Learn More

you can download chess from here.

the code for this post is available here.

the Chess web page is available here.

Channel 9 talk is available here.

Chess at the PDC is available here.

 

kick it on DotNetKicks.com Shout it


Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>