Chess - Deterministic parallel testing
Chess - Deterministic parallel testing
modern software development is drifting in the direction of Parallel Computing.
even those the tools and libraries for Parallel Computing is continuously improving,
we still left with the old problem of how to test the non deterministic execution nature
of Parallel Computing programming.
this post will focus on the Chess testing framework, which is trying to solve the
non deterministic testing issue.
you can download the chess framework from here.
the code for this post is available here.
Why does load testing isn't enough?
common practice for testing Parallel Computing is to use load test.
this practice is a good one and shouldn't be abandoned,
but it has some profound weaknesses in terms of:
- reproducing the bugs.
- and the ability to guaranty the optimal coverage of thread switching scenarios,
sometimes the rare switching timing will never happens during the load testing,
but they are exactly the one that will happens on the customer production environment.
What does Chess bring to the table?
as analogy I can describe the load testing as statistic weapon,
while the Chess can be describe as smart accurate weapon.
Chess will run the test code in loop, each time it will control the
context switching (through custom scheduler) in a way that it will be fully deterministic
and therefore can be reproduce.
this way Chess will gain a better thread switching coverage which will lead for better
code quality (and it will required shorter execution time, because it doesn't have to run for days to fill up the statistic).
How to define Chess test?
Chess test is define as regular MS Test with one additional attribute
Code Snippet
- [TestMethod()]
- [HostType("Chess")]
- public void ParallelTest()
- {
- MyLogic target = new MyLogic();
- decimal expected = target.Read();
- var trd = new Thread (() => target.Deposit(500));
- trd.Start();
- target.Withdraw(500);
- trd.Join();
- decimal actual = target.Read();
- Assert.AreEqual(expected,actual);
- }
line 2, decorate the test for running under Chess.
we already saw this technique with Moles.
running the test without the Chess hosting will result with success 99.9% of the times.
while adding Chess hosting, the test will consistently fail each time it run.
if you want to see what Chess is doing during the test execution
you can add the following attribute
[TestProperty("ChessDebug","true")] (as shown in line 3)
Code Snippet
- [TestMethod()]
- [HostType("Chess")]
- [TestProperty("ChessDebug","true")]
- public void ChessParallelTest()
- {
- ...
- }
How to reproduce failures?
when ever Chess test execution failed, the test failure page supply useful information
of how to create reproduced failure test (it is recommended to create new test for reproducing
because this test will not goes though all the steps the original Chess test goes, and we
don't want to loose any of those steps).
the reproduced test decoration will look as follow:
Code Snippet
- [TestMethod()]
- [HostType("Chess")]
- [TestProperty("ChessDebug","true")]
- [TestProperty("ChessMode","Repro")]
- [TestProperty("ChessBreak","BeforePreemption,AfterPreemption")]
- #region ChessScheduleString (not human readable)
- [TestProperty("ChessScheduleString",@"bpilaiaaaaaaaaaaaeaaonlnahgabmejjgcfcgcpgnmkhlhpekpfeknhoahekbaiiagabdcenijaeabaommbiimnogjcombngjeh
- cdcjklckibmkgffggffnggbgeammonjnlmphnohloplnphnohloplnphlkdljneochphnpppdpfmgggeabgmpgmoeknkmjjocbia
- kkmibpdphohmbpdpcchomnfpodnhpidfpappbpndjpppphkpcjdpmnoplpifpopoglpmfkpmdpplplkpkdlpodldfpnhplnnmaij
- lfpplfpepplpjopooeopcchnjjppjlohpppohpaannppfghgkfaaaaaa")]
- #endregion
- public void ReproduceChessParallelTest()
- {
- ...
- }
lines 6-11, are non user readable decoration, which is the Chess instruction for
reproducing the failure (setting the right context switching).
line 4, is setting the reproduce mode.
line 5, is not mandatory but it is one of the great Chess feature,
it is actually instructing Chess to place break points at the right spots, so you no longer
have to search in the darkness for the right place and time for your break point.
obviously this save tones of debugging hours.
Summary
as you do write parallel coding, it will be wise to add Chess into your
testing toolbox, for gaining better deterministic of the parallel test result.
Chess is a powerful tool which will lead your products to better quality and shorter
debug efforts.
one last issue is that currently there is no bits compatible with VS 2010 RC so we
will have to be patient till we got it.
Downloads and Learn More
you can download chess from here.
the code for this post is available here.
the Chess web page is available here.
Channel 9 talk is available here.
Chess at the PDC is available here.
