A Mini-Review-Benchmark of Ruby’s Different Testing Frameworks

[Note: this post was cross posted from my main blog at IronShay.com]

On of the most shining features of the Ruby language for .NET developers is, in my opinion, its testing frameworks. Ruby has got such amazing testing frameworks that it is such a shame that people still use other languages to test code.

The goal of this post is simple – to show you how to test a given code in various different frameworks, so you can choose the one for you. In the meanwhile I will also take the amount of time taken for each framework to run the tests and compare the times at the end.

For that I’m using my Ruby implementation of choice – IronRuby RC4 (1.8.6 compatible) and my powerful computer (running Windows 7 64-bit). Each testing framework will contain the same 7 tests of the tested code and be executed from the command line.

Note: The testing frameworks I bring in this post are my own random picks. I tried to bring the popular ones and some other small but interesting ones. There are more testing frameworks in Ruby and if you think I should add another ones, let me know and I’ll add them to the list too.

The Tested Code

The code I’m going to test is a C# code (IronRuby FTW!) which resides in an assembly named WaterHelper.dll. The code is as follows:

namespace Demo
{
    public class WaterHelper
    {                
        public bool IsWaterBoiled(decimal celsius)
        {
            return celsius >= 100;
        }

        public bool IsWaterFrozen(decimal celsius)
        {
            return celsius <= 0;
        }

        public string GetWaterStatus(decimal celsius)
        {
            if (IsWaterBoiled(celsius))
            {
                return "Steam";
            }
            else if (IsWaterFrozen(celsius))
            {
                return "Ice";
            }
            return "Liquid";
        }
    }
}

Not much of complication here – three methods that do some water temperature related calculation.

Test::Unit

Official site: http://ruby-doc.org/stdlib/libdoc/test/unit/rdoc/classes/Test/Unit.html

The first testing framework I’m using is Ruby’s built-in one – Test::Unit. It is very similar to NUnit (in .NET) or JUnit (in Java).

The Test::Unit test code which tests the WaterHelper class is as follows:

require 'test/unit'
require "WaterHelper.dll"

class TC_WaterHelper < Test::Unit::TestCase
  def setup
    @instance = Demo::WaterHelper.new
  end
  
  def test_water_boiling_point
    result = @instance.is_water_boiled(100)
    assert_equal true, result
  end
  def test_water_is_boiled
    result = @instance.is_water_boiled(150)
    assert_equal true, result
  end
  
  def test_water_freezing_point
    result = @instance.is_water_frozen(0)
    assert_equal true, result
  end
  def test_water_frozen
    result = @instance.is_water_frozen(-50)
    assert_equal true, result
  end
  
  def test_water_status_steam
    result = @instance.get_water_status(300)
    assert_equal "Steam", result
  end  
  def test_water_status_liquid
    result = @instance.get_water_status(70)
    assert_equal "Liquid", result
  end
  def test_water_status_Ice
    result = @instance.get_water_status(-5)
    assert_equal "Ice", result
  end
end

Execution time: 0.082005 seconds.

RSpec

Official site: http://rspec.info/
Version: 1.3.0

By following BDD principles and providing a clean and elegant DSL (Domain Specific language), RSpec has gained a lot of fans. It is maybe the most popular testing framework in the Ruby world currently.

The code to test the WaterHelper.dll with RSpec is as follows:

require "rubygems"
require "spec"
require "spec/autorun"
require "WaterHelper.dll"

describe "Testing WaterHelper class" do
  before(:each) do
    @instance = Demo::WaterHelper.new
  end
  
  it "should be boiling water when it is 100 degrees" do
    result = @instance.is_water_boiled(100)
    result.should be_true
  end  
  it "should be boiling water when it is 150 degrees" do
    result = @instance.is_water_boiled(150)
    result.should be_true
  end
  
  it "should be frozen water when it is 0 degrees" do
    result = @instance.is_water_frozen(0)
    result.should be_true
  end
  it "should be frozen water when it is -50 degrees" do
    result = @instance.is_water_frozen(-50)
    result.should be_true
  end
  
  it "returns Steam for 300 degress" do
    result = @instance.get_water_status(300)
    result.should == "Steam"
  end
  it "returns Liquid for 70 degress" do
    result = @instance.get_water_status(70)
    result.should == "Liquid"
  end
  it "returns Ice for -5 degress" do
    result = @instance.get_water_status(-5)
    result.should == "Ice"
  end
end

Execution time: 0.201012 seconds.

Cucumber

Official site: http://cukes.info/
Version: 0.6.4

Cucumber is one of the most innovative frameworks out there. It took BDD one step further by providing a simple way to write requirement documents in a language called Gherkin (which is plain English with a few rules) and interpret them via code.

The next Gherkin code contains the requirements for the WaterHelper class:

Feature: WaterHelper
  As all users
  I want to know the status of the water
  To know how to treat it
  
  Scenario: Water are boiled at 100 degrees
    Given the temperature is 100 degrees
    When I check whether the water is boiled
    Then I should find out that it is
    
  Scenario: Water are boiled at 150 degrees
    Given the temperature is 150 degrees
    When I check whether the water is boiled
    Then I should find out that it is
    
  Scenario: Water are frozen at 0 degrees
    Given the temperature is 0 degrees
    When I check whether the water is frozen
    Then I should find out that it is
    
  Scenario: Water are frozen at -50 degrees
    Given the temperature is -50 degrees
    When I check whether the water is frozen
    Then I should find out that it is
    
  Scenario Outline: Water status
    Given the temperature is <temperature> degrees
    When I check the water status
    Then I should find out it is "<status>"
    
    Examples:
      | temperature | status |
      |     300     | Steam  |
      |     70      | Liquid |
      |     -5      |  Ice   |

And this is the Ruby code file to interpret the requirements:

require "WaterHelper.dll"

Before do
  @instance = Demo::WaterHelper.new
end

Given /the temperature is (.*) degrees/ do |temperature|
  @temperature = temperature.to_f
end

When /I check whether the water is boiled/ do
  @result = @instance.is_water_boiled(@temperature)
end
When /I check whether the water is frozen/ do
  @result = @instance.is_water_frozen(@temperature)
end
When /I check the water status/ do
  @result = @instance.get_water_status(@temperature)
end

Then /I should find out that it is/ do
  @result.should == true
end
Then /I should find out it is "(.*)"/ do |status|
  @result.should == status
end

Execution time: 0.883 seconds.

Shoulda

Official site: http://github.com/thoughtbot/shoulda
Version: 2.10.3
Modification: there is currently a bug in IronRuby which prevents Shoulda from running. I altered the IronRuby code to make it work and I’m in contact with the IronRuby team about the problem.

Shoulda is another popular testing framework. It is built on top of the Test::Unit testing framework and it makes it more developer-friendly.

The test code is as follows:

require 'rubygems'
require 'shoulda'
require "WaterHelper.dll"

class WaterHelperTest < Test::Unit::TestCase
  context "WaterHelper" do
    setup do
      @instance = Demo::WaterHelper.new
    end
    
    should "be boiling water when it is 100 degrees" do
      result = @instance.is_water_boiled(100)
      assert_equal true, result
    end
    should "be boiling water when it is 150 degrees" do
      result = @instance.is_water_boiled(150)
      assert_equal true, result
    end
    should "be frozen water when it is 0 degrees" do
      result = @instance.is_water_frozen(0)
      assert_equal true, result
    end
    should "be frozen water when it is -50 degrees" do
      result = @instance.is_water_frozen(-50)
      assert_equal true, result
    end
    should "return Steam for 300 degress" do
      result = @instance.get_water_status(300)
      assert_equal "Steam", result
    end
    should "return Liquid for 70 degress" do
      result = @instance.get_water_status(70)
      assert_equal "Liquid", result
    end
    should "return Ice for -5 degress" do
      result = @instance.get_water_status(-5)
      assert_equal "Ice", result
    end
  end
end

Execution time: 0.096005

riot

Official site: http://github.com/thumblemonks/riot
Version: 0.10.13
Modification: changed code to use iron-term-ansicolor (IronRuby’s equivalent to term/ansicolor)

The riot testing framework is a cute little thing. Its main goal is to make it quicker to write tests and to execute them. The code turns out very minimalistic, which makes this framework a good choice when you need to write some quick unit tests.

The next code contains the unit tests for the WaterHelper class written with the riot framework:

require "rubygems"
require "riot"
require "WaterHelper.dll"

context "Tests WaterHelper class" do
  setup {  Demo::WaterHelper.new }  
  
  asserts("the water is boiling when it is 100 degrees") {  topic.is_water_boiled(100) }    
  asserts("the water is boiling when it is 150 degrees") {  topic.is_water_boiled(150) }    
  asserts("the water is frozen when it is 0 degrees") {  topic.is_water_frozen(0) }    
  asserts("the water is frozen when it is -50 degrees") {  topic.is_water_frozen(-50) }
  asserts("you get steam when it is 300 degress") { topic.get_water_status(300) == "Steam" }
  asserts("you get liquid when it is 70 degress") { topic.get_water_status(70) == "Liquid" }
  asserts("you get ice when it is -5 degress") { topic.get_water_status(-5) == "Ice" }
end

Execution time: 0.083005 seconds.

Protest

Official site: http://rubyprotest.org/
Version: 0.3

Protest is another small and fun testing framework. It is described as a “small, simple and easy-to-extend testing framework” on its web site. It keeps it promise, I tell ya!

The next code tests the WaterHelper class using the Protest framework:

require "rubygems"
require "protest"
require "WaterHelper.dll"

Protest.context "WaterHelper class" do
  setup do
    @instance = Demo::WaterHelper.new
  end
  
  test "should be boiling water when it is 100 degrees" do
    result = @instance.is_water_boiled(100)
    assert result == true
  end
  test "should be boiling water when it is 150 degrees" do
    result = @instance.is_water_boiled(150)
    assert result == true
  end
  test "should be frozen water when it is 0 degrees" do
    result = @instance.is_water_frozen(0)
    assert result == true
  end
  test "should be frozen water when it is -50 degrees" do
    result = @instance.is_water_frozen(-50)
    assert result == true
  end
  test "returns Steam for 300 degress" do
    result = @instance.get_water_status(300)
    assert result == "Steam"
  end
  test "returns Liquid for 70 degress" do
    result = @instance.get_water_status(70)
    assert result == "Liquid"
  end
  test "returns Ice for -5 degress" do
    result = @instance.get_water_status(-5)
    assert result == "Ice"
  end
end

Execution time: 0.164009 seconds.

Stories

Official site: http://github.com/citrusbyte/stories
Version: 0.1.3

The Stories testing framework is built on top of Test::Unit and provides an entire different syntax for it. It has a convenient DSL for writing tests as “stories”.

The next code tests the WaterHelper class using the Stories framework:

require "rubygems"
require "stories"
require "WaterHelper.dll"

class WaterHelperClass < Test::Unit::TestCase
  story "As a user I want to know the status of the water" do
    setup do
      @instance = Demo::WaterHelper.new
    end
    scenario "Given 100 degrees, the water should be boiled" do
      result = @instance.is_water_boiled(100)
      assert_equal true, result
    end
    scenario "Given 150 degress, the water should be boiled" do
      result = @instance.is_water_boiled(150)
      assert_equal true, result
    end
    scenario "Given 0 degress, the water should be frozen" do
      result = @instance.is_water_frozen(0)
      assert_equal true, result
    end
    scenario "Given -50 degress, the water should be frozen" do
      result = @instance.is_water_frozen(-50)
      assert_equal true, result
    end
    scenario "On 300 degrees, water status should be steam" do
      result = @instance.get_water_status(300)
      assert_equal "Steam", result
    end
    scenario "On 70 degrees, water status should be liquid" do
      result = @instance.get_water_status(70)
      assert_equal "Liquid", result
    end
    scenario "On -5 degrees, water status should be ice" do
      result = @instance.get_water_status(-5)
      assert_equal "Ice", result
    end
  end
end

Execution time: 0.118007 seconds.

Lemon

Official site: http://proutils.github.com/lemon/
Version: 10.03.06

Lemon is an interesting unit testing framework. It provides a DSL which makes it very clear to identify which class and method you are testing. Moreover, it has code coverage capabilities which can report you which methods are not covered by your code. It is dependant on a bunch of other gems which makes it a bit slower than other frameworks.

The next code tests the WaterHelper class using the Lemon framework:

require "WaterHelper.dll"

TestCase Demo::WaterHelper do
  Concern "Water statuses are returned as expected."

  Before { @instance = Demo::WaterHelper.new }

  Unit :is_water_boiled => "returns true for 100 degress" do
    result = @instance.is_water_boiled(100)
    result.assert == true
  end
  Unit :is_water_boiled => "returns true for 150 degress" do
    result = @instance.is_water_boiled(150)
    result.assert == true
  end
  
  Unit :is_water_frozen => "returns true for 0 degress" do
    result = @instance.is_water_frozen(0)
    result.assert == true
  end
  Unit :is_water_frozen => "returns true for -50 degress" do
    result = @instance.is_water_frozen(-50)
    result.assert == true
  end
  
  Unit :get_water_status => "returns Steam for 300 degress" do
    result = @instance.get_water_status(300)
    result.assert == "Steam"
  end
  Unit :get_water_status => "returns Liquid for 70 degress" do
    result = @instance.get_water_status(70)
    result.assert == "Liquid"
  end
  Unit :get_water_status => "returns Ice for -5 degress" do
    result = @instance.get_water_status(-5)
    result.assert == "Ice"
  end
end

Execution time: 1.204097 seconds.

bacon

Official site: http://rubyforge.org/projects/test-spec
Version: 1.1

bacon is a small RSpec clone which is written in 300 lines of code. Its syntax is very similar to RSpec with a small difference on the expectation method (should).

The next code tests the WaterHelper class with the bacon framework:

require "rubygems"
require "bacon"
require "WaterHelper.dll"

describe "Testing WaterHelper class" do
  before do
    @instance = Demo::WaterHelper.new
  end
  
  it "should be boiling water when it is 100 degrees" do
    result = @instance.is_water_boiled(100)
    result.should.be.true
  end
  it "should be boiling water when it is 150 degrees" do
    result = @instance.is_water_boiled(150)
    result.should.be.true
  end  
  it "should be frozen water when it is 0 degrees" do
    result = @instance.is_water_frozen(0)
    result.should.be.true
  end
  it "should be frozen water when it is -50 degrees" do
    result = @instance.is_water_frozen(-50)
    result.should.be.true
  end
  
  it "returns Steam for 300 degress" do
    result = @instance.get_water_status(300)
    result.should.equal "Steam"
  end
  it "returns Liquid for 70 degress" do
    result = @instance.get_water_status(70)
    result.should.equal "Liquid"
  end
  it "returns Ice for -5 degress" do
    result = @instance.get_water_status(-5)
    result.should.equal "Ice"
  end  
end

Execution time: 0.120007 seconds.

Custom Testing Framework

This is a custom testing framework built in one line of code. I found it in a blog post by Paul Berry. It is not really for production purposes but it works well, it’s cool and it shows you how powerful Ruby is.

This next piece of code contains both the tests and the testing framework implementation. I surrounded everything with the Benchmark library to get the amount of time it takes to execute the tests:

require "benchmark"
require "WaterHelper.dll"

total_time = Benchmark.measure do
  tests = {
    "is_water_boiled(100)" => true,
    "is_water_boiled(150)" => true,
    "is_water_frozen(0)" => true,
    "is_water_frozen(-50)" => true,
    "get_water_status(300)" => "Steam",
    "get_water_status(70)" => "Liquid",
    "get_water_status(-5)" => "Ice"
  }

  instance = Demo::WaterHelper.new
  
  # The next line goes over all tests and executes them
  tests.each{|e,v| puts((r=eval("instance." + e))==v ? ". #{e}" : "! #{e} was '#{r}', expected '#{v}'")}
end

puts total_time

Execution time: 0.156001 seconds.

Benchmark Conclusion

Benchmarking testing frameworks is not such a good idea. Performance is not something you should care about when you write tests. It is much more important to have a maintainable set of tests instead of fast running ones that need a week of work when a requirement changes.

But, it’s cool and interesting to have charts in blog posts! so, here it is… the comparison between the execution time of all testing frameworks in this post:

Ruby's testing framework execution time comparison

It turned out that Test::Unit is the quickest one but except Cucumber and Lemon, the rest of the frameworks are behind only by a small difference.

Interesting facts:

All frameworks finished the tests under 1 second.
Test::Unit is more than 2 times faster than RSpec.
The custom testing framework got a real nice spot at the middle.
Cucumber is not the quickest one at all but it is still the coolest one :-)

Conclusion

IronRuby opens a whole new world of opportunities for .NET developers and Rubyists. In this post I focused on testing code but of course there is much more to it than just that. However, this showcase gets my point through – this post includes a variety of 10 (!!!) different testing frameworks. Each frameworks has its own uniqueness, making it very easy for you to choose the framework that works best for you.

And just for you to know - I had so much fun writing this post! Ruby is just awesome. Period.

All the best,
Shay.

Share: DZone | RubyFlow | Reddit

Published Friday, April 09, 2010 6:58 PM by shayf

Comments

No Comments

Title (required)
Name (required)
Your URL (optional)
Comments (required)
Remember Me?
Enter the numbers above:

IronShay

This Blog

Syndication

Search

Tags

News

I work at:

My Book

I'm a Microsoft C# MVP

RSS

My main blog is:

Navigation

Archives