ATDD Explained

July 26, 2011

In few of my previous posts I mentioned ATDD (and a related tool called FitNesse), but I never really explained what it is, and how it works. So here I go:

ATDD stands for Acceptance Test Driven Development. ATDD is somewhat related to TDD (Test Driven Development), but there are also some key differences (see this comparison). In my opinion, it is best to use both techniques together, as each has its own benefits.

ATDD is an agile development methodology (complementary to SCRUM or any other agile methodology), which puts the focus on the customer requirements and their automatic and continuous validation.


I added this paragraph after almost completing this post, because I realized that no one will reach the bottom line of such a long post… So I urge you to scroll down, read the paragraph “So what’s the big deal?” and then return and read from here. The reason I didn’t move that paragraph over here is that only after you’ll read the entire post (after-all), you’ll really believe that what’s written there is true.

Specifications in the Agile world

In most development methodologies (other than ATDD), the developers start implementing a feature after they receive some sort of specifications. In Waterfall, the specifications of the final product must be complete and detailed before the development begins, while in agile methodologies, these are often given per-feature, without a great level of details, and often even without any written documentation. They can be given by mail, phone, face-to-face conversation or whatever. There are problems with both methods:

In Waterfall:

· Writing all the detailed specifications up-front, leaves very little flexibility if or when requirements change, or when other constraints call for changing the specifications (like time-to-market, performance issues or development complexity)

· It’s very hard to determine whether there are conflicting requirements, until the developers face this conflict in the code (or even until a bug is found later on…)

· It’s very difficult to verify that all the requirements are met

In Agile:

· Lack of written documentation makes it difficult to communicate about expected behavior, especially after some time has passed

· If there is some sort of written documentation, it is difficult to maintain in up-to-date when requirements change. This makes the documentation useless over time

· It is extremely difficult to verify (e.g. by QA or by the customer) that all the requirements are met

Now, imagine for a moment that you have a way to run a tool that verifies that what’s written in the documentation really works. Wouldn’t it be great?! This way you could write the documentation little by little and see if it works, and only change what you already wrote if the requirements really change. In addition, if you have CI (Continuous Integration) or Gated Check-in system in-place (which you should really have if you don’t already!), you can ensure that every build meets all requirements! WOW, that we be awesome! Isn’t it?!

But wait a minute! “You’ve told me to imagine that I have a tool that verifies my software against the requirements, but I don’t have such tool, and I don’t believe it is even possible, because it refutes the Halting Problem!” you must say. “Are you trying to make a full out of me?!”

Well, it’s true, there’s no such a tool that reads a normal specifications document and verifies the software against it. (I guess that if there were, you could let this tool write the code for you too Smile). But if, instead of writing the specifications in an abstract way you write the requirements as a set of example scenarios, then, yes, it is possible, and this tool already exist! This tool is called FitNesse, and it’s even not such a complicated tool as you might think…

Specifications as a set of scenarios

Let me explain what’s the difference between “normal” specifications and specification by scenarios (or “by example”). Suppose we want to define the factorial function on a calculator. In a “normal” specification document, we would probably write something like that:

When the user enters a number and then clicks the “!” button, the factorial of the number is displayed on the screen. The factorial is calculated as the product of all the integers from 1 up till the entered number (i.e. 1*2*3*…*n).

In a specification as a scenario, the same specification will be written as few scenarios like the following one:

The user enters 10

The user clicks “!”

The display should show “3628800

Each of the scenarios in the set will have a different input and output numbers (the ones in bold). Specifically, when the scenario is simple (as above) and only the input and output values vary, the entire set of scenarios can be written as a table:

User enters

Display shows













More complex scenario

The above example is very simple one, so let’s consider a more complex one. Suppose we’re writing a store management system, we can define the following scenario:

A Coke costs $2

A Sandwich costs $3

A Candy bar costs $4

The manager defines a promotion: Buy Sandwich and get Coke for $1

A customer buys the following items:



Candy bar

The total price should be $8

(In a similar scenario, but without the promotion the total price should be $9).

A scenario is always linear, in that it has no branch statements (‘if’) or loops, and has a deterministic outcome that can be verified. In fact, it can contain more than one verification if needed. However, the steps that make up a scenario can be re-used between different scenarios to define new scenarios.

So what is ATDD?!

If you’re writing the specifications as scenarios, then a tool like FitNesse can execute them (with only little help from the developer) and check if the application works as expected. Once these scenario specifications can be executed, they become acceptance tests, as they test that the requirements are met. (Actually in ATDD, once they pass, they become regression acceptance tests, because they test that we didn’t break anything that already worked).

What is it about the “Driven Development” part of ATDD?

Here’s how it goes:

The product owner (or system analyst or whatever you call the person in your team that defines the requirements) define one or a few scenarios that describe new feature(s) (a.k.a. User Stories) that he wants implemented in the next iteration. Ideally, the product owner should be technical enough (though not a programmer) to be able to specify these scenarios by himself. Often in reality this is not the case, and so either a developer or a tester should help him write or translate his requirements to such scenarios.

Before the developer start implementing the new feature, he should first read the new scenarios and discuss them with the product owner to find possible gaps or mismatches, as well as to make sure he understands the scenario and the User Story. Once the developer feels comfortable with the scenario, he must first implement the little helper code that makes the scenario run-able. This code is called “fixture” and should be located in a dedicated assembly (i.e. not where the production code is). At this point the developer (and anyone else) should be able to run the scenario and see that it fails, because it is not implemented yet.

Finally the developer can start implementing the feature. He should implement just enough functionality to make one scenario (which can now be referred to as a test) pass before going to the next one. The DoD (Definition of Done) of every development task is clear: the test should pass.

It is advisable that the implementation phase will be done using regular TDD (i.e. using unit-tests), but this is not mandatory.

Of course that the old scenarios should be kept and run every build to ensure that no functionality has been broken. (if running all the scenarios take long it is possible to run only a subset every build and the full cycle once every few builds/days).

Due to this safety net of full automatic regression tests, the developer can feel free to make refactoring to the code as often and as thorough as he feels needed without fear of breaking anything. This has the following benefits:

1. It helps to keep the code clean, in a good and maintainable shape

2. The previous benefits implies that it’s easy to add or change features in the future

3. There’s no need to plan too much ahead, because you can always refactor. This implies that you can show the customer (or whoever wants to see) a working software often and get real feedback soon!

What’s the role of the QA person in ATDD?

Besides helping the Product Owner define the scenarios, ATDD frees the QA to do what every QA person likes to do the most: Exploratory Testing! Instead of running the same scenarios over and over again, or running after the developers to make their code more testable, the tester can simply try-out and play with the system to find real bugs. There are few things that are not well covered in ATDD and also remain the realm of the tester:

1. UI testing and usability. UI automation can be written before there’s a UI… also, if automatic UI testing doesn’t test the usability of the UI (e.g. a button can be 1 pixel, or the text can be written in white over white…). Scenarios can test the View Model, but not the View itself.

2. Integration with external systems. While this is possible to test in a scenario, it is often problematic and unreliable, or very difficult to control the other system in order to get predictable results.

3. Performance and load tests. This is a gray area. It can and should be tested automatically using scenarios, but it requires a lot of effort and usually deserves a dedicated lab and experimental abilities

When a tester finds a deterministic bug, he should write a new scenario to reproduce it, and then it is treated as every new feature.

So what’s the big deal?

hmmm… much! don’t you get it yourself yet?!

1. The amount of bugs that make it into the product drops down dramatically! This is because each new feature is tested as it is written, and keeps being tested as new features are added! You can decide to hand-off a version to the customer at any given moment, and know that it’s pretty stable.

2. By enabling this short release cycle you’re shortening the feedback loop. This leads to a better product, happier customers, and ensures a successful project!

3. The project manager (and all stakeholders) get an accurate status of the project at any given time: all the scenarios that were already implemented are green, others are red.

4. Reduces misunderstandings between developers, product owners, customers, QA… Needless to say that this saves valuable time…

5. Helps maintain a clean code base over time, which means that the code-base can last much longer and still be maintainable!

It’s a well-known fact that the sooner a bug is discovered, the less it costs to fix it – so the bottom line: it saves you money…

You keep mentioning this “FitNesse” tool, what is it exactly?

You’ll have to wait for my next post… this post became pretty long without describing FitNesse so I promise to write a new post just for it.

Where do I begin?

“To tell the story of how great a love can be… NoteNoteNote…”

No, really…

You can try to learn more about it from books, then try to implemented it in your own team. But frankly, from my experience, it really takes a renowned consulting company like E4D Solutions to introduce a new methodology successfully in a development organization. Or you can simply leave me a comment here and I’ll get back to you.

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>