Functional Programming Talk

Yesterday I had the pleasure of speaking in front of a number of my colleagues in Sela about functional programming.  My talk’s title was ‘The Essence of Functional Programming’ and it dealt with what I consider to be the primary differences between functional and other languages.  Thanks to everyone who came – it was pleasure (broken air-conditioning not withstanding..)! 

Slide deck is available here.

Testing a Longest-Common Prefix Implementation Using Model-Based Testing

In  my last post I discussed how to implement a Longest-Common Prefix (LCP) algorithm in both an imperative and functional manner in C#.  I also mentioned the fact that for the imperative implementation I needed some to write some unit tests.  Here is the test code:

[TestClass]
public class LCPTests
{
    [TestMethod]
    public void Get_WithNoValues_ShouldReturnEmptyString()
    {
        string result = LCP.Get(new string[] {});
        Assert.AreEqual(0, result.Length);
    }

    [TestMethod]
    public void Get_WithOneValue_ShouldReturnTheValue()
    {
        string result = LCP.Get(new[] {"dir1/dir2/dir3"});
        Assert.AreEqual("dir1/dir2/dir3", result);
    }

    [TestMethod]
    public void Get_WithTwoValues_ShouldReturnTheLongestMatch()
    {
        string result = LCP.Get(new[] {"dir1/dir2/dir3", "dir1/dir2/dir4"});
        Assert.AreEqual("dir1/dir2", result);
    }
}

The idea for these tests is that we have a single test each for 3 equivalence classes – the case where the input to the LCP method is empty, the case where it contains exactly 1 directory and the case where it contains 2 or more directories.  These 3 tests pass and seem to cover everything they need to cover.  But 3 tests for all possible cases?  This doesn’t seem right.  How can we be sure that we’re not missing something?

The short answer is - we can't, at least not with tests (recall that tests can only show the presence of - not the lack of - bugs).  With more tests we might be more confident that there aren't any obvious bugs, but we can never be absolutely sure.  So - more tests it is.  But this seems like such a tedious job, going over a large number of cases and figuring out what the proper result should be.  At this point, it's pretty obvious that the computer itself should somehow generate both the test data and the test results.  In fact, when we do this we have what is known as a Test Oracle.

So how to write an oracle? One way is by reaching for our trusted copy of Visual Studio 2010 that has the Spec Explorer 2010 extension installed.  If you're not familiar with Spec Explorer, it's a Model-Based Testing  (MBT) tool that's integrated directly into Visual Studio.  You write your model's rules using C# and define scenarios and model parameters using a concept known as machines. 

For our model we want the ability to determine how many directories are compared in the LCP method, and how many parts each directory contains.  The general idea is that we have an 'initial' directory that contains a predefined number of parts.  We then add a number of other directories, each with the same number of parts and finally run the method.  The model produces both the inputs (directory parts) and the output (the longest common prefix).  Based on the above two parameters we can decide how few or how many tests Spec Explorer generates.

Model Mechanics

Here's the model code:

public static class ModelProgram
{
    public static int DirectoryPartCount = 1;
    public static int MaxNumberOfDirectories = 1;
        

    private static readonly SequenceContainer<string> FirstDirectoryParts = 
        new SequenceContainer<string>();


    private static readonly SequenceContainer<SequenceContainer<string>> 
        OtherDirectoryParts = new SequenceContainer<SequenceContainer<string>>();

    private static int _otherDirectoryPartsIndex;


    private static bool _needsInitialization = true;


    public static IEnumerable<string> ExistingValues
    {
        get { return FirstDirectoryParts; }
    }

    [AcceptingStateCondition]
    public static bool IsAcceptingState
    {
        get { return OtherDirectoryParts.All(x => x.Count == DirectoryPartCount); }
    }

    [Rule]
    public static void Initialize()
    {
        Condition.IsTrue(_needsInitialization);
        for (int partIndex = 0; partIndex < DirectoryPartCount; partIndex++)
            FirstDirectoryParts.Add("dir" + partIndex);
        _needsInitialization = false;

        OtherDirectoryParts.Add(new SequenceContainer<string>());
    }

    [Rule]
    public static void AddNewDirectoryToCompare()
    {
        Condition.IsFalse(_needsInitialization);
        Condition.IsTrue(OtherDirectoryParts.Count == 0 ||
            OtherDirectoryParts[_otherDirectoryPartsIndex].Count == 
            DirectoryPartCount);

        _otherDirectoryPartsIndex++;
        OtherDirectoryParts.Add(new SequenceContainer<string>());
    }


    [Rule]
    public static string AddDirectoryPart([Domain("ExistingValues")] string part)
    {
        Condition.IsFalse(_needsInitialization);
        Condition.IsTrue(OtherDirectoryParts.Count <= MaxNumberOfDirectories && 
            OtherDirectoryParts.Count > 0);
        Condition.IsFalse(OtherDirectoryParts[_otherDirectoryPartsIndex].Count == 
            DirectoryPartCount);


        OtherDirectoryParts[_otherDirectoryPartsIndex].Add(part);


        return GetCommonPrefix();
    }

    private static string GetCommonPrefix()
    {
        int numberOfMatchingParts = 0;
        int maxLength = OtherDirectoryParts.Min(x => x.Count);
        for (int i = 0; i < maxLength; i++)
        {
            if (OtherDirectoryParts.All(x => x[i].Equals(FirstDirectoryParts[i])))
                numberOfMatchingParts++;
            else
                break;

        }

        return String.Join("/", FirstDirectoryParts.Take(numberOfMatchingParts));
    }
}

Let's start with the rules.  Rules are actions that affect the state of the model.  Spec Explorer uses rules to determine the valid states (as in state-machine) of the model and how it should behave (that is, what output it should produce) for a given set of inputs.  The LCP model has the following rules:

1.  Initialize - Only allowed to run when the _isInitialized flag is false.  This rule sets up the initial directory's parts and prepares an empty directory that is ready to accept new parts.

2.  AddDirectoryParts - Only allowed to run when the _isInitialized flag is true, the number of 'other' directories is below the set threshold and that the current 'other' directory is not full (which requires that we set up another 'other' directory).  This rule adds another part to the current 'other' directory.  It is up to Spec Explorer to supply the values of these parts (with a little user assistance, of course, to give it the set of permissible values).  Finally, this rule is responsible for computing the value that should be returned by the actual LCP function (using a simplified imperative implementation).

3.  AddDirectoryForComparison - Only allowed to run when the _isInitialized flag is true and the current 'other' directory is full.  This rule adds a new 'other' directory and makes it the current one.

In addition to the rules, we have some private state variables and helper methods.  Something worth mentioning is the IsAcceptingState property that is decorated by a [AccptingStateCondition] attribute.  This property indicates to the model that it is in an 'accepting' state - i.e., a state that can be used by the test code generator to end the test.  We'll see this again in a minute.

Let's look now at the .cord (short for 'coordination' file).  This file sets up the test scenarios that we want to generate:

using LCP.Adapter;
using LCP.Model;
using Microsoft.Modeling;

config Main 
{
 
    action all LongestCommonPrefix;
    
    switch StateBound = 65536;
    switch StepBound = 65536;
    switch TestClassBase = "vs";
    switch GeneratedTestPath = "..\\MBT.TestSuite";
    switch GeneratedTestNamespace = "MBT.TestSuite";
    switch TestEnabled = false;
    switch ForExploration = true;
}



machine ModelProgram() : Main 
{
    construct model program from Main
}

machine ModelProgramWithParameters() : Main
{
    {. 
       ModelProgram.DirectoryPartCount = 3;
       ModelProgram.MaxNumberOfDirectories = 2;
     .}:
    ModelProgram
}


machine ModelProgramWithAcceptingStates() : Main
{
    construct accepting paths
    for ModelProgramWithParameters
} 

machine TestSuite() : Main where TestEnabled=true 
{
    construct test cases where Strategy="shorttests", StopAtAccepting=true
    for ModelProgramWithAcceptingStates
}

Inside the .cord file we have a config section that sets up some bounds on the exploration that the model performs and determines where the generated code goes.  Based on this configuration we have a number of machines:

1.  ModelProgram - This sets up a machine that is based on the model code and uses the configuration values from the Main config section.
2.  ModelProgramWithParameters - This machine uses the user-specified parameters for determining how much exploration the model will do.  This is where we determine how many tests the model will generate.
3.  ModelProgramWithAcceptingState - This machine 'cuts off' the paths generated by the ModelProgramWithParameters machine so that paths end with accepting states.  It is the basis for the test generation phase.
4.  TestSuite - This machine is used for generating the actual test suite.  It uses a 'short test' strategy which means that it produces a large number of short tests as opposed to a small number of long tests.

Results

Lets look at a part of the explored ModelProgramWithAcceptingState machine:
DirectoryPartCount = 2
MaxNumbrOfDirectories = 2

See how Spec Explorer products all possible values for the parts as well as the expected LCP output?
And this is a part of the explored TestSuite machine:

And finally, let's see the results of running the tests:

DirectoryPartCount = 2
MaxNumbrOfDirectories = 2

DirectoryPartCount = 3
MaxNumbrOfDirectories = 2

DirectoryPartCount = 2
MaxNumbrOfDirectories = 3

(Yes, it’s the same number of tests as before, but trust me, they're different).

Summary

So there we have it - many different cases for testing the LCP function.  Rather than writing test cases and computing the output values in a manual fashion, we used a model to generate them automatically.  At this point, we can be reasonably sure that our function is correct.

The code for this post is available here.

Longest Common Prefix with C# and LINQ

I recently can across an interesting problem:  Given a set of n directories, find the most nested directory that is an ancestor of all of them.  This is equivalent to finding the longest common prefix. For example, if I have the following paths:

/dir1/dir2/dir3
/dir1/dir2/dir4

Then the longest common prefix is just /dir1/dir2.  If we now add /dir1/dir5 to the set, we get:

/dir1/dir2/dir3
/dir1/dir2/dir4
/dir1/dir5

Then the longest common prefix changes to /dir1.  In order to do this, I start by comparing the first and second directories by splitting them on the / symbol, and then comparing the elements of the resulting arrays.  The common elements are the components of the common prefix, and the common prefix ends at the index where the elements in the arrays are different.  I then take this common prefix, compare it with the next directory of the set and repeat.

For example, for /dir1/dir2/dir3 and /dir1/dir2/dir4 above:

OK, so nothing overly complex there.  There are probably better algorithms for doing this, but the code is not performance-critical and I have more important things to spend my time on. 

Here is the C# code for implementing the algorithm:

public string GetLongestCommonPrefix(string[] directories)
{
    if (directories == null || directories.Length == 0)
        return String.Empty;

    const char SEPARATOR = '/';

    IList<string> directoryParts = new List<string>(
        directories[0].Split(SEPARATOR));

    for (int index = 1; index < directories.Length; index++)
    {
        IList<string> first = directoryParts;
        string[] second = directories[index].Split(SEPARATOR);
        int maxPrefixLength = Math.Min(first.Count, second.Length);
        var tempDirectoryParts = new List<string>(maxPrefixLength);

        for (int part = 0; part < maxPrefixLength; part++)
        {
            if (first[part] == second[part])
                tempDirectoryParts.Add(first[part]);
        }
        directoryParts = tempDirectoryParts;
    }

    return String.Join(SEPARATOR.ToString(), directoryParts);
}

This works and gets the job done.  But it’s ugly.  In particular:

• Where’s the algorithm?  All I see is a bunch of assignments and loops.  It’s really difficult to tell what’s going on here and see the big picture.
• With so many loops and indices and assignments, it’s practically guaranteed that you’ll get lost and have off-by-1 errors and other such fun bugs.  I actually needed tests (plural) for this ‘simple’ piece of code.

Let’s compare this with an implementation based on LINQ (which in essence is an implementation based on the functional programming paradigm):

public string GetLongestCommonPrefix(string[] directories)
{

    if (directories == null || directories.Length == 0)
        return String.Empty;

    const char SEPARATOR = '/';


    string[] prefixParts =
        directories.Select(dir => dir.Split(SEPARATOR))
        .Aggregate(
            (first, second) => first.Zip(second, (a, b) => 
                                    new { First = a, Second = b })
                                .TakeWhile(pair => pair.First.Equals(pair.Second))
                                .Select(pair => pair.First)
                                .ToArray()
        );

    return string.Join(SEPARATOR.ToString(), prefixParts);
}

MUCH nicer.  Note how I have no state management of any kind (no array indices, temporary lists, etc.), which means that the chances I’ll miss an edge case are much slimmer.  The code is just function composition, the kind that programmers do all day, every day.  Also note that the algorithm discussed previously is pretty much right in your face.  We’re telling the compiler WHAT we want done rather then HOW to do it.  Granted, you need to know the LINQ syntax and methods to understand what’s going on.  But LINQ has been around long enough so that many people are familiar with it and there are plenty of examples available. 

But you know what’s really cool about this code?  The fact that once it compiles it Just Works.  So once the compiler and I finished arguing about the types, all of my tests passed without a hitch.  That’s impressive.

I look forward to giving my C# code a functional touch!

SDP 13 Talks

This week we’re running the Sela Developer Practice conference and this year it’s bigger and better than ever, with speakers coming from outside of Israel to deliver talks.  I was fortunate enough to see the 10 Reasons Software Sucks talk by Caleb Jenkins and had a chance to grab a beer and some good food with him later for some one-on-one chat.  Seriously people, if you get a chance – go watch this guy speak.  He  ROCKS! 

I gave two sessions at this SDP:

1.  Model-Based Testing with Spec Explorer 2010 – I discussed what Model-Based Testing (MBT) is and how software teams can use it to enhance their testing effectiveness.  This talk included a practical demonstration on how to use MBT for testing a WCF service.  Code for this session is available here.

2.  Pull-Based Agile Development Methods – I talked about the problems that are inherent in push-based workflows, including both Waterfall and Scrum models.  We discussed the concept of pulling and the idea of Work-In-Progress (WIP), which then naturally led us to a discussion of Kanban. 

As in every  previous SDP, I learn much more than I teach.  A big THANK YOU to everyone who attended my sessions – you had some great and insightful questions and comments that I’ll be pondering.

Hope to see you all in the next SDP!

Getting Started with Getting Started: Yet Another TDD Post

Sit back for a second and think:  what’s the scariest moment when starting a new project?  For me, it’s sitting in front of an empty code file.  I mean, I have the latest versions of Visual Studio and Windows on my machine (2012 and 8, respectively, at the time of writing), everything is humming along nicely, all this power at my fingertips – and nothing to apply it to.  I’m staring at an abyss.  It’s absolutely terrifying.

I had always thought that it’s just me and my personal biases, but recently I had the chance to witness this same phenomenon in others as well.  Some background:  part of my job at Sela is teaching basic software development skills to students.  Among these skills we also teach them C#.  So we would go over some feature of the language, I would give them a coding assignment and let them have a go at it.  And several times I saw two of my brightest students stare at the screens in front of them with a look of absolute horror on their faces.  So I walk over and ask them what’s wrong.  The answer":  “I don’t know where to begin”.    They understood the assignment and had a rough idea of how they would approach it in terms of class and method design.  But actually writing the first lines of code required some kind of leap that was paralyzing them.  I guided them through the first class and method declarations and it was smooth sailing from there on.

And this got me thinking – how do I handle these situations?  What is it that allows me to overcome this dread of empty projects?  In my case, I apply Test-Driven Development.  You’ve probably heard of TDD – if not, look it up and you’ll find lots of material on the Net (try here for a concise introduction).  The key for me is the first principle – writing a failing unit test before writing production code.  If it’s the first test in the project (or, for that matter in a completely new part of the application), I write what I call a warm-up test.  It’s a test whose only purpose in life is to get me from nothing (an empty project) to something.  Enhancing or fixing existing code is nowhere near as frightening as starting something new.

The contents of a warm-up test are almost trivial.  It’s a class declaration, maybe a method or property call, and an assertion that ensures the most basic conditions on a new instance of the actual production class (also known as System-Under-Test or SUT).  That is, it answers the question – what can I say about a brand-new, just-out-of-the-factory instance?  By answering the question I get a test.  Test code is code – in fact, it’s code that obviously can’t run since the SUT does not exist yet.  But hey, my project is not empty anymore! 

If we look closely, we can also see that what we just did is more than just write a bit of trivial code.  We also did some design on the way.  Granted, not a lot of design – but enough to get us started and pointing in the right direction.  Adding any additional code that has applicative value is now easier, since we have an existing object and well-defined behavior for a new instance.

Let’s look at a simple example:  the venerable BankAccount class.  A warm-up test checks this new instance and ensures that its balance is exactly 0 – no debt and no credit.  So my first lines of code would be:

[TestClass]
public class BankAccountTests
{
    [TestMethod]
    public void ctor_HasZeroBalance()
    {
        var account = new BankAccount();
        Assert.AreEqual(0, account.Balance);
    }
}

The end result – my project is not empty and there is a list of tasks that await me: I have a class to declare, a property to add along with a simple implementation that I’m comfortable with.  Plenty of work.  Phew, that was scary there for a while!

TFS Reporting Recipes #2: Get Daily Values for a Date Range

Purpose

One of the most commonly required features in a report is a list of the daily values of some metric over a range of dates (for example, how many open bugs were in the system at the end of the day for each day in the last month).  This query will retrieve these metric values for each day in the date range.

Prerequisites

• TFS 2010 or higher
• @StartDate parameter – The beginning of the required date range
• @EndDate parameter – The end of the required date range

Query

SELECT dd.[Date], query.Value
FROM dbo.DimDate dd
CROSS APPLY
(
    SELECT 1 AS Value     -- Replace with your own query, use dd.[Date] in a WHERE clause to limit the contents based on the date
) query
WHERE dd.[Date] Between @StartDate AND @EndDate

Example

The following example retrieves the number of open bugs for a given date range:

SELECT dd.[Date], query.Value
FROM dbo.DimDate dd
CROSS APPLY
(
    SELECT SUM(RecordCount) AS Value
    FROM dbo.WorkItemHistoryView
    WHERE System_WorkItemType = 'Bug'
        AND ProjectNodeGUID = @ProjectGUID
        AND System_State = 'Open'
        AND System_ChangedDate <= dd.[Date]
        AND System_RevisedDate > dd.[Date]
) query
WHERE dd.[Date] Between @StartDate AND @EndDate

TFS Reporting Recipes #1: Get the Iteration Path/GUID for the Current Sprint

Purpose

Many times you want to filter the contents of reports based on sprints or iterations (for example, the number of re-opened bugs in each sprint in the project).  In this case, you often want the default of the report filter to show the iteration path of the current sprint.  This query will retrieve the current sprint’s iteration path.

Prerequisites

• TFS 2010 or higher
• The Scrum process template
• A @ProjectGUID  parameter – The GUID of the Team Project in which you’re interested

Query

SELECT IterationPath
FROM dbo.CurrentWorkItemView
WHERE System_WorkItemType = 'Sprint'
    AND ProjectNodeGUID = @ProjectGUID
    AND Microsoft_VSTS_Scheduling_StartDate <= GetDate()
    AND Microsoft_VSTS_Scheduling_FinishDate >= GetDate()

TFS Reporting Recipes #0: Introduction

Well, it’s been a while since I’ve last blogged and it’s time to get back into the rhythm!  In this series, which I’m calling TFS Reporting Recipes, I’d like to show some tricks, tips and other patterns that help me along when I’m writing reports for customers.  I almost always use the relational warehouse since I feel more comfortable in SQL than in MDX, so for the recipes I assume you’re in the Tfs_Warehouse database.

Feel free to sound off in the comments if you have questions/issues that you’d like to see me address.

Happy Reporting!

The following posts are available:

1.  Get the iteration path of the current sprint

2.  Get Daily Values for a Date Range

Quality-Jitsu, or Learning to Empty Your Cup

Aikido, and I suppose all martial arts, are painful.  Mostly physically painful, depending on how skilled – or rather, how unskilled – the person on the other end of your joint is.  But there is also a great deal of figurative pain.  It’s the pain of letting go of your ego and the pain of emptying your cup.  It’s the acceptance that someone half your size (or less) can be twice (or more) as effective as you are, disabling and throwing you around seemingly without even trying.  It’s understanding that no, it’s not as trivial as your teacher makes it look and that yes, it’s going to require work to do it effortlessly.  A LOT of work.  

In the years that I’ve been doing Aikido, I’ve seen many people come and go through our school doors.  For some, it truly was physical – too much inter-personal contact, extra-sensitive joints and at least in one case, an almost-broken knee.  But for the grand majority it was not.  It was the inability to learn something ‘easy’ from someone else.  These are the kind of people who came to teach junior students, rather than learn from the senior ones.  In fact, one of the things I admire most about my teacher is his endless patience, which eventually resulted in many students changing from the former kind into the latter (myself included).

But this is a tech blog.  So what’s all this talk of martial arts?  Well, getting software quality to look effortless and easy takes work.  A LOT of work.  And with the state of the practice that I see at some software organizations today, it definitely feels like a battlefield.  Here’s a typical scenario:  a reasonably successful organization will happily spend days squeezing the last ounce of performance out of some algorithm.  And they’ll readily acknowledge that they're not able to iterate quickly enough, or ship with a low enough number of bugs (zero-bugs policy, anyone?), or deliver on-time enough.  Yet when the talk turns to actually doing something about it, the same organizations suddenly know everything there is to know, have done everything there is to do and can generally find a 6-digit number of reasons for why THEIR product quality simply CANNOT be improved.

So, if you’re not willing (notice I said willing – not a word about able) to change anything you’re doing today, what in the name of Miyamoto Musashi makes you think your quality will improve?  What, some fairy will drop down from the sky and magically exorcise all bugs from your code?  Yes, it IS as ridiculous as it sounds.  Wearing a keikogi or not, I’m seeing the same type of people that cannot cut it in a dojo.  You WILL have to write some kind of automated tests.  You WILL have to set up some kind of automated build.  And it’s going to take work.  A LOT of work.  Whether you eventually succeed or not depends on how empty your cup is and how much you’re willing to learn from others, be they younger, older, bigger or smaller than you.

So when someone approaches me at a conference, sends me an email or asks my advice I start by asking them two questions:

1. What have you tried already?
2. Why do want to do whatever it is that you want done?

The answer to #1 helps me make sure my cup is empty (well, not completely full is a good start…).  It’s amazing how resourceful and smart people never cease to surprise me.  My favorite answer to #2:  ‘It hurts.  How do I make it hurt less?’. 

Onegai Shimasu!

TFS 11 מעט על התצורות החדשות הצפויות לנו עם

More on Team Build 2010 Extensions

In my last post, I touched briefly on the concept of build extensions, and explained that they are an implementation of WF4 extensions.  I thought it would be useful to list the other build extensions that are available inside of a build template:

• IBuildDetail – This build extension enables you to notify the build server about important aspects of your build (compilation/test status, overall build status, the label associated with the build, etc.).  In fact, this is the how the SetBuildDetails built-in activity is constructed.  So a typical use-case for this extension could be, for example, to fail the build when your activity encounters an error condition, despite having succeeded in compiling or running the tests.  However, if you choose this route, I strongly urge you to use the composition approach rather than the code approach.
• IBuildLoggingExtension – This extension is useful for controlling which activities are logged into the build log. 
• IAgentReservationExtension – This extension seems to be used for actually requesting the server to reserve/free an agent for running a specific workflow.  This looks to be very interesting for doing complex dependency-management stuff – when I build component A, I’d like to make sure that the latest version of component B (upon which A is dependent) is built, which is a separate build altogether.  I intend to try this out soon.
• IBuildAgent – This extension allows you to retrieve details of the build agent that is currently running your build.  Of course, this is only relevant for activities that run inside an AgentScope.

As I mentioned before, using extensions requires 2 stages:

1.  Declare that you will need the extension inside the activity’s CacheMetadata method, using the metadata’s RequireExtension<T> method

2.  Retrieve the extension inside the execution method (Execute/BeginExecute) using the activity context’s GetExtension<T> method

Add Chuck Norris to Your Build!

In this post I’ll show how to write a somewhat more complex and hopefully fun activity for TFS 2010 Team Build (code is available here).  In particular, I’ll make use of the following features:

• The ActivityTrackingAttribute class
• Build Extensions (a specialization of WF4 Workflow Extensions)

Since our builds are sometimes long and dull processes, we need some stuff to talk about while they run.  What could be better than discussing the prowess of the one-and-only Chuck Norris?  We’ll have Team Build produce a new Chuck Norris (CN) fact every time a build runs:

And now we have a good conversation starter!  Who knows, maybe this is a good way to convince team members to run the build more often during the day…

Design of the CN Activity

In order to implement the activity we need to deal with the following issues:

1. Finding a source of Chuck Norris facts to display and choosing a random fact each time
2. Displaying the fact in the Build Log View
3. Maintaining good Separation of Concerns (SOC) – that is, making sure the code is well-factored

Let’s tackle these one at a time.

Finding the Data

This was actually rather easy.  A quick Google search turned up The Internet Chuck Norris Database, a hilarious site which even contains a REST API!   Great, so this means we can get a single joke back each time by writing appropriate WCF service and data contracts and using the WebHttp binding for interacting with REST services (many thanks to Yaniv Rodensky for helping me out with the WCF details).

Displaying the Data

We want to display the data in the Build Log View – that is, the log in which TFS shows the on-going activity of the build process.  Fortunately, we have the built-in WriteBuildMessage activity for doing precisely this.  It would probably make sense to structure our code in terms of two separate activities:

• GetChuckNorrisFact – Performs the Web access itself and returns the joke text.  Since this is in fact an IO operation, we want to derive it from AsyncCodeActivity<string> in order to take advantage of the WF4 runtime’s ability to run activities in an asynchronous manner for making the workflow as efficient as possible.
• DisplayChuckNorrisFact – Compose the GetChuckNorrisFact and WriteBuildMessage activities into a single building block, which we’ll use in our builds.  This enclosing activity will also include error handling by way of the built-in TryCatch activity.   Since DCNF is the only user of GCNF, it makes sense that we make the latter internal rather than public, which in turn means that the WCF interfaces and contracts can be internal as well.  Encapsulation galore!

The interesting question here is why would we choose to create DCNF as a composition of activities rather then simply deriving from CodeActivity and writing everything out in code.  I would say that the reason for doing it this way has to do with staying at the proper abstraction level and making good use of the underlying WF4 technology.  If we consider Team Build 2010 to be a domain-specific language (where our domain is the automated building of software), than activity composition is in fact the creation of compound words.  Taking the language metaphor even further -  GCNF and DCNF are content words while the other activities (TryCatch, WriteBuildMessage, etc.) are function words and fortunately for us, have already been implemented by Microsoft.  I find this similarity to natural language to probably be the least-understood aspect of WF4, but that’s for some other time (and some other post).

Another interesting point with the composite activity has to do with what exactly is displayed in the log.  The structure of DCNF is as follows, with the Try/Catch/Finally boxes being properties in the TryCatch activity:

Recall the way TFS displays the activities in the template – the DisplayName of each activity is used in the Build Log View and indentation is determined by how deep it is in the build workflow template.  So what we would have expected to see is the name for each one of the internal composed activities, like such:

The fact that this is not so is due to the use of the ActivityTrackingAttribute class.  Notice the attribute on the DisplayChuckNorrisFact activity class:

    [ActivityTracking(ActivityTrackingOption.ActivityOnly)]
    public sealed class DisplayChuckNorrisFact : Activity
    {
        private const string FailureText =
            "Chuck Norris must have brought down the Internet, " +
            "that's why you can't have any facts!";

        public DisplayChuckNorrisFact()
        {
            Implementation = () => CreateBody();
        }

        private Activity CreateBody()
        {
            var factTextVariable = new Variable<string>("factText");
            var tryCatch = new TryCatch();
            tryCatch.Variables.Add(factTextVariable);

            tryCatch.Try = new GetChuckNorrisFact {Result = factTextVariable};
            tryCatch.Catches.Add(new Catch<Exception>
            {
                Action =
                    new ActivityAction<Exception>
                        {
                            Handler =
                                new Assign<string>
                                    {
                                        To = factTextVariable,
                                        Value = FailureText
                                    }
                        }
            });
            tryCatch.Finally = new WriteBuildMessage
            {
                Importance = BuildMessageImportance.High, 
                Message = factTextVariable
            };

            return tryCatch;
        }
    }
}

• ActivityTrackingOption.ActivityTree – Output each and every activity’s DisplayName, both top-level and composed  (this is the default option)
• ActivityTrackingOption.None – Do not output any activity’s DisplayName, neither top-level nor composed

Using this attribute adds a level of control that is more involved (but still possible) to achieve with ‘plain’ C# code (using the CodeActivity route).

Maintaining Separation of Concerns

We already saw some SOC-related issues – namely, the separation of the logical ‘display a random fact’ operation into multiple physical WF4 activities.  We also saw that there are some pretty good reasons to do so, even if it means additional work.  In a similar manner, we would also like to separate the GetChuckNorrisFact activity into two separate parts – the WCF service proxy and a WF4 activity that uses it.

If we were calling a regular SOAP service it would have been very easy to use Visual Studio’s Add Service Reference functionality (or the equivalent svcutil.exe) to generate code for this service’s proxy so we could use it as a black box.  Unfortunately this isn’t currently possible with a REST service, so we need to create this proxy ourselves.  So here’s what we end up with in the Visual Studio project:

• IChuckNorrisFactService.cs: This file contains several DataContracts and a ServiceContract.  Note that the latter is decorated with:
  • A WebInvoke attribute - signifies that this method is activated using an HTTP GET verb, along with the URI template which maps method parameters (on the client) to the URL passed to the service
  • An AsyncPattern property in the OperationContract – signifies that this method is part of an APM method pair that will be used for asynchronous communication with the service
• ChuckNorrisWebServiceConsumer.cs:  This is our service proxy.  It uses the WebChannelFactory<T> class along with a WebHttpBinding to connect to the service and retrieve the raw data.  It then uses a DataContractJsonSerializer to deserialize the returned JSON text into .NET objects that we can pass back to the caller.  (As a side note,  the reason for the existence of this class is because I could not get WCF to deserialize the JSON text automatically.  This might have to do with the fact that the service returns its response with a Content-Type header of ‘text\html’ rather than ‘application\json’ or something similar.)
• The two activities discussed earlier, implemented to use the APM pattern

So now we need to make sure that the GetChuckNorrisFact activity has access to the service proxy.  The easiest way to do this is to simply create a new instance of the proxy inside the activity.  However, this would increase our coupling as GCNF would now need to know how to create an instance of the proxy (not really an issue with this particular implementation, but I hope the point is clear).  Instead, we opt to use a different mechanism, whereby we expect the WF4 host – TFS, in our case – to supply us with an instance of the proxy. 

Note the attribute on the ChuckNorrisWebServiceConsumer class:

   [BuildExtension(HostEnvironmentOption.All)]
   public class ChuckNorrisWebServiceConsumer
   {
       private const string ServiceAddress = "http://api.icndb.com";
       private IChuckNorrisFactService channel;

       public IAsyncResult BeginGetFact(AsyncCallback callback, object state)
       {
           using (
               var cf = 
                   new WebChannelFactory<IChuckNorrisFactService>(
                       new WebHttpBinding(), new Uri(ServiceAddress)))
           {
               channel = cf.CreateChannel();
               return channel.BeginGetFacts("1", callback, state);
           }
       }

       public string EndGetFact(IAsyncResult result)
       {
           var serializer = new DataContractJsonSerializer(typeof (Data));
           var data = (Data) serializer.ReadObject(channel.EndGetFacts(result));

           return data.Jokes.First();
       }
   }

An activity requests the extension by doing two things (see GetChuckNorrisFact.cs):

1.  In the CacheMetadata method, it must notify the workflow host that it expects to use the extension:

protected override void CacheMetadata(CodeActivityMetadata metadata)
{
    base.CacheMetadata(metadata);
    metadata.RequireExtension<ChuckNorrisWebServiceConsumer>();
}

Since CacheMetadata is run before the workflow starts executing, WF4 can verify that the activity has indeed been registered with the server and is available for use.  If it has not (for example, we forgot to deploy the custom activity DLL to the build server), then the WF4 runtime will throw an exception.  In our terms, this means we will immediately get a build failure – before any  build-related activities have run.

2.  Inside the activity’s execution methods (Execute for CodeActivity and BeginExecute for AsyncCodeActivity), it must request the extension from the activity’s context:  

protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context,
    AsyncCallback callback, object state)
{
    var serviceProxy = context.GetExtension<ChuckNorrisWebServiceConsumer>();
    context.UserState = serviceProxy;
    return serviceProxy.BeginGetFact(callback, state);
}

Summary

So there we have it – our very own Chuck Norris fact-retrieving activity for integration into the build.  We saw how to achieve this task using WF4 activity composition, workflow extensions and WCF REST bindings.  While not overly complex, it does show that there is some effort and knowledge required to properly write build activities.  Happy building!

Writing a Simple TFS Report – Part 4: Publishing the Report

In the last post of this series we completed our report so that it contains all the graphical annotations we wanted.  It is now time to publish it to the server and make it available to our users. 

Publishing the Report to the Server

Once we have a report running on the local machine, publishing it is the easy part! 

1.  Open the report that we worked on throughout the previous parts of the series

2.  From the main ribbon button, choose the Save As menu option.  The Save As Report file dialog will appear.

3.  Now comes the easy part:  In the Name textbox, enter the URL for your TFS installation of Reporting Services.  This will typically be of the form http://<TFS SERVER>/ReportServer. 

4.  You should now see a folder named simply TfsReports.

5.  Open the TfsReports folder.  You will now see a list of folders matching the names of the Team Project Collections on the server that are available to you.  Choose the appropriate TPC.

6.  You will now see the list of Team Projects in the TPC that are available to you.  Choose the appropriate project and inside the TP, the appropriate folder in which to save the report.  It is worthwhile to note that the names of folders inside each TP are not hard-coded into TFS – rather, they depend on the process template that was selected for each project (for example, MSF for Agile Software Development or MSF for CMMI Process Improvement).

7.  Click Save.  The report has now been uploaded to the server.

8.  Open Visual Studio (or Visual Studio Team Explorer) and navigate to the relevant project and folder.  Your report should now be visible.  If VS was already open before you saved the report, you need to right-click the Reports node in the Team Explorer window and choose Refresh.

And there it is!  The report is now available to our users.  Of course, permissions are available in SSRS to make sure that not everyone can see everything…

Summary

In this series of posts we saw how to build a basic TFS report step by step: 

1. Choose the appropriate TFS data source and create a query
2. Prepare a report and integrate the query into it, including any parameters that need to be entered by the user
3. Add any additional features to the report, such as graphical annotations
4. Publish the report to the server

From here on, the way is open for you to explore the capabilities of both the TFS databases (which contain A LOT of useful information on pretty much anything that happens inside of TFS) and the SSRS infrastructures.  Good Luck!

Writing a Simple TFS Report–Part 3: Adding the Final Touches

In the last post of this series, we created a report that visually displayed the data we extracted from the TFS 2010 warehouse.  In this post, we’ll add some additional enhancements to the report in order to increase its effectiveness. 

As you recall, we want our report to look like this:

While right now our report looks like this:

This means we need to add the following enhancements:

1.  Change the report’s title and fields

2.  Make sure that cells which have a reason of Fixed have a green background

Changing the Title and Field Headers

Since we haven’t added a title yet, ReportBuilder prominently asks us to. 

Simply click the label with the contents Click to add title and change it to A Bug’s Life

In a similar manner, click the header for each one of the cells in the table (which is called a Tablix in ReportBuilder parlance) and change the text and width to fit the new text:

Another small modification we would like to make to the report is in the System_ChangedDate field.  You have probably noticed that state change dates are displayed using a full date/time format.  We would like to only show the dates.

1.  Click the System_ChangedDate field in the tablix (not the header)

2.  In the ribbon at the top of the screen, find the section labeled Number.  Note the combobox that currently has the value Default

3.  The default format does not work for us, since it shows the entire the contents of the date/time value we pulled from the database.  Change the combo to read Date instead.

Annotating Selected Fields

We want to explicitly bring out the fields in which the reason for the state change is that the bug was fixed.  This will allow us to tell at a glance which bugs were resolved and which were not.  In order to do this, we’ll need to write some code:

1.  Right-click the System_Reason field in the tablix (not the header) and choose Text Box Properties….  The Text Box Properties dialog should come up:

This dialog allows to modify a lot of things in the way the field text box is displayed, both visually and functionally. 

2.  Click the Fill option in the left pane of the dialog. 

3.  Find the control labeled Fill Color.  Notice the small button next to it labeled Fx?  This will allow us to specify a (code) function for the text box’s fill color. 

4.  Click the Fx button.  The Expression Editor will launch, allowing us to enter an expression.  The current value of the fill color is No Color – that is, no fill color is selected.

5.  Enter the following code in the expression editor:

=IIF(Fields!System_Reason.Value = "Fixed", "Green", "Transparent")

This code is a VB Immediate If construct – it simply say that if the value of the System_Reason field is equal to the string "Fixed”, then the value of the expression (and that of the fill color) is “Green”.  Otherwise, the value is “Transparent”.  Just what we need. 

6.  Click OK to close the Expression Editor and OK again to close the Text Box Properties dialog. 

7.  Save and run your report.  Success!

What’s Next?

We have the report looking just like we wanted it to.  Now it is time to upload it to the server and make it available for everyone to see.  We’ll do this next time.  Happy New Year!

Writing a Simple TFS Report – Part 2: Creating the Report

Welcome back!  In the last part of this series we discussed the actual query needed to extract the data from the TFS relational data warehouse.  In this post, we’ll see how to use this query in order to create an actual report. 

Choosing a Report Authoring Tool

In order to actually produce our report we need a report authoring tool.  This tool should allow us to design our report in a WYSIWYG manner and specify the query for getting the data.  We would normally choose from the following options:

• Business Intelligence Development Studio (BIDS) – This tool is available from the SQL Server 2008 installation media.  It is a special version of Microsoft Visual Studio 2008 which is tailored for database-oriented projects.  As such, it supports the developer-oriented functionality of Visual Studio – managing projects, integrating with source-control systems, etc.  BIDS is a good choice if you want to treat your report-authoring endeavors as development projects.
• Report Builder – A freely-available tool from Microsoft that is geared more towards power users who are not developers.  It does not have the project-management features that Visual Studio has, so its UI is easier to use.  ReportBuilder version 3 is needed for SQL Server 2008 R2 while version 2 is used for earlier versions.

It is important to note, though, that both BIDS and ReportBuilder offer the same capabilities as far as the job-at-hand is concerned – it’s just a question of how much additional functionality you would like your tool to contain.  In order to stay as focused as possible, we’ll be using ReportBuilder version 3 for this post.

Creating a New Report

So let’s get going!  Launch ReportBuilder from the Start Menu.  If you get a Getting Started dialog, just choose Blank Report.  Otherwise, you’re good to go.

This is what it looks like on my machine:

On the left we have a pane dealing with the data for this report.  In the middle we have the layout editor.  Finally, on the right there is the properties window for dealing with specific object properties.

Here are the steps we’ll be taking to create our Bug’s Life report:

1.  Add a user parameter

2.  Add a data source

3.  Add a specific data set

4.  Add a table for showing our data

Adding a User Parameter

In this step, we’ll add a user parameter to the report.  This means that prior to running the report the user will be able to supply some values to customize the output.  In our case, this is the list of bug ID numbers for which we would like to see the state transitions.

1.  In the Report Data pane, right-click the Parameters folder and choose Add Parameter….  You should get the Report Parameter Properties Dialog. 

2.  Change the Name field to read bugIdsParam.  For convenience, I gave the report parameter the same name as the parameter name that’s used in the SQL query.

3.  Change the Prompt field to read Bug Ids:.  This is what will be displayed to the user.  Your dialog window should resemble this:

4.  There are many options you can set for parameters.  For example, you can set the available and/or default values of a parameter from queries or based on the values of other parameters.  In our case we have no need for this functionality, so just click the OK button.  You should have a new parameter in the Report Data pane:

Adding a Data Source

In this step, we’ll specify what the source for our data is.  As you recall, we opted to use the TFS 2010 relational data warehouse, so we can use SQL.  We now need to enter this information into the report.

1.  In the Report Data pane, right-click the Data Sources folder and choose Add Data Source….  You should get the Data Source Properties dialog.

2.  In the dialog, choose the Tfs2010ReportDS data source.  This is the relational data warehouse.  If it is not available in the list of data sources, click the Browse button and enter the name of your reporting server in the Name field.  This will typically be something like http://<TFS Server Name>/ReportServer.  Once you connect to the server, the available data sources are typically located directly under the ReportServer folder.

3.  For clarity, change the name of the data source from the default DataSource1 to Tfs2010ReportDS.  This is especially useful if you’re using both the relational and OLAP data sources and you need to differentiate between them.  Your dialog window should resemble this:

4.  Click OK.  You should now have a new data source in the Report Data pane:

Adding a Data Set

In this step, we’ll add an actual data set – that is, a set of data that we would like to do something with.  As mentioned before, data sets can be used for determining the values of parameters.  In our case, however, we would like a data set for the much more basic purpose of simply showing it in the report.  This means we’ll have to define the data set as containing the query we developed earlier.  So let’s do it:

1.  In the Report Data pane, click the Tfs2010ReportDS data source (under the Data Sources folder) and choose Add Dataset….  You should get the Dataset Properties dialog.

2.  Change the Name field to read dsBugs. 

3.  Copy the SQL query from the previous post into the Query field.  Alternatively, you may click the Query Designer button in order to develop your query directly from ReportBuilder.  This is useful when you have need to develop quick and simple queries on-the-spot, but I would recommend that you use SQL Server Management Studio to develop your queries ahead of time.  The designer is useful, however,  for making modifications and corrections once the initial query is in place.  Your dialog window should resemble this:

4.  In the left-hand side of the dialog, choose the Parameters item.  This will allow us to connect parameters in your dataset (i.e., the SQL query) to the parameters we defined a the report level (i.e., the values that need to come from the user).  Note that ReportBuilder has already recognized that the query requires a parameter, and is offering you the chance to connect this parameter at the report level.  Choose the value [@bugIdsParams] from the combo box under the Parameter Values column.  Your dialog window should resemble this:

5.  Click the OK button.  You should now have a new data set in the Report Data pane:

Note that you now have access to each field in the original query. 

Adding a Table for Showing Data

And now for the fun part!  We’ve done all the behind-the-scenes work for connecting our report to user-specified parameters and queries.  Let’s now add the dataset to the report!

1.  In the ribbon bar at the top of screen, move to the Insert tab. 

2.  Click the Table ribbon button and choose Table Wizard…. You should now get the table wizard.

3.  In the Choose a dataset wizard page, choose the dataset you would like to add to your report.  We currently have only one (our dsBugs dataset) so select it and click the Next button.

4.  In the Arrange fields wizard page, drag the System_Id field to the Row Groups list box and all other fields to the Values list box.  Your wizard page should look like this:

In essence, what we have done is told ReportBuilder that we wish to group our bugs according to their ID number – that is, have multiple lines showing the fields in the Values list box for each field in the Row groups list box.

5.  Click the Next Button.

6.  In the Choose the layout wizard page, click Next.

7.  In the Choose a style wizard page, select the Slate style and click Finish.  We will now have a table on the design surface:

Running the Report

We’re ready to run the report for the first time!  On the ribbon bar, click the Run button.  After a short wait, in which SSRS is processing the report, your screen should resemble the following:

Enter a comma-separated list of bugs and click the View Report button on the right-hand side.  You should now see this:

We’ve done it!

You can now use the Design button (that has replaced the Run button) to go back to design mode.  Make sure you save your report – we’ll continue working on it next time.

What’s Next?

We created a report and have shown it using live data.  However, we’re not done.  The report works, but we still need to change the title, column names and add a green background whenever the Reason is set to Verified.  I’ll explain how to do that next time.  See you then!