Quality-Jitsu, or Learning to Empty Your Cup

Aikido, and I suppose all martial arts, are painful.  Mostly physically painful, depending on how skilled – or rather, how unskilled – the person on the other end of your joint is.  But there is also a great deal of figurative pain.  It’s the pain of letting go of your ego and the pain of emptying your cup.  It’s the acceptance that someone half your size (or less) can be twice (or more) as effective as you are, disabling and throwing you around seemingly without even trying.  It’s understanding that no, it’s not as trivial as your teacher makes it look and that yes, it’s going to require work to do it effortlessly.  A LOT of work.  

In the years that I’ve been doing Aikido, I’ve seen many people come and go through our school doors.  For some, it truly was physical – too much inter-personal contact, extra-sensitive joints and at least in one case, an almost-broken knee.  But for the grand majority it was not.  It was the inability to learn something ‘easy’ from someone else.  These are the kind of people who came to teach junior students, rather than learn from the senior ones.  In fact, one of the things I admire most about my teacher is his endless patience, which eventually resulted in many students changing from the former kind into the latter (myself included).

But this is a tech blog.  So what’s all this talk of martial arts?  Well, getting software quality to look effortless and easy takes work.  A LOT of work.  And with the state of the practice that I see at some software organizations today, it definitely feels like a battlefield.  Here’s a typical scenario:  a reasonably successful organization will happily spend days squeezing the last ounce of performance out of some algorithm.  And they’ll readily acknowledge that they're not able to iterate quickly enough, or ship with a low enough number of bugs (zero-bugs policy, anyone?), or deliver on-time enough.  Yet when the talk turns to actually doing something about it, the same organizations suddenly know everything there is to know, have done everything there is to do and can generally find a 6-digit number of reasons for why THEIR product quality simply CANNOT be improved.

So, if you’re not willing (notice I said willing – not a word about able) to change anything you’re doing today, what in the name of Miyamoto Musashi makes you think your quality will improve?  What, some fairy will drop down from the sky and magically exorcise all bugs from your code?  Yes, it IS as ridiculous as it sounds.  Wearing a keikogi or not, I’m seeing the same type of people that cannot cut it in a dojo.  You WILL have to write some kind of automated tests.  You WILL have to set up some kind of automated build.  And it’s going to take work.  A LOT of work.  Whether you eventually succeed or not depends on how empty your cup is and how much you’re willing to learn from others, be they younger, older, bigger or smaller than you.

So when someone approaches me at a conference, sends me an email or asks my advice I start by asking them two questions:

1. What have you tried already?
2. Why do want to do whatever it is that you want done?

The answer to #1 helps me make sure my cup is empty (well, not completely full is a good start…).  It’s amazing how resourceful and smart people never cease to surprise me.  My favorite answer to #2:  ‘It hurts.  How do I make it hurt less?’. 

Onegai Shimasu!

TFS 11 מעט על התצורות החדשות הצפויות לנו עם

More on Team Build 2010 Extensions

In my last post, I touched briefly on the concept of build extensions, and explained that they are an implementation of WF4 extensions.  I thought it would be useful to list the other build extensions that are available inside of a build template:

• IBuildDetail – This build extension enables you to notify the build server about important aspects of your build (compilation/test status, overall build status, the label associated with the build, etc.).  In fact, this is the how the SetBuildDetails built-in activity is constructed.  So a typical use-case for this extension could be, for example, to fail the build when your activity encounters an error condition, despite having succeeded in compiling or running the tests.  However, if you choose this route, I strongly urge you to use the composition approach rather than the code approach.
• IBuildLoggingExtension – This extension is useful for controlling which activities are logged into the build log. 
• IAgentReservationExtension – This extension seems to be used for actually requesting the server to reserve/free an agent for running a specific workflow.  This looks to be very interesting for doing complex dependency-management stuff – when I build component A, I’d like to make sure that the latest version of component B (upon which A is dependent) is built, which is a separate build altogether.  I intend to try this out soon.
• IBuildAgent – This extension allows you to retrieve details of the build agent that is currently running your build.  Of course, this is only relevant for activities that run inside an AgentScope.

As I mentioned before, using extensions requires 2 stages:

1.  Declare that you will need the extension inside the activity’s CacheMetadata method, using the metadata’s RequireExtension<T> method

2.  Retrieve the extension inside the execution method (Execute/BeginExecute) using the activity context’s GetExtension<T> method

Add Chuck Norris to Your Build!

In this post I’ll show how to write a somewhat more complex and hopefully fun activity for TFS 2010 Team Build (code is available here).  In particular, I’ll make use of the following features:

• The ActivityTrackingAttribute class
• Build Extensions (a specialization of WF4 Workflow Extensions)

Since our builds are sometimes long and dull processes, we need some stuff to talk about while they run.  What could be better than discussing the prowess of the one-and-only Chuck Norris?  We’ll have Team Build produce a new Chuck Norris (CN) fact every time a build runs:

And now we have a good conversation starter!  Who knows, maybe this is a good way to convince team members to run the build more often during the day…

Design of the CN Activity

In order to implement the activity we need to deal with the following issues:

1. Finding a source of Chuck Norris facts to display and choosing a random fact each time
2. Displaying the fact in the Build Log View
3. Maintaining good Separation of Concerns (SOC) – that is, making sure the code is well-factored

Let’s tackle these one at a time.

Finding the Data

This was actually rather easy.  A quick Google search turned up The Internet Chuck Norris Database, a hilarious site which even contains a REST API!   Great, so this means we can get a single joke back each time by writing appropriate WCF service and data contracts and using the WebHttp binding for interacting with REST services (many thanks to Yaniv Rodensky for helping me out with the WCF details).

Displaying the Data

We want to display the data in the Build Log View – that is, the log in which TFS shows the on-going activity of the build process.  Fortunately, we have the built-in WriteBuildMessage activity for doing precisely this.  It would probably make sense to structure our code in terms of two separate activities:

• GetChuckNorrisFact – Performs the Web access itself and returns the joke text.  Since this is in fact an IO operation, we want to derive it from AsyncCodeActivity<string> in order to take advantage of the WF4 runtime’s ability to run activities in an asynchronous manner for making the workflow as efficient as possible.
• DisplayChuckNorrisFact – Compose the GetChuckNorrisFact and WriteBuildMessage activities into a single building block, which we’ll use in our builds.  This enclosing activity will also include error handling by way of the built-in TryCatch activity.   Since DCNF is the only user of GCNF, it makes sense that we make the latter internal rather than public, which in turn means that the WCF interfaces and contracts can be internal as well.  Encapsulation galore!

The interesting question here is why would we choose to create DCNF as a composition of activities rather then simply deriving from CodeActivity and writing everything out in code.  I would say that the reason for doing it this way has to do with staying at the proper abstraction level and making good use of the underlying WF4 technology.  If we consider Team Build 2010 to be a domain-specific language (where our domain is the automated building of software), than activity composition is in fact the creation of compound words.  Taking the language metaphor even further -  GCNF and DCNF are content words while the other activities (TryCatch, WriteBuildMessage, etc.) are function words and fortunately for us, have already been implemented by Microsoft.  I find this similarity to natural language to probably be the least-understood aspect of WF4, but that’s for some other time (and some other post).

Another interesting point with the composite activity has to do with what exactly is displayed in the log.  The structure of DCNF is as follows, with the Try/Catch/Finally boxes being properties in the TryCatch activity:

Recall the way TFS displays the activities in the template – the DisplayName of each activity is used in the Build Log View and indentation is determined by how deep it is in the build workflow template.  So what we would have expected to see is the name for each one of the internal composed activities, like such:

The fact that this is not so is due to the use of the ActivityTrackingAttribute class.  Notice the attribute on the DisplayChuckNorrisFact activity class:

    [ActivityTracking(ActivityTrackingOption.ActivityOnly)]
    public sealed class DisplayChuckNorrisFact : Activity
    {
        private const string FailureText =
            "Chuck Norris must have brought down the Internet, " +
            "that's why you can't have any facts!";

        public DisplayChuckNorrisFact()
        {
            Implementation = () => CreateBody();
        }

        private Activity CreateBody()
        {
            var factTextVariable = new Variable<string>("factText");
            var tryCatch = new TryCatch();
            tryCatch.Variables.Add(factTextVariable);

            tryCatch.Try = new GetChuckNorrisFact {Result = factTextVariable};
            tryCatch.Catches.Add(new Catch<Exception>
            {
                Action =
                    new ActivityAction<Exception>
                        {
                            Handler =
                                new Assign<string>
                                    {
                                        To = factTextVariable,
                                        Value = FailureText
                                    }
                        }
            });
            tryCatch.Finally = new WriteBuildMessage
            {
                Importance = BuildMessageImportance.High, 
                Message = factTextVariable
            };

            return tryCatch;
        }
    }
}

This will ensure that only the top-level activity’s DisplayName will be shown in the Build Log View.  Other options are:

• ActivityTrackingOption.ActivityTree – Output each and every activity’s DisplayName, both top-level and composed  (this is the default option)
• ActivityTrackingOption.None – Do not output any activity’s DisplayName, neither top-level nor composed

Using this attribute adds a level of control that is more involved (but still possible) to achieve with ‘plain’ C# code (using the CodeActivity route).

Maintaining Separation of Concerns

We already saw some SOC-related issues – namely, the separation of the logical ‘display a random fact’ operation into multiple physical WF4 activities.  We also saw that there are some pretty good reasons to do so, even if it means additional work.  In a similar manner, we would also like to separate the GetChuckNorrisFact activity into two separate parts – the WCF service proxy and a WF4 activity that uses it.

If we were calling a regular SOAP service it would have been very easy to use Visual Studio’s Add Service Reference functionality (or the equivalent svcutil.exe) to generate code for this service’s proxy so we could use it as a black box.  Unfortunately this isn’t currently possible with a REST service, so we need to create this proxy ourselves.  So here’s what we end up with in the Visual Studio project:

• IChuckNorrisFactService.cs: This file contains several DataContracts and a ServiceContract.  Note that the latter is decorated with:
  • A WebInvoke attribute - signifies that this method is activated using an HTTP GET verb, along with the URI template which maps method parameters (on the client) to the URL passed to the service
  • An AsyncPattern property in the OperationContract – signifies that this method is part of an APM method pair that will be used for asynchronous communication with the service
• ChuckNorrisWebServiceConsumer.cs:  This is our service proxy.  It uses the WebChannelFactory<T> class along with a WebHttpBinding to connect to the service and retrieve the raw data.  It then uses a DataContractJsonSerializer to deserialize the returned JSON text into .NET objects that we can pass back to the caller.  (As a side note,  the reason for the existence of this class is because I could not get WCF to deserialize the JSON text automatically.  This might have to do with the fact that the service returns its response with a Content-Type header of ‘text\html’ rather than ‘application\json’ or something similar.)
• The two activities discussed earlier, implemented to use the APM pattern

So now we need to make sure that the GetChuckNorrisFact activity has access to the service proxy.  The easiest way to do this is to simply create a new instance of the proxy inside the activity.  However, this would increase our coupling as GCNF would now need to know how to create an instance of the proxy (not really an issue with this particular implementation, but I hope the point is clear).  Instead, we opt to use a different mechanism, whereby we expect the WF4 host – TFS, in our case – to supply us with an instance of the proxy. 

Note the attribute on the ChuckNorrisWebServiceConsumer class:

   [BuildExtension(HostEnvironmentOption.All)]
   public class ChuckNorrisWebServiceConsumer
   {
       private const string ServiceAddress = "http://api.icndb.com";
       private IChuckNorrisFactService channel;

       public IAsyncResult BeginGetFact(AsyncCallback callback, object state)
       {
           using (
               var cf = 
                   new WebChannelFactory<IChuckNorrisFactService>(
                       new WebHttpBinding(), new Uri(ServiceAddress)))
           {
               channel = cf.CreateChannel();
               return channel.BeginGetFacts("1", callback, state);
           }
       }

       public string EndGetFact(IAsyncResult result)
       {
           var serializer = new DataContractJsonSerializer(typeof (Data));
           var data = (Data) serializer.ReadObject(channel.EndGetFacts(result));

           return data.Jokes.First();
       }
   }

The attribute instructs TFS to register this class as a WF4 workflow extension at the build server level, where it can be requested by any activity (think ‘Service Locator Pattern’).  Similar to the BuildActivityAttribute class, it takes a HostEnvironmentOption enumeration which tells it whether the extension may be used by activities that run on the build controller, the build agent or both.

An activity requests the extension by doing two things (see GetChuckNorrisFact.cs):

1.  In the CacheMetadata method, it must notify the workflow host that it expects to use the extension:

protected override void CacheMetadata(CodeActivityMetadata metadata)
{
    base.CacheMetadata(metadata);
    metadata.RequireExtension<ChuckNorrisWebServiceConsumer>();
}

Since CacheMetadata is run before the workflow starts executing, WF4 can verify that the activity has indeed been registered with the server and is available for use.  If it has not (for example, we forgot to deploy the custom activity DLL to the build server), then the WF4 runtime will throw an exception.  In our terms, this means we will immediately get a build failure – before any  build-related activities have run.

2.  Inside the activity’s execution methods (Execute for CodeActivity and BeginExecute for AsyncCodeActivity), it must request the extension from the activity’s context:  

protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context,
    AsyncCallback callback, object state)
{
    var serviceProxy = context.GetExtension<ChuckNorrisWebServiceConsumer>();
    context.UserState = serviceProxy;
    return serviceProxy.BeginGetFact(callback, state);
}

Summary

So there we have it – our very own Chuck Norris fact-retrieving activity for integration into the build.  We saw how to achieve this task using WF4 activity composition, workflow extensions and WCF REST bindings.  While not overly complex, it does show that there is some effort and knowledge required to properly write build activities.  Happy building!

Writing a Simple TFS Report – Part 4: Publishing the Report

In the last post of this series we completed our report so that it contains all the graphical annotations we wanted.  It is now time to publish it to the server and make it available to our users. 

Publishing the Report to the Server

Once we have a report running on the local machine, publishing it is the easy part! 

1.  Open the report that we worked on throughout the previous parts of the series

2.  From the main ribbon button, choose the Save As menu option.  The Save As Report file dialog will appear.

3.  Now comes the easy part:  In the Name textbox, enter the URL for your TFS installation of Reporting Services.  This will typically be of the form http://<TFS SERVER>/ReportServer. 

4.  You should now see a folder named simply TfsReports.

5.  Open the TfsReports folder.  You will now see a list of folders matching the names of the Team Project Collections on the server that are available to you.  Choose the appropriate TPC.

6.  You will now see the list of Team Projects in the TPC that are available to you.  Choose the appropriate project and inside the TP, the appropriate folder in which to save the report.  It is worthwhile to note that the names of folders inside each TP are not hard-coded into TFS – rather, they depend on the process template that was selected for each project (for example, MSF for Agile Software Development or MSF for CMMI Process Improvement).

7.  Click Save.  The report has now been uploaded to the server.

8.  Open Visual Studio (or Visual Studio Team Explorer) and navigate to the relevant project and folder.  Your report should now be visible.  If VS was already open before you saved the report, you need to right-click the Reports node in the Team Explorer window and choose Refresh.

And there it is!  The report is now available to our users.  Of course, permissions are available in SSRS to make sure that not everyone can see everything…

 

Summary

In this series of posts we saw how to build a basic TFS report step by step: 

1. Choose the appropriate TFS data source and create a query
2. Prepare a report and integrate the query into it, including any parameters that need to be entered by the user
3. Add any additional features to the report, such as graphical annotations
4. Publish the report to the server

From here on, the way is open for you to explore the capabilities of both the TFS databases (which contain A LOT of useful information on pretty much anything that happens inside of TFS) and the SSRS infrastructures.  Good Luck!

Writing a Simple TFS Report–Part 3: Adding the Final Touches

In the last post of this series, we created a report that visually displayed the data we extracted from the TFS 2010 warehouse.  In this post, we’ll add some additional enhancements to the report in order to increase its effectiveness. 

As you recall, we want our report to look like this:

While right now our report looks like this:

This means we need to add the following enhancements:

1.  Change the report’s title and fields

2.  Make sure that cells which have a reason of Fixed have a green background

 

Changing the Title and Field Headers

Since we haven’t added a title yet, ReportBuilder prominently asks us to. 

Simply click the label with the contents Click to add title and change it to A Bug’s Life

 

In a similar manner, click the header for each one of the cells in the table (which is called a Tablix in ReportBuilder parlance) and change the text and width to fit the new text:

Another small modification we would like to make to the report is in the System_ChangedDate field.  You have probably noticed that state change dates are displayed using a full date/time format.  We would like to only show the dates.

1.  Click the System_ChangedDate field in the tablix (not the header)

2.  In the ribbon at the top of the screen, find the section labeled Number.  Note the combobox that currently has the value Default

3.  The default format does not work for us, since it shows the entire the contents of the date/time value we pulled from the database.  Change the combo to read Date instead.

 

 

Annotating Selected Fields

We want to explicitly bring out the fields in which the reason for the state change is that the bug was fixed.  This will allow us to tell at a glance which bugs were resolved and which were not.  In order to do this, we’ll need to write some code:

1.  Right-click the System_Reason field in the tablix (not the header) and choose Text Box Properties….  The Text Box Properties dialog should come up:

This dialog allows to modify a lot of things in the way the field text box is displayed, both visually and functionally. 

2.  Click the Fill option in the left pane of the dialog. 

3.  Find the control labeled Fill Color.  Notice the small button next to it labeled Fx?  This will allow us to specify a (code) function for the text box’s fill color. 

4.  Click the Fx button.  The Expression Editor will launch, allowing us to enter an expression.  The current value of the fill color is No Color – that is, no fill color is selected.

5.  Enter the following code in the expression editor:

=IIF(Fields!System_Reason.Value = "Fixed", "Green", "Transparent")

This code is a VB Immediate If construct – it simply say that if the value of the System_Reason field is equal to the string "Fixed”, then the value of the expression (and that of the fill color) is “Green”.  Otherwise, the value is “Transparent”.  Just what we need. 

6.  Click OK to close the Expression Editor and OK again to close the Text Box Properties dialog. 

7.  Save and run your report.  Success!

 

What’s Next?

We have the report looking just like we wanted it to.  Now it is time to upload it to the server and make it available for everyone to see.  We’ll do this next time.  Happy New Year!

Writing a Simple TFS Report – Part 2: Creating the Report

Welcome back!  In the last part of this series we discussed the actual query needed to extract the data from the TFS relational data warehouse.  In this post, we’ll see how to use this query in order to create an actual report. 

 

Choosing a Report Authoring Tool

In order to actually produce our report we need a report authoring tool.  This tool should allow us to design our report in a WYSIWYG manner and specify the query for getting the data.  We would normally choose from the following options:

• Business Intelligence Development Studio (BIDS) – This tool is available from the SQL Server 2008 installation media.  It is a special version of Microsoft Visual Studio 2008 which is tailored for database-oriented projects.  As such, it supports the developer-oriented functionality of Visual Studio – managing projects, integrating with source-control systems, etc.  BIDS is a good choice if you want to treat your report-authoring endeavors as development projects.
• Report Builder – A freely-available tool from Microsoft that is geared more towards power users who are not developers.  It does not have the project-management features that Visual Studio has, so its UI is easier to use.  ReportBuilder version 3 is needed for SQL Server 2008 R2 while version 2 is used for earlier versions.

It is important to note, though, that both BIDS and ReportBuilder offer the same capabilities as far as the job-at-hand is concerned – it’s just a question of how much additional functionality you would like your tool to contain.  In order to stay as focused as possible, we’ll be using ReportBuilder version 3 for this post.

 

Creating a New Report

So let’s get going!  Launch ReportBuilder from the Start Menu.  If you get a Getting Started dialog, just choose Blank Report.  Otherwise, you’re good to go.

This is what it looks like on my machine:

On the left we have a pane dealing with the data for this report.  In the middle we have the layout editor.  Finally, on the right there is the properties window for dealing with specific object properties.

Here are the steps we’ll be taking to create our Bug’s Life report:

1.  Add a user parameter

2.  Add a data source

3.  Add a specific data set

4.  Add a table for showing our data

 

Adding a User Parameter

In this step, we’ll add a user parameter to the report.  This means that prior to running the report the user will be able to supply some values to customize the output.  In our case, this is the list of bug ID numbers for which we would like to see the state transitions.

1.  In the Report Data pane, right-click the Parameters folder and choose Add Parameter….  You should get the Report Parameter Properties Dialog. 

2.  Change the Name field to read bugIdsParam.  For convenience, I gave the report parameter the same name as the parameter name that’s used in the SQL query.

3.  Change the Prompt field to read Bug Ids:.  This is what will be displayed to the user.  Your dialog window should resemble this:

4.  There are many options you can set for parameters.  For example, you can set the available and/or default values of a parameter from queries or based on the values of other parameters.  In our case we have no need for this functionality, so just click the OK button.  You should have a new parameter in the Report Data pane:

 

Adding a Data Source

In this step, we’ll specify what the source for our data is.  As you recall, we opted to use the TFS 2010 relational data warehouse, so we can use SQL.  We now need to enter this information into the report.

1.  In the Report Data pane, right-click the Data Sources folder and choose Add Data Source….  You should get the Data Source Properties dialog.

2.  In the dialog, choose the Tfs2010ReportDS data source.  This is the relational data warehouse.  If it is not available in the list of data sources, click the Browse button and enter the name of your reporting server in the Name field.  This will typically be something like http://<TFS Server Name>/ReportServer.  Once you connect to the server, the available data sources are typically located directly under the ReportServer folder.

3.  For clarity, change the name of the data source from the default DataSource1 to Tfs2010ReportDS.  This is especially useful if you’re using both the relational and OLAP data sources and you need to differentiate between them.  Your dialog window should resemble this:

4.  Click OK.  You should now have a new data source in the Report Data pane:

 

Adding a Data Set

In this step, we’ll add an actual data set – that is, a set of data that we would like to do something with.  As mentioned before, data sets can be used for determining the values of parameters.  In our case, however, we would like a data set for the much more basic purpose of simply showing it in the report.  This means we’ll have to define the data set as containing the query we developed earlier.  So let’s do it:

1.  In the Report Data pane, click the Tfs2010ReportDS data source (under the Data Sources folder) and choose Add Dataset….  You should get the Dataset Properties dialog.

2.  Change the Name field to read dsBugs. 

3.  Copy the SQL query from the previous post into the Query field.  Alternatively, you may click the Query Designer button in order to develop your query directly from ReportBuilder.  This is useful when you have need to develop quick and simple queries on-the-spot, but I would recommend that you use SQL Server Management Studio to develop your queries ahead of time.  The designer is useful, however,  for making modifications and corrections once the initial query is in place.  Your dialog window should resemble this:

4.  In the left-hand side of the dialog, choose the Parameters item.  This will allow us to connect parameters in your dataset (i.e., the SQL query) to the parameters we defined a the report level (i.e., the values that need to come from the user).  Note that ReportBuilder has already recognized that the query requires a parameter, and is offering you the chance to connect this parameter at the report level.  Choose the value [@bugIdsParams] from the combo box under the Parameter Values column.  Your dialog window should resemble this:

5.  Click the OK button.  You should now have a new data set in the Report Data pane:

Note that you now have access to each field in the original query. 

 

Adding a Table for Showing Data

And now for the fun part!  We’ve done all the behind-the-scenes work for connecting our report to user-specified parameters and queries.  Let’s now add the dataset to the report!

1.  In the ribbon bar at the top of screen, move to the Insert tab. 

2.  Click the Table ribbon button and choose Table Wizard…. You should now get the table wizard.

3.  In the Choose a dataset wizard page, choose the dataset you would like to add to your report.  We currently have only one (our dsBugs dataset) so select it and click the Next button.

4.  In the Arrange fields wizard page, drag the System_Id field to the Row Groups list box and all other fields to the Values list box.  Your wizard page should look like this:

In essence, what we have done is told ReportBuilder that we wish to group our bugs according to their ID number – that is, have multiple lines showing the fields in the Values list box for each field in the Row groups list box.

5.  Click the Next Button.

6.  In the Choose the layout wizard page, click Next.

7.  In the Choose a style wizard page, select the Slate style and click Finish.  We will now have a table on the design surface:

 

Running the Report

We’re ready to run the report for the first time!  On the ribbon bar, click the Run button.  After a short wait, in which SSRS is processing the report, your screen should resemble the following:

Enter a comma-separated list of bugs and click the View Report button on the right-hand side.  You should now see this:

We’ve done it!

You can now use the Design button (that has replaced the Run button) to go back to design mode.  Make sure you save your report – we’ll continue working on it next time.

 

What’s Next?

We created a report and have shown it using live data.  However, we’re not done.  The report works, but we still need to change the title, column names and add a green background whenever the Reason is set to Verified.  I’ll explain how to do that next time.  See you then!

Writing A Simple TFS Report – Part 1: Getting the Data

In the last post we saw the final result of what we’re aiming for – the Bug’s Life report.  In this post, we’ll start working on the report by actually extracting the data.

The TFS Reporting Data Sources

TFS supplies us with two different data sources for reporting needs:

1. The data warehouse – A relational data store that is designed using a snowflake schema and is named Tfs_Warehouse.  By default, this database is updated within 30 minutes of an anything ‘interesting’ happening in TFS.  We query this database using the SQL language.
2. The cube – A multidimensional OLAP cube.  By default, the cube updates within 2 hours of something happening in TFS, and takes its data from the relational store.  We query the cube using the MDX language.

For further details on these sources, see the “Running TFS 2010 Reports” module of the Introduction to TFS 2010 Training Kit.  In this post, we’ll use the data warehouse and write our query in SQL.  For this purpose, I’ll be using SQL Server Management Studio.

Choosing The Right View

Since we’re dealing with work items, the proper place to start is with the relevant views.  These contain columns for all work item fields, across all team projects and team project collections.  There are two main views that might be interesting for our type of query:

1. The CurrentWorkItemView – This view holds the current state of all work items.  Using this view is similar to running work item queries, in that we get back the most recent data.
2. The WorkItemHistoryView – This view differs from the previous one in that it also contains the entire history of each work item – including all changes to all fields.  In essence, this view contains the same information that you might see in a work item’s History field.

Writing the Actual Query

Obviously, the second view is what we’re interested in.  So let’s try this simple query:

  

  SELECT System_Id, System_ChangedDate, System_State, System_Reason, 
    System_ChangedBy, RecordCount, StateChangedCount 
  FROM dbo.WorkItemHistoryView 
  WHERE System_Id IN (10158, 11025, 11026)

Notice that we’re pulling out all of the fields that are of interest to us (and a couple more as well, as I will explain momentarily).  The warehouse stores fields according to their reference names, as set in the work item definition.

Here are the results: 

So much data for 3 bugs, each with 3 state transitions?  Turns out that history tables/views in TFS work with a technique called compensating records.  In simple terms, this means that whenever we wish to change the value of some field, we need to insert two records into the database:  the first cancels the existing state (‘compensates’ for it) while the second contains the new values.  The RecordCount field contains the value –1 for a compensating record, and 1 when actual values are set.  

So given a hypothetical work item with fields Field1, Field2 and Field 3, if we changed the value of Field1 from ‘value1’ to ‘value4’, we’ll see something like this in the database:

RecordCount	Field1	Field2	Field3
1	value1	value2	value3
-1	value1	value2	value3
1	value4	value2	value3

The records with the bolded values are the ones that enter new information.  The additional record in between effectively ‘zeros out’ the state of the work item during the transition.  For a more complete explanation see here.

In a similar manner, the StateChangeCount field will contain the value 1 when a change has taken place in the System_State column – just what we need!  So we need to add an additional clause to our WHERE statement:

  

  SELECT System_Id, System_ChangedDate, System_State, System_Reason, 
    System_ChangedBy, RecordCount, StateChangedCount
  FROM dbo.WorkItemHistoryView
  WHERE System_Id IN (10158, 11025, 11026) AND StateChangeCount = 1

The results now:

Much better!  Note also that for each record in our results, the RecordCount field is also equal to 1.  This tells us that we are indeed looking at information that was new at the time it was entered.

The Final Query

Here now is the final query that we end up with:

  


-- String splitting code adapted from http://stackoverflow.com/questions/2647/split-string-in-sql
DECLARE @tblBugIds table
(
    BugId varchar(10)
)

DECLARE @bugId varchar(10)
DECLARE @intPosition int

DECLARE @allBugIds nvarchar(255);
SET @allBugIds = LTRIM(RTRIM(@bugIdsParam)) + ','
SET @intPosition = CHARINDEX(',', @allBugIds, 1)

IF REPLACE(@allBugIds, ',', '') <> ''
BEGIN
    WHILE @intPosition > 0
    BEGIN
        SET @bugId = LTRIM(RTRIM(LEFT(@allBugIds, @intPosition - 1)))
        IF @bugId <> ''
        BEGIN
            INSERT INTO @tblBugIds (BugId) VALUES (@bugId)
        END
        SET @allBugIds = RIGHT(@allBugIds, LEN(@allBugIds) - @intPosition)
        SET @intPosition = CHARINDEX(',', @allBugIds, 1)
    END
END



SELECT System_Id, System_ChangedDate, System_State, System_Reason, 
  System_ChangedBy
FROM dbo.WorkItemHistoryView
WHERE System_Id IN (SELECT * FROM @tblBugIds)
      AND StateChangeCount = 1
      AND System_WorkItemType = 'Bug'

  

This query contains a few additional things:

• Code for breaking up a user parameter containing a string of comma-separated bug ID values into a temporary table.  This is used to handle a variable number of bug IDs in the query.
• A further stipulation in the WHERE clause that a work item must be of type ‘Bug’ (although the query would work just as well with all other work item types)

What’s Next?

Now that we know how to get the data, it’s time to put it into an actual report.  In the next post we’ll open up Report Builder, build a layout for our report and add our query into it.  Till next time!

“IBuildDetail not defined” error in TFS 2010 Build Templates

Recently I started receiving the following error when trying to edit TFS 2010 build templates on my machine:

Type ‘IBuildDetail’ is not defined

This was exceptionally weird as TFS was running happily for quite a while.  Turns out the culprit was the Developer Preview edition of Visual Studio 11 that I installed alongside the existing VS2010. 

Anyway, in order to solve the problem:

1.  Open your build template (the .xaml file) in code mode.  (From the Source Explorer, right-click the file, choose ‘View With…’ and select ‘XML (Text) Editor’.

2.  Locate the first line, which starts with <Activity mc:Ignorable=”sad” … >

3.  Replace the following:

 

xmlns:mtbc="clr-namespace:Microsoft.TeamFoundation.Build.Client;
assembly=Microsoft.TeamFoundation.Build.Client"

with

xmlns:mtbc="clr-namespace:Microsoft.TeamFoundation.Build.Client;
assembly=Microsoft.TeamFoundation.Build.Client, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

and

xmlns:mtbwa="clr-namespace:Microsoft.TeamFoundation.Build.Workflow.Activities;assembly=Microsoft.TeamFoundation.Build
.Workflow”

with

xmlns:mtbwa="clr-namespace:Microsoft.TeamFoundation.Build.Workflow.Activities;assembly=Microsoft.TeamFoundation.Build
.Workflow, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

That is, use the full assembly name from VS2010.  Otherwise, the workflow editor gets confused between versions from VS2010 and VS11.

Happy Building!

 

Writing A Simple TFS Report– Part 0: Introduction

In this series of blog posts, I will show how to write a simple report that makes use of the integrated business intelligence (BI) capabilities that are available in TFS 2010.  If you’re unfamiliar with how to use this great feature, this series is for you!  This first post will describe the final “product'” we’d like to construct.

What We’re Aiming For

Our end result for this series is the aptly-named Bug’s Life report:

For each specific bug ID, the report shows us the various states that the bug went through, along with the date, reason and the person responsible for setting this state.  Here are some things that I plan to cover:

1. How to pull information from the TFS 2010 relational data store

2.  How to use the Report Builder tool to create a report and store it in a Report Definition Language (.rdl) file.  In particular, we’ll see how to:

• Design the report
• Add a parameter for the user to fill in
• Change the appearance of a field based on its contents

3.  How to upload the report to a TFS 2010 Team Project, so it is accessible from the Team Explorer window

Why Write a Report?

The question arises – why bother?  We can view the history for each bug directly through Team Explorer.  The answer is fairly simple – using reports we can precisely decide how to present the information.  Proper use of charts, graphics and color can help us identify trends and bring out important characteristics of our data. 

Reporting in TFS 2010

TFS 2010 is based on the enterprise-grade SQL Server 2008 database.  As such, it also makes use of SQL Server Reporting Services (SSRS), the component that enables us to combine a visual layout with the relevant data to produce a useful report.  This means that we can produce useful visualizations of the hugely-useful amount of data that we put into TFS.  In fact, you can access SSRS reports directly from the Team Explorer window using the Reports node:

What’s Next?

In the next post, we’ll start the actual work.  The first thing to do is to design the actual query:  we’ll see how we can pull the data out of TFS based on user input.  Stay tuned!

 

Writing My First Metro-Style App

Having attended BUILD in Anaheim, CA last month, I was looking forward to trying out the various technologies that were introduced:  Windows 8, WinRT + XAML and of course, running it all on the Samsung tablet.  This post is a summary of my experiences.  In general, it’s been good – the tools are appropriate, the APIs pretty much cover what I had in mind and the end result is similar (though not identical) to what I had in mind.

Development Environment

The first issue I encountered was setting up a comfortable development environment.  While the Samsung tablet is a fully capable PC (Intel Core i5, 4GB memory, 64 GB HDD), I had a hard time working on it directly.  I connected my own keyboard and mouse (instead of the Bluetooth keyboard that came with the tablet), but the screen was simply too small when working with Visual Studio and especially the XAML editor. 

I then tried accessing the tablet via Remote Desktop.  While this worked and was usable, it’s not an option I would recommend.  The main problem is one that was mentioned in one of the Big Picture sessions at BUILD – Metro-style apps are full-screen, so when only a single monitor is available, it’s either the IDE or the application.  The same session also discussed the fact that you can debug in an ‘application window’, which is a small window that does not take up the entire screen.  Unfortunately, I couldn’t get this functionality to work – whenever I would click a text box, the application would jump to full-screen.  The conclusion – separate screens for the IDE and the application.

At this point, it was pretty clear that I could use the tablet as my application screen – especially since that’s the only way that I can debug with touch capabilities.  So I would write in one instance of VS11 (on a separate Win8 machine) and deploy to the tablet using VS’s remote debugging feature.  I tried running the Win8 Developer Preview in VirtualBox on my laptop, but was not happy with the performance, especially where the display was concerned (nowhere near the ‘fast and fluid’ that was expected).   Finally, I installed the image on a desktop machine (6 GB memory, quad-core i7) where it dual-boots with an existing copy of Windows 2008.  This worked fine, and supplied a very pleasant development experience.

Concept

I wanted to write something useful and that would take advantage of the tablet and the touch interface.  Finally, I opted to use the API exposed by Wordnik, a great on-line dictionary and a site I’m very fond of.  Using Balsamiq, I came up with this initial design for a metro-style UI:

Things that I tried paying attention to (and taking into account that I’m a, shall we say, *very mediocre* designer)

• Aligning everything to a grid
• Making use of font sizes for effective information display
• Showing images along with definitions (probably using the Flickr API)

 

Writing the Application

When it came down to actually writing the application, I decided to pass on the existing templates in VS, and write everything from scratch.  Since this is basically just an exercise, I wanted to do everything myself and get a better feel for the tools and the platform.  In order to do so, I used the code and .xaml files from the Metro-style app samples and basically just copied stuff across (testing and experimenting as I go).

Some interesting points came up during the development:

• Rather than using the System.Net.WebClient class which is available in .NET 4, I needed to use System.Net.Http.HttpClient instead.
• Integrating Windows 8’s Search contract was as easy as handling the OnSearchActivated event in the Windows.UI.Xaml.Application class.
• I’m not familiar with Expression Blend at all, so I ended up writing the XAML manually.  Therefore, I missed out on all the new Blend goodness that was integrated to VS11.  And the XAML I DID write is probably quite bad.

Things I did not do and would like to experiment with the in future:

• Use the async APIs – right now, all the web access is done via synchronous APIs.  In order to really achieve ‘fast and fluid’, it’s necessary to use the async methods of HttpClient.
• Create a more immersive tile – perhaps displaying a random word with a single definition.
• Learn Expression Blend and write a better UI!

Some screenshots of the deployed app:

 

Conclusion

All in all, it was a fun experience.  The entire application took around 4 hours to write, from the minute I created the project until I had something that I was reasonably happy with.  During that time, I did not experience a single crash of VS, although it did freeze once for about a minute.  I used my existing development knowledge, and apart from the new APIs there wasn’t much that I needed to learn.  I personally am quite fond of the Metro design style (and have been since it appeared in WP7), so I’m looking forward to further work on this kind of applications in the future.

Impressions of the First Day of BUILD 2011

Just came back from the first day of BUILD.  I have to say – wow.  Just one big WOW.  This day has absolutely blown my mind with all the new technologies and philosophies.

Rather than describing the actual details of the various sessions – and in particular the keynote (see one such description here) – I’d like to give my high-level impression of what I saw.  First and foremost – Windows 8 is simply better.  Less memory, less processes that run in the background and less overhead.  That means you get more space for your apps on one hand, and a better user experience on smaller and less powerful devices on the other hand.  And speaking of devices – Win8 runs on everything.  And I mean everything.  The keynote showed an incredible variety of hardware that supports Windows 8, from handheld tablets all the way to fantastic water-cooled machines that have a trio of high-end NVidia graphics cards doing DirectCompute workloads.  And everything that ran on Windows 7 will continue to run.

But what was really interesting is what you can now do.  You can write applications using whatever technology you want.  Managed code (C# or VB.Net), unmanaged code (C++) or JavaScript – take your pick.  You can develop in any of those thanks to the new WinRT API and the native support for XAML.  I personally see this as a huge step forward, as we no longer need to differentiate between a ‘web’ and a ‘desktop’ developer.  Those are just two sides of the same coin – the gap no longer exists.

And speaking of bridging the gaps – I was completely blown away by the concept of contracts.  This has got to be one of the most useful metaphors I have ever seen.  The ability to seamlessly communicate between disparate applications in such a useful, and most importantly, intuitive manner means that the experience of using a home computer is now at a totally different level.  I honestly believe that we have reached a stage where a layperson can be just as productive with their PC as a computing professional (I am, of course, not referring to tasks such as software development). 

The Windows 8 UI is absolutely stunning.  Microsoft has really done an incredible job with Metro.  I am very fond of the Windows Phone 7 UI, so this looks like the natural step forward.  The UI is sleek, professional and responsive.  I’m really looking forward to seeing what people will be doing with these UI capabilities.

And of course – I now have a super-cool Samsung tablet to play with…

Some pictures from the conference and the Sela guys:

 

Tomorrow is the second day – I’m especially looking forward to the Visual Studio and ALM sessions.

It is time for BUILD!

It is currently 6:45 in the morning in Anaheim, California.  The Sela delegation (19 experts!) is all here and we are all anxiously waiting for the conference to begin.  We had a great steak dinner last night (thanks Sasha!) to kick things off on the right foot.

The conference center is huge – which is probably a sign of things to come.  In about an hour we’ll start heading out, hopefully to get some breakfast and get some good seats.  Currently, we have very little knowledge of what the agenda is -  but it’s definitely going to be great.

Stay tuned!

Hi Ho, Hi Ho, it’s off to BUILD we go!

Well, the time is finally here – tomorrow night I’m going to BUILD 2011 in Anaheim, CA.  Looks like it’s going to be a blast!  Apart from the fact there will be a whole bunch of us from Sela (orange shirts galore!), I’m looking forward to seeing some great new technology.  If you’re coming, do drop me a line so we arrange to meet in person.

 

See you all there, and if you’re not coming – see you when we get back.

Introduction to TFS 2010 Training Kit is Released!

This morning we received some good news:  Microsoft has released the TFS 2010 training kit that Assaf and I co-authored.  Happy days!

You’ve probably heard of TFS and in particular of TFS 2010, the latest released version.  It’s a powerful tool packed with features – but where do you begin?  The Training Kit is designed to help you understand what TFS 2010 can do for your organization and software process.  It applies to all members of the team – developers, testers, business people and of course, managers.

So if you thought TFS 2010 was just the next version of Visual Source Safe – this kit is for you!   

Download the Introduction to TFS 2010 Training Kit here.