September 2007 - Posts
Let me start with an out-loud recap of this post: Agile is not something you can put on a bread nor is it "a certain path to success".
It's about STATE OF MIND.
If I had to describe the meaning of "Agile" to a new teammate I would say: Agile is a constant thinking about how we, AS A TEAM, can produce working features to our users with high quality within a short time-frame.
Don't worry:
It's really OK to provide only a subset of feature(s) in one sprint.
It's really OK to leave SOME designing\architecture issues for later on as long as the high-level architecture is good enough (=you're comfortable with it) to answer the big questions.
It's really OK to implement only two REALLY-DONE-HIGH-QUALITY features in one sprint over four semi-working-not-demoable features.
The key here though is not really the practices, it's about the big bullets(again, state of mind):
1). Produce value for your customers and adjust\adopt early.
2). Build a Team (self leadership).
These ideas are hard to implement and require special kind of people. Putting the Team in front of yourself is not a very job-secure attitude. The ability to help your teammates, shift tasks, taking ownership, critisize yourself and your teammates and getting better, produce high quality design, tests and code - all of it - requires versatile people with unique state of mind (and unique abilities, of course). It's worth it. When things glue, it's a real magic; Things start to get going by the Team, improvements and features starting to come from the developers\QA\Graphic Designer, adjustments are made on a regular basis, changes are welcome and productivity is celebrated.
You can feel something is going right.
That's Agile.
The way WCF proxies are designed is to live until shi* happens.
Let's assume that we have a CalcualtorService with one method named Divide(int a, int b). Sasha, a cool programmer-dude, trying to produce some usefull software writes:
public MyCalcualtorForm : Form {
private CalculatorProxy _calc = new CalculatorProxy();
Calc_Click(...) {
_calc.Divide(firstNumber, secondNumber);
}
}
What is the first error you can think of that could happen? Yep, DivideByZeroException.
Once the proxy gets an exception, it enters into a "Faulted" state which makes the proxy unusable(=you cannot use it again).
The quickest solution is to work "by the book" and create a new instance each and every time we need to call the service:
Calc_Click(...) {
using (CalculatorProxy calc = new CalculatorProxy())
calc.Divide(firstNumber, secondNumber);
}
But what's bad in this solution?
- Performance - you pay (not a lot but neither little) for each creation of the proxy. Sure, it will probably not be your bottleneck, but heck, why is it useful? Most of the time the proxy will not throw an exception and yet we need to create it every time just to avoid the faulted state scenario.
- Design - If we declare this exception BY CONTRACT, I would expect that the proxy will still be usable afterwards. Do we really want to return Enum\int\string as status instead of throwing exception just because of poor design?
- TDD - you know that I'm in love with it. Now imagine Dependency Injection. Component A recieve ICalculatorProxy and use it to... calculate something. Working "by the book" is no good as we want to recieve an instance of the proxy from the outside in order to mock it. Right, so we inject a proxy from the outside (got to love Windsor) and life is pretty sweeet. Darn! Wait! one poor (even by design) exception and our proxy goes dead. Very un-TDDish of Microsoft.
I had to come with a solution as no one will take TDD away from me. I present to you ProtectedProxy: this little IL-code-at-the-end-of-the-day will able you to recover from faulted state by creating a new proxy on each exception thus making your proxy... useable (couldn't think about a better word to describe it). Think about a situation where your proxy is trying to call the service but the service is down; In Semingo, we decided that we want to keep trying until the service is up. Via ProtectedProxy, you can determine how many times do you want to recover and when you should finally kill the proxy. Oh yea, ProtectedProxy uses Windsor in order to create new proxies if needed and logging messages to log4net. Good stuff.
In the example above, all Sasha had to do was to:
1). Initialize the _calc field by:
ProtectedProxy<ICalculatorProxy> _calc = new ProtectedProxy<ICalculatorProxy>(new CalculatorProxy());
2). call _calc via:
_calc.Instance.Divide(firstNumber, secondNumber);
But enough said, code please:
// Written by Oren Ellenbogen (07.08.07) - trying to protect our proxies so they could recover from:
// (A) The service is not up yet, but we want to try again later.
// (B) The service throws (ANY) exception, we still want our proxy to function (bubble the exception, but still keep on working).
// Microsoft intended to use a NEW proxy per call, but for TDD this is not ideal as we would like to inject proxies from outside as mocks
// in order to simulate multiple scenarios.
#region using
using System;
using System.Reflection;
using System.ServiceModel;
using Castle.Core.Resource;
using Castle.Windsor;
using Castle.Windsor.Configuration.Interpreters;
using log4net;
#endregion
namespace Semingo.Services.Proxies.Helpers
{
public interface IProxy : ICommunicationObject {
bool Ping();
}
/// <summary>
/// Protect proxy from entering Faulted state by re-creating the proxy via Windsor Container on Faulted.
/// IMPORTANT: that even if a fatal exception is raised by the service (for example: the service is not up yet), the proxy will be raised again.
/// Use it wisely (TIP: you CAN determine the number of 'recovery' attempts).
/// </summary>
/// <typeparam name="I">The proxy interface to protect</typeparam>
/// <remarks>
/// The way WCF works is that ANY exception on the service will cause the proxy to enter "faulted" state which means you can not use it anymore.
/// Imagine a service of CalculatorService that expose the method float Divide(int a, int b). Sending b=0 will raise an exception in the service
/// and the proxy will get into faulted state. This is not ideal as the proxy itself should be used again.
/// </remarks>
public class ProtectedProxy<I> : IDisposable
where I : IProxy
{
private static readonly ILog _logger = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
private I _instance;
private readonly IWindsorContainer _container;
private int _faultedCounter = 0;
private bool _disposed = false;
private const int AlertableNumberOfFaultedTimes = 10;
public ProtectedProxy(I instance)
: this(CreateXmlBasedWindsorContainer(), instance)
{
}
public ProtectedProxy(IWindsorContainer container, I instance)
{
_container = container;
ShieldInstance(instance);
}
/// <summary>
/// Returns the number of faults this proxy had so far
/// </summary>
public int NumberOfFaults
{
get { return _faultedCounter; }
}
public I Instance
{
get
{
ThrowIfInstanceAlreadyDisposed();
if (_instance.State == CommunicationState.Faulted || _instance.State == CommunicationState.Closed || _instance.State == CommunicationState.Closing)
{
_logger.Warn("Notice: The proxy state is invalid (" + communicationObj.State + "). The Faulted event should have been raised and handle this state - this need to be checked.");
HandleFaultedInstance();
}
return _instance;
}
}
public void Close()
{
Dispose();
}
private void ThrowIfInstanceAlreadyDisposed()
{
if (_disposed)
throw new ObjectDisposedException("The protected proxy for the type: " + _instance.GetType().FullName + " was closed. Cannot return a live instance of this type.");
}
private void ShieldInstance(I instance)
{
_instance = instance;
_instance.Faulted += delegate { HandleFaultedInstance(); };
}
private void HandleFaultedInstance()
{
ThrowIfInstanceAlreadyDisposed();
_faultedCounter++;
if (_faultedCounter >= AlertableNumberOfFaultedTimes)
_logger.Warn("ALERT! The proxy for the type " + _instance.GetType().FullName + " got faulted for the " + _faultedCounter + " time. Recreating the proxy but we must verify if this is valid.");
else if (_logger.IsDebugEnabled)
_logger.Debug("Proxy for type " + _instance.GetType().FullName + " got faulted (current state: " + ((ICommunicationObject)_instance).State + ") - recreating the proxy. Number of faulted instances so far: " + _faultedCounter + ".");
ProxyHelper.CloseProxy(_instance); // close current proxy
ShieldInstance(_container.Resolve<I>()); // re-create the proxy, faulted proxies are no good for further use.
}
private static IWindsorContainer CreateXmlBasedWindsorContainer()
{
try
{
return new WindsorContainer(new XmlInterpreter(new ConfigResource("castle")));
}
catch (Exception err)
{
_logger.Warn("Unable to create xml based windsor container, using empty one.", err);
return new WindsorContainer(); // for testing (the proxy will be mocked anyway).
}
}
#region IDisposable Members
///<summary>
///Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
///</summary>
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
_logger.Info("Attempting to dispose the protected proxy for the type: " + _instance.GetType().FullName + ", disposed already? " + _disposed);
if (_disposed) return;
if (disposing)
{
ProxyHelper.CloseProxy(_instance);
}
_disposed = true;
}
#endregion
}
public static class ProxyHelper
{
private static readonly ILog _logger = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
public static void CloseProxy(object proxy)
{
if (proxy == null) return;
CloseProxy(proxy as ICommunicationObject);
}
/// <summary>
/// Close the proxy in a safe manner (will not throw exception)
/// </summary>
/// <param name="proxy">The proxy to close</param>
public static void CloseProxy(ICommunicationObject proxy)
{
if (proxy == null) return;
try
{
if (proxy.State == CommunicationState.Closing || proxy.State == CommunicationState.Closed || proxy.State == CommunicationState.Faulted)
proxy.Abort();
else
proxy.Close();
}
catch (CommunicationException)
{
proxy.Abort();
}
catch (TimeoutException)
{
proxy.Abort();
}
catch (Exception err)
{
_logger.Error(err);
proxy.Abort();
}
finally
{
proxy = null;
}
}
}
Hours of joy...
Almost forgot, on the next post - "How to TDD WCF code" - stay tuned...
One of the most common question in moving towards Agile Development is "Where should I start from?". If you'll ask me, setup a Continuous Integration (aka "CC") would be the first thing you should start with.
Step 1 (Check for compilation bugs): Code Quality ~= 30%
The CC should be able to identify check-ins to your source control, get the latest source and compile it. The output should be either "green" (Everything compiles successfully with 0 warnings) or "red" (more than one warning or compilation errors). In addition to the fast feedback, the output should also include the files that were changed from the last build (and by whom, so people could know where to look).
The immediate value is priceless. the ability to SEE whether your source-code is stable enough to allow other programmers perform Get Latest and continue their work and the "fail-fast" attitude can save you a lot of time in the long run. It's important to realize though that even if the code compiles without warnings, it still doesn't mean that you could count on the quality of the code.
Step 2 (Check for component-based quality): Code Quality ~= 70%
If you can go one extra mile, write a few automated tests (via one of the available XUnit frameworks) for your components. This means that you are able to inject the component's dependencies from the outside and simulate mini use-cases on component's level. This step is crucial even if you write those after the code itself was written. Let the CC run them if Step 1 is OK. This should allow you to catch the majority of your bugs (I'll leave the "how to write good tests" to another post). If all is green, you know that the system behaves as expected, at least to some extent.
This step is not trivial as it requires you to design for testability and invest in proper testing. Don't let go of this step though, automated tests on this level will make your life much easier. It will take your code one (major) step ahead in "write code that could be changed". Agile is all about that state of mind.
Step 3 (Check for integration-based quality): Code Quality ~= 80%
Now that you're components are behaving as expected, you should try to write a few (automated, of course) tests that simulate the entire flow of 2 or more components (DB is a component as well) in the system. As the system grows and more uses cases are added, you should try to improve these tests as they give a solid proof that the SYSTEM works.
Step 4 (Make everything visible): Code Quality ~= 90%
The state & quality of your source control should be visible to the Team and Management as you want to insure IMMEDIATE response time in case someone check-ins a low quality code (on any level). Fixing a failing test three days after the change itself is a bad symptom of low visibility or low perception, by the Team, regarding the importance of the quality of the system.
Step 5 (Automated deployment): Code Quality ~= 95%
After successful build you would like to deploy the latest source on a dedicated environment which the developers could play with before deploying to another (testing?) environment. This won't be a stable environment, but at least it will give a quick look at the current state of the system - the way customers would see it.
Step 6 (Procedures checking): Code Quality > 95%:
You can add many more checks to the flow, such as Tests Coverage or FxCop. Leave those to the end. From my experience, Time .vs. Value in these features will vary from team to team. You'll gain much more from investing in Steps 2-3.
Semingo CC:
Each developer & manager on our team have a CCTray(the little red circle in the little picture above) which is either Red(source control is damaged), Yellow(Build in action) or Green(Life is sweet). We're using Cruise Control.Net, CCTray, MSBuild (and TFS plugin) and NUnit to perform all of the above.
A few teasers:
* Image of Steve Urkel (from the famous Family Matters TV series) is shown for failing build.
* 
On the right you can find Pasha (with a V sign I've added), one of our finest hackers modeling a successful build. You can also notice the 582 green tests (including Integration tests) and 2 changes made by Sagie since the last build.
* Going to NUnit Details, you can get the full details:
I had to cut the pictures in order to keep a sane width for the post, but you can get the drift.
btw - Aviel, yet another Semingo hacker add a "Doh!" (Simpson) each time the CC is red on his computer. It can be quite funny (and scary, if fully concentrated on code).
