Offir Shvartz

AppFabric Caching Service Using SSL

Offir Shvartz — Fri, 01 Feb 2013 12:50:00 GMT

I really like AppFabric distributed cache (previously called Velocity). It’s so simple to use and it has really strong features like notification based local cache. It basically has two versions, one is targeting for on premise and one is for the cloud (Azure). The API is similar which makes it very easy to move your application in later phases to the cloud. But back to my customer which security was a strong requirement because of the nature of his system which I won’t mention here but I’m sure you get the idea. His security requirements are that all connection between machines will be sign and encrypted using a given certificate.

App Fabric Caching Security mode

When looking for information regarding AppFabric security model you find only few pages (like this) that says that AppFabirc Cache is secured in transport level by default and you can choose whether you want to enable encryption and signing (they are enabled by default) but I also found bunch of Stackoverflow question that wanted to understand more regarding how the security is implemented like this and this.

To conclude both of the posts they say that app fabric transport relay on WCF – the binding protocol is TCP and the security is similar to WindowsStreamSecurityBinding which means the encryption and signing using the windows identity token. unfortunately This security model is not sufficient for my client so…

Try #1– Microsoft Support

It sounded to me like a reasonable request to extend/control the security model and it should be simple considering the fact that as far as I understand the communication relay on WCF. so we tried Microsoft support and this is what Microsoft answered:

“Following our discussion last week, I contacted our developers to confirm if AppFabric caching has a feature to extend or there’s any provider which allows to change binding for communication between cache client and caching service i.e. from NetTcpBinding to BasicHttpBinding or WsHttpBinding to use SSL. As discussed earlier, there’s NO feature or provider which allows this. So, it leaves only two security options to be set on the cache cluster: None and Transport (default being Transport with Encrypt and Sign).”

dead end

Try #2 – let’s look inside

As a last resort I decides looking inside the code. It’s a longshot but still AppFabic is written in .NET meaning you can use decompiler tool to look at the code. I use DotPeek which is nice, gives great navigation functionality and most important free.This is what I learned:

DistributedCachingService code has some implementations parts specific for azure meaning that it could be that the same code is used in Azure caching which is very nice in my opinion allowing system that use the cache to move easily to the cloud and change only the configuration.
DataCacheSection– when browsing them I discovered the configuration section has exactly what I needed – SSL

The Element AdvanceProperty contains ServerSecurityProperty this is how it looks:

// Type: Microsoft.ApplicationServer.Caching.ServerSecurityProperties
// Assembly: Microsoft.ApplicationServer.Caching.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
// Assembly location: C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\2012-10\ref\Caching\Microsoft.ApplicationServer.Caching.Core.dll

using System;
using System.Configuration;
using System.Runtime.Serialization;

namespace Microsoft.ApplicationServer.Caching
{
  [Serializable]
  internal class ServerSecurityProperties : ConfigurationElement, ISerializable
  {
    internal const string MODE = "mode";
    internal const string PROTECTION_LEVEL = "protectionLevel";
    internal const string AUTHORIZATION = "authorization";
    internal const string ALLOW = "allow";
    internal const string SERVER_ACS_PROPERTIES = "serverAcs";
    internal const string USE_ACS_FOR_CLIENT = "useAcsForClient";
    internal const string SHARED_KEY_AUTH = "sharedKeyAuth";
    internal const string SSL_ENABLED = "sslEnabled";
    internal const string SSL_PROPERTIES = "sslProperties";

    [ConfigurationProperty("mode", DefaultValue = DataCacheSecurityMode.Transport)]
    public DataCacheSecurityMode DataCacheSecurityMode
    {
      get
      {
        return (DataCacheSecurityMode) this["mode"];
      }
      set
      {
        this["mode"] = (object) value;
      }
    }

    [ConfigurationProperty("protectionLevel", DefaultValue = DataCacheProtectionLevel.EncryptAndSign)]
    public DataCacheProtectionLevel DataCacheProtectionLevel
    {
      get
      {
        return (DataCacheProtectionLevel) this["protectionLevel"];
      }
      set
      {
        this["protectionLevel"] = (object) value;
      }
    }

    [ConfigurationCollection(typeof (AuthorizationElement), AddItemName = "allow")]
    [ConfigurationProperty("authorization", IsDefaultCollection = false, IsRequired = false)]
    public AuthorizationElement Authorization
    {
      get
      {
        return (AuthorizationElement) this["authorization"];
      }
      set
      {
        this["authorization"] = (object) value;
      }
    }

    [ConfigurationProperty("serverAcs", IsDefaultCollection = false, IsRequired = false)]
    public ServerAcsSecurityElement AcsSecurity
    {
      get
      {
        return (ServerAcsSecurityElement) this["serverAcs"];
      }
      set
      {
        this["serverAcs"] = (object) value;
      }
    }

    [ConfigurationProperty("useAcsForClient", DefaultValue = false)]
    public bool UseAcsForClient
    {
      get
      {
        return (bool) this["useAcsForClient"];
      }
      set
      {
        this["useAcsForClient"] = (object) (bool) (value ? 1 : 0);
      }
    }

    [ConfigurationProperty("sharedKeyAuth", IsRequired = false)]
    public SharedKeyAuthorization SharedKeyAuth
    {
      get
      {
        return (SharedKeyAuthorization) this["sharedKeyAuth"];
      }
      set
      {
        this["sharedKeyAuth"] = (object) value;
      }
    }

    [ConfigurationProperty("sslEnabled", DefaultValue = false, IsRequired = false)]
    public bool SslEnabled
    {
      get
      {
        return (bool) this["sslEnabled"];
      }
      set
      {
        this["sslEnabled"] = (object) (bool) (value ? 1 : 0);
      }
    }

    [ConfigurationProperty("sslProperties", IsRequired = false)]
    public SslProperties SslProperties
    {
      get
      {
        return (SslProperties) this["sslProperties"];
      }
      set
      {
        this["sslProperties"] = (object) value;
      }
    }

    internal ServerSecurityProperties()
    {
    }

    protected ServerSecurityProperties(SerializationInfo info, StreamingContext context)
    {
      this.DataCacheSecurityMode = (DataCacheSecurityMode) info.GetValue("mode", typeof (DataCacheSecurityMode));
      this.DataCacheProtectionLevel = (DataCacheProtectionLevel) info.GetValue("protectionLevel", typeof (DataCacheProtectionLevel));
      this.Authorization = (AuthorizationElement) info.GetValue("authorization", typeof (AuthorizationElement));
      try
      {
        this.AcsSecurity = (ServerAcsSecurityElement) info.GetValue("serverAcs", typeof (ServerAcsSecurityElement));
        this.UseAcsForClient = (bool) info.GetValue("useAcsForClient", typeof (bool));
        this.SharedKeyAuth = (SharedKeyAuthorization) info.GetValue("sharedKeyAuth", typeof (SharedKeyAuthorization));
      }
      catch (SerializationException ex)
      {
        this.UseAcsForClient = false;
        this.SharedKeyAuth = new SharedKeyAuthorization();
      }
      try
      {
        this.SslProperties = (SslProperties) info.GetValue("sslProperties", typeof (SslProperties));
        this.SslEnabled = info.GetBoolean("sslEnabled");
      }
      catch (SerializationException ex)
      {
        this.SslEnabled = false;
      }
    }

    internal DataCacheSecurity GetDataCacheSecurity()
    {
      return new DataCacheSecurity(this);
    }

    public void GetObjectData(SerializationInfo info, StreamingContext context)
    {
      info.AddValue("mode", (object) this.DataCacheSecurityMode);
      info.AddValue("protectionLevel", (object) this.DataCacheProtectionLevel);
      info.AddValue("authorization", (object) this.Authorization);
      info.AddValue("serverAcs", (object) this.AcsSecurity);
      info.AddValue("useAcsForClient", this.UseAcsForClient);
      info.AddValue("sharedKeyAuth", (object) this.SharedKeyAuth);
      info.AddValue("sslEnabled", this.SslEnabled);
      if (!this.SslEnabled)
        return;
      info.AddValue("sslProperties", (object) this.SslProperties);
    }
  }
}

As you can see it has SSL enabled property and SSLProperties which include "certificateSubject”. So I tried setting this values with a legal certificate and the service worked . But I celebrated too early because when drilling down to the usage of this property I discovered two things:

The communication channel relays on WCF: the type WcfTransportChannel with two derives WcfClientChannel and WcfServerChannel – I learned exactly the binding that is used:

protected void SetupBindings()
   {
     BinaryMessageEncodingBindingElement encodingBindingElement = new BinaryMessageEncodingBindingElement();
     XmlDictionaryReaderQuotas.Max.CopyTo(encodingBindingElement.ReaderQuotas);
     TcpTransportBindingElement transportBindingElement = this.PrepareTcpTransportBindingElement();
     BindingElementCollection bindingCollection = new BindingElementCollection();
     bindingCollection.Add((BindingElement) encodingBindingElement);
     this.AddSecurityBinding(bindingCollection);
     this.AddCustomBindingElements(bindingCollection);
     bindingCollection.Add((BindingElement) transportBindingElement);
     this._tcpBinding = new CustomBinding((IEnumerable<BindingElement>) bindingCollection);
     this._pipeBinding = new NetNamedPipeBinding();
   }

In security aspect certificate is ignored when using transport mode meaning the fact you can put it in configuration (SSLEnabled=”true”) file doesn’t mean it will really use SSL.

Here is the code from the server side:

protected override void AddSecurityBinding(BindingElementCollection bindingCollection)
    {
      if (this.DataCacheSecurity.SecurityMode == DataCacheSecurityMode.Transport)
      {
        this._securityBindingElement = new VelocityStreamSecurityBindingElement(this._serviceConfigurationManager, true);
        bindingCollection.Add((BindingElement) this._securityBindingElement);
      }
      else
      {
        if (!this._sslEnabled)
          return;
        bindingCollection.Add((BindingElement) new SslStreamSecurityBindingElement()
        {
          RequireClientCertificate = false
        });
        string sslCertIdentity = this._serviceConfigurationManager.AdvancedProperties.SecurityProperties.SslProperties.SslCertIdentity;
        int certificateCount = WcfServerChannel.GetCertificateCount(sslCertIdentity);
        if (certificateCount == 0 && Provider.IsEnabled(TraceLevel.Error))
          EventLogWriter.WriteError(this._logSource, "Certificate with name {0} doesn't exist", new object[1]
          {
            (object) sslCertIdentity
          });
        if (certificateCount > 1 && Provider.IsEnabled(TraceLevel.Error))
          EventLogWriter.WriteError(this._logSource, "More than one certificate with the name {0} exist.", new object[1]
          {
            (object) sslCertIdentity
          });
        ServiceCredentials serviceCredentials = new ServiceCredentials();
        serviceCredentials.ServiceCertificate.SetCertificate(StoreLocation.LocalMachine, StoreName.My, X509FindType.FindBySubjectDistinguishedName, (object) sslCertIdentity);
        this._listenerParameters.Add((object) serviceCredentials);
      }
    }

The cannel type the client uses is IDuplexSessionChannel – from WcfClientChannel ctor:

this._tcpFactory = this._tcpBinding.BuildChannelFactory<IDuplexSessionChannel>(new object[0]);

So I learn a lot but unfortunately I didn’t find a way to override the current binding and security methods.

The Solution

I was really frustrated at this point but then I remembered one of the new features of WCF 4 was Routing Service which one of his target scenarios are: Protocol bridging (“You receive messages over one transport protocol, and the destination endpoint uses a different protocol” – MSDN). the only thing you should know is exactly what is your binding to do the routing correctly and here is the full solution:

Host Routing Service in the client side that listen to App Fabric request meaning:
- Implement the contract IDuplexSessionRouter
- Has custom binding with: TcpTransport,BinaryMessageEncounding and windowsStreamSecurity
- Route all communication using “MatchAll” Filter to the target Routing Service at the App Fabric Server with the required binding/security (Transport security using certificate)
Host Routing Service In the AppFabric cache Server machine
- Implement the contract IDuplexSessionRouter
- Has matching binding to the target service of the other router
- Route all communication using “MatchAll” Filter to the App Fabric Server using custom bindingbinding with: TcpTransport,BinaryMessageEncounding and windowsStreamSecurity

Well it works the client is agnostic to the fact that the messages are going through the routing services – the only down side is the double hopes the messages does but it doesn’t open the soap message (doesn’t Serialize /Deserialize) so I guess this solution it’s not bad.

That’s all for now

Cheers Offir

Azure Table Storage – limitations and solutions/workarounds(Part 2)

Offir Shvartz — Thu, 20 Sep 2012 03:18:00 GMT

This is part two of 2 parts posts discussing the limitations in Azure table storage. It’s highly recommended to read first Part 1. In this part I will discuss the querying limitations. Using table storage as your persistence layer – you need to be able to query the storage efficiently. The .NET layer above Table Storage (CloudTableClient) exposes the CreateQuery method that returns DataServiceQuery that implements IQueryable<T>

public DataServiceQuery<T> CreateQuery<T>(string entitySetName);

meaning you can write whatever LINQ query you want and it will compile – but only a very small subset is supported as described here – Query Operators (Table Service Support). The only supported query operators (it’s easier than writing what is not supported) are: From,Where,Take,First,FirstOrDefault and select.

Limitation #1 – query returns a large number of rows (more than 1000)

Querying table storage using the API above will return a maximum of 1000 entities – this is a good practice and in this case the table storage RESTful service will return a response that contains a continues token to retrieve the next data – to be precise if the query takes more than 5 second or we cross to other partition the query will return. But what if we need to get more then 1000 entities? In this case we have two options:

Using Paging

As I mentioned before if not all data returns a continues token is returned as part of the response – query using this token will return the next result set part of the complete result – this can be implemented using an object called CloudTableQuery<T> – read the details in this excellent post by Jim O`Neil

Query As TableServiceQuery

The simpler case is when we don’t really need paging we just need all the results (and we know that it won’t be a lot more than 1000) – the same CloudTableQuery<T> implements IQueryable<T> but query using it will do the paging under the hood returning all the data needed – more then 1000 rows. simply instead of writing:

context.Blogs.ToList();

Modify it to:

context.Blogs.AsTableServiceQuery().ToList();

and it will query the service as much as needed (using the continues token) – but beware this can be very slow if you have a lot of data so use it wisely.

Limitation #2 – order by

As I mentioned in the beginning of the post most of the LINQ operators are not supported – like order by. So what can we do ? Well if we want the data to retrieve by one specific order – for all queries we can use the fact that the data is stored and retrieved in lexicographic order meaning query the data in a partition will return by their lexicographic order in the partition and the partitions will be returned by the partition key lexicographic order. Let’s look at real life example (that can come in handy) – we want our entities to be ordered from the most recent creation date to the least recent. here is how it can be done:

public sealed class StoredEntity : TableServiceEntity
{      
     public StoredEntity (DateTime creationDate)
     {            
        RowKey = string.Format("{0:D19}",DateTime.MaxValue.Ticks -
                 creationDate.ToUniversalTime().Ticks);
        PartitionKey = string.Format("{0:D19}",DateTime.MaxValue.Ticks -
                 creationDate.Date.ToUniversalTime().Ticks);     
     } 
}

Explanation of the above: format D19 meaning decimal number with 19 digits (turns 1 to 0000…..0000000001) which is in inverse proportion to the max date.We set the partition key to be a 19 digits string by the date (without time) and the row key inside is 19 digits string by the data include the time. we divided the data to partition keys by dates and inside the partition the data is ordered by the full date and time.

Limitation #3 – query like operator

String operators can be very useful especially when we need to implement a free text search functionality on our data. In this case we don’t have a real solution beside using additional storage that has free text capabilities like lucene and query it. The only string operator that is supported is CompareTo. So we cannot turn compartTo to like but we can implement “starts with” functionality. Lets say we want to ask if the property “Col” starts with the string “yes” we find the following lexicographic string which is “yet” (T comes after S) and query if:

c.Col.CompareTo("yes") >= 0 && c.Col.CompareTo("yet") < 0

and you have <==>

c.Col.StartsWith("yes")

It’s not like but just maybe it can match your needs.

Summery

Azure table storage is not bad (it’s actually amazing ) giving unlimited scalable storage of strong typed entities. I’m just saying that using Azure table storage as your only persistence layer can be problematic if your requirements cannot be answered because of the limitations above and in the previous post. So check carefully your needs not only for the first phase but for the full implementation and future planing because changing from one storage infrastructure to the other can be very painful – I don’t believe that abstraction will work in this case because sooner or later you start using specific table storage features and change it may be too expensive.

Cheers,Offir

Azure Table Storage – limitations and solutions/workarounds(Part 1)

Offir Shvartz — Tue, 14 Aug 2012 03:12:12 GMT

Azure table storage – is the NoSQL storage key/value based part of Microsoft Azure cloud services. If you are not familiar with Azure table storage I recommend you to first read here. Recently I was working on a very simple application that is hosted on the cloud (simple web role). My previous experience with NoSQL was mostly using RavenDB (which I really love…) but in this application it was the natural decision to use the out of the box cloud NoSQL storage. Getting started was really easy thanks to the great documentation we have online but when I started to implement the business logic I encountered the limitations and searched for solutions/workarounds – in the next post I want to share with you the limitations I encountered and how to solve them.

Limitation type #1 – Storing your domain object types

My application low level architecture is domain driven meaning I have classes/entites that represent my domain – but I cannot store any of them in the table storage due to the following limitations:

Derive from TableServiceEntity

Stored entity must derive from TableServiceEntity or to be more specific you cannot store any object in azure table storage unless it has the properties: RowKey, PartitionKey and TimeStamp.

Disallow charters in the keys property

let’s say you want to store a URL in the PratitionKey – well you can’t because these are the chars you cannot use:

The forward slash (/) character
The backslash (\) character
The number sign (#) character
The question mark (?) character

Supported property types

Let’s say you want to store property of complex type (like Address) well you can’t because the supported types are limited to the following list:

WCF Data Services type	Common Language Runtime type	Details
Edm.Binary	byte[]	An array of bytes up to 64 KB in size.
Edm.Boolean	bool	A Boolean value.
Edm.DateTime	DateTime	A 64-bit value expressed as Coordinated Universal Time (UTC). The supported DateTime range begins from 12:00 midnight, January 1, 1601 A.D. (C.E.), UTC. The range ends at December 31, 9999.
Edm.Double	double	A 64-bit floating point value.
Edm.Guid	Guid	A 128-bit globally unique identifier.
Edm.Int32	Int32 or int	A 32-bit integer.
Edm.Int64	Int64 or long	A 64-bit integer.
Edm.String	String	A UTF-16-encoded value. String values may be up to 64 KB in size.

Sizes limitations

Stored entities can have up to 255 properties – and combined size <= 1M and RowKey size must be <= 1KB

Solutions

So at this point you may ask what’s new, we could read about it in MSDN (and you’re right), but now I want to discuss the possible solutions/workaround and their pros and cons.

Divide our objects to domain object and stored objects

This solution comes from a lot of other persistence issues in other solutions (like OR\M) and kind of takes the edge from NoSQL power that can store an entity – but maybe it’s good practice to have this separation to keep our Domain Model object free from limitations and restrictions (sometimes we need object that is stored on more than one table). The downside here is obvious: maintenance of classes duplication – you can try using mapping frameworks like automapper but the mapping may be complicated and I think you’ll end up writing your own mapping .

Hooking into DataServiceContext Reading and Writing events

This is needed to complete the solution above to help us dealing with Illegal characters by replacing them before writing and after reading. It can be a place holder for dealing with complex types (if they are private they won’t be stored) we can serialize them to a primitive property – as for me I was using JSON.NET to serialize complex type and I’m able to read it when exploring the stored entities (better than using binary serialization then you cannot understand the stored serialized object). The downside here is that you have to create another layer above the DataServiceContext and add your own logic there – not ideal but get’s the job done.

Use “Fat Entities”

There are some other alternative, one of them is an open source library to store Strong type entities – under the hood the entity is kept serialized in one property of the TableStorageEntity.

Here is the entity class:

     public class CloudEntity <T>

    {

            public string RowKey { get; set; }

            public string PartitionKey { get; set; }

            public DateTime Timestamp { get; set; }

            public T Value { get; set; }    

    }

It’s nice and enables working with one set of entity classes but the downsides are major here: first you cannot query on the entity fields (you can query only on the partition and table keys you set) and second you have to use specific layer that is not exposing all possibilities of the DataServiceContext (downside if you need to use something that is not exposed). This sollution is good if you need simple storing and retrieving by the keys values. get it from here.

Use Lucifure

Another interesting library developed in F# and gives a lot of awesome capabilities – I didn’t try it my self but in general: by using custom attributes on your you can store any entity without any limitations (It’s not an open source so I can’t see the code but I guess it’s done using the dynamic capabilities of F# and creates the rest URL calling the azure table storage service on it’s own). It also gives solutions to other limitations like the size limitations. I will try it and update but it’s really worth looking into.

here is entity example:

    [StashEntity(Mode=StashMode.Explicit)]

      public class Employee

      {

              // Give the partition, row key and timestamp more meaningful names

              [StashPartitionKey]

              public string Department;

              [StashRowKey]

              public string EmployeeId;

              [StashTimestamp]

              DateTime AzureInternal;

      }

Summery

Well I’m sure that by now you get the point and I’m sure there are more limitations I missed. Don’t get me wrong Azure Table Storage is great – it gives you out of the box high availability, enable scale and high performance but if your requirement needs a feature that is not supported then you better check the solutions and alternatives before you choose azure table storage and make sure it satisfied your needs.

In the next post I will talk about the querying limitation and what can we do to overcome them.

Cheers,Offir

TPL Data Flow Debugger Visualizer

Offir Shvartz — Tue, 05 Jun 2012 14:06:00 GMT

TPL Data Flow

I first encounter TPL data flow (part of .NET 4.5 - TPL DTF) during the build conference (I attended last year) but I must say I didn’t get it then. It sounded too much like RX which I was familiar with. Recently during one of my projects that required high performance CPU bounded – high throughput and low latency my colleague Alon the gave the idea – why not using TPL data flow and from then my world changed forever. It’s amazing to how many systems this technology can be suitable to,and for the project I was working on it was perfect. For those of you that are not yet familiar with this technology I suggest reading the article from here and see this amazing example of kinect and TPL DTF here – you’ll thank me later.

So I created some crazy TPL data flow networks on runtime (~100k different networks) and it worked getting 100% CPU most of the time, leveraging all the cores and getting really good performance – but it was really hard to debug and to understand the flow from the code. It was Alon again that suggested to write debug visualizer – so I did. This is my insights from the process.

Writing Debugger Visualizer

I don’t know how many of you had the chance to implement debugger visualizer but it is really straightforward if you need only view on a simple object just follow 5-6 steps:read here. Finally put your assembly in the visualizer folder (e.g. for VS 11 C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers) and you’re done. This is very simple if your debugging target is simple and serializable. I from the other hand had few issues.

The Issues I encountered during the implementation

Debugger Visualizer Target cannot be interface

This really surprised me because I know the target I wanted was IDataFlow which is the common interface of TPL data flow which all the blocks need to implement and I discovered that it’s impossible,the target must be a type (can be common base class but not an interface). My solution to this issue was not perfect – I added multiple attributes with all the build in blocks as target. It’s not that bad because the build in blocks are a good start and as long as you start from build in block (what you normally do) then it will discover the rest of the linked block even if they are not part of the build in blocks. In addition it’s not recommended to implement IDataFlow block on your own – there is a lot to take care of: locking synchronization,buffering and more. If one of you Microsoft guys are listening I think adding support of Interface as target of debug visualizer can be very handy.

DataFlow blocks are not serializable

As I wrote in the previous paragraph writing debugger visualizer is simple (in most cases…). Well somehow the running process (debugee) needs to transfer the debugged object to the IDE (debugger) this is done by serialization. the thing is non of the IDataFlow block are serialzable. In order to resolve it I implemented derive to VisualizerObjectSource – overriding the GetData method that gets the object and output stream and write it to the stream, the debugger can read the stream from his own size. So I created a serializable object that holds the debug info needed and serialized it to the stream and deserialized on the debugger side. I worked O.K until I tested more complicated networks containing shared blocks meaning two blocks or more are connected to the same block. when serializing I got different reference on the debugger side the solution for this problem was DataContrat attribute:

[DataContract(IsReference=true)] and we get the same reference.

Retrieving the information from the IDataFlow block

This experience made me learn a lot about the internal implementation of the build in blocks. If you look on this blocks you get a lot of data like Linked blocks Input Queue ext. This data is not part of the block properties – apparently each block has private inner class call Debug view that holds this data. This made me use some hard core reflection (made me understand how powerful reflection is). One more thing was the internal implementation of Joins and links. when you link to Join block it is linked to internal block called JoinBlockTarget<,> which has no data about the linked targets but has reference to the owner Join block .Another example is adding a link with a filter is adding another block called FilterLinkPropagator (see pic above)

So Start using it now…

Go to http://dataflowdebuggerview.codeplex.com/ download the source/binaries and start using it. I’ll be happy to get your comments and suggestions.

VS 2012 RC Update:TPL dataflow was moved to NuGet (http://nuget.org/packages/Microsoft.Tpl.Dataflow/4.5.1-rc), pay attention you have to add reference to System.Threadin.Tasks.dll (under the C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5)

Cheers Offir.

WCF 4.5 Configuration from external source

Offir Shvartz — Sun, 01 Apr 2012 18:48:38 GMT

WCF 4.5

There is a lot new in WCF 4.5 you can get the full list Here. It always amaze me how much is added from version to version – and the additions are important (UDP, Web Socket binding ext.)

One specific feature caught my eye – Configuring WCF Services in Code. It adds the possibility to setup WCF parameters after it was deployed or when you don’t have access to SerivceHost object.

This is done by a simple static function you can add to your service implementation:

public static void Configure(ServiceConfiguration config)

ServiceConfiguration allowing to add endpoints, behaviors ext. This is nice but why is it so interesting… let’s see the following problem and solution.

Problem - WCF Configuration in enterprise application

I think almost every one that developed enterprise application with WCF service as the server side entry point felt this pain. You want to be able to manage your WCF services configurations from single place – if you have more then one service defined you don’t want to use files for configuration and you want to persist the WCF configuration in one place. sure you can configure WCF in code (if you have access to ServiceHost) but you cannot use the same system.serviceModel configuration section – load it from external source.

Solutions – before WCF 4.5

I seen a lot of solutions along the way:

Implementing your own structure for WCF configuration and write a code that apply it on the service host object – the major downside here is that you have to duplicate the code that exists somewhere inside WCF that reads system.serviceModel configuration and build the matching WCF objects: Bindings, Behaviors ext.
I saw complicated solution that use reflection of the ServiceHost object and inject the system.serviceModel configuration section – the downside here is the risk (you can fail on runtime and the internal may change)
Nice solution by Alon Fliess was to create another app domain and associate specific file to the new domain allowing to read configuration from external file – read about it here.

All the solution above has some downside and this is because ServiceHost was not build to get the configuration from external source – all the solution above are workarounds.

Solve it WCF 4.5 style

This solution is not perfect too (read limitation) but it uses given WCF API which means it is the best we currently have. So as I mention the static function Configure was added but what is doing the trick it the function LoadFromConfiguration of the ServiceConfiguration object. This allows us to write this code:

public class EchoService : IEchoService
   {
       public static void Configure(ServiceConfiguration config)
       {
           var cfg = MyConfigurationManager.GetConfigurationByServiceType(typeof(EchoService));
           config.LoadFromConfiguration(cfg);
       }

       public string Echo(string input)
       {
           return string.Format("Echo:{0}", input);
       }
   }

All the magic is done in the “MyConfigurationManager” which can have is own logic – reading from DB by given parameters (could be service type, machine name ext.) and returns System.Configuration object that is loaded using the LoadFromConfiguration.

to complete my example my configuration manager read it from a file:

public static class MyConfigurationManager
   {
       public static Configuration GetConfigurationByServiceType(Type serviceType)
       {
           var fileMap = new ExeConfigurationFileMap() { ExeConfigFilename = Path.Combine(Environment.CurrentDirectory, "ExternalConfiguration.xml") };
           return ConfigurationManager.OpenMappedExeConfiguration(fileMap, ConfigurationUserLevel.None);

       }
   }

Limitation:Note that LoadFromConfiguration ignores <host> settings within the <service> tag of <system.serviceModel>. Conceptually, <host> is about host configuration, not service configuration, and it gets loaded before the Configure method executes.

Summery

I would like to be able to pass to ServiceHost Configuration object. we are still not there…maybe at WCF 5.0

Cheers, Offir

Windows Build- my first impressions

Offir Shvartz — Thu, 15 Sep 2011 12:15:19 GMT

Well I’m here at Anaheim enjoying every second (I’m a geek what can I say). Wanted to share some of my thoughts. I’ll just note that these word are written using new cool Win 8 Samsung tablet.

So I heard a lot of keywords and pieces of information in the passed days here are some:

Immersive
Fast and Fluid
Modern
Metro style
WinRT
HTML5/JS
Application Contracts
…

I’m still digesting all the information but there is one issue that really amazed me because it can really cause revolution in application capabilities. Windows 8 really bring new perspective on how applications should behave and interact.

I’m talking about the keyword "Application Contracts” (e.g. sharing contract, picker contract, Search contract…) , when looking on it from the software architecture perspective: the infrastructure is the OS (windows) and application may considered as modules all part of a one large application. Everybody knows that a good design is keeping the modules decoupled but enable interaction between them using plugin/IoC infrastructure – like MEF. Microsoft is bringing the same notion to the Metro Application using Win 8 (the infrastructure) enabling plugin infrastructure – Application Contracts. This way application may expose itself in decoupled way and consume other applications services without even know about each other. So Microsoft is introducing new application plugin infrastructure – thinking about it this can really give us new generation of application that speaks with one another and enhance each other – the example I can think of is GPS locator application that expose his search capabilities consumed by social network application allows the user to know what are his friends location.

Re-imagine this…

Cheers Offir

WCF Performance Using Datasets – Part 2

Offir Shvartz — Sat, 23 Jul 2011 19:44:00 GMT

start with reading WCF Performance Using Datasets – Part 1.

Can it be ?

I asked this question over and over – because WCF is clearly faster (see Eyal Vardi comment in part 1). I tried several Binding options including some custom binding , I asked some people to look at my code and up to now no one has manage to contradict my benchmark. I invite each one of you to try (write you own tester) my contract is simple:

[ServiceContract] public interface IPerfService  {       [OperationContract]       DataSet LoadDataSet(DataSet ds);   }

Why is WCF slower for Datasets ?

I decided to use a profiler and look at the time consuming deviation. I like using Dot Trace profiler because it is very simple and clear. I was sampling one of 1000 iteration after 100 iteration.

Hold tight we are going deep.

You can see that the WCF call (ClientLoadDataSetSelfHost) took 719 ms and the Web Service call (LoadDataSet) took 619 ms

Drilling down I discovered that this gap was a result in the gap of desterilize method duration – makes sense because this is most of the work (DataSet of 15,000 rows).

it's Important to remember that when serializing / deserializing an object that is implementing IXmlSerializable it doesn't meter what kind of serializer is defined it will use the object IXmlSerializable implementation – this can be very problematic when using hybrid option when object that not implementing IXmlSerializable holds members that implements IXmlSerializable but discussing this can write full other post.

Both are using: System.Data.DataSet.ReadXml(XmlReader) (from System.Xml.Serialization.IXmlSerializable) - in this function we see the 100 ms diff in duration between Web Service and WCF.

Digging deeper going to the method: System.Data.XmlDataLoader.LoadTable(DataTable, Boolean) called from System.Data.XmlDataLoader.LoadData(XmlReader) called from System.Data.DataSet.ReadXmlDiffgram(XmlReader).

In WCF:

Web Service:

you can see that most of the method takes more time and this is due to the XMLReader specific type that DataSet is choosing in runtime: in web service XMLTextReader in WCF XMLBinaryReader – reading using BinaryXMLReader is taking more time.

When I used text encoding still in WCF it was using XMLBufferReader and not XMLTextReader. In addition I tried most of the binding options but still DataSet is choosing the XMLBufferReader – I’m open for your suggestions. If we could define specific binding that will get DataSet to behave the same as in WebService it could be very useful. The thing is we cannot modify DataSet Code – the XMLReader specific type choosing part.

What is the solution

I’ll start with what is not the solution for sure which is going back to Web Services – we cannot allow our application not moving to WCF because we need WCF power:Security, full separation of binding from service implementation , easy maintaining , tracing ,error handling and most important this is the technology Microsoft is developing and supporting NOW.

I’ll divide my solution to two main options:

The most correct but annoying solutions: Do not use data set – it’s a large cumbersome out dated object. In addition it’s not strong typed meaning it can cause to fail on runtime. Now the way to go is Domain driven.

I know the fist option is not always possible – sometimes too costly and too risky still this is the way to go.

DataSet Surrogate – I saw it in several places over the web (here is one). This solution is to hold the all dataset data in other fast serialized object and past the surrogate object over the wire. You can change the WCF method signature to pass this SurrageDataSet or better use Parameter Inspectors (read more here) to do it using WCF extensibility options.

Conclusion

You may encounter the situation of passing DataSets in WCF only on legacy solutions but it’s important that when refactoring old application, you should be aware that moving to WCF not necessary means memory improvement – mainly if all your object model is data sets (very common in legacy solutions).

I implemented and tested my own DataSet surrogate (not supporting relations) and in my bench mark WCF was faster then Web Service in ~15%. I’m sure it depends on number of tables and rows and to rebuild DataSet takes time too. Still I think this can bring significant performance improvement.

WCF Performance Using Datasets – Part 1

Offir Shvartz — Mon, 04 Jul 2011 02:51:24 GMT

Background

It’s all started when I encountered a strange behavior in one of the systems I work on.

The system is a legacy system that uses a simple client server architecture implementing using web service – this was pre-WCF time so the options were only .NET remoting and web services. Now as part of major refactoring process of the system we decided to replace the communication layer from WS to WCF.

The motivation is obvious:

Alignment with Microsoft latest infrastructure.
performance improvement – http to TCP/ Binary encoding.
Easy maintenance.
Easy monitoring and configuration.
Extendible – interception points (behaviors).

After the modification was completed I found that the system became slightly slower – how can it be.

WCF performance vs. Web Service – What do we know

This bring me to article from 2007 written by Microsoft: http://msdn.microsoft.com/en-us/library/bb310550.aspx

“To summarize the results, WCF is 25%—50% faster than ASP.NET Web Services, and approximately 25% faster than .NET Remoting. Comparison with .NET Enterprise Service is load dependant, as in one case WCF is nearly 100% faster but in another scenario it is nearly 25% slower. For WSE 2.0/3.0 implementations, migrating them to WCF will obviously provide the most significant performance gains of almost 4x.”

My Benchmark

Still articles are articles and facts are facts – “the costumer is always right”

In addition in the article there was no mention of datasets specifically.

In the legacy system we mostly use DataSet (I know pretty ugly but this is what we did in a legacy code what can I do…).

So I decided doing my own simple benchmark.

I tested two scenarios:

Pass 10000 objects – the object include two fields: integer and string.
Pass “large” dataset – includes one table the has two string columns contains 10000 rows.

The services types:

Simple ASMX web service.
Self hosted WCF service.

the client makes sequential call to the services: 1 then 2 and I measure the duration in milliseconds.

I repeat the same test 1000 time and use the average duration of the last 100 test runs.

here are the results (duration in milliseconds):

Conducting a Benchmark Notes

Not doing the following can get you faulty results.

Remember to run the tests not using Visual Studio (not using the .vshost.exe process).
Run the sources complied in “Release”.

Conclusions

My conclusion is for dataset transfer Web Service is faster than WCF (around ~40% faster).

So when choosing to upgrade legacy code to use WCF – will not always increase your system performance – if you use mostly datasets your system may become slower.

What’s next

This is only the start because:

First we need to understand what is the cause for the performance decrease – why is WCF slower only for DataSets.
Second we need to find a solution to use WCF but still keep our performance or better improve – going back to Web Service is not an option.

So stay tune for part 2, I promise it’s we’ll be worthwhile.

http://blogs.microsoft.co.il/blogs/oshvartz/archive/2011/07/23/wcf-performance-using-datasets-part-2.aspx

Cheers

Offir

Publish/Subscriber Using Routing Service (WCF4)

Offir Shvartz — Mon, 21 Feb 2011 02:05:00 GMT

Hi all.

Download Code

Few weeks ago I went to an interesting lecture of Ido Flatow called: “What’s new in WCF4”. The lecture was very good mostly because it was the first time somebody gathered and organized the new abilities of WCF 4, and the material level in my opinion was pretty high. One of the features Ido talked about was the Routing Service. That feature just amazed me, I found it very powerful and useful in many scenarios: load balancing, on service failure, content base routing, bridge and more (MSDN Link).

Pub/Sub Solutions

I was at the time examining some pub/sub solutions. Discussing them can easily fill a series of posts. I will only mention few points on the subject: First there are many options available:

Rhino Service Bus.
NService Bus.
MassTransit.
WCF solution – contract base Juval Lowy Solution.

Second, the Pub/Sub solution I looked for have several significant requirements:

The usage must be very simple and clear – strong typed without complicated configuration – something like the Mass Transit interface(this ruled out the last option).
I didn’t want the MSMQ based solution (the first 3) – one of the reasons is that MSMQ has size limitation (4 Mega) – although there are ways to reduce the size limitation.
The pub/sub host must be completely loose coupled from publishers and subscribers.

I wanted WCF based solution but I had some bad experience using the Duplex capabilities of WCF – it was causing many problems, especially when the publisher server goes down and back up.

My Pub/Sub POC

During Ido’s lecture I realized that Routing service is perfect for Pub/Sub solution. Routing Service Distribute massage to all subscribers – and we get out of the box loose coupled of Binding, Security and contract. Each subscriber is hosting one way WCF service per massage type to get notification, so my first step was to define the client interface (requirement number 1).

The interface:

public interface IPubSubClient{     void Publish<T>(T message);     void Subscribe<T>(ref Action<T> handler);     void Unsubscribe<T>(ref Action<T> handler);}

This is how to Configure and run the routing service:

ServiceHost pubServiceHost = new ServiceHost(typeof(RoutingService));ConfigureRouter(pubServiceHost); pubServiceHost.Open();

Configure Router:

var routerContract = typeof(ISimplexDatagramRouter); string routerAddress = "net.tcp://localhost:8888/publisher"; Binding routerBinding = new NetTcpBinding(); //add the endpoint the router will use to recieve messagesserviceHost.AddServiceEndpoint(routerContract, routerBinding, routerAddress); //create a new routing configuration objectRoutingConfiguration rc = new RoutingConfiguration();rc.RouteOnHeadersOnly = false; serviceHost.Description.Behaviors.Add(new RoutingBehavior(rc));

Writing the POC was quite fluent, which made me more sure that the Routing Service is really suitable in this case.

Interesting Points

The code lies in front of you, still I wanted to point out some points I think are important:

Configure the routing service is very simple using the class RoutingConfiguration we can add/remove keys from the filter table – which holds the routing table.
We can change the Filter table during run time by calling ApplyConfiguration:

internal void Subscribe(string messageType, string endpointAddress){    lock (_routingConfiguration)    {        ContractDescription cd = ContractDescription.GetContract(typeof(IRequestReplyRouter));         ServiceEndpoint servEndpoint = new ServiceEndpoint(cd, new NetTcpBinding(), new EndpointAddress(endpointAddress));         _routingConfiguration.FilterTable.Add(new MessageTypeMessageFilter(messageType),                      new List<ServiceEndpoint>() { servEndpoint });         _routingHost.Extensions.Find<RoutingExtension>().ApplyConfiguration(_routingConfiguration);    }}

The subscribers open the the notification service channel during runtime , using the same port (although they can use other random port numbers) – in order for it to work you must enable net.tcp port sharing by going the windows services (running services.msc) and start the service “NetTcpPortSharing”.

private ServiceHost CreateNewService<T>(List<Action<T>> handlers){    ServiceHost pubServiceHost = new ServiceHost(new PublisherService<T>() { Handlers = handlers });    string pubAddress = string.Format( "net.tcp://{0}:7777/pub/{1}/{2}",Environment.MachineName,typeof(T).Name,Guid.NewGuid());    var portSharingBinding = new NetTcpBinding();    portSharingBinding.PortSharingEnabled = true;     pubServiceHost.AddServiceEndpoint(new ServiceEndpoint(ContractDescription.GetContract(typeof(IPublisherService<T>)), portSharingBinding, new EndpointAddress(pubAddress)));     return pubServiceHost; }

I’m using XPathMessageFilter for matching massage type to his subscribers.

I am printing the latency of the message (avg. of ~60 MilliSec in my computer) and I notice that even when running ~50 subscribers the latency remain the same – this lays in the Routing Service implementation that distribute the massages efficiently.

Summery

Download and run the POC (don’t forget to start the port sharing service) – run the Pub/Sub Server and as much clients as you like. Please share your comments or/and thoughts.

Cheers Offir.

EF4 In real life and using Code Only – Yes We Can ?

Offir Shvartz — Wed, 09 Jun 2010 02:58:37 GMT

Hi all it's been a while but i didn't have anything interesting to write about. Lately I was involved in a process of choosing the technology used in Data access infrastructure of new application, and let's just say it wasn't a simple decision but a fascinating one.

General

The decision process was interesting and long, the dilemma was reduced at the end to decide between two ORM solutions: NHibernate and Entity Framework 4 – EF4 (I'll give some details below). Finally we decided to go with EF4. The risky part in this decision is that EF4 is only lately released (with vs 2010 RTM) and we don't have any known enterprise application using it in production.I Just want to add into considerations that the application Client is developed at Silverligth 4.

Special requirements – Code Only.

The specific application had one additional special need: defining the DB mapping in runtime which means do not use a static edmx file. The solution was to use the Code Only feature – which is currently part of CTP (Microsoft ADO.NET Entity Framework Feature CTP 3) and not part of the release. This issue was added to the danger factor – can we use it to define a complicated model – does it answers our needed scenarios and is it good enough from performance point of view.

EF4 vs NHibernate

There is a lot to say in this section but I'll be brief.

First NHibernate has many advantages over EF4:

Has more fine tuning abilities – scalability and performance.
Is very mature – works for years in enterprise applications.
Has more features.
More Extendable – Open Source.
Supports more DB types out of the box.

So why to choose EF4 you ask. Well for my specific needs, EF4 is more suitable from the following reasons:

EF4 is more suitable for working in N-Tear architecture:
- It has disconnected out of the box solution - the Self Tracking.
- In order to use NHibernate for N-Tear you may need to use DTO because: working N-Tear with NHibernate is not straightforward especially with silverligth for example the many to one navigation property are defined in POCO as interface and the strong but complex lazy loading capabilities of NHibernate are not suitable for N-Tear when your entity is disconnected from the session.
EF4 is more easy to start working with – you have the model designer and code generation items – and it all comes out of the box (NHibernate has some similar solutions but they supply only part of the features EF4 gives – comes out of the box in VS 2010) .
EF4 is Microsoft – we cannot ignore this fact. We develop in a Microsoft environment, the .Net framework is evolving (LINQ, Dynamic ,Generics,Covariance …) it makes sense that Microsoft will evolve EF as well and do it more easily because it's all develops in the same organization.

EF4 problematic issues

while working with EF4 I encountered some problems, some of them are pretty critical.

Code only

Using the code only was a must because of my special needs (as described above). Although using CTP I must say the work was quite smooth and very intuitive. The problems I encountered were not related to the code only feature specifically, but to EF4 in general. The downside was the lack of documentation and information in general. the best information source I found was the All-In-One Code Framework from codeplex and the ADO.NET team blog.

The inheritance TPH (table per hierarchy) issue

This issue was very surprising for me, I tried to define TPH inheritance with code only (using the example from All-In-One Code Framework), then I got strange exception ("InvalidException previous cannot be null"). After a short investigation I discovered that the reason was that the derived classes and the base class weren't located in the same assembly. I checked it without code only feature (with simple edmx file) and the result was the same – an ugly exception. Please tell me if I am doing something wrong or missed something because if not it's a bug and a big one – most likely that in enterprise applications there will be a separation between the base classes and derived classes. So I did a workaround (too ugly to mention) for this issue, but I'm open for you suggestions how to deal with this problem.

The inheritance TPT (table per type) issue

This is a known issue of Entity Framework (from EF1 and still not resolved in EF4). When you define a very common inheritance strategy: each derive is stored in separated table. EF fully supports this strategy but the queries it produces when querying the model is "evil". evil meaning very long (in some scenarios it can cause exceeding the query string limit if we have many inheritances) and has very poor performance – union of left joins of all the derived tables. EF also supports Hybrid mapping meaning define TPT and gives a condition for each specific type (called discriminator in NHibernate) but when producing the Query EF still generates these "evil" queries.

I must say this issue is critical and Microsoft should attend it as soon as possible because currently it's is one of the biggest obstacles when using EF in enterprise applications.

The Solution is to avoid working from the base class when querying, meaning add and delete should be done from the base (we don't really have other options), but when querying work only from the derived use the syntax:

from user in context.Users.OfType<SpecificUser>();

this will generate an efficient inner join query.

* If you need to query your model from the base type consider using TPH or even consider not using EF4 because currently there is no good solution for this issue.

Summery

Having said that, what is the answer to the question in the title: yes we can ?

well the answer is still not clear so I'll say: at this early point it looks like yes :-). I'm sure all the problems I mentioned will be eventually solved by Microsoft, but for now it works (with some workarounds).In my DAL infrastructure – abstraction layer above EF4 (using Repository and UnitOfWork pattern) I use EF in two ways:

Using code only context.
Using edmx file - The usual static model.

I'll update you as the application will get more mature. That's all, waiting for your comments and\or suggestions.

Cheers, Offir.

Self tracking “Poco’s” (self tracking data transfer objects)

Offir Shvartz — Fri, 04 Dec 2009 23:21:39 GMT

Recently I worked on a POC with the guidance of Bernie Almosni – Building a layer above the business entities (stored in a data base) with the use of an ORM (as the data access engine). During the process we encountered the disconnected scenario issue (it’ll describe in details below). The Self tacking DTO component was develop by Bernie and myself as a part of the POC – aimed at addressing the disconnected scenario. We ended up with a generic framework not bound to specific ORM, that I think can be useful to many.

POC Requirements

The business entities are POCOs (Plain old CLR object).
We use inversion of control (specifically unity) so each of the business entities implement an interface. We work only with the interface in our business logic components.

ORM Context

Definition: this term is common for all ORMs that supports using POCOs as persistent entities. It describes the component that manages the entities by tracking the entities modifications. Then persist these specific modifications by sending SQL command to the database. In Entity Framework it is the ObjectContext and in NHibernate it is the Session.

The Disconnected Scenario

the flow:

Load an entity.
Send it to the client – N-Tier maybe by WCF. (the client doesn’t hold the ORM context so the entity is in a disconnected state).
The client modifies the entity.
The client sends the entity back to the server.
The server synchronizes the modifications with the database – this is the hard part since the server doesn’t know what modifications where made, if any.

The Self Tracking DTO Framework

The framework enables the scenario above by using special objects – self tracking DTOs. The server works with the POCOs as usual but when it needs to send an entity to the client it uses a different object (a DTO).It implements the same interface but in addition provide special functionality:

It Populates itself by a given server entity.
Creates a new server side entity that contains its own value – by the common interface.
Applies the modification made on it on a given server side object.
Bonus: INotifyPropertyChange – implemented by the DTO implements so if you use WPF you get it out of the box.

The framework provides two two core interfaces:

1. The ISelfTrackingDataTransferObject interface

Provides the self tracking DTO needed functionality:

void Populate(T entity) - Populates the DTO object by the given entity.

T Clone(bool getOriginalValue) - Creates a DTO clone - Server side object that holds the same value as the DTO.

void ApplyChanges(T enitity) - Simulate the changes made on the DTO object on the given entity ("plays" the changes).

2. The IDtoInstanceCreator interface

ISelfTrackingDataTransferObject<T> CreateDtoInstance<T>(T entity) - creates a new DTO instance by T and a given entity.

The class that implements this interface is the DTO factory – it create DTO instance by using the two parameters (T) and specific entity object. Note: if we use inheritance some times entity type is derive from T and we need to create the DTO instance by the entity type.

Example

This example is part of the unit tests in the source, you can download and run it. I use Unity as my IoC so I use the UnityDataTransferObjectBase but it’s not mandatory, you can use the SelfTrackingDataTransferObjectBase as base class of the DTO object instead.

We have a simple entity that inherits from base class and has a matching interface:

public interface ISimpleObject : IObjectBase
   {
       int Id { get; set; }
       string Name { get; set; }
   }
public class ObjectBase : IObjectBase
   {
       public int Id { get; set; }
   }

public class SimpleObject : ObjectBase ,ISimpleObject
   {
       public string Name { get; set; }
   }

Now let’s look how at the DTO implementation:

[DataContract]
public class SimpleObjectDto : UnitySelfTrackingDataTransferObjectBase<ISimpleObject>, ISimpleObject
{
    public SimpleObjectDto(IUnityContainer unityContainer) : base(unityContainer) { }
    #region ISimpleObject Members
    [DataMember]
    private int _Id;
    [DataMember]
    private string _Name;

    public int Id
    {
        get
        {
            return _Id;
        }
        set
        {
            Set("Id", ref _Id, value);
        }
    }
    
    public string Name
    {
        get
        {
            return _Name;
        }
        set
        {
            Set("Name", ref _Name, value);
        }
    }

    #endregion

    protected override void SpecificPopulate(ISimpleObject entity, IDataTrasferObjectToEntityMapper dataTrasferObjectToEntityMapper)
    {
        _Id = entity.Id;
        _Name = entity.Name;
    }

    protected override void SpecificGetValue(ISimpleObject entity, bool getOriginalValue, IDataTrasferObjectToEntityMapper dataTrasferObjectToEntityMapper)
    {
        entity.Id = RetrievePropertyValue<int>("Id", getOriginalValue);
        entity.Name = RetrievePropertyValue<string>("Name", getOriginalValue);
    }

    protected override void SpecificApplyChanges(ISimpleObject entity, IDataTrasferObjectToEntityMapper dataTrasferObjectToEntityMapper)
    {
    }     
}

Building the DTO is a technical action (no need to think here), in the source I have a readme guide that explains how do it and examples for each of the possible scenarios:

Entity holds only simple types.
Entity holds a reference to an Entity.
Entity holds a list of entities.

Now we implement the IDtoInstanceCreator interface, using unity. ( I use unity singleton which is just a singleton object holding UnityContainer)

public class MyDtoInstanceCreator : IDtoInstanceCreator
    {
        #region IDtoInstanceCreator Members

        public ISelfTrackingDataTransferObject<T> CreateDtoInstance<T>(T entity)
        {
            return (ISelfTrackingDataTransferObject<T>)UnitySingleton.Instance.UnityContainer.Resolve<T>(DataTransferObjectsTester.STR_DTO);
        }

        #endregion
    }

Setup our container:

UnitySingleton.Instance.UnityContainer.RegisterInstance<IDtoInstanceCreator>(new  MyDtoInstanceCreator());
UnitySingleton.Instance.UnityContainer.RegisterInstance<IUnityContainer>(UnitySingleton.Instance.UnityContainer);

//concrete types
UnitySingleton.Instance.UnityContainer.RegisterType<ISimpleObject, SimpleObject>();
//DTOS
UnitySingleton.Instance.UnityContainer.RegisterType<ISimpleObject, SimpleObjectDto>(STR_DTO);

Now we are ready to go!

First loading and sending the entity to the client:

ISimpleObject serverSideSimpleObject = LoadSimpleObject();
IDtoInstanceCreator dtoCreator = UnitySingleton.Instance.UnityContainer.Resolve<IDtoInstanceCreator>();
ISelfTrackingDataTransferObject<ISimpleObject> dtoObject = dtoCreator.CreateDtoInstance<ISimpleObject>(serverSideSimpleObject);
dtoObject.Populate(serverSideSimpleObject);
return dtoObject;

The client gets the DTO and ,modifies it and send it back to the server.The server is updates the entity state in data base (this is done inside ORM context scope):

using (IContext context = GetORMContext())
{
   ISimpleObject serverSide =  dtobject.Clone(true); //server side object holding the original values
   context.Attach(serverSide); // attach the server side instance to context.
   dtobject.ApplyChanges(serverSide);
   context.SaveChanges();
}

Next Phases

Code Generator to generate the DTO class from the server side – I’m on it…
Support more actions on the collection – currently supports only add and remove.
More… I am open to your suggestions.

Entity Framework 4

EF4 that is currently in beta (part of VS 2010 beta) has a CTP with another very similar solution to the disconnected scenario (read about it here). This solution varies from the Self tracking DTO for example it uses the same objects in the client and server side. Note this solution is EF bound (cannot be used with other ORM) and currently it is only a CTP. In addition if you use Oracle DB you currently don’t have a working EF4 provider ( I believe this will change in the near future).

So that’s all, this code is open for you to use and upgrade.

Download it from SelfTrackingDTO in CodePlex.

Cheers Offir.

Passing Event Handlers Over WCF

Offir Shvartz — Sun, 11 Oct 2009 00:40:08 GMT

It’s been a really long time since my last post. Lately I been working on a POC aimed at building an infrastructure layer for accessing the data in the application. Currently this layer is based on Nhibernate and supports several functionalities like WCF interface and more. As part of the POC I encountered a problem and managed to give it a satisfying solution. I want to share the problem and the solution with you.

The Problem

I was building a complex object hierarchy – each object has a reference to a different object or to a collection of other objects. I build this object hierarchy in the server side and pass it on to the client (using WCF). The problem was that part of my logic needed to use the objects events (to be specific I use the ProperyChanged event). Objects in the objects tree are subscribers of the other objects events, to be more specific each object is registered to the PropertyChange event of its child objects.The events should be fired only on the client side but the registration is done in the server side.

I know the Delegate object is serializable so first I tried using the default WCF serializer – DataContractSerializer. I mark my class with the serializable attribute – i could not use DataContract attribute because you can only put [DataMemeber] on fields and properties (not events) .

The code:

[Serializable]
   public class ClassContainsEvents : INotifyPropertyChanged
   {
       #region INotifyPropertyChanged Members

       public event PropertyChangedEventHandler PropertyChanged;

       private void OnPropertyChanged(string properyName)
       {
           if (PropertyChanged != null)
           {
               PropertyChanged(this, new PropertyChangedEventArgs(properyName));
           }
       }
       #endregion

       private int _testProperty;

       public int TestProperty
       {
           get { return _testProperty; }
           set
           {
               _testProperty = value;
               OnPropertyChanged("TestProperty");
           }
       }

       public void RegisterEvent()
       {
           PropertyChanged += new PropertyChangedEventHandler(TestClass_PropertyChanged);
       }

       void TestClass_PropertyChanged(object sender, PropertyChangedEventArgs e)
       {
           Console.WriteLine("Property Changed: {0}", e.PropertyName);
       }
    
   }

Then I got SerializationException:

“Type 'System.DelegateSerializationHolder+DelegateEntry' with data contract name 'DelegateSerializationHolder.DelegateEntry:http://schemas.datacontract.org/2004/07/System' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer”.

I tried to use XmlSerializer and didn’t get the exception I just got the object without handlers on the client side.

The Solution

This problem has several solutions each of them has their own advantages and disadvantages.

Solution 1 – Use NetDataContractSerializer

This solution is the simplest one, just modify your WCF to use the NetDataContractSerializer (detailed description in the link) instead of the DataContractSerializer and it works like a charm. So this solution is really simple and quick, but the disadvantage in this solution is clear:

Using this serialzer binds the service to only for .NET client making the WCF service platform dependent.
This serialzer is much less efficient than the DataContractSerializer because it adds CLR types meta data in addition to object data .

Solution 2 – Use Serialize events

The serialization process fires events throughout the process – we can hook the OnDeserialized event (fired after the deserialize process finished) and reconnect the events subscribers. This solution has many disadvantages:

We have to save state (the subscribers data) in the object holding the event in addition to the event holding it’s handlers – redundancy.
Add specific logic to the object for serialization – in the object there is specific code for serialzation.

The solution in code (added the OnDeserialized method) :

[DataContract]
   public class ClassContainsEvents : INotifyPropertyChanged
   {
       #region INotifyPropertyChanged Members

       [OnDeserialized]
       private void ReconnectEvents(StreamingContext context)
       {
           PropertyChanged += new PropertyChangedEventHandler(TestClass_PropertyChanged);
       }

       
       public event PropertyChangedEventHandler PropertyChanged;

       private void OnPropertyChanged(string properyName)
       {
           if (PropertyChanged != null)
           {
               PropertyChanged(this, new PropertyChangedEventArgs(properyName));
           }
       }
       #endregion

       private int _testProperty;
       [DataMember]
       public int TestProperty
       {
           get { return _testProperty; }
           set
           {
               _testProperty = value;
               OnPropertyChanged("TestProperty");
           }
       }

       public void RegisterEvent()
       {
           PropertyChanged += new PropertyChangedEventHandler(TestClass_PropertyChanged);
       }

       void TestClass_PropertyChanged(object sender, PropertyChangedEventArgs e)
       {
           Console.WriteLine("Property Changed: {0}", e.PropertyName);
       }
    
   }

Solution 3 – Use the SerializableEventHandler

This solution is an upgrade of solution 2: giving a generic way to save the handlers in a generic object called SerializableEventHandler. Saves the handlers only once – remove the redundancy. When you want or must use the DataContractSerializer for your own reasons you can use this generic class.

So let’s look at the code – implementing INotifyPropertyChanged.

[DataMember]
private List<SerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>> _notifyPropertyHandlers = new List<SerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>>();

public event PropertyChangedEventHandler PropertyChanged
{
    add
    {

        _notifyPropertyHandlers.Add(SerializableEventHandler.InitSerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>(value));
    }
    remove
    {
        _notifyPropertyHandlers.Remove(SerializableEventHandler.InitSerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>(value));
    }
}

private void OnPropertyChanged(string properyName)
{
    if (_notifyPropertyHandlers != null && _notifyPropertyHandlers.Count > 0)
    {
        foreach (var handler in _notifyPropertyHandlers)
        {
            handler.Invoke(this, new PropertyChangedEventArgs(properyName));
        }
    }
}

#endregion

We define a List of SerializableEventHandler, and use Event Accessors to override the += and –= operators.

The firing event methods just runs on the handlers list and call the Invoke method.

So let’s look at the SerializableEventHandler code:

[DataContract(IsReference = true)]
public class SerializableEventHandler
{
    public static SerializableEventHandler<TT, TTEventArgs> InitSerializableEventHandler<TT, TTEventArgs>(Delegate handler)
     where TTEventArgs : EventArgs
    {

        return new SerializableEventHandler<TT, TTEventArgs>()
        {
            MethodName = handler.Method.Name,
            Traget = (TT)handler.Target
        };
    }
}

[DataContract(IsReference = true)]
public class SerializableEventHandler<T, TEventArgs> : SerializableEventHandler
    where TEventArgs : EventArgs
{
    private MethodInfo _handlerMethodInfo = null;

    [DataMember]
    public T Traget { get; set; }
    [DataMember]
    public string MethodName { get; set; }

    public void Invoke(object sender, TEventArgs args)
    {
        initMethodInfo();
        _handlerMethodInfo.Invoke(Traget, new object[] { sender, args });
    }

    private void initMethodInfo()
    {
        if (_handlerMethodInfo == null)
        {
            _handlerMethodInfo = Traget.GetType().GetMethod(MethodName, BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance);
        }
    }



    public override bool Equals(object obj)
    {
        if (obj is SerializableEventHandler<T, TEventArgs>)
        {
            var handler = obj as SerializableEventHandler<T, TEventArgs>;
            return Traget.Equals(handler.Traget) && MethodName.Equals(handler.MethodName);
        }
        throw new ArgumentException("Cannot comper to other types than SerializableEventHandler<T,TEventArgs>");
    }

    public override int GetHashCode()
    {
        return Traget.GetHashCode() + MethodName.GetHashCode();
    }
}

Note the class gets two generic types:

T – The type of the instance holding the handler method.
TEventArg – The type of the second parameter in the handler method.

We save the method name as string – because we can’t serialize the MethodInfo object, the reason is the delegate holds the MethodInfo member that is marked as serializable but holds RuntimeMethodInfo type instance which is internal and has only private constructor (you cannot create instance). So we use reflection to invoke the method but the Method info retrieval will be called only once in the first call – we save a MethodInfo as a member after he first Invoke call.

The class full code:

[DataContract]
public class ClassContainsEvents : INotifyPropertyChanged
{

    private int _testProperty;
    private Guid _id;
    
    public ClassContainsEvents()
    {
        Id = Guid.NewGuid();
    }

    #region Properties

    [DataMember]
    public Guid Id
    {
        get { return _id; }
        set
        {
            _id = value;
            OnPropertyChanged("Id");
        }
    }
    [DataMember]
    public int TestProperty
    {
        get { return _testProperty; }
        set
        {
            _testProperty = value;
            OnPropertyChanged("TestProperty");
        }
    }
    #endregion

    #region EventHandlers

    public void RegisterEvent()
    {
        PropertyChanged += new PropertyChangedEventHandler(TestClass_PropertyChanged);
    }

    public void UnregisterEvent()
    {
        PropertyChanged -= new PropertyChangedEventHandler(TestClass_PropertyChanged);
    }

    void TestClass_PropertyChanged(object sender, PropertyChangedEventArgs e)
    {
        Console.WriteLine("Property Changed: {0}", e.PropertyName);
    }
    #endregion

    #region INotifyPropertyChanged Members

    [DataMember]
    private List<SerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>> _notifyPropertyHandlers = new List<SerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>>();

    public event PropertyChangedEventHandler PropertyChanged
    {
        add
        {

            _notifyPropertyHandlers.Add(SerializableEventHandler.InitSerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>(value));
        }
        remove
        {
            _notifyPropertyHandlers.Remove(SerializableEventHandler.InitSerializableEventHandler<ClassContainsEvents, PropertyChangedEventArgs>(value));
        }
    }

    private void OnPropertyChanged(string properyName)
    {
        if (_notifyPropertyHandlers != null && _notifyPropertyHandlers.Count > 0)
        {
            foreach (var handler in _notifyPropertyHandlers)
            {
                handler.Invoke(this, new PropertyChangedEventArgs(properyName));
            }
        }
    }

    #endregion

    public override bool Equals(object obj)
    {
        if (obj is ClassContainsEvents)
        {
            var cce = obj as ClassContainsEvents;
            return Id.Equals(cce.Id);
        }
        throw new ArgumentException("Cannot comper to other types than ClassContainsEvents");
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode();
    }
}

}

Limitation:

T must be serializable.
In order to use –= in the client side the class T must implement Equal method because the instance of the Target is different between the client and server.
We have to hold List of SerializableEventHandler for each target of type(T) we want to support containing handlers (act as a Target) – the better solution is to set T to be one common base class that all event subscribers derive from.

Summery

Well in this post I gave 3 solutions to a the passing event handlers problem, this problem is not common (it took me 6 years before I first encounter it) but the solutions are helping us understand other issues regarding serialization. I recommend reading about the different WCF serializer types see the link, and you may use one of the ideas above to solve other issues. I’ll be glad to hear your comments.

Cheers, Offir.

Layered Application wannabe

Offir Shvartz — Mon, 23 Feb 2009 00:05:20 GMT

Hi it's been a while since my last post (just had nothing interesting to write about). In this post I wanted to share an issue I encountered recently while reviewing a medium scale web application.

Layered Application

This is a common architecture of an enterprise application that has a large number of components (read about it here).

The Problem

I was told that the web application I was reviewing conformed with the Three-Layered Services Application pattern. I started taking a closer look at the code and found that the web app had one solution that contains a web site and three class library projects: BL (Business Logic), DAL (Data access Layer) and Common. It seemed like the layers of the app were really independent and separated, until I encountered something like:

public void DoSomeLogic() //Some Input{    DataSet entitiesDs = DAL.ReadEntities();    foreach(DataRow dr im entitiesDs.Tables[0].Rows)     {        runSomeAction(dr["entityId"])     }    //More code}

The above is a simplified example of code in the BL assembly (Business Logic layer) - It uses the DAL in order to read the entities from data base.

So what's wrong with this code ?

The BL is using the first table from the data set assuming it exists and contains the right data - can cause runtime error.
In the Data Row object we read column name "entityId" (even if it is const string) - causing two problems:
- can cause runtime error if the column doesn't exists.
- dr["columnName"] returns an object we need to cast or parse - this can cause invalid cast exception and performance issues in case we Parse the object result.
The use of DataSet - first I must confess that I HATE DataSets because:
- it uses a large amount of memory that in most cases contains more then we need (especially when we build a layered application) like relations.
- it is weak type (the opposite of strong type) you just don't know what you have - you just can't understand from your code what the data set contains causing maintenance to be very hard, essentially requiring you to debug to understand the code.

The main issue: We don't really have layer separation --> changes in the Data Base schema may cause changes both the Business Layer (BL) and in the data access layer (DAL).

Layered Application Benefits

When I think back to the years I develop without Layer separation, I just can't understand how I could do it. Layer separation seems so natural and so required now. There are a lot of benefits (I'll try to give most of them)

Maintenance - in case of schema change we need to modify only the Data Layer.
Reusable - Blocks of functionality are modular and can be used in other applications due to good separation and abstraction.
Readable - good separation cause clear code. The objects that pass between the layers are simple (containing no logic) making the function flow clear and understandable.
Testable - if you use TDD you have to keep this separation because you want to test only a specific unit (in the BL for example) - no separation makes this task impossible.
Flexibility - replacing Layers while using the same interface is a simple task when the separation is complete.
Scalability - when the data objects are simple and small size we can easily deal with them in chunks, clear code allows developing scalable application.

Solution

The solution is to use simple data container objects to pass between layers ("object model"). In most cases I think they shouldn't hold any logic - leave the logic to the BL layer.

You already know what I think of DataSets, but still there are some cases (not many...) we need to use them. For example - a loose coupled functionality (when we don't know and care what is returned and just want to bind it to a data grid in our UI). but if out BL needs to implement some functionality using the DataSet (like in the example above) - we must make only the data we use strong typed and init the objects strong type properties in the Data layer. Inherit from DataSet and add the needed properties for your logic.

For example the solution for the code above:

public class EntityDataSet: DataSet{  public int EntityTableIndex {get; set;}  public DataTable EntityDataTable     {        get         {             return this.Tables[EntityTableIndex];        }    } public string EntityIdColumnName {get; set;}  }

and the function looks like this:

public void DoSomeLogic() //some input{    EntityDataSet entityDs = DAL.ReadEntities();    foreach(DataRow dr in entityDs.EntityDataTable.Rows)     {        doSomeLogic(dr[entityDs.EntityIdColumnName]);    }    //more code}

Summery

To conclude using layered application pattern can be very profitable. Ask yourself is you application architecture is really layered application, or just layered application wannabe. Using three separate projects is just not enough, make sure that changing the data base schema will cause only a change in the Data access layer and changing the business logic will need only modification in your business logic layer.

Cheers Offir :-)

T3 Service Studio - The ultimate tool for testing your web services and WCF services

Offir Shvartz — Mon, 10 Nov 2008 18:39:00 GMT

During my work I developed many web services and now WCF services, I encountered a very useful tool called Web Service studio which allows invoking web method without writing any code - gives the ability to test web service without any client code.

The tool was great but far from perfect: I encountered some bugs, the UI was in my opinion not as clear and comfortable as it could be and the main problem was not supporting WCF.

So me and my friends: Ami Yolovich and Anat Katz started developing in our spare time our own web service studio - we called him T3 Service Studio.

In the past few weeks this tool become very useful for us (using it for WCF testing) so I decided to upload it to the web, it can be useful to many others and I'll be glad to get your comments and bug reports.

So here it is:

Partial feature list:

WCF support - include TCP binding.
Multi sessions UI.
Copy paste of requests object.
DataSet viewer.
XmlNode viewer.
Much more...

Requirements

Microsoft .NET Framework 3.5 Service Pack 1 installed.

Download T3 Service Studio

Cheers Offir

Using Generic Parameter as an Attribute constructor parameter

Offir Shvartz — Fri, 31 Oct 2008 17:44:56 GMT

Hi all, it's been long while since the last time I wrote (I just wasn't in the mood....). recently I encountered something interesting in my opinion related to generic so I wanted to share it and hear your comments.

It all started with a simple task: my task was to implement generic custom control (Win Forms) that contains PropertyGrid and sets the selected object property to given object. the additional requirement was that in case the object has default constructor allowing creating new instance on the given object.

the control implementation is very simple (my example is in form only for testing purpose):

public partial class Form1 : Form{    private System.Windows.Forms.PropertyGrid propertyGrid1;    private object m_obj;    public Form1(object obj)    {        InitializeComponent();        m_obj = obj;     }     private void Form1_Load(object sender, EventArgs e)    {                  propertyGrid1.SelectedObject = m_obj;    }}

In order to implement the instance creation part I implemented before my own UITypeEditor, I called it GenericTypeUIEditor and by adding the EditAttribute to the selected field the property grid object apply my editor to the object (read about it Walkthrough: Implementing a UI Type Editor).

My GenericTypeUIEditor implementation was very simple just calling the Activator.CreateInstance<T>()

public class GenericTypeUIEditor<T> : UITypeEditor where T:new(){      public override UITypeEditorEditStyle GetEditStyle(System.ComponentModel.ITypeDescriptorContext context)    {        return UITypeEditorEditStyle.Modal;    }     public override object EditValue(System.ComponentModel.ITypeDescriptorContext context, IServiceProvider provider, object value)    {        if (value == null)        {            return Activator.CreateInstance<T>();        }        else            return null;    }}

I wrote this UITypeEditor a while ago and it works perfectly for many object types just by adding to and property the attribute:

[Editor(typeof(GenericTypeUIEditor<MyClass>), typeof(UITypeEditor))]public MyClass MyClass {get;set;}

I wanted a control that supports all types (that implements default constructor) so I wrote generic value wrapper to hold the object and added my generic attribute.

public class ValueWrapper<T> where T:new(){    public ValueWrapper(T val)    {        Value = val;    }    [TypeConverter(typeof(ExpandableObjectConverter))]    [Editor(typeof(GenericTypeUIEditor<T>), typeof(UITypeEditor))]    public T Value { get; set; }}

So the form constructor parameter gets the wrapper object, you can create new instance of this generic type in runtime, in this stage I thought I was done and the code is very clear but then...

error CS0416: 'PropertyGridTester.GenericTypeUIEditor<T>': an attribute argument cannot use type parameters.

I was really surprised because I didn't understand why?

I looked around, and I knew that generic type is resolved during runtime (read about it in the next link:Generics in the Runtime), but what about attribute or in my case the parameter of attribute. during my search I discovered that Attributes are not allowed to be generic types. The only reason I can think of is that a generic type parameter is resolved at runtime. However, an attribute parameter must be resolved at compile time. Therefore, you cannot use a generic type parameter as an argument to an attribute.

So still I needed to solve my problem, well my workaround was to modified my UIEditor: removing the Generics type and using the ITypeDescriptorContext parameter in the method:

public override object EditValue(System.ComponentModel.ITypeDescriptorContext context, IServiceProvider provider, object value)

By using the Instance property retrieving the generic type:

context.Instance.GetType().GetGenericArguments()[0] - the first generic type of the context instance

here is the new code:

public class GenericTypeUIEditor : UITypeEditor{    public override UITypeEditorEditStyle GetEditStyle(System.ComponentModel.ITypeDescriptorContext context)    {        return UITypeEditorEditStyle.Modal;    }     public override object EditValue(System.ComponentModel.ITypeDescriptorContext context, IServiceProvider provider, object value)    {        if (value == null)        {            Type valType = context.Instance.GetType().GetGenericArguments()[0];            return Activator.CreateInstance(valType);        }        else            return null;    }}

and removed the <T> from the attribute parameter and it works perfectly.

So to conclude attributes cannot be generics (maybe in .NET framework 4.0) because they are resolved in compilation time and generic types during runtime.

That's all for now, Offir.

Offir Shvartz

AppFabric Caching Service Using SSL

Azure Table Storage – limitations and solutions/workarounds(Part 2)

Azure Table Storage – limitations and solutions/workarounds(Part 1)

TPL Data Flow Debugger Visualizer

WCF 4.5 Configuration from external source

Windows Build- my first impressions

WCF Performance Using Datasets – Part 2

WCF Performance Using Datasets – Part 1

Publish/Subscriber Using Routing Service (WCF4)

EF4 In real life and using Code Only – Yes We Can ?

Self tracking “Poco’s” (self tracking data transfer objects)

Passing Event Handlers Over WCF

The Solution

Layered Application wannabe

T3 Service Studio - The ultimate tool for testing your web services and WCF services

Requirements

Microsoft .NET Framework 3.5 Service Pack 1 installed.

Using Generic Parameter as an Attribute constructor parameter