I’ve been reading about StreamInsight lately, there’s a lot of buzz around it nowadays.
StreamInsight is mainly a data processing engine which has great performance and latency achievements, as well as massive throughput handling capabilities.
It generally targets systems which need to handle events in massive concurrency or in cases where short latency is needed for resolving the information results.
Let’s look at High Level Architecture -
The architecture resembles BizTalk to say the truth, and is common for such processing engines.
The main terms that you need to be familiar with -
- Event – An event can be any .NET class which represents an occurrence that should be processed by the processing engine. An event can have different shapes to describe the specific scenario.
- Event Sources – Any given source for inputting events into the processing engine. You build the input adapters yourself so you can implement whatever you wish to do with any given source.
For example, you may have a source which publishes events via WCF Service, Push / Polling style, Feeds, Database, etc. - Input Adapters – You implement input adapters as you need. The role of the input adapter is to get events from sources and push it over to the processing engine.
- Streams – The input adapters create an event stream that you can query against to extract any desired information.
- Complex Event Processing (CEP) Engine – The processing engine gets events from the input adapters, and handles the events altogether.
- Query – You create and define queries over input streams to retrieve any information you like.
- Output Adapters – Once you define the input streams and the desired query, you need to do something with the results. The processing engine distributes the proper results to your predefined output adapters where you can do anything you like.
- Event Targets – The target components to which you want to send the results to. For example, database, SharePoint, applications, logging, etc.
To get a notion of the order of things, let’s look at the following scenario -
You have a system which monitors a very busy high-way road and publishes events when every car passes by a certain point.
Let’s assume a new requirement is being presented - write an analytic processing system on top of that.
For example, let’s assume we need to extract the daily average amount of black cars that drive through the road.
This is a classic example for using StreamInsight.
Our first step would be to create the input adapter to get the events when a car passes by and push it over to the processing engine.
Then, we will define the query that interests us. You do that using LINQ. In this example I would query all the black cars that passed on a daily basis and get the average amount.
Afterwards, I will connect everything to my output adapter which persists the processed results into my database.
Finally, I can view the results on any given application since I have it already persisted.
General Pros -
- Great Performance and Scale-Up – This product is written in native C++ and does great work regarding performance and resource allocations. Furthermore, it knows how to scale-up. It uses more resources if available and take advantage of multi-core environments.
- Good architecture – Such architecture which consists of Input/Output and query models enables good and clean separation of things.
- Great product for dealing with massive concurrent events or short latency requirements
General Cons -
- Customization and External Code Support – The query model generates XML configuration which the processing engine can execute in its unmanaged code environment. This means that the code is executed in their environment. Such thing does enable very good performance on the one hand, but once you need to customize it or call external infrastructures, it becomes a more difficult task, especially if you don’t wish to affect the performance to the worse.
- Scale-Out – Currently, the processing engine is designed to work in-memory in one process. This means it doesn’t scale-out by design. You could perhaps write your own solution for that, but doesn’t come out-of-the-box.
There’s much more that you can learn deeper, such as event shapes, windowing, liveliness, CTI, configuration, deployment, testing, debugging, etc.
After six months as a senior consultant and architect at Sela, I have decided to take on a new venture.
Myself and some other great guys are starting a new company, named CodeValue.
The company’s main focus is developing products for developers and ALM. We already have a product at heart, which we are really excited about, and some other very cool ideas.
Furthermore, I don’t intend to stop providing consultations and training, it’s a part of me.
As a company, the purpose is to make things simpler and better for you, the developers, and we will continue to implement projects in the software industry and help bring things to life.
I had a great time during my stay at Sela. I participated as part of the experts group which consists of highly skilled architects and consultants.
Throughout my stay, a relative short one, I had done some really nice things, such as co-authored the WCF 4.0 MOC, passed lectures, consultations, and had the chance of working in highly advanced technology development groups.
I really enjoyed working at Sela, and I’d like to thank the people for giving me the chance to be a part of it – Caro Segal, Dudu Bassa, Ishai Ram.
This doesn’t mean goodbye. The goal is to join hands and work together in the future.
In continuation of the previous posts -
WPF – Editing Mode with Save and Cancel Capability
WPF – Editing Mode with Save and Cancel Capability – Dynamic ViewModelProxy
In this post I will demonstrate an elegant and simple solution that you can use regarding the second option presented in the first post series (cloning via serialization).
The Use-Case
Often, in more advanced scenarios, you need to duplicate your object for a different view. (read the first post for more detail)
A commonly used technique is using serialization to clone the object. Unfortunately, in many times, our view models aren’t cloneable easily in their nature by using serialization, especially when we use PRISM within our view models. When using PRISM, your view models may contain references to all sorts of CAL services for example, which are usually non-serializable.
The common approach for dealing with this is using serialization events. You can set the fields not be serialized (or simply set them to null before serializing) and set their values after deserialization is complete.
You need to design your object to fit such cloning technique. For example, where would you register/instantiate your commands? constructor alone is no good due to the fact the we need to support duplication via serialization.
What I have to offer is..
- ViewModelBase – My ViewModelBase provides a method that you can override - InitializeCore(bool deserializing)
- This method occurs only once!
- If the object is being instantiated normally – part of the constructor
- If the object is being serialized – after deserialization completes
- This method provides easy access for implementing things needed either in cases of instantiation or serialization
One more thing..
I mentioned this above – in PRISM (and other cases), your view models are usually not easy to duplicate, look at the following example -
Note: The code here and in my samples are using the BinaryFormatter.
Code Snippet
public class MyViewModel : ViewModelBase
{
IEventAggregator _eventAggregator;
IUnityContainer _container;
public MyViewModel(IUnityContainer container)
{
_container = container;
_eventAggregator = _container.Resolve<IEventAggregator>();
//Use the event aggregator to subscribe to events..
}
}
Currently, MyViewModel is not serialiable.
In order to make it serializable, I need to decorate my class with the Serializable attribute, but that is not enough.
The serialization will fail because the fields _eventAggregator and _container are not serializable too.
In order to make the serialization work, here is what I need to do -
Code Snippet
[Serializable]
public class MyViewModel : ViewModelBase
{
[NonSerialized]
IEventAggregator _eventAggregator;
[NonSerialized]
IUnityContainer _container;
public MyViewModel(IUnityContainer container)
{
_container = container;
_eventAggregator = _container.Resolve<IEventAggregator>();
//Use the event aggregator to subscribe to events..
}
}
2 things have been made – 1) I set my class to be serializable 2) Marked the fields to not participate in the serialization process.
Is that all? Well, no.
The serialization and deserialization back will work fine and I will have a duplicate object. However, the state of the duplicate object would be incomplete – the _eventAggregator and _container would be nulls and no processing had been done.
Now what? well, in order to make all things work properly, I would have to encapsulate the duplication in one method (be that in my view model or somewhere else) to do the following -
Serialize the view model, call an exposed method to complete operation, return the duplicate.
For example -
Code Snippet
[Serializable]
public class MyViewModel : ViewModelBase
{
[NonSerialized]
IEventAggregator _eventAggregator;
[NonSerialized]
IUnityContainer _container;
public MyViewModel(IUnityContainer container)
{
LoadState(container);
}
public void LoadState(IUnityContainer container)
{
_container = container;
_eventAggregator = _container.Resolve<IEventAggregator>();
//Use the event aggregator to subscribe to events..
}
public MyViewModel Duplicate()
{
MyViewModel other = Helpers.CloneBySerializing(this);
other.LoadState(_container);
return other;
}
}
This is quite fine and gives us a neat solution.
My goal was to make such cases simpler though, since it is quite common in PRISM-based business applications.
Here is the solution
I built a nice wrapping on top of the binary serialization that you can use for such cases. (you can switch the actual serialization provider as you may need)
You can instruct it to serialize fields by ref.
Furthermore, it supports the entire object graph. If your view model contains other view models which have such decorated fields, it will process them as well.
This means, that upon serialization, it will set null to these fields and set their value back both in the duplicate and the original object.
So, using that, my view model looks as follows - (note the use of the InitializeCore I mentioned in the beginning of the post)
Code Snippet
[Serializable]
public class MyViewModel : ViewModelBase
{
[NonSerialized, SerializeByRef]
IEventAggregator _eventAggregator;
[NonSerialized, SerializeByRef]
IUnityContainer _container;
public MyViewModel(IUnityContainer container)
{
_container = container;
_eventAggregator = _container.Resolve<IEventAggregator>();
//need to call it once more because in the base ctor the fields are null
InitializeCore(false);
}
protected override void InitializeCore(bool deserializing)
{
if (_eventAggregator != null)
{
//Use the event aggregator to subscribe to events..
}
}
}
As you can see, I no longer need to worry about state preserving. I simply tag the fields I need to be set as references and do what I need in the InitializeCore implementation which occurs at the constructor or after deserialization when the fields already have their by-ref values.
While in such a simple case it is hard to see the benefit, when it comes to more complicated classes with deeper object graphs, it becomes much easier using this approach.
So.. how do you get the actual duplicate?
You need to call my wrapper in order to generate the duplicate. the internal implementation uses the BinaryFormatter but you can change that to NetDataContractSerializer/DataContractSerializer/XmlSerializer if you need.
Code Snippet
MyViewModel other = ObjectProducer.CloneByBinarySerialize(myViewModel);
Important Note – this serialization technique goes through the entire object graph, so it is advisable to use that in scenarios where you need to serialize a single object at a certain given time. (in this example, when the user edits an employee, the performance hit isn’t an issue for the benefit of using that)
You can download the source here - Zuker.WpfSamples.zip. (Edit State Preserve – EditWithCloneSerialize)
The sample shows a case of a really simple view model, but you can go ahead and try it with the example shown here, with the use of ‘SerializeByRef’.