One of the ways to add another security level to our application is to use SSL. SSL gives the ability to encrypt messages between two endpoints. If you will search the web you’ll probably notice that this ability is given to you [almost] out of the box.
If you are already using IIS as your web server and you want to add this security level, all you need to do is to check some check boxes in your web site configuration. In order to accomplish the configuration part you will be asked to provide a certificate.
The certificate (combination of identity and public key) is helping the client to authenticate the server. One of the common examples is your browser, when a user enters a secured web site the browser check its certificate using CA services, and verifies that the server is really who it claims to be.
There are many companies (VeriSign is one of the popular ones) which define the CA (certificate authority) and provide a certification services (it’s my recommendation for public internet applications). If you don’t want to pay money to a third party, you can create a free public certificate using OpenSSL tools, create self-singed certificate using makecert utility (you must remember to protect your private key), or even to build your own CA server (all this three are recommended for intranet applications).
Assume you choose one of the cheap ways, you still need to configure your software to validate the certificate. There are two ways to do it, the first one is using code, and the second by setting the server certificate in the client machine. In order to use a code you need to register to ServicePointManager.ServerCertificateValidationCallback event, this delegate passes the certificate details as a parameter and return whenever the certificate is valid or not. For example:
public void GetData(string url)
{
using (WebClient webClient = new WebClient())
using (StreamReader webClientStreamReader =
new StreamReader(webClient.OpenRead(url), Encoding.UTF8))
{
ServicePointManager.ServerCertificateValidationCallback +=
new RemoteCertificateValidationCallback(customXertificateValidation);
if (webClientStreamReader != null)
{
string returnedStringFromMoma = webClientStreamReader.ReadToEnd();
}
}
}
private static bool customXertificateValidation(object sender, X509Certificate cert, X509Chain chain, System.Net.Security.SslPolicyErrors error)
{
return (cert.Subject == "CN=MyCustomCert, OU=Dev, …");
}
In case you decided to set the server certificate in the client machine, you need to export the certificate from the server, and then to import the certificate into the client machine. You can do the import/export actions from the Windows Certificates manager.
I had another important issue I forget to mention, I know a lot of people which thought the using the request credential is secured, don’t let it to confuse you, the credential is just encoded it’s not encrypted.
Few months ago I heard a lecture about "Velocity", in the begging of the lecture the lecturer asked the audience the next question: "Assume that you have a client which needs to communicate with a server, what is the best communication pattern to work with, pushing or pooling?" 99% of the audience voted for the pushing. It was a tricky question with a tricky answer – it depends.
I'll explain these two methods for all of those who aren't familiar with them.
Pushing – in this concept, the responsibility to transfer the data is on the server. Each time the server has some new data to publish to the client the server will connect directly to the client and will send it all the relevant data immediately.
Pros:
- Using minimum resources to transfer a message (without extra traffic, network overload and etc)
- The data will be sent and received immediately (without any delay).
Cons:
- Scalability problem: The server needs to be aware for all clients, and which data should be sent for each client.
- Availability problem: The client has high dependency on the server. In a case the server fell down, it'll be very difficult to determining it by the client.
- Reliability and Durability problems: on the other hand, in a case that the client fell down, the server needs to implement a retries mechanism and store the message in some persistence storage, in order to insure the message will be sent successfully and completely to the client.
Pulling – in this method the responsibility for retrieving the data is on the client. The main challenge here is that the client doesn't know when he will need to connect the server in order to retrieve new data from it. In this case the client will need to use some monitoring techniques, every pre-defined interval.
Pros:
- Independency - The client is fully autonomy. Each client only needs to be aware of its server.
- Scalability: this method is fully scalable.
Cons:
- Very "expensive" (using more resources, increase the network traffic, increase the processing time on server and client sides)
- Delay time: the message won't arrived immediately, it'll be sent with a delay.
One of the solutions for this problem is to use publisher-subscriber pattern (aka "pub-sub"). In this pattern we have a third component which call the "pub-sub". Each client will subscribe to the pub-sub to get only messages which is relevant to him. The server will send all the messages to the pub-sub and will publish it to all subscribers' clients. In this pattern the server only need to send one message to one address without to be aware of all client locations.
If we'll return to the question for the beginning, next time you will need to decide which one of them to use, just think about the components architecture, analyze the communication requirements and then do the appropriate choose for your specific design.
In my pervious post I explained how to define the wcf service concurrency and instance context modes. In this post I will show you how to control these specific numbers (number of calls and number of instances). In wcf you can set these values using throttling settings. Like all other wcf configuration you can set it by using a configuration file or to implement it directly in your source code. The throttling configuration contains three elements:
1. maxConcurrentCalls [default: 16]:
Refer to the service concurrency mode. In a case we define the service as a multiple concurrency mode, it'll define the number of threads we open in the service to handle requests. But, in a case we define the service as a single concurrency mode the service will ignore this value (because we have only single thread in the service anyway).
Increase this number if your service needs to process large number of requests at the same time
2. maxConcurrentInstances [default: Int32.Max]:
Like the maxConcurrentCalls configuration it depends on other configurations.
If we define the service instance context mode to be Single the service will be ignore this value. In case the instance context mode set to per session, the real sessions number will be the minimum value between the maxConcurrencySessions and maxConcurrancyInstances. In case you define the service instance context mode as per call, it'll limit the number of active service instances. If the maximum number of instances exceeded all, the new requests will be queued.
Increase this number if you are using InstanceConcurrancyMode.PerCall or InstanceConcurrancyMode.PerSession and you have lightweight instances with heavy operations.
3. maxConcurrentSessions [default: 10]:
Refer only to a channel which supports sessions (like TCP) in all combinations of InstanceContextMode and InstanceConcurrencyMode configurations. Limits the number of active sessions in the service. Each time the client creates a new channel to the service it creates a new session in the service side (again, only for supported channels). Only when you close the channel gracefully (or after a timeout in the service side) the session will be released.
Increase this value if you have a lot of open channels to the service (and especially when some of them aren’t been closed gracefully…)
Two of the important configuration settings in WCF refer to InstanceContextMode and ConcurrencyMode. These two are very important to control the system resources in your live service. The first (InstanceContextMode) controls the number of instances for your service.
In this case you have three options:
Single - Creates only one service instance in the system.
PerSession (default value) – Create a new service instance per session.
PerCall – Creates a new service instance per call.
The second (ConcurrencyMode) controls how many threads can be run in the service at the same time.
For this configuration we have several options:
Single – Single thread, only one thread in the entire service. You are in thread safe situation; don't worry about protecting your data, synchronization and etc. But in case your service is processing a request, all other request will wait (or failed on request timeout).
Reentrant – This is a tricky one, its means that only one thread can be executed in the service but in case you call to another service the WCF infrastructure will release the lock from it (and then you should protect your service state like in the Multiple mode). You should use it to avoid cases of deadlock when two services with a single concurrency mode are calling each other.
Multiple – Multi-thread, multiple threads can be run on the same time. In most cases you will reduce yours client's timeouts. But remember, this is not a magic solution, high number of thread will cost you in a context switch time (I'll cover the "how to control the number of service active threads" issue in one of my next posts).
Everything sounds simple by now, but the combination between the InstanceContextMode and the ConcurrencyMode is the fascinating part. To clarify this issue I summarized all possible combination in the next table:
| Concurrency Mode.Single | Concurrency Mode.Reentrant | Concurrency Mode.Multiple |
| Instance Context Mode. Single | Creates only one instance from the service (singleton) and only one thread can be executed in the service at the same time. Classic case of single thread. | Creates only one instance from the service (singleton), multiple threads can be executed in callback scenarios. | Creates only one instance from the service (singleton), multiple threads can be executed in the service at the same time. Almost without a bottleneck, must handle the synchronization issues. |
| Instance Context Mode. PerSession | Each proxy gets is own instance, but because only one thread can be executed in the service at the same time all other requests will be queued. bottleneck in case of many clients with a lot of calls. | Each proxy gets is own instance, multiple threads can be executed in the service at the same time in callback scenarios, you have to handle synchronization and data safe. | Each proxy gets is own instance, multiple threads can be executed in the service at the same time. You have to handle synchronization and data safe. Overhead in serialized access to shared data. |
| Instance Context Mode. PerCall | Each call gets a new and dedicated service instance (the service ignore the Concurrency mode in this case) Overhead in instance creation for each call. No synchronization issues. |
In our application we chose WCF as a primary communication component. WCF provide several binding protocols and each one of them had its cons and pros. In our system we needed a high performance communication between several servers, it was the reason we used the WCF with NetTcpBinding. If you want to maximize your system utilization and having a better control on your services, you should make some fine tuning in the NetPctBinding. The default configurations for this binding are good for some classic scenarios, but like every other software if you need something more, you should override this default settings. It sounds like a simple task, just copy an existing binding configuration from the net and put it in your application, but actually it is very hard. The complication is caused by some serious reasons: the errors aren’t clear, the documentation is awful and some of the parameters are affected by each other and are also not documented. In my next posts I’ll write about some of this advanced configurations:
- InstanceContextMode and ConcurrencyMode
- Wcf configuration - Throttling settings
During the last few months I had some thoughts about developing a high performance and scalability systems. The idea of designing a complete system which can support almost unlimited traffic is very exciting and challenging. The main goal is to have maximum utilization of the current resources. Furthermore, is having the ability to add another box and to [almost] double the server performance. Each component should have the ability to scale and to work with several copies in different machines at the same time.
In my next posts I decided to focus on some of interesting issues from the architecture design world, like:
- Communication.
- Configuration.
- Performance
- State sharing
and more…
During the previous week Rami (my teammate) and I had an interesting problem. We had to develop two WCF services with some shared collection. One of the solutions we had was to store this collection as a static collection, and then each one of them will be able to get it. But we found a better solution. We created only one service handler class which implements the both contract interface at the same time. At the beginning we weren’t sure that this idea can be implemented, surprisingly it worked. As you probably know you can define several endpoints to the same service, each one of them with a different binding like this one:
<services>
<service
behaviorConfiguration="WcfService.Service1Behavior"
name="WcfService.ServiceHandler">
<endpoint
address="http://localhost:2010/ServiceHandler.svc"
binding="wsHttpBinding"
contract="WcfService.ICustomerService">
</endpoint>
<endpoint
address="net.tcp://localhost:2020/ServiceHandler.svc"
binding="netTcpBinding"
contract="WcfService.ICustomerService">
</endpoint>
</service>
</services>
But our question was: Can we associate more than one contract to a single service?
So we created something like this:
<services>
<service
behaviorConfiguration="WcfService.Service1Behavior"
name="WcfService.ServiceHandler">
<endpoint
address="http://localhost:1056/ServiceHandler.svc"
binding="wsHttpBinding"
contract="WcfService.IOrderService">
</endpoint>
<endpoint
address="http://localhost:1056/ServiceHandler.svc"
binding="wsHttpBinding"
contract="WcfService.ICustomerService">
</endpoint>
</service>
</services>
After few tests it worked. Now our service handler class looks like this:
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]
public class ServiceHandler : IOrderService, ICustomerService
{
…
}
We added the InstanceServiceMode service behavior attribute with the “Single” value to define only one service handler class instance for the entire service requests. Now we have one service handler class which implements more than one service contract.
According to this idea you can work with inheritance interfaces too, for example assume you have this interface:
[ServiceContract]
public interface ICRM : ICustomerService, IOrderService
{
}
You can easily define only one service and get two in the same price.
I attached a code sample of it.
In the beginning of this month I was in a business trip to Asia to train our technical team members on a new software component we developed in our R&D team. The main goal of this component is to monitor net events and map it to actions. The business user has the ability to creates flows in a web interface and to defines the relevant actions for each event. In our solution we used WCF to communicate between our services, in addition, we used FTP to transfer batch files to our backend server.
You can look on some of my pictures here.
It's very useful to take an enum and bind it into bind-able controls (like drop down list, check box, grid view and etc). For example, if we have an enum which represents bug statuses and you want to bind it into a drop down list.
public enum Bug
{
Opened = 1,
Closed = 2,
Rejected = 3,
Resolved = 4,
InProgress = 5
}
In addition, usually in a UI we want to separate between the enum words (for example to change the "InProgress" to "In Progress").
I search over the net and I found some solution but in all of them I needed to extend my current enum (like using attribute in this solution), so I decided to implement one of myself:
public static class EnumHelper
{
public static IDictionary<string, int> GetList<TEnum>()
{
return GetList<TEnum>(false);
}
public static IDictionary<string, int> GetList<TEnum>(bool splitString)
{
Type type = typeof(TEnum);
Dictionary<string, int> list = new Dictionary<string, int>();
Array enumValues = Enum.GetValues(type);
foreach (Enum value in enumValues)
{
int enumValue = Convert.ToInt32(((TEnum)Enum.Parse(type, value.ToString())));
string sb = (splitString) ? SplitString(value) : value.ToString();
list.Add(sb, enumValue);
}
return list;
}
private static string SplitString(Enum value)
{
StringBuilder sb = new StringBuilder();
char[] chars = value.ToString().ToCharArray();
sb.Append(chars[0]);
for (int i = 1; i < chars.Length - 1; i++)
{
if ((chars[i].ToString() == chars[i].ToString().ToUpper())
&& (chars[i - 1].ToString() != chars[i - 1].ToString().ToUpper()))
{
sb.Append(" ");
}
sb.Append(chars[i]);
}
sb.Append(chars[chars.Length - 1]);
return sb.ToString();
}
}
How do we use this code?
In case we just want to bind it without separate between the enum words, it will be like this:
var bugs = EnumHelper.GetList<Bug>();
And in case we want to separate between the words will have something like that:
var bugs = EnumHelper.GetList<Bug>(true);
Challenge:
How do you synchronies between enum (which execute in web/service context) and a table/database context?
For example you have this bug statuses enum and in addition you have a bug table with numeric status column. The status column values are the same enum numeric values. After deployment, in order to understand this column value you must look at the enum that represent it. But in order to do so, you must open your code…
Windows Workflow Foundation is a powerful framework to develop a workflow component in your application. This framework has two workflow types: Sequential workflow (pre-defined flow) and State Machine (event driven flow). In this post I’ll explain and show you how to define a workflow from the later kind.
Each state machine consists of two major components: state (unique configuration) and Transitions (between the states, based on events or input). Each state machine should have initialize state and completion state.
For example let see a state machine of bug(You can download the entire example from here):
After we defined the state machine diagram we can implement it easily.
The first thing we should do is to open a state machine workflow console application:
In order to separate between the workflow design and the implementation we’ll create an interface of the state machine events. In the next step we’ll associate between the interface events to the workflow events. The ExternalDataExchange is necessary for exposing this interface to the workflow state machine.
[ExternalDataExchange]
public interface IBugService
{
event EventHandler<BugEventArgs> BugOpened;
event EventHandler<BugEventArgs> BugRejected;
event EventHandler<BugEventArgs> BugSolved;
event EventHandler<BugEventArgs> BugClosed;
}
As you probably notice I used a custom EventArgs to pass some relevant arguments as parameters to the event handler. This custom EventArgs should inherit from the ExternalDataEventAargs.
[Serializable]
Public class BugEventArgs : ExternalDataEventArgs
{
public BugEventArgs(Guid instanceId, Bug bug)
:base(instanceId)
{
Bug = bug;
WaitForIdle = true;
}
public Bug Bug { get; set; }
}
A very important issue is the WaitForIdle property. You must set the value of this property to “True” (believe me it took me some time to figure it out), it tells to the workflow runtime to raise the event only when the workflow became idle. In case that this property set to “False” and you try to have quick transition between states, you probably will get this unclearly error message:
“Event "BugRejected" on interface type "Wortzel.WF.SimpleStateMachine.IBugService" for instance id "50943c87-348d-4a88-9ada-d440c4989eca" cannot be delivered.” (With inner exception : "Queue 'Message Properties Interface Type:Wortzel.WF.SimpleStateMachine.IBugService Method Name:BugRejected CorrelationValues: is not enabled.")
The service class which going to be the outer API for our state machine, will implement the IServiceBug interface. In addition, this class will contain the entire raise event methods:
[Serializable]
public class BugService:IBugService
{
public void RaiseBugOpenedEvent(Guid instanceId, Bug bug)
{
if (BugOpened != null)
{
BugOpened(null, new BugEventArgs(instanceId, bug));
}
}
public void RaiseBugRejectedEvent(Guid instanceId, Bug bug)
{
if (BugRejected != null)
{
BugRejected(null, new BugEventArgs(instanceId, bug));
}
}
public void RaiseBugSolvedEvent(Guid instanceId, Bug bug)
{
if (BugSolved != null)
{
BugSolved(null, new BugEventArgs(instanceId, bug));
}
}
public void RaiseBugClosedEvent(Guid instanceId, Bug bug)
{
if (BugClosed != null)
{
BugClosed(null, new BugEventArgs(instanceId, bug));
}
}
#region IBugService Members
public event EventHandler<BugEventArgs> BugOpened;
public event EventHandler<BugEventArgs> BugRejected;
public event EventHandler<BugEventArgs> BugSolved;
public event EventHandler<BugEventArgs> BugClosed;
#endregion
}
And now we’ll create the workflow itself. We’ll create an event-driven state machine, each event represents a single transition in our state machine. One of the drawbacks in this model is in case we want to share a single event across multiple states. For example, if we have 2 states (BugSlovedState and BugRejectedState state) which both of them need to define a transition to the BugClosedState state we must define two different events (OnBugClosed1 and OnBugClosed2). I understand the needs to define two events but it’s still annoying…
Each state has two activities: HandleExternalEventActivity and SetStateActivity. The former associates the state machine and the event (in the IBugService interface) and the later set the next state after this event fired.
The last thing we have to do is to activate this state machine. In order to do it we need to write some code which creates the workflow runtime, associates the BugService with our workflow and then creates a workflow instance and starts it.
static void Main(string[] args)
{
using(WorkflowRuntime workflowRuntime = new WorkflowRuntime())
{
ExternalDataExchangeService dataExchangeService = new
ExternalDataExchangeService();
workflowRuntime.AddService(dataExchangeService);
BugService bugService = new BugService();
dataExchangeService.AddService(bugService);
WorkflowInstance instance =
workflowRuntime.CreateWorkflow(typeof(BugWorkflow));
instance.Start();
Bug bug = new Bug() { BugId = 1, AssignTo = "Someone" };
bugService.RaiseBugOpenedEvent(instance.InstanceId, bug);
bugService.RaiseBugRejectedEvent(instance.InstanceId, bug);
bugService.RaiseBugClosedEvent(instance.InstanceId, bug);
}
}
And that it all, we created a state machine workflow. If you want, you can add a tracking service (SqlTrackingQuery) which can help you to determine the entire flow for a specific state machine instance.
When I used it I had a wired behavior: when I printed the current state to the console after I changed several states one after another, I got an inconsistent data (the state name doesn’t always update). My guess that is caused due to racing problem in the state machine engine (only when I gave it much time between the states transition it shows me the right data). Is it bug or feature?
Few days ago we faced an interesting challenge, we needed to implement a component which can move some code execution from a web context to a different context (like a service host in a windows service).
The request first came up when we needed to perform a heavy processing work which started in a web context. Furthermore, we had some additional constraints like: This process shouldn’t use the same web application thread pool and we needed it will execute in a high isolation level. We chose to implement it with anonymous delegate and serialization. When we want to perform an operation in a web context, we’ll send our request to a lower layer which serializes the request in web context and store it in some kind of storage (database or file). Then in a “batch processing” context (which execute asynchronously) we’ll retrieve the request, deserialize and execute it.
You can download the entire example from here.
(1) We prefer to hide this logic from the web layer, therefore, in web context we should call to a lower layer in my example the web page (CustomerPage.aspx) will call to CustomerBL:
protected void CalculateOrders_Click(object sender, EventArgs e)
{
CustomerBL customerBL = new CustomerBL();
customerBL.DoSomeOperation();
}
(2) In my CustomerBL we’ll use a generic utility to request a batch processing. The API is using an anonymous delegate (in this example we’ll use the Action<> method) :
protected void CalculateOrders_Click(object sender, EventArgs e)
{
CustomerBL customerBL = new CustomerBL();
customerBL.DoSomeOperation();
}
(3) In RequestProcess method we’ll serialize the request and store it into some storage (in our example it’ll be a file). The FileStorage class is just implementing the storing/retrieving operations from a file. This was the last operation in web context.
protected void CalculateOrders_Click(object sender, EventArgs e)
{
CustomerBL customerBL = new CustomerBL();
customerBL.DoSomeOperation();
}
(4) Now we’ll see the asynchronous process. In our batch service we’ll retrieve the request from the file, deserialize it. Our last step is to use some reflection magic to restore the original anonymous delegate and to invoke it.
public class BatchProcess
{
public void StartProcessing()
{
string serializationResult =
FileStorage.GetInstance().ReadAllText();
object obj = Serializer.Deserialize(serializationResult);
InvokeProcess(obj);
}
private void InvokeProcess(object deserializeObject)
{
Type parameterType =
Type.GetType(deserializeObject.GetType()
.GetGenericArguments()[0].AssemblyQualifiedName);
Type funcType = typeof(Action<>)
.MakeGenericType(new Type[]
{ parameterType });
// Create the original Action<T> delegate
// the first parameter is the generic parameter
// the second parameter is the Target
Delegate func = Delegate.CreateDelegate(funcType,
deserializeObject.GetType()
.GetProperties()[1].GetValue(deserializeObject,
null), deserializeObject.GetType()
.GetProperties()[0]
.GetValue(deserializeObject, null) as MethodInfo);
object proxy = Activator.CreateInstance(parameterType);
// Invoke the anonymous delegate
func.DynamicInvoke(proxy);
}
}
The serialization of Func<> is very similar to its relative Action<> you just need to create appropriate Generic Func<,> with the relevant return type.
For example, if we have this anonymous delegate: Func<HeavyProcessBL, bool>, we should retrieve this two parameters and to create the original Func<> like in the example below:
Type funcType = typeof(Func<,>).MakeGenericType(new Type[] { parameterType, returnType});
Summary:
We saw how we can serialize a simple anonymous delegate (Action<>), deserialize and invoke it. It’s important to know that the anonymous delegate serialization depends on class create the anonymous delegate (CustomerBL in our example). Therefore, both of the sides (web & batch process) must have a reference to it. In order to pass some parameters they should be serializable.
Did you ever need to do some string manipulation? For example, in a case you have a collection of string and you want to display it nicely in GUI with a separator? Or, if you want to check if a string contains a value? Or just to build your SQL statement in a case you need to create an "IN" clause and you have a collection as a parameter?
String class gives us some powerful helper methods which help us to manipulate our strings. I'm really sure that you already familiar with some of them (like string.Format) and in this post I would like to focus on two major method that maybe help you in your daily work.
(1) string.Join:
In case you want to join a list of strings to one string with a separator, you probably use it.
For example:
In case we have this list
//Join array of strings into one string with seperator
List<string> list = new List<string>() { "a", "b", "c" };
And we want to create this string: "a, b, c", we have two ways:
// (1) The hard way
StringBuilder sb = new StringBuilder();
for (int i = 0; i < list.Count(); i++)
{
sb.Append(list[i]);
if (i != list.Count() - 1)
{
sb.Append(",");
}
}
// (2) The easy way
string entireStrings = string.Join(",", list.ToArray());
// (3) And another example in case you have a numeric collection
List<int> numericList = new List<int>() {1, 2, 3};
string entireStrings = string.Join(",",
numericList.Cast<string>().ToArray());
(2) string.IsNullOrEmpty(someString):
We always want to check if a string is equals to null or if string is empty. We have some techniques to implement it like:
bool isValid = line != null && line.Length != 0;
But the most short way is to use string.IsNullOrEmpty(…) method.
For example:
bool isValid = string.IsNullOrEmpty(line);
As I already mentioned in one of my previous post, the SqlBulkCopy is a powerful tool which gives us an option to perform insertion for a large amount of data. The evolution of the ADO.NET creates us the Linq to Sql, as a great O/R Mapping framework from Microsoft kitchen. One of the missing features in Linq to Sql is the ability to use some bulk insert capabilities. In order to improve our Linq to Sql infrastructure I decided to implement the bulk insert as part of the Table class, and I will show you how I implemented it.
The SqlBulkCopy has various input types, I decided to use the IDataReader as my data source when using the WriteToServer method. The minimum implementation of the IDataReader interface as part of the SqlBulkCopy, requires to override three method: Read, GetValue and fieldCount methods.
public abstract class SqlBulkCopyReader : IDataReader { // derived must implement only these three public abstract bool Read(); public abstract object GetValue(int i); public abstract int FieldCount { get; } ... }
Next, I built our LinqBulkCopyReader as a generic class which inherits from the SqlBulkCopyReader class. In this class I also added a constraint that the generic type should be a class (like all other Linq to Sql entities). In our class constructor I retrieved all class properties and for each one of them I checked if this property has a changeable attribute column (column which isn't automatic generates/updates by the database). I also created a sub-class which called ColumnMapping that helps us to store the association between the class properties and the table columns. For each one of the column I stored the column id, column name and a function which gets the appropriate value from the enumerator.
public class LinqBulkCopyReader<TEntity> : SqlBulkCopyReader
where TEntity : class
{
public LinqBulkCopyReader(IEnumerable<TEntity> entities)
{
_Enumerator = entities.GetEnumerator();
_ColumnMappingList = new List<ColumnMapping>();
var entityType = entities.GetType()
.GetInterface("IEnumerable`1")
.GetGenericArguments()[0];
_TableName = (entityType.GetCustomAttributes(
typeof(TableAttribute), false) as TableAttribute[])[0].Name;
var properties = entityType.GetProperties();
for (int index = 0; index < properties.Length; index++)
{
var property = properties[index];
var columns = property.GetCustomAttributes(typeof(ColumnAttribute), false)
as ColumnAttribute[];
foreach (var column in columns)
{
//Check if this property repserent a updatable column
if ((!column.DbType.Contains("IDENTITY")) &&
(!column.IsDbGenerated) &&
(!column.IsVersion))
{
_ColumnMappingList.Add(new ColumnMapping()
{
ColumnIndex = index,
ColumnName = column.Name ?? property.Name,
ColumnGetter = row => property.GetValue(row, null)
});
}
}
}
}
private readonly IEnumerator<TEntity> _Enumerator;
private readonly IList<ColumnMapping> _ColumnMappingList;
private readonly string _TableName;
public string TableName
{
get { return _TableName;}
}
public IDictionary<string, int> ColumnMappingList
{
get
{
return _ColumnMappingList.Select(column =>
new {column.ColumnName, column.ColumnIndex})
.ToDictionary(x => x.ColumnName, y => y.ColumnIndex);
}
}
private class ColumnMapping
{
public int ColumnIndex { get; set; }
public string ColumnName { get; set; }
public Func<object, object> ColumnGetter { get; set; }
}
…
}
I implemented the IDataReader three requires methods (Read, GetValue and FieldCount). Furthermore, I decided to implement the GetOrdinal method too, in order to give another overload for mapping between the entity properties and the table columns.
public class LinqBulkCopyReader<TEntity> : SqlBulkCopyReader where TEntity : class { …
public override bool Read() { return _Enumerator.MoveNext(); } public override object GetValue(int i) { return _ColumnMappingList.Where(column => column.ColumnIndex == i) .Single() .ColumnGetter(_Enumerator.Current); } public override int FieldCount { get { return _ColumnMappingList.Count; } }
public override int GetOrdinal(string name) { return _ColumnMappingList .Where(column => column.ColumnName == name) .Single() .ColumnIndex; } }
The next step was to use this code inside the data context. I used the extension method feature in order to extend the current Table class, and to provide a standard Linq to Sql API to execute it (like all other data modification methods: insert, delete and etc). For example, in order to insert a new customer your probably write something like this:
Customer customer = new Customer()
{
FirstName ="Avi",
LastName ="Wortzel"
};
LinqDB db = new LinqDB();
db.Customers.InsertOnSubmit(customer);
db.SubmitChanges();
My motivation is to create a similar API to insert a collection of customer in almost the same way:
List<Customer> customers = new List<Customer>()
{
new Customer()
{
FirstName ="Avi",
LastName ="Wortzel"
},
new Customer()
{
FirstName ="Yuval",
LastName ="Wortzel"
}
};
LinqDB db = new LinqDB();
db.Customers.BulkCopy(customers);
Our extension method will be defined on the generic table class and will get the collection of customer as a parameter. All we need to do is to initiate the LinqBulkCopyReader and then to iterate on the column mapping list and to synchronize the column mapping collection in the SqlBulkCopy class. The last part of this code will be to set the database working table and to call to call to WriteToServer method which will perform the bulk insert:
public static class LinqDBExtension
{
public static void BulkCopy<TEntity>(this Table<TEntity> table,
IEnumerable<TEntity> entities)
where TEntity:class
{
SqlBulkCopy bulk = new SqlBulkCopy(ConfigurationManager.ConnectionStrings
["LinqDbConnectionString"].ConnectionString);
LinqBulkCopyReader<TEntity> reader = new LinqBulkCopyReader<TEntity>(entities);
foreach (var column in reader.ColumnMappingList)
{
bulk.ColumnMappings.Add(column.Key, column.Value);
}
bulk.DestinationTableName = reader.TableName;
bulk.WriteToServer(reader);
}
}
Summary:
In order to improve our Linq to Sql framework, we added a significant functionality for bulk insert which works side by side with the DataContext. It gives us the opportunity to use the Linq to Sql without losing a great feature which helps us to improve our system performance.
Update: I added the entire source code in the attachment.
As you probably know Microsoft Israel gave each of the Tech-Ed bloggers an option to ask a question about Tech-Ed Israel, and the price for answering the right answer is a Pizza from Pizza Hut Israel.
So, my question is:
Name at list two speakers who used musical instrument during their lectures, and which instruments they used?
More than a half of Tech-Ed convention is over, and I would like to share with you some of my favorite lectures.
The first lecture was about “SOAP/WS-* and REST: Complementary Communication Styles” by David Chappell. At the beginning of the lecture, David told us that in the past he really didn’t understand the necessity of REST in the new SOAP/WS-* world, furthermore, he didn’t understand how REST creates an entire community around it. After a while, he started to understand the glory around the REST concept. In this lecture David share his thoughts and his conclusions about the different between these two approaches. In addition, he shows us when we should prefer REST against SOAP/WS-*. It was very interesting lecture with a great lecturer.
Photo: Shani Raba
The second lecture was about “Consuming and Creating RESTful Web Services with .NET” by Ron Jacobs. It was a continue lecture for David lecture. Ron wasn’t a WCF/SOAP/WS-*/REST man in his origin, but he really got deep into it. If David lecture was about the theory, Ron’s lecture was about the how to implement it. Ron showed us how to make it work. During the presentation he showed how to take a new service which configured as regular SOAP WCF service, and to translate it to WCF service that support REST style. He told us about real problems which he dealt with in his first try to create a WCF with REST approach. Ron suggested some good ideas to solve them and also mentioned that a lot of it is not documented.
My picture with Ron Jacobs (From the right) Photo: Shani Raba
In the next post I’ll talk about some more good lectures that I really enjoyed them.
More Posts
Next page »