January 2008 - Posts
A client had the need to use BizTalk to send files to an FTP server via a proxy server. There is no place to specify a proxy in the FTP send adapter configuration and there is nothing about how to do it in the BizTalk documentation nor my search on the web come out with any info on that.
After playing with different configuration variations it was found that the following setup will work:
- In the 'Server' field put the IP of the proxy server
- In the 'User Name' put EndTargetServerIP@UserName.
I want to thank Shuki Licht who is the one who did this research and provided the solution.
Recently I have started creating item templates for visual studio. The first was for RfidEventHandler and there are more on the way. In this post I want to present why I think developers and architects should consider the use of item templates more often.
I use quite a few project templates, most of which I found on the web. Item templates, on the other hand, are not so common and so far, I haven't encountered any I can use. I think this feature is under used. It is overshadowed by its bigger and more popular brother: the project template and for really no reason. After getting acquainted with item templates, I can say that more than once I realized the project templates I find would have served me much better if they were implemented as item templates.
My main use for VS templates is when I need to write components. I work a lot with BizTalk and there are quite a few extension components you can write for it. They need to implement various interfaces and it is convenient to get a class with the stub code for all the function already generated for you. When I use a project template to create a component I get a separate project for each component. I usually prefer to put all my components in the same DLL. Many of the components are a single file and there is no reason to have a few projects with one file each.
Item templates solve my problem. I create an empty project with a properly general name like RfidEventHandlers and then start adding components to it using the component template. Except creating the project, an item template works the same way as a project template would:
- It can add to the project more than one file if the component needs this.
- The template can specify which references to other projects\DLLs the component needs. The references will be added if they are not already in the project.
- It does text replacements the same way as the project template etc.
One thing I always find problematic in project templates is that the project name and the namespace given to the classes are never what I need them to be. Project names should represent the content of the project. I prefer to give the project a short name. The namespace, on the other hand, should embody the project name but also represent its place in the overall hierarchy of code. For example MyCompany.MySystem.DataObjects). By default the namespace is created the same as the project name so I always need to set it manually in the project properties after it is created. The project name and namespace created by a template are either a fixed string or based on the given project name. In case of a project for a component this for sure will not be the name I wanted. It is easier to create the project with the name I want and set its default namespace before I start adding classes to it then to change this in a project that was created by the template. In item templates I just specify that the generated classes will get the default namespace of the project and when I add then I know they will get the namespace I wanted.
Don't get me wrong, project templates are a good thing and should be used. They are the right choice if your project needs artifacts which are relevant to the entire project. Things like configuration, setups and tests. I recommend, as I mentioned before, to create a combination of a project template for the sets up a project for components of a specific type and an item template to add components of that type.
Creating item templates is very easy and I really recommend that you to consider creating some also for your own classes. For example I found it helps to enforce the use of an infrastructure layer. My generated classes include code that initializes the infrastructure classes and contains code samples with explanations of how it should be used (in comments). In some cases the generated class will come with utility functions like a log function that add internally the class name to each log line.
Another use is to help maintain design patterns. Many of the patterns require a set of classes that will inherit from a common interface. New classes might be added by developers, sometime even a long time after the design was done and by developers that knows nothing about the architecture. To write such a class, a developer needs to look into the documentation to find which interfaces the component should inherit from and find what each of the functions in it should do. Item templates enable a developer who knows nothing about how to create the class, to do so in a few clicks. All that is needed is to know the class type. The template will generate the class for the developer with the right inheritance and stubs for the functions with default implementations. Most importantly- there will be explanations inside the code about what each function should do. This is my favorite way of learning- I think a sample with explanations in the code is the best kind of documentation.
When I first started to learn how to create an item template for Visual Studio I found this blog post by Jason Kemp very helpfully and obviously had a look at the MSDN for all the small details.
Next thing I did was to look for samples. First I searched my computer and what I found was where VS keeps the templates for the standard items that comes with the installation (class, config, etc). This is the best resource since I can pick the item most similar to the one I want to create and look how it is implemented. These templates are placed for VS2008 in the folder: 'C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplates'. If you use VS2005, the VS folder name is 'Microsoft Visual Studio 8'. You can find the unzipped versions of the templates in 'C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplatesCache'.
I'll be happy for more thoughts and comments on the subject if you have any.
I have been working recently with Microsoft RFID and been writing various event handlers. I'm using Damir Dobric event handler project template to create my projects.
When I want to have a few handlers in the same project currently created a copy of the existing handler file and then edit it to become the new handler. After a few time I have decided to add a Visual Studio item template for adding event handler classes to an existing project. I have another post about my thoughts when to use Visual Studio item templates which you can find here.
All you have to do to work with it is copy the zip file from here and put it as is (meaning left zipped) in the folder ...\My Documents\Visual Studio 2008\Templates\ItemTemplates\Visual C#. If you use VS 2005 this should work as well. The path is the same just with Visual Studio 2005 (if you still use VS2003 its time to upgrade yourself).
I found that creating item templates is very easy and I plan to do a few more (for example for BizTalk pipeline components) and will post them as here as well.
When backing up the BizTalk databases you need to use BizTalk backup job for that since creating a regular maintenance task is not enough. The reason is that the BizTalk mechanism ensures that all the DBs BizTalk needs are backed up in a way that can be later restored to a synchronized point, this is done by entering a transaction marker to all the DBs before the backup. This marker will appear in their log. When performing the restore action, all the DBs should be restored up to that marker.
The steps for performing a restore of a DB to a marker are:
- Restore the last full backup.
- Restore all the log backups since the full backup except the last.
- Restore the last log backup up to the marker
If you have applicative DBs that you need to be synchronized with BizTalk it is possible to add them to use the same mechanism so they will be backed up by the BizTalk backup job (see instruction how to here).
When performing a disaster recovery restore you need to perform the restore steps to all the DBs so they will all be restored to the same point and be synchronized, otherwise there is a risk BizTalk will fail to operate correctly.
There are detailed instructions how to perform such a restore in the BizTalk documentation but this is not an easy task. For example a common configuration of the backup job is to run a full backup once a week and do a log backup a few times a day. This means that in the case of a disaster you might end up having to restore over 50 files which is a lot. It is important you restore them in the right order and you have to figure out which files need to be fully restored and which restored just up to the marker.
I doubt if during a crisis anybody can remembers this complex logic or will have the time to read the documentation to learn how to do it properly. Also if there is somebody who is an expert with the procedure there are so many steps to perform that if done manually the chances for a mistake are too high.
This is why I decided to generate the restore script automatically as part of the backup process. I added a step to the backup job that creates the SQL script for restoring the DBs. This way each backup run creates the SQL script needed for restoring the DBs that were backed up and to that specific point in time (= the marker point created on that run).
To write the SQL to a file I used the job step 'output' option. The step has an SQL which returns the lines needed for the restore. I do this by creating a temp table with one column and insert into it the SQL lines I need. I then perform a select on that table which is outputted by the step to a file.
To add the step perform the following:
- Add a new step to the BizTalk backup job (Backup BizTalk Server...) and name it "Create Restore Script"
- Set the step type to T-SQL
- Set the Database to your BizTalk management DB (default is BizTalkMgmtDb)
- Enter as the SQL command this script.
- Switch to the advanced tab and set in the output file where you want the script to be written. I recommend it will be placed in the same directory as the log backup files.
- Check the "Append output to existing file". This will make sure you don't override the file and so keep also the script of the previous runs so it will be possible to restore your environment back also to them.
- Go to step 3 (the last in the original job) and change in the advance tab so on success it will continue to the next step.
I have also prepared a script which adds the step to the job. Before you run it you should do a 'replace text' in the file to all the places where written %RestoreDataDir% with the directory you want your script file to be written to (for example D:\DBRestoreData). You can also do that after you run the script by opening the new step configuration and changing the output file location in the advanced tab configuration.
I don't wish anyone to ever have to use the generated restore script but if the need comes I hope this will help restoring BizTalk a bit easier.
In BizTalk we often cache data in memory to improve performance, for example conversion tables or configuration data. Cached data is usually kept in static variables. When a DLL is released from memory, all of his static members are released as well. When we write our application, DLLs are not unloaded until the application is shutdown, unless we specify this explicitly.
BizTalk implements an internal logic of unloading DLLs from its memory. By default it unloads DLLs that have not been active for 30 minutes. If there is no instance of any of the orchestration types in a DLL, it will be unloaded already after 20 minutes. This mechanism also releases the regular NET DLLs that are referenced by the BizTalk DLLs.
The rational is that we do not want to keep in memory DLLs we do not need, as it would be a waste of memory resources. The problem is that it means that when we create caches BizTalk default setup unloads them if they have not been used for over half an hour.
We encountered a few cases of clients that complained about slow response from a service on the first call in the morning. This is because the first call included the loading time of the cache that was released during the night, when there was no activity. Another case was with a service that was called only a few times a day. We needed to improve its latency, and tried to do so by caching some conversion data in order to save roundtrips to the DB. We found that we increased the response time, as it turned out that each service call was loading an entire table and building the cache hashtable.
BizTalk enables us to control this mechanism. In the documentation there is a “hidden” page that describes various
sections you can add to the BizTalk config file, one of which tells BizTalk how the unload logic should be for specific DLLs.
In this section you define AppDomains and which DLLs belong to them. Then you instruct each AppDomain how many seconds to wait before it should unload DLLs. If you specify "-1", the DLLs will never be unloaded. It is possible to specify which DLLs belong to an AppDomain by explicitly using a DLL full name, or through regular expression filters to match DLLs name patterns.
When writing a cache we plan to control when the data is loaded, and more importantly when it is released. If we are not aware of the automatic unloading mechanism of BizTalk, our caches will not behave as we planned. We need to add a configuration section and specify that the DLLs that do the caching will not be unloaded.
Here is an example of what it should look like:
<?xml version="1.0" ?>
<configuration>
<configSections>
<section name="xlangs"
type="Microsoft.XLANGs.BizTalk.CrossProcess.
XmlSerializationConfigurationSectionHandler,
Microsoft.XLANGs.BizTalk.CrossProcess" />
</configSections>
<xlangs>
<Configuration>
<AppDomains AssembliesPerDomain="10">
<DefaultSpec SecondsIdleBeforeShutdown="1200"
SecondsEmptyBeforeShutdown="1800"/>
<AppDomainSpecs>
<AppDomainSpec Name="MyCachingDomain"
SecondsIdleBeforeShutdown="-1"
SecondsEmptyBeforeShutdown="-1"/>
</AppDomainSpecs>
<ExactAssignmentRules>
<ExactAssignmentRule AssemblyName="MyCache1, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=9c7731c5584592ad"
AppDomainName="MyCachingDomain="/>
</ExactAssignmentRules>
<PatternAssignmentRules>
<PatternAssignmentRule AssemblyNamePattern="MyCache*"
AppDomainName="MyCachingDomain" />
</PatternAssignmentRules>
</AppDomains>
</Configuration>
</xlangs>
</configuration>
You can download the sample above from here BizTalk DLL Unloading.zip. The zip contains also a more detailed example with all the config options including explanations about each field can be downloaded here This is a corrected version I made of the sample in the BizTalk documentation which is erroneous. The source there includes garbage text that makes the config XML invalid. Also, in my version I simplified the explanations and removed parts that are not related to the unload logic. The parts that have been removed are additional configurations of the AppDomain which can also be very useful but are out of scope for this post.