January 2011 - Posts
As you all know windows event log is one of the most common infrastructure for writing logs in windows applications.
When migrating an existing application that happen to use event log we run into the following problem: Application running on windows Azure does not have the permissions to create a new event log source.
You can only use an existing event log source, any attempt to create a new one will cause a security exception. Even at full trust, some operations are not allowed.
An alternative to using the default Azure logger is to put in any file based trace listener you want to use and then persist that file using WAD's directory data sources. Optionally, there is also a AppFabric service bus based trace listener which can broadcast events to any listener. However, make sure you understand the costs associated with that approach should you decide to pursue it.
A nice solution is to create a startup task running an executable with privileged permissions that will create the event log source (using System.Diagnostics code)
I hope this will be simplified in the future.
Manu
Dependency Injection and IOC are very common patterns in modern applications.Such patterns are based on dynamically loading an assembly on runtime usually form the file system. When running in the cloud instead of using the file system we will use blob storage.
In this post I want to show how to load an assembly from blob storage.
Let us say we have a calculator (MyCalc) in the assembly "calculator.dll" and a simple web role that wants to use it but has no direct reference to it. The web role will load the calculator dynamically from blob storage.
The first thing we do is upload the dll to a blob.
private static void CreateBlob()
{
//This demo runs on the development emulator
var cloudStorageAccount =
CloudStorageAccount.DevelopmentStorageAccount;
var blobStorage = cloudStorageAccount.CreateCloudBlobClient();
var container = blobStorage.GetContainerReference("assemblies");
container.CreateIfNotExist();
CloudBlob blob = container.GetBlobReference("calculator.dll");
blob.UploadFile("calculator.dll");
//Continue and load other required assemblies
}
static void Main(string[] args)
{
CreateBlob();
Console.WriteLine("Press enter to exit");
Console.ReadLine();
}
Now let us load the assembly using simple reflection code:
ICalc _calc;
protected void Page_Load(object sender, EventArgs e)
{
if (!Page.IsPostBack)
{
var cloudStorageAccount =
CloudStorageAccount.DevelopmentStorageAccount;
var blobStorage = cloudStorageAccount.CreateCloudBlobClient();
var container = blobStorage.GetContainerReference("assemblies");
var strm = new MemoryStream();
var blob = container.GetBlobReference("calculator.dll");
blob.DownloadToStream(strm);
byte[] asseblyBytes = strm.ToArray();
var assembly = Assembly.Load(asseblyBytes);
var calcType = assembly.GetType("Calculator.MyCalc");
_calc = Activator.CreateInstance(calcType) as ICalc;
}
}
This is simple.
In .Net 4.0 Microsoft released a great framework called MEF which is targeted for dynamic loading of types and assigning them correctly.
Let us try to use MEF in the cloud.
private Assembly LoadAssebly()
{
var cloudStorageAccount = CloudStorageAccount.DevelopmentStorageAccount;
var blobStorage = cloudStorageAccount.CreateCloudBlobClient();
var container = blobStorage.GetContainerReference("assemblies");
var strm = new MemoryStream();
var blob = container.GetBlobReference("calculator.dll");
blob.DownloadToStream(strm);
byte[] asseblyBytes = strm.ToArray();
return Assembly.Load(asseblyBytes)
}
[Import(typeof(ICalc))]
ICalc _calc;
protected void Page_Load(object sender, EventArgs e)
{
var catalog = new AssemblyCatalog(LoadAssebly());
var Container = new CompositionContainer(catalog);
Container.ComposeParts(this);
}
This case was simple because we know exactly which assembly to load. Sometimes all we know is that the assembly is located in a certain directory. How do we simulate directories in the cloud? The answer is local storage.
private string GetAssemblyPath()
{
var cloudStorageAccount = CloudStorageAccount.DevelopmentStorageAccount;
var blobStorage = cloudStorageAccount.CreateCloudBlobClient();
var container = blobStorage.GetContainerReference("assemblies");
var localResource = RoleEnvironment.GetLocalResource
("assembliesStorage");
foreach (var blobitem in container.ListBlobs())
{
var blob = container.GetBlobReference(blobitem.Uri.ToString());
string name = blobitem.Uri.LocalPath.Split('/').LastOrDefault();
blob.DownloadToFile(localResource.RootPath + name);
}
return localResource.RootPath;
}
[Import(typeof(ICalc))]
ICalc _calc;
protected void Page_Load(object sender, EventArgs e)
{
var catalog = new DirectoryCatalog(GetAssemblyPath());
var Container = new CompositionContainer(catalog);
Container.ComposeParts(this);
}
Enjoy
Manu
On my last post I wrote how to enable ping between the local machine and an Azure VM running your role.
Now it is time to enable a full Folder Share the two machines.
To enable file sharing on Azure VM, follow the steps below:
First make sure the Windows Azure Connect connection is established. e.g. make sure you can ping your Azure VM from your home machine and vice versa. (see last post)
1. Remote Desktop to your Azure VM
2. Run "wf.msc" to open the Windows Firewall console
3. Create a new inbound rule to allow file share:
a. Right click "Inbound Rules", choose "New Rule ..."
b. Select "Predefined", choose "File and Printer sharing" from the drop down list
c. Follow through the wizard, leave everything checked as they are
d. "Allow the connection" and Finish
4. Create a folder on the VM machine, e.g. d:\Share
5. Create a network share by running this command: "net share MyShare=d:\share /grant:account_on_vm,FULL"
7. Restart the SMB service (net stop lanmanserver && net start lanmanserver)
8. Now you should be able to connect to this share from the other Connect machine. e.g. Run "net use * \\RD00155D328A7A\MyShare", where RD00155D328A7A is the hostname of your Azure VM.
Enjoy
Manu
One of the first things we do after establishing an Azure Connect connection between local machines and Azure VMs is ping.
The problem is it will fail. The reason is that ICMP protocol is closed by the firewall.
To open ICMP execute on both the local machine and the Azure VM the following script:
netsh advfirewall firewall add rule name="ICMPv6" dir=in action=allow enable=yes protocol=icmpv6
To execute it on the Azure VM simply connect using Remote Desktop and execute in a command prompt.
Enjoy
Manu
To execute asymmetric cryptography it is required to have a public and a private key.
The common way to pack such keys is using an X.509 certificate.
System.Security.Cryptography.X509Certificates provides the API to open the certificate store and find a certificate.
public static X509Certificate2 FindCertificate(string certificateName,
StoreName storeName, StoreLocation storeLocation)
{
X509Store store = new X509Store(storeName, storeLocation);
X509Certificate2 certificate = null;
try
{
store.Open(OpenFlags.OpenExistingOnly);
X509Certificate2Collection collection = store.Certificates.Find(
X509FindType.FindBySubjectName, certificateName, false);
if (collection.Count > 0)
certificate = collection[0];
else
throw new Exception(
string.Format("CertificateNotFound {0}", certificateName));
}
finally
{
store.Close();
}
return certificate;
}
This code is great but there are number of scenarios it will not work (for example the user does not have the permissions to access the certificates or there private keys)
So there is a solution:
Instead of using a certificate it is possible use key strings.
The following code creates a key string of a public and private key. (this is done on the dev machine)
var cert = Encryptor.FindCertificate(certName, StoreName.My, StoreLocation.LocalMachine);
RSACryptoServiceProvider rsa = cert.PrivateKey as RSACryptoServiceProvider;
var keyString = rsa.ToXmlString(true);
This string can be written to the config file.
The following code imports the string to initialize RSACryptoServiceProvider
var rsa = new RSACryptoServiceProvider();
rsa.FromXmlString(keysXml);
Now we can execute asymmetric cryptography using the private key without the need for the certificate.
Manu
Few weeks ago a post about asymmetric encryption using RSACryptoServiceProvider. Asymmetric encryption is an expensive operation so it is not meant for very long data encryption. If you have an article or a book you should not encrypt it asymmetrically but what about a 200 byte long information? If you will try to encrypt 200 byte using RSACryptoServiceProvider you will get an exception. It seems that the longest data that can be encrypted using is RSACryptoServiceProvider is 80 bytes.
I thinks that there are many scenarios where it makes to encrypt few hundreds bytes long data asymmetrically and I was surprised to find out that the limit is so low.
The solution is parallel encryption. Simply split the long data to chunks of up to 80 bytes and encrypt them in parallel. There are no dependencies between these chunks of data so they can be parallelized very well.
Encryption:
public static byte[] EncryptAsyncPublic(byte[] data, X509Certificate2 certificate)
{
try
{
RSACryptoServiceProvider rsa =
certificate.PublicKey.Key as RSACryptoServiceProvider;
if (data.Length <= 80)
return rsa.Encrypt(data, true);
//The max encryption size is 85 so we split the data into chunk of 80 and encrypt them in parallel
List<byte[]> splits = data.Split(80);
List<byte[]> encryptedSplits = new List<byte[]>();
Parallel.For(0, splits.Count,
i => encryptedSplits.Add(rsa.Encrypt(splits[i], true)));
//Collect all the encrypted splits and create the final encrypted array
return encryptedSplits.Merge();
}
catch (Exception ex)
{
Logger.Log(ex.ToString(), EventLogEntryType.Error);
return null;
}
}
Decryption:
In the decryption method we will do exactly the same.
Asymmetric encryption using RSACryptoServiceProvider always produces 128 bytes. If the byte array to decrypt is longer than 128 bytes (256, 384 etc) it means that it is the result of parallel encryption of long data, therefor it should be decrypted in parallel.
public static byte[] DecrypyAsyncPrivate(byte[] cipher, X509Certificate2 certificate)
{
try
{
RSACryptoServiceProvider rsa =
certificate.PrivateKey as RSACryptoServiceProvider;
if (cipher.Length == 128)
return rsa.Decrypt(cipher, true);
// the encrypted data was split so we have to split
the result and decrypt in parallel
List<byte[]> splits = cipher.Split(128);
List<byte[]> decryptedSplits = new List<byte[]>();
Parallel.For(0, splits.Count,
i => decryptedSplits.Add(rsa.Decrypt(splits[i], true)));
Collect all the encrypted splits and bind them together
return decryptedSplits.Merge();
}
catch (Exception ex)
{
Logger.Log(ex.ToString(), EventLogEntryType.Error);
return null;
}
}
Enjoy
Manu
Last December I gave 12 talks about Windows Azure together with David Platt in Redmond.
All the talks were recorded and can now be download.
There are 12 chapters. Each chapter is about 45 min.
So go to the JumpStart event, watch and enjoy.
Manu
When developing any azure application the development is done against the developer emulator. Yet in the storage emulator it looks likes there are very important differences between the emulator and the real cloud.
Let us think of the classical scenario of inserting an entity to a table. Sometimes you might want to check before Inserting an entity maybe it already exist.
if (!EntityExist(entityID))
InsertEntity(newEntity)
Well in the development storage emulator if this is the first time such entity (schema) is being inserted the first lookup will fail with the error: "One of the request inputs is not valid."
It is documented that: "In development storage, querying on a property that does not exist in the table returns an error. Such a query does not return an error in the cloud."
To confirm this I craeted a table using CloudTableClient.CreateTablesFromModel, and looked in SQL Management Studio for the backing DB of the storage emulator. I saw all of the tables in the TableContainer table, however the SchemaXml column does not contain any properties, as there are currently no records.
So the solution is simple: In the development storage simply make sure you create a dummy entity from any schema before querying the table storage.
Manu
Microsoft learning had just released the Windows Azure training portal.
In the portal you will find learning content as well as a free license for 30 days.
Enjoy
Manu
Traditional symmetric cryptography is all about hiding a secret using an algorithm and a key. The same key is used for encryption and decryption.
Asymmetric encryption does much more. In Asymmetric encryption there are two key: one is kept secret (private) and the other is distributed (public). Both keys are mathematically the same – What makes the public key public is the fact that it was distributed.
To perform a full cycle both keys are required (i.e. encryption with one key and decryption with the other).
There are two possible scenarios:
- "Send a secret" : Encrypt some data with the public key and send it to the one who has the private key. Only she can decrypt the secret.
- "Proof of origin" : Encrypt some data with your private and send it to anyone who has your public. Assuming that "everyone" has your public it means that everyone can read your data. What you gained from the encryption is the fact that anyone who decrypted your data is absolutely certain that you are the source of the data because only the holder of the private key could have encrypted it.
Asymmetric encryption in .Net is extremely simple as I will demonstrate here. (I do not know why the samples published by Microsoft are so long and complicated)
To obtain public and private key you can wrap them in X.509 certificate. You can put it in a file (cer, pfx) or install it in the certificate store. Certificates can be created using Certmgr.exe , IIS manager or plenty of other tools.
It is also possible to create special xml strings from certificates that will hold the private and public keys.
The Class System.Security provides us to execute Asymmetric Cryptography is called: RSACryptoServiceProvider and it is very simple to use.
Remember: Asymmetric encryption is expensive. It cannot be used for large data! In the next post I will show how to use symmetric encryption together with Asymmetric encryption to handle large data.
public byte[] EncryptAsyncPublic(byte[] data, X509Certificate2 certificate)
{
try
{
RSACryptoServiceProvider rsa =
certificate.PublicKey.Key as RSACryptoServiceProvider;
return rsa.Encrypt(data, true);
}
catch (Exception ex)
{
...
return null;
}
}
public byte[] DecrypyAsyncPrivate(byte[] cipher,
X509Certificate2 certificate)
{
try
{
RSACryptoServiceProvider rsa =
certificate.PrivateKey as RSACryptoServiceProvider;
return rsa.Decrypt(cipher, true);
}
catch (Exception ex)
{
...
return null;
}
}
As you can see I implemented the first scenario. It happens that RSACryptoServiceProvider does not allow to execute the second scenario as is. If you will try to encrypt with the private and decrypt with the public you will get an exception! What you are trying to do is to prove to the world that you are the origin of the data. To do that RSACryptoServiceProvider provides digital signature methods.
public byte[] SignPrivate(byte[] data, X509Certificate2 certificate)
{
try
{
RSACryptoServiceProvider rsa =
certificate.PrivateKey as RSACryptoServiceProvider;
return rsa.SignData(data, new SHA1CryptoServiceProvider());
}
catch (Exception ex)
{
...
return null;
}
}
public bool VerifySinatutePublic(byte[] data,
byte[] signature, X509Certificate2 certificate)
{
if ((data == null) || (signature == null))
return false;
try
{
RSACryptoServiceProvider rsa =
certificate.PublicKey.Key as RSACryptoServiceProvider;
return rsa.VerifyData(data,
new SHA1CryptoServiceProvider(), signature);
}
catch (Exception ex)
{
...
return false;
}
}
Enjoy – Security is fun !
Manu
Windows Azure is provides a reliable service. My service and my data are replicated between fault domains and upgrade domains. Great! But what happens if the data center crashes. The data center itself is a fault domain by itself yet all my replications are hosted in the same data center! When speaking about disaster recovery plan (DRP) a scenario where a data center explodes (see Sep 11) is something we have to think about.
Well today data is not replicated across data centers but in the future the Windows Azure Storage Team promised such replication will be available. see the end of the following post.
Sql Azure is also not replicated across data centers but the solution here is simple (yet expensive). All you have to do is create another database and deploy it manually in another data center. Now use Sql Azure Data Sync to synchronize the two databases and there you go.
Windows Azure Services geo replication is not supported! this means that your service is located in a single data center. It is possible to manually bypass this by deploying the service on two data centers and use WCF routing or other routing service to spread the load between them as the two location will have a different URI.
I want to hope that next versions will provide a better solution.
Manu
In the last post I wrote about elasticity policy that defines when to change the number of instances.
In this post I will write about how to actually do that.
Let us say you have a service with 2 instances and you want to increment the number of instances by 1.
The simplest way to that is using the portal.
Well it is nice and easy but it is not automated. Maybe this is a good thing. Remember: More instances means more money to pay ! The fact that the process of creating an instance is not automated means that we stay in full control. On the other hand humans are slow and expensive and if there is no human around (weekends…) we might find ourselves with not enough computing resources for quite some time.
The second option is to use powershell
The scripts to increment the number of instances would look something like this
$cert = Get-Item cert:\CurrentUser\My\CertName
$sub = "CCCEA07B-1E9A-5133-8476-3818E2165063"
$servicename = 'myazureservice'
$package = "c:\publish\MyAzureService.cspkg"
$config = "c:\publish\ServiceConfiguration.cscfg"
Add-PSSnapin AzureManagementToolsSnapIn
Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub |
Get-Deployment -Slot Production |
Set-DeploymentStatus 'Running' |
Get-OperationStatus -WaitToComplete
Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub |
Get-Deployment -Slot Production |
Set-DeploymentConfiguration {$_.RolesConfiguration["WebUx"].InstanceCount += 1}
To automate this you can spin a process that will run powershell yourscript.cmd .As I wrote in the last post you can create rules and monitor performance characteristics to decide when to run the script.
Another option is to use the management API.
The management API is a REST API that can be used to execute any management task that can be executed in the portal.
csManage is a command line tool that abstracts the API. This is a sample provided with Windows Azure Samples so you have the code. It is easy to convert this to a simple API that you can use in your project
you can read interesting post franksie wrote about csManage.
The last Option I would like to talk about is "Elasticity using tools".
Here we are in the beginning of the road… I imagine that many tools are being developed right now (Including System Center Support)
Igor Papirov pointed me to nice tools (but a little expensive) called AzureWatch that automate exactly what I described in these two posts. It will monitor the performance of your app and then it will spin new instances or kill existing instances according to your rules.
Well now it is for you to decide how to implement elasticity.
Enjoy
Manu