Azure Table Storage – limitations and solutions/workarounds(Part 1)

13 בAugust 2012

4 comments

Azure table storage – is the NoSQL storage key/value based part of Microsoft Azure cloud services. If you are not familiar with Azure table storage I recommend you to first read here. Recently I was working on a very simple application that is hosted on the cloud (simple web role). My previous experience with NoSQL was mostly using RavenDB (which I really love…) but in this application it was the natural decision to use the out of the box cloud NoSQL storage. Getting started was really easy thanks to the great documentation we have online but when I started to implement the business logic I encountered the limitations and searched for solutions/workarounds – in the next post I want to share with you the limitations I encountered and how to solve them.

Limitation type #1 – Storing your domain object types

My application low level architecture is domain driven meaning I have classes/entites that represent my domain – but I cannot store any of them in the table storage due to the following limitations:

Derive from TableServiceEntity

Stored entity must derive from TableServiceEntity  or to be more specific you cannot store any object in azure table storage unless it has the properties: RowKey, PartitionKey and TimeStamp.

Disallow charters in the keys property

let’s say you want to store a URL in the PratitionKey – well you can’t because these are the chars you cannot use:

  • The forward slash (/) character
  • The backslash (\) character
  • The number sign (#) character
  • The question mark (?) character

Supported property types

Let’s say you want to store property of complex type (like Address) well you can’t because the supported types are limited to the following list: 

WCF Data Services type

Common Language Runtime type

Details

Edm.Binary

byte[]

An array of bytes up to 64 KB in size.

Edm.Boolean

bool

A Boolean value.

Edm.DateTime

DateTime

A 64-bit value expressed as Coordinated Universal Time (UTC). The supported DateTime range begins from 12:00 midnight, January 1, 1601 A.D. (C.E.), UTC. The range ends at December 31, 9999.

Edm.Double

double

A 64-bit floating point value.

Edm.Guid

Guid

A 128-bit globally unique identifier.

Edm.Int32

Int32 or int

A 32-bit integer.

Edm.Int64

Int64 or long

A 64-bit integer.

Edm.String

String

A UTF-16-encoded value. String values may be up to 64 KB in size.

Sizes limitations

Stored entities can have up to 255 properties – and combined size <= 1M and RowKey size must be <= 1KB

Solutions

So at this point you may ask what’s new, we could read about it in MSDN (and you’re right), but now I want to discuss the possible solutions/workaround and their pros and cons.

Divide our objects to domain object and stored objects

This solution comes from a lot of other persistence issues in other solutions (like OR\M) and kind of takes the edge from NoSQL power that can store an entity – but maybe it’s good practice to have this separation to keep our Domain Model object free from limitations and restrictions (sometimes we need object that is stored on more than one table). The downside here is obvious: maintenance of classes duplication – you can try using mapping frameworks like automapper but the mapping may be complicated and I think you’ll end up writing your own mapping .

Hooking into DataServiceContext Reading and Writing events

This is needed to complete the solution above to help us dealing with Illegal characters by replacing them before writing and after reading. It can be a place holder for dealing with complex types (if they are private they won’t be stored) we can serialize them to a primitive property – as for me I was using JSON.NET to serialize complex type and I’m able to read it when exploring the stored entities (better than using binary serialization then you cannot understand the stored serialized object). The downside here is that you have to create another layer above the DataServiceContext and add your own logic there – not ideal but get’s the job done.

Use “Fat Entities” 

There are some other alternative, one of them is an open source library to store Strong type entities – under the hood the entity is kept serialized in one property of the TableStorageEntity.

Here is the entity class:

public class CloudEntity <T>

{

        public string RowKey { get; set; }

        public string PartitionKey { get; set; }

        public DateTime Timestamp { get; set; }

        public T Value { get; set; }    

} 

It’s nice and enables working with one set of entity classes but the downsides are major here: first you cannot query on the entity fields (you can query only on the partition and table keys you set) and second you have to use specific layer that is not exposing all possibilities of the DataServiceContext (downside if you need to use something that is not exposed). This sollution is good if you need simple storing and retrieving by the keys values. get it from here.

Use Lucifure

Another interesting library developed in F# and gives a lot of awesome capabilities – I didn’t try it my self but in general: by using custom attributes on your you can store any entity without any limitations (It’s not an open source so I can’t see the code but I guess it’s done using the dynamic capabilities of F# and creates the rest URL calling the azure table storage service on it’s own). It also gives solutions to other limitations like the size limitations. I will try it and update but it’s really worth looking into.

here is entity example:

[StashEntity(Mode=StashMode.Explicit)]

  public class Employee

  {

          // Give the partition, row key and timestamp more meaningful names

          [StashPartitionKey]

          public string Department;

 

          [StashRowKey]

          public string EmployeeId;

 

          [StashTimestamp]

          DateTime AzureInternal;

  }

Summery

Well I’m sure that by now you get the point and I’m sure there are more limitations I missed. Don’t get me wrong Azure Table Storage is great – it gives you out of the box high availability, enable scale and high performance but if your requirement needs a feature that is not supported then you better check the solutions and alternatives before you choose azure table storage and make sure it satisfied your needs.

In the next post I will talk about the querying limitation and what can we do to overcome them.

Cheers,Offir

Add comment
facebook linkedin twitter email

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*

4 comments

  1. Irotteaxoreiz28 בJanuary 2013 ב 0:47

    [url=http://sarahsalter.com/download/cipro/]cipro back pain[/url]
    cipro cleocin allergy
    ciprofloxacin er
    ciprofloxacin for sti

    Reply
  2. Choate7 בJuly 2013 ב 5:24

    Hello, I enjoy reading through your article post. I like to write a little comment to support you.

    Reply