I'm on a mission from God object : DEV, Azure

My SDP 13 sessions demo code

Yaniv Rodenski — Tue, 07 May 2013 20:54:00 GMT

Sela Developer Practice for 2013 is finishing it’s breakout sessions stage, and I had three fun sessions:

Things Every Developer Most Know about HTTP
Big-Time: Introducing HDInsight service (Hadoop on Azure)
Get your Node.js Under Control with TypeScript

I had great time, and would like to thank all attendees for attending my sessions. You can find my demo’s code in my Sky Drive.

Next on Thursday, I have a full day workshop on Node.js, with lot’s of good stuff planned like: Express and hosting node (on premises and in Windows Azure). I also want to thank two of my colleagues who help me develop the workshop:

Eyal Ben-Ivri – who help me develop the hosting module (and is among other things a Linux ninja).
Danny Vernovsky – who is going to join me in the workshop and talk about socket.io.

So hope to see you there,

Yaniv

SDP 2013 Bigger Longer & Uncut

Yaniv Rodenski — Thu, 17 Jan 2013 02:56:00 GMT

After speaking in few international conferences outside of Israel in the past 6 months, I have the pleasure of taking part in another conference which is near and dear to me: Sela’s own SDP. This year our conference is bigger and stronger than ever with an international lineup, including Jesse Liberty, Shawn Wildermuth, Udi Dahan and Mike Martin who will also be speaking in my user group, The Windows Azure community, that month.

This SDP I have the pleasure of giving four breakout sessions:

Things Every Developer Should Know About HTTP
Hadoop on Azure
The Gentle Art of Building Web APIs
Get your Node.js under control with TypeScript

I am very proud to be a part of such a conference, and I believe it is about time Israel will take it’s place in the conferences world as well. If you feel the same join us in the coolest conference in Israel E V E R.

Some more Node.js

Beside the breakout sessions I am going to give one full day tutorial, on Node.js. The tutorial is based on a course I am currently creating (soon to be available publically).

So if you want to get up to speed with Node.js I would love to see you there.

Yaniv

BUILD 2012 - Day 2

Yaniv Rodenski — Thu, 01 Nov 2012 21:57:00 GMT

uild is starting to pick up and today was actually packed with Windows Azure actions. The keynote had a couple of cool announcements and a couple of extremely cool hidden jams. The keynote was led by Satya Nadella. The first thing that caught my eye was a slid showing strategic Windows Azure based operations. One logo belonged to my good friends PlayMyTone and it's a good opportunity to congratulate Ohad and the gang for the great job they have been doing.

Satya Nadella (Photography by Bnaya Eshet)

The next couple of cool staff that ware kind of hidden in Scott Hansellman's demo:

EF 6 Alpha bits
New Windows Azure storage explorer in visual studio now supports CRUD capabilities and Queues
Help pages for ASP.NET web APIs

And last but not least, there was David Campbell talking about Hdinsight, the new name for Hadoop on Azure. With a couple of brand new cool features including:

Linq provider for HIVE
A brand new C# SDK on codeplex that includes a MapReduce API similar to the new Java API (AKA the context API)

Well now I really got to start going over the new bits so you can expect interesting stuff soon

Upcoming events: Øredev, SDP and the Windows Azure/Web Developers Communities Holidays special

Yaniv Rodenski — Sun, 16 Sep 2012 14:59:41 GMT

As always, September is a very special time for me. This year, in addition to my b-day and the Jewish high holidays September marks the beginning of a very intensive period, packed with both public events and very special projects. So without farther ado here are the events that have been keeping me busy lately:

The first thing in my very busy calendar is the Windows Azure/Web Developers Communities Holidays special which is going to take place in the Microsoft R&D center in Herzliyya. The meeting is taking place thanks to the great help of the guys at the Windows Azure Accelerator program that helped us greatly and provided us with an alternative venue while the Microsoft Raanana office is closed for the holidays. As you know we are devoted for helping the developer community in Israel (with a passion for windows azure) and feel that the Windows Azure Accelerator program is a great platform to introduce to our groups.

We are also very excited about the session for this meeting, Ken Egozi (who you might remember from the ALT.NET group) is visiting Israel and is going to talk about the project he was recently working on: Windows Azure Mobile Services.

After this excitement (and a short vacation) I am going to give a couple of sessions myself, the first one is going to be in Malmö Sweden, where I am going to speak in Øredev, about two of my favorite topics: Hadoop and Windows Azure all wrapped up in one session.

The next couple of sessions are going to be in Sela's own SDP. Where I am going to give two fully booked one day tutorials along side Ido Flatow titled HTTP Fundamentals using ASP.NET Web API and I am very excited about these sessions as well.

I hope for another few sessions to pop-up, but for now I guess I should go back to work.

So Shana Tova everyone and see you soon.

Troubleshooting Windows Azure diagnostics using the Diagnostic Infrastructure Logs

Yaniv Rodenski — Mon, 06 Aug 2012 02:45:00 GMT

One of my favorite jams of windows azure is the use of the diagnostics.wadwcg file for setting up windows azure diagnostics. This is a great alternative for configuring the The Windows Azure diagnostics infrastructure in code.

Last week I had an issue working on a customer deployment where I just could not get trace logs using the diagnostics.wadcfg configuration. I was getting the Windows Event Logs that was also configured just fine, but for some reason the trace logs would just not come through.

Setting the diagnostics.wadcfg (just like setting the configuration in code) basically generates another configuration file (per role per deployment) in a container called wad-control-container:

Opening the configuration file I have discovered that wile the the diagnostics infrastructure loaded my Windows Event logs configuration correctly, the configuration for the trace logs was not loaded at all:

Another thing you can monitor using the Windows Azure diagnostics infrastructure is the Windows Azure diagnostics infrastructure. To set up the Windows Azure diagnostics infrastructure logs, I just added the DiagnosticInfrastructureLogs section to the diagnostics.wadcfg file:

<?xml version="1.0" encoding="utf-8" ?>
<DiagnosticMonitorConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration"
                                configurationChangePollInterval="PT1M"
                                overallQuotaInMB="4096">
  <Logs bufferQuotaInMB="1024"
        scheduledTransferLogLevelFilter="Verbose"
        scheduledTransferPeriod="PT1M"/>
  <DiagnosticInfrastructureLogs bufferQuotaInMB="1024" 
                                scheduledTransferPeriod="PT1M" 
                                scheduledTransferLogLevelFilter="Verbose"/>
  <WindowsEventLog bufferQuotaInMB="1024" 
                   scheduledTransferPeriod="PT1M" 
                   scheduledTransferLogLevelFilter="Information">
    <DataSource name="System!*"/>
  </WindowsEventLog>
</DiagnosticMonitorConfiguration>

Now I could start my solution and the Windows Azure diagnostics infrastructure created a table called WADDiagnosticInfrastructureLogsTable:

looking around I have found the following exception:

Message string Polling for configuration changes:System.InvalidOperationException: There is an error in XML document (8, 8). ---> System.FormatException: Input string was not in a correct format.
   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at System.Xml.XmlConvert.ToInt32(String s)
   at Microsoft.WindowsAzure.Diagnostics.XmlSerializationReader1.Read7_BasicLogsBufferConfiguration(Boolean isNullable, Boolean checkType)
   at Microsoft.WindowsAzure.Diagnostics.XmlSerializationReader1.Read11_DiagnosticMonitorConfiguration(Boolean isNullable, Boolean checkType)
   at Microsoft.WindowsAzure.Diagnostics.XmlSerializationReader1.Read14_ConfigRequest(Boolean isNullable, Boolean checkType)
   at Microsoft.WindowsAzure.Diagnostics.XmlSerializationReader1.Read27_ConfigRequest()
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at System.Xml.Serialization.XmlSerializer.Deserialize(Stream stream)
   at Microsoft.WindowsAzure.Diagnostics.ControlChannel.<>c__DisplayClass10.<ConfigMonitoringPoll>b__f()
   at Microsoft.WindowsAzure.Diagnostics.ControlChannel.TryExpectError(HttpStatusCode status, Action act)
   at Microsoft.WindowsAzure.Diagnostics.ControlChannel.ConfigMonitoringPoll(Object sender, ElapsedEventArgs args); TraceSource 'Microsoft.WindowsAzure.Diagnostics' event

Apparently the was something wrong with my configuration and one of the integer values (probably the bufferQuotaInMb). I actually spent half a day trying to figure out what was wrong (I was just curious) but could not find out what was wrong, so I just rewrote the XML element.

Conclusion

There are actually two things I am taking with me from this experience. The first thing I have learned is that even if the Windows Azure diagnostics infrastructure can not parse parts of the diagnostics.wadcfg file it will use the parts it can. But most of all I learned how informative and helpful the Windows Azure diagnostics infrastructure logs can be.

Windows Azure Community Israel - Here We Go...

Yaniv Rodenski — Wed, 06 Jun 2012 22:14:00 GMT

A while ago I posted about my busy June, speaking in two major conferences NDC Oslo (I’m actually posting this from Oslo right now) and QCon New-York. Well June just got even more special for me. On the 25th we are kicking off the Israeli Windows Azure Community that is being managed by Gal Kogman and yours truly. The community is going to meet every month on Monday of the forth week, in ILDC hertzelia.

This community is special for me. I’ve been working for a while now to create a home for Windows Azure developers in Israel (both online in our msdn forum and now physically) and it is nice to see that vision comes alive, with a lot of help from the guys at Microsoft Israel.

For our grand opening we have Noam King over as well as lot’s of cool new things lined up, but we just can’t tell you about them quite yet . So for now just register here for the event and keep following us.

See you there

Yaniv

Hadoop on Azure - Creating and Running a simple Java MapReduce

Yaniv Rodenski — Mon, 04 Jun 2012 14:33:00 GMT

Apache Hadoop has a variety of APIs for developing MapReduce applications: you can use the streaming API to create MapReduce applications with almost any programming language, Hadoop pipes adds native support for C++ applications and Hadoop on Azure provides it’s IsotopeJS library for creating JavaScript MapReduce jobs. You can also use a variety of higher-level abstractions and libraries such as Pig and Hive. With that said and done, it is also useful to know how to develop MapReduce applications using Hadoop’s most natural and primal Java API. This API allows you to develop richer, more powerful MapReduce apps and has a staggering amount of samples around the Internet.

If you come from a .NET background (like I do), creating your first MapReduce job might be a little bit frustrating the first time around. But once you get past some of the initial hurdles you will be writing Java MapReduce code in your sleep.

Setting up your IDE

After playing around with quite a few IDEs I have found JetBrains IntelliJ IDEA to be the best choice. I am currently using the community edition that can be downloaded free of charge from here.

Now we need to create a new project. The easiest way to create a project using Maven. Maven is basically a build automation tool but one of it’s coolest features is the ability to to create a new project based on a template (called an archetype). I used this post to setup Maven on my machine

Next we need to get an archetype that matches the Hadoop version used for the Microsoft Hadoop Distribution (0.20.2). Matthias Friedrich created such a repository which creates a sample word count project. In order to use this Maven repository you need to run the following command line from your working folder:

mvn archetype:generate -DarchetypeCatalog=http://dev.mafr.de/repos/maven2/ -DgroupId=com.hadoop.example -DartifactId=MyFirstMapReduce

Note that MyFirstMapReduce is the name of my package.

Next you will be prompted to select the archetype from the list of archetypes in the repository:

Select 1 and press Enter.

You will be asked to change the version property. Press Enter to keep the default value of 1.0-SNAPSHOT and when prompted again select Y to confirm the details of your project.

A Look Around Our Project

Open IntelliJ IDEA, select File->Open Project and the folder created by Maven, Click OK:

The newly created project should load. This project is created with the classic MapReduce sample: word count including all its dependencies as well as some unit tests.

Open the TokenizingMapper class and view its implementation:

public class TokenizingMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private static final IntWritable one = new IntWritable(1);

    protected void map(LongWritable offset, Text value, Context context)
            throws IOException, InterruptedException {

        StringTokenizer tok = new StringTokenizer(value.toString());
        while (tok.hasMoreTokens()) {
            Text word = new Text(tok.nextToken());
            context.write(word, one);
        }
    }
}

This common sample reads the input file line by line, and generates a key-value output with every input word as its key and the digit 1 as its value. This sample actually uses the IntSumReducer class that basically sums the values of the inputs based on their key. Lets take a look at the WordCount class that implements the Tool interface. The Tool interface is used to handle the command-line arguments and configure the MapReduce job.

public class WordCount extends Configured implements Tool {
    
    public static void main(String[] args) throws Exception {
        int res = ToolRunner.run(new Configuration(), new WordCount(), args);
        System.exit(res);
    }

    @Override
    public int run(String[] args) throws Exception {
        String[] remainingArgs 
                = new GenericOptionsParser(getConf(), args).getRemainingArgs();
        
        if (remainingArgs.length < 2) {
            System.err.println("Usage: WorldCount <in> <out>");
            ToolRunner.printGenericCommandUsage(System.err);
            return 1;
        }
        
        Job job = new Job(getConf(), "WordCount");
        job.setJarByClass(getClass());
        
        job.setMapperClass(TokenizingMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        
        FileInputFormat.addInputPath(job, new Path(remainingArgs[0]));
        FileOutputFormat.setOutputPath(job, new Path(remainingArgs[1]));
        
        boolean success = job.waitForCompletion(true);
        
        return success ? 0 : 1;
    }
}

Now we can build the package. To do this we will use Maven again by running the following command from the project’s directory:

mvn package

after maven finishes building your package you should see the following output:

deploying and running the job

before we start, we need some text to analyze. you can use any text file for that, i specifically am going to use the the 2010 cia world factbook i have downloaded from project gutenberg.

to deploy the job you will first need to setup a hadoop on azure cluster. this is fairly easy all you need to do is just give your cluster a dns name, select the size of cluster you would like to setup and create a cluster login using the cluster request form:

once the cluster is up, click on the remote desktop icon, placed in the “your cluster” group and log in using the credentials you set during the cluster request.

copy your input file to the local file system on the remote server and open the hadoop command shell using the desktop shortcut. copy the file to hdfs using the following command:

hadoop fs -copyfromlocal c:\ciabook.txt input/ciabook.txt

let’s list the files in our hdfs using the following command:

hadoop fs -lsr

we’re now ready to run the word count application. let’s go back to the portal and create a new job. click on the create job icon, placed in the “your tasks” group

in the create job page, give the job a friendly name and the jar file. the jar file can be found under target in the project’s directory, and it should be named [package name]-1.0-snapshot-job.jar where [pakagename] is the name of your package.

next you need to add two parameters by clicking the add parameter button (once for each parameter). set the value of the first parameter (parameter 0) to the input directory (the input directory we just created) and set the second parameter (parameter 1) to output (this will be the output directory create by the mapreduce job). click the execute job button.

your job should start executing. for the cia factbook (approx. 12mb in size) the job executes in less than 2 minutes on average. once the job has completed you should see that its status is “completed sucessfully” (the typo is embarrassingly from the hadoop on azure portal)

open the interactive console icon placed in the “your cluster” group and run the #lsr command to list the files in our hdfs:

and use the #tail command to see the first few lines in the file output file:

summary

coming from a .net background writing hadoop java mapreduce code can be a little bit intimidating. but just like in the .net world, the right tool can make all the difference.

Upcoming Gigs: REST, Web API and Hadoop

Yaniv Rodenski — Fri, 30 Mar 2012 15:51:08 GMT

It seems that every time I have a talk about Web API’s something big happens, last time Microsoft moved it from WCF to ASP.NET and this week Microsoft announced about the release of ASP.NET web stack under an open source license.

This announcement comes a the beginning of a couple of very exciting months for me covered with talks about few things that ware the focus of my professional life for the past year:

REST, Web API

On Monday, April 2, I will be talking @ the Israeli web developers community (WDCIL) about REST via ASP.NET web API. In this session we will discuss REST\Hypermedia services and how ASP.NET Web API can help us develop them.

You can register here for the event, hosted at Microsoft office in Ra'anana

Apache Hadoop-based services for Windows Azure

For the lost 6 months I have been a part of a wonderful team at Sela that created training content focusing on Apache Hadoop-based services for Windows Azure. In June, I will have a couple of exciting sessions sharing the experiences we had developing and using Hadoop on Azure:

June 6-8 – Norwegian Developers Conference (NDC). Oslo, Norway.

On the beginning of June, I’ll be traveling to Oslo, with my friend and colleague Ido Flatow where we both will be speaking @ the Norwegian Developers Conference (NDC). If you are there I must recommend Ido’s Fiddler session (I have seen myself a couple of times).

June 18-23 – QCon New-York.

Towards the end of June, I will be traveling to New-York to speak at the first ever QCon New-York. I am honored to be joining real all-star panel of speakers and can’t wait to be a part of this exciting conference.

If you are planning on attending QCon New-York, you are also invited to use my discount code: RODE100 and get a 100$ discount for all prices.

There are some more talks coming on soon, but for now I have a lot of work ahead of me. I hope to see you in one of my talks, in a conference/develpers community near you.

Yaniv

MSDN Israel Windows Azure and SQL Azure forum

Yaniv Rodenski — Thu, 19 Jan 2012 19:23:00 GMT

I am very happy to announce the opening of a new Windows Azure and SQL Azure forum in Hebrew. The new forum is managed by Shay Friedman and myself.

If you are new to Windows Azure, I would like to use this opportunity to invite you to experience cloud development with our support and guidance. If you are already experienced with Windows Azure development I would like to assure you that you can find in our forum help from highly experienced professionals.

With your participation I am sure we can create an awesome and vibrant community.

Hope the catch you there soon

Yaniv

Understanding Widows Azure Queue Storage Throughput

Yaniv Rodenski — Tue, 23 Aug 2011 15:42:00 GMT

The Asynchronous Queuing Pattern describes a classic way to improve service throughput in distributed applications. Over the years I have seen quite a few implementations of this pattern, from the use of MSMQ to ReactiveQueue, each with its own strengths and weaknesses. Windows Azure queue storage is designed for passing messages between applications in a persisted, scalable and controlled manner. With the above attributes, queue storage is a natural choice for enabling the Asynchronous Queuing Pattern, as described in detail in this MSDN magazine article.

A recent implementation I ran across at a client challenged the performance of the Azure queue storage, especially when dealing with a large queue. Their initial implementation was too slow due to a design issue we identified easily, but now they were stuck with a queue containing millions of records and they could not retrieve the messages fast enough. I decided to measure the length of the different queue operations they were using.

The code I used to measure the performance is very simple and can be found here so you can reproduce the tests for yourself. Keep these considerations in mind:

We are using a public storage infrastructure that is prone to preemption by other applications.
The Windows Azure storage infrastructure and API implementations are subject to change.

The following totals reflect 1000 iterations (minus the first 2 to remove the additional cost of the JIT compiler and other potential initialization overhead) of a standard consumer/producer use of Windows Azure queue storage:

	Total ticks	% Execution
new CloudQueueMessage	3013727	0.000513214
AddMessage	444328027497	75.66556694
GetMessage	79718072883	13.57536056
DeleteMessage	63164926400	10.75648996
AsString	12151612	0.002069324

The first thing we notice is that we can easily improve is the message retrieval code. In the above code we used the GetMessage method to retrieve the messages one by one. However the Windows Azure Queue API also exposes an API that allows the retrieval of up to 32 messages at a time using the GetMessages method. As you can see in the results from the following run, messages retrieval was over 6 times faster.

Note: since I omitted the first two iterations of GetMessages, I also omitted the first 64 iterations of every other queue operation, so at the end of the day we are looking at 936 messages rather than 998, but still the improvement is clearly noticeable.

	Total ticks	% Execution
new CloudQueueMessage	2907419	0.000599733
AddMessage	428481361062	88.3858044
GetMessages(32)	12041770036	2.483938924
DeleteMessage	44255399085	9.128866277
AsString	3833020	0.000790663

The next stop on our quest for throughput improvement is the deletion of messages from the queue after we retrieve them. The consumer has to perform this operation in order to clear the message from the queue and ensure reliability. The call to DeleteMessage can also be easily improved. If you take a closer look at the code, you can see that we are using the DeleteMessage method, which is a synchronous call to the Azure Queue service. However there is no real need to wait for this call, so we can use its async implementation by calling BeginDeleteMessage. The results of this run (again for 1000 iterations minus 64) are shown here:

	Total ticks	% Execution
new CloudQueueMessage	4904719	0.001183763
AddMessage	401853371789	96.98804476
GetMessages(32)	12041770036	2.906303177
BeginDeleteMessage	429024316	0.103545802
AsString	3822202	0.000922495

In our sample code, we do not handle exceptions for BeginDeleteMessage (as well as for DeleteMessage) but we can easily do so by passing a callback function to BeginDeleteMessage, which calls the EndDeleteMessage method inside a try/catch block.

Until this point, we have dramatically improved the consumer code for our queue, which I must admit the easy part. For the producer part it is going to be a bit more problematic. Windows Azure Queue Storage exposes an APM based API for adding messages to the queue (using the BeginAddMessage/EndAddMessage methods). If you are adding to the queue from a client application you can use this API to release the calling thread and using the network card to perform the majority of the heavy lifting.

If you are adding to the queue from a WCF service this will not be enough, you should consider using an asynchronous service contract. More information about implementing asynchronous services (and asynchronous calls in WCF in general) can be found in this blog post by Wenlong dong.

Summary

Windows Azure Queue Storage was created with the SOA Asynchronous Queuing Pattern in mind. Using it’s async APIs (based on WCFs awesome async capabilities) and calling the GetMessages batch method we ware able to improve it’s throughput and lower the need for more compute instances.

There's nothing like a job well done

Yaniv Rodenski — Thu, 14 Apr 2011 13:21:00 GMT

Today @ MIX Ido and I got some very good news. The first part of our work for the HPC Azure Burst training kit was released today for download by Microsoft. Oh and we got a Kinect.

The document released provides on overview of the architectural considerations for using Azure compute nodes as a part of your cluster.

There is a lot of work still ahead of us and we will kick out some very cool content on MSDN (and cooler stuff on our blogs ) so stay tuned.

Yaniv