LINQ to HPC (Formerly known as DryadLINQ) Tutorial: Part 2–Data Partitioning (DSC)

יום שלישי, אוגוסט 9, 2011

A new beta has been released since I wrote part 1 of this tutorial. While very little was changed in the product, we have a new name. Another thing held me back personally from publishing this part was the fact that LINQ to HPC is not a part of Windows HPC R2 SP2. So without farther ado I am proud to present the second part of my tutorial about LINQ to HPC. In part 1 of this tutorial we discussed the fundamentals of DSC: how to manually write data to DSC files and how to use the FromEnumerable<T>...

DryadLINQ Tutorial: Part 1 – Distributed Storage Catalog (DSC) Basics

יום שלישי, יוני 14, 2011

One of the most exciting additions to Windows HPC Server 2008 R2 SP2 (currently in beta) is the support for DryadLINQ. DryadLINQ is an API that allows the creation and execution of large scale, data-parallel compute tasks. One of the core capabilities of Dryad (the underlying framework used by DryaLINQ) is the ability to distribute the data over the cluster and maintain data locality by executing the code on the node storing the data. In order to do both, Dryad utilizes a mechanism called The Distributed Storage Catalog (DSC) which I will cover in this post. ...