WPF Image Processing

September 21, 2007

16 comments

With the release of Windows Vista and WPF, Microsoft has also released the WIC: “an extensible framework for working with images and image metadata. WIC makes it possible for independent software vendors (ISVs) and independent hardware vendors (IHVs) to develop their own image codecs and get the same platform support as standard image formats (for example, TIFF, JPEG, PNG, GIF, BMP, and WMPhoto)” (MSDN). Unsurprisingly, WPF uses WIC in its core (for example, BitmapSource). Unfortunately, the WPF team chooses to keep the WIC low level API internally, in such a way that there isn’t a simple solution for dealing with imaging without consuming a lot of memory and CPU time.


About few months ago, my customer asked me to develop an image processing control based on WPF for a medical-system. This includes several post-processing operations such as zooming, panning, windowing (for B/W) and other implemented on the server side. So I started to investigate the WPF imaging system. First I tried to load raw B/W images (1~18Mb), using the BitmapSource.Create method and the Image visual type. This method worked fine until it came to image-processing. Think that you have a 1Mb-pixel image, and you have to iterate through all of its pixels (1048576) on every mouse movement, calculating the next pixel value, and update the screen. With an Intel Duo Core2 CPU the calculation part is insignificant, but what about updating the screen? Now that I have a new buffer, how do I update the BitmapSource instance?


I was thinking about “disposing” the old BitmapSource image, and then create a new one using the Create method again, since it consumes a lot of memory. Alas! The BitmapSource type is not disposable! Also, there is no Close method. So I putted a null in the BitmapSource reference (just for fun), and created a new BitmapSource. Since the image size is 1M, The CLR considering it as a large object. Large objects are kept in the large-object-heap for a long term (calling the GC.Collect method is not an option!).


The best way of course is to manipulate the already allocated BitmapSource internal buffer. But let put it that way: WPF has no public methods for manipulating the BitmapSource internal buffer (which is WIC of course).


Digging inside BitmapSource with Reflector, I figured out that BitmapSource calls WIC native API’s for handling its buffer internally. As I already said, the WPF team did not expose these methods.


Based on this knowledge, I have developed a .NET component for manipulating the internal buffer, by accessing these methods using the .NET great tool ever, reflection.


Before I will show you my code, I want to forward you to Jeremiah Morrill post. He did exactly the same, but posted it three months before me. He owns all the credit for its great job.


My solution is composed of three types: WICBitmap, WICBitmapBuffer and WICBitmapLock.


WICBitmap – encapsulates a BitmapSource instance by providing operations for locking (accessing) the internal WIC image buffer.


WICBitmapLock – represents a WIC buffer lock.


WICBitmapBuffer – encapsulates a WIC buffer pointer and its size.


The code snippet bellow demonstrates how to create a BitmapSource image, and how to access its internal buffer using the WICBitmap* types.

private void InitializeImage()
{
Uri imageUri = new Uri(@”Images\View.jpg”, UriKind.Relative);
BitmapSource bitmapSource = new BitmapImage(imageUri);
_imageVisual.Source = bitmapSource; // WPF Image type
_wicBitmap = new WICBitmap(bitmapSource); // WICBitmap type
}
void button_Manipulate(object sender, RoutedEventArgs e)
{
try
{
using (WICBitmapLock bitmapLock = _wicBitmap.Lock())
{
WICBitmapBuffer bitmapBuffer = bitmapLock.Data;
unsafe
{
Int32* pStart = (Int32*)bitmapBuffer.Buffer.ToPointer();
Int32* pEnd = pStart + bitmapBuffer.Size / sizeof(Int32);
for (; pStart != pEnd; ++pStart)
{
// RRGGBB
*pStart &= 0x0000FF;
}
}
}
_imageVisual.InvalidateVisual();
}
catch (Exception ex)
{
MessageBox.Show(“Failed to manipulate image:\n” + ex.Message);
}
}
image image image image 

As you can see, the code above is very simple so there is no need for explanation.


There are some disadvantages for using this mechanism:



  1. You should call Image.InvalidateVisual() explicitly after manipulating the buffer (this informs the Image that the BitmapSource data was changed and that it needs to render the image again).
  2. Sometimes the Lock/Copy fail. I think that it because I’m trying to lock the buffer while it is in use by the render thread (use the TryLock method instead of Lock, and try again and again if fails. The second or third lock should succeed).
  3. You have to copy at least the first image with BitmapSource. I didn’t find an option to wrap the first/original buffer like in GDI+ (if you have a lot of images, I suggest to create a pool of BitmapSource instances and reuse them).
  4. This solution is temporary and may not work in the future versions of WPF (it works on 3.5), since it uses internal methods. Use it on your own risk.

NOTE: There is another half documented solution for manipulating an image: Custom Bitmap Effects. Creating a custom bitmap effect is involved with C++/COM programming. Maybe I will have time to demonstrate this in a future post.


Now that you have an access to the BitmapSource WIC buffer, you don’t need to create BitmapSource instances for each post-frame, which consumes a lot of memory and CPU time.


Download the code (Orcas) from here.


In the next post I will explain how the WIC interop code works, and how to create an image stream (Video) based on my solution.

Add comment
facebook linkedin twitter email

Leave a Reply to John Melville Cancel Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*

16 comments

  1. ליאור צורףSeptember 25, 2007 ב 14:21

    Hi Tomer,

    Did you get the invitation to bloggers event @ Microsoft on October 8th?
    If not, please contact me at http://blogs.microsoft.co.il/blogs/liorz/contact.aspx

    We’d love to see you there!

    Lior

    Reply
  2. John MelvilleDecember 17, 2007 ב 06:49

    I also needed to do some image postprocessing for a medical application. I chose to go the BitmapEffect route and I was really pleased with the results. I have a custom bitmap effect that will adjust the brightness and contrast, then apply a convolution operator to sharpen the image and finally apply a level operator to the output. (All three steps are optional.) My optimized c++ filter can do these operations at an accepable realtime speed on my old tablet PC. Part of the reason I can do this is I am filtering at screen resoltuion and only filtering visible pixels.

    I “reprocessed” an API sample for the BitmapEffects API to make it easier to get started. Check it out at http://johnmelville.spaces.live.com/blog/cns!79D76793F7B6D5AD!115.entry

    John Melville

    Reply
  3. Tomer ShamamDecember 17, 2007 ב 09:01

    Indeed, this is a possible unmanaged solution for the image procesing in WPF, which I didn’t have time to write about, but I mentioned it in one of my posts. Also you can use the Imaging class for interop with unmanaged memory. See my post about “WPF Official Image Interop”.

    Great work!

    Reply
  4. Brian J. CardiffMarch 14, 2008 ב 01:07

    I’m working with many 4800×4800 pixel images at the same time. One interaction is to toggle the visibility of that image. (I implement this removing Image control).

    I use some kind of lazy load for the BitmapSource since the total number of images are aprox 250.

    The issue I have is that the first time the image is effectively displayed on screen I have ~1 second delay, but on later toggles over same BitmapSource (and re adding the old Image control) the action is done immediately.

    Do yo know how can I improve first load? Maybe storing Images with a certain codec ?

    Thanks in advance

    Reply
  5. vbtricksMay 22, 2008 ב 15:24

    Really speedy sample! Unfortunately I have no luck in putting the processing in a background thread. See http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=3385024&SiteID=1

    Thanks in advance

    Reply
  6. Tomer ShamamMay 22, 2008 ב 17:11

    Hi vbtricks,

    See my reply on your post.

    Reply
  7. SantoshMay 5, 2009 ב 07:36

    Hi Tomer,

    I want to implement ZOOM-IN, ZOOM-OUT on image in WPF application. Currently I’m increasing the Height & Width of Image Control. But it consumes large memory and causing application to throw OutOfMemory/InsufficientMemory exception.

    Also, I’ve a doubt whether it’s a right way or not. But after googling a lot I didn’t find another approach for implementing Zoom Functionality.

    Please, help/suggest me the right way where in memory usage should increase/decrease w.r.t. zoom in/zoom out.

    It’ll be of my greater pleasure !! looking for your reply.

    Thanks!!!

    – Santosh.

    Reply
  8. Tomer ShamamMay 5, 2009 ב 20:24

    Hi Santosh,

    In case that you just want to render the image scaled (zoom-in/zoom-out) you should write:




    In case that you want to manipulate the image pixels you should look at this:
    http://msdn.microsoft.com/en-us/library/system.windows.media.imaging.transformedbitmap.aspx

    Hope it helps.

    Reply
  9. MichaiłJanuary 21, 2010 ב 02:41

    I think that there is no need to do all that kung-fu 🙂 in order to get internal buffer access. (Al least with 3.5 SP1 and later versions of WPF)

    Please use standard, well known interface:

    http://msdn.microsoft.com/en-us/library/system.windows.media.imaging.writeablebitmap.aspx

    Reply
  10. Tomer ShamamJanuary 21, 2010 ב 09:48

    Considering the time of this post, there was no .net3.5 🙂

    Reply
  11. Meuwissen FabriceJune 7, 2010 ב 01:51

    I have a stupid question but there is no alternative to that WPF bitmap object for Winforms ( using c# for example ), I still can’t understand why the Microsoft documentation is so difficult to find to know how to use a bitmap loaded from the wic gateway

    Reply
  12. DudiOctober 31, 2010 ב 16:34

    It is not working for .NET Framework 4.
    What shall i do to have it work in framework 4?

    Reply
  13. Tomer ShamamOctober 31, 2010 ב 16:38

    Hi Dudi,

    It’s not relveant for .NET 4.0 anymore. Use WriteableBitmap instead.

    Reply
  14. polinzhuoJuly 6, 2011 ב 10:23

    WriteableBitmap always contains 32bits data. For 3000*3000 images, 32bit memory space will hold 30M memory space. It is not good for my app.
    how to create 8bits data image using WriteableBitmap ?

    Reply
  15. TitanFebruary 13, 2012 ב 14:18

    I’m facing the same issue. WriteableBitmap takes unwanted memory space even if I use Gray8 pixel format. Is there any workaround to reduce the memory usage?

    Reply
  16. nbFebruary 7, 2013 ב 03:23

    Thanks for the code. I tried it to an InteropBitmap (to get its internal lock, see social.msdn.microsoft.com/…/291b66fc-8c8f-4f9a-969c-86478c5858a6), but it fails during the call to the following line:

               //Execute our static Lock() method

               hr = (int)lockmethod.Invoke(null, args);

    Reply