Webcam control with WPF or how to create high framerate player with DirectShow by using InteropBitmap in WPF application

23 באפריל 2008

29 תגובות

[This blog was migrated. You will not be able to comment here.
The new URL of this post is http://khason.net/blog/webcam-control-with-wpf-or-how-to-create-high-framerate-player-with-directshow-by-using-interopbitmap-in-wpf-application/]


Did you ever see, that MediaElement “eats” about 30% of CPU while playing movie in WPF? Did you thought, that you  can display live camera capture in WPF with 60 fps full screen (I have really high resolution 1920×1200) and 2% of  CPU? You did not? Let’s see how it can be done. Today we’ll create simple WebCam player control that can show you live video capturing with high frame rate. In order to do it, we’ll use DirectShow, WPF and make them work together.

  image

You, probably do not believe me. Let’s start. In order to build this application, we need to make DirectDraw working in C# managed code. We can use DirectShow.NET, but this time we’ll do it manually. Why? because I love to do things manually. So let’s understand what we need? Actually, not very much: one Sample Grabber (ISampleGrabber) and one Device input filter(IBaseFilter). Both we should connvert with Graph Builder (IGraphBuilder) and point to some grabber implementation (ISampleGrabberCB). Also, we do not want DirectShow to render video for use, thus we’ll send it’s default Video Window (IVideoWindow) to null with no AutoShow and then run the controller (IMediaControl). Do you tired enough to lost me? Let’s see the code. One Filter graph with one Device Filter and one Sample grabber.

graph = Activator.CreateInstance(Type.GetTypeFromCLSID(FilterGraph)) as IGraphBuilder;
sourceObject = FilterInfo.CreateFilter(deviceMoniker);

grabber = Activator.CreateInstance(Type.GetTypeFromCLSID(SampleGrabber)) as ISampleGrabber;
grabberObject = grabber as IBaseFilter;

graph.AddFilter(sourceObject, "source");
graph.AddFilter(grabberObject, "grabber");

Set media type for our grabber

using (AMMediaType mediaType = new AMMediaType())
                {
                    mediaType.MajorType = MediaTypes.Video;
                    mediaType.SubType = MediaSubTypes.RGB32;
                    grabber.SetMediaType(mediaType);

And then connect device filter to out pin and grabber to in pin. Then get capabilities of video received (thiss stuff come from your web camera manufacturer)

if (graph.Connect(sourceObject.GetPin(PinDirection.Output, 0), grabberObject.GetPin(PinDirection.Input, 0)) >= 0)
                    {
                        if (grabber.GetConnectedMediaType(mediaType) == 0)
                        {
                            VideoInfoHeader header = (VideoInfoHeader)Marshal.PtrToStructure(mediaType.FormatPtr, typeof(VideoInfoHeader));
                            capGrabber.Width = header.BmiHeader.Width;
                            capGrabber.Height = header.BmiHeader.Height;
                        }
                    }

Out pin to grabber without buffering and callback to grabber object (this one will get all images from our source).

graph.Render(grabberObject.GetPin(PinDirection.Output, 0));
grabber.SetBufferSamples(false);
grabber.SetOneShot(false);
grabber.SetCallback(capGrabber, 1);

Dump output window

IVideoWindow wnd = (IVideoWindow)graph;
wnd.put_AutoShow(false);
wnd = null;

And run the controller

control = (IMediaControl)graph;
control.Run();

We done. Now our video is captured and can be accessed from BufferCB method of ISampleGrabberCB. Next step is to do WPF related stuff

First of all, we’ll use InteropBitmap. This one will provide us with real performance bust. So, one our DirectShow graph is ready and we know result image capabilities, we can create memory section and map it in order to provide ISampleGrabberCB with place to put images. This will be always the same pointer, so all we have to do is to .Invalidate interop image.

if (capGrabber.Width != default(int) && capGrabber.Height != default(int))
                {

                    uint pcount = (uint)(capGrabber.Width * capGrabber.Height * PixelFormats.Bgr32.BitsPerPixel / 8);
                    section = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04, 0, pcount, null);
                    map = MapViewOfFile(section, 0xF001F, 0, 0, pcount);
                    BitmapSource = System.Windows.Interop.Imaging.CreateBitmapSourceFromMemorySection(section, capGrabber.Width, capGrabber.Height, PixelFormats.Bgr32,
                        capGrabber.Width * PixelFormats.Bgr32.BitsPerPixel / 8, 0) as InteropBitmap;
                    capGrabber.Map = map;
                    if (OnNewBitmapReady != null)
                        OnNewBitmapReady(this, null);
                }

Now in capGrabber (ISampleGrabberCB) we’ll copy buffer, comes from our webcam to the mapped location for WPF usage

public int BufferCB(double sampleTime, IntPtr buffer, int bufferLen)
        {
            if (Map != IntPtr.Zero)
            {
                CopyMemory(Map, buffer, bufferLen);
                OnNewFrameArrived();
            }
            return 0;
        }

All we have to do is to call InteropBitmap.Invalidate() each frame to reread the image bytes from the mapped section.

if (BitmapSource != null)
                    {
                        BitmapSource.Invalidate();

How do display all this stuff? Simple – subclass from Image and set it’s Source property with the interop bitmap.

public class CapPlayer : Image,IDisposable
    {

void _device_OnNewBitmapReady(object sender, EventArgs e)
        {
            this.Source = _device.BitmapSource;
        }

Now, the usage from XAML is really simple

<l:CapPlayer x:Name="player"/>

We done 🙂 As always, download full source code for this article

…and be good people and don’t tell anymore, that WPF performance in terms of imaging is sucks 🙂

P.S. small ‘r’ if you have more, then one WebCam connected. Inside CapDevice class there is member, public static FilterInfo[] DeviceMonikes, that provides you with all DirectShow devices installed. So, the only thing you should do in order to change the device is to set deviceMoniker = DeviceMonikes[0].MonikerString; with the moniker of your device. This sample works with first one.

הוסף תגובה
facebook linkedin twitter email

כתיבת תגובה

האימייל לא יוצג באתר. שדות החובה מסומנים *

29 תגובות

  1. Jeremiah Morrill23 באפריל 2008 ב 22:34

    Great example of how to use the InteropBitmap! I've heard about it, but never tried it.

    I do disagree about the MediaElement. It still is the highest performing way to get video into WPF. Maybe different GPUs handle it differently…

    I downloaded the example and indeed it was pretty low CPU. Better than I expected! Around 4% – 6% CPU at 640×480@20 FPS.

    I tested my VideoRendererElement(http://www.codeplex.com/VideoRendererElement) and it was able to do the same video settings at 1% – 2% CPU tax. Give it a shot and let me know what you think.

    -Jer

    הגב
  2. Tamir Khason24 באפריל 2008 ב 8:45

    Jeremiah, MediaElement uses WMP, that rather resource greedy. I'll take look into your videoRendererElement

    הגב
  3. PierreMF26 במאי 2008 ב 16:19

    Thanks for the sample code. I am know using extensively InteropBitmap in my app. It is very fast. WPF happens to be faster than GDI+ : don't need to lockbits/unlockbits !

    Just a question: how did you figure out that CreateBitmapSourceFromMemorySection needs a memory file mapping ? In MSDN, they say it is "A pointer to a memory section" 😉

    הגב
  4. Tamir26 במאי 2008 ב 16:41

    It *is* CreateBitmapSourceFromMemorySection other words pointer to memory section. So if you'll put some image structure in this memory section it will be handled and displayed by WPF

    הגב
  5. cindex16 ביוני 2008 ב 6:11

    Thank you.
    You help me.

    הגב
  6. cindex16 ביוני 2008 ב 20:40

    How can I fix it?

    הגב
  7. cindex16 ביוני 2008 ב 21:20

    webcam image is inversed.

    How can I fix it?

    הגב
  8. Franz18 ביוני 2008 ב 3:13

    Help!
    Why is the portrait shows the inverted?

    הגב
  9. Tamir Khason18 ביוני 2008 ב 10:40

    Franz, this is direct show behaviour. All you have to do, either set image invert option on for DS graph options, or use TranslateTransform for WPF image

    הגב
  10. EAM18 ביוני 2008 ב 13:01

    Thank you very much for the great example!

    I tried to modify the source code to capture Rgb24 format video, by doing the following:

    mediaType.SubType = MediaSubTypes.RGB24;

    And I changed the format of the memory section by doing the following:

    uint pcount = (uint)(capGrabber.Width * capGrabber.Height * PixelFormats.Rgb24.BitsPerPixel / 8);

    section = CreateFileMapping(new IntPtr(-1), IntPtr.Zero, 0x04, 0, pcount, null);

    map = MapViewOfFile(section, 0xF001F, 0, 0, pcount);

    BitmapSource = System.Windows.Interop.Imaging.CreateBitmapSourceFromMemorySection(section, capGrabber.Width, capGrabber.Height, PixelFormats.Rgb24,
    capGrabber.Width * PixelFormats.Rgb24.BitsPerPixel / 8, 0) as InteropBitmap;

    (I tried PixekFormats.Bgr24 also).

    But I can only see pure black in the capture window. Does anyone happen to know why?

    Thank you very much!

    הגב
  11. EAM20 ביוני 2008 ב 6:35

    No one met this problem before?

    Can someone help me? Thanks

    הגב
  12. cindex20 ביוני 2008 ב 8:25

    I have solved Inversed Image broblem.





    But I have another problem.
    I load video in new window.

    In that window button click event.
    player.Dispose();
    this.close();

    After that, soon I float that window. Video is not shown In new window .
    probably, because fo deferred dispose cause that broblem.

    Who do you know solution?

    הגב
  13. Tamir Khason20 ביוני 2008 ב 9:39

    EAM, it looks like you have a problem with pixelformat
    cindex, you have to recreate new handler

    הגב
  14. EAM20 ביוני 2008 ב 12:47

    Thank you very much for your response!

    I also believe it is a problem with the pixel format. I tried to save the content in "map" (the memory) to a file. The avi file is ok. I can play it with media player.

    But any idea which format I shall change? For "mediaType.SubType", I guess we don't have any other choice besides "MediaSubTypes.RGB24". For CreateBitmapSourceFromMemorySection, I tried both PixelFormats.Rgb24 and PixekFormats.Bgr24. I got the same results. Any other formats I shall try?

    Would you mind trying the modification on your source code? I believe you can see the same results by just changing the formats.

    I really appreciate your help!

    הגב
  15. Tamir Khason20 ביוני 2008 ב 13:28

    I can do nothing, 'cos I do not know what video format your camera outputs. Try to play with argb and abgr also, you can look into 32 bit formats

    הגב
  16. cindex25 ביוני 2008 ב 16:19

    Thank you for your answer,Tamir Khason.
    But, I do not understand yours.

    Inspite fo "player.Dispose();"
    My webcam`s lamp is not turned off.(my webcam`lame turned on when it run)
    Because of that, probably new window is non shown webcam video.
    It`s lamp is turned off when application is closed.

    הגב
  17. cindex25 ביוני 2008 ב 16:22

    Web cam video window is opened by "showdialog()"

    הגב
  18. cindex25 ביוני 2008 ב 16:48

    …hmm
    Lamp turned off when debugging stop, not when application is closed.

    הגב
  19. cindex30 ביוני 2008 ב 21:16

    Please, answer my problem.

    T.T

    Inspite of "player.Dispose();"
    webcam is not released.
    Because of that, new window do not load webcam.

    Help,me

    הגב
  20. Tamir Khason1 ביולי 2008 ב 13:36

    cindex, I do not understand and cannot repro the problem. It looks like a problem with your webcam software or drivers

    הגב
  21. Ted24 ביולי 2008 ב 17:26

    Excellent article Tamir!

    I was playing around with it using a firewire-video as source, and it worked just fine (except for beeing flipped upside down, but that was easily corrected with a a scale transform), BUT I have a question though, is there any route I could take to add de-interlacing of the video as well? Do you know? Thanks!

    Cheers!
    -Ted

    הגב
  22. Mike6 באוגוסט 2008 ב 16:23

    Wow, excellent article/code, thanks!

    I'm trying to get this to work with an IP Camera (Linksys WVC54GC) but can't figure out how to get the FilterInfo for the ip camera (device enum won't work since it's not directly connected to my pc). I can get IFilterInfo for the ip cam by going through FilgraphManager instead, but connect fails when I try to use this filter. Any ideas on how I can get this to work with an IP Camera? Any help would be greatly appreciated.

    Thanks!
    Mike.

    הגב
  23. Brandon10 בספטמבר 2008 ב 16:16

    How do you stop the feed of the images? I called the Stop() method located _device.Stop() and made a public method to access. When I call this it will stop but it takes a crazy long time and is very random, does it not dispose properly?

    הגב
  24. C.S. Finch15 בספטמבר 2008 ב 17:25

    Hi, I noticed someone posted a question about the video being upside down, and I must admit I have the same problem, however, I fixed it with an easy fix.

    2 options.

    First option is you can go into Expression Blend 2.0, and just rotate the video player 180 degrees.

    if you don't have Expression Blend 2.0 you can do the following

    replace the following <1:CapPlayer x:Name="player"/>
    with the following code






    Hope it helped.

    הגב
  25. Ivan21 בנובמבר 2008 ב 6:32

    What about webcam options? Is it possible to change them?

    הגב
  26. Matt26 בנובמבר 2008 ב 22:35

    How would you specify the resolution you want the camera to produce? Sorry about the newbie question.

    הגב
  27. Rodrigo11 בדצמבר 2008 ב 5:00

    I'm starting in WPF. There is some method in the classes of this project that capture one frame of the CapPlayer control to put in one image control, for example?

    הגב
  28. Geert van Horrik22 בדצמבר 2008 ב 10:44

    I have extended your CapPlayer class a little bit by adding 2 properties:

    1) Rotation
    2) CurrentBitmap

    You can find a blog entry about it here:

    http://blog.catenalogic.com/post/2008/12/22/Retrieving-snapshots-from-a-webcam.aspx

    הגב
  29. Rich31 בדצמבר 2008 ב 2:54

    Very impressive. +1 for changing the resolution. I've been searching for a solution and could not find any that I understand.

    הגב