Kinect – Getting Started – Become The Incredible Hulk

17/06/2011





In my last two posts I’ve talked about Kinect SDK from Kinect .NET SDK–Getting Started and Kinect – Getting Started – Control Camera Angle, but now it’s time to do some cool things with the Kinect Sensor.


Now I’ll show you how to become The Incredible Hulk using Skeleton Tracking.


image


Download Demo Project


One of the big strengths of Kinect for Windows SDK is its ability to discover the skeleton of joints of an human standing in front of the sensor, very fast recognition system and requires no training to use.


The NUI Skeleton API provides information about the location of up to two players standing in front of the Kinect sensor array, with detailed position and orientation information.


The data is provided to application code as a set of points, called skeleton positions, that compose a skeleton, as shown in the picture below. This skeleton represents a user’s current position and pose.


Applications that use skeleton data must indicate this at NUI initialization and must enable skeleton tracking.


The Vitruvian Man has 20 points that called Joints in Kinect SDK.


imageimage


Step 1:  Register To SkeletonFrameReady


Make sure you Initialize with UseSkeletalTracking, otherwise the Skeleton Tracking will not work.

_kinectNui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);
_kinectNui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(SkeletonFrameReady);

The Kinect NUI cannot track more than 2 Skeletons,  if (SkeletonTrackingState.Tracked != data.TrackingState) continue;    
means the Skeleton is tracked, untracked Skeletons only gives their position without the Joints, also Skeleton will be rendered if full body fits in frame.

Debugging isn’t a simple task when developing for Kinect – Get Up Each time you want to test it. Smile


Skeleton Joints marked by TrackingID enum that defined its reference position:

public enum JointID
     {
         HipCenter,
         Spine,
         ShoulderCenter,
         Head,
         ShoulderLeft,
         ElbowLeft,
         WristLeft,
         HandLeft,
         ShoulderRight,
         ElbowRight,
         WristRight,
         HandRight,
         HipLeft,
         KneeLeft,
         AnkleLeft,
         FootLeft,
         HipRight,
         KneeRight,
         AnkleRight,
         FootRight,
         Count,
     }

Step 2: Get Joint Position


The Joint position defined in Camera Space, and we need to translate to our Size and Position.


Depth Image Space

Image frames of the depth map are 640×480, 320×240, or 80×60 pixels in size, with each pixel representing the distance, in millimeters, to the nearest object at that particular x and y coordinate. A pixel value of 0 indicates that the sensor did not find any objects within its range at that location. The x and y coordinates of the image frame do not represent physical units in the room, but rather pixels on the depth imaging sensor. The interpretation of the x and y coordinates depends on specifics of the optics and imaging sensor. For discussion purposes, this projected space is referred to as the depth image space.


Skeleton Space

Player skeleton positions are expressed in x, y, and z coordinates. Unlike the coordinate of depth image space, these three coordinates are expressed in meters. The x, y, and z axes are the body axes of the depth sensor. This is a right-handed coordinate system that places the sensor array at the origin point with the positive z axis extending in the direction in which the sensor array points. The positive y axis extends upward, and the positive x axis extends to the left (with respect to the sensor array), as shown in Figure 5. For discussion purposes, this expression of coordinates is referred to as the skeleton space.


image

private Point getDisplayPosition(Joint joint)
{
     float depthX, depthY;
     _kinectNui.SkeletonEngine.SkeletonToDepthImage(joint.Position, out depthX, out depthY);
     depthX = Math.Max(0, Math.Min(depthX * 320, 320)); 
//convert to 320, 240 space
     depthY = Math.Max(0, Math.Min(depthY * 240, 240)); 
//convert to 320, 240 space
     int colorX, colorY;     ImageViewArea iv = new ImageViewArea();

   
// only ImageResolution.Resolution640x480 is supported at this point
     _kinectNui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(ImageResolution.Resolution640x480, iv,
(int)depthX, (int)depthY, (short)0, out colorX, out colorY);
   
// map back to skeleton.Width & skeleton.Height
     return new Point((int)(imageContainer.Width * colorX / 640.0) – 30, (int)(imageContainer.Height * colorY / 480) – 30);
}

Step 3: Place Image Based On Joint Type


A position of type Vector4 (x, y, z, w – The first three attributes define the position in camera space. The last attribute (w) gives the quality level (between 0 and 1)) of the position that indicates the center of mass for that skeleton.


This value is the only available positional value for passive players.



void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
    { 
        if (SkeletonTrackingState.Tracked != data.TrackingState) continue;

        foreach (Joint joint in data.Joints)
        {
            if (joint.Position.W < 0.6f) return;// Quality check
            switch (joint.ID)
            {
                case JointID.Head:
                    var heanp = getDisplayPosition(joint);

                    Canvas.SetLeft(imgHead, heanp.X);
                    Canvas.SetTop(imgHead, heanp.Y);

                    break;
                case JointID.HandRight:
                    var rhp = getDisplayPosition(joint);

                    Canvas.SetLeft(imgRightHand, rhp.X);
                    Canvas.SetTop(imgRightHand, rhp.Y);
                    break;
                case JointID.HandLeft:
                    var lhp = getDisplayPosition(joint);

                    Canvas.SetLeft(imgLefttHand, lhp.X);
                    Canvas.SetTop(imgLefttHand, lhp.Y);
                    break;
            }
        }
    }
}

Download Demo Project


Enjoy


Add comment
facebook linkedin twitter email

Leave a Reply

13 comments

  1. Pixel20/06/2011 ב 07:43

    Hi, still there is no orientation information available from the sdk. Your example only works with positions for what i see right? Any ideas on how to compute the orientations?

  2. shair20/06/2011 ב 08:12

    I’m doing a demo now, but the basic idea is to calculate also the other Joints. Shoulder Left + Elbow Left you can calculate the angle and based on that create the orientation you want.

  3. dunno28/10/2011 ב 08:59

    Erm do u know how to set the y and X axis for the hand. I am now doing a project on kinect. It is like this: when your left and right hand are down, the flame will appear. Do u know how to do this? If you do, can reply asap?

  4. Dunno11/11/2011 ב 18:34

    Hey really need your help..Hope that you could help! May i Know why did u put the start and the stop button? Is it for counting down of the time?
    Anyway do u know how to insert a game timer because I am doing a game and the user must complete the game within one minute. If you have the source code, can give me?

  5. shair12/11/2011 ב 10:59

    Hi,

    The stop and start are for the Kinect Runtime engine, you can using your own timer there is no connection to Kinect in this point.

  6. Dunno12/11/2011 ב 11:45

    The stop and start are for the Kinect Runtime engine? what do u mean? can u explain? Does it mean that u stop the running of the program?

  7. Dunno12/11/2011 ב 11:47

    I do not know how to write the code to set up a countdown timer in my game so i need your help. Do u know? LIke those normal game, there will be a countdown of time at the top right hand corner of the page and from 60s to 0 s.

  8. Dunno12/11/2011 ב 16:44

    The stop and start are for the Kinect Runtime engine..what do u mean by this statement? could you explain to me because i am just a beginner. Dp u actually you start and run the program or wat?

  9. Dunno15/11/2011 ב 03:04

    void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
    {
    foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
    {
    //Tracked that defines whether a skeleton is ‘tracked’ or not. The untracked skeletons only give their position.
    if (SkeletonTrackingState.Tracked != data.TrackingState) continue;

    //Each joint has a Position property that is defined by a Vector4: (x, y, z, w).
    //The first three attributes define the position in camera space. The last attribute (w)
    //gives the quality level (between 0 and 1) of the
    foreach (Joint joint in data.Joints)
    {
    if (joint.Position.W < 0.6f) return;// Quality check
    switch (joint.ID)
    {

    case JointID.HandRight:
    var rhp = getDisplayPosition(joint);

    Canvas.SetLeft(imgRightHand, rhp.X);
    Canvas.SetTop(imgRightHand, rhp.Y);
    break;
    case JointID.HandLeft:
    var lhp = getDisplayPosition(joint);

    Canvas.SetLeft(imgLeftHand, lhp.X);
    Canvas.SetTop(imgLeftHand, lhp.Y);
    break;
    }
    }
    }
    }

    Hey how to change to part of code such that when the left and right hand are below the hip…the green thing on the hand will not appear? can u giv me the code

  10. Dunno15/11/2011 ב 05:37

    Hi, could you explain the coding inside the void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) to me? I am just a beginner and my coding not good. And what does if (joint.Position.W < 0.6f) return;// Quality check does?

  11. Dunno16/11/2011 ב 03:33

    void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
    {
    foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
    {
    //Tracked that defines whether a skeleton is ‘tracked’ or not. The untracked skeletons only give their position.
    if (SkeletonTrackingState.Tracked != data.TrackingState) continue;

    //Each joint has a Position property that is defined by a Vector4: (x, y, z, w).
    //The first three attributes define the position in camera space. The last attribute (w)
    //gives the quality level (between 0 and 1) of the
    foreach (Joint joint in data.Joints)
    {
    if (joint.Position.W < 0.6f) return;// Quality check
    switch (joint.ID)
    {
    case JointID.HandRight:
    var rhp = getDisplayPosition(joint);

    Canvas.SetLeft(imgRightHand, rhp.X);
    Canvas.SetTop(imgRightHand, rhp.Y);
    break;
    case JointID.HandLeft:
    var lhp = getDisplayPosition(joint);

    Canvas.SetLeft(imgLefttHand, lhp.X);
    Canvas.SetTop(imgLefttHand, lhp.Y);
    break;
    }
    }
    }
    }

    Can explain this part of code to me bcos i do not understand.

  12. Dunno17/11/2011 ב 02:49

    Hey for ur project..ur green hand actually get out of the screen border..i want to make it move only within the screen resolution border…which part of the code should i change?

  13. LALA20/11/2011 ב 05:10

    // map back to skeleton.Width & skeleton.Height
    return new Point((int)(imageContainer.Width * colorX / 640.0) – 30, (int)(imageContainer.Height * colorY / 480) – 30);

    i dun understand this part..can u explain to me