Sorting out the Kinect skeleton feature

The Kinect has some very clever software and firmware which enables it to capture the skeleton patterns of up to two people at the same time.  I have been tracking the progress of the Kinect since it's introduction and bought a commercial, as opposed to an Xbox release, when it became available in 2012.  With the rapid completion of the Inmoov robot print here and the successful testing of the Mini Maestro 24 servo controller with the Hitech HS-805BB servos used to move the arms and shoulders, I began to delve into the Kinect SDK.

The Kinect can capture skeleton patterns in two modes; full body...

...and upper body {seated}

Extracting the Cartesian coordinates of the bones in three dimensions is straightforward.  Being a naturally suspicious sort, however, I decided to check to see how consistent the length of the individual bones actually was.  As I suspected, they vary.  This shouldn't be surprising considering the wickedness of trying to figure out the position of bones buried in flesh and muscle starting only with surface measurements of the body and on top of that making the calculations at a rate of thirty frames per second.

Just eyeballing the captured data it appeared that the bone lengths were varying wildly.  When I took a mean and standard deviation for each bone, however, I discovered that the calculation was remarkably consistent. I made a test data set using myself as the subject.  The measurements are in meters.

Mean SD Shoulder Left 0.217 0.017 Right 0.218 0.013 Humerus  Left 0.242 0.025 Right 0.241 0.026 Forearm Left 0.235 0.024 Right 0.235 0.026 Palm Left 0.089 0.034 Right 0.086 0.029

The numbers are not too bad.  The wrist/palm measurements are the worst.  I have extra code which I hope will give better results for the hands and fingers.

The next step will be to see if I can transform the coordinates into something more useful with the Inmoov by fixing the bone lengths to calibrated means.

Kinect Progress 13-02-23

This morning, I got a note from the UP 3D printer people that they wanted to do a Skype video session to put my printer right.  I thought they meant right then, but was wrong, apparently.  While the new extruder heater block helped matters it didn't fix my UP.

In the meantime, I worked on the Kinect motion capture task.  I have it capturing the upper torso, head and arms and transferring it to a disk file.  My programme is very crude.  I simply identify the joints and put the coordinates in a list box.  Surprisingly, my PC and Visual Basic 2010 is able to keep up with the 30 frames/second that the Kinect does that process at with no effort whatsoever.  After I record a session, I simply save the XML formatted information to a file.

My next task is to restrict the saved output to the left upper torso, shoulder and arm and to covert the cartesian coordinates to joint specific coordinate systems which match the kinematics that Gael has designed into the Inmoov arm.

Digging into the Kinect

I've been avoiding getting into .NET for years, but now I am having to get on with learning it.  The first thing I've encountered is a whole new set of controls called WPF {Windows Presentation Foundation} controls.  So far, so good.

Starting with the Skeleton tracking routine written in VB.NET 2010, I dug around in the data structure and managed to locate the Cartesian coordinates of the major body joints that Kinect is so adept at locating.  They are generated at 30 times/second and fill up a listbox rather quickly.

Watching the graphics display, it appears that adding a simple motion smoothing routine would be very useful.  I'm surprised that something of the sort wasn't included in the firmware.

The scale of the coordinates appears to be meters.

Kinect is active

I was able to get Kinect going and run it through most of its paces.  It can capture a 3D radar image out to about 4-5 meters as you can see with this colour coded screen grab.

This perspective projection gives you, perhaps, a better feel for the depth of the space.

Oddly, it flips the data from left to right.  I don't know what that is about yet.

It does face tracking down to about a meter away from the Kinect and skeleton tracking down to about 2 meters.  I was unable to capture that with the very primitive Visual Studio apps that I have so far.  I thought it showed the orientation of the palm, but with what I have I apparently don't have that possibility.

I was also able to videotape the skeleton tracking option.  Here is the full body tracking.

I understand that Microsoft used quite a large farm of servers to develop the parameters for this feature.  Notice that you have trouble when a foot leaves the visual field of the Kinect.  Also notice the problem when you accidentally get one leg behind the other or behind an obstacle, in this case a stool.

Here we restrict tracking to the upper body, viz, "seated mode".

Notice that when the hands leave the visual field of the Kinect the same thing happens that happened when the feet left the visual field in full body mode.  Also note what happens when I placed one and the both hands on my head.I was able to locate some open source finger tracking software for the Kinect that was written in Spain.  I will be trying that next.