Senior Design Group 16Client / Adviser: Dr. Tom Daniels

Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer

Cpre 185 Wii/Kinect Lab Software

Freshmen computer engineering college students and high school students need an interesting way to be introduced to programming and problem solving.

In the past, Iowa State students have been taught to create simple programs in the C programming language that use Nintendo Wiimotes as input devices.

Software needs to be developed similar to Wii Wrap, but uses the Microsoft Kinect™ sensor as input to student-written programs.

Problem Statement

RGB Webcam Camera 640x480 @ 30 Hz (center) IR Projector Projects Grid for Depth Mapping IR Camera at 640x480 @ 30 Hz collects depth data

(right) Microphones give room audio

Hardware - Kinect

Small Panning motor turns sensor bar from side to side

Onboard image preprocessing

Projector power draw requires extra AC power

Wii Remote / Wiimote 11 buttons + Power Roll/Pitch/Yaw Gyro Infrared Sensor Speaker (unsupported) Optional Nunchuck

Hardware - Wii

Schedule / Cost

Three Modules:1. Hardware

interface to read in raw data (provided)

2. Interface simplification / data processing

3. Graphical interface to display output

Architecture

Flow Diagram

Issue: Decrease delay in printf output Solution: Added a fflush(stdout) statement

immediately after printf

Issue: IR data becomes erratic if sensor loses source for one frame of data

Solution: Save last good frame of IR data (x,y) and loop the last good data until new data arrives

WiiMote Solutions

OpenKinect◦ Tricky to get started with in windows◦ Natively supports Unix◦ Body tracking not built-in◦ Needs a calibration pose for body tracking◦ No event when new Video or Depth frame is available

Microsoft Kinect SDK◦ Sample Projects◦ Documentation◦ Simple installer◦ Natively supports Windows◦ Full body tracking built-in

Kinect Design Approaches

Data Available◦ RGB Video Frames (640x480 max resolution)◦ Depth (mm) Data Frames (320x240 max

resolution)◦ Skeletal tracking points (20 points per player)

How we get the data◦ Pass an event object to alert us when new data is

available◦ Pass a video/depth/skeleton frame pointer to read

in next frame◦ Run event loop and process each frame as

RGB/Depth/Skeleton data

Microsoft Kinect SDK

Windows File Mapping is an inter-process communication tool that maps a block of memory by name

Shared Memory in Kinect Wrap◦ File Mapping is set up by Kinect Wrap backend

program with a specific name for each block◦ Backend gets data from Kinect SDK and dumps it into

shared memory space when each new frame becomes available

◦ Our API Library copies frame from shared memory into local memory space for student program to use.

Shared Memory Use

Provides an easy-to-use interface for students while maximizing functionality

Functions:◦ getVideoFrame(rgb** pFrame)◦ getDepthFrame(short** pFrame)◦ getSkeleton(Point3D* pSkel)◦ setOutputDisplay(rgb** pFrame)◦ setOutputText(char* sText)◦ setPixel(rgb** pFrame, int x, int y, KCOLOR color)

Kinect Wrap API

Kinect Wrap Library is a static Windows library that students must include their C projects

Library handles backend processing of API Abstracts all of the technical workings of the

Kinect SDK, File Mapping API, and GUI design

Kinect Wrap Library

Abstracts away the details of drawing Kinect data via a graphical API

Kinect GUI receives data via shared memory Data is drawn as a grid of vertices, each

with a custom RGB value via DirectX Intend to port GUI code to OpenGL and work

on constructing a more intuitive GUI in future

Kinect GUI

Field Testing◦ Compared time delay of data transmission

between the current and previous iterations of the Wii Wrap code.

◦ Found the apparent delay seen in the Wii Wrap GUI to be significantly decreased.

◦ Test results corroborated by CPRE 185 lab TAs during classroom use.

◦ No previous functionality was affected by these changes.

Testing – Wiimotes

Assumed that the Kinect SDK functions deliver correct data from Kinect sensor

Shared Memory◦ Wrote known data into shared memory space with

one program and had second program read it out and verify that the data has not been altered

Student API Interface◦ Verified that buffers of known data sent to the

student program are not altered before being read◦ Verified that a known function run by student code

produces the expected output frame data given a known input frame

Kinect Black Box Testing

GUI Output◦ Tested whether frames were being updated

consistently by writing specific colors to known pixels in a given output frame

Kinect Black Box Testing

setPixel()◦ Tested for boundary values.◦ Attempted to set pixels outside the frame (640x480)◦ Attempted to set pixel RGB values outside of 0-255

getVideoFrame() & setOutputDisplay()◦ Check that output frame RGB data matches input data◦ Found that with simple student processing code, any

reduction in output fps from device frame rate is minimal getDepthFrame()

◦ Used a plane at a known distance from the Kinect to find how accurate the depth values in mm are vs. reality.

Kinect Integration Testing

int main()

{

KinectLib_Init(); // Initialize the library

rgb* frame[VIDEO_HEIGHT]; // Create 2D frame of rgb data

short* dframe[DEPTH_HEIGHT]; // Create 2D frame of short depth data

Point3D* skeleton; // Create array of skeleton data

while(1){

getVideoFrame(frame); // Map video frame to pointers

getDepthFrame(dframe); // Map depth frame to pointers

if(skeletonTracked()) {

skeleton = getSkeleton(); // Map Skeleton data

}

setPixel(frame, 23, 18, BLUE); // Arbitrarily set pixel (23, 18) to Blue

displayVideoFrame(frame); // Display the modified video frame in the GUI

}

KinectLib_Destroy(); // Destroy Kinect library instance

return 1;

}

Kinect Lab Example

Multi-Kinect Support More comprehensive GUI program Additional Library Functionality Create code to share Kinect data over

Ethernet

Future Work

Questions?seniord.ece.iastate.edu/dec1116/