Hand Gesture Based User Hand Gesture Based User InterfaceInterface

For Generic Windows For Generic Windows ApplicationsApplications– Computer InterfaceComputer Interface

• A 2D/3D input device based on real time hand trackingA 2D/3D input device based on real time hand tracking• Translation of gestures to Windows commandsTranslation of gestures to Windows commands• More natural interface for 3D design applicationsMore natural interface for 3D design applications

– Aid for disabled and deaf peopleAid for disabled and deaf people• Sign language analysis, translationSign language analysis, translation• Sign language teachingSign language teaching• Using multimodality to enable communication between disabled peopleUsing multimodality to enable communication between disabled people

– EntertainmentEntertainment• Applications involving hand gesture recognition, from computer games to sound and Applications involving hand gesture recognition, from computer games to sound and

image design applicationsimage design applications

LeftLeftImageImage

3D Reconstruction3D Reconstruction

3D Kalman Filter3D Kalman Filter

Vector QuantizerVector Quantizer

Gesture SpotterGesture Spotter

ApplicationApplicationRightRightImageImage

2D Kalman Filter2D Kalman Filter

MarkerMarker SegmentationSegmentation

CapabilitiesCapabilities• Camera calibration and gesture training toolsCamera calibration and gesture training tools• Registration of new markers through the interfaceRegistration of new markers through the interface• Recognition of 2D and 3D gestures in real time, modeled with HMMsRecognition of 2D and 3D gestures in real time, modeled with HMMs• Real time 2D and 3D Kalman filtering of the trajectoriesReal time 2D and 3D Kalman filtering of the trajectories• Support for two hands wearing different colored glovesSupport for two hands wearing different colored gloves• Real time mouse simulation based on hand trackingReal time mouse simulation based on hand tracking• Capability of triggering windows input device events – controlling third party applicationsCapability of triggering windows input device events – controlling third party applications• Generic interface that can be linked to any application through configurationGeneric interface that can be linked to any application through configuration

RequirementsRequirementsHand Gesture Interface is specifically designed to not require Hand Gesture Interface is specifically designed to not require any special hardwareany special hardwareHardware RequirementsHardware Requirements• Smooth colored hand glove as the markerSmooth colored hand glove as the marker• Simple WDM or VFW based camera(s)Simple WDM or VFW based camera(s)• Good lighting conditionsGood lighting conditions• Mainboard should be able to handle two cameras for the 3D modeMainboard should be able to handle two cameras for the 3D modeSoftware RequirementsSoftware Requirements• Windows Operating System (Tested on XP and Me)Windows Operating System (Tested on XP and Me)

ExecutionExecution• The user registers the marker by waving the hand to the camera(s)The user registers the marker by waving the hand to the camera(s)• The user trains the system any 2D or 3D gesture (2D uses only x-y)The user trains the system any 2D or 3D gesture (2D uses only x-y)• The user modifies or replaces the configuration file to link it to any The user modifies or replaces the configuration file to link it to any

application and mouse or keyboard eventsapplication and mouse or keyboard events• The user starts controlling the interfaceThe user starts controlling the interface

RemarksRemarks• Gestures are only recognized when the hand is held verticallyGestures are only recognized when the hand is held vertically• Keeping the hand horizontal simulates the left click of the mouseKeeping the hand horizontal simulates the left click of the mouse• The ends of a gesture are automatically found by the system.The ends of a gesture are automatically found by the system.• The system comes with 8 pre-trained gestures, which can be The system comes with 8 pre-trained gestures, which can be

changed anytimechanged anytime• The system comes with a pair of calibration matrices, which should The system comes with a pair of calibration matrices, which should

work with most camera setups. If they don’t, the calibration tool work with most camera setups. If they don’t, the calibration tool needs to be usedneeds to be used