handy fb (gesture recognition and facebook manipulation project)
DESCRIPTION
Used gesture recognition to perform events like " chat, write on wall , like this and poke" on face-book look alike page.TRANSCRIPT
Handy Facebook(Hand gestures to manipulate social networking website)
Team Members
Jaskaran Uppal (0419)
Sandeep Mallela (9769)
Darpan Dhamija (0550)
Rahul Perhar (4562)
Project Objective
Identify hand gestures in front of a webcam
Navigate the website depending on the gestures recognized
Tasks to be performed
• Making of gestures in front of the camera
• Gesture detection at a suitable frame rate
• Capturing the gestures and storing them in a .jpg file
• System training to recognize the gestures with a low error rate
• Execution of events upon the successful gesture recognition on the webpage
• Notification to be sent to the user
Gesture Making
Usage of a small set of gestures (fingers).
Every finger raised will perform some predefined navigation of the webpage
System capabilities can be programmed to accommodate other human gestures as well
Error in detection can be reduced by training
Gesture Detection
Gestures are detected at a suitable frame rate.
The camera captures the hand gesture and we apply canny edge detection algorithm to store the gestures in the following format
System Training
System training is done using “Neuroph” an open source Image Recognition tool that takes images as input and produces a neural network.
This Neural network can be trained to recognize the gestures
This can be used with Java Classes to be integrated in our application, using plug-in provided with the tool
Website Navigation
The default page shown to the user
User makes gesture
System recognizes
Website navigates
Facebook profile loaded
Furthermore the user can use other gestures to navigate though additional WebPages
Website Navigation Contd.
After initial gesture recognition, user is navigated to a personal profile page where he is given additional options
The user can make gestures to perform either of the actions
1 Chat
2 Write on wall
3 Like a post
4 Poke a person
Implementation Details
The application is implemented using the following:
OpenCV libraries for gesture recognition code
Using Java to capture the image and convert it into a BufferedImage for easy processing
Neuroph tool is used to train the system
The output from Neuroph is the recognition of Gesture upon which we have actions defined
Results
Home Page
Results
Personal Page of a user
Results
Opening Chat for a user
Results
Writing on the wall of a user
Results
“Like” a user post
Results
“Poke” a user
Limitations
The Limitations to the system includes the following:
The error rate in gesture recognition is persistent
It is a Lo-Fi prototype of what can be done on a larger scale further improvements can be done
Gesture recognition is dependent upon on available light.
Future Additions
Improvement in Hand gesture recognition. Making the system more refined and gestures easily recognizable
We can Integrate this into a number of applications like Google maps to get the address of a particular place.
A lot more different gestures can be used and trained in the system
We can have a real chat window in the future
Credits & References
Prof. Suya You, for all the support and knowledge of various User Interface Designs
Vijayakumar Gopalakrishnan, TA for giving an initial idea and helping us in realizing the project till the completion
Neuroph and related documentation for gesture recognition (http://neuroph.sourceforge.net/documentation.html)
Thank You