wii3d – extending the wiimote into 3d
DESCRIPTION
Final Honours Presentation. Wii3D – Extending the Wiimote into 3D. Principal Investigator: João Lourenço Supervisor: Dr Hannah Thinyane. Presentation Outline. Introduction Design and Implementation User Study Future Work Conclusion. Introduction. Problem Statement - PowerPoint PPT PresentationTRANSCRIPT
Final Honours Presentation
Principal Investigator: João LourençoSupervisor: Dr Hannah Thinyane
Presentation Outline Introduction Design and Implementation User Study Future Work Conclusion
2
Introduction Problem Statement Research Motivation and Goals
3
Introduction Problem Statement Research Motivation and Goals
4
Problem Statement Migration from 2D output to 3D output
3D TVSurround sound speakers
Suggests a need for migration from 2D input to 3D inputMicrosoft Kinect (previously Project Natal)Wii3D
5
Introduction
6
Research Motivation and Goals Intuitive 3D Interaction Techniques are
needed Develop 3D Interaction using Nintendo
Wii Remote Compare Proposed interaction
technique with existing techniques
7
Design and Implementation Hardware Software
OS and FrameworksBluetoothLibrariesPoint TrackerGestures and Gesture RecognitionUser Interface
8
Design and Implementation Hardware Software
OS and FrameworksBluetoothLibrariesPoint TrackerGestures and Gesture RecognitionUser Interface
9
Hardware 2 Nintendo Wii Remotes Personal Computer with Bluetooth
Capabilities 2 WiiGloves
10
Nintendo Wii Remotes Layout Maximizing Volume
11
Nintendo Wii Remotes Layout Maximizing Efficiency
12
WiiGlove Circuit
2 Configurations2 IRs per hand4 IRs on one hand
13
Design and Implementation
14
OS and Frameworks Windows 7 Microsoft C#.NET 4.0
15
Design and Implementation
16
Bluetooth Microsoft Windows built-in Bluetooth
Stack
17
Design and Implementation
18
Libraries WiimoteLib 1.7.0.0 – Wiimote
Connection Accord.NET 2.1.1.0 – Statistics Math.NET Iridium 2008.8.16.470 –
Linear algebra 32feet.NET 2.2.0.0 – Bluetooth
Connection
19
Design and Implementation
20
Point Tracker Point tracking and probabilistic estimation
were required to ensure that the system would handle multiple inputs (multitouch) and continue to function even in the event of the loss of points. A loss of points can occur for several reasons:An infra-red source can leave the field of view of
the Wii RemoteAn infra-red source can move behind another
infra-red sourceTwo infra-red sources can come together and
merge
21
Point Tracker Several Components
Stereoscopic Triangulation SolverDistance MinimizerPredictive Interpolator
22
Stereoscopic Triangulation Achieved by solving for the point at the
near intersection of two rays Camera intrinsic properties needed
Viewport widthViewport heightHorizontal FOVVertical FOV
23
Stereoscopic Triangulation
24
Stereoscopic Triangulation Using the fields of view, a vector from the
camera position to the detected point was obtained. These vectors can be thought of as rays from the infra-red source to the camera.σ the horizontal field of viewψ be the vertical field of viewx be the horizontal component of the two
dimensional coordinatey be the vertical component of the two
dimensional coordinate
25
Stereoscopic Triangulation
26
Stereoscopic Triangulation With the two rays from the two Wii
Remote Cameras, the intersection of the two rays can be calculated using linear algebra.
27
Stereoscopic Triangulation
28
Stereoscopic Triangulation Physical System – rays would never
perfectly intersect Need to take an estimate
Estimate distance along first vectorEstimate distance along second vectorAverage the two points and make note of
the distance between them
29
30
31
Distance Minimization The points that are reported by the two
Wii Remotes are not necessarily sorted in the same order
32
Distance Minimization Pairwise permutation of points in matrix
form Minimize distance error per row
33
Predictive Interpolation Try to predict where a point should be –
expected pointsInstantaneous velocityAverage velocity over a period
Minimize the errors between the observed and expected points
34
Predictive Interpolation
35
Predictive Interpolation The selected point per row is used as
the point that the Wii3D System reports
36
Design and Implementation
37
Gestures and Gesture Recognition Supported gestures Recognition methods
38
Click Gesture A finger going down and forward, and
then up and backward
39
Pan/Scroll Gesture Two fingers that are close together
moving in similar directions
40
Zoom Gesture Two fingers moving in opposite
directions with the distance between them increasing
41
Rotate Gesture Two fingers moving in opposite
directions with the distance between them constant
42
Circle Gesture Hidden Markov Model Gesture
43
Multitouch Gestures Number of points used for each gesture
has been linked to the intensity of the gesture
44
Recognition Movement – polling Simple gestures – Finite State Automata Complex Gestures – Hidden Markov
Models
45
Design and Implementation
46
User Interface States:
47
User Interface Upon start-up, the Wii3D System
attempts to connect to the Wii Remotes
48
User Interface Two possible errors – not 2 Wiimotes
connected or no Wiimotes found
49
User Interface Search for devices
50
User Interface Found devices
51
User Interface Allow the devices to install
52
User Interface Connection success
53
User Interface Start polling
54
User Interface Currently polling
55
User Interface Events received
56
User Study Goals of the Experiment Design and Methodology
Tasks Results
57
User Study Goals of the Experiment Design and Methodology
Tasks Results
58
Goals of the Experiment To investigate whether the Wii3D
System is a useful system To investigate whether the Wii3D
System could be used as an alternative to the mouse and keyboard
59
User Study
60
Design and Methodology 20 participants 15 minutes per user User consent required No individual data stored, only collated
results Supervised by researcher
61
User Study
62
Tasks Introductory Questions Tasks Overview Questions
63
Introductory Questions How long have you been using a mouse
and keyboard for?8 years
How often do you use a mouse and keyboard?3 hours/day
64
Task 1 – Pointer Movement
65
Move pointers over targets
Task 2 - Clicking Move over targets Click gesture
66
Task 3 – Panning/Scrolling Pan/Scroll and Image
67
Task 4 – Zooming Zoom into and out of an image
68
Task 5 - Rotating Rotate an image about the Z and X axes
69
Task 6 – Complex Gesture Hidden Markov Models were used Circle gesture
70
Overview Questions Would you use the Wii3D System as an
alternative to the mouse and keyboard? Why or why not?
If no, do you believe that the Wii3D System would be useful to someone else? Why?
Would you add/remove/change anything in the Wii3D System? Why?
You have undoubtedly had more experience using a mouse and keyboard than this type of system. What tasks would you find Wii3D more suitable for than a mouse and keyboard?
71
User Study
72
Results Preliminary Observations Principal Results Participant Subjective Feedback Other Findings
73
Preliminary Observations Confusion between rotation and zoom
gestures People are more used to mouse and
keyboard Hidden Markov Models not performing
efficiently Hidden Markov Models only successfully
recognizing a circle gesture 50% of the time
74
Principal Results None of the users would use Wii3D as a
replacement for a mouse and Keyboard:Lack of typing supportGestures are not always recognizedAccuracy of the Wii3D System pointers was
not as good as a mouse
75
Principal Results The majority of the participants stated
that they thought that the Wii3D System would be useful
76
Movement Mouse better
for a single pointer
Wii3D better when there are more pointers
77
Clicking Mouse better
for single pointer clicks
Wii3D slightly better for multiple pointer clicks
78
Panning/Scrolling Panning and scrolling is preferred with a
mouse Average and above ratings for Wii3D
79
Zooming Zooming is preferred with the mouse Best ratings out of all the gestures for
Wii3D
80
Rotating Preferred,
again, is the mouse
Many users made zoom gestures instead
81
Complex Gesture Circle gestures were
recognized correctly 50% of the time using Hidden Markov Models
This result is reflected in the ten participants who rated this task as “Very Unintuitive”
82
Participant Subjective Feedback Users thought that the Wii3D System
would be useful for gaming and other three dimensional interactions
The users want typing functionality
83
Other Findings The longer people have used
computers, the more time they spend on them daily
84
Future Work Improvements Additions
85
Future Work Improvements Additions
86
Improvements Multiple Wii Remotes Acceleration in predictive interpolation Dynamic Hidden Markov Models Gaussian Mixture Models in HMM
(continuous)
87
Future Work
88
Additions Gestures
Double ClickFlickScaling an ObjectTyping
Artificial Neural Networks for Recognition
89
Conclusion There is a use for the Wii3D System The Wii3D System would not work as a
replacement for the traditional mouse and keyboard
90
Questions/Comments
91