multimo3-d: a testbed for multimodel 3-d pc presenter: yi shi & saul rodriguez march 14, 2008
Post on 15-Jan-2016
213 views
TRANSCRIPT
MUltimo3-D: a Testbed for MUltimo3-D: a Testbed for Multimodel 3-D PC Multimodel 3-D PC
Presenter: Yi Shi & Saul RodriguezPresenter: Yi Shi & Saul Rodriguez
March 14, 2008March 14, 2008
OutlineOutline
► Motivations and conceptsMotivations and concepts
► Operating SystemOperating System
► 3-D Display3-D Display
► Input DevicesInput Devices
► ApplicationsApplications
► DiscussionDiscussion
Motivations and concepts in 3D PCMotivations and concepts in 3D PC
► 3D display3D display
► Contact free on human bodyContact free on human body
► Higher bandwidth: hands, head, eyes, etc.Higher bandwidth: hands, head, eyes, etc.
► Intelligent intention interpretationIntelligent intention interpretation
► Less human strainLess human strain
Operating System of mUltimo3-DOperating System of mUltimo3-D
► Developing toolDeveloping tool
Based on a C++ class library, which is based on Based on a C++ class library, which is based on
and extended the virtual reality development and extended the virtual reality development
environment dVS/DIVISION from PTC Ltd.environment dVS/DIVISION from PTC Ltd.
Operating System of mUltimo3-D Operating System of mUltimo3-D (cont.)(cont.)
► Interface agencyInterface agency Collector agentCollector agent
► Collects and interprets data on the users and makes this Collects and interprets data on the users and makes this
information available to other agents.information available to other agents.
► All the data collected is stored in the form of user model for All the data collected is stored in the form of user model for
later uselater use
Placer agentPlacer agent► Arranges the media objects in the interaction space.Arranges the media objects in the interaction space.
► It avoids overlapping and collision of objects.It avoids overlapping and collision of objects.
► Gradually move away and remove obsolete objectsGradually move away and remove obsolete objects
Operating System of mUltimo3-D Operating System of mUltimo3-D (cont.)(cont.)
► Interface agency (cont.)Interface agency (cont.) Starter agentStarter agent
► Automatic starts applications.Automatic starts applications.
Visualization agentVisualization agent► Critical action confirmation, undo/redo.Critical action confirmation, undo/redo.
► Status display: planning, working, satisfied, surprised, Status display: planning, working, satisfied, surprised,
confusedconfused
Figure 1. Communication states of the avatar: idle state, confirmation request and inquiry.
3-D Display of mUltimo3-D3-D Display of mUltimo3-D
► Autostereoscopic displaysAutostereoscopic displays
Lenticular lens based 3D display.Lenticular lens based 3D display.
Figure 2. 3D display using an array of lenticular lenses to channel the left and right eye images to the respective eye (L and R refer to image stripes of the left and right images of a stereo pair).
3-D Display of mUltimo3-D (cont.)3-D Display of mUltimo3-D (cont.)
► Face to FaceFace to Face
The display screen is placed on a platform that turns The display screen is placed on a platform that turns
around to continuously orient the screen to the user’s around to continuously orient the screen to the user’s
eyes.eyes.
The L/R picture changes when the screen moves which The L/R picture changes when the screen moves which
avoids the spatial and optical distortions and makes the avoids the spatial and optical distortions and makes the
view angle wider.view angle wider.
3-D Display of mUltimo3-D 3-D Display of mUltimo3-D (cont.)(cont.)
► Full focus problemsFull focus problems Physical and virtual positions are different, thus undesirable Physical and virtual positions are different, thus undesirable
when supporting direct virtual object manipulation with for when supporting direct virtual object manipulation with for
example, hands.example, hands.
Figure 3. With conventional 3D displays, interaction between virtual and real objects in the grasp area is affected by blurred effects, either the virtual object is sharp (left) or the real object is sharp (middle), depending on what object the user is looking at. On the accommodation display both objects are seen in full focus (right).
3-D Display of mUltimo3-D 3-D Display of mUltimo3-D (cont.)(cont.)
► Possible solutionPossible solution
Accommodation 3-D displayAccommodation 3-D display
Figure 4. The user accommodates and perceives object on the aerial image plane. The Fresnel lens projects the exit pupils of the stereo projector to the left and right eye, respectively (field-lens principle) and, hence, completely separates the constituent stereo images (in the prototype system there is virtually no crosstalk).
Input Devices of mUltimo3-DInput Devices of mUltimo3-D
► Multiple input Multiple input signalssignals
Figure 5. The overall system diagram of the mUltimo3D testbed
Input Devices of mUltimo3-DInput Devices of mUltimo3-D
► Multiple input signalsMultiple input signals
Figure 6.1. The input devices of mUltimo3-D
Input Devices of mUltimo3-DInput Devices of mUltimo3-D
► The mUltimo3-D systemThe mUltimo3-D system
Figure 6.2. The mUltimo3-D system
Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)
►Voice inputVoice input IBM Via VoiceIBM Via Voice is integrated in the mUltimo3-D. is integrated in the mUltimo3-D.
Logox3 speech synthesis software from G-DATA is used as Logox3 speech synthesis software from G-DATA is used as
speech outputspeech output
Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)
► Video head trackerVideo head tracker Skin color is used as tracking featureSkin color is used as tracking feature
Eye blinking is used to locate the positions of eyes on the Eye blinking is used to locate the positions of eyes on the
headhead
Able to cope with changes in size, orientation and Able to cope with changes in size, orientation and
illumination.illumination.
(demo)(demo)
Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)
► Video hand trackerVideo hand tracker Two cameras and infrared are used to capture and Two cameras and infrared are used to capture and
segment handssegment hands
Figure 7. Video hand tracker (the device attached to the keyboard) at the mUltimo3D user interface and the segmented hand.
Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)
► Video hand trackerVideo hand tracker Hand center is determined and Hand center is determined and radial lines are radial lines are
derived from the centerderived from the center
Different line features are used to recognize the hand Different line features are used to recognize the hand
shape pattern as well as movementshape pattern as well as movement
(demo)(demo)
Input Devices of mUltimo3-D (cont.)Input Devices of mUltimo3-D (cont.)
► Video gaze trackerVideo gaze tracker Used cornea-reflex methodUsed cornea-reflex method
Low-intensity infrared lightLow-intensity infrared light
Center of the pupil and the reflection of the light from the Center of the pupil and the reflection of the light from the
cornea are found to determine the direction of the eyes gazing cornea are found to determine the direction of the eyes gazing
atat
(demo)(demo)
Figure 8. The eye image captured by the gaze tracker (left) and an enlarged part of the eye showing the pupil and light reflections on the cornea (right).
Applications in mUltimo3-DApplications in mUltimo3-D
Figure 9. The placer agent arranges the display of the media objects in the interaction space. The tool objects (cubes) show icons for gaze-controlled selection of applications. The cubes turn overproportionally when the head moves, so that all sides become visible with slight head movements.
Applications in mUltimo3-D (cont.)Applications in mUltimo3-D (cont.)
Figure 10. Screenshots of the Internet browser (left) and the multimodal 3D CAD application (right). By changing the viewing perspective (head movement), hidden objects can be viewed and addressed by eye gaze. The 3D CAD object shelf covering parts of the video windows has currently been moved to the front by gaze interaction.
DiscussionDiscussion
► Combining and optimizing multimodel information Combining and optimizing multimodel information in the best wayin the best way
► Real-time database updatingReal-time database updating
► More user testing and feedbackMore user testing and feedback
Thank you!Thank you!