projection mapping implementation
TRANSCRIPT
Projection Mapping ImplementationEnabling Direct Externalization of Perception Results
and Action Intent to Improve Robot Explainability
Zhao Han, Alexander Wilkinson, Jenna Parrillo,Jordan Allspaw, Holly A. Yanco
November 13, 2020 AI-HRI 2020
IntroductionImproving understanding of robots
● Improves trust in real-time● Leads to greater efficiency during more
difficult human-robot collaboration scenarios
Due to embodiment, HRI researchers focused on
● Enabling robots to communicate their intention through non-verbal means● E.g., eye gaze, arm movement
(Desai et al. HRI ‘13)
(Admoni et al. HRI ‘16)
(Moon et al. HRI ‘14)(Admoni, Scassellati. JHRI ‘17)
(Dragan et al. HRI ‘13)(Kwon et al. HRI ‘18)
2
ProblemNon-verbal cues in human-life
● Plays an important role in supporting and improving communication
But can also cause confusion
E.g., how can a robot communicate
● Which objects it has detected?● Which one is it going to grasp?
Use eye gaze and arm motion?
● Pointing can be vague
Verbal explanation?
● Gestures can still be underspecified● It requires follow-up questions
(Admoni et al. AI-HRI ‘15)
3
One Approach: Projection MappingProjection mapping of perception results and action intent
We present a tool for
● Implementing projection mapping● Using an off-the-shelf projector
We will discuss
● The high level architecture● Low level technical details● Hardware (robot) platform
Objectsdetected
Object to grasp
Open-sourced on GitHub: github.com/uml-robotics/projection_mapping
4
Why Projection Mapping Implementation?Projection mapping in robotics is not a new idea, but rather relatively rare
● Most are simple navigation projection● Few are projection mapping
Implementation effort has been missing
● Blocks knowledge of the effects of accurate externalization through it
● Makes it hard to compare it with other methods
Chadalavada et al., ECMR ‘15.
Andersen et al., RO-MAN ‘16.
Watanabe et al., ROS ‘15.
Coovert et al., Computers in Human Behavior ‘14.
6
Why Projection Mapping?1. Direct and accurate externalization
● Objects are directly externalized○ Perceived or to be
manipulated
● Completely removes the need for mental inference○ cf. non-verbal methods and
verbal explanations
2. More salient
● Compared to other direct methods, e.g., display screen
● Projection can be seen farther away
3. Eliminates mental mapping
● From another media, e.g., display screen○ Can cause misjudgment and○ Lead to undesired consequences
7
Implementation Overview
Object Clusters (ROS point cloud pub.)
Rviz Visualization
Virtual Camera (Image pub.)
4. Subscribes to
3. Rviz plugin(rviz_camera_stream)
Image Viewer(Full screen GUI)
Projector Lens Intrinsics
(CameraInfo pub.)
1. Calibration
2. Subscribes to 6. Subscribes to
5. Virtual world
7. Output (HDMI)Pose is the same as projector’s(Pub.: ROS publisher)
Interactive Markers(Other displays in Rviz)
(Alternatives)
8
Why This Projection Mapping Implementation Works?
Principal: a projector is the dual of a camera
● Camera maps 3D world coordinates to 2D image plane coordinates● Projector maps 2D image plane coordinates to 3D world coordinates
● In a camera, light rays from the world pass through the lense and hit sensor● In a projector, light rays pass through the lens and hit a surface in the world
The virtual camera with the same intrinsics and pose as the real projector
● Allows us to map points from the virtual world’s 3D space to the 3D space in the real world
Low level details next... 9
Projector Selection ConsiderationProjection must be visible and legible under bright light conditions
● Unlike common uses: present slides and watch movie (lights are off or dimmed)● Robots often operate indoors with lights on or outdoor under sunshine
Brightness - Standard measure is ANSI lumens, total light output per unit of time (e.g., 3,800)
Contrast - Ratio between the darkest and lightest areas of an image (e.g., 22,000:1)
Projection technology - Choose DLP over LCD for higher brightness● DLP is based on Digital Micromirror Device (DMD)● It has a reflective and high-fill factor digital light switch
○ Active-matrix LCDs are transmissive, generated heat cannot be dispatched well - Hornbeck 1997
ViewSonicPA503W
10
Virtual Camera: Connecting virtual & real worldsOur lab developed rviz_camera_stream Rviz plugin
● It outputs a 2D image of what a camera sees in the Rviz virtual world○ Open sourced at github.com/uml-robotics/rviz_camera_stream
The virtual camera is placed at the real projector’s pose
The pose is specified to the same robot transform hierarchy as perception sensor
● Everything in Rviz can be transformed from the perception sensor’s frame to the one of the virtual camera, such as point clouds and interactive markers.
● This allows the projection to be in the projector’s viewpoint of what the perception sensor sees.
11
Projection Outputrviz_camera_stream publishes the image from the virtual camera
We used the image_view ROS package to subscribe and show the image
The image viewer GUI is made full screen
● No additional window managers(e.g., i3) are used
● In Ubuntu, one can enable the“Toggle full screen” keyboard shortcut to remove the window chrome
12
Hardware PlatformWe demonstrate our implementation on a Fetch Robot
● A custom structure to mount the projector○ If mounted on its head, the neck won’t
bear the weight● To pan and tilt the projector, we attached a
ScorpionX MX-64 Robot Turret Kit
The projection mapping technique isrobot agnostic
13
Being Robot AgnosticApplied it on an assistive robot before
Minimum Requirement: Projector frame integrated into robot’s TF tree
● Our setup: projector lens & attachment point (intermediate frame)
It does not mean that the projector must be attached to the robot; it only needs to be co-located with the robot● Convenient for robot arms
Wang et al., ISRR ‘19.14
Steps to Implement Projection Mapping for Robotics
15
2. Get lens intrinsics
- Calibrate lense (guide on GitHub) - Publish it in a CameraInfo message
4. Add Rviz displays
- Point cloud clusters - Any other displays, e.g., interactive markers
1. Set up projector
- Buy and mount projector - Calculate transform
5. Output projection
- Point projector w/ turret unit- Output image w/ image_viewer
3. Set up virtual camera
- Use rviz_camera_stream - Subscribe to the CameraInfo message
Thanks!
Open source github.com/uml-robotics/projection_mapping
Zhao Han [email protected] www.cs.uml.edu/~zhan
Lab website: robotics.cs.uml.edu
Projection Mapping Implementation: Enabling Direct Externalization of Perception Results and Action Intent to Improve Robot Explainability
Zhao Han, Alexander Wilkinson, Jenna Parrillo, Jordan Allspaw, and Holly A. Yanco
17
Why This Projection Mapping Implementation Works?
Principal: a projector is the dual of a camera
● Camera maps 3D world coordinates to 2D image plane coordinates
● Projector reverses this process
● In a camera, light rays from the world pass through the lense and hit sensor● In a projector, light rays pass through the lens and hit a surface in the world
The virtual camera with the same intrinsics and pose
● allows us to map points from the virtual world’s 3D space to the 3D space in the real world.
Low level details next... 18
More at High-LevelThe code repository1 includes
● sample Rviz config file,
● pcd file (point cloud),
● launch file to publish CameraInfo message and the projector pose,
● A well-documented readme file
1. github.com/uml-robotics/projection_mapping
19
Steps to Implement Projection Mapping for Robotics
- Buy and install - Calculate transform
1. Set up projector
- Point projector w/ turret unit - Output image w/ image_viewer
5. Output projection
- Use rviz_camera_stream - Subscribe to the CameraInfo message
3. Set up Virtual camera
- Point cloud clusters - Any other displays, e.g., interactive markers
4. Add Rviz displays
- Calibrate lense (guide on GitHub) - Publish it in a CameraInfo message
2. Get lens intrinsics
20