collaborative robotics heads-up display major qualifying ...€¦ · collaborative robotics...
Post on 28-Jul-2020
4 Views
Preview:
TRANSCRIPT
Christopher Bove (RBE)
Alexander Wald (CS)
Advised by:
William Michalson (WPI, ECE)
Mark Donahue (MITLL, 76)
Jason LaPenta (MITLL, 76)
Collaborative Robotics Heads-up Display Major Qualifying Project
This material is based upon work supported by the Department of the Navy under Air Force Contract No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions,
findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of the Navy.
DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.
C. Bove, A. Wald 10/12/2016
Slide 2
Research Goals
• Support multiple platforms
• Robust, real-time data processing
• Intuitive interfaces for data interaction
• Intelligent autonomous behavior
Problem: Harness Talents of Humans and Autonomous Systems
• Dirty, Dull, and Dangerous Jobs
• Hazard Agnostic
• Enhanced Sensing - Produce LOTS of data!
• Remote Operations
• Expendable and Replaceable
Autonomous Systems
Humans
• Excellent High Level Decision Making
• Fast Task Distribution and Planning
• Ethical Decisions
• Adaptive in Uncertainty
Merge
via Hybrid
Human Robot
Teaming
Distill Information and Present Intuitively to User
C. Bove, A. Wald 10/12/2016
Slide 3
Relies on pose estimate to relate user with environment.
Augment Reality on Heads-Up Displays to Improve Collaboration and Situational Awareness
Display Augmented Reality Objects
to user through HUD.
C. Bove, A. Wald 10/12/2016
Slide 4
• Inertial Navigation System (INS) – Inertial Measurement Unit (IMU) and GPS
– IMU provides orientation: magnetometer, accelerometer, and gyroscope
– GPS for position
– Position unavailable with obstructed view of sky
• Simultaneous Localization and Mapping (SLAM) – Determine current position by examining world
– Use further examinations to build map
– Close the loop by recognizing previously visited areas
• Hector SLAM utilizing LIDAR scans – Measure ranges and particle filter for probable
locations, provides X,Y, Yaw
– Can utilize IMU or spinning LIDAR to develop 3D pose and mapping
– LIDAR has large SWaP, emits energy into environment
Variety of Methods for Obtaining Pose
Hector SLAM Utilizing LIDAR Sensor
C. Bove, A. Wald 10/12/2016
Slide 5
Monocular Visual Simultaneous Localization and Mapping
Oriented FAST and Rotated Brief (ORB) SLAM – Feature based mapping, sparse mapping – Robust pose determination and scale-drift
aware loop closure – Focus of work for MQP
Large Scale Direct (LSD) SLAM – Gradient pixel based stereo depth-matching – Dense point cloud generation, processor
intensive
C. Bove, A. Wald 10/12/2016
Slide 6
Problem: Maps with Monocular Vision Lack a Real Scale
Frame at t2
C. Bove, A. Wald 10/12/2016
Slide 7
Problem: Maps with Monocular Vision Lack a Real Scale
Possible
Distances
An unknown baseline between two
images results in an unknown
distance to objects.
Frame at t1 Frame at t2
Possible Baselines
Feature
C. Bove, A. Wald 10/12/2016
Slide 8
Envisioning the Future of Augmented Reality
Research Grade Equivalents to ODG Glasses + LIDAR for Truth
Objective System
Research System
Tasks for MQP:
Data Collection
Scale and Scale Drift Correction
Runtime Performance Evaluation
C. Bove, A. Wald 10/12/2016
Slide 9
Goal: Determine Position with Prototype Helmet Mapping System
Orthogonal
Color Cameras
Inertial Navigation
System
ODG HUD
Glasses
NVIDIA Jetson TX1
Embedded 15W
System On Chip
Portable Data Collection
tethered to Helmet
LIDAR (Truth Data) GPS Receiver
C. Bove, A. Wald 10/12/2016
Slide 10
The Big Picture Approach
Our Project
Dense 3D Point Cloud
C. Bove, A. Wald 10/12/2016
Slide 11
Fuse IMU Data into ORB SLAM to Correct Scale and Drift
Our Project
C. Bove, A. Wald 10/12/2016
Slide 12
Envisioning the Future of Augmented Reality
Research Grade Equivalents to ODG Glasses + LIDAR for Truth
Objective System
Research System
Tasks for MQP:
Data Collection
Scale and Scale Drift Correction
Runtime Performance Evaluation
C. Bove, A. Wald 10/12/2016
Slide 13
Collected Data to Check Assumptions and Evaluate Performance
• Tests of several motion patterns
– Static
– Linear
– Circle
– Square
– Erratic
– Head Turns
– Jumping
• Rotational Testing
– Increasingly fast rotations
• Sanity checks for coordinate systems
– XYZ translation
– Roll, pitch, yaw tests
– 360 degree RPY tests
C. Bove, A. Wald 10/12/2016
Slide 14
Collected Data to Check Assumptions and Evaluate Performance
• Tests of several motion patterns
– Static
– Linear
– Circle
– Square
– Erratic
– Head Turns
– Jumping
• Rotational Testing
– Increasingly fast rotations
• Sanity checks for coordinate systems
– XYZ translation
– Roll, pitch, yaw tests
– 360 degree RPY tests
C. Bove, A. Wald 10/12/2016
Slide 15
• Sources of Drift:
– Gravity
– DC biases
• Approach:
– Calculate gravity vector from orientation
– Calculate DC bias using a calibration phase
Implemented Gravity-Compensated Dead Reckoning on IMU
Accelerometer
DC Bias
Gravity Vector
C. Bove, A. Wald 10/12/2016
Slide 16
Envisioning the Future of Augmented Reality
Research Grade Equivalents to ODG Glasses + LIDAR for Truth
Objective System
Research System
Tasks for MQP:
Data Collection
Scale and Scale Drift Correction
Runtime Performance Evaluation
C. Bove, A. Wald 10/12/2016
Slide 17
Determined Transformations Between Systems
C. Bove, A. Wald 10/12/2016
Slide 18
Scaled Initial Baseline Estimate to Fix Scaling
Transform
Mocap or IMU
Position
ORB SLAM
Frames
Transform
Inject Extra
Pose Data into
Frames
Transform
Mocap or IMU
Position
ORB SLAM
Frames
Transform
Scale
Transformation
to Match
Scale Map Points
with Same Scaling
C. Bove, A. Wald 10/12/2016
Slide 19
• Modified ORB SLAM implementation
• Deliver inertial velocity and absolute orientation
Corrected Motion Model and Injected Absolute Orientation to Mitigate Drift
Mocap or IMU
Position
Last Pose x Velocity Mocap or IMU
Orientation
C. Bove, A. Wald 10/12/2016
Slide 20
Results
C. Bove, A. Wald 10/12/2016
Slide 21
IMU Gravity Compensated and Position Integrated
C. Bove, A. Wald 10/12/2016
Slide 22
Results: Coordinate Systems Correctly Aligned
Transform
Heading Error:
50.2 degrees
Correct Heading to Object
C. Bove, A. Wald 10/12/2016
Slide 23
ORB Scale Correction Achieved Accurate Results: ~10cm Error w/ Truth Data Injected
Heading Error:
2.66 degrees
Correct Heading to Object
C. Bove, A. Wald 10/12/2016
Slide 24
Scale Drift Correction Produced Mixed Results
Before Drift Correction After Drift Correction
Implemented Method to Correct Drift. Future Refinements are Necessary.
C. Bove, A. Wald 10/12/2016
Slide 25
Envisioning the Future of Augmented Reality
Research Grade Equivalents to ODG Glasses + LIDAR for Truth
Objective System
Research System
Tasks for MQP:
Data Collection
Scale and Scale Drift Correction
Runtime Performance Evaluation
C. Bove, A. Wald 10/12/2016
Slide 26
• Real time – System eventually on Jetson TX1
• Analyzed ORB SLAM run time performance
• Step 1: Compare each thread’s average cycle duration
Conducted System Performance Evaluation of ORB SLAM
Frame Received
Tracking Thread
Local Mapping
Loop Closing
Key Frame Key Frame
Loop Closing
Key
Frame Processing
Local Bundle Adjustment
Global Bundle Adjustment
C. Bove, A. Wald 10/12/2016
Slide 27
• Step 2: Analyze specific tasks in call hierarchy
Conducted System Performance Evaluation of ORB SLAM
Tracking Thread
Track
Monocular
Grab Image
Monocular
Cvt Color Track
Track Reference
KeyFrame
Track Local
Map
Track With Motion
Model
a) Find average duration for each task
b) Find total child time for each node
Frame
Extract ORB
Construct
Frame
C. Bove, A. Wald 10/12/2016
Slide 28
ORB Performance Evaluated and Bottlenecks Exposed
Track
Monocular
Grab Image
Monocular
Frame
Extract ORB
Construct
Frame
195 sec
230.5 sec
195 𝑠𝑒𝑐
230.5 𝑠𝑒𝑐= 0.8460 = 85% of run time
230.5 sec (total)
195 sec
𝟖𝟓% 𝐨𝐟 𝐓𝐨𝐭𝐚𝐥 𝐑𝐮𝐧 𝐓𝐢𝐦𝐞
…
…
C. Bove, A. Wald 10/12/2016
Slide 29
• Scale correction
– With truth, worked well
– With inertial position, dead reckoning drift impacted accuracy
– Staking the scale at one point does not introduce feedback
• Scale drift correction
– Created internal inconsistencies between perceived motion in map and estimated pose
– Produced jagged motion as estimates switched
– Need to optimize position of points or mix estimates externally
Conclusion: Passive Sensor Position Estimate Achievable
• Performance evaluation
– 84.60% of tracking run time spent on ORB feature extraction
– GPU optimized calls exist
C. Bove, A. Wald 10/12/2016
Slide 30
Future Improvements to Provide Robustness
Frame Filter
C. Bove, A. Wald 10/12/2016
Slide 31
Thank you!
C. Bove, A. Wald 10/12/2016
Slide 32
• https://www.ptgrey.com/flea3-20-mp-color-usb3-vision-e2v-ev76c5706f-3
• https://developer.nvidia.com/sites/default/files/akamai/embedded/images/jetsontx1/JetsonTX1_DevKit_3Quarter_ON.png
• http://www.govconnection.com/shop/samsung-512gb-ssd-950-pro-pcie-3.0-nvme-m.2-2280-internal-solid-state-drive/30785312?cac=CartItem
• http://www.hokuyo-aut.jp/02sensor/07scanner/download/products/utm-30lx-ew/
• https://www.hokuyo-aut.jp/02sensor/07scanner/img/utm_30lx_top.jpg
• https://www.microstrain.com/sites/default/files/styles/larger__550x550_/public/gx4-45_0.jpg?itok=R4F6eE8L
• https://www.inition.co.uk/wp-content/uploads/2014/09/s250e1.png
• http://www.digitalartsonline.co.uk/cmsdata/reviews/3444105/MaxPNG688.jpg
Photograph Links
C. Bove, A. Wald 10/12/2016
Slide 33
Monocular Vision Accumulates Errors without Loop Closure
Scale-Drift Aware Loop Closure
And Optimization Corrects Path
C. Bove, A. Wald 10/12/2016
Slide 34
Sensors Capable of Providing Positions
Passive Sensors are Ideal for Use
in the Field and Mobile Systems
• LIDAR Systems:
– Directly provide accurate position data
– High cost
– Emit laser infrared energy
• Inertial Measurement Units:
– No energy emission
– Low power
– Low weight
– Only provide linear acceleration and angular velocity
• Global Position Systems:
– Accurate positioning on Earth
– Passive, could be interrupted
• Vision Systems
– No energy emission
– Low power
– Low weight
– Requires extensive computer processing
top related