yuping lin and gérard medioni. introduction method register uav streams to a global reference...
TRANSCRIPT
![Page 1: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/1.jpg)
Yuping Lin and Gérard Medioni
![Page 2: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/2.jpg)
Introduction Method
Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic
Synchronization of multiple video streams Conclusion
![Page 3: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/3.jpg)
Input: Multiple UAV video streams Position of moving objects in each video
stream
Goal: Synchronize using a common moving object
![Page 4: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/4.jpg)
Register UAV streams to a global reference image (a map), then
Synchronize the streams using the unique path of a common moving object on the map
![Page 5: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/5.jpg)
Introduction Method
Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic
Synchronization of multiple video streams Conclusion
![Page 6: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/6.jpg)
Input: Global reference image (Map) UAV stream The homography of the first frame of the
UAV stream to the map
![Page 7: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/7.jpg)
ISSUES
UAV images and the map are different in terms of viewpoints, sensors and time of capture
Direct matching is difficult
APPROACH
Given the homography of the first UAV frame to the map,
Two step registration Consecutive UAV image
registration, then UAV to Map registration
![Page 8: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/8.jpg)
Introduction Method
Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic
Synchronization of multiple video streams Conclusion
![Page 9: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/9.jpg)
Method: extract features in each frame Establish feature correspondences
between consecutive images estimate the transformation
![Page 10: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/10.jpg)
ISSUES
Features should be descriptive for matching and sufficient to give good transform estimation
Feature matching
Transform estimation
APPROACH
SIFT feature extraction 128 dimension feature
descriptor Avg. 2000 features in each
image Nearest neighbor
matching Avg. 1000 matches in each
pair of images RANSAC
Avg. 600 inliers in each pair of images
![Page 11: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/11.jpg)
Illustration
![Page 12: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/12.jpg)
Illustration
![Page 13: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/13.jpg)
Illustration
![Page 14: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/14.jpg)
Illustration
MiH ,1
![Page 15: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/15.jpg)
Illustration
MiH ,1 1,,1', iiMiMi HHH
![Page 16: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/16.jpg)
Problem: error is accumulated
i
kkkMiiMiMi HHHHH
1,1,01,,1,
![Page 17: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/17.jpg)
Introduction Method
Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic
Synchronization of multiple video streams Conclusion
![Page 18: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/18.jpg)
Method: Perform local search for
correspondences between the UAV image and the map
![Page 19: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/19.jpg)
ISSUES
UAV images are very different from the map, SIFT features cannot always match
APPROACHES
Sample points in the map
For each point, locally search for the most similar image patch in the UAV image
Use Mutual Information as similarity measurement
![Page 20: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/20.jpg)
Illustration
![Page 21: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/21.jpg)
Illustration
',MiH
![Page 22: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/22.jpg)
Illustration
',MiH
H
![Page 23: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/23.jpg)
Illustration
![Page 24: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/24.jpg)
Illustration
',, MiMi HHH
![Page 25: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/25.jpg)
Introduction Method
Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic
Synchronization of multiple video streams Conclusion
![Page 26: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/26.jpg)
Method: Perform consecutive UAV image
registration and UAV to Map registration iteratively▪ Consecutive UAV image
registration produce good initials for UAV to Map registration▪ Register the partial local
mosaic to the map
![Page 27: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/27.jpg)
ISSUES
Correspondences in a single frame are not enoughRegistration is unstable
APPROACH
Multiple frames in a time window forms a partial local mosaic which spans a larger region and provides more correspondences More robust Smooth transition
![Page 28: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/28.jpg)
ISSUES
Correspondences in a single frame are not enoughRegistration is unstable
APPROACH
Multiple frames in a time window forms a partial local mosaic which spans a larger region and provides more correspondences More robust Smooth transition
![Page 29: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/29.jpg)
Result
Register single frame
Register partial local mosaic
![Page 30: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/30.jpg)
Illustration
![Page 31: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/31.jpg)
Illustration
![Page 32: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/32.jpg)
Illustration
![Page 33: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/33.jpg)
Illustration
![Page 34: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/34.jpg)
Result
![Page 35: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/35.jpg)
Introduction Method
Register UAV streams to a global reference image▪ Consecutive UAV image registration▪ UAV to Map registration▪ Interleaving image to image and image to map▪ Partial local mosaic
Synchronization of multiple video streams Conclusion
![Page 36: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/36.jpg)
Input: UAV image sequences of different views, different frame rates, but capture the same area and overlap in time
An moving object on the ground plane which serves as a “clock” to synchronize the sequences
![Page 37: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/37.jpg)
The moving object should generate a single path on the map
Use sequence alignment algorithm to synchronize the UAV streams
![Page 38: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/38.jpg)
![Page 39: Yuping Lin and Gérard Medioni. Introduction Method Register UAV streams to a global reference image ▪ Consecutive UAV image registration ▪ UAV to](https://reader036.vdocuments.site/reader036/viewer/2022081520/56649dd25503460f94ac977d/html5/thumbnails/39.jpg)
Two steps to register an UAV image to the map Register each frame to its previous frame
to derive an initial estimate Register UAV image to the map to derive
Limitations Initial estimate should be given Unable to recover from a bad estimate
MH ,0',MiH
',MiH
MiH ,