stefano mattoccia -...
TRANSCRIPT
Learning-based methods for depth from images and confidence estimation
Stefano Mattoccia
Department of Computer Science and EngineeringAlma Mater Studiorum, University of Bologna
Naver Labs Europe - Grenoble, FranceDecember, 14 2018
Outline
● Introduction● Confidence estimation ● Depth from stereo ● Depth from monocular images
Introduction
● Our group mainly focuses on depth estimation from images ● In most cases with machine learning● Team
○ Stefano Mattoccia, Associate Professor ○ Matteo Poggi, PostDoc researcher○ Fabio Tosi, PhD student (second year)○ Filippo Aleotti, PhD student (first year)
Depth sensing: applications
Depth sensing is paramount for many applications:
● Autonomous driving● Augmented reality ● Robotics● Biometrics● ...
Active depth sensing
● LiDAR, structured and unstructured light, Time-of-flight ● Example: Velodyne, Kinect 1, Intel RealSense and Kinect 2 ● Effective and quite popular● Limitations: indoor/outdoor, sparse, expensive, cumbersome, short range
Passive depth sensing
● Based on standard cameras (cheap, fast, lightweight)● Do not perturb the sensed environment● Suited for indoor and outdoor
Therefore, (potentially) able to address issues of active sensors
● Stereo
● Monocular
In both cases, deep-learning is state of the art
Passive depth sensing: setups
Countless devices
Few devices
Stereo setup
● Given a stereo setup● The first and crucial problem consists in determining corresponding points
○ Very challenging task
● Then, depth can be easily inferred by triangulation
Reference Target Disparity map
Cost volume
● Cost volume: likelihood C(x,y,d) of each potential correspondence ● Census + Hamming a popular, yet effective, matching cost● Learning-based methods (eg, MC-CNN [Zbontar, CVPR15]) more accurate
SGM [Hirschmuller, CVPR05]
● The raw cost volume is further processed for better accuracy● SemiGlobal Matching (SGM) is an excellent trade-off (accuracy vs speed)● Performs independent Scanline Optimizations (SOs) along different paths
Deep-stereo matching
● State-of-the-art networks trained end-to-end often do not explicitly determine corresponding points (eg, DispNet [Mayer, CVPR16])
● Initially, trained on large synthetic datasets ● Then, fine-tuned in the target environments (eg, autonomous driving)
Domain adaptation
● Deep-networks yield excellent results (top on Kitti leaderboards)● However, domain adaptation (eg, outdoor ➞ indoor) is critical ● Requires large datasets with depth labels for fine-tuning● Adaptation is a major issue for practical deployment of deep networks
DispNet without fine-tuning on Kitti
Monocular setup 1/2
● Given a single camera, the task is even more challenging (and ill-posed)
http://drawingpencilarts.com/ https://scambieuropei.info
Monocular setup 2/2
● Despite these facts, deep-learning enabled astonishing results● Unsupervised training (eg, [Godard, CVPR17]) is very effective
Input image Monodepth [Godard, CVPR17]
Confidence estimation
● Regardless of the depth estimation strategy, detecting outliers is crucial
Confidence map
Disparity map (SGM)
Learning-based confidence measures (CMs)
● Conventional methods based on heuristic assumptions● For stereo, mainly analyzing the cost volume● [Haeusler, CVPR13] the first learning-based method● Much more effective than conventional approaches ● Other methods improved this method with better features
○ [Spyropoulos, CVPR14]○ [Park, CVPR15]
Confidence ∈ [0..1]
Early learning-based confidence measures
● Most features extracted from the cost volume (required)
[Haeusler, CVPR13]
[Spyropoulos., CVPR14]
[Park, CVPR15]
Cost volume
Random Forest (RF)
How to deal with COTS depth sensors?
● Commercial stereo cameras or prototypes do not provide a cost volume● The same (substantially) applies for monocular and stereo deep networks
Eg, Intel RealSense
Do we really need the cost volume for CMs?
● Given the output of a conventional stereo algorithm (eg, SGM), by looking at the disparity map, we can easily detect outliers
● Thus, can a ML framework learn this task?
Confidence estimation in the disparity domain
● In O1 [Poggi-a, 3DV16] local behaviour captured by 5 hand-crafted features ● Each feature, computed at 4 scales in constant time, from disparity map● Outperforms [Haeusler, CVPR13], [Spyropoulos, CVPR14], [Park, CVPR15]
O1, [Poggi-a, 3DV16]
Learning from scratch confidence measures
● Motivated by these results, we moved forward ● Learning a CM with a shallow CNN (CCNN) [Poggi, BMVC16]● Patch-based (9x9) strategy, without hand-crafted features
Patch Based Confidence Prediction (PBCP)
● [Seki, BMVC16] proposed a disparity domain approach ● However, requires Left and Right disparity maps not always available ● Left-Right features are fed to a shallows CNN● As for any other methods, trained on patches
Left-Rightfeatures
Performance evaluation [Poggi, ICCV17]
● Overall, evaluated 76 CMs (for embedded system see [Poggi, ICIAP17] )● Stereo algorithms: BM, SGM and MC-CNN ● Datasets: Kitti12, Kitti15 and Middlebury14
⇒ Learning-based methods much more effective⇒ In particular disparity domain methods O1, CCNN and PBCP⇒ Generalize better (eg, Kitti12➞Kitti15, Kitti12➞Middlebury) ⇒ CCNN outperforms other methods
● Main issue: these methods require training (and depth labels)
Training without GT labels 1/2
● [Mostegel, CVPR16] inferred depth labels from stereo sequences● In [Tosi, BMVC16] a different strategy based on stereo pairs ● Reliable good and bad samples selected according to a pool of std CMs
[Mostegel, CVPR16]GT[Tosi, BMVC17]
Training without GT labels 2/2
● Lower density (11.1% vs 28.2%) ● Better accuracy (94.7% vs 85.2%)
O1 & GT O1 & [Mostegel, CVPR16] O1 & [Tosi, BMVC17]
Exploiting local consistency of CMs
● [Poggi, CVPR17] argued that confidence estimation is locally consistent● Proposed a shallow CNN trained to take advantage of this assumption● Improved, up to 75%, 23 out of 23 CMs (including learning-based ones)
PKR PKR+
Beyond local reasoning 1/2
● The main limitation of SOTA CMs is their limited perceptive field (a patch)● In [Tosi, ECCV18] proposed a local/global method to get rid of this issue
CCNN LGC-Net [Tosi, ECCV18]
Beyond local reasoning 2/2
LGC-Net
CMs applied to deep stereo networks
DispNet [Mayer, CVPR16] CCNN [Poggi, BMVC16]Disparity map errors
● CCNN applied to the output of deep stereo network DispNet [Poggi-c, under review]
Left image
CMs applied to monocular deep networks
Input image Depth map errors
Monodepth [Godard, CVPR17] CCNN [Poggi, BMVC16]
● CCNN applied to the output of Monodepth [Poggi-c, under review]
Applications of confidence measures
● Depth sensors fusion [Marin, ECCV16]● Improving SGM accuracy [Poggi-a, 3DV16]● Stereo disparity fusion [Poggi-b,3DV16] ● Disparity refinement with Non Local Anchoring (NLA) [Poggi-a, under review]
On Kitti 15, with O1:BM from 37% to 8%MC-CNN from 17% to 6%SGM from 8% to 6%
Handling domain adaptation
● Deep stereo networks achieve top performance● Perform poorly when not appropriately fine-tuned● Large datasets with GT labels required ● [Tonioni, ICCV17] tackled this issue with a confidence guided loss● Reliable disparity labels from BM or SGM filtered according to CCNN● Eventually [Tonioni, under review] improved and extended to monocular
depth estimation suffering of the same issue (see next indoor video)
Monodepth fine-tuned on Kitti
DispNet and Monodepth with the proposed adaptation method [Tonioni, under review]
Monocular depth estimation enforcing a trinocular assumption (3Net)
● In [Poggi, 3DV18] proposed a method to get rid of stereo artefacts ● Trinocular setup for training● Unfortunately trinocular datasets are not available● Interleaved training protocol to enforce trinocular assumption from
standard binocular datasets (e.g., Kitti)● 3Net outperforms Monodepth [Godard, CVPR17] by a significant margin
L C R
Other works about depth from mono
● [Zama Ramirez, ACCV18] semi-supervised framework to improve monocular depth estimation leveraging semantic labels
● The output of the network: depth map + semantic map
● [Aleotti, 3DRW18] cast monocular depth estimation within a GAN framework
PydNet: fast monocular depth estimation
● SOTA monocular depth estimation methods: accurate but slow (on CPU)● Monocepth [Godard, CVPR17] counts about 30 M parameters● [Poggi, IROS18] a lightweight pyramidal architecture (PyDNet) ● Accuracy close to Monodepth● Only 1.9 M parameters, suited for CPUs and embedded systems● About 1 fps on a Raspberry PI and 10 fps on Intel i7 CPU● In [Peluso, DATE19] further optimizations aimed at improving energy
efficiency (accuracy ≊PyDNet)
PydNet2: a faster and better network
● In [Poggi-b, under review], PyDNet was further improved ➞ PyDNet2:
⇒ less parameters: ≈ 2.5% vs Monodepth and ≈ 37% vs PydNet ⇒ faster architecture: ≈ 10+ fps on a Jetson TX2 (without the GPU)
● Moreover, a peculiar training allows to close the gap with Monodepth:
⇒ same accuracy at H and Q resolution⇒ marginal drop at E resolution
PydNet2, lightweight and accurate network at three scales (H,Q,E res)
Pyd-Net2 [Poggi-b, under review]
Real-time depth from mono on a CPU
Monodepth
PyDNet
PyDNet2
H Q E
ARMv8 (no GPU)
● Accuracy: PyDNet2 ≊ Monodepth● Much faster: model size ≈ 2.5%
● In its fastest version, up to 40+ fps on ARM CPU (Jetson TX2) [Poggi-b, under review]
Conclusions
● Deep learning paved the way to major improvements for depth from (stereo and monocular) images and confidence estimation
● Learning-based depth and confidence estimation methods very effective ● Several applications:
○ AR○ autonomous driving ○ Robotics
● Open problems:○ Confidence measure for stereo and monocular deep networks ○ Self-adapting, possibly real-time, monocular systems
Acknowledgements (alphabetical order)
Filippo Aleotti, Andrea Calimera, Antonio Cipolletta, Luigi Di Stefano, Valentino Peluso, Matteo Poggi, Alessio Tonioni, Fabio Tosi, Pierluigi Zama Ramirez
We gratefully acknowledge the support of NVIDIA Corporation with the donation of a Titan Xp and a Titan X Pascal GPU
Bibliography
[Zbontar, CVPR15] J. Zbontar, Y. LeCun, “Computing the stereo matching cost with a convolutional neural network”, CVPR 2015[Mayer, CVPR16] N. Mayer and E. Ilg and P. H{\"a}usser and P. Fischer and D. Cremers and A. Dosovitskiy and T. Brox , “A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation”, CVPR 2016[Hirschmuller, CVPR05], H. Hirschmuller, “Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information”, CVPR 2005[Godard, CVPR17], C. Godard, O. Mac Aodha, G. J. Brostow, “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, CVPR 2017[Haeusler, CVPR13], R. Haeusler, R. Nair, D. Kondermann, “Ensemble learning for confidence measures in stereo vision”, CVPR 2013
[Spyropoulos, CVPR14], A. Spyropoulos, N. Komodakis, P. Mordohai, “Learning to detect ground control points for improving the accuracy of stereo matching”, CVPR 2014[Park, CVPR15], M. G. Park, K. J. Yoon, “Leveraging stereo matching with learning-based confidence measures”, CVPR 2015 [Poggi-a, 3DV16], M. Poggi, S. Mattoccia, “Learning a general-purpose confidence measure based on O(1) features and a smarter aggregation strategy for semi global matching”, 3DV 2016[Poggi, BMVC16], M. Poggi, S. Mattoccia, “Learning from scratch a confidence measure”, BMVC 2016[Seki, BMVC16], A. Seki and M. Pollefeys, “Patch based confidence prediction for dense disparity map”, BMVC 2016[Poggi, ICCV17], M. Poggi, F. Tosi, S. Mattoccia, “Quantitative evaluation of confidence measures in a machine learning world”, ICCV 2017[Mostegel, CVPR16], C. Mostegel, M. Rumpler, F. Fraundorfer, and H. Bischof, “Using self-contradiction to learn confidence measures in stereo vision”, CVPR 2016[Tosi, BMVC17], F. Tosi, M. Poggi, A. Tonioni, L. Di Stefano, S. Mattoccia, “Learning confidence measures in the wild”, BMVC 2017[Poggi, ICIAP17], M. Poggi, F. Tosi, S. Mattoccia, “Efficient confidence measures for embedded stereo”, ICIAP 2017[Poggi, CVPR17], M. Poggi, S. Mattoccia, “Learning to predict stereo reliability enforcing local consistency of confidence maps”, CVPR 2017
[Tosi, ECCV18], F. Tosi, M. Poggi, A. Benincasa, S. Mattoccia, “Beyond local reasoning for stereo confidence estimation with deep learning”, ECCV 2018[Marin, ECCV16], G. Marin, P. Zanuttigh, S. Mattoccia, “Reliable fusion of ToF and stereo depth driven by confidence measures”, ECCV 2018[Poggi-b, 3DV16], M. Poggi, S. Mattoccia, “Deep Stereo Fusion: combining multiple disparity hypotheses with deep-learning”, 3DV 2016[Poggi-a, under review], M. Poggi, F. Tosi, S. Mattoccia, “Confidence prediction by O(1) features in the disparity domain and applications to stereo”, paper under review [Tonioni, ICCV17], A. Tonioni, M. Poggi, S. Mattoccia, L. Di Stefano, “Unsupervised Adaptation for Deep Stereo”, ICCV 2017[Tonioni, under review], A. Tonioni, M. Poggi, S. Mattoccia, L. Di Stefano, “Unsupervised Domain Adaptation for Depth Prediction from Images”, paper under review[Poggi, 3DV18], M. Poggi, F. Tosi, S. Mattoccia, "Learning monocular depth estimation with unsupervised trinocular assumptions", 3DV 2018[Zama Ramirez, ACCV18], P. Zama Ramirez, M. Poggi, F. Tosi, S. Mattoccia, L. Di Stefano, "Geometry meets semantic for semi-supervised monocular depth estimation", ACCV 2018[Aleotti, 3DRW18], F. Aleotti, F. Tosi, M. Poggi, S. Mattoccia, “Generative Adversarial Networks for unsupervised monocular depth prediction”, 3DRW 2018, ECCV 2018 workshop[Poggi, IROS 18], M. Poggi, F. Aleotti, F. Tosi, S. Mattoccia, “Towards real-time unsupervised monocular depth estimation on CPU”, IROS 2018
[Peluso, DATE19], V. Peluso, A. Cipolletta, A. Calimera, M. Poggi, F. Tosi and S. Mattoccia, “Enabling Energy-Efficient Unsupervised Monocular Depth Estimation on ARMv7-Based Platforms”,[Poggi-b, under review], M. Poggi, F. Tosi, F. Aleotti, S. Mattoccia, “Real-time monocular depth estimation without GPU”, paper under review [Poggi-c, under review], M. Poggi, F. Tosi, S. Mattoccia, “Learning from scratch confidence measures for depth data”, paper under review