Institut für Photogrammetrie und GeoInformation
60 years CASM: 1959 - 2019
International Symposium on Innovation, Integration and Development of Surveying and Mapping
Research and development in
photogrammetry and remote sensing
- and the role of ISPRS
Christian Heipke
IPI - Institut für Photogrammetrie und GeoInformation,
Leibniz Universität Hannover
President, ISPRS
Institut für Photogrammetrie und GeoInformation
Greetings from ISPRS Council …
… and the whole ISPRS community
Institut für Photogrammetrie und GeoInformation
The International Society for
Photogrammetry and
Remote Sensing - ISPRS
• … an international NGO with a focus on
– science and development
• in photogrammetry, remote sensing, spatial information
– cooperation between all relevant stakeholders
• academia, private industry, government, end users
– truly global cooperation
• education, technology transfer, capacity building
• more than 100 years old
• approx. 180 institutional members
Institut für Photogrammetrie und GeoInformation
Table of contents
• Introduction
• Trends in research and development
• Deep learning
• The role of ISPRS
Institut für Photogrammetrie und GeoInformation
• Globally increasing population – in particular in emerging countries and coastal areas
– shrinking population in some developed areas, aging society
• Increasing globalisation – mobility of people, markets, capital, education, information, …
– digitisation
• Global companies entered the scene
• Changing communication – request for 24/7 availability, real-time response
– social media, a totally connected world
• Global change (only ONE earth, no plan[et] B …) – „Fridays for Future“
– 2030 UN Agenda and SDGs, Sendai Framework, Paris Agreement
Social and political frame
Institut für Photogrammetrie und GeoInformation
UN Sustainable Development Goals
… Earth Observation supports virtually all of them
Institut für Photogrammetrie und GeoInformation
Global ICT trends
Digitisation Mobility
Autonomous
driving
Internet of things,
AI
adapted from Volker Liebig, ESA 2016
Industry 4.0
SDG goals
Block chain Quantum computing
Institut für Photogrammetrie und GeoInformation
A connected world – location is key
• location uniquely ties items
together, in 2D and 3D, and
thus unlocks value in other‘s
data (N. Clifford, then CEO
Ordnance Survey, 2016)
• we are part of a more
complex, connected world,
not an isolated island
• interoperability
• crowd sourcing
BIM
Institut für Photogrammetrie und GeoInformation
Trends in research and
development
Institut für Photogrammetrie und GeoInformation
Principle of photogrammetry & remote sensing
• determination of characteristics
of the electromagnetic radiation
of a given wave length, reflected
or emitted from some surface
• energy, phase, polarisation, signal
form, travel time
• derivation of object and surface
characteristics from these
measurements
• geometry: position, size, shape
• radiometry: orthophoto mosaic
• semantics: object-ID, attributes
Institut für Photogrammetrie und GeoInformation
• new and better sensors, new platforms
– UAV, mobile mapping
– NewSpace: small COTS satellites, swarms, space video, …
• sensor and data fusion
– images + point clouds, maps, VGI, text, voice, …
• multi-temporal data analysis
– change detection, update
– process monitoring (data continuity !)
• real-time applications: traffic, environment, security, …
– robotics for mapping / mapping for robotics
Research and development trends
big data – machine / deep learning (data mining) big data, no alternative to automation
Institut für Photogrammetrie und GeoInformation
Sentinel-1A mosaic of Europe
using 400 scenes collected between May and July 2015,
contains modified Copernicus Sentinel data
© (2015) ESA/Array SC Canada Sentinel-2A cloudfree mosaic of Africa
using almost 7000 scenes collected between Dec 2015 and April 2016
contains modified Copernicus Sentinel data
© (2016) ESA/Brockmann Consult, GER/ Uni. Catholique de Louvain, BEL
Institut für Photogrammetrie und GeoInformation
Satellite constellations
• Planet: “The whole Earth in high resolution every day”
• standard cube sat’s (nano-sat’s), 10*10*30 cm3, 4 kg
• 3 – 5 m GSD, 4 bands, design constantly adapted
• flock of 28 sat’s. deployed from ISS (in 2014/2016, …)
• Feb. 2017: 88 doves (flock 3p) launched with Indian PSLV rocket
• currently 150+ sat’s in orbit (incl. 5 RE + 13 Skybox sat.)
Institut für Photogrammetrie und GeoInformation
Increased data volume © Bamler, 2018
Institut für Photogrammetrie und GeoInformation
Deep learning - the answer to automation needs ?
Institut für Photogrammetrie und GeoInformation
YOLO: You Only Look Once (version 2) – a Convolutional Neural Network (CNN)
for object detection
J. Redmon, S. Divvalay, R. Girshick, A. Farhadiy, Uni. Washington, 2016
Institut für Photogrammetrie und GeoInformation
AI / ML / DL
• AI / DL (“data science”) recently caught lots of attention in
science and society alike – big data
– „2nd best for everything“
– hardware developments (in part. GPU)
– abundant (training) data
• one of 10 breakthrough techno-
logies in MIT Tech. Review 2013
• recently pushed by Internet companies – Google, Facebook, Microsoft, Baidu, …
• said to rival (even resemble?) human cognitive ability
– victory of AlphaGo software against Go champion Lee Sedol (Oct.’17)
– „super human“ performance in computer vision challenges
Artificial
Intelligence
Institut für Photogrammetrie und GeoInformation
Deep learning - what is it, what is new?
• traditional classification:
Cleverly-Designed Features
Input Data ML model
most of the “heavy lifting” here, final performance only as good as feature set
… “learning” an input-output mapping from examples by machines (supervised classification)
• deep learning:
features and model learned together, mutually reinforcing each other
Features Input Data model Deep Learning
Institut für Photogrammetrie und GeoInformation
Development of deep learning
• Cybernetics („the perceptron“) – f: step function, linear decision boundary
– McCulloch, Pitts, 1943, Rosenblatt, 1958/62
• Connectionism, ANN, MLPs – f: sigmoid/tanh function, non-linear decision boundary
distributed representation, back propagation for training
– Rumelhart et al., 1986, Hinton et al., 1986, LeCun, 1987
• Deep learning – learn representation (features)
– learn from actions (e.g. to write SW)
– fast hardware (GPU), vast amount of training data:
efficient training of networks with „many“ layers
– Hinton et al., 2006
– Krizhevsky et al., 2012: CNN wins ILSVRC, decreasing the error by nearly 50%
.
.
.
x1
x2 x3
xn bj
S f
wj1
wjn
wj3
wj2 aj
Input Neuron j Output
wjb
Institut für Photogrammetrie und GeoInformation
CNN: Convolutional neural networks
• convolution
• (max-)pooling
• activation function e.g. ReLU(x) =
max(0,x)
7 5 9 8
5 0 3 1
0 4 7 3
8 6 6 2
9 8
7 4
• for regularly structured data
• repeated execution of those steps
based on “some“ architecture
• training of hierarchical feature
representation (filter coefficients)
by back propagation
• classification: one label per patch
Institut für Photogrammetrie und GeoInformation
AlexNet architecture (2012)
Krizhevsky, A., Sutskever, I., Hinton, G. E., 2012: ImageNet Classification with Deep
Convolutional Neural Networks, NIPS 2012.
• Goal: Classification of entire (small) images
predict one class label per image
ImageNet Large-Scale Visual Recognition Challenge (ILSVRC):
1.2 million training images, 1000 classes
5 convolutional layers, two FC layers, 60 Mio. parameters
with AlexNet, CNN really took off (16.4% top-5 error)
Institut für Photogrammetrie und GeoInformation
Classification (CV terminology), examples
Institut für Photogrammetrie und GeoInformation
Success story of DL/CNN in computer vision
ImageNet Large Scale Visual Recognition Comp. (INLSVRC):
• 1,2 Mio test images, 1000 classes
• one class label per image
• ever increasing no. of layers
Institut für Photogrammetrie und GeoInformation
Other ILSVRC tasks solved using CNN
© Fei Fei Li et al., Stanford CS 231, 2017
good
semantic segmentation (CV) = pixel-level classification (Photo/RS)
Institut für Photogrammetrie und GeoInformation
Computer vision data discussed so far
from:
Net,
2014
https://ai.googleblog.com/2014/09/building-deeper-understanding-of-images.html
Institut für Photogrammetrie und GeoInformation
Typical remote sensing data
Institut für Photogrammetrie und GeoInformation
Photogrammetry, remote sensing - CNNs
• bird‘s eye view (sky is not „up“), geolocated + known GSD
• large images, complex scenes, multiple objects
• multi-modal information (spectral bands, 3D point clouds,
maps, social media data, …)
• monitoring of geo-processes, time-series processing,
spatio-temporal-spectral modelling
• prior knowledge (in part. of observed processes) essential
• geo-/bio-parameters vs. classification labels
• shortage of training data, but outdated DB available
• global challenges -> global transferability of approaches
• more stringent quality requirements (geom., semantics)
Institut für Photogrammetrie und GeoInformation 28
Example I: Road extraction / refinement
using CNNs
– first CNN application using
aerial images
– binary classification
(road / no road)
– training data: OSM
• incomplete
• partly with geometric errors
– models for errors in training data, learning of the related model
parameters
Mnih & Hinton, 2012
Mnih V., Hinton G, 2012: Learning to label aerial images from noisy data. In: McCallum, Roweis
(eds.), ICML 2012
Mnih & Hinton, 2012
Institut für Photogrammetrie und GeoInformation
Example II: land cover / land use update
DSM
CNN-based land cover
classification
CNN-based land use
classification
WO
RK
LF
LO
W
DOP DTM
Building
Sealed surface
Bare soil
Grass
Tree
Water
Rails
Car
Others
Land
cover
DTM
Settlement
Road
Railway
Water
Agriculture
Forest
Others
GIS
© Yang et al., 2018 - IPI
Yang C., Rottensteiner F., Heipke C., 2018: Classification of land cover and land use based on
convolutional neural networks. ISPRS Annals of the Ph, RS IV-3, 251-258.
Institut für Photogrammetrie und GeoInformation
Ensemble classifier: RGB, IR, nDSM, SkipNet architecture
training from scratch incl. data augmentation
Land cover classification (sem. seg.)
convolution block max-
pooling
upsample skip
connection
softmax
Institut für Photogrammetrie und GeoInformation
Land use update
Network architecture
predict the class for an input image of 256 x 256
require the generated patches scaled to 256 x 256
large objects (e.g. roads) subdivided and re-scaled
64 96 128 256 2 x 256 2 x 256
mask RGB/IR/nDSM land cover
posteriors
…
convolution block max-
pooling
average-
pooling
ROI
location
softmax
Institut für Photogrammetrie und GeoInformation
Results, land cover
Datasets Hameln (12 km2), Schleswig, 8 classes
Data CRF-Approach1 CNN-Approach
OA [%] Av. F1 [%] OA [%] Av. F1 [%]
Hameln 83.7 66.7 89.1 81.8
Schleswig 82.5 64.6 87.3 79.3
Dataset Mecklenburg-Vorpommern, 10 classes
Data CNN-Approach
OA [%] avg. F1 [%]
MV 83.5 79.7 © Yang et al., 2019 - IPI
1 Albert L., Rottensteiner F., Heipke C., 2017: A higher order conditional random field model for
simultaneous classification of land cover and land use. JPRS (130) 63-80.
Institut für Photogrammetrie und GeoInformation
Results, land cover
Input Our result Groundtruth
Building
Asphalt
Soil
Grass
Tree
Water
Car
Others
Institut für Photogrammetrie und GeoInformation
ISPRS Semantic Labelling Challenge
www2.isprs.org/commissions/comm3/wg4/tests.html
Institut für Photogrammetrie und GeoInformation 36
Example III: Dense matching using CNNs
Kang J., Chen L., Deng F., Heipke C., 2019: Encoder-Decoder network for local structure
preserving stereo matching Illumination. Proc. DGPF, Vol. 28
left image gt/right image Our baseline
Zbontar J, LeCun Y, 2015: Computing the Stereo Matching Cost with a Convolutional Neural
Network, CVPR, pp. 1592-1599, DOI: 10.1109/CVPR.2015.7298767
Institut für Photogrammetrie und GeoInformation
Conclusions - evaluation
• many other examples in the last few years (see e.g.
review Zhu et al., 197 reference entries)
– more flexible wrt. different object characteristics
– strength: learning features, classifier not so important
– key to good performance: network depth, no. of training data
– has outperformed just about all classical algorithms by a
large margin, provided enough training data is available
– is about to do the same in geometric tasks
• no expert-free totally automatic processing chain, but
integration with human capabilities
Zhu X., Tuia D., Mou L., Xia G., Zhang L., Xu F., Fraundorfer F., 2017. Deep Learning in Remote
Sensing: A comprehensive review and list of resources. IEEE GRSS Magazine 5(4): 8-36.
Institut für Photogrammetrie und GeoInformation
Conclusions - evaluation
• Open-source CNN implementations available Tensorflow (Google): https://www.tensorflow.org
CAFFE2 (Facebook) https://caffe2.ai/
…
• CNN has arrived in the commercial world
– Descartes Lab, https://www.descarteslabs.com/
– Simularity, simularity.com/solutions/satellite-anomaly-detection/
– Orbital Insight, https://orbitalinsight.com/
– spire, https://spire.com/
– NVIDIA (computer games!), https://www.nvidia.com/
– …
Institut für Photogrammetrie und GeoInformation
Conclusions - evaluation
• CNN is not magic - fundamentally a classifier – based on correlation between data sets
– an enormous amount of unknowns to be estimated
• lots of training data, long training times (days … weeks)
– sensitive to
• overfitting (“curse of dimensionality”)
• non-rep. training data (incorrect / biased / unbalanced / …)
• user choices, incl. hyper-parameters
– network architecture, (non-lin.) activation function
– design of loss function (function to be optimised)
– training parameters (weight initialisation, learning rate, drop
out rate, “momentum”, batch size, regularisation, …)
– generalisation and prediction capabilities unclear
• the system can’t learn what it never saw
Institut für Photogrammetrie und GeoInformation
The G
uard
ian,
Int.
Editio
n,
July
1st, 2
017
Institut für Photogrammetrie und GeoInformation
Conclusions – to do’s
• integrate prior knowledge, e.g. physical models
– don’t try to learn what we already know (e.g. laws of nature)
• CNNs for point clouds and other irregularly rep. data
• tackle training data shortage
– weakly, semi-, unsupervised learning, reinforcement learning
– data augmentation and simulation
– deal with errors in training data, e.g. when using outdated DB
– combination with crowd sourcing and social media data
– investigate limits of pre-training + fine tuning
• pay attention to geometric accuracy of object delineation
• sequential data, time series processing - RNNs
– streaming (big) data, online CNN learning
Institut für Photogrammetrie und GeoInformation
Special thanks to
Prof. Franz Rottensteiner
Chair, IPI research group
Photogrammetric image analysis
Chen Lin, Max Coenen, Philo Humberg, Uyen Nguyen, Dennis Wittich, Chun Yang, Junhua Kang