damage detection fromaerial images via convolutional ...5/8/2017 mva2017@nagoya university 2 in the...

Damage Detection from Aerial Images

via Convolutional Neural Networks

*National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan

†Nagoya University, Aichi, Japan

5/8/2017 MVA2017@Nagoya University 1

Aito Fujita*

Riho Ito*

Ken Sakurada† Tomoyuki Imaizumi*

Shuhei Hikoska* Ryosuke Nakamura*

Aerial images for damage detection

In the event of catastrophic disasters, fast assessment ofthe extent of damage is crucial for recovery.

E.g. “Which buildings got washed away by tsunami? And how many?”

Satellite/aerial images as source for damage assessment:

◦ The wide coverage can facilitate the assessment.

◦ However, currently, the assessment is performed manually (by human eye).

Aerial view of Tohoku quake

Post-quake imagePre-quake image

Damage assessment in practice

This building is “washed-away”.

This building is “surviving”.

Repeated for all buildings.

Manual assessmentPost-quake imagePre-quake image

RED: washed-away bldg.

YELLOW: surviving bldg.

RED: washed-away bldg.

YELLOW: surviving bldg.

Manual assessment

Manual assessment takes too much time.

Can we automate such assessment process?

Background and Contributions

Background of damage detection:

◦ Lack of labeled dataset

◦ No study on Deep Learning applied to satellite/aerial images

Related works in this context:

◦ Gueguen+(CVPR2015): hand-designed features (tree-of-shapes); satellite images

◦ Cooner+(Remote Sensing 2016): shallow neural networks; aerial images

◦ Nia+(CRV2017): Convolutional networks but ground-level images

Our contributions:◦ A new labeled dataset (ABCD dataset)

◦ Comprehensive analyses of CNNs for washed-away building detection

Image coverage

New dataset for washed-away building detection

AIST Building Change Detection (ABCD) dataset:

◦ Based on images taken before/after the Tohoku earthquake

◦ Over 10K pairs of pre/post-tsunami patches

◦ Collected from 66km2 of aerial images of the Tohoku coastal region

◦ A target building at the center of the patch

◦ A damage label assigned to each pair

Washed-away

buildings

Surviving

buildings

◦ The damage label data is based on the survey result of the Great East Japan earthquake,

which was conducted by the Japanese government (MLIT) after March 11, 2011.

◦ Now preparing to release to the public.

Overview of washed-away detection framework

Practical considerations

From a practical viewpoint, we investigated the following:

Input scenario (to address availability of pre-tsunami image)

Input scale (to address variability of building size)

Practical considerations: Input scenario

◦ Both pre- and post-tsunami images are available.

◦ Only post-tsunami images are available.

Input to a CNN: a pair of pre/post patches

Two configurations inspired by Zagoruyko+(CVPR15), Lin+(CVPR15), Simo-Serra+(ICCV15)

Input to a CNN: a pair of pre/post patches

Two configurations inspired by Zagoruyko+(CVPR15), Lin+(CVPR15), Simo-Serra+(ICCV15)

Fully-connected

Convolution

Max pooling

In practice, pre-tsunami images may not be available.

Input to a CNN: a post-tsunami patch

◦ Multiple crop sizes

◦ Fixed-scale

◦ Size-adaptive

◦ Multi-scale CNN

◦ Fixed-scale -> Aims to encompass most of the buildings.

◦ Size-adaptive -> Makes tiny buildings more conspicuous.

◦ Multi-scale CNN

◦ Fixed-scale: 160 x 160 pixels (64m x 64m) so that most buildings are encompassed.

160 pixels

◦ Fixed-scale -> Aim to encompass most of the buildings.

◦ Size-adaptive: the crop size depends on buildings—more emphasis on small buildings.

Size-adaptively crop and resize

Fixed-scale (160x160) Resized version

◦ Fixed-scale -> Aim to encompass most of the buildings.

◦ Size-adaptive -> Make tiny buildings more conspicuous.

◦ Multi-scale CNN

◦ Fixed-scale

◦ Size-adaptive

◦ Multi-scale CNN

Inspired by Zagoruyko+(CVPR15)

Experiment

Experimental setting, the same across all conditions (input scenario and scale)

The number of data◦ Randomly sampled 8,500 pairs; balanced class distribution

Evaluation◦ 5-fold cross validation (train/val per fold = 6800 : 1700)

◦ Classification accuracy

CNN hyper-parameters◦ Conv-Pool-Conv-Pool-Conv-Conv-FC-FC with ReLU nonlinearity

◦ No batch normalization (due to degradation in performance)

◦ SGD with momentum; constant learning rate; weight decay

◦ Shared or unshared weights between branches for Siamese case

Data pre-processing◦ Zero mean and unit variance (pixel-wise)

◦ Augmentation with vertical and horizontal flip

Accuracy

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Accuracy

6-channel Siamese Only post image

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Accuracy

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Accuracy

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Fixed Adaptive

Accuracy

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Wrongly classified

as “surviving”

Correctly classified

as “washed-away”

Correctly classified

as “washed-away”

Wrongly classified

as “surviving”

Fixed-scale Adaptive

The different scales are complementary w.r.t. prediction.

Accuracy

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Accuracy

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Accuracy

Without multi-scale

With multi-scale

Accuracy: in terms of input scenario

Only post-tsunami images may be sufficient in the context of washed-away building detection.

Both pre/post images Only post image

Accuracy: in terms of input scale

No good-and-bad between fixed-scale and adaptive-scale.The ensemble is always better.

Accuracy: in terms of input scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Without multi-scale

With multi-scale

Multi-scale CNNs always beat single-scale counterparts.

Qualitative result

Ground truthCNN prediction

RED: washed-away

YELLOW: surviving

Conclusion

This work in a nutshell:

◦ Effective use of CNNs for washed-away detection was explored.

◦ To this end, we compiled a new labeled dataset.

◦ On this dataset, we performed some types of experiments from an application viewpoint (input scenario and input scale).

◦ Overall, the accuracy was reasonably good.

Future work:

◦ Generalizability (other regions, other events, other data and so on)

◦ End-to-end framework from localization to classification such as instance-segmentation (Pinheiro+ECCV16)

Reference:◦Cooner et al.: Detection of urban damage using remote sensing and machine learning algorithms:

Revisiting the 2010 Haiti earthquake, in Remote Sensing, 2016.

◦Gueguen and Hamid: Large-scale damage detection using satellite imagery, in CVPR, 2015.

◦Simo-Serra et al.: Discriminative learning of deep convolutional feature point descriptors, in ICCV, 2015.

◦Zagoruyko and Komodakis: Learning to compare image patches via convolutional neural networks, in

CVPR, 2015.

◦Lin et al.: Learning deep representations for ground-to-aerial geolocalization" in CVPR, 2015.

◦Ministry of Land, Infrastructure, Transport, and Tourism: First report on an assessment of the damage

caused by the Great East Japan earthquake", http://www.mlit.go.jp/common/000162533.pdf

(published in Japanese)

◦Pinheiro: Learning to re ne object segments, in ECCV, 2016.

◦Nia and Mori: Building Damage Assessment Using Deep Learning and Ground-Level Image Data, in CRV,

Acknowledgement: This presentation is based on results obtained from a project commissioned by the New Energy and

Industrial Technology Development Organization (NEDO).

Appendix

Related work

Damage detection◦ Gueguen & Hamid (CVPR, 2015)

◦ Pre- and post-disaster satellite images

◦ Semi-supervised classification (BoF + SVM)

◦ Cooner et al. (Remote Sensing, 2016)

◦ Pre- and post-disaster satellite images

◦ Two-layer neural network

◦ Nia & Mori (CRV, 2017)

◦ Only post-disaster ground-level image

◦ Dilated CNN

Related work

Two image patches as input to CNN

◦ Simo-Serra et al. (ICCV, 2015)

◦ Lin et al. (CVPR, 2015)

◦ Zagoruyko & Komodakis (CVPR, 2015)

9 days after the quake (20/Mar/2011)114 days before the quake (17/Nov/2010)

Human annotations derived from the field survey are superimposed.

Red arrows are buildings that were washed away by tsunami.

Visual changes of buildings with tsunami are clear, but how can we describe these?

Cross validation result

15 22 29 36 43 50 57 64 71 78 85 92 99

Iteration (x10)

accuracy (fixed_size/noAug)

testloss (fixed_size/noAug)

trainloss (fixed_size/noAug)

Size-adaptive classification

Improved:

Worsened:

0 1 2 3 4

alidati

f accura

Number of conv layers shared

Effect of # of shared layers on accuracy

Visualization via Global Average Pooling

Global Average Pooling is the technique for:

1) Regularizing big networks by forgoing FC layers (e.g., NIN and GoogLeNet)

2) Visualizing feature maps learned by networks in an intuitive way for human

3) Localizing object instances even under the weakly-supervised setting

damage detection fromaerial images via convolutional ...5/8/2017 mva2017@nagoya university 2 in the...

Documents

issue six, november 2017 6.pdf · catastrophic damage was...

catastrophic events. catastrophic event an event that...

industry leaders forum: catastrophic injury claims can...

passive acoustic monitoring of recovery boiler dissolving...

nagoya 2 calendar - nagoya international center

catastrophic events

global economic turmoil, catastrophic loss and … ·...

catastrophic events catastrophic event –any event...

vermont national historic landmarks · introduction ....

nagoya visitors guide information center tourist attractions...

catastrophic risks

catastrophic event. an event that results from earth...

rapid damage assessment methodology for catastrophic souris...

hilton nagoya, nagoya city, japanhilton nagoya, nagoya city,...

asset performance management · significant or catastrophic...

global catastrophic risks:...

˚ kansai nagoya kansai nagoya

certification of civil composite structures based on ... -...

catastrophic dragons

catastrophic failures