applying computer vision techniques to web public images for the environmental monitoring purposes

DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA

SnowWatch

Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring

Roman Fedorov: [email protected]


UGC

2

Media User Generated Content (UGC) publicly available on the Web is reaching unprecedented size nowadays:• Facebook ~ 350 MM daily photo uploads• Flickr ~ 8.000 MM photographs with 3.5 MM daily

uploads• Panoramio reached ~ 75 MM geo-tagged photograph

It contains an enormous amount of latent implicit knowledge, the challenge is to extract it.



Introduction

Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring 3

• Challenge: transform geolocated and timestamped public images depicting natural scenarios into virtual environmental monitoring stations.

• Complement existing measurements.


Project

4

SnowWatch aims at:• improving snow analysis and water management through

the acquisition and analysis of public media content depicting alpine mountains

• Creating alpine snow media dataset with open contribution of new media and access to processed media and its results for any environmental researchers



Content Providers

SnowWatch Engine

SnowWatch Architecture

User Photos

5

Flickr Photos

Content Consumers

Webcams

API Media Storage

Get Image Orientation, Identify Mountain Peaks

Get Image Snow Pixel-level classification



Flickr Photographs Crawling

6

Does an image contain a clear a mountain slope view?

- Heavily depends on thephotograph shooting point altitude



Flickr Crawling Implementation

7

SVM Image Content-based Classifier:� Bag of Visual Words / Fisher Vector� SIFT/SSIM/GIST/HOG2x2 Descriptors� Spatial Histogram, Vertical Bands

Manually annotated 1658 samples (~70%) for training and validation and 710 samples (~30%) for testing.



Webcam Images Aggregation

8

Identified ~2.000 webcams depicting mountains in Alpine regions.Each webcam produces an image every 1’ – 15’.• 67% of the images are not suitable due to bad weather• More than 50.000 images and 6 GB of data acquired dailyThrough the extraction of the edges we can compute the percentage of the skyline clearly visible, and discard all images with visibility lower than a certain threshold, obtaining the following ROC curve varying the threshold:



Webcam Images Aggregation

9

Remaining small moving obstacles (such as low clouds, persons, vehicles) and varying illumination conditions are managed through the median aggregation of all daily images



Photo Peak Identification

10

An orientation of a geo-tagged photograph is needed.

Given a Digital Elevation Model (a 3D terrain model) we can simulate the terrain silhouettes that should be seen from the photographer point of view.

Steps� Panorama� Scale� Edges� Filter� Match



Photo Panorama

11




Photo Scaling

12

Photo must be scaled properly, so the mountains have the same pixel dimension on the photograph and on the panorama.This can be obtained by calculating the Field Of View (FOV) of the photo:




Photo Edge Extraction

13




Photo Edge Filtering

14

Too many edges on the image that do not belong to a terrain contour!• Skyline detection algorithm is employed, and all the edge

pixels above the skyline are removed, being considered obstacles or clouds.

• Weighting mechanism is applied, which assigns decreasing weights to the edge pixels as the distance from the skyline increases.




Photo Alignment

15

Find the best overlap maximizing the vector cross correlation score:




Photo Peak Alignment

16

A global overlap between the photo and the panorama is often not enough, a neighborhood vector cross correlation adjustment is applied to each identified peak:

Steps� Panorama� Scale� Edges� Filter� Match� Local Match



Photo Peak Identification

17

Fraction of photographs with orientation correctly identified



Snow Pixels Classification

18

Classifying each pixel as snow covered/terrain is much more than simply checking bright/dark pixels:




19

SnowGround GroundGroundSnow Snow Snow




20

Random Forest Classifier that considers pixel and pixel neighbors color, global and local intensity of the image.Trained on 59 manuallylabeled images(~ 7 Million pixels),outperform currentmethods:



Environmental Impact Evaluation

21

Environmental consistency evaluation of extracted snow cover indexes.� No direct ground truth, how do we

evaluate it?� Practical utility w.r.t. an environmental

model



Environmental Impact Evaluation

22

Use case: Como Lake water managementFlooding vs Irrigation trend, what is the best solution?



SnowWatch 2°°°° year

23

If the physical snow estimations provided by ARPA (Italian EPA) are not available, Adding Virtual Snow Measures improves the performance by ~100%!



SnowWatch 2°°°° year

24

With some irrigation <-> flooding target solutions, using SnowWatch virtual measurements is even better than using ARPA snow estimation!



Current and Future Work

25

Current:• MountainWatch: real time augmented reality mobile

application• Expand tests on environmental impact

Future:• Expand area of interest and content providers• Orthophotos and new snow indexes


applying computer vision techniques to web public images for the environmental monitoring purposes

Software