cloud computing & big data - morris ... outline of the course 1. cloud computing & big data 2....

Download Cloud Computing & Big Data - Morris ... Outline of the Course 1. Cloud Computing & Big Data 2. Machine

Post on 20-May-2020

0 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • Dr. Gabriele Cavallaro Postdoctoral Researcher High Productivity Data Processing Group Juelich Supercomputing Centre, Germany

    Remote Sensing Systems and Applications (1)

    October 16th, 2018 Room V02 – 138

    Cloud Computing & Big Data PARALLEL & SCALABLE MACHINE LEARNING & DEEP LEARNING

    PRACTICAL LECTURE 6.1

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Outline

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Outline of the Course

    1. Cloud Computing & Big Data

    2. Machine Learning Models in Clouds

    3. Apache Spark for Cloud Applications

    4. Virtualization & Data Center Design

    5. Map-Reduce Computing Paradigm

    6. Deep Learning driven by Big Data

    7. Deep Learning Applications in Clouds

    8. Infrastructure-As-A-Service (IAAS)

    9. Platform-As-A-Service (PAAS)

    10. Software-As-A-Service (SAAS)

    11. Data Analytics & Cloud Data Mining

    12. Docker & Container Management

    13. OpenStack Cloud Operating System

    14. Online Social Networking & Graphs

    15. Data Streaming Tools & Applications

    16. Epilogue

    + additional practical lectures for our

    hands-on exercises in context

     Practical Topics

     Theoretical / Conceptual Topics

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Outline

     Remote Sensing Background

     Data Acquisition

     Preprocessing

     Feature Extraction and Selection

     Copernicus: Sentinel 2 Mission

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  •  The term remote sensing was first used in the United States in the 1950s by Ms. Evelyn Pruitt of the U.S. Office of Naval Research

    Remote (without physical contact) Sensing (measurement of information)

    • Measurement of radiation of different wavelengths reflected or emitted from distant objects or materials

    • They may be categorized by class/type, substance, and spatial distribution

    [1] Satellite (1960)

    Remote Sensing

    [2] The Earth-Atmosphere Energy Balance Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • • Suitable for many applications

    • Non-invasive method

    • Satellite platforms • Invaluable view • Repetitive and consistent

    Application Domain

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • • How to assess the damage? • Which are the most hit points?

    • How to plan the Humanitarian aid?

    A devastating tsunami hit many coastal regions in the Indian Ocean (2014)

    • One of the most devastating natural disasters in recorded history • 14 countries were hit

    [3] The 2004 Indian Ocean Tsunami [4] Five years after Indian Ocean tsunami

    Application Examples (1)

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  •  Global change detection and monitoring: Deforestation

     Environmental assessment and monitoring: Urban growth

    [6] Deforestation in Bolivia from 1986 to 2001

    [5] Population growth from 1975 - 2010 of Manila

    Application Examples (2)

    1986 2001Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • • Perhaps the most common form of image interpretation

    • Applications: environmental management, agricultural planning, health studies, climate and biodiversity monitoring, and land change detection

    Classification of Remote Sensing Images

    Generation of thematic maps

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  •  Different tasks of equal importance

     Successful classification results depend on all the steps

    Pipeline for Classification

     New analysis challenges

     5Vs: Volume, Variety, Velocity, Veracity and Value

     Need of scalable methods and underlying infrastructures

    DATA ACQUISITION PRE-PROCESSING

    EXTRACT/SELECT FEATURES CLASSIFICATION

    POST- PROCESSING

    CLASSIFICATION SYSTEM

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Platforms and Sensors

     Active Sensor: own source of illumination

     Capture image in day and night

     Any weather or cloud conditions

     Passive Sensor: natural light available

     Great quality satellite imagery

     Multispectral and Hyperspectral technology

     Platform: selected according to the application

    [7] Active-and-passive-remote-sensing

    DATA ACQUISITION PRE-PROCESSING

    EXTRACT/SELECT FEATURES CLASSIFICATION

    POST- PROCESSING

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  •  Data are complex, noisy and may contain errors

     Improve image quality as the basis for later analyses that will extract information

     Detection and restoration of bad lines

     Geometric rectification

     Image registration

     Radiometric calibration

     Atmospheric correction

    Preprocessing

    DATA ACQUISITION PRE-PROCESSING

    EXTRACT/SELECT FEATURES CLASSIFICATION

    POST- PROCESSING

    CLASSIFICATION SYSTEM Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Radiometric Calibration and Correction Process

    [8] Radiometric Corrections

     The value recorded for a given pixel includes:

     reflected or emitted radiation from the surface

     radiation scattered and emitted by the atmosphere

     Most of the applications are interested in the actual surface values

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • The Feature Domain

     Select suitable features for successfully implementing an image classification

     How to obtain discriminating and independent features?

    [9] G. Hughes

    Too many features may decrease the classification accuracy

    (1) : few raw features

    (2): hundreds of raw features

    DATA ACQUISITION PRE-PROCESSING

    EXTRACT/SELECT FEATURES CLASSIFICATION

    POST- PROCESSING

    CLASSIFICATION SYSTEM

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Normalized Difference Vegetation Index (NDVI)

     Create additional relevant features from the existing raw features in the data

     Increase the predictive power of the classifier

    NDVI = (NIR – R) / (NIR + R)

    [11] NDVI

    R G

    B NIR

    Bellingham, WA , US

    [10] NDVI & ClassificationPractical Lecture 6.1 – Remote Sensing Systems and Applications

  • Spatial Information (1)  The scene complexity and the spatial resolution determines the number of mixed pixels

     The spectral unmixing problem:

     Identify the pure materials (endmembers)

     Estimate their corresponding proportions (abundances)

    mixed spectral signature mixed pixel

    65% 20% 5% 6% 4%

    soft classification

     Two models to analyze the mixed pixel

    [12] A. Plaza et al.

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Spatial Information (2)

     When spatial resolution increases, structures are larger than the pixel size

     The correlation between neighboring pixels increases

     Adjacent pixels of a roof pixel belong to the same class with a high probability

     Structures can be represented as regions of spatially connected pixels

    The presence of mixed pixels can not be avoided

     Spatial contextual classifiers can exploit the correlation of pixels within a subset domain

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Classification

    DATA ACQUISITION PRE-PROCESSING

    EXTRACT/SELECT FEATURES CLASSIFICATION

    POST- PROCESSING

    CLASSIFICATION SYSTEM

     Model choice

     Mapping between low level features and high level information

     Training

     Training set must be representative

     Evaluation

     Measure the error rate (or performances)

     Computational Complexity

     Scalability

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  •  World's largest single earth observation programme

     Directed by the European Commission in partnership with the European Space Agency (ESA)

     Monitors the Earth, its environment and ecosystems

     Free and open data policy

     Features: Continuity, Global coverage, Frequent updates and Huge data volumes

     Served by a set of dedicated satellites (the Sentinel families)

    [13] Copernicus programme

    [14] Sentinel Space

    Copernicus

    Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Copernicus

    [15] Sentinels for Copernicus Practical Lecture 6.1 – Remote Sensing Systems and Applications

  • Sentinel 2 Mission

    • Platform: Twin polar-orbiting satellites, phased at 180° to each other

    • Temporal resolution of 5 days at the equator in cloud-free conditions

     Provides