11. image data analytics - jacobs university bremen...data analytics 405 visualization and computer...

Visualization and Computer Graphics LabJacobs University

11. Image Data Analytics

Data Analytics 377


Motivation

• Images (and even videos) have become a popular data format for storing information digitally.

Data Analytics 378


Motivation

• Traditionally, scientific and medical imaging techniques were considered.

Data Analytics 379


Motivation

• Nowadays, photographs form the majority.

1 pixel

Data Analytics 380


Image Analytics?

• Similarity computation is difficult.• Similar images may look very different.

– E.g., two photographs of cars may have completely different colors and shapes.

• Digital image processing allows for the detection of features in the images.

• Semantic queries on images is difficult without metadata.

Data Analytics 381


Semantic gap

• Image processing allows for the detection of regions of certain properties (typically homogeneous regions).

• Heuristics can be used to combine homogeneous regions to larger structures.

• Larger structures can be considered as low-level semantics.– E.g., detect human hands.

• The problem is to bridge the low-level semanstics to high-level semantics as in queries:– E.g., show me all images, where presidents are shaking hands.

• The gap between the low-level and high-level semantics is called the semantic gap.


11.1 Digital Image Processing

Data Analytics 383


Digital Image Processing

Image Acquisition

Image Restoration

Morphological Processing

Segmentation

Representation & Description

Image Enhancement

Object Recognition

Problem Domain

Colour Image Processing

Image Compression

Data Analytics 384


Image Aquisition

Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 385


Image Enhancement

Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 386


Image Restoration

Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 387



Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 388


Segmentation

Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 389


Object Recognition

Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 390



Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 391


Image Compression

Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 392



Image Acquisition

Image Restoration


Segmentation


Image Enhancement

Object Recognition

Problem Domain


Image Compression

Data Analytics 393


11.2 Image Segmentation

Data Analytics 394


Definition

• Image segmentation is the operation of partitioning an image into a collection of connected sets of pixels.

Data Analytics 395


Definition

A segmentation is a partition of an image I into a set ofregions S satisfying the following conditions:1. Partition covers the whole image.2. No regions intersect.3. Each region is homogeneous within itself.4. Adjacent regions form no homogeneous region when

united.

Data Analytics 396


Region Growing• Region growing techniques start with one pixel of a

potential region and try to grow it by adding adjacent pixels till the pixels being compared are too disimilar.

• The first pixel selected can be just the first unlabeled pixel in the image or a set of seed pixels can be chosen from the image.

• Usually a statistical test is used to decide which pixels can be added to a region, e.g., threshold on

to decide whether pixel with intensity y is added, where X denotes the mean, S standard deviation, and N is the number of pixels in the region.

(N-1) * NT = -------------- (y - X) / S

(N+1)

221/2

Data Analytics 397


Region Growing

• Result:

image

segmentation

Data Analytics 398


Clustering• Apply a clustering approach on the pixel intensities.• Histogram-based approaches try to automatically

split the range of the intensities by looking into minima of the histogram.

Data Analytics 399


Clustering

• We can use the clustering methods we have been looking into.

• For example, we can use k-means with a guessed number of segments k.

• There exist modifications to k-means that look into local statistics to consider spatial distribution of pixels.

Data Analytics 400


Clustering• Results for k-means:

Data Analytics 401


Clustering• Graph cut algorithms are widely used.• Let G = (V,E) be a graph. • The vertices V represent the pixels.• Each edge (u,v) has a weight w(u,v) that represents

the similarity between u and v.• The goal is to partition the vertices into disjoint sets

with high similarity within each set and low similarity across sets.

• Graph G can be broken into 2 disjoint graphs by removing edges that connect these sets.

• The segmentation is obtained by finding minimal cuts of G.

Data Analytics 402


Clustering

• Graph cuts with normalized cut:

2

2 2

2 2

41 3

2

2 2

3

22

2

1

3 3Ncut(A,B) = ------- + ------

21 16

A B

cut(A,B) = ∑ w(u,v).uεA, vεB

cut(A, B) cut(A,B)Ncut(A,B) = ------------- + -------------

asso(A,V) asso(B,V)

asso(A,V) = ∑ w(u,t)u∈A, t∈V

Data Analytics 403


Clustering

• Graph cut results:

Data Analytics 404


11.3 Image Collection Data Analytics

Data Analytics 405


Image collections

• Image segmentation helps to identify certain features in the image.

• They do not work perfectly.• However, if we are trying to perform a data analytics

approach on a collection of images, the data segmentation results are only of use for low-level semantics queries.

• Is there a way to analyze a collection of images on a higher level?

Data Analytics 406


Data analytics

• Assume that we can characterize an image by a number of image descriptors, we can try to compute similarities based on those descriptors.

• Then, we can build a similarity (or distance) matrix of pairwise (dis-)similarities of images.

• Based on the similarity (or distance) matrix, we can apply the data analytics approaches:– Cluster approaches,– Classification approaches,– Interactive visual analysis based on MDS projections.

Data Analytics 407


Data analytics example

• Corel data set includes 1,000 photographs on 10 different themes, described by 150 dimensions (SIFT descriptors).

• The Medical data set is of magnetic resonance (MRI) images and has 540 objects and 28 dimensions (Fourier descriptors and energies derived from histograms, plus mean intensity and standard deviation).

Data Analytics 408


Projection-based visualization of labeled data

Data Analytics 409


11.4 Image Descriptors

Data Analytics 410


Image descriptors

• Fourier analysis• Wavelet analysis• SIFT features• Color statistics

Data Analytics 411


Image Transforms

• Many times, image processing tasks are best performed in a domain other than the spatial domain.

• Key steps(1) Transform the image(2) Carry the task(s) in the transformed domain.(3) Apply inverse transform to return to the spatial domain.

Data Analytics 412


Fourier Series Theorem

• Any periodic function f(t) can be expressed as a weighted sum (infinite) of sine and cosine functions of varying frequency:

is called the “fundamental frequency”

Data Analytics 413


Fourier Series

α1

α2

α3

Data Analytics 414


Continuous Fourier Transform (FT)

• Transforms a signal (i.e., function) from the spatial (x) domain to the frequency (u) domain.

where

Data Analytics 415


Example: Removing undesirable frequencies

remove highfrequencies

reconstructedsignal

frequenciesnoisy signal

To remove certainfrequencies, set theircorresponding F(u)coefficients to zero!

Data Analytics 416


How do frequencies show up in an image?

• Low frequencies correspond to slowly varying pixel intensities (e.g., continuous surface).

• High frequencies correspond to quickly varying pixel intensities (e.g., edges)

Original Image Low-passed

Data Analytics 417


Example of noise reduction using FT

Input image

Output image

Spectrum (frequency domain)

Band-reject filter

Data Analytics 418


Extending FT in 2D

• Forward FT

• Inverse FT

Data Analytics 419


Discrete Fourier Transform

• Assume that f(x,y) is M x N.

• Forward DFT

• Inverse DFT:

Data Analytics 420


Extending DFT to 2D2D cos/sin functions

Interpretation:

Data Analytics 421


Magnitude and Phase of DFT

only phase

only magnitude phase (woman)magnitude (rectangle)

phase (rectangle)magnitude (woman)

Data Analytics 422


Wavelet transform

• An alternative to Fourier transforms are wavelet transforms.

• They have the advantage that they represent the image at multiple levels of details.

Data Analytics 423


B-spline representation

coarsening

details

Data Analytics 424


B-spline representation

Haar wavelet transform

Data Analytics 425


Haar wavelets

Basis function

Wavelet function

Data Analytics 426


Haar wavelets

Data Analytics 427


Multiresolution representation

Data Analytics 428


Multiresolution representation

• Object is represented as a sequence of resolutions.• The resolutions are called levels (levels of detail,

LOD)• The differences are called detail coefficients.• The levels build a multiresolution hierarchy:

• The level is the base level.• The base level does not need to be represented by a

regular mesh. All levels use then semi-regular meshes.

Data Analytics 429


Multiresolution representation with Haar wavelets

Data Analytics 430


2D Haar wavelet transform

• 2D basis and wavelet functions are tensor productsof 1D basis and wavelet functions.

Data Analytics 431


2D Haar wavelet transformBasis:

Data Analytics 432



Data Analytics 433


2D Haar wavelet transformAlternative construction:

Use 2D basis function

and three 2D wavelet functions

Data Analytics 434


2D Haar wavelet transformBasis:

Data Analytics 435



Advantage: One obtains undistorteddownscaled versions of the 2D image.

Data Analytics 436


2D wavelet transform in RGB space

Data Analytics 437


Image compression

100% 21% 4% 1%Error: 0% 5% 10% 15%

Haar wavelets:

Data Analytics 438


Image compressionJPEG 2000: Cohen-Daubechies-Feauveau wavelets

Data Analytics 439


Image compressionJPEG 2000: lossy compression leads to blurring.

Data Analytics 440


SIFT features

• The scale-invariant feature transform (SIFT) is another transform that can be used to describe image characteristics.

Data Analytics 441


Canonical Frames

Data Analytics 442


Multi-Scale Oriented Patches

Extract oriented patches at multiple scales

Data Analytics 443


Application: Image Stitching

Data Analytics 444


Multi-Scale Oriented Patches

1. Detect an interesting patch with an interest operator. Patches are translation invariant.

2. Determine its dominant orientation.3. Rotate the patch so that the dominant orientation

points upward. This makes the patches rotation invariant.

4. Do this at multiple scales, converting them all to one scale through sampling.

5. Convert to illumination “invariant” form

Data Analytics 445


Idea of SIFT

Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

Data Analytics 446


Claimed Advantages of SIFT

• Locality: features are local, so robust to occlusion and clutter (no prior segmentation)

• Distinctiveness: individual features can be matched to a large database of objects

• Quantity: many features can be generated for even small objects

• Efficiency: close to real-time performance• Extensibility: can easily be extended to wide range of

differing feature types, with each adding robustness

Data Analytics 447


Overall Procedure at a High Level1. Scale-space extrema detection

2. Keypoint localization

3. Orientation assignment

4. Keypoint description

Search over multiple scales and image locations.

Fit a model to determine location and scale.Select keypoints based on a measure of stability.

Compute best orientation(s) for each keypoint region.

Use local image gradients at selected scale and rotationto describe each keypoint region.

Data Analytics 448


11/15/2016 448

Using SIFT for Matching “Objects”

Data Analytics 449


Color statistics

• We have been looking at greyscale images (application: medical imaging data).

• We have been using histograms on color distribution.• However, photographs are typically color images.• How can we process color statistics?

Data Analytics 450


11.5 Color Models

Data Analytics 451


Electromagnetic spectrum

purple blue green yellow orange red

Data Analytics 452


Visible light spectrum

Data Analytics 453


Relative sensitivity of human eye

• The ability of the human eye to distinguish colors isbased on the sensitivity in the retina to light of different wavelength.

Data Analytics 454


Tristimulus

• The retina contains 3 types of receptor cells, which iscalled tristimulus.

• The receptors are most responsive to light of wavelengths 420nm, 534nm, and 564nm.

Data Analytics 455


RGB color model

Idea: • Use three wavelengths R, G, and B that reflect

monochromatic light and represent a tristimulus.• Other colors are obtained by combining/mixing the

three components R, G, and B.

Data Analytics 456


RGB color model

Implementation: • Choose three colors:

– R = red (700 nm)– G = green (546.1 nm)– B = blue (435.8 nm)

• Arrange them in a 3D Cartesian coordinate systemsuch that

Data Analytics 457


RGB color model

• This model allows the generation of colors c with

where .

Data Analytics 458


RGB color cube

• All colors c that can be generated are represented bythe unit cube in the 3D Cartesian coordinate system.

red

green

blue

black

white

grey

yellow

magenta

cyan

Data Analytics 459


Additive color scheme

• RGB color model is additive, i.e., adding colors makesthe resulting color brighter.

• Application: color monitors.

Data Analytics 460


Composition example

Data Analytics 461


Compositing in the RGB color cube

Data Analytics 462


Caveat

• It is often believed that the RGB color modelreflects the tristimulus of the human eye.

• This is wrong.• In particular, the large wavelength of the human eye‘s

tristimulus is 565 nm, which is not red but ratheryellow-green.

Data Analytics 463


Choice of RGB wavelength

• The choice of the wavelength in the RGB model has historical and practical reasons.

• When first monitors were developed, generatingmonochromatic light was a difficult task.

• The chosen wavelength were those that could begenerated most easily.

Data Analytics 464


Reconstruction of visible spectral light• In order to reconstruct all wavelengths of the visible

spectral light, we have to add the R, G, and B components with the following weighting factors:

Data Analytics 465


Reconstruction of visible spectral light

• The negative values indicate that not all visible colorscan be produced with the RGB color model.

• Nevertheless, close approximations can be achieved.


11.6 Assignment

Data Analytics 467


Assignment 9• Download and extract the archive of an image collection data set from

http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/oxbuild_images.tgz. The data set consists of around 5,000 images of different buildings in Oxford. If processing of the entire data set takes too long, you may use a subset, but make sure the subset contains images of different buildings. Then, convert the color images to grayscale images.

• Approach 1: Interpreting the grayscale images as 1D vectors of pixel intensities, run a clustering algorithm with an appropriate distance metric. What do the clusters represent?

• Approach 2: Transform the greyscale images into the space of SIFT descriptors. How many dimensions does this space have?Hint: You may use the Python bindings of OpenCV. Here is a tutorial: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/ py_feature2d/py_sift_intro/py_sift_intro.html.Apply a clustering approach to the SIFT descriptors and visualise the result in an MDS projection (in 2D). What do the clusters represent?p.t.o

Data Analytics 468


Assignment 9• Approach 3: Exchange the order of the two analysis steps in

Approach 2, i.e., first project the SIFT descriptors to a 2D space using an MDS approach and then cluster the 2D points using the same clustering approach as above. Again, visualise the result. What do the discovered clusters represent?

• How do the three approaches compare?

11. image data analytics - jacobs university bremen...data analytics 405 visualization and computer...

Documents