Hiroyuki Takeda, Hae Jong Seo, Peyman MilanfarEE Department

University of California, Santa Cruz

Jan 11, 2008

Statistical Image Quality Measures

UCSC MDSP Lab

Overview

BackgroundBackground

CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)

Slide 1

ConclusionConclusion

SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)

UCSC MDSP Lab

Develop quantitative measures that Develop quantitative measures that automatically predict the perceived image automatically predict the perceived image qualityquality

Objective Quality AssessmentObjective Quality Assessment

Full-reference Full-reference

No-reference No-reference

Reduced-reference Reduced-reference

Applications Applications

Image acquisition, compression, communication,Image acquisition, compression, communication,

displaying, printing, restoration displaying, printing, restoration

Slide 2

UCSC MDSP Lab

Overview

BackgroundBackground

CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)

ConclusionConclusion

SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)

UCSC MDSP Lab

Full-Reference Image Quality MeasureFull-Reference Image Quality Measure

Structural Similarity Measure [1]Structural Similarity Measure [1]

Focus on Focus on perceived changes in structural informationperceived changes in structural information variationvariation unlike error based approach ( i.e. MSE or unlike error based approach ( i.e. MSE or PSNR )PSNR )

MSE : 210MSE : 210

Original imageOriginal image

Contrast stretchedContrast stretched

JPEG compressedJPEG compressed Blurred Blurred

[1] [1] Zhou Wang et al,  “ et al,  “Image Quality Assessment: From Error Visibility to Structural SimilarityImage Quality Assessment: From Error Visibility to Structural Similarity ”, ”, IEEE TIP ‘IEEE TIP ‘ 04 04

Slide 3

Salt-pepperSalt-pepper

Mean shiftedMean shifted

UCSC MDSP Lab

Structural Similarity MeasureStructural Similarity Measure

Three components : Luminance , Contrast , StructureThree components : Luminance , Contrast , Structure

Small constant Small constant

Slide 4

Image patches being Image patches being compared compared

UCSC MDSP Lab

Drawback of SSIMDrawback of SSIM

Sensitive to spatial translation, rotation, and scale Sensitive to spatial translation, rotation, and scale changeschanges

due to due to simple correlation coefficientsimple correlation coefficient

A powerful statistical toolA powerful statistical tool

: Canonical Correlation Analysis (Hotelling, : Canonical Correlation Analysis (Hotelling, 1936)1936)

Solution Solution

OriginalOriginal RotatioRotationn

Zoom OutZoom Out TranslatioTranslationn

SSIM: 0.549SSIM: 0.549 SSIM: 0.551SSIM: 0.551SSIM: 0.505SSIM: 0.505

Slide 5

UCSC MDSP Lab

: : canonical correlationcanonical correlation

New Statistical Image Quality MeasureNew Statistical Image Quality Measure

Canonical Correlation Analysis (CCA)Canonical Correlation Analysis (CCA): : Find out a pair of direction vectors whichFind out a pair of direction vectors which

maximally correlatemaximally correlate the two datasets the two datasets

: : Useful propertyUseful property Affine–invariance Affine–invariance

Slide 6

UCSC MDSP Lab

CCA

CCA

CCA

New Statistical Image Quality MeasureNew Statistical Image Quality Measure

Canonical Correlation Structural Similarity Canonical Correlation Structural Similarity MeasureMeasure

original image

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

noisy, sigma= 35

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500Gx GyP Gx GyP

P P

GyGx Gx Gy

: : Local Search Window at i Local Search Window at i thth position position

Slide 7

P : Pixel intensity

Gx,Gy : Gradients

AA BB

UCSC MDSP Lab

New Statistical Image Quality MeasureNew Statistical Image Quality Measure

Mathematical SolutionMathematical Solution

1) Calculate Covariance Matrix1) Calculate Covariance Matrix

2) Solve coupled eigen-value problems2) Solve coupled eigen-value problems

3) Define CCSIM as largest canonical 3) Define CCSIM as largest canonical correlation correlation

Slide 8

UCSC MDSP Lab

original image

Examples (1)Examples (1)

Original Original ImageImage

Zoom OutZoom Out

Slide 9

11 22

UCSC MDSP Lab

Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.73212) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 SSIM: MSSIM = 0.3419WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

original image

Examples (2)Examples (2)

CCSIMCCSIMSSISSIMM

11

0.340.34 0.730.73

22

Zoom OutZoom Out

Dis

trib

ution o

f C

C1(C

om

pre

ssed): P

ixel -->M

ean(0

.3098) B

lock S

ize:5

50

100

150

200

50

100

150

200

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2211Original ImageOriginal Image

Slide 10

UCSC MDSP Lab

original image

Examples (2)Examples (2)

Original Original ImageImage

TranslationTranslation

11 33

Slide 11

UCSC MDSP Lab

Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.75026) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 SSIM: MSSIM = 0.38452WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

original image

Examples (2)Examples (2)

CCSIMCCSIMSSISSIMM

11

0.380.38 0.750.75

33

Original ImageOriginal Image

TranslationTranslation

Dis

trib

ution o

f C

C1(C

om

pre

ssed): P

ixel -->M

ean(0

.3098) B

lock S

ize:5

50

100

150

200

50

100

150

200

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

3311

Slide 12

UCSC MDSP Lab

original image

Examples (3)Examples (3)

Original Original ImageImage

RotationRotation

11 44

Slide 13

UCSC MDSP Lab

Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.77315) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 SSIM: MSSIM = 0.41067WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

original image

CCSIMCCSIMSSISSIMM

11

0.410.41 0.770.77

44

Original ImageOriginal Image

RotationRotation

Dis

trib

ution o

f C

C1(C

om

pre

ssed): P

ixel -->M

ean(0

.3098) B

lock S

ize:5

50

100

150

200

50

100

150

200

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

4411

Slide 14

Examples (3)Examples (3)

UCSC MDSP Lab

JPEG Compression ExampleJPEG Compression Example

Slide 15

Clean imageClean image(QF=100)(QF=100) JPEG(QF=10)JPEG(QF=10)JPEG(QF=50)JPEG(QF=50)

11 22 33

0.899 bits/pixel0.899 bits/pixel 0.352 bits/pixel0.352 bits/pixel8 bits/pixel8 bits/pixel

UCSC MDSP Lab

22

11

SSIM: MSSIM = 0.90153WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.900.90

Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.8533) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.850.85

SSISSIMM

Clean ImageClean Image

JPEG (QF =50)JPEG (QF =50)

JPEG (QF =10)JPEG (QF =10)

Slide 16

JPEG Compression ExampleJPEG Compression Example

11

22

33

Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5

50 100 150 200

50

100

150

200

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

CCSIMCCSIM

SSIM: MSSIM = 0.786WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Distribution of CC1(compressed): Pixel + Gradient Value -->Mean(0.79459) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

33

11

SSISSIMM

0.790.79

0.790.79

CCSIMCCSIM

UCSC MDSP Lab

Clean Image VS Compressed ImagesClean Image VS Compressed Images

Slide 17

QualityQuality

JPEG quality JPEG quality factorfactor

10 20 30 40 50 60 70 80 90 1000.75

0.8

0.85

0.9

0.95

1

Pixel + Gradient : Window Size = 5SSIM : Window Size = 5

SSISSIMM

CCSIMCCSIM

UCSC MDSP Lab

Denoising ExampleDenoising Example

Clean ImageClean Image Denoised by Denoised by SKR[2]SKR[2]

WGN(sigma=15WGN(sigma=15))

Slide 18

[2] Takeda[2] Takeda et al.,  “ Kernel Regression for image processing and reconstruction ”, et al.,  “ Kernel Regression for image processing and reconstruction ”, IEEE TIP ‘IEEE TIP ‘ 07 07

denoised by SKR, sigma= 15noisy, sigma= 15 original image

11 22 33

UCSC MDSP Lab

SSIM: MSSIM = 0.47514WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Distribution of CC1(Noisy): Pixel -->Mean(0.4755) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Denoising ExampleDenoising Example

CCSIMCCSIM

SSISSIMM

Slide 19

Clean ImageClean Image

WGN( sigma =15 )WGN( sigma =15 )

Denoised by SKRDenoised by SKR

SSIM: MSSIM = 0.88668WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Distribution of CC1(Denoised): Pixel + Gradient Value -->Mean(0.88536) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

denoised by SKR, sigma= 15

noisy, sigma= 15

original image

Distribution of CC1(Compressed): Pixel -->Mean(0.3098) Block Size:5

50 100 150 200

50

100

150

200

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

CCSIMCCSIM

SSISSIMM

11

22

33

22

11

33

110.470.47

0.470.47

0.890.89

0.890.89

UCSC MDSP Lab

Clean VS (Noisy & Denoised images)Clean VS (Noisy & Denoised images)

WGN: Noise levelWGN: Noise level

QualityQuality

Slide 20

QualityQuality

WGN: Noise levelWGN: Noise level

5 10 15 20 25 300.5

0.6

0.7

0.8

0.9

Pixel+Gradient: Window Size = 5SSIM : Window Size = 5

5 10 15 20 25 300.2

0.4

0.6

0.8

1

Pixel: Window Size = 5SSIM : Window Size = 5

Clean VS NoisyClean VS Noisy Clean VS DenoisedClean VS Denoised

SSISSIMM

CCSIMCCSIM

SSISSIMM

CCSIMCCSIM

UCSC MDSP Lab

MotionEstimation

Resolution enhancement from video frames captured by a commercial webcam(3COM Model No.3719)

Steering Kernel Regression

Super-resolutionSuper-resolution

Slide 21

UCSC MDSP Lab

original image superresolved by SKR

Super-resolution ExampleSuper-resolution Example

Clean ImageClean Image

(512 x 512)(512 x 512)Super-resolved Super-resolved by SKRby SKR

Low resolution Low resolution

SequenceSequence

(64x64 32 frames)(64x64 32 frames)

Slide 22

11 22 33

UCSC MDSP Lab

Distribution of CC1(Denoised): Pixel + Gradient Value -->Mean(0.91575) Block Size:3

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 SSIM: MSSIM = 0.86996WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Super-resolution ExampleSuper-resolution Example

SSISSIMM

Slide 23

Clean ImageClean Image

Super-resolved by Super-resolved by SKRSKR

Low resolution Low resolution Sequence( 32 frames)Sequence( 32 frames)

Dis

trib

utio

n of

CC

1(C

ompr

esse

d): P

ixel

-->

Mea

n(0.

3098

) B

lock

Siz

e:5

5010

015

020

0

50 100

150

200

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

CCSIMCCSIM

331111

22

33

0.870.87 0.910.91

original image

superresolved by SKR

UCSC MDSP Lab

Overview

BackgroundBackground

CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)

ConclusionConclusion

SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)

UCSC MDSP Lab

No-Reference SVD-Based MeasureNo-Reference SVD-Based Measure

SVD N x NSingular value decomposition of local gradient matrix:Singular value decomposition of local gradient matrix:

Local orientation dominanceLocal orientation dominance

It becomes close to 1 when there is one dominant It becomes close to 1 when there is one dominant orientation orientation

in a local area.in a local area.It takes on small values in flat or highly textured (or pure It takes on small values in flat or highly textured (or pure noise) noise)

area. area. So, this quantity tells us about the “edginess” of the So, this quantity tells us about the “edginess” of the region beingregion being

examined.examined.

UCSC MDSP Lab

Properties of Local Orientation Dominance(1)Properties of Local Orientation Dominance(1)

Slide 25

Density function for i.i.d. white Gaussian noiseDensity function for i.i.d. white Gaussian noise

N: the window size

[1] A. Edelman. Eigenvalues and condition numbers of random matrices, SIAM Journal on Matrix Analysisand Applications 9 (1988), 543-560.

Note : the PDF is independent from the noise variance, but depends on the window size.

[2] X. Feng and P. Milanfar. Multiscale principal component Analysis for Image Local Orientation Estimation, Proceedingof 36th Asilomar Conference on Signals, Systems, andComputers, Pacific Grove, CA, Nov. 2002

N=3

N=5

N=7

N=9

N=11

UCSC MDSP Lab

Properties of Local Orientation Dominance(2)Properties of Local Orientation Dominance(2)

Slide 26

The mean values for a variety of test images with The mean values for a variety of test images with added white Gaussian noise.added white Gaussian noise.

N = 11

The mean values for pure noise are always constant.

Rememberthe number

0.06

UCSC MDSP Lab

The Performance AnalysisThe Performance Analysis

Suppose we have a noisy image and a denoised version Suppose we have a noisy image and a denoised version

using some filter:using some filter:

The residual image is essentially just noise. The residual image is essentially just noise.

of the residual image must be close to the value of the residual image must be close to the value expectedexpected

for pure noise.for pure noise.

Slide 27

If the filter cleans up the given image effectively, If the filter cleans up the given image effectively,

: a given noisy image

: the estimated (denoised) image

: the residual image

UCSC MDSP Lab

Example (1)Example (1)

Image denoising by bilateral filterImage denoising by bilateral filter

Slide 28

Denoising experimentDenoising experiment

Bilateral filter has two parameters:Bilateral filter has two parameters:

Spatial smoothing parameterSpatial smoothing parameter , and , and radiometric smoothing parameterradiometric smoothing parameter

The original image A noisy image, Added white Gaussian noise,SNR=20dB, PSNR=29.25dB, RMSE = 8.67

C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images”, Proceedings of the 1998 IEEEInternational Conference of Computer Vision, Bombay, India, pp. 836-846, January 1998.

UCSC MDSP Lab

The Performance Analysis of Bilateral FilterThe Performance Analysis of Bilateral Filter

Slide 29

N = 11

The plot of as a function of the smoothing parameters: The plot of as a function of the smoothing parameters:

UCSC MDSP Lab

Denoising ResultDenoising Result

Slide 30

Bilateral filterPSNR = 42.87dB,RMSE = 1.833

The noisy imageResidual

UCSC MDSP Lab

The Performance Analysis of Bilateral FilterThe Performance Analysis of Bilateral Filter

Slide 31

The plot of as a function of the smoothing parameters: The plot of as a function of the smoothing parameters:

N = 11

UCSC MDSP Lab

Denoising ResultDenoising Result

Slide 32

Bilateral filterPSNR = 39.57dBRMSE = 2.68

The noisy imageResidual

The filter also removes image

contents.

UCSC MDSP Lab

What If We Pick the Parameters by the Best RMSE?What If We Pick the Parameters by the Best RMSE?

Slide 33

The plot of RMSE as a function of the smoothing The plot of RMSE as a function of the smoothing parameters: parameters:

UCSC MDSP LabSlide 34

Denoising Result Denoising Result

Bilateral filter, PSNR = 42.87dBRMSE = 1.832

The noisy imageResidual

UCSC MDSP LabSlide 35

Example (2)Example (2)

Iterative Steering Kernel RegressionIterative Steering Kernel Regression

The original image The noisy image,Added white Gaussian noise,SNR=5.6dB, PSNR = 20.22dB RMSE = 24.87

Iteratively cleaning up noisy imagesIteratively cleaning up noisy images

Using the local orientation dominance, we find the optimal Using the local orientation dominance, we find the optimal number of iterations.number of iterations.

UCSC MDSP LabSlide 36

Denoising Result (1)Denoising Result (1)

The plot of as a function of the smoothing parameters:The plot of as a function of the smoothing parameters:

UCSC MDSP LabSlide 37

Denoising Result Denoising Result

ISKR, IT = 15,PSNR = 31.33 dB

RMSE = 6.92

The noisy imageResidual

UCSC MDSP LabSlide 38

If the Ground Truth is Available,If the Ground Truth is Available,

The plot of RMSE as a function of the smoothing The plot of RMSE as a function of the smoothing parameters:parameters:

RMSERMSE

UCSC MDSP LabSlide 39

Denoising Result Denoising Result

ISKR, IT = 12, PSNR = 31.69 dB

RMSE = 6.64

The noisy imageResidual

UCSC MDSP Lab

Overview

BackgroundBackground

CCA-based Similarity Measure (Full-reference)CCA-based Similarity Measure (Full-reference)

ConclusionConclusion

SVD-based Quality Measure (No-reference)SVD-based Quality Measure (No-reference)

UCSC MDSP Lab

ConclusionConclusion

Two new statistical quality measuresTwo new statistical quality measuresCCSIM(CCA-based)CCSIM(CCA-based) : full-reference : full-reference

Slide 40

SVD-based measure: SVD-based measure: no-referenceno-reference

CCSIM is a general version of SSIMCCSIM is a general version of SSIM

The proposed methods can be easily The proposed methods can be easily extended to video using 3-d local window.extended to video using 3-d local window.

We showed examples of JPEG compression, denoising , and We showed examples of JPEG compression, denoising , and supersuper

Resolution with comparison to SSIMResolution with comparison to SSIMSVD-based measure is applicable for any SVD-based measure is applicable for any denoising filter.denoising filter.

We illustrated application to global parameter We illustrated application to global parameter optimization.optimization.Locally adaptive parameter optimization is also possible.Locally adaptive parameter optimization is also possible.

UCSC MDSP Lab

AuthorsAuthors

[1] Hiroyuki Takeda : [1] Hiroyuki Takeda : hiro@soe.ucsc.edu

www.ucsc.edu/~htakedawww.ucsc.edu/~htakeda

[2] Hae Jong Seo : [2] Hae Jong Seo : rokaf@soe.ucsc.edu

www.ucsc.edu /~rokafwww.ucsc.edu /~rokaf

[3] Peyman Milanfar : [3] Peyman Milanfar : milanfar@soe.ucsc.edu

www.ucsc.edu/~milanfarwww.ucsc.edu/~milanfar

UCSC MDSP Lab

Thank you !Thank you !

UCSC MDSP Lab

Super-resolution ExampleSuper-resolution Example

Clean ImageClean ImageSuper-resolved Super-resolved by SKRby SKR

Down-sampled(2)Down-sampled(2)

+WGN(sigma=15)+WGN(sigma=15)

Extra 1

noisy, sigma= 15 original image

11 22 33

UCSC MDSP Lab

SSIM: MSSIM = 0.70632WinSize = 5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Super-resolution ExampleSuper-resolution Example

SSISSIMM

Extra 2

Clean ImageClean Image

Super-resolved by Super-resolved by SKRSKR

Distribution of CC1(Noisy): Pixel + Gradient Value -->Mean(0.85761) Block Size:5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

original image

Down-sampled(2)Down-sampled(2)+WGN (sigma=15)+WGN (sigma=15)

noisy, sigma= 15

Dis

trib

utio

n of

CC

1(C

ompr

esse

d): P

ixel

-->

Mea

n(0.

3098

) B

lock

Siz

e:5

5010

015

020

0

50 100

150

200

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

CCSIMCCSIM

331111

22

33

0.710.71 0.850.85