Image Quality Assessment

Rajiv Soundararajan

Department of ECE

Indian Institute of Science

Classification of QA

• Full reference QA

• Applications – compression, transmitter end processing

• SSIM, VIF, VSNR, JND (Sarnoff) – examples of FR IQA algorithms

• PSNR/MSE is also a FR IQA

No reference (NR) QA

• Relaxation - know distortion type/types that image has been subjected to e.g. JPEG2000, JPEG, blur

• Application - camera tuning

Is this image of good quality?

Reduced reference (RR) QA

• General purpose or distortion specific

• Design questions - what and how much information is required

Application

• Quality monitoring in networks

• Particularly useful where it is expensive/impossible to obtain reference for quality computation

MSE/PSNR

• Mean squared error (MSE) – most widely used image QA algorithm

• Peak signal to noise ratio (PSNR) – equivalent to MSE

– L: max pixel value (typically equal to 255)

Z. Wang and A. C. Bovik, “Mean squared error: love it or leave it? – A new look at signal fidelity measures,” IEEE Signal Processing Magazine, vol. 26, no. 1, pp. 98-117, Jan. 2009

Analysis of MSE/PSNR

• Advantages– Computationally simple algorithm– Distance metric properties

• Nonnegativity - MSE(x,y) >=0• Identity – MSE(x,y) = 0 iff x=y• Symmetry – MSE(x,y) = MSE(y,x)• Triangle inqeuality – MSE(x,y)

Einstein w/different distortions:

(a) original image

(b) mean luminance shift

(c) contrast stretch

(d) impulse noise

(e) Gaussian noise

(f) Blur

(g) JPEG compression

(h) spatial shift (to the left)

(i) spatial scaling (zoom out)

(j) rotation (CCW).

Images (b)-(g) have nearly

identical MSEs but very

different visual quality.

9

(b) MSE = 309

(a)

(c) MSE = 306 (d) MSE = 313

(e) MSE = 309 (f) MSE = 308 (g) MSE = 309

(h) MSE = 871 (i) MSE = 694 (j) MSE = 590

Failure of MSE – independent of spatial relationships

Failure of MSE – independent of relationship between signal and error

Failure of MSE – independent of sign of error samples

Failure of MSE – all signal errors are equally important

Human opinion of quality

• Human opinion – ultimate measure of quality

• Subjective study – collect human opinion scores of images viewed under controlled calibrated conditions

• Mean opinion score (MOS) – averaged over all subjects with removal of outliers

• QA algorithm needs to correlate well with MOS

Full reference IQA

• Seek full reference image QA algorithm that correlates well with human judgment of quality

• Various approaches

– Human vision based – JNDMetrix

• Uses complex models of human visual system

– Structural – SSIM, MS-SSIM

– Statistical – VIF, IFC

Structural similarity based metrics

• Measure loss of structure in the image as opposed to just any deviation with respect to reference

• Loss of image structure measured locally through

– Luminance similarity

– Contrast similarity

– Structural similarity

• Perform average of local measure across the image

Structural similarity index (SSIM)

• SSIM has revolutionalized QA – many applications in other domains as well

• L(i,j) – luminance similarity at location (i,j)

• C(i,j) – contrast similarity

• S(i,j) – structure similarity Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Transactions on Image Processing , vol. 13, no. 4, pp. 600 - 612, April, 2004

Luminance similarity

• Luminance similarity term

• Weighted local average intensity

• Isotropic unit area weighting function

• C_1 - numerical stabilizing constant

Contrast similarity

• Contrast similarity term

• Weighted local standard deviation of intensity

• C_2 - numerical stabilizing constant

Structural similarity

• Structural similarity term

• Weighted local correlation of intensities

• C_3 – numerical stabilizing constant

SSIM properties

• Symmetric

• Boundedness

• Unique maximum

if and only if images are locally identical

Stabilizing Constants

• The constants C1, C2 and C3 stabilize the three terms – in case the local means or contrasts are very small.

• For gray-scale range 0-255, C1 = (0.01·255)2, C2 = (0.03·255)

2, C3= C2/2 work well and robustly.

• If C1 = C2 = C3 = 0, then

• This original version of SSIM is known as Universal Quality Index (UQI).

22

Quality map - SSIM

• Visualize quality degradations at a pixel level

• SSIM map displays masking of distortions

– Less noise visible near edges

– More noise in flat regions

– Connections with contrast masking principles of human visual system

(a) reference image; (b) JPEG compressed;

(c) absolute difference; (d) SSIM Index Map.

24

(a) (b)

(d)(c)

(a) reference image; (b) JPEG2000 compressed;

(c) absolute difference; (d) SSIM Index Map25

(a) (b)

(d)(c)

(a) original image; (b) additive white Gaussian noise;(c) absolute difference; (d) SSIM Index Map.

26

(a) (b)

(d)(c)

Mean SSIM

• The SSIM map is averaged over all pixel locations in an image

• Mean SSIM correlates very well with human response as measured through subjective studies and MOS

SSIM vs. MOS

28

On a broad database of imagesdistorted by jpeg, jpeg2000,white noise, gaussian blur,and fast fading noise.

Best fitting logistic function

Pearson linear correlation coefficient

x_i – logistic fit of objective scores of ith imagey_i – MOS of ith image

Spearman rank order correlation coefficient

p_i – rank of objective score of ith imageq_i – rank of MOS of ith image

SSIM vs. The Competition

29

(a)

(b) (d)

(c)

Multi-scale SSIM

Exponents learned via subjective experiments

Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multi-scale Structural Similarity for Image Quality Assessment," Proc. 37th IEEE Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, November, 2003