mehdi rezagholizadeh: image sensor modeling: color measurement at low light levels

Color Perception at Low Signal

to Noise Levels

By:

Mehdi REZAGHOLIZADEH

MCGILL UNIVERSITY

DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING

Supervisor:

Prof. James J. CLARK

Brief Overview

DECEMBER 2014

Statement of Problems

• Developing a Fast and Accurate Color

Constancy Method

• Color Appearance Modeling at Low Light

Levels

• Image Sensor Modeling and Color

Measurement at Low Light Levels

2

Definition of Color Constancy

• Discounting the illuminant effect on the color

of objects

• Color constancy is a great feature of our

visual system

3Computational Color Constancy

Importance of Color Constancy• Applications:

� Object Recognition

� Image Enhancement

� Robot Vision

� Object Tracking

� Photography and Film Industry

The last three cases are

real-time applications of color constancy.


Problem of Color Constancy


Problem of Color Constancy

• Image Formation Model:

�� = �� , � � �� : the illuminant spectrum

�(�, ): the surface spectral reflectance function

at location .

�(�): the sensor spectral sensitivity

��: the sensor response of ith channel


Problem of Color Constancy• The Transformation Imposed by a Change in Illumination:

�� = �� , � � ��Givens:

�(�): the sensor spectral sensitivity

��: the sensor response of ith channel

Unknowns:

� � : the illuminant spectrum

�(�, ): the surface spectral reflectance function at location .

Spectral Color Constancy Approaches try to find the entire spectrum of the illuminant and the surface spectral reflectance function.

It is an ill-posed problem.


Methods of Color Constancy

Computational Color Constancy

Spectral Methods

Non-spectral Methods

Static Methods

Gamut-Based

Learning-Based


Non-Spectral Methods:• MAIN Objective of this problem is to obtain:

�� = �� , � � ��It is equivalent to obtaining the sensor responses when � � = 1.

• Assuming that color of the illuminant can be estimated:

�� = �� • The Transformation Imposed by an illuminant can be

obtained through an Over-simplification:

�� ≅ ��

= �� , � � ��

9Computational Color Constancy, March. 2014

Non-Spectral Methods:

Over-simplification leads to:

- Estimating the illuminant color

rather than

- Estimating the entire spectrum of the illuminant �(�)• Corrective Transformation:

��=

1�� 0 0

0 1��

0

0 0 1��

��

10Computational Color Constancy, March. 2014

Problem II: Color Appearance Modeling at Low Light Levels

• Color Appearance Model (CAM):

• An ideal color appearance model:

The output resembles human perception in all conditions including different light levels

• Lack of a good color appearance model

for low light conditions

Color Perception at Low Signal to Noise Levels 11

TransformTransformTristimulus

values (RGB)

Perceptual attributes of

color:

lightness, hue, chroma

Biophysical Background

• Our eye can work in three different modes:

1- Photopic condition (Luminance>5 cd/m2 )

2- Mesopic condition (0.005<Luminance<5 cd/m2 )

3- Scotopic condition (Luminance<0.005 cd/m2 )

• Photopic Condition: (High Light Levels)


• Mesopic Condition: (Low Light Levels)• Scotopic Condition: (Very Low Light Levels)

Background and Preliminaries • Existing Models for Mesopic & Scotopic Vision:

� Modeling Blue Shift in Moonlit scenes [1]– Addresses scotopic vision by adding some blue to the initial image

– The output of this algorithm does not look natural and realistic

� Cao’s Model of Mesopic Vision [2]– It is a two stage model based on the gain control and cone opponent

mechanisms

– Model is fitted to the psychophysical experiment data

� iCAM06 Tone Compression Model for Mesopic Vision [3]– iCAM06 includes rod responses in a linear fashion

� Shin’s Color Appearance Model [4]– Boynton two-stage model is fitted to the behavioral experiment data

13

[1] S. M. Khan and S. N. Pattanaik, “Modeling blue shift in moonlit scenes by rod cone interaction,” Journal of VISION, vol. 4, no. 8,

2004.

[2] D. Cao, J. Pokorny, V. C. Smith, and A. J. Zele, “Rod contributions to color perception: linear with rod contrast," Vision research, vol.

48, no. 26, pp. 2586-2592, 2008.

[3] J. Kuang, G. M. Johnson, and M. D. Fairchild, “iCAM06: a rened image appearance model for HDR image rendering," Journal of

Visual Communication and Image Representation, vol. 18, no. 5, pp. 406 -414, 2007.

[4] J. Shin, N. Matsuki, H. Yaguchi, and S. Shioiri, “A color appearance model applicable in mesopic vision," Optical review, vol. 11, no.

4, pp. 272-278, 2004.

Physics & Color Perception


Problem

• Lack of a good color appearance model (CAM) for the low light conditions

Physics

• The basic physical principles governing the probabilistic nature of color perception at low light levels

Analysis

• Photon Detection and Color Perception at low light levels

Proposed Method: Maximum Entropy Spectral Modeling Approach to the

Low Light Levels Color Appearance Modeling

• Under very low light conditions:

The photoreceptor responses more uncertain

• Hypothesis:


Visual Processing Center reconstructs a part of the information being lost in the projection of light spectra into the space

of photoreceptor responses



• The spectral theory of color perception [Clark and Skaff,

2009]:

�Provides a tool to address the issues of uncertain

measurements

� Estimates the spectral power distributions corresponding

to these uncertain measurements.




• Spectral Model of Mesopic Vision

� Clark and Skaff proposed a spectral model for color perceptionwhich is valid for photopic conditions

� During the mesopic condition, both cones and rods contribute to thevision

� Given the measurement vector �, we can model the rod intrusioninto the perception as follows:

�� = � � �� + �� ! � �� + "# $ ∈ {',(, �}

- +,(-) and +.(-): cone and rod spectral sensitivity functions respectively

- /(-): normalized mesopic spectral power distribution

- �: intensity factor

- 0 [0, 1]: a parameter which determines relative rod intrusion

- 1 = [�3 �4 ��]: a diagonal matrix specifies the relative contribution of rod response to each conechannel.

- ": additive noise

17Color Perception at Low Signal to Noise Levels




� Given the measurement vector �, we can model the rod intrusioninto the perception as follows:

�� = � � �� + �� ! � �� + "# $ ∈ {',(, �}

18

19



� An exponential family is employed to estimate ! � :

!̂ � = exp(< � � , ; > −>(;))� � = �� + �1� �;: parameter vector which should be estimated

>(;): normalizing function

� Parameters can be estimated as follows:

;? = minC { DE − D FG(DE − D)} − HI(;)}D = �/� normalized measurement

I(;): entropy function corresponding to !̂(�)A: positive definite matrix

K: regularization factor


Color Perception at Low Signal to Noise Levels

Results:

• Simulation of Munsell patches

– surrounded by a white background

– viewed under different light levels from scotopic to

photopic.


Image Sensor Modeling:

Color Measurement at Low Light Levels

By:

Mehdi REZAGHOLIZADEH

James J. CLARK

MCGILL UNIVERSITY

November 2014

22nd Color and Imaging Conference

The Presentation Outline:

Image Sensor Modeling: Color Measurement at Low Light Levels 22

Conclusion

Experiments and Results:

Preparation Caveats Analysis Experiment Scenarios

Solution: Image Sensor Modeling

Physical Background Noise Model Pixel Measurement Model

Introduction:

Motivation Statement of the Problem: Color Measurement at Low Light Levels

Introduction:Motivation

Importance of Studying low light levels:

• Color Measurement at low light level becomes more uncertain due to the low signal to noise ratio

• Most of the theories, measures, models and methods in color science are developed for high intensities

• The quality of the human color vision at low light levels is much better than existing handy cameras

23Image Sensor Modeling: Color Measurement at Low Light LevelsKirk, Adam G., and James F. O'Brien. "Perceptually based tone

mapping for low-light conditions." ACM Trans. Graph. 30.4 (2011): 42.

Introduction:Statement of the Problem

24

Problem:

• What is the impact of noise at low lightlevels on the color measurements ofimaging devices?

Image Sensor Modeling: Color Measurement at Low Light Levels

Applications of the Study:

� Spectral Imaging

� Image Processing

� Low Light Photography

� Characterizing the Noise of Image Sensors

�Developing Denoising and Enhancement Algorithms

� Photon Limited Imaging (biosensors, astronomy, etc)

25Image Sensor Modeling: Color Measurement at Low Light Levels

What Next…

26

Conclusion





Introduction:



Physical Background

27

• Simulating the effect of Photon Noise (given the high

intensity description of the light):

• For each bin:

L M(��), N � O PQ RSTU VQW!

0

2

4

6

8

104

00

42

0

44

0

46

0

48

0

50

0

52

0

54

0

56

0

58

0

60

0

62

0

64

0

66

0

68

0

70

0

Av

era

ge

Ph

oto

n C

ou

nt

: g

(λ)

Wavelength (nm)

ᵟ


Physical Background

28

• A set of Poisson distributions (one for each bin)

characterizes the targeted light.

• To estimate the spectral radiance at a lower intensity:

• The estimated quantal spectral radiance:

'YZ[ �� = \]Z(��)^

0

10

g(λ

)

Wavelength (nm)


_ = lowintensityhighintensity Draw samples from

hijk l m n -j op qrl -j ~hijk l m n -j��

Simulation:

How Does Spectral Power Distribution Change with Intensity?

• The estimated spectral power distribution at

different intensities. _ = 5 m 10u��1vww

^ � 5Nxw � 0.2{|}


Simulation:



different intensities.


_ � 5 m 10u��1vww^ � 5Nxw � 0.2{|}

Simulation:



different intensities.


_ � 5 m 10u�~1vww^ � 5Nxw � 0.2{|}

Image Sensor Modeling

32

• Image sensor pipeline (for a single channel):

Noise Model

Photon Shot Noise

Dark Current Noise

Read Noise

Quantization Noise



33


Noise Model

Photon Shot Noise

Dark Current Noise

Read Noise

Quantization Noise



34


Noise Model

Photon Shot Noise

Dark Current Noise

Read Noise

Quantization Noise



35


Noise Model

Photon Shot Noise

Dark Current Noise

Read Noise

Quantization Noise



36


Noise Model

Photon Shot Noise

Dark Current Noise

Read Noise

Quantization Noise


Image Sensor Modeling: Noise Model

37

• Variations in the number of emitted photons

• Can be modeled by a Poisson DistributionPhoton Shot Noise

• The current produced inside the image sensor

• �� (�, �)~L�${( �� )Dark Current

Noise

• The noise in the readout circuit

• � S��~�(0, � S��)Read Noise

• The error introduced in the quantization stepQuantization

Noise


Image Sensor Modeling: Pixel Measurement Model

38

Output of the Image Sensor

• �� , � � \�ST m �� m �'YZ[ �, �, � �S� � �� m �� , ��

Measured Voltage

• �] � �, � � �� , � � � S�� , ��

Raw Output

Image

• �� , � � q m �r� �, ��


What Next…

39

Conclusion





Introduction:



Experiments & Results:Dataset and Preparation

� Dataset: “A data set for Color Research”

� By: Barnard et al.

� Includes:

- The Sony DXC-930 sensor sensitivity curves

- The spectra and color measurements of 598 color samples

made by the Sony camera

40[1] K. Barnard, L. Martin, B. Funt, and A. Coath, “A data set for color research,” Color Research & Application, vol. 27, no. 3, pp.

147-151, 2002.

Experiments & Results:Dataset and Preparation

� Preparation:

− 20 samples from the 598 color measurements are selected

for our experiments

− By scaling the initial spectra, the luminance values of color

samples are set to 100

41

− The luminance of each color

sample is modified by applying

the intensity factor, F.


Experiments & Results:Caveats

42

• Temperature is assumed constant, hence the dark noise

parameters are fixed during the experiments.

• Noise model is additive

• The Sony DXC-930 camera is nearly linear for most of its

range, provided it is used with gamma disabled.

• Raw output images are considered for our analysis.

• The effects of reset noise, photodetector response

nonuniformity (PRNU), dark signal nonuniformity(DSNU) are

considered negligible.


Experiments & Results:Analysis

43

Experiments:

Scenario I: Ideal Image Sensor

Scenario II: Effects of Dark Current

Scenario III: Real Image Sensor Model


Experiments & Results:Scenario I: Ideal Image Sensor

44

Assumptions:

• Sensor is ideal (no internal noise in the model)

• Photon shot noise may corrupt the measurements

• log _ ∈ &0, =7,=8,=9,=10,=11,=12,=13,=14)




Chromaticity of Measured Samples at

Different Light Levels

Magnified Result of the Data Point Indexed 3

at Different Intensity Factors

Experiments & Results:Scenario II: Effects of Dark Current

46

Assumptions:

• Only photon shot noise and dark noise may corrupt the measurements

• Only boundary color patches are used (index: 1-13)

• _ ∈ &1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001)


Experiments & Results:Scenario III: Real Image Sensor Model

48

Assumptions:

• A model of real image sensor is considered

• _ ∈ &1, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001)


Experiments & Results:Comparing the Three Scenarios


Scenario I Scenario II Scenario III

What Next…

52

Conclusion





Introduction:



Conclusion

53

− Photon noise

− read noise

− quantization error

The physical limitation imposed by the photon noise

Dark current dominates the other sensor noise types

in the image sensor


Uncertain measurements distributed

around the noise free measurements

Dark

current

noise

dynamic

effects

on color

measur

ements

Shifting

chromaticities

towards the

camera black

point

1

2

3

4

Prevents stable measuring of color (even for an ideal image sensor)

Image Sensor Modeling: Color Measurement at Low Light Levels 54

Thank You for Your Attention!

Questions…


55


Noise Model

Photon Shot Noise

Dark Current Noise

Read Noise

Quantization Noise



56

Output of the

Image Sensor

• \�ST: conversion gain (volts/|u)

• ��: saturation function of the sensor

• �S� � : the quantum efficiency function of the sensor

• 'YZ[: the quantal radiance at the intensity factor F (photons/sec/x�/sr/nm)

�� , � � \�ST m �� m �'YZ[ �, �, � �S� � �� m �� , ��



57

Output of the

Image Sensor

• �� , � �\�ST m �� m �'YZ[ �, �, � �S� � �� m �� , ��

Measured

Voltage

• �r� �,� � �� ,� � p.�� , ��


mehdi rezagholizadeh: image sensor modeling: color measurement at low light levels

Engineering

illuminant color

definition of color

color measurement

good color appearance

ideal color appearance

sensor spectral sensitivity

illuminant spectrum

illuminant effect