7/31/2019 Research 2003

1/24

Logarithmic CMOSLogarithmic CMOS

image sensorsimage sensorsDr. Dileepan JosephDr. Dileepan Joseph

Dept. of Engineering ScienceDept. of Engineering ScienceUniversity of Oxford, UKUniversity of Oxford, UK

7/31/2019 Research 2003

2/24

OutlineOutline

MotivationMotivation

BackgroundBackground

MethodMethod

ConclusionsConclusions

Future workFuture work

7/31/2019 Research 2003

3/24

Motivation: socialMotivation: social

Society has invested over many millennia inSociety has invested over many millennia indeveloping technology to record observeddeveloping technology to record observedscenes on an independent mediumscenes on an independent medium

Artistic license aside, the aim has been to renderArtistic license aside, the aim has been to render

images with a maximum of perceptual accuracyimages with a maximum of perceptual accuracyusing a minimum of effortusing a minimum of effort

The digital camera is a culmination of the aboveThe digital camera is a culmination of the abovebut its development is far from completebut its development is far from complete

Although digital cameras have in many waysAlthough digital cameras have in many wayssurpassed film cameras, human vision routinelysurpassed film cameras, human vision routinelyoutperforms the best camerasoutperforms the best cameras

7/31/2019 Research 2003

4/24

Motivation: economicMotivation: economic

A digital camera consists of many componentsA digital camera consists of many components

(optics, housing, battery, memory etc.), of which(optics, housing, battery, memory etc.), of whichthe image sensor is considered principalthe image sensor is considered principal

With market revenues of $1.7 billion in 2003,With market revenues of $1.7 billion in 2003,there is widespread research and developmentthere is widespread research and development

in a variety of image sensor designsin a variety of image sensor designs

Modern designs may be either charge coupledModern designs may be either charge coupled

device (CCD) sensors or complementary metal-device (CCD) sensors or complementary metal-

oxide-semiconductor (CMOS) sensorsoxide-semiconductor (CMOS) sensors

7/31/2019 Research 2003

5/24

Motivation: technologicalMotivation: technological

Criterion Human eye Film photo CCD sensor CMOS sensor

Pixel pitch 23 m 1020 m 510 m 510 m

Image pitch 3 cm Film size 1 mm11 cm 1 mm2 cm

Dynamic range 25 decades 14 decades 4 decades 35 decades

Max. frame rate 15 Hz 1 shot only 10 kHz >> 10 kHz

Pre-processing Extensive None None Possible

Unit price Invaluable 0.1 100 10

7/31/2019 Research 2003

6/24

Background: CCD image sensorBackground: CCD image sensor

Marches photo generatedMarches photo generatedcharge systematicallycharge systematicallyfrom an array of pixels tofrom an array of pixels to

an output amplifieran output amplifier

Established technologyEstablished technology

High resolution, highHigh resolution, highsensitivity, low noisesensitivity, low noise

Fabrication process isFabrication process isoptimised for imagingoptimised for imaging

Market share of 93% inMarket share of 93% in1999 (49% in 2004?)1999 (49% in 2004?)

7/31/2019 Research 2003

7/24

Background: CMOS image sensorBackground: CMOS image sensor

Works like memory arrayWorks like memory arraywith photosensitive pixelswith photosensitive pixelsinstead of memory cellsinstead of memory cells

Signal processing may beSignal processing may be

included on the same dieincluded on the same die

High yield and good videoHigh yield and good videoperformanceperformance

May be fabricated by theMay be fabricated by themakers of microchipsmakers of microchips

Market share of 7% inMarket share of 7% in1999 (51% in 2004?)1999 (51% in 2004?)

7/31/2019 Research 2003

8/24

Background: linear pixelsBackground: linear pixels

Linear pixels (CCD orLinear pixels (CCD orCMOS) integrateCMOS) integratephotons over discretephotons over discreteperiods of timeperiods of time

They produce aThey produce avoltage directlyvoltage directlyproportional to theproportional to thelight intensitylight intensity

The response mayThe response maysaturate white orsaturate white orblack easilyblack easily

7/31/2019 Research 2003

9/24

Background: logarithmic pixelsBackground: logarithmic pixels

Logarithmic pixelsLogarithmic pixels(CMOS only) can(CMOS only) canmeasure photon fluxmeasure photon fluxcontinuouslycontinuously

They produce aThey produce avoltage proportional tovoltage proportional tothe logarithm of lightthe logarithm of lightintensityintensity

The response isThe response issimilar to that ofsimilar to that ofhuman visionhuman vision

7/31/2019 Research 2003

10/24

Background: image qualityBackground: image quality

Images are noisy withImages are noisy with

logarithmic sensorslogarithmic sensors

Colour is worse thanColour is worse than

with linear sensorswith linear sensorsQuality improves withQuality improves with

digital processingdigital processing

No comprehensiveNo comprehensivetreatment of eithertreatment of eitherproblem or solutionproblem or solution

7/31/2019 Research 2003

11/24

Method: theoryMethod: theory

Model logarithmic CMOS image sensorsModel logarithmic CMOS image sensors

using optical & integrated circuit theoryusing optical & integrated circuit theory

Use the model to hypothesize the causeUse the model to hypothesize the cause

and solution of image quality problemsand solution of image quality problems

Calibrate the model and test hypothesesCalibrate the model and test hypotheses

using constrained regression theoryusing constrained regression theory

Optimise digital image processing usingOptimise digital image processing using

multilinear (or array) algebramultilinear (or array) algebra

7/31/2019 Research 2003

12/24

Method: simulationMethod: simulation

Simulation of integrated circuits is moreSimulation of integrated circuits is moreaccurate than a theoretical analysisaccurate than a theoretical analysis

Cost of simulation in time and money isCost of simulation in time and money is

small compared to that of experimentsmall compared to that of experimentIntegrated circuits may be studied underIntegrated circuits may be studied undercontrolled and well-defined conditionscontrolled and well-defined conditions

Internal states and variables may beInternal states and variables may beobserved without specialised equipment,observed without specialised equipment,circuit disruption and/or foresightcircuit disruption and/or foresight

7/31/2019 Research 2003

13/24

Method: experimentMethod: experiment

Experiments were performed using a FugaExperiments were performed using a Fuga

15RGB camera from C-Cam Technologies15RGB camera from C-Cam Technologies

The camera was operated from a portableThe camera was operated from a portable

PC via a custom Windows applicationPC via a custom Windows application

The image sensor had 512The image sensor had 512 512 pixels 512 pixels

and a full frame rate of about 8 Hzand a full frame rate of about 8 Hz

Until recently, it was the most successfulUntil recently, it was the most successful

commercial logarithmic image sensorcommercial logarithmic image sensor

7/31/2019 Research 2003

14/24

Conclusions: fixed pattern noiseConclusions: fixed pattern noise

yy== aa ++ bb ln (ln (cc++ xx) +) + for illuminancefor illuminance xxandandresponseresponse yyof a pixelof a pixel

Variation of offsetVariation of offset aa,,gaingain bb, bias, bias ccor aor a

combination thereofcombination thereofcauses FPNcauses FPN

Calibration possibleCalibration possible

within limits of thewithin limits of the

stochastic errorstochastic error

7/31/2019 Research 2003

15/24

Conclusions: fixed pattern noiseConclusions: fixed pattern noise

Left to right: FPNLeft to right: FPNcorrection for single,correction for single,double and tripledouble and triplevariation modelsvariation models

Top to bottom: twoTop to bottom: twodecade attenuation ofdecade attenuation ofilluminance in halfilluminance in halfdecadedecade stepssteps

Inter-scene plus intra-Inter-scene plus intra-scene dynamic rangescene dynamic rangeof 3.5 decadesof 3.5 decades

7/31/2019 Research 2003

16/24

Conclusions: transient responseConclusions: transient response

The transient response ofThe transient response ofa pixel is fast enough fora pixel is fast enough formost applicationsmost applications

Greater demands areGreater demands are

placed on the row andplaced on the row andcolumn readoutscolumn readouts

Premature digitizationPremature digitization

results in a predictableresults in a predictable

non-uniformity or FPNnon-uniformity or FPN

Affects only a few rowsAffects only a few rows

due to slow scanningdue to slow scanning

7/31/2019 Research 2003

17/24

Conclusions: transient responseConclusions: transient response

Premature digitization isPremature digitization ismore serious for columnmore serious for columnreadout due to speedreadout due to speed

For example, columnsFor example, columns

need scanning at 100need scanning at 100MHz for HDTV videoMHz for HDTV video

Column-to-column gainColumn-to-column gain

variation is caused byvariation is caused by

transient responsetransient response

Resolve with carefulResolve with careful

timing and designtiming and design

7/31/2019 Research 2003

18/24

Conclusions: temperatureConclusions: temperature

dependencedependence

Unlike with humans,Unlike with humans,

digital cameras do notdigital cameras do notregulate temperatureregulate temperature

Hence, responses toHence, responses toilluminance dependilluminance depend

on temperatureon temperature

When temperatureWhen temperature

dependence variesdependence varies

from pixel to pixel,from pixel to pixel,

FPN occursFPN occurs

7/31/2019 Research 2003

19/24

Conclusions: temperatureConclusions: temperature

dependencedependenceThe dark response ofThe dark response ofa pixel depends onlya pixel depends onlyon temperatureon temperature

It may be used toIt may be used tocorrect FPN due tocorrect FPN due totemperature in thetemperature in thelight responselight response

Experiments supportExperiments supportthis conclusion butthis conclusion butsimulation results aresimulation results areshown for clarityshown for clarity

7/31/2019 Research 2003

20/24

Conclusions: colour renditionConclusions: colour rendition

Combine the theories ofCombine the theories ofcolour linear sensors andcolour linear sensors andb/w logarithmic sensorsb/w logarithmic sensors

Calibrate FPN, usingCalibrate FPN, usingimages of uniform stimuli,images of uniform stimuli,by aby a relativerelative analysisanalysisCalibrate colour, usingCalibrate colour, usingimages of a colour chart,images of a colour chart,by anby an absoluteabsolute analysisanalysis

Fuga 15RGB competesFuga 15RGB competeswith conventional digitalwith conventional digitalcameras (which have acameras (which have aperceptual error of 15)perceptual error of 15)

7/31/2019 Research 2003

21/24

Conclusions: colour renditionConclusions: colour rendition

Image of a colour chart, inImage of a colour chart, in11 lux of illuminance, was11 lux of illuminance, wasrendered using calibratedrendered using calibrated

modelsmodels

Single, double and tripleSingle, double and triplevariation results and idealvariation results and ideal

colours are showncolours are shown

As with vision, renditionAs with vision, rendition

improves in brighterimproves in brighterlighting and worsens inlighting and worsens in

dimmer lightingdimmer lighting

7/31/2019 Research 2003

22/24

Future workFuture work

Digital cameras aim to render images withDigital cameras aim to render images witha maximum of perceptual accuracy usinga maximum of perceptual accuracy usinga minimum of efforta minimum of effort

By modelling and calibrating logarithmicBy modelling and calibrating logarithmicCMOS image sensors, problems withCMOS image sensors, problems withimage quality may be solvedimage quality may be solved

Past work has focused on maximisingPast work has focused on maximisingperceptual accuracy but future work willperceptual accuracy but future work willfocus on minimising effortfocus on minimising effort

7/31/2019 Research 2003

23/24

Future workFuture work

Shrinking featureShrinking feature

sizes may be used tosizes may be used toimprove imagingimprove imaging

There are challengesThere are challengeswith deep submicronwith deep submicron

CMOS processes thatCMOS processes thatneed overcomingneed overcoming

What about industrialWhat about industrial

and biomedical usesand biomedical uses

of the technology?of the technology?

7/31/2019 Research 2003

24/24

AcknowledgementsAcknowledgements

This work was funded thanks to the engineeringThis work was funded thanks to the engineering

research councils of both Canada and the UKresearch councils of both Canada and the UK

Thanks also to colleagues at the MicroelectronicThanks also to colleagues at the Microelectronic

Circuits and Analogue Devices research groupCircuits and Analogue Devices research group