tomihisa (tom) welsh michael ashikhmin klaus mueller tomihisa (tom) welsh michael ashikhmin klaus...

Tomihisa (Tom) WelshTomihisa (Tom) Welsh

Michael AshikhminMichael Ashikhmin

Klaus MuellerKlaus Mueller

Tomihisa (Tom) WelshTomihisa (Tom) Welsh

Michael AshikhminMichael Ashikhmin

Klaus MuellerKlaus Mueller

Center for Visual ComputingCenter for Visual Computing

Stony Brook UniversityStony Brook University

Transferring Color to Greyscale ImagesTransferring Color to Greyscale Images

The ProblemThe Problem

• How to colorize greyscale images?How to colorize greyscale images?• How to colorize greyscale images?How to colorize greyscale images?



•Is his tie blue or green?Is his tie blue or green?



• Red!!Red!!


• How to colorize greyscale images?How to colorize greyscale images?

• Issues:Issues:• No “correct” solution

• Need to be creative

• How to minimize the manual labor involved?

• How to colorize greyscale images?How to colorize greyscale images?

• Issues:Issues:• No “correct” solution

• Need to be creative

• How to minimize the manual labor involved?

MotivationMotivation

• Enhance Scientific Data Enhance Scientific Data • Medical Imaging (MRI, CT, X-Ray)




• Satellite Images (Landsat)




• Enhance Scientific DataEnhance Scientific Data • Medical Imaging (MRI, CT, X-Ray)


• Scanning Electron Microscopy (SEM)

• Enhance Scientific DataEnhance Scientific Data • Medical Imaging (MRI, CT, X-Ray)







• Colorize black/white photographs and moviesColorize black/white photographs and movies










• Artistic EffectsArtistic Effects





• Artistic EffectsArtistic Effects

The TaskThe Task

• The problem is fundamentally ill-posedThe problem is fundamentally ill-posed

• It is an attempt to extrapolate from 1-D to 3-D It is an attempt to extrapolate from 1-D to 3-D • Map scalar luminance (intensity) to vector RGB

• The problem is fundamentally ill-posedThe problem is fundamentally ill-posed

• It is an attempt to extrapolate from 1-D to 3-D It is an attempt to extrapolate from 1-D to 3-D • Map scalar luminance (intensity) to vector RGB

Previous MethodsPrevious Methods

• Coloring Book MethodColoring Book Method• Photoshop: manually paint color with low opacity




• Movie Industry: track polygons [Cinesite Press Article]






• Pseudo-coloringPseudo-coloring• Global Transformation/Color Map








• Satellite ImagesSatellite Images• Registration [R2V Software], Orthoimagery [Premoze]










• Image AnalogiesImage Analogies• Grey Source : Color Source :: Grey Target : Result





• Image AnalogiesImage Analogies• Grey Source : Color Source :: Grey Target : Result

Related WorkRelated Work

• ““Color Transfer between Images” Color Transfer between Images” [Reinhard et al. 2001][Reinhard et al. 2001]





TargetSource

+


TargetSource Final

+ =



Reinhard et al.Reinhard et al.

• Decorrelated color space (Decorrelated color space (llαβ) [Ruderman et al., 1998]αβ) [Ruderman et al., 1998]

• Scale and shift color distributions globallyScale and shift color distributions globally(Using mean and standard deviation)



SourceTarget






Target (Scaled) Source






Target (Scaled & Shifted) Source

Our Approach Our Approach

Target

Our ApproachOur Approach

• Select color source imageSelect color source image• Select color source imageSelect color source image

+???

SourceTarget

???


• Select color source imageSelect color source image

• Match each target pixel with a few sourceMatch each target pixel with a few source pixelspixels


• Match each target pixel with a few sourceMatch each target pixel with a few source pixelspixels

+

SourceTarget



• Match each target pixel with a few sourceMatch each target pixel with a few source pixelspixels• Choose best match using local pixel neighborhood statistics



+

SourceTarget




• Transfer colorTransfer color




+

SourceTarget

=





• Repeat for all pixelsRepeat for all pixels




• Repeat for all pixelsRepeat for all pixels

=+

SourceTarget Final

Global Image Matching ProcedureGlobal Image Matching Procedure

1.1. Convert images to Convert images to llαβ color spaceαβ color space

2.2. Image MatchingImage Matching

3.3. Color TransferColor Transfer

1.1. Convert images to Convert images to llαβ color spaceαβ color space

2.2. Image MatchingImage Matching

3.3. Color TransferColor Transfer

1. Convert to lαβ Space1. Convert to lαβ Space

• Luminance (Luminance (ll), alpha (α) and beta (β) channels), alpha (α) and beta (β) channels

• Minimizes correlation between axes Minimizes correlation between axes (i.e. cross-channel artifacts)

• Luminance (Luminance (ll), alpha (α) and beta (β) channels), alpha (α) and beta (β) channels

• Minimizes correlation between axes Minimizes correlation between axes (i.e. cross-channel artifacts)

RGB lαβ

0 50 100 150 200 250 3000

1

2x 104 red channel

0 50 100 150 200 250 3000

1

2x 104 green channel

0 50 100 150 200 250 3000

1

2x 104 blue channel

Color

Image

-5 -4 -3 -2 -1 0 1 2 3 4 50

1

2x 104 luminance channel

-5 -4 -3 -2 -1 0 1 2 3 4 50

1

2x 104 alpha channel

-5 -4 -3 -2 -1 0 1 2 3 4 50

1

2x 104 beta channel

2. Image Matching2. Image Matching

1.1. Remap luminance histograms between src/targetRemap luminance histograms between src/target1.1. Remap luminance histograms between src/targetRemap luminance histograms between src/target

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

5000

10000

15000luminance channel-source

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

5000

10000

15000luminance channel-target

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

5000

10000

15000luminance channel-remapped

Target

Source-Before

Source-After


1.1. Remap luminance histograms between src/targetRemap luminance histograms between src/target

2.2. Precompute neighborhood statistics for imagesPrecompute neighborhood statistics for images






3.3. Reduce samples using jittered samplingReduce samples using jittered sampling- Faster computation due to smaller search space




10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100








10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

10010 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100








10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

10010 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100

10 20 30 40 50 60 70 80 90 100

10

20

30

40

50

60

70

80

90

100





4.4. Find best neighborhood match from samplesFind best neighborhood match from samples- Weighted metric of luminance, mean, standard deviation




4.4. Find best neighborhood match from samplesFind best neighborhood match from samples- Weighted metric of luminance, mean, standard deviation

3. Color Transfer3. Color Transfer

• Transfer only alpha and beta channels (color)Transfer only alpha and beta channels (color)• Transfer only alpha and beta channels (color)Transfer only alpha and beta channels (color)

Target Image Colorized Results

3. Color Transfer3. Color Transfer

• Transfer only alpha and beta channels (color)Transfer only alpha and beta channels (color)

• The original luminance value remains unchangedThe original luminance value remains unchanged

• Transfer only alpha and beta channels (color)Transfer only alpha and beta channels (color)

• The original luminance value remains unchangedThe original luminance value remains unchanged

Target Image Colorized Results Convert to Greyscale

(Photoshop)

Results: satelliteResults: satellite

+

TargetSource

Results: satelliteResults: satellite

+ =

Final

TargetSource

Results: TexturesResults: Textures

+

TargetSource


+ =

TargetSource Final

Limitations (Global Approach)Limitations (Global Approach)

+ =

TargetSource Final

Limitations (Global Approach)Limitations (Global Approach)

+ =

TargetSource Final

+ =

User-Assisted ApproachUser-Assisted Approach

1.1. User selects small number of swatchesUser selects small number of swatches

2.2. Transfer color only to swatches Transfer color only to swatches (Global Matching Procedure)

3.3. Color entire target imageColor entire target image (Only use swatch samples)

1.1. User selects small number of swatchesUser selects small number of swatches

2.2. Transfer color only to swatches Transfer color only to swatches (Global Matching Procedure)

3.3. Color entire target imageColor entire target image (Only use swatch samples)

1. Selection of Swatches1. Selection of Swatches

TargetSource

2. Transfer Color to Swatches2. Transfer Color to Swatches

• Transfer color using Global Matching ProcedureTransfer color using Global Matching Procedure(described previously)

• Transfer color using Global Matching ProcedureTransfer color using Global Matching Procedure(described previously)

3. Colorize Entire Image3. Colorize Entire Image

• Discard original source imageDiscard original source image• Discard original source imageDiscard original source image

3. Colorize Entire Image3. Colorize Entire Image

• Colorize the full imageColorize the full image

• Match using L2 Norm

• Colorize the full imageColorize the full image

• Match using L2 Norm

Swatches: NotesSwatches: Notes

• Expect good results Expect good results betweenbetween swatches swatches• Expect good results Expect good results betweenbetween swatches swatches


• Expect good results Expect good results betweenbetween swatches swatches• Expect good results Expect good results betweenbetween swatches swatches

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

5000

10000

15000luminance channel

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

5000

10000

15000luminance channel


• Expect good results Expect good results betweenbetween swatches swatches

• Expect better matching within an imageExpect better matching within an image (Allows more precise metric: L2 Norm)


• Expect better matching within an imageExpect better matching within an image (Allows more precise metric: L2 Norm)

(2L )



• Expect better matching within an imageExpect better matching within an image (L2 Norm is more sensitive to image differences)


• Expect better matching within an imageExpect better matching within an image (L2 Norm is more sensitive to image differences)

Between WithinBetween Within

Results: SwatchesResults: Swatches

TargetSource Final (Previous Method)

+ =


TargetSource Final

+ =

+ =


+


Final

+ =


+


Final

+ =


TargetSource

+


TargetSource Final

+ =

LimitationsLimitations

CommentsComments

• Works well when Works well when targettarget can be segmented well can be segmented well• Works well when Works well when targettarget can be segmented well can be segmented well

CommentsComments

• Works well when Works well when targettarget can be segmented well can be segmented well

• Large shadows present a problem Large shadows present a problem • (partial volume effect)

• To be more useful, combine with other toolsTo be more useful, combine with other tools

• Works well when Works well when targettarget can be segmented well can be segmented well

• Large shadows present a problem Large shadows present a problem • (partial volume effect)

• To be more useful, combine with other toolsTo be more useful, combine with other tools

Running TimeRunning Time

• Pentium 3 (800 Mhz): 15 sec – 1 minPentium 3 (800 Mhz): 15 sec – 1 min

• Typical Image size: 640x480Typical Image size: 640x480

• Implemented using MATLAB (Optimized)Implemented using MATLAB (Optimized)

• Factors:Factors:• Image size

• Neighborhood Size

• Pentium 3 (800 Mhz): 15 sec – 1 minPentium 3 (800 Mhz): 15 sec – 1 min

• Typical Image size: 640x480Typical Image size: 640x480

• Implemented using MATLAB (Optimized)Implemented using MATLAB (Optimized)

• Factors:Factors:• Image size

• Neighborhood Size

VideoVideo

1.1. Colorize one frame using swatches Colorize one frame using swatches

2.2. Use swatches to colorize the entire sequenceUse swatches to colorize the entire sequence

If a single frame in a sequence is colorized well, If a single frame in a sequence is colorized well, then the entire sequence can be colorized wellthen the entire sequence can be colorized well



If a single frame in a sequence is colorized well, If a single frame in a sequence is colorized well, then the entire sequence can be colorized wellthen the entire sequence can be colorized well

Video ProcedureVideo Procedure

1.1. Colorize one frame using swatches Colorize one frame using swatches 1.1. Colorize one frame using swatches Colorize one frame using swatches

Video ProcedureVideo Procedure





… …

… …=

Video: WavesVideo: Waves

Video: HorsesVideo: Horses

Video: Visible HumanVideo: Visible Human

ConclusionsConclusions

• Keep original luminance valuesKeep original luminance values

• Use local pixel neighborhood statistics to matchUse local pixel neighborhood statistics to match

• Simple algorithms provide fast (and good) resultsSimple algorithms provide fast (and good) results

• Keep original luminance valuesKeep original luminance values

• Use local pixel neighborhood statistics to matchUse local pixel neighborhood statistics to match

• Simple algorithms provide fast (and good) resultsSimple algorithms provide fast (and good) results

Future WorkFuture Work

• Robustness: more sophisticated matching Robustness: more sophisticated matching • Multi-resolution, other pattern matching metrics

• VolumesVolumes

• Color CorrectionColor Correction• Use local neighborhood statistics

• Color correct movies automatically

• Robustness: more sophisticated matching Robustness: more sophisticated matching • Multi-resolution, other pattern matching metrics

• VolumesVolumes

• Color CorrectionColor Correction• Use local neighborhood statistics

• Color correct movies automatically

Questions?Questions?

tomihisa (tom) welsh michael ashikhmin klaus mueller tomihisa (tom) welsh michael ashikhmin klaus...

Documents