1 asu mat 591: opportunities in industry! asu mat 591 image processing science and robotic vision...

1

ASU MAT 591: Opportunities in Industry!

ASU MAT 591ASU MAT 591

Image Processing ScienceImage Processing Scienceand Robotic Visionand Robotic Vision

Rod PickensRod PickensPrincipal Research EngineerPrincipal Research Engineer

Lockheed Martin, IncorporatedLockheed Martin, Incorporated

2


Signals and Processing

Signals– Analog and discrete signals– Dimensionality of signals

1-D signals Sounds (temporal), echocardiogram, seismic signal

2-D signals (this presentation) Images (spatial)

3-D signals Video sequences of images (spatial and temporal)

Signal processing– Synthesize and analyze signals– Filter signals using low-pass, band-pass, and high-pass filter– Modify signals such as warp, delay, stretch, rotate, shrink, …– Restore and enhance signals– Recognize patterns and detect signals

3


Signal Processing: Now

Animal

Touch

Vision

Hearing

Smell

Taste

Robotic

Touch

Vision

Hearing

Smell

Taste

4


The Processing Analogy

5


Analysis and Synthesis of Light

)(tf )(tf

White Light Out

)(tf )(tf

)(wF

dtetfwF jwt)()(

dtetfwF jwt)()(

Fourier Analysis

dwewFtf jwt)()(

dwewFtf jwt)()(

Fourier SynthesisWhite Light In

Inverse Functions

6


Fourier Transforms are Inverse Functions

)((

))),(((

Functions Inverse

*),()(

Synthesize

*),()),((

Analyze

1

1

)(1

)(

),ωF(ωFTFT),ωF(ω

yxfFTFTf(x,y)

dxdyeF),ωF(ωFTf(x,y)

dxdyeyxfyxfFT),ωF(ω

yxyx

yxjyxyx

yxjyx

yx

yx

11 xx

7


Inverse Functions

DerivativeInv Fourier TransInv Radon TransWarp Correction

IntegralFourier TransformRadon Transform

Warp Data

()1f

()f

)(1 xffx )(xfy

)(1 yffy )(1 yfx 11 xx

8


Filtering

)(tf )(tf

)(tf f )(tf f

)(wF

White Light In

Filtering removes all but red colorsFiltering removes all but red colorsRed Light Out

9


Television

)(tf )(tf

)(tf f )(tf f

)(wF

Channel 6Filtering removes all but Channel 6Filtering removes all but Channel 6

Television Stations 3, 5, 6, 13, 15, …

Television

10


Television

)(tf )(tf

)(tf f )(tf f

)(wF

Television Stations 3, 5, 6, 13, 15, …

Channel 15Filtering removes all but Channel 15Filtering removes all but Channel 15

Television

11


Radio

)(tf )(tf

)(tf f )(tf f

)(wF

Radio Stations

Station 100.7Filtering removes all but Station 100.7Filtering removes all but Station 100.7

Radio Stations 91.5, 96.9, 100.7

Radio

12


Radio

)(tf )(tf

)(tf f )(tf f

)(wF

Radio Stations

Station 96.9Filtering removes all but Station 96.9Filtering removes all but Station 96.9

Radio Stations 91.5, 96.9, 100.7

Radio

13


Vision

)(tf )(tf

)(tf f )(tf f

)(wF

BookFiltering removes all but a bookFiltering removes all but a book

Scene of a Room: walls, books, desks, chairs,

windows,…

Robot vision

14


Vision

)(tf )(tf

)(tf f )(tf f

)(wF

Scene of a Room

TableFiltering removes all but a tableFiltering removes all but a table

Scene of a Room: walls, books, desks, chairs,

windows,…

Robot vision

15


Graphics to build a scene

dwewFwDtf jwtd )()()(

dwewFwDtf jwtd )()()(

Synthesis

)(wF

)(tfd )(tfd

Scene of a RoomDescriptor of scene is D(w)

All Room Contents

16


Data compression

)(tf )(tf

)(wF

Signal

Filter that eliminates less important data.

)(~tf )(

~tf

Approximation of Signal

17


Data compression goal

)(tf )(tf

)(~tf )(

~tf

)(wF

Signal

Filter that eliminates less important data.

Approximation of Signal

)()(~

tftf )()(~

tftf

18


An Example of a Processing Architecture

19


FormatCorrect Errors

Preprocess Restore

Analyze Recognize

The Example Architecture

Format

Descriptions

DataCorrect Errors

Communications

Preprocess

NormalizeRemove NoiseRemove Distortions

Restore

Remove Sensor Effects

Analyze

Decompose Signals

Recognize

Label Signals

Will Discuss in more detail!

20



Preprocess Restore

Analyze Recognize

Preprocess

Preprocess

Descriptions

Data


21


Noisy Input Image

Fourier Based Noise Filtering

From Jason Plumb at http://noisybox.net/weblog/

Clearer Output Image

Mostly Noise so is Zeroed

Mostly Signal

Fourier Synthesis

Fourier Analysis

Fourier Transform and Filter the Noise

22


Filtering and Enhancing Data

From Mathworks homepage at http://www.mathworks.com/

Math to follow

23


Filtering: Analysis

)(tf )(tf

)(wF

Image

dxdyeyxf),ωF(ω yxjyx

yx )(*),(

Analyze

Analysis

24


Filtering: Removing Noise

)(tf )(tf

)(wF

Filtering: removes noiseFiltering: removes noise

Image

otherwise 0

if

Filter22

yx

yxyxf

),ωF(ω),ω(ωF

25


Filtering: Synthesis

Enhanced

)(tf )(tf

)(tf f )(tf f

)(wF

Image

dxdyeF(x,y)f yxjyxff

yx )(*),(

Synthesize

Synthesis

26


Filtering

Enhanced

)(tf )(tf

)(tf )(tf

)(wF

Filtering: removes noiseFiltering: removes noise

Image

dxdyeFf(x,y)

),ωF(ω),ω(ωF

dxdyeyxf),ωF(ω

yxjyx

yxyx

yxf

yxjyx

yx

yx

)(

22

)(

*),(

Synthesize

otherwise 0

if

Filter

*),(

Analyze

Analysis

Synthesis

27


Enhancing the Data: Linear map

I=Intensity

I1

p(I1)

Input ImageIntensity Histogram

I2

p(I2)

Output ImageIntensity Histogram

(more contrast)

I1

I2

I2 = m* I1

Enhance (stretch) Using Linear Mapping

28


Warping data


Suppose we have unwanted camera motion.

29


Warping data


We can correct motion errors if we know motion model.

30


Warping data


31


Warping Correction is an Inverse Function

)(1 xffx )(1 yffy

WarpingCorrection

Warping

()1f

()f

)(xfy

)(1 yfx

32


Linear Algebra to Flip

x1

y1

x2

y2

33



x1

y1

x1

x2

x2=- x1

y1

y2

y2=y1

1

1

2

2

10

01

y

x

y

x

112

112

)(

)(

yygy

xxfx

x2

y2

34



x1

y1

x1

x2

x2=- x1

y1

y2

y2=y1

1

1

2

2

10

01

y

x

y

x

112

112

)(

)(

yygy

xxfx

I(x1,y1)

x2

y2

x2

y2

35



x1

y1

x1

x2

x2=- x1

y1

y2

y2=y1

1

1

2

2

10

01

y

x

y

x

112

112

)(

)(

yygy

xxfx

I(x1,y1)

I(x2,y2)

x2

y2

x2

y2

x2

y2

37



x1

y1

x1

x2

x2=- x1

x2

y2

y1

y2

y2=y1

1

1

2

2

10

01

y

x

y

x

112

112

)(

)(

yygy

xxfx

I(x1,y1)

I(x2,y2)=I(f(x1),g(y1))

x2

y2

x2

y2

x2

y2

38



x1

y1

x2

x1

x1=- x2

x2

y2

y2

y1

y1=y2

2

2

1

1

10

01

y

x

y

x

1221

1221

)(

)(

yyyg

xxxf

I(x2,y2)

39



x1

y1

x2

x1

x1=- x2

x2

y2

y2

y1

y1=y2

2

2

1

1

10

01

y

x

y

x

1221

1221

)(

)(

yyyg

xxxf

I(f-1(x2), g-1(y2))

I(x2,y2)

40



x1

y1

x1

x2

x2=- x1

x2

y2

y1

y2

y2=y1

1

1

2

2

10

01

y

x

y

x

12

12

yy

xx

I (x1,y1)=I(f-1(x2), g-1(y2))

I (x2,y2)

41


Linear Algebra to Flip and Shrink

x1

y1

x2

y2

42


Linear Algebra to Flip and Shrink

x1

y1

x1

x2

x2

y2

y1

y2

y2 = -0.5 * y1

x2 = 0.5 * x1

12

12

*5.0

*5.0

yy

xx

1

1

2

2

5.00

05.0

y

x

y

x

43


Correcting warped data (camera motion)


))(),(( 21

121

1 ygyxfxI

))(),(( 1212 ygyxfxI ),( 11 yxI

),( 22 yxI

If we can determine f(), g(), f-1(), and g-1(), then we can correct camera motion!

11 xx

44



Preprocess Restore

Analyze Recognize

Restoration

Restore

Descriptions

Data


45


Restoring data for smear, optics,…


UsesLinear Systems

Theory

Next

Smear and optics can be viewed as filters that can degrade an image!

46




UsesLinear Systems

Theory

Next

Restoration

47


Restoration: Analysis

),( yxf ),( yxf

),( yx wwF

Image

dxdyeyxf),ωF(ω yxjyx

yx )(*),(

Analyze

Analysis

48


Filtering: Removing Smear

Smr-1(wx,wy) is a filter that removes smear or restores the original object.

Smr-1(wx,wy) is a filter that removes smear or restores the original object.

Image

)(

Filter Restoring1 ),ωF(ω,ωωSmr),ω(ωF yxyxyxf

),( yxf ),( yxf

),( yx wwF

50


Filtering

),( yxf ),( yxf

),( yx wwF

Image Restored to best look like original Object

Image Restored to best look like original Object

Image

dxdyeF(x,y)f

),ω)F(ω,ω(ωSmr),ω(ωF

dxdyeyxf),ωF(ω

yxjyxff

yxyxyxf

yxjyx

yx

yx

)(

1

)(

*),(

Synthesize

Restore

*),(

Analyze

Object

),( yxf f ),( yxf f

Smear inverted as a filterSmear inverted as a filter

51




UsesLinear Systems

Theory

Image(wx,wy) Next

52




UsesLinear Systems

Theory

Image(wx,wy)

Smr(wx,wy)*Image(wx,wy)

Next

53




UsesLinear Systems

Theory

Image(wx,wy)

Smr(wx,wy)*Image(wx,wy)

Image(wx,wy) *Smr-1(wx,wy)* Smr(wx,wy)

Image(wx,wy)= Image(wx,wy) *1(wx,wy )11 xx

54



Preprocess Restore

Analyze Recognize

Synthesis and Analysis

Descriptions

Data

Decompose / Compose Signals - Transforms: Fourier, SVD, Wavelets - Statistical Analysis: parametric and non-parametric

Synthesize

Analyze

55


Fourier Transform

)(tf )(tf

White Light Out

)(tf )(tf

)(wF

dtetfwF jwt)()(

dtetfwF jwt)()(

Fourier Analysis

dwewFtf jwt)()(

dwewFtf jwt)()(

Fourier SynthesisWhite Light In

56


Fourier Transform

Magnitude Phase

From Wolfram homepage at http://documents.wolfram.com

57


Radon Transform


58


Wavelet Transform

From Wolfram homepage at http://documents.wolfram.com

59


Common Transforms

Fourier Discrete fourier Cosine Sine Hough Hadamard Slant Karhunen-Loeve Fast KL SVD Sinusoidal

60


Statistics


61



Preprocess Restore

Analyze Recognize

Recognition

Recognize Descriptions

Data

Label Signals - Signal Detection - Pattern Recognition - Artificial Intelligence

62


Fea

tur e

1Feature 2 *

Fea

ture

1

Feature 2

Class 1(daisy)

Class 2(rose)

Class 3(sun flower)

* Features are mathematical measurements

Pattern Recognition

Classification

BayesianNeural netsNearest neighborsLinear

Transforms: Fourier, Wavelet, …Statistics: mean, st. dev, …Shape: Fourier, Hough, MomentsTexture: Cooccurrence, Eigen Filters, …

Analysis Tools Features

Feature 1: Hough measureFeature 2: 3rd Eigen Filter

Analysis

63


Mathematical Decisions

z

o

Class 1 is z

Class 2 is o

o

o

o

o

o

o

oo

oo

o

o

z

z

z

z

z

z

z

zz

z

zz

z

f1

f2

z

How do we separate the

classes?

o

oo

o

o

64


Mathematical Decisions

z

o

Class 1 is z

Class 2 is o

o

o

o

o

o

o

oo

oo

o

o

z

z

z

z

z

z

z

zz

z

zz

z

f1

f2

z

Linear decision

o

oo

o

o

65


Mathematical Decision

z

o

Class 1 is z

Class 2 is o

o

o

o

o

o

o

oo

oo

o

o

z

z

z

z

z

z

z

zz

z

zz

z

f1

f2

z

Linear decision

o

oo

o

o

66



z

o

Class 1 is z

Class 2 is o

o

o

o

o

o

o

oo

oo

o

o

z

z

z

z

z

z

z

zz

z

zz

z

f1

f2

z

Quadratic decision

1or 1

BoundaryDecision Reasonable2

122

12 ffff

o

oo

o

o

67



zClass 1 is z

z

z

z

z

z

z

z

zz

z

zz

z

f1

f2

zone class isobject then 12

12 ff

68



o

twoclass isobject then 1212 ff

Class 2 is o

o

o

o

o

o

oo

oo

o

of1

f2

o

oo

o

o

69



z

o

Class 1 is z

Class 2 is o

o

o

o

o

o

o

oo

oo

o

o

z

z

z

z

z

z

z

zz

z

zz

z

f1

f2

z

-1

3

one class isobject so 113)1(3 2 o

oo

o

o

twoclass isobject then 1

one class isobject then 12

12

212

ff

ff

70


Isolate Object: Segmentation


Analysis Synthesis

71


Analyze Object: Features

- Length- Width- Contour- Orientation

- Edges- Skeleton- Texture Details- Intensity


72


Matched Filtering (registration)


Input Image or Iin(x,y)

73




Exemplar (reference) or Iref(x,y)


74






2112222 )),(),((min(, yxIyxIyx refin

x2

error

x

75






2112222 )),(),((min(, yxIyxIyx refin

x2

error

x

Actually search form min of x,y simultaneously!

76



Preprocess Restore

Analyze Recognize

Image Processing: Summary

Format

Descriptions

DataCorrect Errors

Communications

Preprocess


Restore


Analyze

Decompose Signals

Recognize

Label Signals

77


References

Fundamentals of Image Processing by Jain

Digital Image Analysis by Gonzalez and Wintz

Pattern Recognition by Fukunaga

Pattern Recognition Principles Tou and Gonzalez

Detection, Estimation, and Modulation Theory by Van Trees

Pattern Classification by Duda and Hart

Robot by Hans Moravec (graphics from www.amazon.com)

78


Touch

Vision

Hearing

Smell

Taste

Signal Processing: 50 years from now

Touch

Vision

Hearing

Smell

Taste

Robotic Evolved

Hmmm.

Vision

79


Touch

Vision

Hearing

Smell

Taste


Touch

Vision

Hearing

Smell

Taste

Robotic Evolved

Wow!

Vision

80


Touch

Vision

Hearing

Smell

Taste


Touch

Vision

Hearing

Smell

Taste

Robotic Evolved

I see, therefore, am I? Hmmm.Vision

1 asu mat 591: opportunities in industry! asu mat 591 image processing science and robotic vision...

Documents