National Center for Supercomputing ApplicationsUniversity of Illinois at Urbana-Champaign

Using Image Data in Your Research

Kenton McHenry, Ph.D.Research Scientist

Image and Spatial Data Analysis Group

Image and Spatial Data Analysis Group

• Research & Development• Cyberinfrastructure: Software development for the sciences

(and industry)• Computer Vision: Information from images• High Performance Computing: Software that scales with

regards to computation and data

Image and Spatial Data Analysis Group• Content Based Retrieval

• Search in digitized collections• Document segmentation• Authorship• 3D models

• Automatic Image Annotation • Assign keywords as metadata

• Tracking• 3D Reconstruction• Image Stitching

Image and Spatial Data Analysis Group

• Digital Preservation• Access to data content independent of format• Access to software functionality independent of distribution• Information loss evaluation• Document similarity

• Environmental Modeling• Workflows• Heterogeneous data sources

• Data Exploration• Data mining• eScience

Goals for Today

• A high level understanding of what Computer Vision is and how YOU might use it.• A sense of what is currently possible• A sense of how these things break• A sense of what might be possible• A sense of what is pure science fiction!• The looming opportunity in “Big Data”

• A little bit of hands on experience

Computer Vision

• Books: • D. Forsyth, J. Ponce, “Computer Vision: A Modern Approach”,

Pearson, 2011.• R. Szeliski, “Computer Vision: Algorithms and Applications”,

http://szeliski.org/Book, 2010.

• CS 543: Computer Vision (UIUC)• Derek Hoiem, Ph.D.• http://www.cs.illinois.edu/class/sp12/cs543

Computer Vision

[Hoiem, 2012]

Computer Vision

• Make a computer understand images and video

• What kind of scene?• Are there cars?• Where are the cars?• Is it day or night?• What is the ground made of?• How far is the building?

[Hoiem, 2012]

Raster Images0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.990.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.910.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.920.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.950.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.850.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.330.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.740.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.930.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.990.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.970.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

[Hoiem, 2012]

Image Creation

Light emitted

Sensor

Lens

Fraction of light reflects into camera

[Hoiem, 2012]

Image Creation • Light(s)• Position• Strength• Geometry• Color

• Surface(s)• Orientation• Color• Material• Nearby surfaces

• Sensor• Lens• Aperture• Exposure• Resolution

Light emitted

Sensor

Light reflected to camera

[Hoiem, 2012]

Surfaces: Reflected Light

incoming lightspecular reflection

ΘΘ

incoming lightdiffuse reflection

absorptionincoming light

[Hoiem, 2012]

Surface: Reflected Light

Surfaces: Orientation

1

2Ix = rxLNx

[Hoiem, 2012]

Surfaces

light sourcetransparency

light source

refraction

[Hoiem, 2012]

Surfaces

λ1

light source

λ2

fluorescence

Surfaces

t=1

light source

t>1

phosphorescence

[Hoiem, 2012]

Surfaces

λ

light source

subsurface scattering

[Hoiem, 2012]

Light

Human Luminance Sensitivity Function

[Hoiem, 2012]

Light.

# P

hoto

ns

D. Normal Daylight

Wavelength (nm.)

B. Gallium Phosphide Crystal

400 500 600 700

# P

hoto

ns

Wavelength (nm.)

A. Ruby Laser

400 500 600 700

400 500 600 700

# P

hoto

ns

C. Tungsten Lightbulb

400 500 600 700

# P

hoto

ns

[Hoiem, 2012]

Light

Light

• [GIMP Demo]

Sensors

• Long (red), Medium (green), and Short (blue) cones, plus intensity rods

[Hoiem, 2012]

Sensors

[Hoiem, 2012]

SensorsR

G

B

[Hoiem, 2012]

Sensors: Perspective

• Projecting a 3D world onto a 2D plane• Parallel lines disappear at vanishing points• Sizes appear smaller further away

Surface Interactions!

[Hoiem, 2012]

Surface Interactions

[Hoiem, 2012]

Surface Interactions

[Hoiem, 2012]

Surfaces: Interactions

Surface Interactions

[Hoiem, 2012]

Raster Images0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.990.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.910.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.920.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.950.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.850.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.330.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.740.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.930.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.990.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.970.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

[Hoiem, 2012]

image(234, 452) = 0.58

Individual Pixels

[Hoiem, 2012]

Neighborhoods of Pixels

• For nearby surface points most factors do not change much

• Local differences in brightness

[Hoiem, 2012]

Neighborhoods of Pixels

[Hoiem, 2012]

Neighborhoods of Pixels

[Hoiem, 2012]

Neighborhoods of Pixels

[Hoiem, 2012]

Changes in Intensity

• Changes in albedo• Changes in surface normal• Changes in distance

[Hoiem, 2012]

Computer Vision

• Make a computer understand images and video• Lots of variables are involved in the creation of an image/frame• Variables are not independent and interact• The problem is underconstraned

• i.e. multiple scenes can result in the same image

Optical Illusions

Optical Illusions

Optical Illusions

Vision is Really Hard!

• Vision is an amazing feat of natural intelligence• More human brain devoted to vision than anything else

[Hoiem, 2012]

State of the Art

• From 1960’s to present…

Barcodes

• Optical machine readable representation of data• 1950’s

http://en.wikipedia.org/wiki/Barcode

Optical Character Recognition (OCR)

Digit recognition, AT&T labshttp://www.research.att.com/~yann/

• Technology to convert scanned documents to ASCII text• If you have a scanner, it probably came with OCR software

License plate readershttp://en.wikipedia.org/wiki/Automatic_number_plate_recognition

[Hoiem, 2012]

Biometrics

Fingerprint scanners on many new laptops, other devices

Face recognition systems now beginning to appear more widelyhttp://www.sensiblevision.com/

[Hoiem, 2012]

Face detection

• Many new digital cameras now detect faces• Canon, Sony, Fuji, …

[Hoiem, 2012]

Medical imaging

3D imaging, MRI, CT

[Hoiem, 2012], http://en.wikipedia.org/wiki/3D_ultrasound

The Matrix movies, ESC Entertainment, XYZRGB, NRC

3D Reconstruction

Pirates of the Carribean, Industrial Light and Magic

Motion capture

[Hoiem, 2012]

Image Stitching

NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007.

[Hoiem, 2012]

Industry

Vision-guided robots position nut runners on wheels

[Hoiem, 2012]

Sports

http://www.sportvision.com/video.html

[Hoiem, 2012]

Object Recognition

Point & Find, Nokia, Google Goggles

[Hoiem, 2012]

LaneHawk by EvolutionRobotics

Human Computer Interaction

• Object Recognition: http://www.youtube.com/watch?feature=iv&v=fQ59dXOo63o• 3D Reconstruction: http://www.youtube.com/watch?v=7QrnwoO1-8A• Robot: http://www.youtube.com/watch?v=w8BmgtMKFbY

[Hoiem, 2012]

Driving

• Oct 9, 2010. "Google Cars Drive Themselves, in Traffic". The New York Times. John Markoff• June 24, 2011. "Nevada state law paves the way for driverless cars". Financial Post. Christine Dobby• Aug 9, 2011, "Human error blamed after Google's driverless car sparks five-vehicle crash". The

Star (Toronto)

[Hoiem, 2012]

State of the Art

• Remember vision is hard!• Most vision applications are “quirky”.

Image and Spatial Data Analysis Grouphttp://isda.ncsa.illinois.edu

Questions?