Computer Vision: CSE 803

A brief intro

MSU/CSE Fall 2017

Today's Class

n  Imaging process n  Image processing operation n  Perceive 3D from images n  CV applications n  Homework #1 brief

MSU/CSE Fall 2017

MSU/CSE Fall 2017

Goal of Computer Vision n  Make useful decisions about real physical

objects and scenes based on sensed images. n  Alternative (Aloimonos and Rosenfeld): goal is

the construction of scene descriptions from images.

n  How do you find the door to leave? n  How do you determine if a person is friendly

or hostile? .. an elder? .. a possible mate?

MSU/CSE Fall 2017

Critical Issues n  Sensing: how do sensors obtain images of

the world? n  Information/features: how do we obtain

color, texture, shape, motion, etc.? n  Representations: what representations

should/does a computer [or brain] use? n  Algorithms: what algorithms process image

information and construct scene descriptions?

MSU/CSE Fall 2017

Images: 2D Projections of 3D n  3D world has color, texture, surfaces, volumes, light

sources, objects, motion, betweeness, adjacency, connections, etc.

n  2D image is a projection of a scene from a specific viewpoint; many 3D features are captured, some are not.

n  Brightness or color = g(x,y) or f(row, column) for a certain instant of time.

n  Images indicate familiar people, moving objects or animals, health of people or machines.

MSU/CSE Fall 2017

Image Receives Reflections n  Light reaches surfaces

in 3D n  Surfaces reflect n  Sensor element

receives light energy n  Intensity matters n  Angles matter n  Material maters

MSU/CSE Fall 2017

Simple Objects: Simple Image?

MSU/CSE Fall 2017

Where is the Sun?

MSU/CSE Fall 2017

CCD Camera has Discrete Elts

n  Lens collects light rays

n  CCD elts replace chemicals of film

n  Number of elts less than with film (so far)

MSU/CSE Fall 2017

Camera + Programs = Display n  Camera inputs to

frame buffer n  Program can

interpret data n  Program can add

graphics n  Program can add

imagery

MSU/CSE Fall 2017

Human eye as a spherical camera

n  100M sensing elts in retina

n  Rods sense intensity n  Cones sense color n  Fovea has tightly packed

elts, more cones n  Periphery has more rods n  Focal length is about

20mm n  Pupil/iris controls light

entry

•  Eye scans, or saccades to image details on fovea

•  100M sensing cells funnel to 1M optic nerve connections to the brain

MSU/CSE Fall 2017

Qualitative Representation of Visual Detail of the Eyes

http://www.cambridgeincolour.com/tutorials/cameras-vs-human-eye.htm

Computational Photography

Dr. Freeman's Talk

MSU/CSE Fall 2017

Some image format issues

Spatial resolution; intensity resolution; image file format

MSU/CSE Fall 2017

MSU/CSE Fall 2017

Resolution is “pixels per unit of length”

n  Resolution decreases by one half in cases at left

n  Human faces can be recognized at 64 x 64 pixels per face

MSU/CSE Fall 2017

Features detected depend on the resolution

n  Can tell hearts from diamonds

n  Can tell face value

n  Generally need 2 pixels across line or small region (such as eye)

Image Super-Resolution •  Scale factor 4 super-resolution results

3

24

Super-Resolution Results

Ying Tai, Jian Yang, Xiaoming Liu, Image Super-Resolution via Deep Recursive Residual Network, in CVPR 2017.

MSU/CSE Fall 2017

Image Processing Operations

Thresholding; Edge detection;

Motion field computation

MSU/CSE Fall 2017

Find Regions via Thresholding

Region has brighter or darker or redder color, etc.

If pixel > threshold then pixel = 1 else pixel = 0

MSU/CSE Fall 2017

Example Red Blood Cell Image n  Many blood cells are

separate objects n  Many touch – bad! n  Salt and pepper

noise from thresholding

n  How useable is this data?

MSU/CSE Fall 2017

sign = imread('Images/stopSign.jpg','jpg'); red = (sign(:, :, 1)>120) & (sign(:,:,2)<100) & (sign(:,:,3)<80); out = red*200; imwrite(out, 'Images/stopRed120.jpg', 'jpg')

MSU/CSE Fall 2017

sign = imread('Images/stopSign.jpg','jpg'); red = (sign(:, :, 1)>120) & (sign(:,:,2)<100) & (sign(:,:,3)<80); out = red*200; imwrite(out, 'Images/stopRed120.jpg', 'jpg')

MSU/CSE Fall 2017

Thresholding is Usually Not Trivial

MSU/CSE Fall 2017

Cluster Pixels by Color Similarity and Adjacency

Original RGB Image Color Clusters by K-Means

MSU/CSE Fall 2017

Detect Motion via Subtraction n  Constant

background n  Moving object n  Produces pixel

differences at boundary

n  Reveals moving object and its shape Differences computed over

time rather than over space

MSU/CSE Fall 2017

Two Frames of Aerial Imagery

Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.

MSU/CSE Fall 2017

Best Matching Blocks Between Video Frames N+1 to N (motion vectors)

The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!

Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)

MSU/CSE Fall 2017

Gradient from 3x3 Neighborhood Estimate both magnitude and direction of the edge.

MSU/CSE Fall 2017

2 Rows of Intensity vs Difference

MSU/CSE Fall 2017

Boundaries not Always Found Well

MSU/CSE Fall 2017

Canny Edge Operator

MSU/CSE Fall 2017

Color and Shading n  Used heavily in human vision n  Color is a pixel property,

making some recognition problems easy

n  Visible spectrum for humans is 400nm (blue) to 700 nm (red)

n  Machines can “see” much more; ex. X-rays, infrared, radio waves

MSU/CSE Fall 2017

Imaging Process (review)

MSU/CSE Fall 2017

Factors that Affect Perception •  Light: the spectrum of energy that illuminates the object surface •  Reflectance: ratio of reflected light to incoming light •  Specularity: highly specular (shiny) vs. matte surface •  Distance: distance to the light source •  Angle: angle between surface normal and light source •  Sensitivity how sensitive is the sensor

MSU/CSE Fall 2017

CV: Perceiving 3D from 2D

Many cues from 2D images enable interpretation of the structure of the 3D world

producing them

MSU/CSE Fall 2017

Many 3D Cues

How can humans and other machines reconstruct the 3D nature of a scene from 2D images? What other world knowledge needs to be added in the process?

MSU/CSE Fall 2017

What about Models for Recognition

“recognition” = to know again; How does memory store models

of faces, rooms, chairs, etc.?

MSU/CSE Fall 2017

Some Methods: Recognize n  Via geometric alignment: CAD n  Via trained neural net n  Via parts of objects and how they join n  Via the function/behavior of an object

MSU/CSE Fall 2017

CV Applications and New Topics

MSU/CSE Fall 2015

Aerial Images & GIS n  Aerial image

of Wenatchie River watershed

n  Can correspond to map; can inventory snow coverage

MSU/CSE Fall 2015

Medical Imaging is Critical n  Visible

human project at NLM

n  Atlas for comparison

n  Testbed for methods

44

Manufacturing case

100 % inspection needed Quality demanded by major buyer

Assembly line updated for visual inspection well before today’s powerful computers

MSU/CSE Fall 2015

45

Simple Hole Counting Alg. n  Customer needs

100% inspection n  About 100 holes n  Big problem if any

hole missing n  Implementation in

the 70’s n  Alg also good for

counting objects

MSU/CSE Fall 2015

46

Imaging added to line n  Camera placed

above conveyor line

n  Back lighting added

n  1D of image from motion of object past the camera

MSU/CSE Fall 2015

47

Critical “corner patterns” n  “external corner”

has 3(1)s and 1(0)

n  “internal corner” has 3(0)s and 1(1)

n  Holes computed from only these patterns!

MSU/CSE Fall 2015

48

Hole (Object) Counting Alg.

MSU/CSE Fall 2015

49

#holes = (#e - #i)/4

MSU/CSE Fall 2015

50

Variations on Algorithm n  Easy if entire image is in memory n  Only need to have 2 rows in memory at any

time * used in the 1970’s * can allow special hardware Relate to driving around city blocks. Check out C++ program and results on web.

MSU/CSE Fall 2015

51

Industrial Vision/Inspection n  Literally thousands of applications n  Usually very specific engineering n  Usually called “image processing” – not

“computer vision”

MSU/CSE Fall 2015

MSU/CSE Fall 2015

Some Hot New Topics n  Phototourism: from hundreds of overlapping

images, maybe some from cell phones, construct a 3D textured model of the landmark[s]

n  Photo-GPS: From a few cell phone images “the web” tells you where you are located [perhaps using the data as above]

n  Deep learning for computer vision

MSU/CSE Fall 2015

Photo Tourism http://phototour.cs.washington.edu/

Snap to Paint

MSU/CSE Fall 2015

MSU/CSE Fall 2015

Snap to Paint

Seated Nude (Femme nue assise)' by Pablo Picasso, 1910

Snap to Paint

"The Shipwreck of the Minotaur" by J.M.W. Turner, 1805

MSU/CSE Fall 2017

Summary n  Images have many low level features n  Can detect uniform regions and contrast n  Can organize regions and boundaries n  Human vision uses several simultaneous

channels: color, edge, motion n  Use of models/knowledge diverse and

difficult n  Last 2 issues difficult in computer vision