Contents• Description of the big picture

• Theoretical background on this work

• The Algorithm

• Examples

Depth reconstruction process

• Image acquisition

• Camera modeling

• Feature acquisition

• Image matching

• Depth determination

• Interpolation

Image acquisition (in aerial photo)

• Images are vertical (the camera directed downwards)

• Images are taken from high point• Images are taken at very small time intervals• Rotation is minimal• The height is almost the same

Aerial photo

Camera modeling

To provide function that maps pairs of corresponding points from two stereo images onto scene points one need the camera model:

• Focal length of the camera Focal length = Zoom

• Camera location• Tilt – No tilt

Feature acquisition

• Pixels grouped into a region contain match more information

• There exists two techniques of image matching:– Area matching:

Compare pixels in both images– Feature based:

Corners, junctions, straight or curved segments or other features are extracted from images and then are being matched

The algorithm combines the both techniques.

Problems in feature based matching – occlusion

Feature acquisition problems – different angle

(not our case)

Image matching• Both images are projections of the same 3D object,

thus along with camera model one could reconstruct the original 3D object.

• Common matching constraints:– Epipolarity constraint: Epipolarity constraint: point on a given epipolar

line can match only to the same point on the corresponding epipolar line on the other image.

– Uniqueness constraint:Uniqueness constraint: no two objects in the scene project to the same object in an image

– Ordering constraint:Ordering constraint: projections of two objects along an epipolar line can't swap order

Image matching continued• Common matching constraints (continued):

– Smoothness constraint: Smoothness constraint: difference of disparities of two nearby objects doesn’t exceed a threshold value.

– Orientation constraint:Orientation constraint: difference of orientations of two matching segments is bounded.

– Contrast constraint:Contrast constraint: matched features has the same contrast sign.

Matching techniques• Best resemblance approachBest resemblance approach

• Coarse to fineCoarse to fine

• Evaluation-propagation approachEvaluation-propagation approach

There are more techniques to do it.

Best resemblance Best resemblance approachapproach

• Use a local pixel information to decide a potential match

• Match decision is made independently of the others (pixels)

• Main match criteria is a good resemblance between regions of pixels in two stereo images.

Coarse to fine approachCoarse to fine approach

• Hierarchical approach – matching information is passed from coarse to finer level of computation.

• Features are matched with different level of resolution.

• The coarse disparities are used to constrain the finer matches.

Evaluation propagation Evaluation propagation approachapproach

• Uses neighborhood relations between matching features.

• Potential matches form a correspondence graph whose nodes are pairs of matching segments. Edges of graph join features that doesn’t violate the smoothness and uniqueness constraints.

Our algorithm’s Our algorithm’s technique.technique.

• The algorithm uses combination of Best resemblance and evaluation-propagation.

• It builds neighborhood graph.

• Initial disparity of some features is propagated to other features.

• Local matches are evaluated with the best resemblance approach.

Theoretical Background

• The algorithm is another implementation of partial Hausdorff distance

• Pixel based method and segment based method

The Algorithm

• Preprocessing

• Boot process

• Evaluate-propagate stage

• Postprocessing

Pixel based methodAmit’s thesis

• Divide both images to grid of search windows• Find the disparity of corresponding windows, while

they are treated as set of black pixels.• If percent of matching pixels is small, repeat the

previous step using the previously found disparity as approximated initial disparity.

• The major disadvantage in our case is the impossibility to obtain objects original height, because the calculated height will be the average per search window!

Segment based method

• There exists a method, which treats both images as set of segments.

• It’s main disadvantage is Runtime complexity

• The algorithm will mix both techniques!

)log( 34 nnO

Preprocessing - 1

• Use edge detection on both images.

• Create non crossing segments.

• Split wiggly segments

• Build weakly visibility graph.

Split wiggly segment

d

l

s

• Find the farthest point P where d>= some threshold value.

•Split the segment at point P.

•Repeat recursively.

Weak visibilityi

j

k

Weakly visibility graph

• Vertices – segments in left image.

• Edges connects vertices if the corresponding segments are weakly visible.

• Edge’s weight is the minimal vertical or horizontal distance between the corresponding segments.

• Graph is built by sweepline technique.

Sweepline structure

Y-A

xis

X1 2 3 4 5 6 7 8

1

2

3

4

5

6

7

8

9

543

456N

L LLL

NNN

Seg_fieldX_field

Sweepline technique• Create events stack

• Iteratively process the event points e from left to right:

Y is the Y coordinate of e (black pixel of segment I)

a. If seg_field(Y) <> Nil then I and J (pointed by seg_field) are neighbors, and their distance is X-X_field(Y); If J is already in the Graph then the distance is the minimum

b. Seg_filed(Y)=I, x_field(Y)=X

Preprocessing - 2

• After the graph was built its MST is constructed by PRIM algorithm.

• Neighboring vertices are visible and close

• Long and strait segment is chosen as the root.

• In the right image distance transform is computed and each pixel hold the coordinates of the closest black pixel and distance from it.

Why initial disparity ?

• Wrong values may be obtained if there is no way to minimize the search window for a specific segment.

L

JK

M

Initial disparity

• Find the most long and straight segment in the right image.

• Find it in the left image.

• Let its corresponding vertex’s initial disparity be its disparity.

Evaluate – propagate stage

• BFS like algorithm is implemented in this stage.

• Calculation for a vertex in BFS:– Find its disparity using the disparity of its

parent vertex– Propagate it’s disparity as the initial disparity

to it’s children

Segment’s disparity

• Segment disparity – disparity of the whole segment.

• Pixel disparity – for pixel of some segment S, distance to the closest black pixel in the right image, under translation t, which is it’s segment’s disparity.

• Each pixel in left image holds the closest black pixels from the right image

Segment’s disparity (2)

• Each segment is being searched in the window defined by it’s initial disparity and double the maximum allowed disparity

• For each tested disparities two values are minimized:– D1 Hausdorf distance of segment– D2 Hausdorf distance of segment with its MST

neighbors

80% of pixels are used during the match processBest disparity is chosen among competing disparities

Post processing

• Pixels may hold more then one close black pixel or none.

– If closest pixel is too far it remains unmatched– If it is unmatched it remains unmatched– If it have more then one matched pixel

1) Remove already matched candidates

2) Among the rest choose pixel from segment which has the most matching pixels with the segment of the pixel

Runtime

• The runtime of algorithm is-

Which may be represented by

Where An is average number of neighbors

L is total number of black pixels

W is the size of search window

)*)))()((((1 1

wnlenilenON

i

n

j iji

)**( wALO n

Depth determination

• When images are rectified (only horizontal displacement of images)

InterpolationInterpolation

• The problem – feature based approach doesn’t provide depth for the whole scene.

• There are two methods:– Suppose that there is a continuous function of

depth that can be approximated to the sparse depth array achieved in previous steps

– Try to fit known geometric models to the sparse depth array.