panoramic viewfinder

patrick baudischdesney tan, drew steedly,eric rudolph, matt uyttendaele,chris pal, and richard szeliski

microsoft research

OZCHI 2005

Cropping frame: this area will survive croppingCropping frame: this area will survive cropping

Preview: overall area covered so farPreview: overall area covered so far

Viewfinder: what the camera sees right nowViewfinder: what the camera sees right now

summary

cropping frame

preview

viewfinder

ultra portable PC + web cam

stitching photos intoa panorama

from photos [Teodosio & Bender, 1993]from video [Irani and Anandan, 1998]

stitching: allows taking wide-angle pictures

... getting hi-res photos

but sometimesit goes wrong…

• missing content (65% respondents)• tricky because of perspective projections

• ghosting (88% respondents)• hard to detect

• stitching failed (38% respondents)• lack of overlap, of texture, of focus

• when users notice flaws it is too late

but sometimesit goes wrong…

panoramicviewfinder

user interface

Cropping frame: this area will survive croppingCropping frame: this area will survive cropping

Preview: overall area covered so farPreview: overall area covered so far

Viewfinder: what the camera sees right nowViewfinder: what the camera sees right now

walkthrough: goalbuilding 114building 115

walkthrough

post-processing

• restitching• upload photo to PC• restitch using our high-quality offline stitcher

maximum image quality

post-processing

• auto cropping to rectangular shape• all desired content is preserved cropping frame guarantees this

• or manual cropping

live demo…

dealing with stitching failure

• if user moves camera too quickly (or lack of texture)• viewfinder remains stationary• but continues to update• turns red, error sound

• to fix the problem• pan camera back halfways inside panorama• and take an additional shot• or zoom out

• alert ends when next match is found

the real-timecropping frame...

1. shoot the desired scene elements

2. fill the bounding boxaround these elements

…is the key component

cropping frame tells users when they are done it is the focus of the user’s attention

(traditional: viewfinder is focus of attention)

interaction

• requirements• goal: make sure we have enough content• too much content is ok

• resulting design: control cropping frame indirectly by adding content• no resize handles!• users can focus on pointing camera• interaction transfers to consumer camera

what it does not do

• for artistic reasons, user may want to crop

• cannot crop panorama while shooting

• crop offline, with all other photos

related work

related work• stitch arbitrary order and arrangement

[Szeliski & Shum ‘97]• real-time [Peleg and Herman, 1997]

our focus

• user interface

• preview to help users create successful panorama

• match the interaction model of existing digital cameras allow port to existing camera interaction mechanisms

stitch assist

canon powershot S230 only horizontal

we made 4-way assist can’t fill area

take content in any ordercan add content post-hoc

HP Photosmart R707stitch in camera layback modestill need to retake panorama

related work on interaction tiltable interfaces[Rekimoto 1996]

chameleon[Fitzmaurice 1993]

peep-hole displays[Yee 2003]

paintable interfaces[Baudisch 1998]

implementation

computing panorama

• written in C, frontend GDI+ and DirectX9• feature-based stitching [Brown et al 92]

• uses same libraries as off-line stitcher1. extracting multi-scale oriented patches2. matching with previous image3. estimating camera orientation4. warping image

• Difference: keep existing panorama unchanged• benefit: faster (~ constant time)• drawback: more accumulation error

capture modes

• “video mode”• accumulates calibration error (Sawney et al. ’98)

• “auto mode”• snap picture only when necessary

• “manual mode”• best control, avoid motion blur

computing cropping frame

• naive: O(n4), but we do it in ~ O(n2)

• step 1: downsample image by factor 4

• step 2: for each pixel• compute hor and vert spans

• step 3: for each pixel• upper bound = hor span x vert span• if (upper bound < current max) move on• else traverse down

length of horizontalspan starting at this pixel

future work &conclusion

future: user study...

• as soon as we have a more reliable stitcher

• consider mocking-up stitcher to simulate 100% reliability

• Losing track breaks the flow

• Even with perfect stitcher there will be stitching failures

• We need to be able to lose track without breaking the flow

code editing

1. Batch compiler

2. Forced syntax

3. Curly underlines

1. Offline stitcher

2. Panoramic viewfinder

3. Asynchronous stitcher

vs. stitching

Panorama: largest sub panorama created so farPanorama: largest sub panorama created so far

2 1

Last frame added to the panoramaLast frame added to the panorama

Viewfinder: what thecamera sees right nowViewfinder: what thecamera sees right now

Burst buffer: # pictures taken, but not processedBurst buffer: # pictures taken, but not processed

Recycle buffer: # of frames not yet matchedRecycle buffer: # of frames not yet matched

future: form factor…

future

• start thinking about using cameras in mobile phones as pointing devices

conclusions

• one step closer to makingprocess of taking panoramic picturessimilar toprocess of taking normal photos

read more & try out

patrickbaudisch.com/projects

END