1070-986X/03/$17.00 © 2003 IEEE Published by the IEEE Computer Society 57

Feature Article

Route panorama is anew image mediumfor digitallyarchiving andvisualizing scenesalong a route. It’ssuitable forregistration,transmission, andvisualization of routescenes. This articleexplores routepanoramaprojection,resolution, andshape distortion. Italso discusses how toimprove quality,achieve real-timetransmission, anddisplay long routepanoramas on theInternet.

In ancient western art, compositionsshowed the perspective of a scene’s pro-jection toward a 2D view plane. In con-trast, an ancient oriental painting

technique exposed scenes of interest at differentlocations in a large field of view, which let view-ers explore different segments by moving theirfocus. A typical and famous example is a longpainting scroll named A Cathay City (see Figure1), painted 900 years ago. It recorded the pros-perity of the capital city of ancient China in theSong Dynasty by drawing scenes and eventsalong a route from a suburb to the inner city.

The invention of paper in ancient Chinaallowed most paintings to be created on foldableand scrollable papers instead of canvases. Asopposed to a single-focus perspective projectionat one end of a street or a bird’s eye view, onebenefit of this scrollable style is that a path-oriented projection can display more detailedvisual information in an extended visual field

than a single-focus perspective projection at oneend of a street or a bird’s-eye view from the top.

Today we can realize a composition approachsimilar to A Cathay City by creating a new imagemedium—Route Panorama, a program that con-tains an entire scene sequence along a route. Wegenerate long route panoramas by using digitalimage-processing techniques and render routeviews continuously on the Internet. Now we cancapture, register, and display route panoramasfor many miles taken from vehicles, trains, orships. We can use this approach for many prac-tical purposes, including profiles of cities for vis-itors or for introductory indexes of localhometowns. In an extended manner, we couldprobably even use it as part of something like anenhanced version of Mapquest for people navi-gating through large cities like Los Angeles orBeijing. Figure 2 (next page) shows an exampleof a route panorama from a one-hour video inVenice, a place where people often get lost.

Route scenes on the InternetThere are many things to consider when cre-

ating a quality route panorama. To begin with,we create a route panorama by scanning scenescontinuously with a virtual slit in the cameraframe to form image memories. We call it a virtualslit because it isn’t an actual lens cover with a slitin it; rather, for each camera image in the videosequence, we’re copying a vertical pixel line at afixed position.

We paste these consecutive slit views (orimage memories) together to form a long, seam-less 2D image belt. We can then transmit the 2D

Figure 1. A Cathay City

painted on an 11 m

scroll in the Song

Dynasty 900 years ago.

(From the collection of

the National Palace

Museum, Taipei,

Taiwan, China.)

Digital RoutePanoramas

Jiang Yu ZhengIndiana University–Purdue University, Indianapolis

image belt via the Internet, enabling end users toeasily scroll back and forth along a route. Theprocess of capturing a route panorama is as sim-ple as recording a video on a moving vehicle. Wecan create the route panorama in real time with aportable PC inside the vehicle or by replaying arecorded video taken during the vehicle’s move-ment and inputting it into a computer later.Nevertheless, the generated image belt with itsconsecutive slit views pieced together has muchless data than a continuous video sequence.

My colleagues and I first invented a routepanorama 10 years ago for mobile robot naviga-tion.1-3 We called it a generalized panoramic viewbecause we discovered it while creating the firstdigital panoramic view. In this early study, wemounted a camera on a moving vehicle and itconstantly captured slit views orthogonal to themoving direction. The route panorama is a specialcase of a more general representation called adynamic projection image,4 which forms the 2Dimage using a nonfixed, temporal projection.Figure 3 (next page) illustrates how a vertical slitcan scan the street scenes displayed in Figure 2when the camera is positioned sideways along asmooth path. This viewing scheme is anorthogonal-perspective projection of scenes—orthogonal toward the camera path and perspec-tive along the vertical slit. Generally speaking,common modes of transportation such as a four-wheeled vehicle, ship, train, or airplane can pro-vide a smooth path for the camera.

Compared with existing approaches to modela route using graphics models,5,6 our routepanorama has an advantage in capturing scenes.It doesn’t require taking discrete images by man-ual operation or texture mapping onto geometrymodels. A route panorama can be ready after dri-ving a vehicle in town for a while. It yields a con-tinuous image scroll that other image stitching ormosaic approaches,7-9 in principle, are impossibleto realize. A mosaicing approach works well forstitching images taken by a camera rotating at astatic position. For a translating camera, howev-er, scenes with different depths have different dis-parities (or optical flow vectors) in consecutiveimages. Overlapping scenes at one depth will addlayers and create dissonant scenes at other depths,much like overlapping stereo images.

A route panorama requires only a small frac-tion of data compared to a video sequence and hasa continuous format when accessed. If we pile asequence of video images together along the timeaxis, we obtain a 3D data volume calledspatial–temporal volume that’s full of pixels (seeFigure 4, p. 60). The route panorama comprisespixel lines in consecutive image frames, whichcorrespond to a 2D data sheet in the spatial–tem-poral volume. Ideally, if the image frame has awidth w (in pixels), a route panorama only has1/w of the data size of the entire video sequence(w is 200~300), since we only extract one pixelline from each frame when viewing through theslit. The route panorama neglects redundant

58

IEEE

Mul

tiM

edia

Figure 2. Segment of

route panorama

recording canal scenes

in Venice.

scenes in the consecutive video frames, which wecan observe from the traces of patterns in theepipolar plane images (Figure 4 shows one EPI).This shows a promising property of the routepanorama as an Internet medium, which candeliver large amounts of information with smalldata. The missing scenes are objects under occlu-sion when exposed to the slit. Also, as normal 2Dimages capturing dynamic objects, a route panora-ma only freezes instantaneous slit views, ratherthan capturing object movements as video does.

Figure 5 (see p. 60) is another example of aroute panorama. It displays a segment of the FirstMain Street of China. We used a small digitalvideo camera on a double-decker bus to recordthe 5-km route panorama. No image device otherthan an image grabber is required in processing.

Projecting scenesCompared with existing media such as photo,

video, and panoramic views, route panorama hasits own projection properties. For instance, muchcan depend on the smoothness of the ride whilethe camera is recording. In an ideal situationroads are flat and the vehicle has solid suspen-sion, a relatively long wheelbase, and travels at aslow speed. Under these conditions, the camerahas less up-and-down translation and can tra-verse a smooth path along a street.

In this article, we assume the camera axis is sethorizontally for simplicity. We define a path-oriented coordinate system SYD for the projec-tion of scenes toward a smooth camera trace S onthe horizontal plane. We denote a point in sucha coordinate system as P(S, Y, D), where S is thecamera-passed length from a local starting point(see Figure 3b), Y is the height of the point fromthe plane containing the camera path, and D isthe depth of the point from the camera focus.Assume t and y are the horizontal and verticalaxes of the route panorama, respectively. Theprojection of P in the route panorama denotedby p(t, y) is

t = S/r y = Yf/D (1)

where r (m/frame) is the slit sampling interval onthe camera trace, which is the division of the vehi-cle’s speed V (m/second) by the camera’s framerate per second (fps). The camera’s focal length f(in pixels) is known in advance after calibration.

Because the route panorama employs anorthogonal-perspective projection, the aspectratio of an object depends on its distance from

the path. Figure 6 displays a comparison of viewsbetween an ordinary perspective image and aroute panorama. The further the object, thelower the object height is in the route panorama.Object widths in the route panorama are propor-tional to their real widths facing the street. In this

59

Camera

Camera trace

S

Slit

Virtual slit(pixel line)

Route panoramain long memory

Automaticcopy and

paste

Camera motion

Camera path

Image frame

P

(a)

(b)

Y

D

Y D

S

S

t

y

Buildings

Viewing plane(plane of sight)

Route panorama

Figure 3. Route scene scanning by constantly cutting a vertical pixel line in the

image frame and pasting it into another continuous image memory when the

camera moves along a smooth path on the horizontal plane.

60

x

t (image number)

y

Origin

Spatial-temporal volume

Route panorama

Image frame

Figure 4. Data size of a

route panorama is one

sheet out of a volume of

video images.

Figure 5. A segment of route panorama generated from a video taken on a bus (before removing shaking components).

HorizonVanishingpoint

Vanishing point

HorizonAsymptoteAsymptote

Perspective projection

Route panorama

Image frameL4

L2

L2

Hf /D1 Hf /D1

Hf /D2

Hf /D2

Wf /D1

Wf /D2W/r

W/r

L1 L1L1 L1

(a)

(c)

(d)

(b)

L4

Figure 6. 2D projections of 3D objects

in a route panorama compared with

a perspective projection image. W

stands for the object width, H is

object height, and D is object depth.

(a) Ordinary perspective projection.

(b) Orthogonal-perspective

projection. (c) A typical object in

perspective projection and (d) in an

orthogonal-perspective projection

images.

sense, distant objects are extended horizontallyin the route panorama. Thus, route panoramasaren’t likely to miss large architectures. Smallobjects such as trees, poles, and signboards areusually narrower than buildings along the t axisand might disappear after squeezing in the tdirection, while buildings won’t disappear. In anormal perspective projection image, a small treeclose to the camera may occasionally occlude alarge building behind it.

We examine several sets of structure lines typ-ically appearing on 3D architectures (see Figures6c and 6d) and find their shapes in the routepanorama from a linear path. Assuming a linearvector V = (a, b, c) in the global coordinate sys-tem with its X axis parallel to the camera path,the line sets are

❚ L1 {V | c = 0}: lines on vertical planes parallelto the camera path. These lines may appear onthe front walls of buildings.

❚ L2 {V | a = c = 0} ⊂ L1: vertical lines in the 3Dspace. These lines are vertical rims on archi-tectures.

❚ L3 {V | c ≠ 0}: lines stretching in depth fromthe camera path.

❚ L4 {V | c ≠ 0, b = 0} ⊂ L3: horizontal 3D linesnonparallel to the camera path.

Obviously, lines in L2 are projected as verticallines in the route panorama through the verticalslit. Denoting two points P1(S1,Y1,D1) andP2(S2,Y2,D2) on the line, where P1P2 = τV, their pro-jections in the route panorama are p1 and p2. Theprojection of the line is then

according to the projection model in Equation 1.For a line in L1 where ∆D = D2 − D1 = 0 and ∆Y/∆Sis constant b/a, its projection becomes

which is linear in the route panorama. Therefore,a line in L1 is still projected as a line in the routepanorama for a linear path.

The most significant difference from perspec-tive projection is a curving effect on line set L3(Figure 6d). For a line in L3, its projection in theroute panorama is a curve, because point P2 inthe route panorama is

which is a hyperbolic function of τ. This curveapproaches a horizontal asymptotic line y = fb/cfrom p1 when τ → ∞. Particularly for lines in L4,their projections are curves approaching towardthe projection of horizon (y = 0) in the routepanorama.

The path curvature also defines the lines’ curv-ing effect if the camera moves on a curved trace.Because of space, we omit the analysis here.Nevertheless, we can obtain reasonably goodvisual indexes of route scenes as long as we allowfor smooth curves in the route panorama.

Another interesting property of the routepanorama is the common asymptote for a set ofparallel lines stretching in depth (parallel lines inL3). Under perspective projection, we project par-allel lines with a depth change in the 3D spaceonto the image plane as nonparallel lines andtheir extensions on the image plane cross at acommon point called the vanishing point (accord-ing to the principle in computer vision). In theroute panorama obtained from a linear camerapath, however, a set of 3D parallel lines stretch-ing in depth has a common asymptotic line inthe route panorama. This is because parallel linesin L3 have the same direction (a, b, c), and theirprojections in the route panorama all approachthe same horizontal asymptotic line y = fb/cwhen τ → ∞.

t ySr

fYD

S Sr

f Y YD D

a Sr

f Y bD c

2 22 2

2

1 1

1

1 1

1

,( )

( )

( ) =

=

+ ++

=+ +

+

∆ ∆∆

τ ττ

v =

=

=

∆ ∆Sr

f YD

ar

fbD

ar

fbD

1 1 1

τ ττ

v = − =

−−

p p

S Sr

fYD

fYD2 1

2 1 2

2

1

1

61

July–Septem

ber 2003

We can obtain reasonably

good visual indexes of route

scenes as long as we allow for

smooth curves in the route

panorama.

If we fix the camera axis so that the plane ofsight through the slit isn’t perpendicular to thecamera path, we obtain a parallel-perspectiveprojection along a linear path because all the per-spective planes of sight are parallel. We can fur-ther extend this to a bended-parallel-perspectiveprojection when the camera moves along acurved path. We can extend most properties ofthe orthogonal-perspective projection similarly.

Stationary image blur and close objectfiltering

When we’re actually recording a routepanorama, we obtain slit views by cutting apixel line in the image frame of a video camera.Every slit view itself then is a narrow-perspec-tive projection. The sampling rate of the slithas a limit lower than the video rate. If thevehicle speed isn’t slow enough, it’s reflectedin the route panorama, because the panoramais actually the connection of narrow perspec-tive projections at discrete positions along theroute (as Figure 7 depicts). Scenes contained ina narrow wedge are projected onto the one-pixel line at each time instance. We examinesurfaces that can appear in three depth rangesfrom the camera path. First, at the depth wheresurface 2 in Figure 7 is located, each part of thesurface is taken into consecutive slit viewswithout overlapping, just as a normal perspec-tive projection does. We call this the just-sam-pling range (depth).

Second, for a surface closer than the just-

sampling range, the consecutive slit views don’tcover every fine part of the surface (surface 3 inFigure 7). Surfaces in this range are undersam-pled in the route panorama. If the spatial fre-quency of intensity distribution is low on thesurfaces—that is, the surface has relativelyhomogeneous intensity—we can recover theoriginal intensity distribution from the sampledslit views (the route panorama), according tothe Nyquist theorem in digital signal process-ing. Otherwise, we may lose some details with-in that range. Therefore, route panorama has afunction of filtering out close objects such astrees, poles, people, and so forth. By reducingthe camera’s sampling rate, this filtering effectbecomes clearer and more distinct. This is help-ful when we’re mainly interested in architec-tures along a street.

Third, if a surface is farther than the just-sampling range, the camera could oversamplethe surface points. Because a slit view accumu-lates intensities in the narrow perspectivewedge, a point on surface 1 may be counted inthe overlapped wedges of the consecutive slitviews. Therefore, a distant object point retain-ing at a position in the perspective projectionmay cause a blur horizontally in the routepanorama. We call this phenomenon stationaryblur, since it’s converse to the motion blureffect in a dynamic image where a fast translat-ing point wipes across several pixels during theimage exposure.

We can give the just-sampling depth’snumerical computation in a more general formto include bended parallel-perspective projec-tion. If we set a global coordinate system O-XYZ, we can describe the smooth camera pathby S[X(t), Z(t)]. If the vehicle is moving on astraight lane without obvious turns, the camerapath almost has zero curvature (κ ≅ 0). Selectinga camera observing side, we can divide a pathroughly as linear, concave, or convex segments,depending on the sign of curvature. For sim-plicity, we assume the radius of curvature R(t) ofa curved camera path is constant between twoconsecutive sampling positions S1 and S2, whereR(t) > 0 for a concave path, R(t) < 0 for a convexpath, and R(t) = ∞ for a linear path. The curvelength between S1 and S2 is r. The wedge’s angleis 2θ where f × tanθ = 1/2, because we cut onepixel as the slit width (see Figure 8).

For bended parallel-perspective projection,the plane of sight—which is the wedge’s centralplane—has an angle α from the camera transla-

62

IEEE

Mul

tiM

edia

Slit view covered regions

Oversampling range

Just-sampling depth

Undersampling range

Focal length of camera

Over sampledSampled surfaceSlit view

S1

PJ

S2

Surface 1

Surface 2

Surface 3

Slit

Camera path

Figure 7. Oversampling

range, just-sampling

depth, and

undersampling range of

the route panorama.

tion direction that is the curve’s tangent vector.The two consecutive wedges of thin perspectiveprojection meet at a vertical line through pointPj in Figure 8. We have a vector relation on thehorizontal plane as

S1Pj = S1S1 + S1Pj

in triangle S1S2Pj. Then we obtain S1Pj and S2Pj as

by using sine theorem. It isn’t difficult to furthercalculate the just-sampling depth Dj in the planeof sight to the just-sampled surface. Figure 9shows the equation for this.

It’s important to note that the just-samplingrange not only relies on the camera’s samplingrate, vehicle speed, image resolution, and camer-a’s focal length, but also depends on the camerapath’s curvature. The just-sampling range tendsto be close when the camera moves on a concavepath and far on a linear path. When the camera

path changes from linear to convex (R(t) variesfrom −∞ to 0−), Dj extends to the infinity (Dj →+∞) and then starts to yield negative values (Pj

flips to the path’s other side). This means that theconsecutive perspective wedges won’t intersectwhen the convex path reaches a high curvatureand the entire depth range toward infinity isundersampled.

For a simple orthogonal-perspective projec-tion toward a linear path, we can simplify theequation in Figure 9 to

by setting α = π/2 and R(t) → ∞. Overall, there aredifferently sampled regions in a route panoramadepending on the subjects’ depths. We can selecta sampling rate so that the just-sampling range is

D

rj =

2 tanθ

S P

R tr

R tr

R t

rR t

S P

R tr

R tr

R t

rR t

j

j

1

2

22 2

2

22 2

2

=+ +

+

=− −

+

( ) sin( )

sin( )

sin( )

( ) sin( )

sin( )

sin( )

α θ

θ

α θ

θ

63

July–Septem

ber 2003

1 pixel

Radius of curvature

Motion vector

S1 S2

=S2Pj

Dj

Pj

R(t)Camera path

Camera path

Camera axis

θ

α

(a)

(b)

Figure 8. Just-sampling range for a curved camera path. (a) Curved camera path and a physical sampling wedge. (b) Two

consecutive slit views and just-sampling depth.

DS P S P

S P S P

R tr

R tr

R tr

R t

rR t

rR t

rR t

jj j

j j

=×

+

=+ +

− −

+

+ +

+ − −

1 2

1 2

22 2 2

22 2

cos

( ) sin( )

sin( )

sin( )

cos

sin( )

sin( )

sin(

θ

α θ α θ θ

θ α θ α θ

))

Figure 9. Calculating the just-sampling depth Dj in the plane of sight to the

just-sampled surface.

approximately at the front surfaces of the build-ings of interest.

The collection of slit views is a process ofsmoothing spatial intensity distribution: outputvalue at a point by averaging intensities over aspace around it. We can estimate the degree ofstationary blur as follows. At depth D, the widthof the perspective wedge is W = D tanθ. We canaverage the color distribution at depth D over Wto produce a pixel value for the slit view, whichis the convolution between the intensity distrib-ution and a rectangular pulse function withwidth W and height 1/W. If we set a standard testpattern at depth D that’s a step edge with unitcontrast or sharpness, we can easily verify thatthe convoluted result is a triangular wave withthe contrast reduced to 1/(D tanθ). Therefore, anedge’s sharpness is inversely proportional to itsdepth. This is important when estimatingobjects’ depth in a route panorama.

If a segment of route panorama mainly con-tains objects far away, we can squeeze it alongthe t axis to reduce the stationary blurring effectand, at the same time, reduce shape distortion.This scaling may visually improve the objects’appearances in the route panorama if there’s norequirement to keep the exact scale or resolutionof the route panorama horizontally.

Dealing with camera shakingImproving image quality is a crucial step

toward the real application of route panoramas inmultimedia and the Internet. In Figure 5, we

pushed a small video camera firmly on a windowframe of the bus to avoid uncontrollable acciden-tal camera movement. We still can observe severezigzags on horizontal structural lines in the routepanorama. This is because the camera shookwhen the vehicle moved over an uneven road. Tocope with the camera shaking, some have tried tocompensate by using a gyroscope. However,adding special devices might increase the diffi-culty of spreading this technology. Our approachwas to develop an algorithm to rectify distortionaccording to some constraints from scenes andmotion.

We can describe the camera movement at anyinstance by three degrees of translations andthree degrees of rotations (as displayed in Figure10). Among them, the camera pitch caused byleft-and-right sway and the up-and-down trans-lation caused by the vehicle shaking on unevenroads are the most significant components affect-ing the image quality. The latter only yields smallup-and-down optical flow in the image if thevehicle bumping is less than several inches.

Overall, camera pitch influences image qual-ity the most and luckily we can compensate forit with an algorithm that reduces camera shak-ing (many algorithms along these linesexist).10,11 Most of these algorithms work bydetecting a dominant motion componentbetween consecutive frames in the imagesequence. For a route panorama, however, weonly need to deal with shaking componentsbetween consecutive slit lines. The idea is to fil-ter the horizontal zigzagged lines in the routepanorama to make them smooth, whichinvolves spatial processing. We do this by usingthe following criteria:

❚ smooth structural lines in the 3D space shouldalso be smooth in a route panorama, and

❚ vertical camera shaking (in pitch) joggles theslit view instantly to the opposite direction.

As Figure 11 illustrates, we estimate the cam-era motion from the joggled curves in the routepanorama and then align vertical pixel linesaccordingly to recover the original straightstructural lines. The way to find an instant cam-era shaking is to check if all the straight linesjoggle simultaneously at that position. We trackline segments horizontally in the route panora-ma after edge detection and calculate their con-secutive vertical deviations along the t axis. At

64

IEEE

Mul

tiM

edia

(Camera pan Ry)Roll Rz

Pitch Rx

Camera

Vehicle turn

Wheelbase

Ty

Tx

Tz

Figure 10. Vehicle and camera model in taking a

route panorama. Note that (Tx, Ty, Tz, Rx, Ry, and

Rz) = (forward translation, up-and-down

translation, translation sideways, pitch, pan, and

roll), respectively. Also, translation sideways

doesn’t occur for a vehicle movement.

each t position, we use a median filter to obtainthe common vertical deviation of all lines toyield the camera’s shaking component. Themedian filter prevents taking an original curvein the scene as a camera-joggled line and result-ing in a wrong pitch value. After obtaining thesequence of camera parameters along the t axis,we prepare a window shifting along the hori-zontal axis and apply another median filter tothe camera sequence in the window, whicheliminates disturbances from abrupt vehicleshaking.

Suppose the original structure lines in thescenes are horizontal in an ideal route panorama(Figure 11a). The camera, however, shakes verti-cally over time (Figure 11b), which joggles thestructure lines inversely in the captured routepanorama (Figure 11c). By shifting all the verti-cal pixel lines according to the estimated cameramotion sequence, we can align curved linesproperly to make horizontal structure linessmooth in the route panorama (Figure 11d).

Figure 12 shows the route panora-ma of Figure 5 after removing thecamera shakes in the pitch. Thisalgorithm is good at removingsmall zigzags on the structure linesto produce smooth curves. Once weapply the algorithm, it’s easy tomodify the route panorama tomake major lines smooth andstraight.

Real-time transmission and displayOur next step is to transmit a long route

panorama on the Internet and to seamlesslyscroll it back and forth. Displaying and stream-ing route panoramas gives users the freedom toeasily maneuver along the route.

We developed three kinds of route panoramadisplays to augment a virtual tour. The first typeis a long, opened form of route panorama (seeFigure 2). The second type is a side-window view

(see Figure 13) that continuously reveals a sectionof the route panorama back and forth. We call ita route image scroll. You can control the directionof movement and the scrolling speed with amouse. The third type is a forward view of thevehicle for street traversing; we call it a traversingwindow (see Figure 14, next page).

We can combine these different displays in var-ious ways. In rendering a traversing window, wemap both side-route panoramas onto two sidewalls

65

July–Septem

ber 2003

Figure 11. Recovering straight structure lines from camera-

joggled curves in a route panorama.

Figure 12. Recovering smooth structure lines in the route panorama by removing the camera-shaking components.

Figure 13. Real-time streaming data transmission over the Internet to show route scenes.

(a)

(b)

(c)

(d)

along the street and then project to a localpanoramic view (a cylindrical image frame). Wethen display the opened 2D form of this panoram-ic view so that users can observe the street stretch-ing forward as well as architectures passing by,while traversing the route. We render the travers-ing window continuously according to the mov-ing speed specified by the mouse. Although thetraversing window isn’t a true 3D display, majorportions in the traversing window have an opticalflow that resembles real 3D scenes. As anotherform of use, it’s even possible to display these pseu-do-3D routes within a car’s navigation system.

As a route panorama extends to several miles,it’s unwise to download the whole image andthen display it. We developed a streaming datatransmission function in Java that can displayroute image scrolls and traversing windows dur-ing download. Because of the route panoramas’small data sizes, we achieved much faster trans-mission of street views than video.

The image belt provides a visual profile of along street for its compactness. By adding clickingregions in the image, the route panorama becomesa flexible visual index of routes for Web page link-ing. On the other hand, we can automaticallyscroll a route panorama in a small window to giveviewers the feeling that they’re viewing architec-tures and shops from a sightseeing bus. With twocameras directing left and right sides of the vehi-cle, we can establish two side views of a route bysynchronizing the route panoramas. If we drivevehicles passing every street in a town for all theroute panoramas, we can generate a visual map ofthe town for virtual tourism on the Internet.

With the tools discussed here, we can registerand visualize an urban area using panoramicviews, route panoramas, route image scrolls, andmaps. All these images have much less data com-pared to video and look more realistic than 3Dcomputer-aided design models. We can link areaswithin a city map on the Web to correspondingareas in the route panoramas so that clicking aspot on the map can a update route image scroll

accordingly (and vice versa). Eventually, we canuse these tools to visualize large-scale spaces suchas a facility, district, town, or even city.

ConclusionStreets have existed for thousands of years

and there are millions of streets in the worldnow. These streets have a tremendous amountof information including rich visual contextsthat are closely related to our lifestyles andreflect human civilization and history.Registering and visualizing streets in an effectiveway is important to our culture and commerciallife. With the Route Panorama software, whenyou click a map to follow a certain route, theroute panorama will also be scrolled according-ly, showing real scenes along the route. This willgreatly enhance the visualization of GeographicInformation Systems (GIS).

Because of the 2D characteristics of capturedroute panoramas, we can use them in a variety ofways. For instance, we can display and scrollthem on wireless phone screens or handheld ter-minals for navigation in cities or facilities. Byconnecting a route panorama database with theGlobal Positioning System, we can locate ourposition in the city and display the correspond-ing segment of a route panorama on a liquidcrystal display. Displaying route panoramas andpanoramic images are basically raster copy of sec-tions of images. Hence, the proposed techniquesare even applicable for 2D animation and gameapplications, potentially providing richer, morerealistic content than before.

Currently, we’re working on several innova-tions for our Route Panorama software. Thisincludes constructing 3D streets from routepanoramas, sharpening distance scenes due tothe stationary blur, establishing route panoramaswith a flexible camera setting, capturing highrises into route panoramas, combining routepanoramas with local panoramic views incityscape visualization, and linking route panora-mas to existing GIS databases. MM

66

IEEE

Mul

tiM

edia

Figure 14. The

traversing window

dynamically displays

two sides of route

panoramas in an open,

cylindrical panoramic

view for virtual

navigation along a

street.

East <- The 5th Street The 4th Street -> West

Move forwardSpeed up

Pause

67

July–Septem

ber 2003

References1. J.Y. Zheng, S. Tsuji, and M. Asada, “Color-Based

Panoramic Representation of Outdoor Environment

for a Mobile Robot,” Proc. 9th Int’l Conf. Pattern

Recognition, IEEE CS Press, vol. 2, 1988, pp.801-803.

2. J.Y. Zheng and S. Tsuji, “Panoramic Representation

of Scenes for Route Understanding,” Proc. 10th Int’l

Conf. Pattern Recognition, IEEE CS Press, vol. 1,

1990, pp.161-167.

3. J.Y. Zheng and S. Tsuji, “Panoramic Representation for

Route Recognition by a Mobile Robot,” Int’l J. Comput-

er Vision, Kluwer, vol. 9, no.1, 1992, pp. 55-76.

4. J.Y. Zheng and S. Tsuji, “Generating Dynamic Projec-

tion Images for Scene Representation and Recogni-

tion,” Computer Vision and Image Understanding, vol.

72, no. 3, Dec.1998, pp. 237-256.

5. T. Ishida, “Digital City Kyoto: Social Information

Infrastructure for Everyday Life,” Comm. ACM, vol.

45, no. 7, July 2002, pp. 76-81.

6. G. Ennis and M. Lindsay, “VRML Possibilities: The

Evolution of the Glasgow Model,” IEEE MultiMedia,

vol. 7, no. 2, Apr.–June 2000, pp. 48-51.

7. Z. Zhu, E.M. Riseman, and A.R. Hanson, “Parallel-

Perspective Stereo Mosaics,” Proc. Eighth IEEE Int’l

Conf. Computer Vision, IEEE CS Press, vol. I, 2001,

pp. 345-352.

8. H.S. Sawhney, S. Ayer, and M. Gorkani, “Model-

Based 2D and 3D Motion Estimation for Mosaicing

and Video Representation,” Proc. 5th Int’l Conf.

Computer Vision, IEEE CS Press, 1995, pp. 583-590.

9. S.E. Chen and L. Williams, “Quicktime VR: An

Image-Based Approach to Virtual Environment

Navigation,” Proc. Siggraph 95, ACM Press, 1995,

pp. 29-38.

10. Z. Zhu et al., “Camera Stabilization Based on 2.5D

Motion Estimation and Inertial Motion Filtering,”

Proc. IEEE Int’l Conf. Intelligent Vehicles, IEEE CS

Press, vol. 2, 1998, pp. 329-334.

11. Y.S. Yao and R. Chellappa, “Selective Stabilization

of Images Acquired by Unmanned Ground

Vehicles,” IEEE Trans. Robotics and Automation, vol.

RA-13, 1997, pp. 693-708.

Jiang Yu Zheng is an associate

professor at the Department of

Computer and Information Sci-

ence at Indiana University–

Purdue University, Indianapolis.

His current research interests

include 3D modeling, dynamic image processing, scene

representation, digital museums, and combining

vision, graphics, and human interfaces. Zheng received

a BS degree from Fudan University, China, and MS and

PhD degrees from Osaka University, Japan. He received

the 1991 Best Paper Award from the Information Pro-

cessing Society of Japan for generating the first digital

panoramic image.

Readers may contact Jiang Yu Zheng at jzheng@

cs.iupui.edu.

January–MarchContributions to Computing

Learn how yesterday’s contributions have shaped today’s comput-ing world. Featured articles cover topics such as the first assembly-line production of a digital computer, a manager’s account ofworking at Computer Sciences Corporation in the mid-1960s, andsearching for tractable ways of reasoning about programs.

April–JuneEvolution of Digital Computers

Digital computers have evolved into powerful workstations, butthat wasn’t always the case. Read about one man’s foray intodigital computing as well as the making of the MCM/70 micro-computer. This issue also features articles on the glory days ofDatamation and the history of the sector, an analog calculatinginstrument.

July–SeptemberHistorical Reconstructions

This special issue is devoted to recording and recounting efforts topreserve computing practice through physical reconstruction,restoration, and simulation. In addition to providing accounts ofsuch projects through substantive articles, the issue will serve as adigest of projects and initiatives as well as societies and organiza-tions active in these areas.

October–DecemberWomen in Computing

Since the days of Ada Lovelace, women have played an importantrole in the history of computing, and this role has received increas-ing attention in recent years. Scholarship in this area has begun tomove beyond simply demonstrating women's presence in thehistory of computing to considering how computing and genderconstructs have shaped one another over time.

2003Editorial Calendar

http://computer.org/annals