We use an annotateddigital photocollection todemonstrate a two-level auto-layouttechnique consistingof a central primaryregion withsecondary regionssurrounding it.Because the objectsizes within regionscan only be changedin discrete units, werefer to them asquantum content.Our real-timealgorithms enable acompellinginteractive display asusers resize thecanvas, or move andresize the primaryregion.

62 1070-986X/06/$20.00 © 2006 IEEE Published by the IEEE Computer Society

Feature Article

Automatic text-flow algorithms havebrought about a revolution in thepage-layout process. Writers andpublishers now take for granted that

text will break correctly at the end of a line, orflow around a drawing within a document.1

Today’s Web browsers work on this same princi-ple of dynamic flow, and most Web pages willreflow the text as the window is resized. If usersprint a document, properly configured text willreflow to meet the paper’s dimensions.

This article explores the possibility of perform-ing rapid, automatic layout with nonoverlapping,2D fixed-aspect-ratio objects, such as photos.These objects can appear in a central primaryregion, which defines the entire layout, or in sec-ondary regions that offer lower-level detail, suchas parents in the primary region, surrounded bysecondary regions containing many pictures ofeach of their children. We address such commontwo-level categories in this article, leaving morecomplex layouts for future work. In all cases, theprimary region highlights one photo, concept, ortext block; the secondary regions serve the pur-pose of showing organized subcollections that the-matically relate to the primary region.

The techniques presented here apply espe-cially to photo libraries, but might have variantsthat apply to newspaper layouts, online maga-zines, and electronic product catalogs. A com-pelling aspect of these algorithms is theinteractive redisplay as users change the rectan-gular canvas size and shape, add or delete sec-ondary regions, and add or delete objects to asecondary region. We combine these algorithmsand refer to them holistically as the bilevel radialquantum, or BRQ algorithm—pronounced as

“brick,” not only for pronunciation of theacronym, but also because of its playful resem-blance to a brick wall. This engaging animatedinteraction (see the “Dynamic behavior” section)is an important feature for consumer applicationssuch as personal photo management.

Case study: Photo layoutIn many commercial photo management

tools—such as ACD’s ACDSee, Adobe’s Photoshop,Apple’s iPhoto, and Google’s Picasa—photos arepresented in a simple grid. Because all of the pho-tos are the same size we refer to them as quantumcontent. While these programs allow advancedmetadata annotation, such annotation doesn’tcarry over to the presentation mode so that userscould view photos with different annotationssimultaneously, sorted appropriately.

Web-based commercial stock photo vendors—such as Punchstock, Corbis, and Getty—also usesimple grid layouts to show search results, eventhough they often have rich metadata thatwould support more orderly presentations.

Our work extends beyond the design of toolssuch as PhotoMesa (designed by the University ofMaryland’s Human–Computer Interaction Labora-tory), which show the benefits of a grid layoutbased on the directory structure or metadata anno-tations. In this article, we present several screenshots with accompanying descriptions that illus-trate how such a bilevel layout presents interestingpossibilities for this domain. More extensive justi-fication and a user study appear elsewhere.2

We generated all of our screen shots with theBRQLayer tool, which implements the algorithmswe describe. The program and source code arefreely available at http://www.cs.umd.edu/hcil/brqlayer. We created each layout by choosing sev-eral tags from the annotated collection (for exam-ple, family members Al, Shuly, Esther, Simmy, andLani in Figure 1), and then choosing a representa-tive photo for the primary region in the center.The layouts were then sized to the desired dimen-sions, and the primary region was filled and scaledto the desired location and size. Throughout, thethumbnails in the secondary regions maintained amaximal size as they resized and moved them-selves in response to changes in the primaryregion; no user input was required in creating theregions or choosing the thumbnail size.

Family photo collectionFigure 1 depicts a family layout, in which the

user has chosen photos for each family member.

HierarchicalLayouts forPhoto Libraries

Jack Kustanowitz and Ben ShneidermanUniversity of Maryland

In it, similar quantities of photos are in eachregion, which lends itself to a balanced view.Layouts with reasonably similar quantitiesshould be able to minimize wasted space, andwith some manipulation of the size and positionof the primary region, the user can create a lay-out that’s efficient and attractive.

Vacation tripsFigure 2 depicts a generated layout of photos

taken during a trip to Italy. This is a case in whichthe ordering is important, so that users can read-ily find a part of the trip that was toward thebeginning, middle, or end. This is also a goodexample of having more than four photo regions,along with the need for dynamic and proper bal-ancing. While other timeline-generation utilitiesexist (Picasa has a particularly appealing interac-tive timeline), this application allows the dynam-ic creation of a printable layout that shows theentire photo collection, supplemented with time-line information.

Organization chartsA common way of grouping is to represent

organizational hierarchies, but they don’t usuallyhave a sequential layout requirement. Figure 3(next page) shows six regions representing someresearch areas of the University of Maryland’sComputer Science Department with a photo forfaculty members. The primary region shows thedepartment chair. In this case, we chose a layoutthat contained enough horizontal blank space forthe titles. As an example, if the primary regionwere enlarged to the left, the thumbnail size couldremain constant, but the text “ScientificComputing” would get truncated.

Real estate browserFigure 4 presents a hypothetical real estate

Web site, containing a bilevel radial quantumphoto layout within a Web page. As userschange the browser window size, the layout,regions, and photos all size dynamically toallow for the largest photo size possible giventhe relative number of photos in each area. Inthis way, the photo flow resembles text reflow-ing or resizing.

Should a new house come on the market inSilver Spring, it would fit nicely in the blankspace under the houses currently visible. A newhouse in one of the other regions could pushthe photo size smaller, or might instead causethe secondary regions to be distributed differ-

ently about the primary region, possibly with-out requiring smaller photos.

Problem definition and requirements As we defined the problem, certain require-

ments became clear. For example, we wanted thecanvas size to remain stable, even if an algorithmcould discover an improved layout by reducingor enlarging the canvas dimension. Similarly,uniform size for all photo thumbnails was animportant goal, even if an algorithm could

63

Figure 1. Family photo collection depicting the main subjects in a primary

region surrounded by grouped photos of family members. Even a collection of

this size can be resized and manipulated in real time.

Figure 2. Honeymoon

in Italy showing six

locations during a two-

week trip. The radial

increase in shading

along with the date

captions reinforce the

sequential nature of

this series.

reduce unused space by reducing or enlargingsome thumbnails. Future research might relaxsome of the requirements or produce interestingvariations that are potentially applicable in otherdomains. Earlier work on layout requirements forphotos and interfaces provided guidance for ourwork1,3-5 (see also the “Related Work” sidebar).

Fixed canvas sizeThe algorithm can’t change the canvas size

(width by height, usually the window size cho-sen by users). As users vary the canvas size (thealgorithm runs again with each resize), the lay-out will change, still filling the canvas exactly,never causing it to grow or shrink under algo-rithmic control.

Relaxing this requirement would let the can-vas size grow if the algorithm determined that abetter layout could be achieved if the canvas were5 percent wider, for example. While possibly use-ful, this introduces a feedback loop in which thecanvas size determines layout, which in turndetermines canvas size. Preserving this require-ment allows the algorithm to be deterministic,avoids nonlinearity, and maintains the user’ssense of control.

Fixed primary region size and locationThe user sets the primary region’s size and

location; the algorithm doesn’t modify this. Ifusers change the primary region’s size and loca-tion, then the layout will be updated to reflectthe new size and position. As we mentioned inthe “Fixed canvas size” section, relaxing thisrequirement also introduces a feedback loop thatwe avoid for similar reasons.

Uniform quantum size and aspect ratioInitially, all of the quanta (in the test case,

photo thumbnails) must be the same size andaspect ratio. Photos that are intrinsically differ-ent sizes can be placed on a background that’sconstant, so that some photos take up the wholebackground and on some the background showsthrough on the margins. Once the layout is inplace, this requirement can be removed to mini-mize wasted space in each region. This shouldoccur as a later step to avoid feedback in the ini-tial layout algorithm.

Secondary regions distributed in quadrantsWe based the algorithms on the idea of having

four fixed quadrants surrounding the primaryregion (see Figure 5a on p. 66). Within each quad-rant we can place several secondary regions, butno secondary region can cross a quadrant bound-ary. Additionally, each quadrant must contain atleast one region. The benefit is a tight alignmentof secondary regions with the edges of the pri-mary region, thus making a visually appealinglayout that enables easy-to-scan grids of photosin each secondary region.

64

IEEE

Mul

tiM

edia

Figure 3. University of Maryland’s Computer Science Department organization

chart with seven research areas. In this layout, the constant thumbnail size

was enforced, causing a layout that doesn’t give artificial prominence to any

group of photos, at the expense of unused space in each region.

Figure 4. Real estate

browser with five

communities and the

price ranges of selected

homes. This illustrates

a potential use of the

BRQ algorithm on the

Web, which depends

heavily on real-time

page rendering and

dynamic reflowing as

the browser is resized.

65

Our work extends the work on ordered and quantumtreemaps1 applied in PhotoMesa.Treemaps map a hierarchy ontoa rectangular region in a space-filling manner; ordered treemapsadd an additional requirement of maintaining order in the layout.

Order-preserving layouts reduce the effect of rectanglesshifting position as the item sizes change. The quantum require-ment is useful in photo layouts, in which each rectangle hassubrectangles of a fixed size. The layout algorithms describedin this article add a central (both in importance and in position)rectangle, which can itself be sized and moved around the lay-out, with the other rectangles in turn rapidly moving and resiz-ing. This bilevel property gives prominence to a certain layoutfeature, whether it’s a graphic, descriptive text, or central fea-ture of a nonphoto application.

While our approach is to define the interaction of requirementsin a way that guarantees a solution, other creative ways to solveproblems with interrelated constraints include using Monte Carlomethods or genetic algorithms.2,3 Genetic algorithms have beenused to solve image placement problems in the context of auto-mated page layout in a digital album, using principles such as bal-ance, spacing, emphasis, and unity.4 These algorithms produceuseful results that are appropriate in cases where the constraints inquestion are fuzzy or subjective. In our work they’re more quan-tifiable, which allows for simpler and faster algorithms.

In this article, we present a radial algorithm for two-level lay-outs. Multilevel radial animations are described elsewhere,5 inthe context of navigating an expanding and contracting graphthat works based on polar coordinates. It’s possible that thiswork could be expanded either to higher-level layouts, or thata collection of photos could be navigated using a radial anima-tion to get from one two-level layout to a related one.

New presentation media could drive the demand for alter-nate 2D dynamic layout techniques. The Personal DigitalHistorian (PDH),6 for example, describes a tabletop display sev-eral times larger than the screens currently used to view photocollections. While PDH groups similar photos based on usermodeling, the algorithms described here could provide an alter-nate way to dynamically adjust the layout of a tabletop or otherlarge displays based on the user’s position at the table or cate-gory of interest. On still larger wall-sized displays, thousands ofphotos could be accommodated, making the utility of mean-ingful organization still greater. Even in a smaller, nontraditionaldisplay,7 our work could allow a control panel to be smoothlymoved around the screen while the surrounding photos resizedand flowed accordingly.

Our layout algorithm’s final result is that it provides a “focus +content” display, similar to the semantic fisheye view (SFEV)described elsewhere.8 The SFEV also uses image annotation datato generate a layout based on relationships between terms andsimilarities in the annotation text. Similar to the SFEV layouts,we can engineer the layouts produced in our work to have a

fisheye look, with the advantages of focusing the viewer’s atten-tion on a primary feature while still showing related images asbeing connected, if at a smaller size.

The Bramble system9 describes a toolkit for a constraint-basedlayout, using a differential, continuous approach. This allows fordynamic enabling or disabling of specific constraints, potentially ina 3D space. While we describe in the “Problem definition andrequirements” section on the problem requirements that resembleconstraint-based programming to a degree, our focus was on real-time, smooth, 2D motion to provide maximum interactivity.Generic constraint-based problem solvers require large memorycapacity and slow down as the problem size increases, while thedomain-specific algorithms we chose often provide real-time per-formance at the expense of generality.10 By avoiding backtrackingin the algorithms and using today’s fast processors, we succeededby simply recomputing the layout when the parameters changed.

References1. B. Bederson, B. Shneiderman, and M. Wattenberg, “Ordered

and Quantum Treemaps: Making Effective Use of 2D Space to

Display Hierarchies,” ACM Trans. Graphics, vol. 21, no. 4, 2002,

pp. 833-854.

2. W. Graf, “Constraint-Based Layout Framework LayLab and Its

Applications,” Proc. ACM Workshop on Effective Abstractions in

Multimedia, ACM Press, 1995; http://www.cs.uic.edu/~ifc/

mmwsproc/graf/mm95.html.

3. L. Purvis et al., “Creating Personalized Documents: An

Optimization Approach,” Proc. ACM Symp. Document

Engineering, ACM Press, 2003, pp. 68-77.

4. J. Geigel and A. Loui, “Using Genetic Algorithms for Album Page

Layouts,”IEEE MultiMedia, vol. 10, no. 4, 2003, pp. 16-27.

5. K. Yee et al., “Animated Exploration of Graphs with Radial

Layout,”Proc. IEEE Symp. Information Visualization, IEEE CS Press,

2001, pp. 43-50.

6. B. Moghaddam et al.,“Visualization & Layout for Personal Photo

Libraries,” Int’l Workshop Content-Based Multimedia Indexing,

2001; http://www.merl.com/publications/TR2001-028.

7. M. Balabanovic, L. Chu, and G. Wolff, “Storytelling with Digital

Photographs,” Proc. Conf. Human Factors in Computing Systems,

ACM Press, 2000, pp. 564-571.

8. P. Janacek and P. Pu, “An Evaluation of Semantic Fisheye Views

for Opportunistic Search in an Annotated Image Collection,”

Int’l J. Digital Libraries, vol. 5, no. 1, 2005, pp. 42-56.

9. M. Gleicher, “A Graphics Toolkit Based on Differential

Constraints,” Proc. 1993 ACM Symp. User Interface Technology,

ACM Press, 1993, pp. 109-120.

10. W. Hower and W. Graf, “Research in Constraint-Based Layout,

Visualization, CAD, and Related Topics: A Bibliographical

Survey,” Proc. Int’l Workshop on Constraints for Graphics and

Visualization (CGV 95), 1995; http://citeseer.ist.psu.edu/

hower95research.html.

Related Work

66

The alternatives are either to tighten thisrequirement or to remove it entirely. Tighteningto eight octants (eight boxes surrounding the pri-mary region) would maintain alignment with theprimary region to some degree, but would allowdegenerate cases with L-shaped and C-shapedregions, which could pose scanning problems asthe viewer tries to determine if the photos shouldbe viewed left-to-right, up-and-down, and so on(see region 4 in Figure 5b). In the case of quad-rants, this is only a problem if there are fewerthan four regions, and it can be avoided by forc-ing the primary region to a corner in these cases.For octants, L- and C-shaped regions remain apossibility for up to eight regions, which coversmany of the most common scenarios.

Removing the quadrant requirement entirelycomplicates things even further, as C-shapes andL-shapes would be allowed with no guarantees oflining edges up with the primary region (seeFigure 5c). Trying to place regular thumbnails inregions 5 or 7 of Figure 5c could prove difficult.Additionally, layouts causing irregularly shapedregions make it difficult to provide an easilyscannable photo grid.

Fixed number of thumbnailsThe number of thumbnails in a given region

is fixed, in that the algorithm cannot arbitrarilyadd or remove thumbnails to create a more bal-anced layout. If one region has dramaticallymore thumbnails than another, an applicationmight find it advantageous to allow a region tohave only x times the number of thumbnails ofanother, or to specify a minimum thumbnail size

which (given a fixed canvas size) necessarilychanges the number of thumbnails that are visi-ble. In these cases, a scrollbar or a “More” buttoncould be added to view the thumbnails that wereleft out.

For the purpose of this discussion, we let usersdecide on the number of thumbnails, but oncethat decision is made, the number of thumbnailsthat the algorithm works with is fixed. Allowingthe algorithm to internally fine-tune these para-meters introduces the undesirable feedback loopdiscussed previously.

Fixed order of regionsThe regions should be laid out in the order in

which they are added. This is useful in applica-tions where order is important, whether it’salphabetic, the age of children, or a numeric pageorder for a table of contents. Preserving orderoffers the additional benefit of stability whenresizing, because it’s distracting and undesirableto have regions jumping around when the can-vas is resized.6

Bilevel radial quantum layoutBased on the requirements that we discuss in

the “Problem definition and requirements” sec-tion, the following algorithms (that are part ofBRQ) will generate regions with the largest pos-sible thumbnail size. We describe the layouts asbilevel to indicate our solution is for two-levelhierarchies, as radial to indicate that the sec-ondary regions wrap around the primary regionin an ordered manner, and as quantum to indi-cate that the items in each region are fixed in sizeand shape.

BRQ layout algorithmThe BRQ layout algorithm comprises three

steps:

1. Distribute the secondary tiles among the fourquadrants, using the RegionSplit algorithm(see Figure 6).

Rgn1 Rgn2 Rgn3

Rgn4

Rgn5Rgn6

Rgn7 PrimaryPrimaryPrimary

Quad2

Quad3

Quad4

Quad1 Rgn1

Rgn4 Rgn3

Rgn2

(a) (b) (c)

Figure 5. (a) Four

quadrants distributed

about a primary region.

This is our chosen

solution. (b) Awkward

L-shape caused by

using octants for

secondary regions. Our

solution prevents this.

(c) Misalignments

caused by arbitrary

region placements. Our

solution prevents this.

Second split Second splitFirst split

Rgn1 Rgn2 Rgn3 Rgn4 Rgn5

Figure 6. Distribution

of five secondary

regions among the four

quadrants.

2. Set the initial quantum width and height,which is guaranteed to be an upper bound onthe possible quantum dimensions, using theInitialQuantumDim algorithm.

3. Reduce the quantum dimensions (keeping thesame aspect ratio) until there is no overflow,using the ReviseQuantumDim algorithm. If thequantum dimensions drop below a specifiedminimum, handle the layout as a special case.

RegionSplit algorithm. This algorithm opti-mally divides regions up among the four quad-rants and works as follows:

1. Choose the region n (0 < n = NumRegions)such that the sum of the thumbnails in allregions up to and including region n is clos-est to 1/2 of the total number of thumbnailsin all regions. This will split the regions intotwo groups.

2. Do Step 1 for each of the two groups, gener-ating a total of four groups.

The RegionSplit algorithm requires three calls,the first splitting the entire collection of regionsin two, and then once again on each half result-ing from the first step. Thus, the time forRegionSplit scales linearly with the number ofsecondary regions.

InitialQuantumDim algorithm. This algo-rithm sets initial dimensions for maximalthumbnail size. This means that the algorithm’soutput (tlength, twidth) is defined to be an upperbound on the thumbnails’ size. In the (rare) casewhere every region has exactly rows × columnsthumbnails, this will also be the final thumbnailsize. In every other case, the thumbnail size willneed to be reduced until there is no overflow.The algorithm works as follows:

1. Create four rectangles with the same dimen-sions as the quadrants, as in Figure 7.

2. Let |ti| {1 < i = 4} = the total number of thumb-nails in the quadrant by summing the totalnumber of thumbnails in each region in thequadrant.

3. If the thumbnails must exactly fit the space,then the following would be true for eachquadrant:

theightCandidate ∗ twidthCandidate ∗ |ti| =clientAreai

4. Solve for quadrant 1, using the known aspectratio of the thumbnails:

theightCandidate =Sqrt(clientArea1 ∗ thumbnailAspectRatio/|t1|)

twidthCandidate =theightCandidate/thumbnailAspectRatio

These two values are now upper bounds on thethumbnail size (the “Optimality discussion” sec-tion explains why).

5. Solve for the other three quadrants, reducingtheightCandidate and twidthCandidate if lowervalues are found.

The InitialQuantumDim algorithm is simplya set of calculations for each of the four regions,and thus operates in constant time.

ReviseQuantumDim algorithm. This algo-rithm takes as its input dimensions that are anupper bound on the possible thumbnail dimen-sions. It revises those dimensions downwarduntil there are no more columns (quadrants 2and 4) or rows (quadrants 1 and 3) than any ofthe quadrants has room for. The algorithm oper-ates as follows:

1. For quadrant 1, determine the number ofcolumns used by the first region as follows:

columns = ceiling(quadrantClientWidth/tWidthCandidate)

67

Octo

ber–D

ecemb

er 2006

Figure 7. Quadrant

rectangles for

computing an upper

bound of the

thumbnail size.

68

IEEE

Mul

tiM

edia

Using the ceiling allows for columns that areonly partially full, but which still take up hori-zontal space, as in Figure 8.

2. Because an uneven number of thumbnailsexist and some regions might contain emptyspace, others might contain overflow.Determine the region that contains thelargest number of overflowing thumbnailsand keep reducing the thumbnail width forall regions by 1 pixel until there is no moreoverflow.

3. Redistribute whatever extra space is at theend (shaded in Figure 8) among all of theregions in the quadrant.

4. Using the current thumbnail size, repeat forthe other quadrants.

5. When all quadrants have been processed, thelast thumbnail size will be the correct one.

It might be possible to do Step 2 analytically,although for the pixel width and height in thiscontext, this progressive technique is a reason-able approach.

The ReviseQuantumDim algorithm starts withthe upper bound on the thumbnail width, andreduces by one each time until it reaches either adefined minimum, or zero in the worst case. If itneeds to go until zero, the algorithm will run inO[maximum thumbnail width]. This factor will onlyget large in the case of a large display with few

thumbnails, and even in that case it will likely ter-minate before the thumbnail width reaches zero.

Edge layouts. The algorithms work well for acentered rectangle. However, if the primaryregion is moved or sized so that a quadrant does-n’t have room for any regions, edge layouts occurin which the primary region is aligned with oneor more edges of the layout, as in Figure 9. Tocorrectly handle these layouts, the RegionSplitalgorithm must be modified. To do this, the sys-tem needs to determine the number of nonemp-ty quadrants and proceed as follows:

1. If there are four, proceed to the normalRegionSplit.

2. If there are three, proceed to TriRegionSplit.

3. If there are two, perform RegionSplit onlyonce, to yield two regions.

4. If there is one, allocate all regions to thatquadrant.

5. If there are none, do not show any regions.

Each of these layout algorithms runs in time thatis linear with the number of regions, as describedin the initial “RegionSplit” section.

TriRegionSplit. This algorithm is a specialversion of RegionSplit to handle division of a col-lection of n discrete-sized pieces as evenly as pos-sible in thirds (see Figure 10). While it’s

Figure 8. Secondary

regions with free space.

Quad 2 Quad 2Q

uad 4

Quad 1

Quad 3 Quad 3

Figure 9. Edge layouts,

in which some

quadrants are empty.

conceptually similar to RegionSplit, it’s worthdiscussing separately, because it might not beimmediately obvious how to make a bifurcatingalgorithm generate three regions:

1. Divide the n regions into two, as described inthe first step of the RegionSplit algorithm. Atthis point, there are two regions closest to theRegionSplit split point; one to the left andone to the right, at distances a and b, wherethe distance is the number of thumbnails inthe adjacent region.

2. Move one of the regions adjacent to theRegionSplit split point to a third pile, suchthat after the move, the average distance fromeach side to the absolute center n/2 is as closeto equal as possible. In Figure 10, b would beadded.

3. Repeat Step 2 until adding a region causes |n/3 −new pile’s area| to be more than it was beforethat iteration of step 2. When this happens,backtrack one and save the result as the answer.

PostprocessingOnce the algorithm has run, certain actions

can incrementally improve the automaticallygenerated layout. We describe two here. In gen-eral, these actions can be suggested by going overthe requirements of the “Problem definitions andrequirements” section, and violating one of themat this later stage, whereas violating them duringthe initial layout would cause undesirable feed-back and backtracking. For example, violatingthe fixed primary region size and locationrequirement suggests resizing or repositioningthe primary region to get a “better” layoutaccording to some metric.

Vary thumbnail size. Once the initial algo-rithm has run, it’s possible to increment thethumbnail size in each region until any furtherincrease would cause overflow of the region. Thisallows each region to have a minimum of wast-ed space, at the expense of the photos in differ-ent regions no longer lining up. This must bedone as a postprocessing step to avoid violatingthe uniform thumbnail size requirement.

Add scrollbar. For the layouts to scale proper-ly, a scrollbar can appear in a region if that regionhas substantially more thumbnails (currently setat 20x) than the smallest region, or than any

other region, depending on the user’s preference.Adding the scrollbar involves deciding how manythumbnails to show (some maximum per region),and then adding to that number on a per-regionbasis to enforce a full grid when not all photos arevisible. For example, if a maximum of 40 thumb-nails out of a region’s 70 are shown, and there areseven columns, the fifth row will have only fivethumbnails. In this case, two thumbnails shouldbe added to the last row so that the grid will befull unless the scrollbar is at the last position.These must be added as a postprocessing step sothat the fixed number of thumbnails requirementisn’t violated during the initial layout.

Dynamic behaviorThe algorithms described previously allow for

interactive, dynamic behavior that encouragesexperimenting with primary rectangle place-ment, size, and overall dimensions. This goal ledus to develop computed layouts, avoiding thehill-climbing or backtracking strategies that areoften used in constraint satisfaction problems.The typical behavior we wished to support is toallow users to move the primary rectangle fromthe upper-left corner into the center, and thenenlarge it to highlight its contents (see Figure 11,next page). Other behaviors include users resiz-ing the canvas and adding and deleting regionsas well as thumbnails.

PerformanceTo confirm the dynamic behavior described

for these algorithms, we conducted three trials,in which regions were incrementally added with10, 50, and 100 thumbnails per region, untilthere were 100 regions added. Execution timewas measured for the three algorithms, demon-strating that the algorithms were rapid enough,with no unforeseen explosions even at the highend of 100 thumbnails in each of the 100regions, for a total of 10,000 quanta being posi-tioned. For this test case, the RegionSplit,InitialQuantumDim, and ReviseQuantumDimalgorithms took 0.037 millisecond (ms), 0.014ms, and 0.14 ms, respectively, on a Pentium IV

69

Octo

ber–D

ecemb

er 2006

Regionsplit

Split pointn/2

ba

Figure 10. Dividing

secondary regions

among three

quadrants.

2.4-gigahertz processor with 1 Gbyte of RAM.Additional tests confirmed the linear growth oftime with the number of thumbnails.

Drawing the photos was more time consum-ing, exerting as much as three orders of magni-tude (350 ms) greater drag on performance thanany of the resizing algorithms. With better imagestorage allocation, we can solve these photo dis-play problems, independent of the algorithmicperformance outlined in this article.

Optimality discussionIt’s hard to quantify what makes one photo

layout better than another, as subjective mea-sures could vary in different applications and fordifferent users. For our purposes, however, we usethe following definition to try to describe anoptimal layout:

A layout is said to be optimal if there is nolayout that results in larger thumbnails.

Because any available space should be used forincreasing thumbnail size, this is equivalent to asecond definition:

A layout is said to be optimal if there’s a min-imum of unused space.

Using these definitions, the algorithmsdescribed here produce optimal layouts given therequirements. The relaxation of one or more ofthese requirements could provide a layout thatbetter meets these optimality criteria, but at theexpense of ease of scanning, consistent regionplacement, equal thumbnail prominence, andother problems. While some of the followingobservations are reasonably intuitive, they’reworth noting in the interest of thoroughness.

Distribution of regions into quadrantsRegionSplit divides the regions among the

quadrants as evenly as possible. Imbalances tendto occur for a thumbnail distribution, which isheavy on one side (for example: five regions,where regions 1 through 4 have five thumbnailseach and region 5 has 50). In this case, RegionSplitwants to put regions 1 through 4 in quadrants 1and 2, but then region 5 would need to get placedin quadrants 3 and 4, violating one of our require-ments and causing the undesirable L-shapedregion. So in this example, regions 4 and 5 wouldgo into quadrants 3 and 4, causing a reduction inthumbnail size.

Upper bound on thumbnail sizeThe InitialQuantumDim algorithm calculates

a value for thumbnail size using height, width,and number of thumbnails, ignoring row versuscolumn layouts and uneven distribution amongregions. To show that this is an upper bound,consider w′, a proposed thumbnail width that iswider than w, the value claimed to be maximal.(Since the aspect ratio is fixed, this also impliesan h′ > h.) Therefore, w′ × h′ × (number of thumb-nails) > clientAreai, which means that the thumb-nails take up more absolute area than is available.

70

Figure 11. Dynamic

behavior of BRQ layout

showing user control

over dragging and

sizing the primary

region.

Original layout

After moving the primary region

After resizing the primary image

Thus, the values w and h are upper bounds fromwhich ReviseQuantumDim can confidentlyrevise only downward.

Final thumbnail sizeThe ReviseQuantumDim algorithm will

always result in at least one quadrant containingall of its rows (quadrants 2, 4) or columns (quad-rants 1, 3) having at least one thumbnail, thustightly fitting the set of thumbnails to the quad-rant’s client area. As a result, the other quadrantsby definition have a minimum of unused spaceand the largest possible thumbnails, since if thethumbnails were made any larger they wouldoverflow the tightest-fitting quadrant (because allthumbnails are required to be the same size).

Improvements by relaxing requirementsThere are several ways in which users might

want to improve the quality of the generated lay-out by manually adjusting some of the require-ments. Users could choose, for example, toincrease the primary rectangle’s dimensions, orchange its position, if an initial layout showedextra space. A manual change in ordering theregions could also result in a better layout. The“same-size thumbnails” requirement might berelaxed, with thumbnails growing to differentsizes in their various regions until there is nowasted space, at the expense of having different-sized thumbnails in each region.

We should view these user actions as occur-ring at the application level: The algorithms dis-cussed here provide a starting point given certainrequirements, and if those are modified, they willagain generate a best layout based on the newrequirements. Any interaction between the appli-cation level and the algorithm layer causes feed-back loops that make deterministic solutionsimpossible.

If the user desires a better solution, at theexpense of the feedback loops previouslydescribed, an approach would be to generate a setof thumbnail dimensions for points about thecurrent solution, for a given requirement. Forexample, once the algorithm finishes, it couldgenerate what-if scenarios, varying the main rec-tangle’s size or its position by up to some thresh-old. Depending on how many requirements wereviolated and to what extent, some number ofhypothetical dimensions could be generated.Because the calculations are relatively trivial,many of these could be generated in a short peri-od of time, and by looking at this space, a better

solution could be found. This type of operationshould only be allowed at the user’s explicitrequest, as someone trying to exactly locate theprimary rectangle could be frustrated by an auto-mated agent “improving” the position to enlargethe thumbnails by changing the position thatthe user wants to obtain.

Usability studyTo gauge user responses to these BRQ layouts

we asked four knowledgeable users of photolibrary software to review our interface for 30 to40 minutes.3 In this modest usability study, theywere shown the on-screen, but static, layouts inFigures 1 through 4 in order and asked what theyunderstood about the layout and relationshipamong the regions in each figure. The usersunderstood why a particular thumbnail size waschosen, although sometimes only after carefullycomparing regions to find a constrained one.Several voiced interest in a feature that wouldrelax the same thumbnail size requirement tohave less wasted space. They all appreciated theability to manipulate the layout in real time.

In response to the user feedback, we added anoption to dynamically change between constantthumbnail size and allowing thumbnails to fillthe available space. After that addition, userstended to choose the variable size to minimizewasted space. There were also several requests toremove photos from the layout and to choose analready-visible photo for the primary region.These were implemented with additional contextmenu items to allow a more intuitive directmanipulation of the final layout. One user com-mented that to truly generate a polished, print-able finished product, full control would have tobe given to the user in terms of relative position-ing of text, line weights, choice of photos withina collection, and so on. While this wouldundoubtedly increase the tool’s value, imple-menting these features was beyond the scope ofthe prototype, which was designed primarily todemonstrate the algorithms and show the poten-tial for this type of layout.

ConclusionMany applications, such as digital photo lay-

outs, could use our algorithms to do dynamic lay-out quickly and deterministically. A vital aspectof the BRQ layout algorithm is its rapid perfor-mance, which enables compelling interactiveexperiences as users resize the canvas, add ordelete regions, and add or delete items to a region.

71

Octo

ber–D

ecemb

er 2006

Our future work includes investigating therequirements we’ve described, with an eye toremoving some of them, such as the fixed sizeand location of the primary region, or the quan-tum size and aspect ratio. A further challenge isto extend these ideas to a three-level hierarchicallayout, which presents additional difficulties. Athree-level layout of digital photos could be use-ful to show grandparents, parents, and grand-children in one large ensemble, or anorganizational chart with the CEO, then vicepresidents, and finally senior managers in con-centric rectangles. MM

AcknowledgmentsWe would like to thank Adobe Corporation

for its support, which has made this research pos-sible. We would also like to thank Ben Bederson,Alexander Loui, and Martin Wattenberg for theirvaluable comments; Jonathan Katz for his sug-gestions regarding the thumbnail sizing algo-rithms; and our reviewers for their detailed andthoughtful comments.

References1. C. Jacobs et al., “Adaptive Grid-Based Document

Layout,” ACM Trans. Graphics, vol. 22, no. 3, 2003,

pp. 838-847.

2. J. Kustanowitz and B. Shneiderman,“Meaningful

Presentations of Photo Libraries: Rationale and

Applications of Bilevel Radial Quantum Layouts,”

Proc. ACM/IEEE Joint Conf. Digital Libraries, ACM

Press, 2005, pp. 188-196.

3. S. Lok and S. Feiner, “A Survey of Automated

Layout Techniques for Information Presentations,”

Proc. SmartGraphics Symp., ACM Press, 2001, pp.

61-68; http://www.smartgraphics.org/sg01/.

4. S. Lok, S. Feiner, and G. Ngai, “Evaluation of Visual

Balance for Automated Layout,” Proc. 9th Int’l Conf.

Intelligent User Interfaces, ACM Press, 2004, pp.

101-108.

5. K. Rodden and W. Basalaj, “Does Organization by

Similarity Assist Image Browsing?” Proc. Special

Interest Group on Human–Computer Interaction

(SIGCHI 01), ACM Press, 2001, pp. 190-197.

6. B. Bederson, B. Shneiderman, and M. Wattenberg,

“Ordered and Quantum Treemaps: Making

Effective Use of 2D Space to Display Hierarchies,”

ACM Trans. Graphics, vol. 21, no. 4, 2002, pp. 833-

854.

Jack Kustanowitz is currently

the director of infrastructure at

the HealthCentral Network. His

research interests include human–

computer interaction, applica-

tions of annotated media,

knowledge management sys-

tems, Internet search, and online communities.

Kustanowitz has an MS in computer science from the

University of Maryland at College Park, where he stud-

ied interactions between individuals and their growing

collections of personal digital photos.

Ben Shneiderman is a professor

in the Department of Computer

Science, founding director of the

Human–Computer Interaction

Laboratory, and member of the

Institute for Advanced Computer

Studies and the Institute for

Systems Research, all at the University of Maryland at

College Park. He’s the author of Leonardo’s Laptop:

Human Needs and the New Computing Technologies.

Readers may contact Jack Kustanowitz at jkustan@

umd.edu and Ben Shneiderman at ben@cs.umd.edu.

72

computer.org/e-News

Available for FREEto members.

Be alerted to

•articles and special issues

•conference news

•registrationdeadlines

Sign Up Today for the IEEE

Computer Society’s

e-News

Sign Up Today for the IEEE

Computer Society’s

e-News