In: Perception, 2001.

The Perception of Texture on Folded Surfaces

MARY J. BRAVO 1, HANY FARID 2

Do judgments of texture similarity reflect surface texture or image texture? To find out,we had observers view a rectangular surface that was folded into three panels much likea brochure. Each panel was textured with an oriented noise pattern and the observers’ taskwas to determine which side panel matched the center panel in surface texture. Informationabout surface geometry was conveyed by binocular disparity and by the boundaries of therectangular surface. We found that observers were often consistently wrong, selecting thetexture that differed in the image and not on the surface. In sharp contrast, when observersjudged the texture orientation on each panel individually, their judgments were accurate re-flections of the surface texture. So even when observers can recover surface texture, theirjudgments of texture similarity may still be based on image texture.

Texture Similarity Grouping Perceptual Constancy

1Department of Psychology, Rutgers University, Camden NJ 08102.2Department of Computer Science, Dartmouth College, Hanover NH 03755.

1

1 Introduction

In his laws of perceptual organization, Wertheimerincluded the tendency for similar stimulus elementsto group together [22]. This law of similarity group-ing has intrigued many vision scientists becausethe idea seems so simple, and yet a precise formu-lation has proven elusive.

The basic phenomenon is shown in Figure 1a.Each side of this stimulus is textured with a pat-tern of repeating lines. The lines differ in orien-tation, and observers report seeing them as form-ing two distinct groups separated by a perceptualboundary. In other words, the textures appear tosegment. Early experiments [1, 15] showed thatdifferences in orientation are especially potent forproducing this effect. But textures that differ inother ways may also segment; for example, pat-terns of “+”s and “L”s segment even though theyhave the same component orientations.

Early attempts to explain texture segmentationwere couched in terms of abstract features [10, 3,20]. Thus, the segmentation of “+”s and “L”s wasattributed to a difference in a line-crossing feature.However, a simpler explanation has since been pro-posed: the spatial frequency filters that are thoughtto be involved in early vision respond differentlyto these textures. This realization led to compu-tational models of texture segmentation that werebased on well-established models of threshold vi-sion. According to these models, textures are ana-lyzed by arrays of spatial frequency filters that arespatially local and selective for orientation. Theoutputs of these filter arrays are then analyzed bya second set of filters which detects spatial discon-tinuities [8, 6, 21, 5, 19, 11, 13].

These models signify an important advance inour understanding of texture segmentation becausethey are elegant and rigorous and yet still havegreat explanatory power. However, they cannotexplain all aspects of texture segmentation. Rocket al. used cast shadows to change the retinal lu-minance, but not the perceived reflectance, of theirtexture stimuli [18]. Contrary to the predictions offilter-based models, they found that grouping wasdetermined by surface reflectance. These authors

(d)(c)

(a) (b)

Figure 1: When a surface is folded, dissimilarsurface textures (a) may appear similar in theimage (b), and similar surface textures (c) mayappear dissimilar in the image (d).

concluded that similarity grouping is not a prod-uct of early vision, but instead involves a surfacerepresentation. Similarly, He and Nakayama ex-ploited the phenomenon of amodal surface com-pletion to alter the perceived, but not the retinal,shape of their texture stimuli. Their stimulus ma-nipulation should not have affected filter activity,and yet it had a significant effect on grouping ([9],see also, [16]). Finally, Palmer and Nelson usedillusory contours to dissociate the retinal and theperceived structure of their stimuli, and they toofound that grouping was determined by the per-ceived surface structure [17]. Thus, a number ofresearchers have provided evidence that filter-basedmodels alone cannot account for human texturesegmentation. They propose that at least some tex-ture segmentation processes act on a surface rep-resentation in which such properties as reflectanceand occlusion are made explicit.

Given the possibility that texture segmentationis based on a surface representation, it seems ap-propriate to reconsider even simple texture stimulisuch as that shown in Figure 1a. What, exactly, areobservers grouping in this display? Is groupingbased on the retinal image textures, as the compu-tational models suggest, or is it based on surfacetexture as recent findings suggest? With a fronto-

2

parallel surface it is impossible to differentiate be-tween these alternatives because the textures onthe stimulus surface correspond closely to the tex-tures in the retinal image. But the similarity of sur-face and image textures can be dissociated by fold-ing this stimulus and viewing it obliquely. Dis-similar surface textures (Figure 1a) may then pro-duce similar image textures (Figure 1c). And con-versely, similar surface textures (Figure 1b) mayproduce dissimilar image textures (Figure 1d).

In this study, we used a folded stimulus to de-termine whether judgments of texture similarityare based on image texture or on surface texture.The stimuli were designed such that judgmentsbased on image texture would produce qualita-tively different from results judgments based onsurface texture.

2 Methods

Our stimulus consisted of a computer generatedsurface that was folded into three panels much likea brochure, Figure 2(c). An oriented texture wasmapped onto each panel and the observers’ taskwas to tell us which of the side panels most closelyresembled the center panel in surface texture. Intwo control conditions, observers performed thesame task on modified versions of the stimulus.The basic stimulus and the experimental and con-trol conditions are described below. First, how-ever, we discuss our choice of method.

As reviewed in [4], researchers have used a va-riety of approaches to study texture segmentation.The most stringent method requires observers tomake judgments about the shape of a texture de-fined region. We could not use this method herebecause, in our stimuli, the texture discontinuitynecessarily coincides with highly salient depth dis-continuity. Another approach to studying textureperception involves presenting observers with pairsof textures and asking them to judge the similar-ity (or conversely, the discriminability) of the tex-tures. We are essentially using a two-alternativeforced choice variant of this approach. Althoughthis second measure does not directly assess group-

ing, it is thought to be closely correlated with group-ing. That is, textures that are judged to be similarwill group together, and textures that are judgedto be dissimilar will segment [3]. For this relation-ship to hold, however, it is essential that the simi-larity judgments be based on an impression of theglobal texture pattern and not on scrutiny of theindividual elements. To discourage such scrutinywe used continuous, stochastic textures in whichlocal measurements of orientation were less reli-able than global measurements. So although thisexperiment measures perceived similarity, we be-lieve the results bear on similarity grouping as well.

2.1 Stimuli

The stimuli were generated in OpenGL on an SGI02. Perspective projections of the right and lefteye views of this stimulus were presented on alter-nating frames of the computer display. Observersviewed the stimuli with liquid crystal goggles (Crys-talEyes by Stereographics) which synchronized theview of each eye with the alternating frames, yield-ing a stereoscopic display. A headrest was used tomaintain the viewing distance of 70 cm.

The rendered stimulus consisted of three panelspresented side-by-side, Figure 2. The two outerpanels were 10cm squares, and the center panelwas a 5cm wide and 10cm tall rectangle. Eachpanel was texture-mapped with white noise thathad been convolved with an oriented band-passfilter. Because the filter was band-pass, contrastwas spread across a small range of orientations.When considered globally, however, each texturehad a clear dominant orientation. Fifteen randomtexture samples were generated for each of the tex-ture orientations used in the experiment.

A −30, 0 or 30 degree texture was mapped ontothe center panel of each stimulus. A different sam-ple of the same texture orientation was mappedonto one of the side panels, Figure 2a. Note thatthe textures on these two panels were not contin-uous; there was always a seam between the pan-els. 3 The other side panel was mapped with a tex-

3This seam is a necessary artifact of our method of stimu-lus generation. One way to create seamless stimuli is to use

3

(a) fronto-parallel

(b) flat

(c) folded

(d) image

Figure 2: (a) The basic stimulus. Here, as in b-d,the right and center panels have the same sur-face texture orientation. Thus the right panel isthe target, and the left panel is the non-target.(b) The flat condition: the panels have been ro-tated about the horizontal axis by 35 degrees.(c) The folded condition: same as b, but the sidepanels have been folded forward. (d) The im-age condition: same as c, but viewed behindan occluder. Conditions b and c were viewedstereoscopically.

ture orientation that differed from the center tex-ture by 10, 20, 30 or 40 degrees. The observers’ taskwas to select the side panel that had the same tex-ture as the center panel. We will refer to this sidepanel as the “target” and to the dissimilar panel asthe “non-target”.

2.1.1 The folded condition

The three panels were rotated by 35 degrees aboutthe horizontal axis so that the top of the stimulusappeared closer to the viewer. The left and rightside-panels were then rotated forward by 50 de-grees, Figure 2c. Under these viewing conditions,the similarity of the image textures can be readilydissociated from the similarity of the surface tex-tures.

To see how folding the stimulus dissociates theimage and surface textures, compare the fronto-parallel and folded stimuli depicted in Figure 2 (aand c). In the fronto-parallel stimulus, the rightand center textures have a 30 degree orientation,while the left texture is −10 degrees. When thissame stimulus is folded, the left and center imagetextures now appear more similar, while the rightimage texture differs. Note that if the surface tex-ture on the left panel had differed by +40 ratherthan −40 degrees (i.e., if we had used a 70 de-gree rather than a−10 degree texture), then the leftpanel would differ from the center panel in bothsurface and image texture. Since our goal was todetermine whether similarity judgments are basedon surface texture or on image texture, we alwaysselected non-target textures that would dissociatethe two. Thus, whenever the left panel was thenon-target, we rotated its texture in a clockwisedirection to compensate for the counterclockwiserotation caused by the image projection. And con-versely, whenever the right panel was the non-target,we rotated its texture in the counterclockwise di-rection.

The plot in Figure 3 shows, for each of the three

discrete textures and position the fold between columns oftexture elements as in Figure 1. We made a few such seam-less stimuli out of cardboard and observed the same percep-tual effects reported here.

4

panels, the relationship between the texture orien-tation on the surface and the texture orientationin the image. 4 The filled circles correspond to a30 degree texture mapped onto each of the pan-els. The open circles correspond to the non-targettextures used in the experiment. Thus if the tar-get was on the right, the non-targets would cor-respond to the four open circles on the left panel.Note that on the left panel, the more the non-targetsdiffered from the center in surface texture, the morethey resembled it in image texture.

2.1.2 The flat condition

In the flat control condition, the three panels wererotated by 35 degrees, but the side panels werenot folded forward (Figure 2b). Because the pan-els were coplanar, the similarity of the textures inthe image closely corresponded to the similarity ofthe textures on the surface. As such, this stimulusresembles stimuli used in a standard texture dis-crimination experiments. It also provides a roughestimate of how observers would perform in thefolded condition if they based their judgments onsurface texture. 5

2.1.3 The image condition

In this condition, the folded stimulus described abovewas presented behind a cardboard mask which ob-scured the edges of the panels (Figure 2d). Thevertical extent of the stimulus was made larger forthis control condition so that the visible stimulusarea would be unchanged. The stimulus was alsopresented non-stereoscopically. Without depth cues,the panels appeared coplanar and fronto-parallel.

4Note that Figure 3 shows only part of the texture trans-formation that is caused by the image projection. In additionto changing the texture orientation, the perspective projec-tion also changes spatial frequency. So while this graph givessome indication of how observers might perform if they basetheir judgments on image texture, the best indicator of this isthe direct empirical measure described in Section 2.1.3.

5This is only a rough estimate because it does not take intoaccount any biases or imprecision in the observers’ assess-ment of surface geometry. Nor does it take into account thenonlinearity of the curves in Figure 3 and the effect this non-linearity would have on texture discrimination thresholds.

−90 −60 −30 0 30 60 90−90

−60

−30

0

30

60

90

right panel

left panel

orientation (surface)

orie

ntat

ion

(imag

e)

Figure 3: The orientation transformationcaused by image projection. On the horizon-tal axis is the orientation of the surface texture,on the vertical axis is the orientation of the im-age texture. The three curves correspond to thethree stimulus panels in the folded condition(Figure 2c).

This image control condition shows how observerswould perform in the folded condition if they basedtheir judgments on image texture.

2.2 Procedure

We used a physical model of the stimulus to ex-plain the task to the observers. The side panels ofthe model could be adjusted so that they were ei-ther flat (coplanar with the center panel) or folded.The panels of the model had plastic covers for mount-ing pieces of cardboard that had been printed withtexture patterns. To explain the task, we first mountedseveral texture triplets (e.g., 0, 0, 30 and 0, 30, 30)on the flat model. We then asked the observers toindicate which side panel had the same texture ori-entation as the center panel. Next, we folded themodel, mounted new textures, and again askedthe observers to judge the similarity of the textures.To ensure that the observers understood that wewere asking about the textures on the surface and

5

not in the retinal image, we explained that their re-sponse should be the same whether the model wasfolded or unfolded.

This same model was then used to asses howaccurately the observers perceived the geometryof the folded surface. Each observer adjusted thephysical model to match the folded surface dis-played on the computer screen. This task was re-peated three times and after each time we mea-sured the angles between the center and side pan-els and between the center panel and vertical. Over-all, there was a slight tendency for observers tounderestimate both angles, but their average judg-ments were accurate within 6 degrees of rotation.

After these preliminaries, observers ran the ex-periment. The flat, folded and image conditionswere run in 30 separate but interleaved blocks of96 trials each. Within a block, the different tex-ture orientations were randomly intermixed. Eachstimulus was displayed for two seconds. The ob-servers responded by pressing a key on the com-puter keyboard and the next stimulus was presentedafter a two second delay. No feedback was given.Each observer ran five, one-hour sessions over aperiod of 2− 3 weeks.

2.3 Observers

Three undergraduates from Rutgers University (fe-males, ages 18 − 38) were paid to participate inthe study. The observers had never before partic-ipated in a psychophysical experiment and wereunaware of the purpose of the study.

3 Results and Discussion

The observers’ task was to determine which of theside-panels matched the center panel in surface tex-ture. The texture orientation on the center panelwas either horizontal or diagonal. Because theseorientations produced very different results, we con-sider them separately.

The results for the horizontal textures are plot-ted in Figure 4 which shows the individual andaverage data for the three observers. The verti-cal axis indicates the percentage of trials in which

s1 s2

0

25

50

75

100

perc

ent c

orre

ct

s3 average

10 20 30 400

25

50

75

100

orientation

perc

ent c

orre

ct

10 20 30 40orientation

Figure 4: Results for stimuli that had a horizon-tal texture (0 degrees) on the center and targetpanels. The x-axis indicates the texture orienta-tion on the non-target panel . The y-axis showsthe percentage of times the observer selectedthe target as similar to the center. Triangles cor-respond to the flat condition, filled circles to thefolded condition and open circles to the imagecondition.

observers correctly selected the target panel as be-ing similar to the center panel. The horizontal axisshows the texture orientation of the dissimilar, non-target panel. Each point is the average of 80 trialsfrom two mirror symmetric stimuli. Thus, the datapoints for the 40 degree orientation reflect the av-erage results for the 0, 0, 40 and −40, 0, 0 stimuli.

The triangles in Figure 4 correspond to the flatcondition in which the three panels of the stimu-lus were coplanar. As expected, the results resem-ble those of a typical texture discrimination exper-iment. Performance improves as the orientationdifference between the non-target and center pan-els increases, reaching an asymptote by 30 degrees.The filled circles show the data for the same stim-ulus but with the side panels folded forward. Inthis condition, the observers must factor-in sur-face geometry to judge the similarity of the sur-

6

Figure 5: A stimulus with 0 degree surface tex-ture on the center and right panels and −30 de-gree texture on the left panel. Note that the hor-izontal surface textures run parallel to the panelboundaries.

face textures. The open circles show the data forthe image condition. These data indicate how ob-servers would perform in the folded condition ifthey based their judgments solely on image tex-ture. As expected, performance here is consistentlybelow chance. This is because, in the image, thetexture of the center panel is more similar to thenon-target panel than it is to the target panel.

For our purposes, the important feature of thesedata is the separation between the image and foldedconditions (i.e., the open and filled circles). Thedisplacement of the folded data away from the im-age data and towards the flat data shows that ob-servers did not rely solely on image texture in mak-ing their similarity judgments. Instead, they basedtheir judgments on surface texture.

Although the results suggest that texture dis-crimination involves a 3D surface representation,these results could instead reflect a purely image-based process. This process cannot be based di-rectly on image texture, of course, but it could bebased on the parallelism between the image tex-ture and the surface edge, since this is invariant toimage projection, Figure 5. Because observers mayhave used the image cue of parallelism to deter-mine which textures were horizontal, these find-ings may not generalize to other texture orienta-tions.

Figure 6 shows the data for the diagonal tex-tures. Again, the vertical axis shows the percent-age of trials in which the observers correctly judged

the target texture as similar to the center texture.The horizontal axis corresponds to the orientationof the non-target texture. Also, as in Figure 4, wehave collapsed the data for mirror symmetric stim-uli. This time, however, the symmetry is betweenstimuli with different textures on the center pane.So, for example, 0, 30, 30 is not the mirror imageof 30, 30, 0, but it is the mirror image of −30, −30,0. Thus, in graphing the results for diagonal tex-tures, the data for 30 and −30 degree stimuli werecombined. The labels on the graph correspond tothe 30 degree stimuli.

The open triangles correspond to the flat con-dition. As before, these data look fairly typicalfor a texture discrimination experiment: perfor-mance increases as the orientation difference be-tween the non-target and center increases up toabout 30 degrees. The filled circles correspond tothe data for the folded condition, while the opencircles show the data for the image condition. Un-like the data in Figure 4, here the folded and imagedata are very similar. Thus instead of selecting thetarget, which resembled the center panel in sur-face texture, observers were consistently selectingthe non-target, which resembled the center in im-age texture. This effect is especially striking on theleft panel. Here, the dissociation between the sim-ilarity of image and surface textures is especiallylarge: the more the textures differ on the surface,the more similar they are in the image. As a result,observers were remarkably bad at this task whenthe left panel was the non-target.

To summarize the results for this first experi-ment, when the center texture was horizontal, sim-ilarity judgments accurately reflected surface tex-ture. In contrast, when the center texture was di-agonal, these judgments were consistently wrong;instead of judging surface texture, observers judgedimage texture. How can we account for this dis-crepancy between the results for horizontal anddiagonal textures?

As noted earlier, the parallelism between the hor-izontal textures and the surface edges is preservedin the image projection, and so our observers couldhave used this image cue to determine which tex-

7

-10

30

45

left right

0

25

50

75

100

perc

ent c

orre

ct

s1

0

25

50

75

100

perc

ent c

orre

ct

s2

0

25

50

75

100

perc

ent c

orre

ct

s3

−10 0 10 200

25

50

75

100

orientation

perc

ent c

orre

ct

40 50 60 70orientation

avg.

Figure 6: Results for stimuli that had a diagonal tex-ture (30 degrees) on the center and target panels. Thex-axis shows the texture orientation on the non-targetpanel. (These stimuli do not have the same bilateralsymmetry as those in Figure 4, and so left and rightpanels are plotted separately.) The y-axis shows thepercentage of times the observers selected the targetas similar to the center. Triangles correspond to theflat condition, filled circles to the folded conditionand open circles to the image condition. The dashedlines indicate the non-target textures that matchedthe center texture in the image. These textures arealso depicted in the schematic at top.

tures were horizontal on the surface. 6 Addition-ally, horizontal is the most common texture ori-entation for man-made surfaces (consider bricks,tiles and aluminum siding), and so it is likely thatour observers had more experience with these tex-tures. And, as discussed below, there is recent evi-dence that observers can use surface texture to re-cover surface shape only for textures that are ori-ented perpendicular to the lines of maximal cur-vature on a surface (e.g., our horizontal textures)[12]. For all these reasons, the horizontal texturesmay be a special case.

Even if we can account for the superior perfor-mance for horizontal textures, we are still left withthe surprisingly poor performance for diagonal tex-tures. If observers base their similarity judgmentson image texture, then these results reveal an un-usually pronounced failure of perceptual constancy.In the next experiment we look more closely at thisfailure of perceptual constancy.

4 Experiment 2

In the previous experiment, observers judged tex-ture similarity on a folded surface. When the tex-tures ran diagonally along the surface, their judg-ments were dominated by image texture. Two con-clusions can be drawn from this result. The first isthat observers do not have orientation constancy. 7

That is, lines on a surface may appear to run oneway from one vantage point, and a different wayfrom a different vantage point. Alternatively, ob-servers may have orientation constancy, but theymay not use it when making similarity judgments.Or, in other words, orientation constancy may becalculated after, or in parallel with, similarity judg-ments. To distinguish between these alternatives,we measured orientation constancy directly by ask-ing observers to make absolute judgments about

6But we should note that the data in Figure 6 showthat observers did not take advantage of this cue when thecenter texture was diagonal and the non-target panel washorizontal.

7We are using this term in a somewhat non-standard way.Traditionally, orientation constancy has referred to the stabil-ity of perceived shape under image rotation.

8

the surface texture on each of the three stimuluspanels.

4.1 Methods

Three new observers were recruited for this exper-iment. All three were Rutgers students who werepaid for their time and who were naive as to thepurpose of the study.

To measure orientation constancy, we first hadthe observers learn a standard texture orientationon the center panel. This center panel had the samedimensions as the center panel in the previous ex-periment, but, rather than being rotated about thehorizontal axis, it was fronto-parallel. Initially, ob-servers passively viewed 10 samples of a 30 or−30

texture mapped onto this panel. We then mea-sured the accuracy of the observers’ internal rep-resentation of the standard by having them per-form an orientation discrimination task. In fourblocks of 40 trials each, the observers were showntextures that deviated from the standard orienta-tion by ± 10, 20, 30 or 40 degrees. Their task wasto judge whether these deviations were in a clock-wise or counterclockwise direction. Each stimu-lus was presented for 1 second. The observer re-sponded by pressing a key on the keyboard, feed-back was provided, and the next stimulus was pre-sented 2 seconds following a 2 second delay.

After discriminating textures on the center panel,the observers repeated the task for the same tex-tures mapped onto the side panels. These sidepanels were identical to those in the previous ex-periment, and so were rotated about both the hor-izontal and vertical axes and offset from the mid-line (left and right panels of Figure 2c). Only onepanel was shown at a time, and the observers’ taskwas to judge whether the texture on the panel wasrotated clockwise or counterclockwise relative tothe standard. Trials involving the right panel wererandomly intermixed with trials involving the leftpanel. In all, eight blocks of 45 trials each were runfor this one-panel, in-depth condition.

Interleaved with blocks of the one-panel, in-depthcondition were blocks of a one-panel, no-depth con-dition. In this condition, observers judged the tex-

tures on the side panels in the absence of depth in-formation. The stimuli were presented non- stereo-scopically behind an aperture which obscured theedges of the panel.

Each observer ran the constancy experiment twice,once with the 30 degree texture as the standardand a second time with the −30 degree texture asthe standard. After completing both texture con-stancy experiments, the same observers repeatedthe texture similarity experiment described in Sec-tion 2.

4.2 Results

The results of the orientation constancy experimentfor three observers are shown in Figure 7. The ob-servers’ task was to judge whether a texture pat-tern was rotated clockwise or counterclockwise rel-ative to a learned standard. The standards were30 and −30 degrees, and because these stimuli aremirror symmetric, we have averaged the data fromthe two conditions. As before, the labels on thegraphs correspond to the 30 degree texture. Thegraph shows the percentage of counterclockwiseresponses plotted against the texture orientationfor the left and right panels. Recall that observersfirst performed this task, with feedback, on the fronto-parallel center panel. These data are plotted asopen triangles in the left and right graphs. Thefilled circles show the data for the in-depth condi-tion in which the observers repeated the task onthe slanted side panels. The observers were givena full stereoscopic view of the panels and so couldpotentially have used information about surfacegeometry to recover the surface texture. Finally,the open circles show the data for the no-depthcondition in which observers were forced to basetheir judgments on the image texture of the slantedpanels.

If, as in the previous experiment, observers basetheir orientation judgments on image texture, thenthe data for the in-depth condition (filled circles)should fall near the data for the no-depth condi-tion (open circles). Although this trend is seen inone observer’s data for one panel (s2, right panel),this is clearly not the general pattern. In most cases,

9

-10 30 45

left right

0

25

50

75

100

perc

ent C

CW

s1

0

25

50

75

100

perc

ent C

CW

s2

0

25

50

75

100

perc

ent C

CW s3

−10 10 30 50 700

25

50

75

100

orientation

perc

ent C

CW

−10 10 30 50 70orientation

avg.

Figure 7: Perceived texture orientation. The x-axis shows the texture orientation on the panel sur-face. The y-axis shows the percentage of times theobservers judged the texture to be rotated counter-clockwise from the 30 degree standard. Trianglescorrespond to the flat condition, filled circles to theone-panel, in-depth condition and open circles to theone-panel, no-depth. Dashed lines indicate whichsurface texture project to a 30 degree image texture.These texture orientations are also depicted in theschematic at top.

left right

−10 0 10 200

25

50

75

100

orientation

perc

ent c

orre

ct

40 50 60 70orientation

Figure 8: Diagonal texture: average data forthree observers. Triangles correspond to the flatcondition, filled circles to the folded conditionand open circles to the image condition.

the orientation judgments for the in-depth stimu-lus are quite similar to those for the center panel(open triangles). Thus these data show that the ob-servers can accurately judge surface texture.

The same three observers also ran in the texturesimilarity experiment described in Section 2. Theiraverage results are shown in Figure 8. Previouslywe had found that observers relied on image tex-ture in judging texture similarity, and this obser-vation holds here as well. Overall, the data for thefolded condition (filled circles) are similar to thoseof the image condition (open circles). These resultscontrast with those in (Figure 7) where the data ofthe one-panel in-depth condition, which is analo-gous to the folded condition, were more similar tothe flat condition.

We should note that the effect in this second tex-ture similarity experiment is not as strong as in theoriginal experiment (Figure 6). One possible ex-planation for this reduced effect is that during theorientation constancy experiment, the observers mayhave learned to associate the image textures withtheir corresponding surface textures. This associa-tion might also explain why the second set of ob-servers did noticeably better than the first set onthe image condition (open circles).

To summarize, observers demonstrated good con-stancy when making absolute judgments of tex-ture orientation. Thus they were able to judge whetherthe surface texture on a slanted panel was rotated

10

clockwise or counterclockwise relative to a learnedstandard. Because they learned the standard ori-entation on a fronto-parallel panel, the observersmust have compensated for the change in surfacegeometry when making these judgments. Nonethe-less, these same observers showed poor constancywhen making relative judgments of texture simi-larity. That is, when asked to compare the orienta-tions of several textures presented simultaneously,their performance was strongly biased by the im-age textures.

At first glance, it may seem that these results donot add up. If observers could accurately judgethe texture orientations on the left, right and centerpanels, then surely they should have been able todetermine which panels had the same orientation.Presumably, if we had instructed our observers toanalyze the stimulus one panel at a time, we wouldhave found this result. However, we instructedour observers to tell us which panels appeared tohave the same surface texture. Thus our resultslikely reflect not what observers can calculate, butwhat they perceive.

5 General Discussion

Our central finding is that observers fail to takesurface geometry information into account whenjudging texture similarity for diagonal textures. Thus,their similarity judgments reflect not the textureson the surface, but the textures in the surfaces’ pro-jected image. Although this finding suggests thatobservers cannot recover surface texture, a con-trol experiment showed that this is not the case.When presented with a single planar surface, ob-servers were able to accurately judge surface tex-ture even when the surface was rotated in depth.So although observers can recover surface texture,they do not use this information when judging tex-ture similarity across a folded surface.

This surprising conclusion has a parallel in re-cent work by Li and Ziadi [12]. While we wereinterested in the recovery of surface texture fromshape information, these authors were interestedin the recovery of surface shape from texture in-

formation. Li and Ziadi mapped oriented texturesonto a surface that was sinusoidally modulated indepth and then asked observers to make judgmentsabout the shape of this surface. These judgmentswere accurate only when the surface textures wereoriented along the lines of maximum and mini-mum curvature. When the textures ran diagonallyalong the surface, performance was poor. Thus,the visual system appears to be limited both inits ability to use image texture to recover surfaceshape and in its ability use surface shape to re-cover surface texture.

Our finding that the visual system might be un-able to use surface shape to recover surface textureis also related to work by Cavanagh et al. [7]. Theyreported a study and demonstration which showthat observers do not compensate for surface ge-ometry when interpreting surface markings. Theirdemonstration is easy to recreate and so we willdescribe it here. It requires folding a dollar billso that it appears as a “∨” when viewed from thetop and so that the crease vertically bisects GeorgeWashington’s face. When the folded bill is viewedstraight-on with the crease orthogonal to the view-ing direction, Washington appears expressionless.But when the bill is rotated about the horizontalaxis, Washington changes expression. Bringing thebottom of the bill closer makes him smile, bringingthe top closer makes him frown. When the top ofthe bill is closer, the retinal image of Washington’smouth is a “∧”, and observers attribute this “∧” toa frown and not to the fold. So just as in our exper-iment, the observers seem to ignore the fold wheninterpreting the markings on the surface.

Our study is also related to an experiment byBeck [2]. His stimulus was an array of intermixedvertical and diagonal lines drawn on cardboard,and his observers’ task was to rate the extent towhich the two orientations formed separate groups.When the stimulus was fronto-parallel, groupingwas near threshold and observers gave very lowratings. When the stimulus was slanted towardsthe ground plane, however, these ratings increasedconsiderably. Beck concluded that since the lineorientation on the surface did not change, observersmust base their judgments on image orientation.

11

However, as Beck himself acknowledged, there aredifficulties with this interpretation (see also [18]).The most fundamental difficulty is that surface ori-entation is calculated from image orientation. Thus,any change in the reliability of the image infor-mation will necessarily affect judgments of surfaceorientation. So his finding is consistent both withthe idea that similarity judgments are based on im-age orientation, and with the idea that these judg-ments are based on a property derived from imageorientation, namely, surface orientation.

This same objection does not apply to our ex-periment, however, because we approached thisquestion in a different way. Rather than presenttwo textures on a single planar surface, we pre-sented two textures on a folded surface and weasked whether observers take the fold into accountwhen judging texture similarity. If they do, theneven if two textures are similar in the image, theyshould not be judged as similar on the surface.But, of course, our experiment rests on the assump-tion that observers had sufficient depth informa-tion to recover the surface geometry. We think theydid because all observers gave accurate verbal de-scriptions of the folded stimulus. In addition, theobservers were able to adjust a physical model ofthe stimulus to match the computer generated sur-face. And finally, when we measured orientationconstancy, observers were able to recover surfacetexture, suggesting that they could recover surfacegeometry.

But if observers could recover surface texture,why did they base their similarity judgments onimage texture? One possibility is that judgmentsof texture similarity and texture segmentation arebased on a process that occurs in early vision. Ofcourse, to accommodate the results mentioned inthe introduction, some segmentation processes mustoccur after occlusions and reflectance have beenrecovered. Thus, texture segmentation may involvean early, rudimentary surface representation thatindicates depth ordering but not surface geometry.Surface geometry and surface texture may still berecovered by the visual system and used for ori-entation constancy, but this occurs only at a laterstage in processing.

Alternatively, the discrepancy between texturesimilarity and orientation constancy may arise notbecause texture processing occurs in early visionbut because the process underlying orientation con-stancy involves cognition rather than visual per-ception. That is, orientation constancy may not beperceived directly, but may involve some form ofdeliberate mental rotation. According to this sec-ond idea, judgments of texture similarity are basedon a high-level surface representation. But even atthe highest levels of visual processing, surface tex-ture is not represented.

In any event, a great deal of research has shownthat observers can readily detect the boundary be-tween two textures with different orientations [1,15, 14, 11, 23]. These texture boundaries may corre-spond to a number of situations in the world. Thatis, a texture boundary could reflect an occlusion, achange in surface texture or a surface fold. Ourresults indicate that observers cannot discriminatebetween texture discontinuities that are due to achange in surface texture and those that are dueto a surface fold. Thus this finding extends ear-lier work by showing that even though observerscan readily detect texture boundaries, they cannotreliably interpret these boundaries.

Acknowledgments

We are grateful for the support from NSF grantSBR-9729015 (Bravo) and NSF grant EIA-98-02068(Farid) and NSF CAREER award IIS-99-83806 (Farid).We thank Stijn Oomes for providing the stereo view-ing OpenGL code and Bob Hoskins for helping tocollect the data.

References

[1] Jacob Beck. Perceptual grouping producedby changes in orientation and shape. Science,154:538–540, 1966.

[2] Jacob Beck. The relation between similaritygrouping and perceptual constancy. AmericanJournal of Psychology, 88:397–409, 1975.

12

[3] Jacob Beck. Textural segmentation, chapter Or-ganization and representation in perception.Erlbaum Associates, 1982.

[4] James R. Bergen. Theories of visual texture per-ception, chapter Spatial Vision, vol. 10 of Vi-sion and Visual Dysfunction, pages 114–134.MacMillan Publishing, 1991.

[5] James R. Bergen and Edward H. Adelson.Early vision and texture perception. Nature,333:363–364, 1988.

[6] Terry Caelli. On discriminating visual tex-tures and images. Perception and Psy-chophysics, 31:149–159, 1982.

[7] Patrick Cavanagh, S. Peters, and M. vonGrunau. Rigidity failure and its effect on thequeen. Perception, 17:27a, 1988.

[8] L.O. Harvey and M.J. Gervais. Visual tex-ture perception and fourier analysis. Percep-tion and Psychophysics, 24:534–542, 1978.

[9] Zijiang He and Ken Nakayama. Perceivingtextures: Beyond filtering. Vision Research,34:151–162, 1994.

[10] Bela Julesz. Textons, the elements of tex-ture perception and their interactions. Nature,290:91–97, 1981.

[11] Michael S. Landy and James R. Bergen. Tex-ture segregation and orientation gradient. Vi-sion Research, 31:679–691, 1991.

[12] Andrea Li and Qasim Ziadi. Perceptionof three-dimensional shape from texture isbased on patterns of oriented energy. VisionResearch, 40:217–242, 2000.

[13] Jitandra Malik and Pietro Perona. Preatten-tive texture discrimination with early visionmechanisms. Journal of the Optical Society ofAmerica (A), 7(5):923–932, 1990.

[14] H.C. Nothdurft. Orientation sensitivity andtexture segmentation in patterns with differ-ent line oreintation. Vision Research, 25:551–560, 1985.

[15] Richard K. Olson and Fred Attneave. Whatvariables pro-duce similarity grouping? American Journalof Psychology, 83:1–21, 1970.

[16] Stephen Palmer, Jonathan Neff, and DianeBeck. Grouping and amodal completion. Psy-chonomic Bulletin and Review, 3:75–80, 1996.

[17] Stephen Palmer and Rolf Nelson. Late influ-ences on perceptual grouping: illusory fig-ures. Perception and Psychophysics, 62:1321–1331, 2000.

[18] Irvin Rock, Stephen E. Palmer Romi Ni-jhawan, and Leslie Tudor. Grouping based onphenomenal similarity of achromatic color.Perception, 21:779–789, 1992.

[19] Anne Sutter, Jacob Beck, and Norma Graham.Contrast and spatial variables in texture seg-regation: Testing a simple spatial frequencychannels model. Perception and Psychophysics,46:312–332, 1989.

[20] Anne Treisman. Features and objects in visualprocessing. Scientific American, 255:114–125,1986.

[21] M.R. Turner. Texture discrimination by ga-bor functions. Biological Cybernetics, 55:71–82,1986.

[22] Max Wertheimer. Untersuchungen zur lehrevon der gestalt ii. Psycologische Forschung,4:301–350, 1923. Translation published in El-lis, W. (1938). A source book of Gestalt psy-chology (pp. 71-88). London: Routledge &Kegan Paul.

[23] Sabina S. Wolfson and Michael S. Landy.Discrimination of orientation-defined textureedges. Vision Research, 20:2863–2877, 1995.

13