looking at op art from a computational viewpoint · 2004-03-26 · looking at op art from a...

Spatial Vision Vol 17 No 1-2 pp 75ndash94 (2004)Oacute VSP 2004Also available online - wwwvsppubcom

Looking at Op Art from a computational viewpoint

JOHANNES M ZANKERcurren

Department of Psychology Royal Holloway College University of London EghamSurrey TW20 0EX UK

Received 14 August 2002 revised 10 November 2002 accepted 11 November 2002

AbstractmdashArts history tells an exciting story about repeated attempts to represent features that arecrucial for the understandingof our environmentand which at the same time go beyond the inherentlytwo-dimensional nature of a at painting surface depth and motion In the twentieth century Opartists such as Bridget Riley began to experiment with simple black and white patterns that do notrepresent motion in an artistic way but actually create vivid dynamic illusions in static picturesThe cause of motion illusions in such paintings is still a matter of debate The role of involuntaryeye movements in this phenomenon is studied here with a computational approach The possibleconsequences of shifting the retinal image of synthetic wave gratings dubbed as lsquoriloidsrsquo wereanalysed by a two-dimensional array of motion detectors (2DMD model) which generates responsemaps representing the spatial distribution of motion signals generated by such a stimulus For atwo-frame sequence re ecting a saccadic displacement these motion signal maps contain extendedpatches in which local directions change only little These directions however do not usuallyprecisely correspond to the direction of pattern displacement that can be expected from the geometryof the curved gratings as an instance of the so-called lsquoaperture problemrsquo The patchy structure ofthe simulated motion detector response to the displacement of riloids resembles the motion illusionwhich is not perceived as a coherent shift of the whole pattern but as a wobbling and jazzing of ill-de ned regions Although other explanations are not excluded this might support the view that thepuzzle of Op Art motion illusions could potentially have an almost trivial solution in terms of smallinvoluntary eye movement leading to image shifts that are picked up by well-known motion detectorsin the early visual system This view can have further consequences for our understanding of howthe human visual system usually compensates for eye movements in order to let us perceive a stableworld despite continuous image shifts generated by gaze instability

Keywords Motion perception illusion gratings computational model Op Art motion signaldistributions

currenE-mail jzankerrhulacuk

76 J M Zanker

1 INTRODUCTION

Thomas Gainsboroughrsquos famous dictum lsquoPainting is a science and should bepursued as an inquiry into the laws of naturersquo (quoted in Zeki 1999) makesneuroscientists curious to learn from their artistic companions about brain functionand at the same time puts the artist in a rather challenging position Throughoutthe history of visual arts we encounter in this pursuit of the laws of naturerepeated attempts to develop a pictorial language for representing features of afour-dimensional world which defy capture on an essentially two-dimensionalpainting surface depth and motion The twentieth century which was marked bythe invention and development of kinematic media (cinema television internet)witnessed various attempts to make motion sensations an integral part of visualarts (such as kinetic arts) and Op artists like Bridget Riley began to experimentwith simple black and white patterns that can create vivid dynamic illusions instatic pictures (Riley 1999) Simple designs such as concentric areas lled withoblong checkerboard patterns of different orientations (Ouchi 1977) attractedmuch interest from psychophysicists (Hine et al 1997 Khang and Essock 1997Mather 2000)

The physiological and perceptual mechanisms that might underlie the illusoryperception of motion in Op Art paintings are a matter of ongoing debate (seeWade 1982 for an introduction and many examples) Ever since the earlydescriptions by Purkinje and von Helmholtz (1924) of dynamic deformationsobserved in patterns composed of ne lines the importance of optical effectsrelated to eye movements and accommodation has been discussed repeatedly andhas been countered by various views that peculiarities of the neural representationcan lead to dynamic image distortions A recent example of such a dispute isthe lsquoEnigmarsquo painting by I Leviant which elicits a variety of moireacute pattern andmotion illusions (Leviant 1996) lsquoEnigmarsquo consists of a pattern of radiating blackand white lines similar to the ray pattern used by MacKay (1957a) on whichconcentric rings of uniform colour are superimposed On the one hand the sensationof a moireacute pattern itself has been attributed to fading afterimages of the originalimage after saccadic eye movements (MacKay 1957b Gregory 1993) whereas theshimmering and deformations that can be seen in the radial line patterns have beeninterpreted as rather trivial consequences of accommodation uctuations (lsquohuntingfor accomodationrsquo Gregory 1994) following an initial suggestion by Campbelland Robson (1958) On the other hand a rather different effect a circular motionwithin the uniformly coloured rings has been attributed to a so far ill-de nedcortical mechanism that does not rely on image shifts This suggestion was basedon evidence from functional brain imaging that suggests some speci c activationin motion-speci c human brain regions related to the perceived illusion and theobservation that aphakic patients can perceive the illusion (Zeki et al 1993 Zeki1994 1995) However this dispute seems to suffer from three unresolved issues(a) The arguments are often stated as alternatives that exclude each other rather thansuggesting a variety of causes that can lead to similar effects What if for instance

Computational analysis of Op Art 77

hunting for accommodation small involuntary saccades and an unknown corticalmechanism can all generate motion illusions in Op Art paintings independently(b) Little is known about the actual characteristics of eye movements carried out byan observer watching an Op Art painting under everyday viewing conditions It iseasy to demonstrate that saccadic eye movements do enhance the sensation of vividpattern motion when looking at such patterns (Gregory 1994) but how stable is thegaze when an observer tries to xate a speci c point in the stimulus (c) There areonly vague speculations about how lsquoafterimagesrsquo and effects of retinal displacementor deformation could generate the described perceptual effects Is it be possible topropose a precisely de ned mechanism in computational terms which would allowus to formulate precise expectations about how the illusion should appear

In order to resolve some of the issues related to the importance of retinaldisplacements a combined experimental and computational approach is requiredFirstly by measuring the eye movements that are carried out by observers lookingat Op Art paintings one will be able to lsquolook through the eyes of the observerrsquoat the painting and knowing exactly what kind of sensory information the brain isdealing with The results from an initial study of such eye movements are presentedelsewhere (Zanker et al 2003) At this point it is only important to note that gazestability is rather limited even under xation conditions and that small involuntarysaccades in apparently random directions are abundant Secondly this fundamentalobservation can be used as a basis from which to deduce precise expectations aboutthe kind of input information available to the cortex when an observer is looking atsuch a painting This analysis of the sensory information available to higher corticalprocessing is in the focus of the work presented here which uses a biologicallymotivated motion detector network to describe the motion information contained indiscrete displacements of simple black and white patterns such as would result fromsmall saccades during observation of such stimuli Bridget Rileyrsquos painting lsquoFallrsquo(1963 Tate Gallery London) was chosen as stimulus pattern because (a) the patternis composed exclusively from ne lines without any other structural elements mdashthis reduces the variety of early visual mechanisms that need to be consideredwith regard to the neural encoding of these patterns (b) it allows for a simplemathematical description (see Section 21 and Fig 2) which provides easy accessfor parametric variations of the critical stimulus features and (c) it does not elicitmultiple perceptual effects but a unitary and strong illusion that may afford a clearerdescription than the rather diffuse lsquoshimmeringrsquo or lsquojazzingrsquo observed in MacKayrsquosray patterns for instance When watching Rileyrsquos lsquoFallrsquo after some short period oftime the line gratings become animated in a peculiar fashion in which a numberof hazily outlined and dynamically changing horizontal pattern sections seemto move in a rather coherent direction which between different horizontal bandshowever can vary considerably

An initial idea about the input information to higher cortical processing can bederived from a geometrical consideration of the effect a single small displacement ofthe Riley pattern has on the retinal image Assuming a limited temporal integration

78 J M Zanker

Figure 1 Additive (left side) and subtractive (right side) interference (moireacute) patterns generated bysuperimposing a phase-modulated grating (riloid) with a slightly displaced copy of the same pattern(diagonal displacement the two frames are shown in the centre) For details see text

time (ie visual persistence) in the response of the retinal neurones one can expectthe two retinal images (ie two frames of such a displacement sequence shownin Fig 1 for a diagonal displacement by a quarter of the grating cycle in bothhorizontal and vertical direction) to be superimposed in the retinal representationof the stimulus leading to an additive moireacute pattern similar to that discussed byGregory (1994) and Spillmann (1993) These interference patterns (left side ofFig 1) illustrate the characteristic regions of image blur in the temporally averaged(or low-passed) intensity distribution hitting the retina and thus correspond nicelyto some of the described static effects on the pattern appearance but do not takeinto account temporal intensity changes that will be crucial for any sensation ofmotion In order to look at this aspect of the two-frame sequence it may behelpful to subtract the intensity values of the two images thus marking the regionsof increasedecrease in luminance by brightdark pixels The resulting differencemoireacute pattern (right side of Fig 1) is complementary to its additive counterpartshowing highest contrast (maximum intensity increase or decrease) in regions wherethe additive moireacute pattern was blurred and vice versa Although the differenceinterference patterns illustrate the areas of dynamic change they do not indicatethe nature of this change and in particular fall short of providing an informativepicture of the distribution of motion signals Conspicuous areas of high contrast ieneighbouring bright and dark stripes for instance indicate a strong change of localintensity in opposite directions Due to the periodic nature of the stimulus thesemay correspond to identical motion signals but they do not tell us the direction


of such motion signals and they could equally be related to incoherent motionsignals or to icker Therefore a more thorough analysis of the motion signaldistributions will be provided in the following which is based on a biologicallymotivated model (Zanker et al 1997) This lsquo2DMD modelrsquo consists of a two-dimensional array of motion detectors which provides response maps that representthe spatial distribution of motion signals generated by a stimulus such as a two-frame sequence re ecting a saccadic displacement

2 METHODS

21 Stimulus patterns Riloids

For the computational analysis stimulus patterns were generated to ll a 512 pound 512pixels array with grey levels varying in arbitrary units between 00 (black) and 10(white) In order to study the effects of different pattern layouts a simple algorithmwas used to generate arti cial patterns which when appropriate parameters arechosen strongly resemble Rileyrsquos lsquoFallrsquo (a reproduction of this painting can befound in Gregory 1998) and are therefore dubbed lsquoriloidsrsquo The fundamental gratingis generated by sinusoidal modulation of intensity I with the grating period cedilalong the horizontal axis x according to the formula

I x y D 051 C sin[2frac14x iexcl Aacutey=cedil] (1)

The phase of the sinewave function Aacute is modulated sinusoidally along the verticalaxis y as described by the function

Aacutey D A sin[2frac14y=sup1y] (2)

with a phase modulation amplitude A and a phase modulation period sup1 decreas-ing linearly between a maximum at the top and a minimum at the bottom of thepattern according to the formula

sup1y D sup1top C ysup1top iexcl sup1bottom=ytop iexcl ybottom (3)

As illustrated in Fig 2 the crucial parameters that in uence the appearance ofthe riolod patterns are the fundamental grating period cedil and the phase modulationperiod sup1 By setting cedil to 5 pixels and the phase modulation amplitude A to 32pixels and varying sup1 between 400 and 130 pixels a pattern is generated that closelyapproaches the appearance of the original Riley painting (centre of Fig 2) Fixingthe phase modulation at a period sup1 of 180 pixels leads to a much more regularpattern (left side of Fig 2) which still elicits the motion illusion but may appear lessvivid Keeping the variation of the phase modulation period unchanged and settingthe grating period cedil to 32 pixels a much coarser version of the closely matchingriloid is generated (right side of Fig 2) which requires a larger viewing distanceto elicit the motion illusion The standard riloid used for most of the simulationspresented here was de ned by cedil D 32 pixels A D 16 pixels sup1top D 256 pixels

80 J M Zanker

Figure 2 Parametric variations of riloid patterns (512 pound 512 pixels) generated using equations (1)to (3) and varying the grating period cedil the phase modulation amplitude A and the phase modulationperiod sup1 The centre panel shows a con guration resembling the original painting cedil D 5 pixelsA D 32 pixels sup1 varies between 400 and 130 pixels left and right panels (half of the patternshown) show simple variants generated by changing sup1 and cedil respectively (parameters given belowthe patterns)

and sup1bottom D 64 pixels Compared to the original painting these parameterslead to a somewhat enlarged pattern which makes it possible to compute localmotion responses for a range of modulation periods at suf ciently high resolutionswhile keeping the computational costs low It should be kept in mind that arbitraryspatial scaling factors can be used in the digital simulations and that the meaningfulvalue is relative size with respect to the grating period which determines thespatial frequency channel in the human visual system that is optimally tuned to theparticular pattern When watching Bridget Rileyrsquos lsquoFallrsquo from a viewing distanceof 2 m the absolute grating period would be about 037plusmn corresponding to a besttuned channel of approx 3 cycles per degree

22 Computational model

A simpli ed version of the dimensional motion detector model that has been usedpreviously (Zanker et al 1997) was applied in the present study to determinethe response to two-frame sequences of riloid pattern displacements The basicbuilding blocks of this 2DMD model are elementary motion detectors (EMDs) ofthe correlation type that have been shown in many psychophysical behavioural andphysiological studies to be a most likely candidate for biologically implementedmotion detectors (for review see Reichardt 1987 Borst and Egelhaaf 1989)This elementary motion detector is used here as a representative of a variety of


luminance-based motion detectors (Adelson and Bergen 1985 van Santen andSperling 1985) while other models (Torre and Poggio 1978 Srinivasan 1990)could be used without affecting the main conclusions we draw from our results Forinstance detecting local image displacements by means of a gradient-type algorithm(Johnston et al 1999) is not expected to change the pattern of motion signals thatare determined by the distribution of local orientation and grating period The2DMD model consists of a two-dimensional network of pairs of such local EMDswhich detect horizontal and vertical motion components (schematically sketched asinset in Fig 3) It has been extensively used to simulate a variety of psychophysicalphenomena (Zanker et al 1997 Patzwahl and Zanker 2000 Zanker 2001)

In a simple implementation each EMD receives input from two points of thespatially ltered stimulus patterns The signals interact in a nonlinear way aftersome temporal ltering to provide a directionally selective signal Differences ofGaussians (DOGs) were used as bandpass lters in the input lines with excitatorycentre and inhibitory surround balanced as to exclude any DC components fromthe input (Marr and Hildreth 1980 Srinivasan and Dvorak 1980) The samplingdistance between the two inputs which is the fundamental spatial model parameterwas xed to 8 pixels This value was chosen in order to use a motion detectorthat is optimally tuned to the grating period of the riloid (ie to deliver thestrongest responses to pattern displacements) For a grating of 03ndash04plusmn such as thatobserved in Rileyrsquos lsquoFallrsquo under lsquonaturalrsquo viewing conditions (see above) this wouldcorrespond to a motion channel tuned to a spatial frequency of 3 cycles per degreeTo prevent aliasing the diameter of the receptive eld (as measured between zero-crossings from excitatory to inhibitory regions) was set to about twice the valueof the sampling distance The signal from one input line was multiplied with thetemporally ltered signal from the other line and two antisymmetric units of thiskind were subtracted from each other with equal weights leading to a fully opponentEMD This kind of detector is known to be highly directionally selective (Borst andEgelhaaf 1989) The time constant of the rst-order lowpass lter which is thefundamental temporal parameter of the EMD was xed to an arbitrary value of 2simulation time steps for an input sequence in which each of the two stimulus frameslasts for 8 simulation time steps The model output was averaged over the rst 2time steps after image displacement (change from stimulus frame 1 to frame 2)during which the EMD response reached its maximum Response patterns at a laterstage are in essential attenuated versions of those presented here The 2DMD modelprovides two arrays of 512pound512 EMDs containing the horizontal and vertical localresponse components respectively These response components can be displayedas two-dimensional motion signal maps (see Section 31 Fig 3) or can be averagedhorizontally across the arrays to generate vertical motion signal pro les as will bepresented in Section 32 of this paper For this purpose a border of 32 pixels on allfour sides of the stimulusresponse patterns was ignored in order to avoid boundaryeffects from spatial lters extending beyond the stimulus pattern leaving outputmaps for the central 448 pound 448 pixels of the stimulus pattern

82 J M Zanker

3 RESULTS AND DISCUSSION

31 The general structure of motion signal maps

Based on the geometry of the two-frame pattern displacement one would expecthorizontal streaks of activity in the motion detector network response as suggestedby the difference moireacute patterns shown in Fig 1 A typical motion signal map (thetwo-dimensional distribution of motion detector outputs) generated by the 2DMDmodel in this case for the standard riloid (cedil D 32 pixels A D 16 pixels sup1

varying between 256 and 64 pixels) displaced diagonally by 8 pixels rightwards anddownwards is shown in the centre of Fig 3 The signal maps are shown in a 2D-colour-code illustrated as inset in Fig 3 in which different hues indicate differentdirections of the 2DMD output (right-green up-yellow left-red down-blue) andsaturation indicates the strength of the motion signal at a given point in the map

The motion signal maps shown in Fig 3 show two striking effects an abundanceof many different colours suggesting the emergence of various motion directions

Figure 3 Motion signal maps generated by the 2DMD model (schematically sketched in top leftinset) for shifting riloids in the diagonal direction The central 448 pound 448 elements of the motiondetector output array are shown in 2D-colour-code representing direction and strength of the localmotion signal by hue and saturation as indicated in top right inset The horizontal bands of variouscolours show that the stimulus leads to motion responses in a variety of directions which are groupedin regions perpendicular to the basic line orientation The three panels show the response maps forthree different displacement step sizes (4 8 and 12 pixels in x- and y-direction from left to rightonly half of the response pattern is shown on the left and right) This gure is also published in colouron httpwwwingentacom


and a strati cation in horizontal bands of identical colour Both effects are not verysurprising if one considers the local stimulus con guration (a) Horizontal bandsmay be expected from the simple fact that the stimulus pattern is strictly periodicalong the horizontal axis so that horizontally neighbouring motion detectors areexposed to identical input patterns as long as they have the same phase relationshiprelative to the fundamental grating cycle Variations in this phase of the localmotion detector relative to the grating cycle can lead to small periodic variationsof the motion signals which create the wiggly borders between different 2DMDoutput regions visible in Fig 3 (b) The phase modulation of the sinewave gratingde ning the riloids leads to overall vertical black and white lines that oscillatehorizontally (see Fig 2) and thus change their local orientation according to thephase modulation function (see equations (2) and (3)) With a restricted eld ofview a local motion detector serving as element of the 2DMD model is exposedto a piece of the overall pattern for which the dominating orientation can varygreatly In consequence the local motion detector is dealing with the so-calledlsquoaperture problemrsquo (Wallach 1935 Marr and Ullman 1981 Adelson and Movshon1982 Nakayama and Silverman 1988) Because the predominant motion detectorresponse for 1D luminance modulations (ie gratings) will always be perpendicularto the contour orientation (Reichardt and Schloumlgl 1988 Castet and Zanker 1999) ithas to be expected that the direction of local motion signals varies together with theorientation of the pattern lines This idealised picture is somewhat complicated byfact that for non-horizontal displacements the input from the two stimulus framescan be different in local orientation and local spatial frequency and that for largepattern displacements grating phase shifts larger than half a pattern cycle can appearwhich result in inverted motion signals (see Section 32)

32 Variations of pattern displacement

What is the effect of changing the direction and amplitude of pattern displacementon the resulting motion signal maps The left and right parts of Fig 3 show the2DMD response for smaller (4 pixels shift in horizontal and vertical direction) andlarger (12 pixels shift in horizontal and vertical direction) pattern displacement thanthe standard con guration depicted in the middle panel (8 pixels shift in horizontaland vertical direction) Whereas the left panel of Fig 3 has more blue-green regions(corresponding to the diagonal displacement) than the standard con guration thelarger displacement step leads to a increase to the red-orange-yellow hues (iethe opposite diagonal directions) as shown in the right panel of Fig 3 Thisphenomenon can be easily understood as a consequence of matching two regionsof the stimulus pattern for which the phase angle between the two positions of thefundamental gratings is larger than half a grating cycle thus leading to an apparentinversion of motion direction Whereas it is obvious that the likelihood of suchinversions happening increases with growing displacement step size it should alsobe noted that the critical step size for the motion signalrsquos zero-crossing is not simplyhalf the fundamental grating period Because the critical stimulus variable for this

84 J M Zanker

effect is half the grating period orthogonal to its orientation (ie line thickness)the critical step size depends on the local pattern orientation which strongly affectsthe line thickness (see Fig 2) The inversion effect is much more prominent forpurely horizontal displacements for which the response depends mainly on thehorizontally periodic structure of the fundamental grating and is not confoundedby matching different vertical regions with different phase modulation

In an attempt to quantify the structure of the motion signal maps by exploiting theemergent horizontal bands in these maps vertical motion pro les were calculatedby averaging the response values across the rows of the data arrays For further datareduction the mean results for 112 sets of four rows were calculated and plotted asa function of vertical position in the stimulusresponse pattern In Fig 4 the resultof this operation is shown for horizontal (a b) and vertical (c d) displacementof the standard riloid with the horizontal motion component plotted in the leftcolumn (a c) and the vertical motion component plotted in the right column (b d)These pro les are shown for different step sizes of the displacement between thetwo stimulus frames (indicated by different symbols in Fig 4) Diagonal patterndisplacements lead to somewhat intermediate results (data not shown)

Not surprisingly due to the strictly periodic nature of the riloids along thehorizontal axis the clearest picture is given by the horizontal components of motiondetector responses to horizontal displacements (Fig 4a) Taking the pro le for adisplacement of 8 pixels (ie a quarter of the grating cycle) as an example (darktriangles in Fig 4a) in the top region ( rst 300 rows) there is a comparativelystrong positive response (corresponding to rightwards displacement) with onlysmall uctuations which are related to the phase modulation ie local orientationof the grating For the lower part of the stimulus (rows 300 and beyond) the responseis still positive on average but it starts uctuating considerably and is reduced forthe regions at the bottom of the riloids mdash re ecting the ever ner modulation of thegratingrsquos phase The data for other displacements can be regarded as scaled versionsof the very same response pro le at each vertical location the purely horizontaldisplacements simply shift the phase of a sinusoidal input function (the amplitudeof which depends on vertical location) and because the displacement step size onlyaffects the size of the phase shift and thus increases or decreases the output by thesame factor at any pattern position In particular (a) phase shifts by any multipleof frac14 (ie 0 16 32 48 pixels) lead to zero response because the pattern isidentical for both frames or counter-phasing (b) phase shifts between 0 and frac14 (eg4 8 12 pixels) lead to positive responses whereas phase shifts between frac14 and2frac14 (eg 20 24 28 pixels) lead to negative responses because in the latter casethe closer matching function (ie stronger EMD output) results from a equivalentleftward phase shift (iexcl12 iexcl8 iexcl4 pixels) (c) the same response pattern is elicitedby a phase shift Aacute and Aacute C 2frac14n eg 4 and 36 pixels for the standard riloid becausean additional phase shift by a full pattern cycle generates an identical input A singleresponse pro le scaled by an amplitude factor depending on displacement step sizeis also found for the vertical response components (Fig 4b) The fundamental


Figure 4 Motion signal pro les (horizontal averages of motion signal maps plotted as a functionof vertical position in pattern) for shifting riloids in horizontal (a b) and vertical (c d) direction byvarious amounts (different symbols as indicated in inset) The horizontal (a c) and vertical (b d)motion signal components are shown separately to illustrate how much of the response is related tothe actual direction of displacement and how much is induced by the structure of the pattern

response pro le in this case is characterised by multiple sign inversions re ectingthe changes of local orientation of the lines that add an upward or downwardcomponent to the motion vector perpendicular to the lines It should be noted thatthe maximum of the vertical components for the standard con guration is in thesame range as that of the horizontal components meaning that motion directionsextend up to sect45plusmn Similar to the horizontal motion components in the lowerregions of the stimulus the vertical response pro les become very irregular andreduced in overall amplitude In summary all possible response patterns for purelyhorizontal pattern displacements are self-similar and only differ by an amplitudefactor including the possibility of sign inversion

The whole picture is more complicated for vertical displacements (Figs 4cand 4d) because the standard riloid is not repetitive along the vertical axis Inconsequence the response pro les are less regular even for the riloid regions withlarger phase modulation periods and only vaguely resemble each other when thedisplacement step size is varied Nevertheless the vertical signal components reveala small overall negative response in the upper pattern regions (corresponding to the

86 J M Zanker

downward displacement see Fig 4d) whereas the horizontal response componentsare centred around zero (see Fig 4c)

For a further simpli ed description of the response patterns the direction andstrength of the average motion signal was calculated to quantify any possible overalltendency and the average of the motion vector length (ie not taking into accountthe direction of the local signal) to quantify the overall motion energy detected bythe 2DMD model This was done separately for the upper and the lower half of thestimulus to show the in uence of pattern structure The results of these calculationsare shown in Fig 5 as a function of displacement step size for horizontal verticaland diagonal shifts of the riloids

Looking at the average direction of the motion detector response (Fig 5c)con rms the rst impression from the example pro les shown in Fig 4 that theactual direction of pattern displacement is captured by the model only undervery restricted conditions Horizontal displacements (squares in Fig 5c 0plusmn)generate responses close to 0plusmn for small displacements then invert to 180plusmn andfor even larger displacements (no direction can be determined for multiples of

Figure 5 Strength (a) and direction (c) of average motion vector and average length of motion vectors(b) for the upper (black symbols) and lower (grey symbols) half of the riloid pattern The riloids weredisplaced by various amounts (abscissa) and in three different directions (horizontal squares verticaltriangles diagonal diamonds)


16 pixel displacements due to zero response) Vertical displacements (triangles inFig 5c iexcl90plusmn) show responses in the neighbourhood for small and intermediatedisplacement step sizes (and preferentially in the upper half of the riloid) but thenstart to assume values unrelated to the actual direction of pattern shift The averageresponse direction for diagonal displacements (diamonds in Fig 5c iexcl45plusmn) showsno clear relation to the veridical direction and varies considerably and apparentlyerratically with displacement step size In conclusion the motion signals on averagecontain little or no reliable information about the veridical displacement of thepattern

Comparing the size of the coherent motion signal (strength of average responseFig 5a) with the overall motion energy contained in the response (average lengthof motion vectors Fig 5b) reveals another interesting aspect of riloids Only forthe horizontal displacement (square symbols in Figs 5a and 5b) can one see a closecorrespondence between these two values For both values the coherent (rightward)motion signal accounts for much of the overall motion energy and they also sharethe periodic modulation with displacement step size that was discussed above as aconsequence of shifting the relative phase of two sinusoidal input functions Forboth diagonal (diamonds) and vertical (triangles) pattern displacements there is alarge amount of motion energy in the 2DMD response mdash perhaps a little more inthe upper than in the lower half of the pattern for all displacement step sizes testedso far (see Fig 5b) On the other hand for these non-horizontal shifts motionenergy is converted to a sizeable coherent motion signal (see Fig 5a) that can beused to retrieve the original image displacement only for rather small displacementstep sizes Again the situation is slightly more favourable but not fundamentallydifferent for the upper half than for the lower half of the pattern To summarisethere is an interesting signal-to-noise effect when it comes to analysing the overallpattern displacement although there are strong and non-random pattern of signalsin the motion signal maps derived from simple riloid pattern displacements onlya small fraction of these signals contributes to coherent motion information thatre ects the displacement in the stimulus with the exception of purely horizontalpattern shifts that generate rather coherent maps

33 Variations of pattern layout

So far all considerations have been restricted to a single arbitrary combinationof pattern layout and elementary motion detector It is well known that thehuman visual system contains an extensive set of detectors with different spatialand temporal frequency tuning (van de Grind et al 1986) and it may be askedwhether different spatiotemporal channels could generate different perhaps moreinformative response patterns for the same stimulus Conversely one can study thedependence of the response of a given motion detector on the spatial layout of thestimulus This approach was followed here by changing the fundamental gratingperiod cedil and then determining the response patterns for displacement steps of aquarter grating cycle that leads to an optimum response (eg 8 pixels displacement

88 J M Zanker

step for a riloid with cedil D 32 pixels cf Fig 5a) The results of this set of simulationsis shown in Fig 6 for grating periods between 4 and 64 pixels (ie between 1=2and 8 times the sampling distance of the elementary motion detector of the 2DMDmodel) for horizontal and vertical displacements

The average response (Fig 6a) representing the coherent motion signal compo-nents in the upper half of the pattern can best be described by a comparatively nar-row tuning function that drops off gently on both sides of the maximum at cedil D 28pixels for both horizontal and vertical shifts For the lower half of the pattern thetuning functions are shifted rightwards with larger periods still leading to substan-tial responses mdash this effect as well as the asymmetry of the tuning curves can bewell understood by considering the decrease of the effective grating period (ie thedimension of a grating cycle perpendicular to its orientation see Fig 2) with de-viations from vertical orientations which extends the sensitivity of the elementary

Figure 6 Strength of average motion vector (a) and average length of motion vectors (b) for theupper (black symbols) and lower (grey symbols) half of the riloid pattern The fundamental gratingperiod cedil of the riloids was varied (abscissa) and the patterns were displaced by a quarter period inhorizontal (squares) or vertical direction (triangles)


motion detector to larger grating periods The tuning curves of the average mo-tion vector lengths (Fig 6b) representing the overall motion energy contained inthe stimulus strongly resemble the coherent tuning functions in their shape but aregenerally ampli ed by approximately 20 This difference in amplitude is expectedfrom the variation of local grating orientation that generates motion components or-thogonal to the average direction Such lsquoincoherentrsquo motion components are lost inthe overall average but are preserved in the pooled motion energy (this effect hasbeen discussed in Section 32) The important aspect to note here is that this reduc-tion in the coherent signal is rather constant over all values of cedil and therefore doesnot depend on how well the motion detector is adjusted to the fundamental gratingperiod In summary the narrow and similar tuning functions for the coherent motioncomponents and the overall motion energy demonstrate that despite the complexityof the riloid structure there is little activation in motion channels that are not tunedto the fundamental grating period and that there is no speci c gain of coherent mo-tion components in other channels Conversely there are no hidden componentsin other spatiotemporal frequency bands and thus the dominant response to a riloidwill come from the population of motion detectors that are tuned to the fundamentalpattern grating

4 CONCLUSIONS

The present work is an attempt to understand the kind and the accuracy of informa-tion that is available to the brain when lsquoriloidsrsquo which are mathematically simpli- ed variations of Bridget Rileyrsquos painting lsquoFallrsquo are subjected to displacement stepssuch as happen during the small involuntary saccades that observers of Riley paint-ings exhibit under lsquonormalrsquo viewing conditions (Zanker et al 2003) A biologicallymotivated model of motion detector networks (2DMD model Zanker et al 1997)was used to determine the motion signal distribution that results from such a stim-ulus which can be regarded as the basic information that is encoded in the earlystages of the nervous system to be used as input for higher cortical processing

The motion signal maps provided by the 2DMD model show a characteristicpattern with horizontal bands of coherent motion the direction of which can varyconsiderably with the location of a patch and the size of the displacement stepFor almost any displacement direction and size the overall picture is characterisedby large amounts of motion energy in a limited spatiotemporal frequency rangewhich however does not lead to any substantial coherent average motion signalthat re ects the actual image displacement In other words the computational modelpredicts that the appearance of a riloid through an eye with imperfect stabilisationwill be characterised by dynamically changing horizontal bands that exhibit strongcoherent motion in various unpredictable directions

It is striking how closely this description of the simulation results correspondsto the illusion that is experienced when observing the Riley painting where thewaves of the wiggling lines are animated into dynamic patches of motion in

90 J M Zanker

various directions Even when patches of motion in different directions cannot besegmented several directions can be detected simultaneously a phenomenon knownas motion transparency (van Doorn and Koenderink 1982 Zanker 2001) Thus thecombined experimental and computational approach followed in the present studycan provide a rather simple explanation of this most impressive phenomenon Thisexplanation is similar to some accounts of the motion illusions elicited by radialgratings (MacKay 1957a) and related patterns in that it relates to retinal imagedisplacements (Campbell and Robson 1958 Gregory 1994 Leviant 1996) but itgoes beyond these suggestions in that it demonstrates by means of simulations thatsuch displacements would be suf cient to elicit the illusion The persistence of theillusion under experimental conditions that intend to minimize or to eliminate imageshifts (MacKay 1958 Zeki 1994) can be considered as an empirical issue becausethe actual quality of image stabilisation is not always easy to verify and even tinydisplacements can elicit the characteristic response patterns Furthermore it needsto be kept in mind that a variety of conditions leading to the same effect do notnecessarily exclude each other so the persistence of the illusion after abolition ofeye movements does not contradict the proposition that they are responsible whenthey are present The major advantage of the suggestion put forward here even if itis not the only possible one is that it is fully satisfactory and that at the same timeit requires only modest and plausible assumptions namely the presence of smallinvoluntary eye movements and a network of low-level motion detectors

A perhaps more puzzling question is why corresponding effects of small invol-untary eye movements are not perceived with other patterns starting from straightgratings or checkerboards through displays containing simple objects and extend-ing to complex natural scenes To illustrate this point motion signal maps weregenerated for the diagonal displacement of two control patterns which are shownin Fig 7 Firstly an artistic modi cation of checkerboards was generated by in-troducing smooth variations of check sizes similar to patterns designed by VictorVasarely and accordingly dubbed lsquovasareloidsrsquo (left side of Fig 7) Secondly aradial grating was generated which has the same structure as the lsquoray patternrsquo stud-ied by MacKay (1957a b) consisting of 128 sectors of alternating black and whitecolour (right side of Fig 7) Both motion signal maps are dominated by green-blue colours which correspond to the actual displacement of the pattern to the rightand downwards In consequence from these distributions of motion signals thedirection of image shift that led to the signal map could be recovered for instanceto compensate for the image displacement elicited by small involuntary saccadesComparison with the corresponding signal maps for riloids shown in Fig 3 demon-strates that such a mechanism would not be successful in that case because there areno systematic signal components that dominate the motion direction distributionsIt should also be noted that for the radial gratings inconsistent motion signals arefound in the central regions which nicely re ect the lsquoshimmering effectrsquo perceivedwhen looking at these patterns


Figure 7 Motion signal maps (bottom row a sampling distance of 4 pixels was used for thelocal motion detector to capture ne pattern details same 2D-colour-code as in Fig 3) generatedby the 2DMD model for shifting two control patterns (shown in top row) in a diagonal direction(displacement rescaled to 4 pixels in x- and y-direction compare with responses for the riloid patternshown in Fig 3) Left side lsquoVasareloidsrsquo mdash checkerboard patterns in which check size variessmoothly between different image regions (eg lsquoVegarsquo Vasarely 1957 Simonyi Collection) Rightside radial grating or lsquoray patternrsquo as studied by MacKay (1957a b) Both motion signal mapscontain strongly biased distributions of motion directions that allow us to recover the direction ofimage shift that led to the signals This gure is also published in colour on httpwwwingentacom

It is a common experience that has attracted considerable scienti c interest that weusually perceive a perfectly stable world although every eye movement does lead toretinal image shifts that should be perceived as movement of the environment Theclassical answer to this puzzle is based on the assumption that extra-retinal signalssuch as the motor commands themselves could be used to compensate for the imageshifts elicited by eye movements (von Helmholtz 1924 von Holst and Mittelstaedt1950) Alternatively the information contained in the image displacement itself

92 J M Zanker

could be used for compensation (MacKay 1973) Furthermore there have beenvarious suggestions that motion perception is actively suppressed during saccades inthe human visual system (Burr et al 1994) or that displacements are invisible due tothe dynamic limitations of motion perceptions (Castet and Masson 2000) and thereis evidence that the retinotopic space is re-mapped with respect to external spacewhen saccades are performed (Ross et al 1997 Lappe et al 2000) Basicallyeach of these compensation mechanisms should always work for a wide variety ofstimulus patterns mdash so why are they not successful for riloids

To answer this question it is crucial to remember that riloids although theygenerate a considerable amount of motion energy in a narrow frequency band do notallow the reliable estimation of the direction of displacement nor the combinationof motion information across different spatiotemporal lters In consequence thevisual system has no means of correcting for the image displacement by estimatingretinal image shifts When a pattern appears at a different position before andafter the saccade even if motion detection is suppressed during the saccade itwould still serve as an ef cient stimulus for the motion detector network due tothe pattern displacement as such just as assumed in the present computationalanalysis So the only source of information to compensate for image displacementof this stimulus class would be extra-retinal such as the proprioceptive or motor-control signals that are related to the eye movements themselves It is questionablewhether the precision of this information source would be suf cient to signal smalleye movements with the necessary accuracy For the original Riley paintingassuming a typical gallery viewing distance of 2 m the fundamental periodof the grating amounts to approximately 037plusmn The simulations presented heredemonstrate that displacements even smaller than a grating period can lead tosubstantial dif culties in extracting the veridical direction of displacement (seeFig 5) so any compensation mechanism would need to encode eye position witha precision going beyond small fractions of a degree The view that the onlyavailable extra-retinal information is not suf cient to correct image displacement issupported by the simple fact that we perceive lsquoillusoryrsquo motion when looking at theRiley painting Conversely the fact that no such motion is perceived for the controlpatterns for which the simulations reveal consistent motion directions would meanthat under more favourable stimulus conditions it is indeed the retinal image shiftitself that is used to make the outside world appear stable

Because the study of illusions elicited by Op Art paintings prompts unexpectedquestions and suggestions such as those raised in the last paragraphs it leads us farbeyond the initial simple explanation of the illusion in terms of small involuntaryeye movement generating image shifts that are picked up by well-known motiondetectors in the early visual system Thus Op Art can prove to be much morefor scientists than just an interesting sensation however pleasant provocative orpuzzling it may be because it can help us to understand fundamental mechanismsof visual processing in highly evolved nervous systems


Acknowledgements

I am very grateful to Robin Walker for various discussions and to him JamesTaylor and the reviewers for many useful comments on an earlier version of thismanuscript I also thank the Vision Group at Royal Holloway for creating a inspiringand critical working environment Special thanks go to Bridget Riley for creatingan amazing visual puzzle that challenges our understanding of the human visualsystem

REFERENCES

Adelson E H and Bergen J R (1985) Spatiotemporal energy models for the perception of motionJ Opt Soc Am A 2 284ndash299

Adelson E H and Movshon J A (1982) Phenomenal coherence of moving visual patterns Nature300 523ndash525

Borst A and Egelhaaf M (1989) Principles of visual motion detection Trends in Neuroscience 12297ndash306

Burr D C Morrone M C and Ross J (1994) Selective suppression of the magnocellular visualpathway during saccadic eye movements Nature 371 511ndash513

Campbell F W and Robson J G (1958) Moving visual images produced by regular stationarypatterns Nature 181 362

Castet E and Masson G (2000) Motion perception during saccadic eye movements NatureNeuroscience 3 177ndash183

Castet E and Zanker J M (1999) Long-range interactions in the spatial integration of motionsignals Spatial Vision 12 287ndash307

Gregory R L (1993) A comment Mackay rays shimmer due to accomodation changes Proc RoySoc London B 253 123

Gregory R L (1994) Under the carpet Perception 23 741ndash744Gregory R L (1998) Eye and Brain Oxford University Press OxfordHine T Cook M and Rogers G T (1997) The Ouchi illusion An anomaly in the perception of

rigid motion for limited spatial frequencies and angles Perception and Psychophysics 59 448ndash455

Johnston A McOwan P W and Benton C P (1999) Robust velocity computation from abiologically motivated model of motion perception Proc Roy Soc London B 266 509ndash518

Khang B G and Essock E A (1997) A motion illusion from two-dimensional periodic patternsPerception 26 585ndash597

Lappe M Awater H and Krekelberg B (2000) Postsaccadicvisual referencesgenerate presaccadiccompression of space Nature 403 892ndash895

Leviant I (1996) Does lsquobrain-powerrsquo make Enigma spin Proc Roy Soc London B 263 997ndash1001MacKay D M (1957a) Moving images produced by regular stationary patterns Nature 180 849ndash

850MacKay D M (1957b) Some further visual phenomena associated with regular patterned stimula-

tion Nature 180 1145ndash1146MacKay D M (1958) Moving visual images produced by regular stationary patterns Nature 181

362ndash363MacKay D M (1973) Visual stability and voluntary eye movements in Handbook of Sensory

Physiology VII3 Jung R (Ed) pp 307ndash331 Springer Verlag BerlinMarr D and Hildreth E-C (1980) Theory of edge detection Proc Roy Soc London B 207 187ndash

217

94 J M Zanker

Marr D and Ullman S (1981) Directional selectivity and its use in early visual processing ProcRoy Soc London B 211 151ndash180

Mather G (2000) Integrationbiases in the Ouchi and other visual illusions Perception 29 721ndash727Nakayama K and Silverman G H (1988) The aperture problem I Perception of non-rigidity and

motion direction in translating sinusoidal lines Vision Research 28 739ndash746Ouchi H (1977) Japanese Optical and Geometrical Art Dover Publications New YorkPatzwahl D R and Zanker J M (2000) Mechanisms for human motion perception combining

evidence from evoked potentials behavioural performance and computational modelling Eur JNeurosci 12 273ndash282

Reichardt W (1987) Evaluation of optical motion information by movement detectors J CompPhysiol A 161 533ndash547

Reichardt W and Schloumlgl R W (1988) A two-dimensional eld theory for motion computationFirst order approximation translatory motion of rigid patterns Biol Cybern 60 23ndash35

Riley B (1999) The Eyersquos Mind Bridget Riley Collected Writings 1965ndash1999 Thames and HudsonLondon

Ross J Morrone M C and Burr D C (1997) Compression of visual space before saccades Nature386 598ndash601

Spillmann L (1993) The perception of movement and depth in moire patterns Perception 22 287ndash308

Srinivasan M V (1990) Generalized gradient schemes for the measurement of two-dimensionalimage motion Biol Cybern 63 421ndash431

Srinivasan M V and Dvorak D R (1980) Spatial processingof visual information in the movement-detecting pathway of the y J Comp Physiol 140 1ndash23

Torre V and Poggio T (1978) A synaptic mechanism possibly underlying directional selectivity tomotion Proc Roy Soc London B 202 409ndash416

van de Grind W A Koenderink J J and van Doorn A J (1986) The distributionof human motiondetector properties in the monocular visual eld Vision Research 26 797ndash810

van Doorn A J and Koenderink J J (1982) Visibility of movement gradients Biol Cybern 44167ndash175

van Santen J P H and Sperling G (1985) Elaborated Reichardt detectors J Opt Soc Am A 2300ndash321

von Helmholtz H (1924) Treatise on Physiological Optics Dover New Yorkvon Holst E and Mittelstaedt H (1950) Das Reafferenzprinzip Wechselwirkungen zwischen

Zentralnervensystemund Peripherie Naturwiss 20 464ndash476Wade N J (1982) The Art and Science of Visual Illusions Routledge and Kegan Paul LondonWallach H (1935) Uumlber visuell wahrgenommene Bewegungsrichtung Psychologische Forschung

20 325ndash380Zanker J M (2001) Combining local motion signals a computational study of segmentation and

transparency in Motion Vision ComputationalNeural and Ecological Constraints Zanker J Mand Zeil J (Eds) pp 113ndash124 Springer Verlag Berlin

Zanker J M Hofmann M I and Zeil J (1997) A two-dimensionalmotion detector model (2DMD)responding to arti cial and natural image sequences Invest Ophthalmol Visual Sci 38 S 936

Zanker J M Doyle M and Walker R (2003) Gaze stability of observers watching Op Art picturesPerception 32 1037ndash1049

Zeki S (1994) The cortical Enigma a reply to Professor Gregory Proc Roy Soc London B 257243ndash245

Zeki S (1995) Phenomenalmotion seen througharti cial intra-ocularlenses Proc Roy Soc LondonB 260 165ndash166

Zeki S (1999) Inner Vision An Exploration of Art and the Brain Oxford University Press OxfordZeki S Watson J D G and Frackowiak R S J (1993) Going beyond the information given the

relation of illusory visual motion to brain activity Proc Roy Soc London B 252 215ndash222

76 J M Zanker

1 INTRODUCTION

Thomas Gainsboroughrsquos famous dictum lsquoPainting is a science and should bepursued as an inquiry into the laws of naturersquo (quoted in Zeki 1999) makesneuroscientists curious to learn from their artistic companions about brain functionand at the same time puts the artist in a rather challenging position Throughoutthe history of visual arts we encounter in this pursuit of the laws of naturerepeated attempts to develop a pictorial language for representing features of afour-dimensional world which defy capture on an essentially two-dimensionalpainting surface depth and motion The twentieth century which was marked bythe invention and development of kinematic media (cinema television internet)witnessed various attempts to make motion sensations an integral part of visualarts (such as kinetic arts) and Op artists like Bridget Riley began to experimentwith simple black and white patterns that can create vivid dynamic illusions instatic pictures (Riley 1999) Simple designs such as concentric areas lled withoblong checkerboard patterns of different orientations (Ouchi 1977) attractedmuch interest from psychophysicists (Hine et al 1997 Khang and Essock 1997Mather 2000)

The physiological and perceptual mechanisms that might underlie the illusoryperception of motion in Op Art paintings are a matter of ongoing debate (seeWade 1982 for an introduction and many examples) Ever since the earlydescriptions by Purkinje and von Helmholtz (1924) of dynamic deformationsobserved in patterns composed of ne lines the importance of optical effectsrelated to eye movements and accommodation has been discussed repeatedly andhas been countered by various views that peculiarities of the neural representationcan lead to dynamic image distortions A recent example of such a dispute isthe lsquoEnigmarsquo painting by I Leviant which elicits a variety of moireacute pattern andmotion illusions (Leviant 1996) lsquoEnigmarsquo consists of a pattern of radiating blackand white lines similar to the ray pattern used by MacKay (1957a) on whichconcentric rings of uniform colour are superimposed On the one hand the sensationof a moireacute pattern itself has been attributed to fading afterimages of the originalimage after saccadic eye movements (MacKay 1957b Gregory 1993) whereas theshimmering and deformations that can be seen in the radial line patterns have beeninterpreted as rather trivial consequences of accommodation uctuations (lsquohuntingfor accomodationrsquo Gregory 1994) following an initial suggestion by Campbelland Robson (1958) On the other hand a rather different effect a circular motionwithin the uniformly coloured rings has been attributed to a so far ill-de nedcortical mechanism that does not rely on image shifts This suggestion was basedon evidence from functional brain imaging that suggests some speci c activationin motion-speci c human brain regions related to the perceived illusion and theobservation that aphakic patients can perceive the illusion (Zeki et al 1993 Zeki1994 1995) However this dispute seems to suffer from three unresolved issues(a) The arguments are often stated as alternatives that exclude each other rather thansuggesting a variety of causes that can lead to similar effects What if for instance





78 J M Zanker





2 METHODS









80 J M Zanker








82 J M Zanker











84 J M Zanker








86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker






























78 J M Zanker





2 METHODS









80 J M Zanker








82 J M Zanker











84 J M Zanker








86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker




























2 METHODS









80 J M Zanker








82 J M Zanker











84 J M Zanker








86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker


























80 J M Zanker








82 J M Zanker











84 J M Zanker








86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker





























82 J M Zanker











84 J M Zanker








86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker






























84 J M Zanker








86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker






























86 J M Zanker










88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker































88 J M Zanker






4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker




























4 CONCLUSIONS




90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker


























90 J M Zanker






92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker





























92 J M Zanker





Acknowledgements


REFERENCES



















217

94 J M Zanker



























Acknowledgements


REFERENCES



















217

94 J M Zanker


























94 J M Zanker


























looking at op art from a computational viewpoint · 2004-03-26 · looking at op art from a...

Documents