visual-proprioceptive intermodal perception in infancyutsc.utoronto.ca/~marksch/schmuckler...

INFANT BEHAVIOR AND DEVELOPMENT 19,22 l-232 (1996)

Visual-Proprioceptive Intermodal Perception in Infancy

MARK A. SCHMUCKLER Universiry of Toronto at Scarborough

Three experiments investigated visual-proprioceptive intermodal perception in infancy. In these studies, 5-month-old infants manually explored a hidden object while simultaneously viewing an on-line image of their own hidden limb (the contingent display) and a prerecorded videotape of a different infant’s limb exploring that same object (the noncontingent display). In Experiment 1, infants looked preferentially towards the noncontingent display, relative to the contingent display, replicating previously reported findings. Experiment 2 reversed the left-right spatial directionality of visual and proprioceptive inputs and found that infants no longer preferred either display. In Experiment 3, infants saw their hidden limb from a novel, unique station point. This experiment similarly found evidence for intermodal perception, with preferential fixation of the noncontingent display relative to the contingent display. Overall, these findings highlight the importance of both temporal contingency and spatial congruence in visual-proprioceptive intermodal perception and have implications for work investigating the development of self-recognition.

intermodal perception visual-proprioceptive integration

Perceptual and cognitive functioning involves coordinating information arising from different perceptual systems. When moving through the world, adults continuously encounter an array of information, available simultaneously from multiple sensory and perceptual systems. Rather than experiencing separate, independent inputs, however, our experience of this multi- tude of information is of unified, distinct objects and events; the ability to form these unified percepts is commonly referred to as “intermodal perception” or “intermodal recognition” (Spelke, 1987). The growth of intermodal perception has fascinated both psychologists and philosophers for years, with our understanding of the processes underlying the formation of intermodal relations addressing very basic ques- tions of our ability to perceive and act within the world (Abravanel, 198 1; Bushnell, 198 1).

Investigations of infants’ abilities to perceive intermodal relationships have, by now, a relatively long history in developmental psy-

This research was supported by a grant from the Natural Sciences and Engineering Research Council of Canada to M.A.S. The author would like to thank Lorraine Bahrick, John Kennedy, Philippe Rochat, and Arlene Walker-Andrews for their helpful comments on earlier drafts of this manuscript. Portions of this work were presented at the 9th International Conference on Infant Studies, June, 1994, Paris, France.

Correspondence and requests for reprints should be sent to Mark A. Schmuckler, Division of Life Sciences, University of Toronto, Scarborough Campus, Scarborough, Ontario, Canada MlC lA4, e-mail: [email protected]. utoronto.ca.

chology and have successfully documented numerous forms of intermodal coordination early in life (for reviews, see Bushnell & Boudreau, 1991, 1993; Lewkowicz & Lickliter, 1994; Rose & Ruff, 1987). For example, there is now reliable evidence that neonates coordinate visual and auditory space, in that they look towards and/or turn their heads to visually localize an auditory sound source (e.g., Clifton, Morrongiello, Kulig, & Dowd, 1981; Muir & Field, 1979; Wertheimer, 1961). Similarly, there exists evidence for visual-haptic intermodal coordination as early as 1 month (Gibson & Walker, 1984; Meltzoff & Borton, 1979). Although this evidence of visual-haptic intermodal perception at such a tender age has met with some scepticism by researchers, recent reviews concur in their acceptance of reliable evidence of intermodal perception, both visual-haptic and visual-auditory, by about the age of 5 months (Bushnell & Boudreau, 1991, 1993; Rose & Ruff, 1987).

Most of these investigations of intermodal perception focus on understanding the integration of the different perceptual systems operat- ing during object and surface exploration. In addition, one can question whether intermodal information is similarly used for obtaining knowledge of one’s own body position and/or posture in space. In this case, the question becomes how one integrates information arising from vestibular and proprioceptive sources with visual, auditory, and haptic inputs (Schmuckler, 1995).

221

222 Schmuckler

Although not as thoroughly researched as other forms of intermodal perception, aspects of intermodal knowledge of body position and posture have been examined by researchers. For example, research in spatial orientation has explored the use of proprioceptive, vestibular, and visual information for keeping track of and the cognitive updating of people’s ideas of their body position in space (Rieser, 1979; Rieser, Guth, & Hill, 1986; Rieser & Rider, 1991; Rider & Rieser, 1988; Schmuckler, 1995). A different example of work investigating intermodal information specifying knowledge of body posture and position involves research on using visual information for postural control. Numerous studies in this vein have convincing- ly demonstrated that visual information is critical for maintaining an upright stance from infancy onwards (Bai, 1991; Bertenthal & Bai, 1989; Butterworth & Hicks, 1977; Delorme, Frigon, & Lagace, 1989; Gapenne & Jouen, 1994; Jouen, 1984; Lee & Aronson, 1974; Lee & Lishman, 1975; Schmuckler, 1995, in press; Schmuckler & Gibson, 1989; Stoffregen, Schmuckler, & Gibson, 1987).

In both of these lines of research, there is strong evidence that toddlers and young children coordinate input from different sources (e.g., visual, vestibular, and proprioception) to produce information concerning their body position (see Schmuckler, 1995, for a review). It is important to note, though, that the evidence for intermodal perception of body position in these studies is indirect, at best. Consequently, although these results are consistent with the idea that intermodal information is used by observers for knowing where they are in space, these studies do not provide evidence for intermodal perception per se.

Relatively recently, two series of experiments have directly examined intermodal perception of body position in younger infants. Investigating 5-month-olds, Bahrick and Watson (1985) manipulated the availability of visual and proprioceptive information arising from leg movements by hiding infants’ legs while simultaneously presenting these infants with a video display containing an image of the movement of their unseen legs. Along with this video display of their own movement, infants also saw a video display containing the previously recorded movement of a different child. Hence, one of these displays contained a stimu-

lus in which the visual information matched the proprioceptive information arising from the hidden limb movement, whereas the other display did not match. Analyses of visual fixations to these two displays indicated significant preferential looking towards one of the two displays; such preferential looking suggested that infants in these studies did, in fact, detect visual-proprioceptive contingencies. Fascinatingly, infants did not look preferentially at the display of their own hidden limbs, but instead they looked more at the display that was not contingently related to their own movement, a result in keeping with earlier studies looking at preferential fixation of contingent and noncontingent displays (Papousek & Papousek, 1974). Although the reason for the direction of such preferential looking is unclear, the results indi- cate intermodal perception by 5-month-old infants of the relation between visual and proprioceptive information arising from limb and body movements.

Following on the heels of Bahrick and Watson (1985), Rochat and Morgan ( 1995) fur- ther explored visual-proprioceptive intermodal perception by 3- and 5-month-old infants. The goal of this work was to thoroughly examine the role of spatial congruence in intermodal perception while holding constant temporal contingencies between visual and proprioceptive information. Towards this end, infants saw two on-line, spatially manipulated video images of their moving legs; the primary dependent measures were their preferential looking and a com- parison between looking behavior and actual leg movement. In Rochat and Morgan’s first study, the spatial orientation (ego- vs. observer- based points of view) and directionality of movement (left-right orientation) were varied in one display, with the other display retaining the appropriate spatial orientation and directionality of movement. Their second study manipulated only the directionality of movement, whereas the third experiment varied spatial orientation alone. The results of this work indicated that infants were sensitive to the directionality of movement in these displays, exhibiting preferential looking towards a left-right reversed display, along with producing more leg activity while fixating this display. In contrast, the spatial orientation of the displays did not influence either measure. Although it is unclear whether this preferential fixation indi-

Visual-Proprioceptive Perception 223

cates a failure to detect the temporal contingency in the left-right reversed display (thereby causing this display to be perceived by infants as noncontingent) or whether the spatial mis- match actually drove preferential looking, irre- spective of the temporal contingency, these findings do suggest that intermodal perception of body movement is at least partially dependent upon information about movement directionality. In contrast, spatial orientation, in the form of the point of observation, appears not to be critical for such intermodal perception.

Although the studies of Bahrick and Watson (1985) and Rochat and Morgan (1995) con- verge on the same general findings, these experiments differ in some critical ways. In Bahrick and Watson’s study, infants saw a contingently moving, on-line presentation of their own moving legs paired with a noncontingent film of moving legs (either their own or those of another infant). These two displays always shared spatial relations, in that they both presented the same egocentric view of the children’s legs (the view children would naturally see if they were looking at their legs in front of them). Thus, these studies highlight the importance of temporally contingent information, with infants demonstrating detection of such information via their preference for the noncontingent display. In contrast, Rochat and Morgan presented two on-line, contingent displays- thereby holding temporal information constant-while manipulating the spatial relations of the two displays. Thus, Rochat and Morgan’s work speaks most directly to sensitivity to spatial relations in intermodal perception, and not necessarily to the importance of temporal information.

The goal of the following series of studies was to confirm the findings of Bahrick and Watson (198.5) and Rochat and Morgan (1995) and extend these results. One extension involved exploring visual-proprioceptive perception of limb movements other than legs and feet, thus, inter-modal perception of arm, hand, and finger movements was examined. Although there is no principled reason for infants to rec- ognize leg but not hand movements, positive findings from this manipulation strengthens the argument that infants are actually inter-modally integrating visual and proprioceptive information per se and not just recognizing specific body parts and/or movements.

These studies also extend these findings by manipulating both spatial relations and temporal contingencies and assessing the impact of these manipulations on intermodal perception. This goal was accomplished by combining methodological aspects of Bahrick and Watson (1985) and Rochat and Morgan (1995). Specifically, infants always saw a contingently moving, on-line presentation of their limbs paired with a noncontingent display (similar to Bahrick & Watson, 1985)-in this fashion, these studies investigate the detection of temporal correspondences. Across the three studies, the spatial relations and/or point of view for both the contingent and noncontingent displays were manipulated-accordingly, these studies assess the impact of such information on the detection of temporal contingencies.

EXPERIMENT 1 Intermodal Perception

of Arm and Hand Movements

The purpose of this initial study was to determine whether the visual-proprioceptive intermodal perception results observed by Bahrick and Watson (1985) could be replicated with arm and hand movements, as opposed to leg movements. To test intermodal perception, an infant manually explored a hidden toy with his or her right hand, while viewing two video images; one showed his or her own hand and the other showed another child’s hand in the same situation. Similar to Bahrick and Watson, this study examined detection of temporal contingencies between visual and proprioceptive information; spatial relations were held constant, in that both displays presented egocentic views of the hidden limbs. Based on Bahrick and Watson’s results, detection of intermodal visual-proprioceptive temporal correspondences should be evident by preferential fixation of the noncontingent display.

Method

Results are reported on 18 infants (9 males, 9 females) with a mean age of 5.1 months (range = 4.8-5.3 months). A large number of additional infants (n = 24) began this experiment but were ultimately not used in the final analyses. Of these 24 infants, 21 were removed prior to reliability coding. The reasons for removing these 21 infants include failing to complete either one or both of the preference trials due to excessive fussiness (n = 15). continuously removing their hands from the box containing the hidden object (n = 2)

224 Schmuckler

and experimenter error (n = 4); more detailed explanations for this high attrition rate are discussed subsequently. The 3 remaining infants were removed because their looking times in the preferential looking procedure proved unreliable. These infants were recruited through a local hospital as well as other available records (thereby constituting a convenience sample) and were initially contacted by letter and/or telephone. All infants were recruited from the Scarborough, Ontario, community and were given a toy and certificate for participating.

Stimuli

The stimuli for this experiment consisted of two video displays. One of these displays was an on-line picture of an infant’s right hand exploring a hidden toy. This stimulus will be referred to as the “contingent” stimulus, given the relationship between the visual movement of the display and the child’s own proprioceptive feedback of this movement. The second display consisted of a previously recorded videotape of a different child playing with the same toy. shown from the same angle. This stimulus will be referred to as the “noncontingent” display. Each child saw the videotape of the hand of a child who had previously com- pleted the experiment; this procedure is similar to Bahrick and Watson (1985).

Apparutus

Each infant was run in an experimental chamber covered with acoustic panelling. This room was lit by a pair of overhead spotlights, providing dim, but not dark, illumination. During this experiment, each infant faced a pair of Sony 14.in. (34.66 cm) CVM-194 color video monitors positioned on a table directly in front of him/her. These monitors were seated approximately 25.4 cm atop of the table, separated by about 20.3 cm. Orange curtains were suspended from the ceiling and draped behind the video monitors, blocking off all view of what lay behind these monitors. A JVC GF-700 video camera was positioned in between the two video monitors such that it could be focused on the infant’s face, with the lens of the camera placed through a slit made in the orange curtains. An observer hidden behind these curtains adjusted this camera so that it maintained a clear image of the infant’s face.

Positioned in front of the table on which the video monitors rested, and situated directly between these two monitors, was a cardboard box that was 81.3 cm tall, 54.6 cm wide, and 43.2 cm deep. This cardboard box had holes cut into the front end (the end facing the infant) and the back end (the end abutting the table with the video monitors), and the inside was lined with black construction paper. The inside of this box was lit using a small clip-on light attached to the back end of the cardboard box near its base. Approximately 6.4 cm from the front end of the box was suspended a small rubber orange elephant hanging in front of the opening in the front end of the box. At the back end of the box was a JVC GS-CDIU video camera, positioned at an angle such that it viewed the front of the box. This camera afforded a view of the suspended toy as well as a view of the infant’s hand when it was placed through the hole in the front end of the box. If this camera is oriented normally (e.g., rightside up), an observer-oriented view is produced; such a view reverses left-right spatial directionality. Accordingly, to achieve an egocentric view of the hand and toy, in which left-right is not reversed, the JVC GS-CDIU video camera must be (and was) positioned

upside down. The hand and toy appeared in color on the video monitors.

Each infant participated in this study while seated on a parent’s lap. The parent sat in a chair situated at a right angle to the front end of the box, facing the side wall and not the video monitors. The infant sat on the parent’s right leg, directly in front of the cardboard box, facing the video monitors. In this position, it was possible to place the infant’s right hand and arm through the opening in the front end of the box, so that the infant could grasp and explore the suspended toy, although it was not possible to remove the toy from the box.

Located in an adjacent control room were three video decks controlling the presentation and recording of both the stimuli and the infant’s face. The image from the JVC GS- CDlU (the infant’s hand) was recorded by a JVC BR- 8600U video deck, with the video output of this deck fed into one of the Sony CVM-194 monitors located in the experimental room; this apparatus provided the on-line video image of the child’s hand movements. The second Sony CVM-194 monitor received its image from a previously recorded videotape played on a JVC 630U video deck. The image of the infant’s face was recorded using a Sharp VCR. A second experimenter, seated in this adjacent control room, controlled the presentation of the stimulus displays to both Sony monitors in the experimental room and recorded the amount of time the infant spent looking at the two Sony monitors using the left and right buttons of a computer mouse. This mouse was attached to an IBM-PC compatible computer, which, along with cumulating the time spent looking at the different stimulus displays, kept track of the length of the experimental trials and signalled the end of the trial to the experimenter. Although this observer was situated in the control room, he/she was blind to the side of presentation on which the contingent and noncontingent displays appeared to the infant.

Procedure

Each infant was tested when he/she was awake and appeared alert, A parent accompanied the child into the experimental chamber, and the experimenter situated the parent and child appropriately in front of the cardboard box, facing the video monitors. The parent then placed the infant’s right hand into the hole in the front of the cardboard box so that the infant grasped the suspended toy and then gently held the child by the elbow so that the child was unable to remove his/her arm from the box.’ Parents were instructed not to look at the video monitors themselves nor to interact with or distract their child during the experiment.

’ Given this procedure, the high number of infants failing to complete this study due to fussiness becomes under- standable. Infants were required to hold their arms in a box, playing with a toy that they could not see, nor could they remove their arms from this box. Needless to say, this situation caused a great deal of distress, resulting in many infants failing to complete the experiment. In fact, the majority of infants who did not complete this study never actually began the experiment. Essentially, the moment their arm was placed in the box with the hidden toy they became extremely upset; the procedure was generally ter- minated at this point, without these infants ever seeing the visual displays.


Once the parent and infant were positioned correctly, and the child had hold of the suspended toy, the experimenter in the control room began the study by turning on the two stimulus displays. One of these two displays showed the contingent stimulus, while the other monitor presented the noncontingent display. The infant observed the two stimulus displays for 1 min, with the amount of time spent looking at each of these two displays recorded. After this trial, the displays were turned off, and a short break occurred. After this break, the infant’s hand was again placed in the cardboard box, and a second 1 -min trial was run. In this second trial, the sides on which the contingent/noncontingent displays appeared were reversed. Subsequent to this second trial, parents were debriefed. For one group of infants, the contingent display appeared on the right monitor during the first trial and on the left monitor during the second trial, whereas for the remaining infants, this arrangement was reversed. The actual experimental phase took approximately 10 min. with the entire visit by the parent and infant to the laboratory lasting about 30 to 45 min.

Reliability

Because the infant’s face was recorded on videotape during this experiment, it was possible to perform reliability assessments of the amount of time spent looking to the left and right monitors from these tapes. Subsequently, a second observer coded all visual fixations to the left and right monitors for both trials. This observer was also unaware of the left-right positioning of the contingent and noncontingent displays. Looking times were considered unreliable if the difference between the original and reliability coding qualitatively changed the pattern of preferential fixation (e.g., more looking to the contingent, relative to the noncontingent, for the original coding, but the reverse pattern for the reliability coding). Generally, looking times proved quite reliable. The mean absolute difference in the two measures of looking times, averaged across the left and right monitors and the two experimental trials, was 1.3 s (SD = 1.2 s). The original and reliability codings were also strongly correlated, ~(70) = .98, p < ,001.

Results and Discussion

For each infant, the proportions of looking times toward the two displays were calculated by dividing the time spent looking towards either display by the total time spent looking towards both displays.* These proportions were then turned into difference scores by sub- tracting the proportion of time spent looking

* Along with their preferential looking data, Rochat and Morgan (1995) provided the compelling dependent measure of looking as a function of leg activity. Unfortunately, the experimental apparatus used in these studies was not designed such that analogous information was obtained, nor was subsequent coding of the videotapes of arm and hand movements able to reliably recover this information. The inability of this work to provide any information concem- ing this relationship is unfortunate.

towards the contingent display from the proportion of time spent looking towards the noncontingent display. A two-way analysis of variance (ANOVA) of these difference scores failed to reveal any effects of either order (contingent on the left during the first trial vs. contingent on the right during the first trial), F( 1, 16) = 0.60, ns, or repeated trials (Trial 1 vs. Trial 2) F(l, 16) = 0.21, ns, nor was there an interaction between the two, F( 1, 16) = 0.21, ns; accordingly, all data were collapsed across these factors. Difference scores were then compared to zero (which represents 50%-50% preferential viewing) using a t test. This analysis revealed significant preferential looking towards the noncontingent display (M proportion looking time = 61.0%, SD = 20.2%) relative to the contingent display (M proportion looking time = 39.0%, SD = 20.2%), t(17) = 2.67, p < .05; these proportion looking times are shown in Figure 1. This pattern of preferential looking occurred for 14 of the 18 infants in this study, a significant difference by a sign test, x2( 1, N = 18) = 4.50, p < .05.

This experiment corroborates the results of both Bahrick and Watson (1985) and Rochat and Morgan (1995) and adds that visual-

Experiment 1 Experiment 2 Experiment 3

Fiiure 1. The mean proportion of lookina times, and dndard errors, toGards he contingent Gnd no&n- tingent visual displays for Experiments 1, 2, and 3. Equivalent looking (SO’%) towards the two displays is notated with a dotted line.

226 Schmuckler

proprioceptive intermodal perception occurs with hand and arm movements as well as leg movements. Just as in these earlier studies, 5- month-old infants demonstrated intermodal perception by preferentially attending to the movements of a different infant. Is it justified to assume that infants were performing visual- proprioceptive intermodal perception? One might be concerned that infants in this study simply recognized features of their hands (relative to the different child on the other display) and discriminated the two displays on the basis of this familiarity, rather than in terms of any form of motion contingency between visual and proprioceptive information. Although this is a possible concern, both Bahrick and Watson and Rochat and Morgan provided compelling evidence that simple familiarity (in terms of recognizing one’s own hand vs. another’s) cannot account for this intermodal perception. Bahrick and Watson addressed this concern by making a videotape recording of the child’s leg movements and using this pre-recorded tape as the noncontingent display. This technique not only equates the displays in terms of the featural information that would underlie recognizing one’s own limbs but also equalizes the two displays for characteristics such as general movement patterns and overall activity level. Even when the displays were equated in this fashion, infants in their studies still preferentially fix- atcd the noncontingent display, as they did in the case reported here.

Rochat and Morgan (1995) addressed this issue by having, in all three of their studies, both displays contain on-line images of the infants’ legs; obviously, such a technique also equates for the familiarity and movement fat- tors described above. Once again, the results of Rochat and Morgan concur with Bahrick and Watson’s (1985) findings, as well as our data: Infants preferred the noncontingent display. Given the independent replication of the basic result in three different laboratories employing divergent experimental paradigms, these studies provide extremely strong evidence that these preferential looking patterns towards the contingent and noncontingent displays are not a function of recognizing familiar features or movements. Instead, such preferential looking appears to be based on intermodal perception of a contingently moving display.

EXPERIMENT 2 The Role of Left-Right Spatial

Directionality

Experiment 2 investigated the importance of the spatial directionality of children’s movement on visual-proprioceptive intermodal perception by reversing the left-right spatial orientation of the hidden limb. As already described, Rochat and Morgan (199.5) provided a similar manipulation for the image of children’s legs and observed that, for two temporally contingent displays, infants preferentially fixated the left-right reversed display. relative to a display in which left-right orientation was retained. In Experiment 2, both displays were left-right reversed, with the on-line view of the display retaining its temporal contingency with children’s movements. Accordingly. this study examined whether left-right spatial alignment was necessary for the detection of temporally contingent information. If left-right orientation is not required, then infants should again preferentially fixate the noncontingent display. If, however, this information is a prerequisite for perceiving temporal contingencies, but is now made discrepant by inverting left-right. the prior pattern of results should no longer hold: that is, the preference for the noncontingent display will disappear.

Method

Kesult\ :,1-t’ reported on 22 infants (9 males, I3 females) with 3 mean qc of 3.9 month\ (range = 4.6-5.3 months). Thirteen other infant? began this experiment but were not usxl due to a failure tn complete the study because of fussi- ncs\ and/or removing their hands from the cnrdboard box (II = 9). an irremovable bracelet on the child’s hand/arm (II = IL the parent putting the wrong hand in the cardboard hox (II = I J, and a fatlure to achieve reliability on the child’\ visual fixntionh 01 = 2). infants were recruited a\ in Experiment I.

As in Experiment I, the stimuli for this experiment cow si\ted of contingent and noncontingent video display?. The contingent display was again an on-line image of the iw fant‘\ O\III hand exploring the suspended toy. and the now contingent display consisted of a previously recorded videotape of a different child’\ hand playing with this sm,c toy. Each child saw the videotape of the hand of a child who had previously taken part in the experiment.

Thi\ study was run using the same apparatu\ as m Experiment I. with the following exceptions. First. the JVC GS-CDI U video camera producing the video image uf the child’s hand was placed rightsideup, as opposed to upside- down, as in Experiment I. This produced an image in which the left-right spatial axis of the image was reversed. Rccaux the noncontlngent display was actually a recording


of the previous child in this study, this display was also left-right reversed. Second, the computer mouse used for recording looking times towards the left and right monitors was moved into the experimental room so that the experimenter hidden behind the orange draperies coded visual fixations. All other aspects of the experimental setup and procedure were similar to Experiment I.

As in Experiment I, a second observer who was similarly naive to the positioning of contingent and noncontingent displays calculated a second measure of the time spent looking towards the two monitors using the videotaped records of the experimental trials. The looking time measures were again reliable: the mean absolute difference in looking times, averaged across the left and right monitors and the two experimental trials, was I.5 s (SD = I.4 s). Original and reliability codings were strongly correlated, r(86) = .98, p < ,001.


For each infant, the proportions of time spent looking towards the contingent and noncontingent displays were calculated as in Experiment 1, and difference scores were then computed. Once again, a two-way ANOVA indicated that there were no significant differences as a function of either order, F( 1, 20) = 0.92, ns, or repeated trials, F( 1, 20) = 0.20, ns, nor was there a significant interaction between the two, F( 1, 20) = 0.65, ns. Most interestingly, and in contrast to Experiment 1, 5-month-old infants did not preferentially fixate either the noncontingent display (M proportion looking time = 52.1%, SD = 23.6%) or the contingent display (M proportion looking time = 47.9%, SD = 23.6%) t(21) = 0.72, p = .48; these proportions are portrayed graphically in Figure 1. Fourteen of the 22 infants looked more at the noncontingent display than the contingent display, an insignificant difference by a sign test, X2(l,N=22)= 1.14,ns.

The import of these results is straightforward. When the left-right spatial contingency between proprioceptive and visual information was reversed, Smonth-old infants did not show preferential viewing. The most obvious explanation for this result is that, with the left-right spatial reversal, infants failed to detect the temporal contiguity between their movements and the image on the contingent display. Alter- natively, these results could have emerged through competing tendencies to watch the noncontingent display (as in Experiment 1) and to watch the contingent display, because it sud- denly became interesting and/or novel given the left-right spatial reversal; unfortunately,

choosing between these alternatives is not possible with these data. Regardless of which explanation is true, these results do suggest that infants are aware of the left-right spatial alignment of these displays, and that such alignment is likely necessary for detecting temporal contingencies.

This result also corroborates the findings of Rochat and Morgan (1995), who similarly found that reversing left-right spatial directionality disrupted intermodal perception. Intriguingly, these two studies reached the same conclusion via different experimental paradigms. In Experiment 2, pairing a noncontingent, left-right inverted display with a contingent, left-right inverted display failed to produce preferential fixation in infants. In Rochat and Morgan’s work, both displays were temporally contingent, with one display retaining the proper spatial alignment, and the other display reversing left and right. In this case, infants gazed preferentially towards the inverted display. As just discussed, both sets of experiments imply that spatial contiguity is critical for visual-proprioceptive intermodal perception, albeit through varying procedures and differing patterns of results. Together, these studies provide a nice example of the principle of converging operations (Garner, Hake, & Erikson, 1956; Proffitt & Bertenthal, 1990), with different experimental setups, procedures, and findings leading to the same con- clusions regarding the importance of spatial directionality in visual-proprioceptive intermodal perception.

One potential concern with this study is that the method by which spatial directionality was manipulated actually produced distortions other than the left-right spatial organization of the world. Along with a difference in terms of the direction of movement of arm and hand movements, the position of the camera similarly resulted in an unusual view of an infant’s hand, one in which the position of the fingers was reversed relative to normal viewing. For example, the thumb on the right hand appeared on the outside rather than the inside of the hand, relative to the body. Unfortunately, it is simply not possible to invert the left-right spatial axis of the world without simultaneously reordering the digits on the hand. Despite this inevitable confound between the two, it seems unlikely

228 Schmuckler

that this simple reordering of the digits on the hand was a determining factor in this study. First, it is important to remember that the thumb is not always on the inside of the hand, relative to the body, given that we can rotate our hands/arms 180”. Such rotations, which position the palm upwards, also place the thumb on the outside of the hand. Accordingly, the thumb being located on the inside of the hand is not invariant. Given that infants moved their hands during this study, the exact orientation of the thumb relative to the body varied in an uncontrollable way; as such, it is unlikely to provide a constant confound. Second, Rochat and Morgan (1995) covered infant’s legs with socks, hiding the individual toes. Despite this unavailability of digit information, spatial directionality was still the determining factor in intermodal perception in their study. Given the already observed close correspondence between these studies, it seems unlikely that this issue is of critical import in this experiment.

EXPERIMENT 3 Intermodal Perception

in Unusual Orientations

The goal of Experiment 3 was to the role of orientation of viewpoint den limb on visual-proprioceptive

investigate of the hid- intermodal

perception. Once again, Rochat and Morgan (1995) examined the effect of the point of observation by providing infants with two temporally contingent and spatially congruent displays, one of which was seen from an egocentric view, whereas the other presented an observer’s view of the infants’ legs. In this study, infants failed to look preferentially towards either display. Although it is unclear from such a pattern of results whether infants were treating both displays as contingently or noncontingently related to proprioceptive input, these results suggest that the point of observation (observer vs. egocentric) matters little in visual-proprioceptive intermodal perception. This study extends these earlier findings by (a) presenting infants with a different and unusual view of the their hidden limba view of the hand from underneath the palm, when their hand is oriented horizontally-and (b) providing a situation that can determine whether infants see such a display as contin-

gently related or not, despite the variation in viewpoint. Evidence of preferential looking in this study would imply, similar to the findings of Rochat and Morgan, that specific viewpoint or orientation is not critical for intermodal perception.

Method

Results are reported on 20 infants (10 males, 10 females) with a mean age of 4.9 months (range = 465.0 months). Twelve additional infants began this experiment but were not used in the final analyses due to a failure to complete the study because of fussiness (n = 7), experimenter and/or equipment error while running the experiment (n = 3). and a failure to look at the stimuli during the experiment (n = 2). Infants were recruited as in Experiments 1 and 2.

The stimuli for this experiment consisted of contingent and noncontingent video displays similar to those of Experiments 1 and 2. Again, each child saw his/her own hand in one image and the image of a child who had just previously taken part in the experiment.

The same apparatus was used as in Experiments 1 and 2, with the following exception. The JVC GS-CDIU video camera producing the video image of the child’s hand was placed on the floor underneath the child’s hand, pointing upwards. This produced an image in which the left-right spatial axis was correct, but now the palm of the hand was in view, with the arm and wrist at the top of the screen and fingers pointing downwards. The experimental setup and procedure were similar to Experiments 1 and 2.

As in Experiments 1 and 2, a second (naive) observer measured fixation times using the taped records of the experimental trials. These looking time measures were once again reliable, with an absolute difference, averaged across the left and right monitors and the two experimental trials, of 1 .S s (SD = 1.5 s). The correlation between the two codings was again significant, r(78) = .97, p < .OOl_


As before, proportions of time spent looking towards the contingent and noncontingent displays were calculated, and difference scores were created. Again, a two-way ANOVA indicated that there were no differences as a function of order, F( 1, 18) = 3.12, ns, or repeated trial, F( 1, 18) = 0.14 izs, nor was there a significant interaction between the two, F( 1, 18) = 0.45 11s. Similar to the results of Experiment 1, but in contrast to Experiment 2, infants preferentially fixated the noncontingent display (M proportion looking time = 56.8%, SD = 19.7%) relative to the contingent display (M proportion looking time = 43.2%, SD = 19.7%). This difference, also shown in Figure 1, was statisti- cally significant, t(19) = 2.44, p < .05. Fifteen of the 20 infants looked more at the noncontingent display than the contingent display, a sig-


nificant difference by a sign test, x2( 1, N = 20) = 4.05, p < .05.

The implications of this experiment are straightforward. Five-month-old infants once again demonstrated visual-proprioceptive intermodal perception, even though the displays contained an unusual view of the hand. Accordingly, the specific view of the hand, provided that it retain the appropriate left-right spatial directionality, does not appear to matter for visual-proprioceptive intermodal perception. Similar to Experiment 2, this study replicates and extends Rochat and Morgan (1995) by providing a divergent experimental setup, with a different pattern of results, that con- verges on the same conclusion.

GENERAL DISCUSSION

In summary, the results of these studies demonstrate that under certain conditions, 5-month- old infants preferentially fixate a display presenting a prerecorded image of a different child’s limb (a noncontingent display) relative to a display showing an on-line presentation of one of their own hidden limbs (a contingent display). Given the methodology of these studies, and taken in conjunction with the data of Bahrick and Watson (1985) and Rochat and Morgan (1995), it seems unlikely that infants preferentially fixated the noncontingent display because the visual image of this hand was simply more interesting or attractive, either due to familiarity of features or more interesting movement patterns.

Along with demonstrating visual-proprioceptive intermodal perception in 5-month-old infants, these studies have identified some of the perceptual information underlying this coordination. In this regard, these results demonstrate that without spatial alignment between visual and proprioceptive information, infants do not show intermodal perception; such a finding concurs with previous research (Rochat & Morgan, 1995). In contrast, the specific viewpoint or orientation of the visual display appears to be unimportant for such intermodal perception-a finding that also accords with previous research.

One important facet in this regard is the fact that, in all three of these studies, the contingent display retained a temporal congruence with infants’ movements while the spatial congru-

ence was systematically manipulated. In contrast, the noncontingent display always differed with respect to both spatial and temporal factors. The fact that temporal contingency by itself was not powerful enough to drive intermodal perception in all situations is striking given the evidence that temporal patterning can act as a powerful cue for detecting intermodal correspondence (e.g., Allen, Walker, Symonds, & Marcell, 1977; Bahrick, 1983; Lewkowicz, 1985, 1986, 1992; Spelke, 1979; Spelke, Born, & Chu, 1983). Spelke (1979), for example, systematically explored the impact of temporal relations on auditory-visual intermodal perception and found that 4-month-old infants showed increased visual search for an auditorially syn- chronized event, suggesting sensitivity to audiovisual temporal synchrony. Subsequent studies found intermodal perception of rate or tempo (e.g., fast vs. slow), even in the absence of strict temporal synchrony.

The fact that left-right spatial reversal disrupted detection of visual-proprioceptive temporal contingencies is consistent with other intermodal studies (see Bahrick & Pickens, 1994, for a review). For example, Spelke et al. (1983) found that 4-month-olds responded to auditory-visual temporal synchrony when a sound coincided with a change in the trajectory of a visual event but did not respond with no change in direction of the event. Similarly, Bahrick observed that temporally synchronous visual and auditory information must be appropriate to the substance (Bahrick, 1983) and the composition (i.e., a tube with one marble vs. many marbles) of the object (Bahrick, 1988) for intermodal perception. Other work suggests that auditory-visual information must match in affect (e.g., Walker, 1982) or the gender of the speaker (Walker-Andrews, Bahrick, Raglioni, & Diaz, 1991) for infants to perceive them as corresponding. Thus, although temporal synchrony is critical, intermodal perception is con- strained by other factors as well, suggesting that synchrony in and of itself does not drive intermodal perception.

One possibility is that spatial information is a stronger factor in perceiving intermodal relations that are, fundamentally, spatial in nature, such as might occur in visual-haptic intermodal perception or in the situations presented here. In contrast, intermodal relations based

230 Schmuckler

more heavily on temporal patterning, for example visual-auditory relations, will rely more on temporal correspondences. Accord- ingly, an important direction for future research involves exploring the importance of spatial and temporal contingencies on intermodal perception, although given the inherent limitations in dealing with infant subjects, these manipulations will likely focus on varying the visual input, relative to constant proprioceptive information.

One obvious subsequent manipulation in this regard would be to perturb or delay this visual information, producing a visual analog to the phenomenon of delayed or perturbed auditory feedback (Fairbanks, 1955; Wing, 1977). Given the critical role immediate feedback plays in recognizing an auditory signal, investigating perception of visual movement under delayed conditions would be instructive, assessing the importance of temporal contingencies directly, as well as providing a situation in which the relative importance of spatial and temporal information in intermodal perception could be compared.

Although not the primary focus of this work, the studies described in this report have implications for research and theory on the development of self-perception (Schmuckler, 1995). In terms of the growth of self-perception, researchers in this area have suggested that one of the first tasks an infant must appreciate is that “. . . he or she exists as an active causal agent, a source and controller of actions separate from other persons and objects in the world” (Hatter, 1983, p. 279). One primary method by which an infant might begin to develop this “existential” representation of self (Lewis & Brooks-Gunn, 1979) is through awareness of various contingency relations occurring in the world. Such contingencies can exist on a motoric, proprioceptive level, such as the relation between banging one’s foot against a crib and the ensuing pain (Harter, 1983), or with external objects and events (dropping a rat- tle and seeing it fall), and so on. In fact, there is a great deal of evidence suggesting infant sensitivity to many such contingencies, such as leg kicks moving an overhead mobile or sucking turning a light on and off (Rovee & Rovee, 1969; Sameroff, 1971; Watson & Ramey, 1972).

Bahrick and Watson (1985) suggested that the contingency relations involved in kicking one’s legs and moving a mobile are distinctly different from those involved in visual-proprioceptive intermodal perception. In the former situation, the stimulation between leg and mobile is conjugately related, but not perfectly correlated. In contrast, an on-line visual image of one’s own hidden moving limbs involves a perfectly contingent relation between visual and prorioceptive inputs, at least in terms of the spatial, temporal, and intensity components of the movement.’ These authors proposed that perfect contingencies arise only in situations in which the self is active, whereas imperfect contingencies may occur when other sources in the environment are active. As such, recognizing perfect contingencies provides a crucial source of information for distinguishing the actions of self from the actions of others. Whether or not visual-proprioceptive intermodal perception truly reflects an appreciation of the distinction between self and other is, in fact, debatable. Such intermodal perception might simply rep- resent a sensitivity to spatial and temporal contingencies, without any strong understanding of self. However, visual-proprioceptive intermodal perception of self-movement might rep- resent a critical first step in the development of a concept of self and learning to distinguish self from others.

Finally, one particularly intriguing result of these studies is the finding that infants dis- played visual-proprioceptive intermodal perception by preferentially watching the noncontingent display. Although this finding replicates previously reported results (Bahrick & Watson, 1985; Papousek & Papousek, 1974; Rochat &

’ There are numerous ways in which these inputs were not perfectly correlated. As a partial list, the visual information was at an incorrect distance, presented at the wrong angle, and the image of the hand was not the proper size. However, and as pointed out in the text, the inputs were correlated spatially, temporally, and in terms of the intensity of movement (small physical movements led to small visual movements). Accordingly, the use of “perfect contingency” is taken with reference to these variables. That intermodal perception was still observed despite the dis- crepancies listed here provides additional evidence that specific components of point of view are relatively unimportant for recognizing self-movement.


Morgan, 1993, the explanation underlying this somewhat nonintuitive result is unclear. Previous researchers (Bahrick & Watson, 1985) have suggested that this pattern of preferential looking represents infants’ growing responsive- ness to social stimulation produced by the outside world. Given that perfectly contingent relations provide strong information for actions of the self, a preference for noncontingent stimulation reflects a preference for stimulation by others. A related, albeit ad hoc, explanation for this preference is that the contingent display is relatively uninformative, providing redundant information to that of the proprioceptive information. In contrast, the noncontingent display provides new, unfamiliar input; accordingly, preferential fixation now represents a preference for novel information.

In sum, these studies replicate and extend the findings of Bahrick and Watson (1985) and Rochat and Morgan (1995) and suggest that 5- month-old infants coordinate visual information arising from limb movement with proprioceptive input occurring in response to that same movement. Such perception is tied to a strict spatial framework, in that disrupting spatial contingencies critically impairs intermodal perception. More generally, these results have intriguing implications for theories of intermodal coordination, and for research on the growth of knowledge of the self, and provide a novel approach for studying these issues.

REFERENCES

Abravanel, E. (198 I). Integrating the information from eyes and hands: A developmental account. In R.D. Walk & H.L. Pick (Eds.), Intersensory perception and sensory integration. New York: Plenum.

Allen, T.W., Walker, K., Symonds. L., & Marcell, M. (1977). Intrasensory and intersensory perception of temporal sequences during infancy. Dei&~pmentul Psychology, 13, 225-229.

Bahrick, L.E. (1983). infants’ perception of substance and temporal synchrony in multimodal events. Infant Behavior und Development, 6, 420-45 I,

Bahrick, L.E. (1988). Intermodal learning in infancy: Learning on the basis of two kinds of invariant relations in audible and visible events. Child De\&pment. 53, 197-209.

Bahrick, L.E., & Pickens, J.N. (1994). Amodal relations: The basis for intermodal perception and learning in infancy. In D.J. Lewkowicz & R. Lickliter (Eds.), The development of inter-sensory perception: Compurutive perspectives. Hillsdale, NJ: Erlbaum.

Bahrick, L.E., & Watson, J.S. (1985). Detection of intermodal proprioceptive-visual contingency as a potential basis of self-perception in infancy. Developmental Psychology, 21,963-973.

Bai, D.L. (1991). Visual control of posture during infancy. Unpublished doctoral dissertation, University of Virginia. Charlottesville, VA.

Bertenthal, B.I., & Bai, D.L. (1989). Infant’s sensitivity to optical flow for controlling posture. Developmental

Psychology. 25, 936945. Bushnell, E.W. (1981). The ontogeny of intermodal rela-

tions: Vision and touch in infancy. In R.D. Walk & H.L. Pick (Eds.), Intersensory perception and sensory integration. New York: Plenum.

Bushnell, E.W., & Boudreau, J.P. (1991). The development of haptic perception during infancy. In M.A. Heller 91 W. Schiff (Eds.), The psychology of touch. Hillsdale, NJ: Erlbaum.

Bushnell, E.W., & Boudreau, J.P. (1993). Motor development and the mind: The potential role of motor abilities as a determinant of aspects of perceptual development. Child De\~elopment. 64, 100-1021.

Butterworth, G., & Hicks, L. (1977). Visual proprioception and postural stability in infancy: A developmental study. Perception, 6, 2.55-262.

Clifton, R.K., Morrongiello, B.A., Kulig, J.W., & Dowd, J.M. (1981). Newborns’ orientation toward sound: Possible implications for cortical development, Child Del,elopment, 52, 833-838.

Delorme, A., Frigon, J., & Lagace, C. (1989). Infants’ reactions to visual movement of the environment. Perception, 18, 667473.

Fairbanks, G. (1955). Selective vocal effects of delayed auditory feedback. Journal of Speec,h and Hearing Disorders. 20, 333-346.

Gapenne. 0.. & Jouen, F. (1994, June). Temporal proper- ties of the \?suo-posturnl couplin,p in nerrhorns. Poster session presented at the 9th International Conference on Infant Studies, Paris, France.

Garner, W.R., Hake, H.W.. & Erikson. C.W. (19%). Operationalism and the concept of perception. Psycholo,+ul Review, 63, 149-l 59.

Gibson, E.J., & Walker, AS. (1984). Development of knowledge of visual-tactual affordancea of substance. Child Development, 55.453-460.

Harter, S. (1983). Developmental perspectives on the self- system. In P.H. Mussen (Series Ed.) & E.M. Hetherington (Vol. Ed.), Handbook cf child psycholo,~y: volume IV. Soc~ialixztion. personulity. und so&l dc\,elopment. New York: Wiley.

Jouen, F. (1984). Visual-vestibular interactions in infancy. Infant Behuvior- und Development, 7, 135-145.

Lee, D.N., & Aronson, E. (1974). Visual proprioceptive control of standing in human infants. Perception & Psychophysics, IS. 529-532.

Lee. D.N.. & Lishman, J.R. (1975). Visual proprioceptive control of stance. Journal of Human Movement Studies, I, 87-95.

Lewis, M., & Brooks-Gunn, J. (1979). Social cognition and the ac,quisition of self New York: Plenum.

Lewkowicz, D.J. (1985). Bisensoty response to temporal frequency in 4-month-old infants. Developmental Psychology. 2 I, 306-3 17.

232 Schmuckler

Lewkowicz, D.J. (1986). Developmental changes in infants’ bisensory response to synchronous durations. 1nfunt Behavior andDevelopment, 9.335-353.

Lewkowicz, D.J. (1992). Infants’ response to temporally based intersensory equivalence: The effect of synchronous sounds on visual preferences for moving stimuli. Infant Behavior and Development. 5, 297-324.

Lewkowicz, D.J., & Lickliter, R. (Eds.). (1994). The development of intersensory perception: Compurative perspectives. Hillsdale, NJ: Erlbaum.

Meltzoff, A.N., & Barton, R.W. (1979). Intermodal match- ing by human neonates. Nature. 2X2,403-404.

Muir, D.. & Field, J. (1979). Newborn infants orient to sounds. Child Delzelopment, 50,431436.

Papousek, H., & Papousek, M. (1974). Mirror-image and self recognition in young human infants: A new method of experimental analysis. Developmental Psychobiology. 7, 149-157.

Proffitt, D.R., & Bertenthal, B.I. (1990). Converging operations revisited: Assessing what infants perceive using discrimination measures. Perception & Psychophysics, 47, l-1 I.

Rider, E.A., & Rieser, J.J. (1988). Pointing at objects in other rooms: Young children’s sensitivity to perspective after walking with and without vision. Child Development. 59, 48&494.

Rieser, J.J. (1979). Spatial orientation of six-month-old infants. Child Development, 50, 1078-1087.

Rieser. J.J.. Guth, D.A., & Hill, E. (1986). Sensitivity to perspective structure while walking without vision. Perception, 15, 173-188.

Rieser, J.J., & Rider. E.A. (1991). Young children’s spatial orientation with respect to multiple targets when walking without vision. Developmental Psychology, 27.97-107.

Rochat, P., & Morgan, R. (1995). Spatial determinants in the perception of self-produced leg movements to 3- to 5month-old infants. Developmental Psychology, 3/,626-636.

Rose, S.A., & Ruff, H.A. (1987). Cross-modal abilities in human infants. In J.D. Osofsky (Ed.), Handbook of infant development (2nd ed.) New York: Wiley.

Rovee. C.K., & Rovee, D.T. (1969). Conjugate reinforce- ment of infant exploratory behavior. Journal of Experimental Child Psychology, 8, 33-39.

Sameroff, A.J. (197 1). Can conditioned responses be estab- lished in the newborn infant? Developmental Psychology, 5, 1-12.

Schmuckler, M.A. (1995). Self-knowledge of body position: Integration of perceptual and action system information. In P. Rochat (Ed.), The self in infancy: Theory and research. Amsterdam: Elsevier Science Publishers.

Schmuckler, M.A. (in press). Children’s postural sway in response to low and high frequency visual oscilla- tion. Journal of Experimental Psychology: Human Perception & Performance.

Schmuckler, M.A., & Gibson, E.J. (1989). The effect of imposed optical flow on guided locomotion in young walkers. British Journal of Developmental Psychology, 7, 193-206.

Spelke, E. (1979). Perceiving bimodally specified events in infancy. Developmental Psychology, IS, 626-636.

Spelke, E. (1987). The development of intermodal perception In P. Salapatek & L. Cohen (Eds.), Handbook of infant perception: Vol. 2. From perception to cognition. Orlando, FL: Academic.

Spelke, E., Born, W.S., & Chu, F. (1983). Perception of moving, sounding objects by 4-month-old infants. Perception, 12,719-732.

Stoffregen, T.A., Schmuckler, M.A., & Gibson, E.J. (1987). Use of central and peripheral optical flow in stance and locomotion in young walkers. Perception, 16, 113-I 19.

Walker, A.S. (1982). Intermodal perception of expressive behaviors by human infants. Journal of Experimental Child Psychology, 33, 5 14-535.

Walker-Andrews, S., Bahrick, L.E., Raglioni, S.S., & Diaz, I. (1991). Infants’ bimodal perception of gender. Ecological Psychology, 3, 55-75.

Watson, J.S., & Ramey, C.T. (1972). Reactions to response contingency stimulation in early infancy. Merrill- Palmer Quarterly, 18, 219-227.

Wertheimer, M. (1961). Psychomotor coordination of audi-

Wing, tory and visual space at birth. Science. 134, 1692. A.M. (1977). Perturbations of auditory feedback delay and the timing of movement. Journal of Experimental Psychology: Human Perception and Performance, 3, 175-l 86.

I7 February 1995; Revised 30 October 1995 n

visual-proprioceptive intermodal perception in infancyutsc.utoronto.ca/~marksch/schmuckler...

Documents