auditory-visual speech perception in normal-hearing and cochlear-implant...

Auditory-visual speech perception in normal-hearing andcochlear-implant listenersa)

Sheetal Desai, Ginger Stickney, and Fan-Gang Zengb�

Departments of Anatomy and Neurobiology, Biomedical Engineering, Cognitive Sciencesand Otolaryngology – Head and Neck Surgery, 364 Medical Surgery II, University of California,Irvine, California 92697-1275

�Received 1 September 2006; revised 30 October 2007; accepted 31 October 2007�

The present study evaluated auditory-visual speech perception in cochlear-implant users as well asnormal-hearing and simulated-implant controls to delineate relative contributions of sensoryexperience and cues. Auditory-only, visual-only, or auditory-visual speech perception was examinedin the context of categorical perception, in which an animated face mouthing /ba/, /da/, or /ga/ waspaired with synthesized phonemes from an 11-token auditory continuum. A three-alternative,forced-choice method was used to yield percent identification scores. Normal-hearing listenersshowed sharp phoneme boundaries and strong reliance on the auditory cue, whereas actual andsimulated implant listeners showed much weaker categorical perception but stronger dependence onthe visual cue. The implant users were able to integrate both congruent and incongruent acoustic andoptical cues to derive relatively weak but significant auditory-visual integration. Thisauditory-visual integration was correlated with the duration of the implant experience but not theduration of deafness. Compared with the actual implant performance, acoustic simulations of thecochlear implant could predict the auditory-only performance but not the auditory-visualintegration. These results suggest that both altered sensory experience and improvised acoustic cuescontribute to the auditory-visual speech perception in cochlear-implant users.© 2008 Acoustical Society of America. �DOI: 10.1121/1.2816573�

PACS number�s�: 43.71.Ky, 43.71.Es, 43.66.Ts �PEI� Pages: 428–440

I. INTRODUCTION

Multisensory integration provides a natural and impor-tant means for communication. The benefit of integratingauditory and visual �AV� cues in speech perception has beenwell documented, particularly in difficult listening situationsand for hearing-impaired listeners �Sumby and Pollack,1954; Erber, 1972; Binnie et al., 1974; Dodd, 1977; Sum-merfield, 1979; Easton and Basala, 1982; Walden et al.,1993; Grant et al., 1998; Massaro, 1998�. The benefit derivedfrom speechreading can be substantial, allowing unintelli-gible speech to become comprehensive, or even exceedingthe benefit derived from the use of assistive listening de-vices, counseling, or training �Sumby and Pollack, 1954;Walden et al., 1981; Montgomery et al., 1984; Grant andBraida, 1991; Grant and Walden, 1996�. Two fundamentalquestions arise naturally concerning this AV integration inspeech recognition: �1� what acoustic and optical cues areintegrated? and �2� how and where are they integrated in thebrain? �Rosen et al., 1981; Braida, 1991; Massaro and Co-hen, 2000; Bernstein et al., 2002; De Gelder and Bertelson,2003�.

Acoustic and optical cues can be complementary to en-hance speech perception. On the one hand, some speech

a�Portions of this work were presented at the Conference on ImplantableAuditory Prostheses, Asilomar Conference Center, Pacific Grove, Califor-nia, 2003.

b�Author to whom correspondence should be addressed. Presently at: Uni-versity of California, Irvine, 364 Med Surge II, Irvine, CA 92697. Elec-
tronic mail: [email protected]
428 J. Acoust. Soc. Am. 123 �1�, January 2008 0001-4966/2008/1

sounds can be more easily distinguished in the visual modal-ity than in the auditory modality, e.g., the bilabial /ba/ versusthe alveolar /da/ stop consonant �Binnie et al., 1974; Dodd,1977; Summerfield, 1979; Walden et al., 1990�. On the otherhand, other speech sounds, called visemes, such as /b/, /p/,and /m/, rely on the acoustic cues to differentiate from eachother because they are visually indistinguishable �Fisher,1968; Binnie et al., 1974�. While normal-hearing listenershave little trouble doing so, hearing-impaired listeners, in-cluding cochlear-implant listeners, often have great difficultydifferentiating these phonemes due to reduced auditory tem-poral and spectral resolution �Miller and Nicely, 1955;Turner et al., 1997; Munson et al., 2003�.

Because the optical cues provided by a speaker’s facialand lip movements are not affected by the presence of noise,they are particularly useful at relatively low signal-to-noiseratios �SNRs�. Sumby and Pollack �1954� demonstrated ro-bust and relatively constant visual contribution to AV speechperception at SNRs over a 30-dB range. A recent study, how-ever, showed the maximal integration efficiency at an inter-mediate SNR of about −12 dB �Ross et al., 2007�.

There is also evidence that different acoustic cues con-tribute differently to the amount of auditory and visual inte-gration. When presented acoustically, voice pitch is virtuallyunintelligible but can significantly improve speechreading.The reason for this improvement is that the voice pitch cueprovides important segmental and suprasegmental informa-tion that is usually invisible �Rosen et al., 1981; Grant,
1987�. Similarly, the temporal wave form envelope cue im-
© 2008 Acoustical Society of America23�1�/428/13/$23.00

proves speechreading as long as the periodicity informationis included �50–500 Hz� �Rosen, 1992�. If only the envelopeinformation ��25 Hz� is included, then this temporal enve-lope cue produces little or no effect on speechreading �Grantet al., 1991; 1994�. Overall, these studies suggest that it isnot necessary to provide accurate information in both mo-dalities, rather complementary acoustic and optical cues aresufficient to support high-level AV speech perception �e.g.,Van Tasell et al., 1987�.

To understand how and where these acoustic and opticalcues are integrated in the brain, researchers have used incon-gruent cues from auditory and visual modalities �e.g., Calvertand Campbell, 2003; De Gelder and Bertelson, 2003; vanWassenhove et al., 2007�. A compelling example showinginteractions between incongruent auditory and visual cues inspeech perception is the McGurk effect �McGurk and Mac-Donald, 1976�. The McGurk effect is evoked by dubbing theaudio recording of one sound �e.g., /ba/� onto the visual re-cording of a different sound �e.g., /ga/�, obligating many lis-teners to report hearing an illusive sound �e.g., /da/ in thiscase�. The McGurk effect has been extended to sentences,different languages, children, hearing-impaired listeners, andspecial patient populations �Green et al., 1991; Sams et al.,1998; Cienkowski and Carney, 2002; Burnham and Dodd,2004�. It is believed that AV speech integration occurs in arelatively early, prelexical integration stage �e.g., Calvert etal., 1997; Reale et al., 2007�.

Recently, there has been an intensified interest in usingthe cochlear implant to study AV speech perception and in-tegration. There are at least two reasons for this intensifiedinterest. First, because the present implant extracts and de-livers only the temporal envelope cue, lacking access to finestructure including the low-frequency voice pitch that is typi-cally accessible to a hearing aid user, the implant users areparticularly susceptible to noise �e.g., Stickney et al., 2004;Kong et al., 2005; Zeng et al., 2005�. The optical cue, whenavailable, is essentially unaffected by noise. Therefore, theimplant users rely more than normal listeners on the visualcue, forcing them to become not only better speechreadersbut also better multisensory integrators �Goh et al., 2001;Clark, 2003; Schorr et al., 2005; Rouger et al., 2007�. In-deed, some cochlear-implant users can integrate AV cues toincrease the functional SNR in noise �Lachs et al., 2001;Bergeson et al., 2005; Hay-McCutcheon et al., 2005;Moody-Antonio et al., 2005�.

Second, the dramatic auditory experience and interven-tion with the cochlear implant provide a unique tool to studybrain plasticity in multiple ways. For example, Schorr et al.�2005� demonstrated a critical period in developing AV inte-gration, with the critical age of implantation being at about2.5 years old. On the other hand, brain imaging studies haveshown a profound cortical reorganization in cochlear implantusers, with good users being able to recruit a larger corticalarea, even the visual cortex, than poor users to perform anauditory task �Giraud et al., 2001b; Lee et al., 2001; Doucetet al., 2006�.

At present it remains unclear how much the cochlear-implant users can integrate auditory and visual information
and whether this integration is related to stimulus and subject
J. Acoust. Soc. Am., Vol. 123, No. 1, January 2008

variables. The primary goal of the present study was to ad-dress the following two questions: �1� Do postlinguisticallydeafened persons fitted with a cochlear implant really inte-grate auditory and visual information? �2� How will alteredstimuli and sensory experience affect AV integration? Wefirst quantified the degree of AV integration by measuringperformance in normal-hearing listeners, actual implant lis-teners, and simulated-implant listeners. We then delineatedthe relative contributions of stimulus and subject variables toAV integration by relating the degree of the AV integration tothe duration of deafness and the duration of the implant ex-perience.

II. METHODS

A. Subjects

1. Normal-hearing listeners

A total of 14 young, normal-hearing listeners partici-pated in this study. All subjects were native English speakerswith reported normal hearing. These young subjects rangedin age from 18 to 36 years. Subjects reported normal orcorrected-to-normal vision.

Because of the large age difference between normal-hearing and cochlear-implant listeners as well as the knowncognitive differences in sensory and cross-modality process-ing �Walden et al., 1993; Gordon-Salant and Fitzgibbons,1997; Humes, 2002; Hay-McCutcheon et al., 2005�, threeelderly, nearly normal-hearing listeners �average age�77�were recruited to evaluate whether age is a significant factorin the present study. Pure-tone averages �across 500, 1000,and 2000 Hz� of two subjects were below 15 dB HL, and thethird subject had a pure-tone average of 22 dB HL. Pure-toneaverages for all three subjects were taken from the right ear.These elderly subjects reported normal or corrected-to-normal vision. Because of time limitation, they only partici-pated in the experiment with the original unprocessedstimuli.

2. Cochlear-implant listeners

A total of eight postlingually deafened, adult cochlear-implant listeners were evaluated in this experiment. Thesesubjects were recruited locally from the Southern Californiaarea. They had a mean age of 66 years old, duration of deaf-ness of 18 years, and �1 year of experience with their de-vice. Table I shows additional information on the individualcochlear-implant listeners evaluated in this study, includingconsonant and vowel identification scores. Significant corre-lation was observed between duration of deafness and con-sonant �r=−0.96� and vowel �r=−0.85� recognition, as wellas between implant experience and consonant �r=−0.70� andvowel �r=−0.91� recognition. All cochlear-implant listenerswere native English speakers and were postlingually deaf-ened. All cochlear-implant listeners reported normal or

1
corrected-to-normal vision.
Desai et al.: Auditory-visual integration in cochlear implants 429

B. Stimuli

1. Unprocessed stimuli

The auditory stimuli were created using a web-basedKlatt Synthesizer �1980�, developed by Bunnell �1996� andcolleagues at the Speech Research Laboratory, A.I. duPontHospital for Children �1996�. Eleven consonant-vowel �CV�tokens were synthesized to represent an auditory continuumalong /ba/, /da/, and /ga/. The continuum was created byvarying the starting frequency of the F2 or F3 formants in200-Hz steps while keeping the other formant frequenciesconstant �see Table II�. The formant frequencies for the /a/sound paired with each consonant were also kept constant.Reference tokens for /ba/, /da/, and /ga/ are highlighted ingray. The formant values for these reference tokens wereadopted from Turner and Robb �1987�.

To test the effects of formant-transition-duration �i.e.,from the onset of the consonant sound to the onset of thesteady state /a/ sound�, two continuums were created: onewith a formant-transition-duration of 20 ms and a secondwith a formant-transition-duration of 40 ms. The fundamen-tal frequency for the consonant sounds started at 150 Hz andchanged after 20 or 40 ms �depending on the type of stimu-lus� and then decreased to 100 Hz for the vowel sound overthe remaining duration.

The total duration of each CV stimulus token was kept

TABLE I. Demographics of the cochlear-implant listeners, including age, imimplant, and percent correct scores in consonant and vowel recognition. No�Hillenbrand et al., 1995; Shannon et al., 1999�.

SubjectAge

�years� Implant Etiology

1 61 Nucleus 22 Genetic2 78 Nucleus 24 Unknown3 70 Nucleus 24 Unknown4 80 Med-El Genetic5 58 Nucleus 24 Genetic6 46 Nucleus 22 Trauma7 68 Clarion II Genetic8 69 Nucleus 24 Virus

TABLE II. Starting formant frequencies for each consonant token as well assteady-state formant frequencies for the vowel �/a / �.

Stimulus tokenF1

�Hz�F2

�Hz�F3

�Hz�

/b / 1 300 700 28002 300 900 28003 300 1100 28004 300 1300 28005 300 1500 2800

/d / 6 300 1700 28007 300 1700 26008 300 1700 24009 300 1700 220010 300 1700 2000

/g / 11 300 1700 1800/a / 720 1250 2500

430 J. Acoust. Soc. Am., Vol. 123, No. 1, January 2008

constant at 305 ms for both 20- and 40-ms stimuli �i.e., 20 or40 ms was allotted to the respective consonant sounds andthe remaining portion of the 305 ms was designated as thevowel sound�. The auditory stimuli were calibrated using theBruel & Kjaer sound level meter �Model No. 2260�. A cali-bration tone, created by using a 1000-Hz sinusoid matched tothe same rms level as the synthesized speech sounds, wasused to adjust the level of the auditory stimuli to a 70 dBSPL presentation level.

Normal-hearing and cochlear-implant listeners wereseated in a sound-attenuated booth during the experiments.Normal-hearing listeners listened to auditory stimuli monau-rally through the right ear with Sennheiser HDA 200 head-phones. Seven of the cochlear-implant listeners were pre-sented with stimuli through a direct audio input connectionto their speech processor and one cochlear-implant listenerwas presented with stimuli through a speaker because herear-level device did not allow a direct audio connection.

Visual stimuli from an animated face �“Baldi”�, whichcorresponded to the /ba/, /da/, and /ga/ sounds, were createdusing the Center for Spoken Language Understanding�CSLU� Speech Toolkit �Barnard et al., 2000; Massaro et al.2000�. The animated face was temporally aligned with eachauditory stimulus token to represent the initial consonant po-sition, the transition �20 or 40 ms�, and the final vowel posi-tion. The computer monitor window displaying the animatedface was modified so that the lips were 2 in. in width and1 in. in height.

The synthetic sound and the animated face, instead of anatural sound and a human face, were used for the followingtwo reasons. First, they can rule out a possible confoundingfactor of idiosyncratic acoustic and optical cues in a small setof stimuli as used in the present study. Second, the “Baldi”program allowed accurate temporal alignment between thecongruent and incongruent acoustic and optical cues. An ap-parent weakness of using these synthetic stimuli was theirrelatively weaker signal strength compared with naturalstimuli, because the synthetic stimuli were limited to thenumber of variables in the models, resulting in lower accu-racy and poorer resolution than the natural stimuli �Massaroet al., 2000�. This weak signal strength could have contrib-uted to relatively low-level AV integration �e.g., the McGurk

t type, etiology, duration of deafness, years of experience with the cochlearhearing listeners typically score �90% on these consonant and vowel tests

uration ofdeafness�years�

ImplantExperience

�years�

Percentconsonants

�%�

Percentvowels

�%�

9 5 72 5912 1 44 3111 3 54 5149 1 8 2544 1 12 221 11 71 79

16 2 59 511 6 77 63

planrmal-

D

effect� found in the present study.

Desai et al.: Auditory-visual integration in cochlear implants

2. Four- and eight-channel processed stimuli

The cochlear-implant simulation was generated using analgorithm developed by Shannon et al. �1995�. The originalstimuli from the 11-token continuum were band-pass filtered�sixth-order elliptical IIR filters� by either four or eight bandsusing the Greenwood map �Greenwood, 1990�. The envelopefrom each band was extracted through full-wave rectificationfollowed by a 500-Hz low-pass filter �first-order Bessel IIRfilter�. The envelope was then used to modulate a sinusoidalcarrier set to the center frequency of the narrowband. Theoutputs of the narrowband signals were combined to createthe four- or eight-channel cochlear-implant simulations.

C. Procedures

We adopted a classical categorical perception paradigmsimilar to that used by Walden et al. �1990� in hearing-impaired listeners and by Clark and colleagues �Clark, 2003�in prelingually deafened pediatric cochlear-implant users. Athree-alternative-forced-choice procedure was used in threeexperimental conditions: auditory-alone �A�, visual-alone�V�, and AV �AV�. In the AV condition, an animated facemouthing /ba/, /da/, or /ga/ was paired with each speechsound from the auditory continuum, creating both congruentand incongruent AV combinations. The V condition evalu-ated the subjects’ ability to lipread the mouthed phonemes.Each condition was tested with both 20- and 40-ms formant-transition-durations. Subjects were given both verbal andwritten instructions to click buttons labeled /ba/, /da/, and/ga/ on a computer-based interface that corresponded to whatthey thought they perceived. The three-alternative forced-choice protocol allowed quantification of perceptual bound-aries but could be a liability if the subject perceived a soundthat was different from the three choices, e.g., a /bg/ responseto an auditory /g/ and visual /b/ stimuli �McGurk and Mac-Donald, 1976�.

Prior to the test session, practice sessions were givenwith feedback for the A, V, and AV conditions. For the A andAV conditions, the reference tokens were presented a total of20 times, resulting in 60 presentations for one complete prac-tice session. For the V practice sessions, each reference tokenwas mouthed a total of ten times, resulting in 30 presenta-tions for one complete practice session. Normal-hearing lis-teners were required to achieve 80% correct identification ofall three unprocessed CV pairs in the A practice session �at a40-ms formant-transition-duration� to continue with the ex-periment. All of the normal-hearing listeners met this crite-rion.

Following the practice sessions, the test sessions weregiven without feedback. The test order of these conditionswas randomized for each subject. For the A condition, eachof the 11 continuum tokens was presented randomly a totalof 20 times, resulting in 220 presentations for one completetest session. For the AV condition, each of the 11 continuumtokens was paired with a /ba/, /da/, and /ga/ face and werepresented randomly a total of ten times, resulting in 330presentations. For the V condition, each /ba/, /da/, and /ga/face was presented randomly a total of 20 times, resulting in
60 presentations for one complete test session. The scores

were calculated as the number of tokens correctly identified�in % correct� or as a distribution in response to each of thethree tokens �in % identification�.

D. Data analysis

The phonemic boundaries were estimated with a four-parameter sigmoidal function fit to each function:2

f�x� = y0 +a

1 + e−�x−x0/b�

where y0�minimum y value; x0=x value corresponding topeak y; a=50% point; and b�slope.

A repeated-measures analysis of variance �ANOVA� wasperformed on both within- and between-subjects factors toexamine the main effects for each condition and stimulus.Chance performance was considered to be 33% identificationfor consonants. If an interaction was found between any ofthe main effects, a simple effects analysis was carried outfollowed by planned comparisons with a Bonferroni correc-tion between the conditions or stimuli in question. A signifi-cant effect in a Bonferroni analysis was calculated by divid-ing the p value of 0.05 by n−1 where n was the number ofconditions or stimuli in question.

III. RESULTS

A. Categorical perception


Since no main effect was found for the formant-transition-duration factor, the data for all conditions wereaveraged across 20- and 40-ms stimuli. The left panels inFig. 1 show perceptual labeling �% identification� by youngnormal-hearing listeners, who listened to the unprocessed au-ditory continuum along /ba/-/da/-/ga/. The top panel showsthe results for the A condition, while the bottom three panelsshow results for the auditory continuum simultaneously pre-sented with a visual /ba/ �second panel�, a visual /da/ �thirdpanel�, or a visual /ga/ stimulus �bottom panel�. Open circles,filled squares, and open triangles represent the subjects’ per-cent identification for /ba/, /da/, and /ga/, respectively. Thecurves represent the best fit of the four-parameter sigmoidalfunction to the data. The left vertical dashed line in eachpanel represents the estimated phonemic boundary between/ba/ and /da/, while the right dashed line represents theboundary between /da/ and /ga/. The asterisk symbol in thevisual /ga/ condition �bottom panel� placed above the /da/response �filled square� at the token 1 location represents oneof the commonly observed McGurk effects �auditory /ba /+visual /ga / =perceived /da/�.

The A condition �left-top panel� shows the classical pat-tern of categorical perception, with a significant main effectbeing observed for the three responses �F�2,12�=5.2; p�0.05�. Post hoc analysis with a Bonferroni correction re-vealed that tokens 1–3 were primarily labeled as /ba/�F�10,4�=1805.3; p�0.017�, tokens 4–8 labeled as /da/�F�10,4�=551.7; p�0.017�, and tokens 9–11 labeled as /ga/
�F�10,4�=963.3; p�0.017�. The estimated /ba-da/ boundary

was located at about token 4 and the /da-ga/ boundary was attoken 8.

The simultaneous presentation of the visual stimulus�bottom three panels� generally increased the subjects’ over-all percent identification towards the visual stimulus whiledecreasing the response to the incompatible auditory stimuli.For example, the visual /ba/ stimulus �second panel� in-creased the response to /ba/ from 0% to about 40% for to-kens between 5 and 11, whereby the peak /da/ response de-creased to about 60% and the peak /ga/ response decreased toabout 50%. Compared with the effect of the visual /ba/stimulus, the visual /da/ and /ga/ stimuli produced a similarbut slightly smaller effect on the subjects’ responses.

The simultaneous presentation of the visual /ba/ stimulussignificantly shifted the /ba/-/da/ boundary but not the /da/-/ga/ boundary �F�3,12�=11.3; p�0.05�. Compared to the A
condition, the visual /ba/ shifted the /ba/-/da/ boundary right-

ward from the 3.9-token position to the 5-token position,indicating that subjects tended to respond /ba/ to more stimu-lus tokens when the corresponding visual cue was present.This shift in the /ba/-/da/ boundary was not significant whenthe A condition was compared to the visual /da/ and /ga/conditions. Additionally, none of the visual stimuli producedany significant shift for the /da/-/ga/ boundary.

The right panels of Fig. 1 show the results obtained un-der the same conditions from three elderly listeners. Exceptfor the larger error bars, reflecting the smaller sample size�n=3� than the young normal-hearing listeners �n=14�, theelderly listeners produced essentially the same results. Forexample, the categorical boundary was at about token 4 be-tween /ba/ and /da/, and at about 8 between /da/ and /ga/.Similarly, the visual /ba/ shifted the /ba/-/da/ boundary right-

FIG. 1. Percent identification as afunction of consonant continua inyoung �left column� and elderly �rightcolumn� normal-hearing listeners. Thetop panel shows the results for theauditory-alone continuum. The datafor the three AV conditions are shownin separate panels �visual /b/: secondrow; visual /d/: third row; and visual/g/: bottom row�. Open circles ��,filled squares ��, and open triangles�� represent the percentage responseto /b/, /d/, and /g/, respectively. Errorbars represent the standard error of themean. Sigmoidal four-parameter func-tions were fitted to the data to reveal/b/-/d/ and /d/-/g/ boundaries. Verticaldashed lines show where these bound-aries occur along the continuum. Anasterisk �*� denotes one of the com-monly observed McGurk effects, i.e.,when subjects responded /da/ when avisual /ga/ face was paired with thereference auditory /ba/ sound �token1�.

ward to about token 5, the visual /da/ and /ga/ slightly shifted


the /ba/-/da/ boundary leftward, whereas the visual stimulihad minimal effect on the /da/-/ga/ boundary. This small setof data from elderly listeners suggests that age per se plays anegligible role in the present AV tasks �see also Walden etal., 1993; Helfer, 1998; Cienkowski and Carney, 2002; Som-mers et al., 2005�.


Figure 2 shows perceptual labeling by cochlear-implantlisteners �left column�. The figure configuration and symbolconventions are the same as Fig. 1. Several differences areapparent between the normal and implant listeners. First,cochlear-implant listeners produced an insignificant main ef-fect for the three responses in the A condition �F�2,6�=3.9;p�0.05�. Because of a significant interaction between re-sponses and stimulus tokens �F�20,140�=4.2; p�0.05�, posthoc analysis with a Bonferroni correction was conducted to
show that only tokens from 1 to 3 produced significantly

higher responses to /ba/ than to /da/ or /ga/ �p�0.025�. Sec-ond, different from the nearly perfect response to the refer-ence tokens by the normal-hearing listeners, the highest /ba/response obtained by the implant listeners was about 60% totokens 1–2, followed by 50% /da/ to tokens from 5–8 and30% /ga/ to tokens 10–11. Although their overall categoricalresponses were much weaker, the implant listeners produceda proper /ba/-/da/ boundary at token 4 and a slightly right-ward shifted /da/-/ga/ boundary at token 9.

Third, a totally different pattern emerged for the visualeffect on categorical perception in cochlear-implant listenersthan in normal-hearing listeners. Independent of the auditorystimulus, the cochlear-implant listeners showed an almosttotal bias toward the visual /ba/ cue �second panel�, and to alesser extent, toward the visual /da/ �third panel� and /ga/ cue�bottom panel�. Except for the separate /da/ and /ga/ labelingat high tokens �10 and 11� in the visual /da/ condition, thedominant visual cue wiped out the relatively weak categories

FIG. 2. Percent identification as afunction of consonant continuum inthe cochlear-implant listeners �firstcolumn�, and the four-channel �middlecolumn� and eight-channel �right col-umn� simulated implant listeners. Thetop panels show the results for theauditory-alone continuum. The datafor the three AV conditions are shownin separate panels �visual /b/: secondrow; visual /d/: third row; and visual/g/: bottom row�. Open circles, filledsquares, and open triangles representthe percentage response to /b/, /d/, and/g/, respectively. Error bars representthe standard error of the mean. An as-terisk �*� denotes the McGurk effect,i.e., when subjects responded /da/when a visual /ga/ face was pairedwith the reference auditory /ba/ sound�token 1�.

that existed in the A condition. On the surface, the McGurk


effect �the asterisk symbol in the bottom panel� appeared tobe much stronger in the implant listeners �60% labeling tothe combined auditory /ba/ and visual /ga/ stimuli� than inthe normal listeners �24%�. However, note that the overallbaseline response to /da/ was also much higher in implantlisteners �e.g., 46% to the combined auditory /ga/, i.e., token11, and visual /ga/ stimuli� than in normal listeners �8%�. Weshall return to this point in Sec. III.

3. Cochlear-implant simulations

Figure 2 also shows perceptual labeling by youngnormal-hearing listeners attending to four- and eight-channelcochlear-implant simulations �middle and right columns�.Like cochlear-implant listeners, normal-hearing listeners pre-sented with cochlear-implant simulations produced relativelyweak categorical perception and showed a strong bias towardthe visual cue, particularly to the visual /ba/. Moreover, thesimulated implant listeners produced a characteristic, albeitrelatively weak, categorical boundary for the /ba/-/da/ pair atthe token 4 position and /da/-/ga/ boundary close to the token9 position for the eight-channel simulation condition only.Finally, the simulated implant listeners produced a McGurkeffect ��40%, represented by asterisk symbols in the middleand right bottom panels� that was greater than normal-hearing listeners attending to unprocessed stimuli ��20% �but smaller than the actual implant listeners ��60% �. Doesthis mean that actual and simulated implant listeners aremore susceptible to the McGurk illusion than normal-hearinglisteners? The following sections analyze congruent and in-
congruent conditions in detail.

B. Congruent AV perception „AV benefit…

Figure 3 shows perceptual labeling results in response toA, V, and congruent AV stimuli in normal-hearing listeners,cochlear-implant listeners, and four- or eight-channelcochlear-implant simulations. The filled, unfilled, andhatched bars correspond to the percent identification of /b/,/d/, and /g/, respectively. In this case, only the reference to-kens were included �i.e., token 1 for /ba/, token 6 for /da/,and token 11 for /ga/�.

We also used two measures to define the amount of rela-tive AV benefit that can be derived relative to either the Acondition or the V condition. The first relative AV benefitmeasures the amount of phoneme recognition improvementrelative to the A baseline and is defined as �AV-A� / �100-A�with AV and A scores expressed as percent scores �Grant andSeitz, 1998�. Similarly, the second AV benefit measures theimprovement relative to the V baseline and is defined as�AV-V� / �100-V� with AV and V expressed as percent scores.These relative measures, as opposed to the absolute differ-ences �i.e., AV-A or AV-V�, were adopted to avoid potentialbiases because a wide range of A and V scores occurred inthe present diverse group of subjects. For example, a biasoccurs because a high A score will certainly produce a lowAV benefit score with the absolute measure but not necessar-ily with the relative measure.


Normal-hearing listeners demonstrated a significantmain effect on modality �F�2,12�=9.0; p�0.05�, showing

FIG. 3. Performance of all four groupsof subjects in the congruent AV, A, andV conditions. Error bars represent thestandard error of the mean.

nearly perfect performance for the identification of the refer-


ence stimuli in the A and AV conditions, but lower and morevariable performance in the V condition. On average, theproper labeling with the original unprocessed stimuli was95% for the A condition, and 94% for the AV condition, butonly 80% for the V condition.

Normal-hearing listeners also demonstrated a significantmain effect on phoneme identification �F�2,12�=23.0; p�0.05�, showing an averaged labeling of /ba/ 98% of thetime, /da/ 88% of the time, and /ga/ 83% of the time. Therewas a significant interaction between modality and consonantidentification �F�4,10�=7.0; p�0.05�. The interaction was dueto the fact that the normal-hearing listeners confuse the pho-nemes /da/ and /ga/ only in the V condition. This result is nottoo surprising because the voiced bilabial stop consonant,/b/, was more visually salient than either /d/ or /g/.

Averaged across all three phonemes, the normal-hearinglisteners produced an AV benefit score of −0.14 relative tothe A baseline and +0.71 relative to the V baseline. The nega-tive score was due to the fact that the 94% AV percent scorewas slightly lower than the 95% A-alone score. These rela-tive AV benefit scores suggest that the relative signal strengthin the auditory domain is strong, producing a ceiling effect.


In contrast to the normal control, the implant listenersproduced much lower overall percent correct scores in allconditions. Average performance collapsed across all condi-tions was 67% for the AV condition, 51% for the A condi-tion, and 52% for the V condition. A significant main effectwas found for modality �F�2,6�=15.7; p�0.05� in cochlear-implant listeners. A significant main effect was also found forphoneme identification �F�2,6�=15.7; p�0.05�. Average per-formance collapsed across all modalities was significantlyhigher for /b/ �80%�, but no difference was shown between/d/ �44%� and /g/ �46%�. A significant interaction was foundbetween modality and phoneme identification �F�4,4�=41.5;p�0.05�. This interaction was attributed to the larger differ-ence in performance between /b/ and /d/ in AV �43 percent-age points� and V �52 percentage points� conditions, as com-pared to the auditory-only condition �11 percentage points�.

The present cochlear-implant listeners were able to inte-grate the auditory and visual cues to significantly improvethe A or the V performance by 15–16 percentage points. Thecorresponding relative AV benefit score was 0.30 relative tothe A baseline and 0.29 relative to the V baseline.


First, a significant main effect was found for modality�F�2,5�=22.8; p�0.05�. Like actual implant listeners, thesimulated implant listeners appeared to benefit from the ad-ditional lipreading cue when presented with the degradedauditory stimuli. The performance was improved from 48%with A stimuli to 70% with AV stimuli in the four-channelcondition, and from 63% to 74% in the eight-channel condi-tion. However, unlike actual implant listeners, the V condi-tion produced the highest performance of about 80% in bothcochlear-implant simulations, suggesting no integration be-
tween the auditory and visual cues.

Second, a significant main effect was found for numberof channels �F�1,6�=37.5; p�0.05�, with the eight-channelcondition producing 72% performance and the four-channelcondition producing significantly lower performance at 66%.A significant interaction between channel and modality wasalso observed �F�2,5�=15.6; p�0.05�, reflecting lower A per-formance for the four-channel condition than the eight-channel condition. This result was expected because thegreater spectral resolution associated with the eight-channelwould produce better A performance than the four-channelcondition.

Third, a significant main effect was found for phonemeidentification �F�2,5�=24.3; p�0.05�. In the four-channelcondition, the mean performance was 83%, 64%, and 52%for /b/, /d/, and /g/, respectively. In the eight-channel condi-tion, the mean performance was 91%, 62%, and 64% for /b/,/d/, and /g/, respectively. Overall, the simulated implant lis-teners performed poorer than the normal-hearing listenerswhen listening to the unprocessed stimuli, but more similarlyto the actual cochlear-implant listeners.

Finally, the simulated four-channel implant listeners pro-duced the following percent correct scores: 50% for A, 81%for V, and 68% for AV; the simulated eight-channel implantlisteners produced the following percent correct scores: 63%for A, 81% for V, and 74% for AV. The corresponding rela-tive AV benefit score was 0.36 relative to the A baseline and−0.70 relative to the V baseline in the four-channel simula-tion, and was 0.30 and −0.37, respectively, in the eight-channel simulation. Similar to the actual implant listeners,the simulated implant listeners produced similar auditory-only percent scores and AV benefit scores relative to theauditory baseline. In contrast to the actual implant listeners,the simulated listeners produced higher visual-only scoresand negative AV benefit scores relative to the visual baseline.

C. Incongruent AV perception „McGurk effect…

Figure 4 shows perceptual labeling in response to theincongruent condition �the reference auditory /ba/ pairedwith the visual /ga/� in normal-hearing listeners listening tothe unprocessed auditory stimuli, cochlear-implant listeners,four-channel, and eight-channel cochlear-implant simula-tions. The filled, unfilled, and hatched bars represent the per-centage of the /ba/, /da/, and /ga/ responses, respectively.While a /ba/ response indicates a bias towards the auditorycue and a /ga/ response indicates a bias towards the visualcue, a /da/ response represents integration of the auditory andvisual cues, also known as the McGurk effect in this case.

We used two different measures to estimate the size ofthe McGurk effect. First, hearing-impaired listeners areprone to numerous auditory errors, with some of the errorsbeing consistent with McGurk-like responses, making it dif-ficult to distinguish between these auditory errors and a trueMcGurk effect �Grant and Seitz, 1998�. To overcome thisauditory error problem, we followed Grant and Seitz’s cor-rective method �Grant and Seitz, 1998� and adjusted the sizeof the McGurk effect by subtracting the subject’s /da/ re-sponse to visual /ga/ and auditory /ba/ �i.e., the solid square
data point above token 1 in the bottom panels of Figs. 1 and

2� from the subject’s averaged /da/ response to the auditory/ba/ token �i.e., the solid square data point above token 1 inthe top panels of Figs. 1 and 2� and the auditory /ga/ token�i.e., the solid square data point above token 11 in the toppanels of Figs. 1 and 2�. We will refer to this measure as theerror-adjusted McGurk effect.

Second, there may be a general subject response bias. Ifthe bias happens to be a McGurk-type response, it will inflatethe size of the McGurk effect. To overcome this responsebias problem, we adjusted the size of McGurk effect by sub-tracting the subject’s /da/ response to the incongruent stimu-lus from the subject’s /da/ response to the congruent /ga/stimulus �i.e., the solid square data point above token 11 inthe bottom panels of Figs. 1 and 2�. We will refer to thismeasure as the bias-adjusted McGurk effect.


A significant effect was found for consonant identifica-tion �F�2,12�=6.5; p�0.05�. The normal-hearing listeners re-sponded 63% of the time to /ba/, 24% to /da/, and 13% to/ga/, implicating a strong auditory bias. The 24% McGurkeffect was relatively weak, possibly due to the weak signalstrength of the present synthetic stimuli. Interestingly, six outof 14 normal-hearing listeners did not experience anyMcGurk effect, responding to the auditory /ba/ cue 100% ofthe time. The remaining eight subjects who experienced theMcGurk effect had a mean /da/ response of 42% �SD�23,with a range from 15% to 85%�. The unadjusted 24%McGurk effect was significantly higher than the averaged 3%/da/ response to the auditory /ba/ and /ga/ tokens �paired-ttest, p�0.05�, resulting in an error-adjusted McGurk effectof 21%. On the other hand, the normal subjects produced an8% /da/ response to the congruent /ga/ token, which wassignificantly lower than the 24% unadjusted McGurk effect�paired-t test, p�0.05�. The bias-adjusted McGurk effectsize is therefore 16%.


A significant effect was found for consonant identifica-
tion �F�2,6�=54.1; p�0.05�. The implant listeners responded

4% of the time to /ba/, 60% to /da/, and 36% to /ga/ with theincongruent auditory /ba/ and visual /ga/ stimulus. The im-plant response pattern was significantly different from thenormal control pattern �F�2,19�=8.6; p�0.05�. For example,the response to the auditory /ba/ cue was greatly reducedfrom 63% in the normal control to 4% in the implant listen-ers, suggesting that the auditory signal strength via a co-chlear implant was much weaker than the visual signalstrength.

On the surface, the McGurk effect appeared to be muchstronger in implant listeners, producing a fused /da/ response60% of the time, compared with the /da/ response 24% of thetime in normal control. The 33% error-adjusted McGurk ef-fect was significant �paired t-test, p�0.05� but the 14% bias-adjusted McGurk effect just missed the predefined signifi-cance criterion �paired t-test, p=0.08�. Overall, the 33%error-adjusted and the 14% bias-adjusted McGurk effects inimplant subjects were statistically indistinguishable fromtheir 21% and 16% counterparts in the normal control �t-testwith two-sample unequal variance; p�0.2�. Overall, theseadjusted measures suggest that although there is some evi-dence for integration between incongruent auditory and vi-sual cues, but this integration is weak, if present at all, inpostlingually deafened, adult implant listeners.


No significant main effect was found for consonantidentification for either the four-channel �F�2,5�=0.9; p�0.05� or eight-channel �F�2,5�=1.0; p�0.05� condition. Inthe four-channel condition, the response was the least to theauditory /ba/ cue �19%�, but was increasingly biased towardthe fused /da/ cue �36%� and the visual /ga/ cue �45%�. In theeight-channel condition, the response was evenly distributedacross the auditory /ba/, the fused /da/, and the visual /ga/cues at near chance performance. Neither adjusted McGurkeffect showed any indication of integration between the in-congruent auditory and visual cues in these simulated im-

FIG. 4. Percent identification of /ba/, /da/, or /ga/ in allfour groups of subjects for the incongruent AV condi-tion, in which an auditory /ba/ cue was paired with avisual /ga/ cue. Error bars represent the standard errorof the mean.

plant listeners.


IV. DISCUSSION

A. Categorical perception

The present results show sharp categories in young andelderly normal-hearing listeners, but greatly weakened cat-egories in actual and simulated cochlear-implant listeners.This finding is consistent with previous studies on hearing-impaired listeners �Walden et al., 1990� and pediatriccochlear-implant users �Clark, 2003� who also showedbroader than normal boundaries in a similar categorical per-ception task. Together, these data suggest that categoricalperception is affected by hearing loss or reduced spectralresolution, but not by age.

B. Congruent auditory and visual benefit

The present AV benefit in normal-hearing subjects wasconfounded by the ceiling effect. The actual implant datashowed similar 0.30 AV benefit scores relative to either theauditory or the visual baseline. This relative AV benefit scorewas about half of the 0.67 AV benefit score obtained byhearing-impaired listeners in the Grant and Seitz study�Grant and Seitz, 1998�. This discrepancy might be due toeither the specific and limited set of stimuli, or the perceptualdifference between hearing-impaired and cochlear-implantsubjects, or both. The simulated implant data suggest thatcurrent acoustic simulation models of the cochlear implant�Shannon et al., 1995� can simulate auditory perception ofdegraded acoustic stimuli but cannot simulate the V and AVperception in actual cochlear-implant users.

C. Incongruent auditory and visual integration

When incongruent acoustic and optic cues are present,listeners may be biased toward either auditory or visual cues.Consistent with previous studies �Easton and Basala, 1982;Massaro and Cohen, 1983; Bernstein et al., 2000; Cien-kowski and Carney, 2002; Clark, 2003; Schorr et al., 2005�,the present results show that normal-hearing listeners werebiased toward the auditory cue �i.e., greater /ba/ response inFig. 4� while cochlear-implant listeners were biased towardthe visual cue �i.e., greater /da/ and /ga/ response in Fig. 4�.

This bias appears to depend on the relative signalstrength between acoustic and optical stimuli and is likely toinfluence the degree of integration �e.g., Green et al., 1991;Green and Norrix, 1997�. In the normal-hearing subjects, theoptical signal strength from the animated “Baldi” was rela-tively weaker than the synthetic acoustic signal strength. Inactual and simulated implant listeners, however, the sameoptical signal became relatively stronger than the improvisedacoustic signal. While the unadjusted McGurk effect wasmuch greater in implant subjects than normal subjects, thisdifference could be totally accounted for by the McGurk-likeauditory error responses and response biases �Grant andSeitz, 1998�. Overall, the present data showed that the ad-justed McGurk effect was weak in normal-hearing andcochlear-implant subjects but totally absent in simulated-
implant subjects.

D. Experience and performance

Typically before receiving their devices, postlinguallydeafened adult-implant users experience a period of deafnessfrom several months to decades and during which they relyon lipreading. After implantation, users usually need severalmonths to years to achieve asymptotic performance �e.g.,Tyler and Summerfield, 1996; Skinner et al., 2002�. Thisunique experience may allow them to use the visual cue andintegrate the auditory and visual cues to a greater extent thanthe normal-hearing listeners �Schorr et al., 2005; Rouger etal., 2007�.

Recent brain imaging studies showed strong associationbetween cortical plasticity and cochlear-implant perfor-mance: In general, good performance was correlated with theamount of overall cortical activation, not only in the auditorycortex but also in the visual cortex when using the implantonly �Giraud et al., 2001a; Lee et al., 2001; Green et al.,2005�. Because good performance is correlated with the du-ration of implant experience, we would expect that both vari-ables are correlated with AV integration. On the other hand,Doucet et al. �2006� found an intramodal reorganization ingood implant performers but a profound cross-modality re-organization in poor performers, suggesting that the durationof deafness is correlated with the AV integration.

To address this experience and performance issue, weperformed correlational analysis between two implant vari-ables �duration of deafness and duration of implant usage�and seven implant performance measures �Table III�. Theimplant performance included three direct measures in re-sponse to A, V, and AV stimuli and four derived measures:the AV benefit score relative to A baseline �AV_A�, the AVbenefit score relative to V baseline �AV_V�, the error-adjusted McGurk effect �M _error�, and the bias-adjustedMcGurk effect �M _bias�.

Consistent with previous studies �e.g., Blamey et al.,1996; van Dijk et al., 1999; Gomaa et al., 2003�, the durationof deafness is negatively correlated with the duration of theimplant usage and the auditory-only performance. However,we found that the duration of deafness is not correlated withany other measures, including the visual-only performance.On the other hand, we found that the duration of implantexperience is directly correlated with the A and AV perfor-mance, as well as the AV benefit relative to the visual base-line and the McGurk effect. Because visual-only perfor-mance is not correlated to any implant performance, thepresent data suggest, at least in the present postlinguallydeafened implant users, that implant experience, rather thanauditory deprivation, determines the implant users’ ability tointegrate both congruent AV cues �i.e., the AV benefit� andincongruent AV cues �i.e., the McGurk effect�.

V. CONCLUSIONS

Auditory-only, visual-only, and AV speech perceptionwas conducted in normal-hearing listeners, postlinguallydeafened actual implant users, and acoustically simulatedcochlear-implant listeners. Given the limitations of using thesynthetic acoustic stimuli, the animated face, and the three-
alternative, forced-choice method, we can reach the follow-

ing conclusions:

�1� Normal-hearing listeners show not only sharp categori-cal perception of place of articulation but also strongreliance on the auditory cue;

�2� Cochlear-implant listeners show weak categorical per-ception of place of articulation but strong reliance on thevisual cue;

�3� The implant listeners can derive significant AV benefitfrom the congruent acoustic and optical cues;

�4� Both normal and implant listeners produced a relativelyweak McGurk effect in response to the incongruentacoustic and optical cues, possibly due to the weak sig-nal strength in the synthetic stimuli;

�5� It is the duration of the implant experience, rather thanthe duration of deafness, that correlates with the amountof AV integration; and

�6� The present acoustic simulation model of the cochlearimplant can predict the actual implant auditory-only per-formance but not the AV speech integration.

ACKNOWLEDGMENTS

We are very grateful for the time and dedication that ourcochlear-implant and normal-hearing listeners have offeredfor this study. We thank Dominic Massaro, Jacques de Villi-ers, and Paul Hosom at the Center for Spoken LanguageUnderstanding, UC Santa Cruz, for their help and for the useof their Baldi software. We also thank Abby Copeland, JeffCarroll, Hongbin Chen, Sara Ghazi, Jeremy Liu, Sheng Liu,and Kaibao Nie for their assistance in technical support andhelpful comments on the manuscript. This work was sup-ported in part by a grant from the National Institutes ofHealth �Grant No. RO1 DC002267�.

1One of the cochlear-implant listeners tested reported that she had correc-tive cataract surgery and that the vision in the right eye was still somewhatlimited. However, she also reported that glasses are sufficient for viewingthe television and objects at close range. She was also able to drive duringdaylight hours.

2Occasionally there were functions that did not optimally fit the parametersof the sigmoid four-parameter equation. In these cases, alternative meth-ods were used to find the x intercepts: �1� if a five-parameter sigmoidal

TABLE III. Correlational analysis between implant variables and implant pevariables included the duration of deafness �Deaf� and duration of implant uscongruent auditory and visual cues: the auditory-only score �A�, the visual-odetails�: the AV benefit score relative to A baseline �AV_A�, the AV benefit sand the bias-adjusted McGurk effect �M _bias�. Pearson correlation coeffici

Deaf CI A V

Deaf 1 −0.67a −0.65a 0.06CI 1 0.87a −0.09A 1 0.02V 1AVAV_AAV_VM _errorM _bias

aSignificant correlation at the 0.05 level.

function was able to fit a boundary instead of a four-parameter sigmoidal


function, then the values for a, b, x0, and y0 were taken from calculationsmade by the graphing software �SigmaPlot�; �2� if two boundary lineswere linear functions, the x intercept was determined from the equation ofthe intersecting lines. When none of these rules applied, for instance whenone function was linear and the second sigmoidal, the point that mostclosely estimated the x intercept was taken upon inspection of the graph.These exceptions only occurred in nine of 672 cases.

Barnard, E., Bertrant, J., Rundlem, B., Cole, R., Cosi, P., Cronk, A.,de Villiers, J., Fanty, M., Hosom, P., Kain, A., Kaiser, E., Macon, M.,Rozman, R., Shobaki, K., Schalkwyk, J., Sutton, S., Tadrist, A.,van Vuuren, S., Vermelen, P., Wouters, J., Yan, Y. �2000�.�http://cslu.cse.ogi.edu/�, Oregon Graduate Institute of Science and Tech-nology, �Last viewed 10/16/2007�.

Bergeson, T. R., Pisoni, D. B., and Davis, R. A. �2005�. “Development ofaudiovisual comprehension skills in prelingually deaf children with co-chlear implants.” Ear Hear. 26, 149–164.

Bernstein, L. E., Auer, E. T., Jr., Moore, J. K., Ponton, C. W., Don, M. andSingh, M. �2002�. “Visual speech perception without primary auditorycortex activation.” NeuroReport 13, 311–315.

Bernstein, L. E., Demorest, M. E., and Tucker, P. E. �2000�. “Speech per-ception without hearing.” Percept. Psychophys. 62, 233–252.

Binnie, C. A., Montgomery, A. A., and Jackson, P. L. �1974�. “Auditory andvisual contributions to the perception of consonants.” J. Speech Hear. Res.17, 619–630.

Blamey, P., Arndt, P., Bergeron, F., Bredberg, G., Brimacombe, J., Facer, G.,Larky, J., Lindstrom, B., Nedzelski, J., Peterson, A., Shipp, D., Staller, S.,and Whitford, L. �1996�. “Factors affecting auditory performance of post-linguistically deaf adults using cochlear implants.” Audiol. Neuro-Otol. 1,293–306.

Braida, L. D. �1991�. “Crossmodal integration in the identification of con-sonant segments.” Q. J. Exp. Psychol. A 43, 647–677.

Bunnell, H. �1996�. �http://wagstaff.asel.udel.edu/speech/tutorials/synthesis/�, Speech Research Laboratory, A.I. duPont Hospital for Chil-dren, �Last viewed 10/16/2007�.

Burnham, D., and Dodd, B. �2004�. “AV speech integration by prelinguisticinfants: perception of an emergent consonant in the McGurk effect.” Dev.Psychobiol. 45, 204–220.

Calvert, A. A., Bullmore, T. T., Brammer, J. J., Campbell, R., Williams, S.C., McGuire, P. K., Woodruff, P. W., Iversen, S. D., and David, A. S.�1997�. “Activation of auditory cortex during silent lipreading.” Science276, 593–596.

Calvert, G. A., and Campbell, R. �2003�. “Reading speech from still andmoving faces: the neural substrates of visible speech.” J. Cogn Neurosci.15, 57–70.

Cienkowski, K. M., and Carney, A. E. �2002�. “AV speech perception andaging.” Ear Hear. 23, 439–449.

Clark, G. �2003�. “Cochlear implants in children: safety as well as speechand language.” Int. J. Pediatr. Otorhinolaryngol. 67, S7–20.

De Gelder, B., and Bertelson, P. �2003�. “Multisensory integration, percep-tion and ecological validity.” Trends Cogn. Sci. 7, 460–467.

ance in eight postlingually deafened cochlear-implant listeners. The implantCI�. The implant performance included three direct measures in response tocore �V�, the AV score �AV�, as well as four derived measures �see text forrelative to V baseline �AV_V�, the error-adjusted McGurk effect �M _error�,as used.

AV AV_A AV_V M _error M _bias

0.38 0.20 −0.41 −0.44 −0.230.82a 0.13 0.87a 0.77a 0.63a

0.84a 0.14 0.80a 0.90a 0.83a

0.24 0.48 −0.27 0.41 0.231 0.37 0.87a 0.77a 0.89a

1 0.17 0.14 −0.111 0.63a 0.65a

1 0.87a

1

rformage �nly scoreent w

−

Dodd, B. �1977�. “The role of vision in the perception of speech.” Percep-


tion 6, 31–40.Doucet, M. E., Bergeron, F., Lassonde, M., Ferron, P., and Lepore, F. �2006�.

“Cross-modal reorganization and speech perception in cochlear implantusers.” Brain 129, 3376–3383.

Easton, R. D., and Basala, M. �1982�. “Perceptual dominance during lip-reading.” Percept. Psychophys. 32, 562–570.

Erber, N. P. �1972�. “Auditory, visual, and AV recognition of consonants bychildren with normal and impaired hearing.” J. Speech Hear. Res. 15,413–422.

Fisher, C. G. �1968�. “Confusions among visually perceived consonants.” J.Speech Hear. Res. 11, 796–804.

Giraud, A. L., Price, C. J., Graham, J. M., and Frackowiak, R. S. �2001a�.“Functional plasticity of language-related brain areas after cochlear im-plantation.” Brain 124, 1307–1316.

Giraud, A. L., Price, C. J., Graham, J. M., Truy, E., and Frackowiak, R. S.�2001b�. “Cross-modal plasticity underpins language recovery after co-chlear implantation.” Neuron 30, 657–663.

Goh, W. D., Pisoni, D. B., Kirk, K. I., and Remez, R. E. �2001�. “Audio-visual perception of sinewave speech in an adult cochlear implant user: acase study.” Ear Hear. 22, 412–419.

Gomaa, N. A., Rubinstein, J. T., Lowder, M. W., Tyler, R. S., and Gantz, B.J. �2003�. “Residual speech perception and cochlear implant performancein postlingually deafened adults.” Ear Hear. 24, 539–544.

Gordon-Salant, S., and Fitzgibbons, P. J. �1997�. “Selected cognitive factorsand speech recognition performance among young and elderly listeners.”J. Speech Lang. Hear. Res. 40, 423–431.

Grant, K. W. �1987�. “Encoding voice pitch for profoundly hearing-impairedlisteners.” J. Acoust. Soc. Am. 82, 423–432.

Grant, K. W., and Braida, L. D. �1991�. “Evaluating the articulation indexfor AV input.” J. Acoust. Soc. Am. 89, 2952–2960.

Grant, K. W., Braida, L. D., and Renn, R. J. �1991�. “Single band amplitudeenvelope cues as an aid to speechreading.” Q. J. Exp. Psychol. A 43,621–645.

Grant, K. W., Braida, L. D., and Renn, R. J. �1994�. “Auditory supplementsto speechreading: combining amplitude envelope cues from different spec-tral regions of speech.” J. Acoust. Soc. Am. 95, 1065–1073.

Grant, K. W., and Seitz, P. F. �1998�. “Measures of AV integration in non-sense syllables and sentences.” J. Acoust. Soc. Am. 104, 2438–2450.

Grant, K. W., and Walden, B. E. �1996�. “Evaluating the articulation indexfor AV consonant recognition.” J. Acoust. Soc. Am. 100, 2415–2424.

Grant, K. W., Walden, B. E., and Seitz, P. F. S. �1998�. “AV speech recog-nition by hearing-impaired subjects: Consonant recognition, sentence rec-ognition, and AV integration.” J. Acoust. Soc. Am. 103, 2677–2690.

Green, K. M., Julyan, P. J., Hastings, D. L., and Ramsden, R. T. �2005�.“Auditory cortical activation and speech perception in cochlear implantusers: effects of implant experience and duration of deafness.” Hear. Res.205, 184–192.

Green, K. P., Kuhl, P. K., Meltzoff, A. N., and Stevens, E. B. �1991�. “In-tegrating speech information across talkers, gender, and sensory modality:female faces and male voices in the McGurk effect.” Percept. Psychophys.50, 524–536.

Green, K. P., and Norrix, L. W. �1997�. “Acoustic cues to place of articula-tion and the McGurk Effect: the role of release bursts, aspiration, andformant transitions.” J. Speech Lang. Hear. Res. 40, 646–665.

Greenwood, D. �1990�. “A cochlear frequency-position function for severalspecies - 29 years later.” J. Acoust. Soc. Am. 87, 2592–2605.

Hay-McCutcheon, M. J., Pisoni, D. B., and Kirk, K. I. �2005�. “Audiovisualspeech perception in elderly cochlear implant recipients.” Laryngoscope115, 1887–1894.

Helfer, K. S. �1998�. “Auditory and AV recognition of clear and conversa-tional speech by older adults.” J. Am. Acad. Audiol 9, 234–242.

Hillenbrand, J., Getty, L. A., Clark, M. J., and Wheeler, K. �1995�. “Acous-tic characteristics of American English vowels.” J. Acoust. Soc. Am. 97,3099–3111.

Humes, L. E. �2002�. “Factors underlying the speech-recognition perfor-mance of elderly hearing-aid wearers.” J. Acoust. Soc. Am. 112, 1112–1132.

Klatt, D. H. �1980�. “Software for a cascade/parallel formant synthesizer.” J.Acoust. Soc. Am. 67, 971–995.

Kong, Y. Y., Stickney, G. S., and Zeng, F. G., �2005�. “Speech and melodyrecognition in binaurally combined acoustic and electric hearing.” J.Acoust. Soc. Am. 117, 1351–1361.

Lachs, L., Pisoni, D. B., and Kirk, K. I. �2001�. “Use of audiovisual infor-
mation in speech perception by prelingually deaf children with cochlear

implants: a first report.” Ear Hear. 22, 236–251.Lee, D. S., Lee, J. S., Oh, S. H., Kim, S. K., Kim, J. W., Chung, J. K., Lee,

M. C., and Kim, C. S., �2001�. “Cross-modal plasticity and cochlear im-plants.” Nature �London� 409, 149–150.

Massaro, D. W. �1998�. Perceiving Talking Faces: From Speech Perceptionto a Behavioral Principle �MIT Press, Cambridge, MA�.

Massaro, D. W., and Cohen, M. M. �1983�. “Evaluation and integration ofvisual and auditory information in speech perception.” J. Exp. Psychol.Hum. Percept. Perform. 9, 753–771.

Massaro, D. W., and Cohen, M. M., �2000�. “Tests of AV integration effi-ciency within the framework of the fuzzy logical model of perception.” J.Acoust. Soc. Am. 108, 784–789.

Massaro, D. W., Cohen, M. M., Beskow, J., and Cole, R. �2000�. “Devel-oping and evaluating conversational agents.” in Embodied ConversationalAgents, edited by J. Chassell, J. Sullivan, S. Prevost, and E. Churchill�MIT Press, Cambridge, MA�, pp. 287–318.

McGurk, H., and MacDonald, J. �1976�. “Hearing lips and seeing voices.”Nature �London� 264, 746–748.

Miller, G. A., and Nicely, P. E. �1955�. “An analysis of perceptual confu-sions among some English consonants.” J. Acoust. Soc. Am. 27, 338–352.

Montgomery, A. A., Walden, B. E., Schwartz, D. M., and Prosek, R. A.�1984�. “Training AV speech reception in adults with moderate sensorineu-ral hearing loss.” Ear Hear. 5, 30–36.

Moody-Antonio, S., Takayanagi, S., Masuda, A., Auer, T. T., Jr., Fisher, L.,and Bernstein, L. E. �2005�. “Improved speech perception in adult con-genitally deafened cochlear implant recipients.” Otol. Neurotol. 26, 649–654.

Munson, B., Donaldson, S. S., Allen, S. L., Collison, E. A., and Nelson, D.A. �2003�. “Patterns of phoneme perception errors by listeners with co-chlear implants as a function of overall speech perception ability.” J.Acoust. Soc. Am. 113, 925–935.

Reale, R. A., Calvert, G. A., Thesen, T., Jenison, R. L., Kawasaki, H., Oya,H., Howard, M. A., and Brugge, J. F. �2007�. “AV processing representedin the human superior temporal gyrus.” Neuroscience 145, 162–184.

Rosen, S. M. �1992�. “Temporal information in speech: Acoustic, auditoryand linguistic aspects.” Philos. Trans. R. Soc. London, Ser. B 336, 367–373.

Rosen, S. M., Fourcin, A. J., and Moore, B. C. �1981�. “Voice pitch as an aidto lipreading.” Nature �London� 291, 150–152.

Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., and Foxe, J. J.�2007�. “Do you see what I am saying? Exploring visual enhancement ofspeech comprehension in noisy environments.” Cereb. Cortex 17, 1147–1153.

Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., and Barone,P. �2007�. “Evidence that cochlear-implanted deaf patients are better mul-tisensory integrators.” Proc. Natl. Acad. Sci. U.S.A. 104, 7295–7300.

Sams, M., Manninen, P., Surakka, P., Helin, P., and Katto, R. �1998�.“McGurk effect in Finnish syllables, isolated words, and words in sen-tences: Effects of word meaning and sentence context.” Speech Commun.26, 75–87.

Schorr, E. A., Fox, N. A., van Wassenhove, V., and Knudsen, E. I. �2005�.“AV fusion in speech perception in children with cochlear implants.” Proc.Natl. Acad. Sci. U.S.A. 102, 18748–18750.

Shannon, R. V., Jensvold, A., Padilla, M., Robert, M. E., and Wang, X.�1999�. “Consonant recordings for speech testing.” J. Acoust. Soc. Am.106, L71–74.

Shannon, R. V., Zeng, F.-G., Kamath, V., Wygonski, J., and Ekelid, M.�1995�. “Speech recognition with primarily temporal cues..” Science 270,303–304.

Skinner, M. W., Holden, L. K., Whitford, L. A., Plant, K. L., Psarros, C.,and Holden, T. A. �2002�. “Speech recognition with the nucleus 24SPEAK, ACE, and CIS speech coding strategies in newly implantedadults.” Ear Hear. 23, 207–223.

Sommers, M. S., Tye-Murray, N., and Spehar, B. �2005�. “AV speech per-ception and AV enhancement in normal-hearing younger and older adults.”Ear Hear. 26, 263–275.

Stickney, G. S., Zeng, F. G., Litovsky, R. Y., and Assmann, P. F. �2004�.“Cochlear implant speech recognition with speech masker.” J. Acoust.Soc. Am. 116, 1081–1091.

Sumby, W., and Pollack, I. �1954�. “Visual contribution to speech intelligi-bility in noise.” J. Acoust. Soc. Am. 26, 212–215.

Summerfield, Q. �1979�. “Use of visual information for phonetic percep-tion.” Phonetica 36, 314–331.

Turner, C. W., and Robb, M. P. �1987�. “Audibility and recognition of stop


consonants in normal and hearing-impaired subjects.” J. Acoust. Soc. Am.81, 1566–1573.

Turner, C. W., Smith, S. J., Aldridge, P. L., and Stewart, S. L. �1997�.“Formant transition duration and speech recognition in normal andhearing-impaired listeners.” J. Acoust. Soc. Am. 101, 2822–2825.

Tyler, R. S.,and Summerfield, A. Q. �1996�. “Cochlear implantation: Rela-tionships with research on auditory deprivation and acclimatization.” EarHear. 17, 38S–50S.

van Dijk, J. E., van Olphen, A. F., Langereis, M. C., Mens, H. H., Brokx, J.P., and Smoorenburg, G. F. �1999�. “Predictors of cochlear implant perfor-mance.” Audiology 38, 109–116.

Van Tasell, D. J., Soli, S. D., Kirby, V. M., and Widin, G. P. �1987�. “Speechwaveform envelope cues for consonant recognition.” J. Acoust. Soc. Am.82, 1152–1161.

van Wassenhove, V., Grant, K. W., and Poeppel, D. �2007�. “Temporal win-


dow of integration in AV speech perception.” Neuropsychologia 45, 598–607.

Walden, B. E., Busacco, D. A., and Montgomery, A. A. �1993�. “Benefitfrom visual cues in AV speech recognition by middle-aged and elderlypersons.” J. Speech Hear. Res. 36, 431–436.

Walden, B. E., Erdman, S. A., Montgomery, A. A., Schwartz, D. M., andProsek, R. A. �1981�. “Some effects of training on speech recognition byhearing-impaired adults.” J. Speech Hear. Res. 24, 207–216.

Walden, B. E., Montgomery, A. A., Prosek, R. A., and Hawkins, D. B.�1990�. “Visual biasing of normal and impaired auditory speech percep-tion.” J. Speech Hear. Res. 33, 163–173.

Zeng, F. G., Nie, K., Stickney, G. S., Kong, Y. Y., Vongphoe, M., Bhargave,A., Wei, C., and Cao, K. �2005�. “Speech recognition with amplitude andfrequency modulations.” Proc. Natl. Acad. Sci. U.S.A. 102, 2293–2298.


auditory-visual speech perception in normal-hearing and cochlear-implant...

Documents