brain and language volume 147 , august 2015, pages 30–40 · 2015-05-25 · italian culture....

Brain and Language

Volume 147, August 2015, Pages 30–40

Semantic brain areas are involved in gesture comprehension: An electrical neuroimaging study

• Alice Mado Proverbioa, , , • Veronica Gabaroa, • Andrea Orlandia, b,

• Alberto Zanib Show more

doi:10.1016/j.bandl.2015.05.002

Get rights and content

Highlights

•

ERPS to 800 spontaneous (e.g., emblematic and iconic) gestures were recorded in Italian healthy non-signers.

•

Congruent gestures were recognized by areas involved in body language

processing as early as 350 ms.

•

Gestural violations activated linguistic brain areas (e.g. N400 generator: medial

temporal lobe).

•

The data hint at a phylogenetic transition between a gestural and a verbal communicative system.

Abstract

While the mechanism of sign language comprehension in deaf people has been widely investigated, little is known about the neural underpinnings of spontaneous gesture comprehension in healthy speakers. Bioelectrical responses to 800 pictures of actors

showing common Italian gestures (e.g., emblems, deictic or iconic gestures) were recorded in 14 persons. Stimuli were selected from a wider corpus of 1122 gestures. Half of the pictures were preceded by an incongruent description. ERPs were recorded from 128 sites while participants decided whether the stimulus was congruent. Congruent pictures elicited a posterior P300 followed by late positivity, while incongruent gestures elicited an anterior N400 response. N400 generators were investigated with swLORETA reconstruction. Processing of congruent gestures activated face- and body-related visual areas (e.g., BA19, BA37, BA22), the left angular gyrus, mirror fronto/parietal areas. The incongruent–congruent contrast

particularly stimulated linguistic and semantic brain areas, such as the left medial and the superior temporal lobe.

Keywords

• ERPs; • Action processing; • Language; • Semantic violation; • N400; • Body language;

• Mirror neurons

1. Introduction

In this study, the neural mechanisms underlying normal speakers’ ability to understand

spontaneous gestures was investigated. Gesture language comprises a set of actions, mostly involving facial mimicry and hand movements (but also of other body parts) that are used automatically either while talking with others to emphasize the message meaning (Beattie and Shovelton, 1999, Beattie and Shovelton, 2002 and Dick et al.,

2009), or in replacement of oral speech (e.g., in noisy environments and/or with distant interlocutors). For example, nodding, very frequently used by infants to signify negation, and later on followed by hand shaking (“baby signs”), are both emblematic

gestures (Kirk, Howlett, Pine, & Fletcher, 2013). Interestingly, it has been shown that people are better at understanding ambiguous utterances if allowed to see the accompanying gestures performed by speakers in videos (Guellai, Langus, & Nespor, 2014). For example, Holle and Gunter (2007) recorded EEG as participants watched videos of a person gesturing and speaking simultaneously. They found that N400 to target words was smaller after a congruent gesture and larger after an incongruent gesture, suggesting that listeners can use gestural information to disambiguate speech. In contrast to standard sign language (e.g., American Sign Language, ASL, British Sign Language, BSL, which is a formal language employing a system of hand gestures for

communication, as by the deaf), spontaneous sign language is not formally taught and, although it does have a sort of grammatik and a relatively fixed set of rules, it is used in a rather flexible way. It presents considerable individual, regional, and cultural differences and it’s implicitly learned and understood by speakers. Spontaneous sign language is defined as any means of communication through bodily movements, especially of the hands and arms, rather than through speech, which is spontaneously used by humans, without formal training. For this reason it has been conceptualized as halfway between a formal sign language and an emotional body language (EBL) system, for example by Andric et al. (2013).

If, on one hand, the neural systems underlying standard sign language comprehension have been widely investigated by neuroimaging and electromagnetic studies (Braun et al., 2001, Hänel-Faulhaber et al., 2014, Husain et al., 2012 and Levänen et al., 2001), the same does not hold for spontaneous gesture comprehension. As for the BSL, MacSweeney et al. (2002) identified a set of regions commonly activated in deaf and hearing sign language users, which included inferior prefrontal regions bilaterally (including Broca’s area) and superior temporal regions bilaterally (including Wernicke’s area). Sign language (vs. audiovisual speech) generated enhanced activation in the posterior occipito-temporal regions (V5), most likely because of its dynamic nature. Interestingly, deaf native signers demonstrated greater activation in the left superior

temporal gyrus in response to BSL than hearing native signers. The role of Broca’s area for sign language production (Braun et al., 2001) and of Wernicke’s area for sign

language comprehension (Petitto et al., 2000) have been demonstrated by a series of neuroimaging and clinical studies (Hickok et al., 1996 and Poizner et al., 1987). To address the neural mechanism supporting the comprehension of spontaneous body

language, some ERP studies have used the N400 linguistic paradigm as a contrast to ERPs for correct vs. incongruent gestures (e.g., Bach et al., 2009 and Gunter and Bach, 2004), which represents a valuable tool for determining the neural processing time course for conceptual, linguistic and semantic information (Kutas and Federmeier, 2000 and Kutas and Hillyard, 1980). Interestingly, this component has been shown to reflect the detection of semantic violations in sign language among deaf individuals (e.g., in American sign language: Neville et al., 1997, or in German sign language: Hänel-Faulhaber et al., 2014), thus suggesting the recruitment of auditory language-devoted structures in sign language processing.

To investigate the electrophysiological correlates of gesture comprehension in normal hearing speakersBach et al. (2009) presented two hand actions as consecutive frames, one showing an instrument and the other one a potential target object of the action. Two mismatches were possible: a tool orientation mismatch or a misuse case (e.g., a screwdriver followed by a keyhole). Both types of violation were associated with the occurrence of similar N400 responses. Although these studies demonstrated the temporal component of brain processing, they only concerned hand actions, which are highly relevant to body language but cannot convey the complex pattern of affective and gestural indices provided by the entire body. One recent neuroimaging study

(Andric et al., 2013) compared the neural processing of object-directed actions (such as grasping) with the processing of emblems, but the study only considered a limited set of hand actions (while the full body of the agent was not visible). To pursue a more ecological approach, in a very recent ERP study on the mechanism of emotional body language (EBL) comprehension (Proverbio, Calbi, Manfredi, & Zani, 2014), we recorded ERPs in 30 Italian University students while they evaluated 280 full-body pictures displaying typical EBL patterns acted out by 8 male and female Italian actors. Half of the stimuli were incongruent with a previous verbal description (for example, “Come here, let me hug you!” followed by the portrait of visibly hostile man. ERP responses showed an anterior N400 response indicating the detection of

incongruent body language, starting as early as 300 ms post-stimulus. SwLORETA was performed on the N400 difference by subtracting ERPs to congruent actions from ERPs to incongruent ones to identify the strongest generators of this effect in the right

rectal gyrus (BA11) of the ventromedial orbitofrontal cortex, the bilateral uncus (limbic system) and the cingulate cortex, and in cortical areas involved in face and body processing (superior temporal sulcus (STS), fusiform face area (FFA) and extra-striate

body area (EBA) and the premotor cortex (BA6), which is involved in action understanding. These findings are consistent with fMRI literature on brain response to emotional (not symbolic) body language (De Gelder et al., 2009). Recently Kana and Travers (2012) found significant activation in brain areas associated with visual representation (EBA and FFA), with action processing (inferior frontal gyrus (IFG) and STS, see also Wurm, Hrkać, Morikawa, & Schubotz, 2014) and inferior parietal lobule (IPL), and with emotion processing (anterior insula, medial prefrontal cortex (MPFC), striatum, superior colliculus and pulvinar) when interpreting the actions and emotions of stick figures. In this case as well, stimuli were not very ecologically valid because stick

figures were used instead of real pictures. The purpose of this study was multifold: (1) We wished to investigate the neural mechanisms underlying normal speakers’ ability to understand spontaneous gestures. (2) We intended to determine whether the comprehension of gestures in healthy users was closer to the mechanism of sign language processing in non-hearing speakers, involving linguistic brain areas, or more similar to affective body language comprehension, as investigated in a previous ERP study (Proverbio et al., 2014). (3) By applying the N400 paradigm we aimed at studying for the first time the processing of a large corpus of spontaneous gestures (including deictic, transformative,

metaphoric, emblematic, iconic and motor gestures (see Goodwin, 2003) engaging both body and face processing. (4) In order to provide the richest stimulation as possible individual and dialectal variations of bodily expressions were considered by recruiting 6 different male and female actors. To accomplish these goals, more than one thousand pictures were taken of actors displaying a large variety of distinctive and highly consistent gestures, which normally accompany social communication in the Italian culture. Emblematic gestures represent a meaning without relying on spoken context (Ekman and Friesenm, 1969 and McNeill, 2005) and, unlike formal sign language, are not combined into complex gesture strings to make longer sentences (Goldin-Meadow,

1999). On the other hand, iconic gestures (Holler & Beattie, 2003) are hand gestures that represent meaning that is closely related to the semantic content of the segments of speech that they accompany (McNeill, 2005). For example, the utterance: “she’s

eating the food”, might be accompanied by the iconic gesture “left hand moves toward the mouth” (Beattie & Shovelton, 1999). Iconic gestures are typically large complex movements that are performed relatively slowly and carefully in the central gesture

space, but they can also be small and fast; speech and gesture refer to the same event, but each presents a somewhat different aspect of it (McNeill, 1992). The meaning of words and gestures is processed in an integrated manner, as suggested by an ERP study showing larger N400 responses to target words preceded by incongruent gestures (Holle & Gunter, 2007). In the present study, gestures used as stimuli did not express an emotional state, but delivered precise semantic information about speakers’ plans, desires, motivations, attitudes, believes and thoughts (such as: “I am getting out of here”, “I am not letting you do this”, “it’s cold”, “it was so tall”, “he is lying”, “it happened a long time ago”,

“there were so many”).

Each gesture was performed by 6 different actors (3 males and 3 females) and specifically validated for its clarity and comprehensibility by an independent group of 18 judges before ERP testing. Each selected gesture was associated with a short descriptive label that could be either congruent or incongruent with the gesture itself. We expected to observe an N400 modulation of ERP responses as a function of sentence-image congruence. Electromagnetic tomography (LORETA) was applied to the bioelectrical activity recorded during the processing of congruent and incongruent

gestures and to the differential activity. Because the gestures and related body language were semantically, rather than emotionally, pregnant in nature, we expected to find a scarce involvement of emotional areas compared to the previous study in which actors displayed emotional states (e.g., “I am rather disappointed”, “I am in love with you”, “I am deeply disgusted” “I am so ashamed”, etc.”; Proverbio et al., 2014). Furthermore, we expected to find a greater involvement of linguistic brain areas, with some similarity to neural circuits involved in sign-language processing (Levänen et al., 2001). Therefore, although the two set of stimuli (people gestures involving face and body expressions) and experimental paradigms were very similar across the two

studies, we expected to find a large difference in the neural mechanism subserving gesture comprehension, based on the different quality of information conveyed (i.e. emotional, vs. semantic).

2. Material and methods

2.1. Participants

Fourteen healthy right-handed Italian University students (7 males and 7 females) were recruited for this experiment. Ages ranged from 20 to 27, with a mean age of 26 years. All had normal or corrected to normal vision and reported no history of neurological illness or drug abuse. Their handedness was assessed by the Italian version of the Edinburgh Handedness Inventory, which is a laterality preference questionnaire that reported strong right-handedness and right ocular dominance in all participants. Data from all participants were included in all analyses. Experiments were conducted with the understanding and written consent of each participant according to the Declaration

of Helsinki (BMJ 1991, 302:1194) with approval from the Ethical Committee of university of Milano-Bicocca and in compliance with APA ethical standards for the treatment of human volunteers (1992, American Psychological Association). The subjects obtained university credits (CFU) for their participation.

2.2. Stimuli and material

A list of spontaneous gestures typically used during conversations was collected for the creation of the stimulus set. This list counted 187 gestures. To illustrate these gestures, we photographed six actors (3 male and 3 female), asking them to use the gestures spontaneously. Photographs were taken in a professional photography studio while the

actors stood in front of the camera in a white hall in light-controlled conditions. The list of gestures was given to each actor, indicating that they should spontaneously express each of the 187 gestures on the list. In this way, 1122 gestures were photographed, six for each of 187 gestures (see Fig. 1 for some examples of stimuli). Half of the pictures were assigned to the congruent condition, and the other half were assigned to the incongruent condition. In the congruent condition, the pictures were associated with congruent verbal descriptions of the gestures; in the incongruent condition, the pictures were associated with incongruent verbal descriptions of the gestures.

Fig. 1. Examples of gestures, displayed by male and female actors, associated with a congruent verbal description (congruent condition).

Figure options Stimuli were balanced so that each actor appeared in an equal number of congruent and incongruent stimuli. Each gesture was associated with three congruent stimuli and three incongruent stimuli so that each verbal description appeared six times, three

times associated with congruent pictures, and three times associated with incongruent pictures.

To test the validity of the pictures, they were presented to a group of 18 judges (8 men, 10 women) between 20 and 43 years of age. These judges were asked to judge the coherence between the gesture shown in each picture and the associated verbal label, on a scale from 0 to 2 (0 = not appropriate, 1 somewhat appropriate, 2 = very appropriate).

All pictures were randomly ordered, one per page, in a Power Point file with their associated verbal descriptions and presented to the 18 judges. The experimenter showed the judges the pictures one by one for a few seconds each and asked them to quickly assess the coherence, as described above.

Following this procedure, 19 gesture categories were discarded for the following reasons:

(1) gestures received low validation scores (between 0.30 and 1.70);

(2) gestures were not recognized by all subjects because of their regional specificity (only used in restricted geographic areas of Italy);

(3) gesture comprehension was reduced by the static property of a picture.

At the end of this process, we selected 800 stimuli (half of which were congruent and

half incongruent).

The stimuli were equiluminant; an ANOVA revealed no difference in picture luminance across the categories (congruent = 13.66 cd/m2, incongruent = 13.58 cd/m2). The verbal descriptions had the same length; an ANOVA revealed no difference in the number of words (congruent = 2.74 words, incongruent = 2.78) and no difference in the number of letters (congruent = 12.12 letters; incongruent = 12.19). The verbal descriptions were presented in the Times New Roman font and were written

in yellow on a black background. Each verbal description was presented in short lines (1–3 words per line) for 700 ms at the center of the PC screen with an inter-stimulus interval (ISI) that ranged from 100 to 200 ms and were followed by the corresponding picture, which was presented for 1200 ms with an ISI of 1500 ms. The outer background was light gray.

To obtain an optimal temporal synchronization between EEG/ERP recordings and brain processing of gestures, images were flashed as static pictures (not as dynamic short movies). Therefore, stimuli were not so ecological from this point of view, but they were

created so that each photograph captured the most salient moment of the gestural action.

2.3. Procedure

The task required participants to respond as accurately and quickly as possible to the congruence of the pictures and descriptions. To respond that a pair was congruent, participants pressed a response key with their index finger (of the left or right hand), and to respond that a pair was incongruent, they pressed a response key with their middle finger (of the left or right hand). At this regard, previous literature has shown no significant difference in response speed between the two fingers (index vs. middle one)

in button press tasks (Lachnit & Pieper, 1990). The hand used was alternated during the recording session. Hand order and the task conditions were counterbalanced across subjects. At the beginning of each session, participants were reminded which

response hand would be used. They were seated comfortably in a dark, acoustically and electrically shielded test area. Participants faced a high-resolution VGA computer screen located 100 cm from their eyes, and they were instructed to gaze at the center of the screen, where a small blue circle served as a fixation point, and to avoid any eye or body movements during the recording session. Stimuli were presented in a random order at the center of the screen in 12 blocks of 66–74 trials that lasted approximately 4 min each. Each block was preceded by a warning signal (a red cross) presented for 700 ms. The experimental session was preceded by a training session that included two runs, one for each hand. The sequence presentation order varied across subjects.

2.4. EEG recording and analysis

The EEG was continuously recorded from 128 scalp sites at a sampling rate of 512 Hz using tin electrodes mounted in an elastic cap (Electro-Cap) and arranged according to the international 10–5 system (Oostenveld & Praamstra, 2001). Horizontal and vertical eye movements were also recorded. Average mastoids served as the reference lead. The EEG and electro-oculogram (EOG) were amplified with a half-amplitude band pass of 0.016–70 Hz. Electrode impedance was maintained below 5 kΩ. EEG epochs were synchronized with the onset of gesture presentation. A computerized artifact rejection criterion was applied before averaging to discard epochs in which eye movements,

blinks, excessive muscle potential or amplifier blocking occurred. The artifact rejection criterion was a peak-to-peak amplitude exceeding 50 μV, and the rejection rate was ∼5%. ERPs were averaged off-line from −100 ms to 1200 ms after stimulus onset. ERP components were identified and measured with reference to the average baseline voltage over the interval of −100 to 0 ms at the sites and latencies at which they reached their maximum amplitudes. EEG epochs associated with an incorrect behavioral response were excluded. The amplitude of the posterior P300 response was measured at occipito/temporal (P9, P10, PPO1, PPO2, TP7, TP8) sites between 350 and 450 ms. The anterior N400 mean area amplitude was quantified at dorsolateral

(F1, F2) and central (FCC1h, FCC2h, Fp1, Fp2) frontal sites in the 380–460 ms time window. The mean area amplitude of the Late Positivity (LP) was quantified at frontal (F1, F2) and central (C1, C2) sites in the 600–800 ms time window. Fig. 2 shows the

surface topographic distribution of brain potentials relative to each ERP component of interest.

Fig. 2. Isocolour topographical maps (top view) relative to surface potentials recorded in between 350 and 450 ms for P300 response, 380–460 ms for N400 response, 600–800 ms for Late Positivity, corresponding to the temporal windows used to quantify the amplitude of ERP components.

Figure options

ERP data were subjected to multifactorial repeated-measures ANOVAs with three within group factors: Condition (Congruent, Incongruent), Electrode (dependent upon the ERP component of interest) and Hemisphere (left, right). Multiple comparisons of

means were performed with Tukey’s post hoc tests.

Topographical voltage maps of the ERPs were made by plotting color-coded isopotentials obtained by interpolating voltage values between scalp electrodes at specific latencies. Low-resolution electromagnetic tomography (LORETA; Pascual-Marqui, Esslen, Kochi, & Lehmann, 2002) was performed on the ERP waveform from the anterior N400 (380–460 ms). LORETA is a discrete linear solution to the inverse EEG problem, and it corresponds to the 3D distribution of neural electric activity that maximizes similarity (i.e., maximizes synchronization) in terms of orientation and

strength between neighboring neuronal populations (represented by adjacent voxels). In this study, an improved version of standardized weighted low-resolution brain electromagnetic tomography (sLORETA) was used; this version incorporates a singular value decomposition-based lead field weighting (i.e., swLORETA, Palmero-Soler, Dolan, Hadamschek, & Tass, 2007). The source space properties included a grid spacing (the distance between two calculation points) of 5 points and an estimated signal-to-noise ratio, which defines the regularization, of 3 (higher values indicating less regularization and therefore less blurred results). SwLORETA was performed on the group data and identified statistically significant electromagnetic dipoles (p < 0.05);

increases in the magnitudes of these dipoles correlated with more significant activation. Reaction times (RTs) that exceeded the mean value ±2 standard deviations were discarded, which resulted in a rejection rate of 2%. Error rate percentages were converted to arcsin values. Both RTs and error percentages were subjected to separate multifactorial repeated-measures ANOVAs with (1) between-subject factor (gender: male or female) and (2) within-subjects factors (condition: congruent or incongruent; and response hand: left or right).

3. Results

3.1. Behavioral results

Analysis of the reaction times (RTs) revealed a main effect of response hand (F1,13 = 16.217, p < 0.0055) that was due to the right hand responses (770 ms, SE = 26.10) being faster than those of the non-dominant hand (805 ms, SE = 30.18). Stimulus congruence (F1,13 = 4.552, p = 0.05) also affected RTs, such that congruent

stimuli (774 ms, SE = 27.09) were evaluated more quickly than incongruent stimuli (800 ms, SE = 29.93). The accuracy data indicated that fewer errors were committed in response to incongruent pictures (3%, SE = 0.72) than in response to congruent

pictures (9%, SE = 0.86), and the corresponding main effect of congruence was significant (F1,13 = 125.72, p < 0.05). No other factors significantly affected accuracy.

3.2. Electrophysiological data

Fig. 3 displays grand-average ERPs recorded at various anterior, central and posterior electrode sites in response to congruent vs. incongruent gestures. Gesture-description incongruence was associated with an increased anterior N400 response, while the congruence was associated with an enhanced posterior P300 component, followed by a huge Late positivity (LP) complex.

Fig. 3. Grand-average ERP waveforms recorded at left and right prefrontal, fronto/central, temporo/parietal and occipito/temporal electrode sites in response to congruent and incongruent gestures.

Figure options

3.3. P300 (350–450 ms)

The ANOVA performed on the mean amplitude values of the P300 revealed a significant effect of Condition (F(1,13) = 18.242; p < 0.05) that was due to a stronger response to congruent (3.09 μV, SE = 0.30) than incongruent (1.38 μV, SE = 0.42) gestures. The significant effect of Electrode (F(2,26) = 10.266; p < 0.05) indicated the presence of a larger P300 at posterior occipital (4.01 μV, SE = 0.71) than parietal (2.14 μV, SE = 0.40) and parieto/temporal (1.24 μV, SE = 0.24) sites. The ANOVA also revealed a significant effect of hemisphere (F(1,13) = 7.915; p < 0.05) that was due to a

greater response on right hemisphere (2.89 μV, SE = 0.40) than on left hemisphere (2.09 μV, SE = 0.33). The ANOVA also revealed a significant Condition × Electrode interaction (F(2,26) = 33.611; p < 0.0001) that was driven by larger P300 responses to congruent gestures at posterior sites (5.17 μV, SE = 0.65; post hoc comparison: p < 0.00013). The significant Condition × Hemisphere interaction (F(1,13) = 20.513; p < 0.00057) indicated much greater responses to congruent stimuli in the right (3.69 μV, SE = 0.38) than the left (2.50 μV, SE = 0.29) hemisphere (post-hoc tests: p < 0.000203).

3.4. N400 (380–460 ms)

The ANOVA performed on the mean amplitudes of the anterior N400 revealed a significant effect of Condition (F(1,13) = 55.023, p < 0.00001) due to greater responses to incongruent (−2.50 μV, SE = 0.96) than congruent images (0.23 μV, SE = 0.99). The significant Condition × Electrode interaction (F(2,26 = 9.754, p < 0.0068) showed a larger effect of incongruence on frontal and centro/frontal then prefrontal sites. The main effect of Hemisphere was also significant (F(1,13) = 6.515, p < 0.024), indicating a greater N400 response over the left hemisphere (−1.26 μV, SE = 0.98) than the right hemisphere (−1.00 μV, SE = 0.94), but the significant Electrode × Hemisphere

interaction (F(2,26) = 5.111, p < 0.014) showed a lack of asymmetry at prefrontal sites (Fp1: −1.32 μV, SE = 1.10; Fp2: −1.35 μV, SE = 1.09; F1: −1.34 μV, SE = 1.01; F2: −0.87 μV, SE = 0.99.; FCC1 h: −1.13 μV, SE = 0.96; FCC2 h: −0.77 μV, SE = 0.88). To locate the possible neural source of the N400 response, three swLORETA source reconstructions were applied to the brain voltage recorded in the Congruent and Incongruent conditions and on the difference-waves obtained by subtracting the ERPs to congruent gestures from those to incongruent gestures in the 380–460 ms time window. Table 1 provides a list of significantly active sources explaining the N400 surface voltage to congruent stimuli, along with the Talairach coordinates of the

corresponding neural generators. The strongest sources of activation in response to congruent gestures corresponded with occipito/temporal visual areas involved in face and body processing, such as: the right lingual gyrus (BA19), right fusiform gyrus (BA37), right precuneus (BA31), right STG (BA22), right Middle temporal gyrus (BA37), and left occipital area (BA18). Conversely, the most active sources during processing of incongruent gestures corresponded to the right precuneus (BA31), the right STG (BA22), the cingulate cortex (BA31) and the left and right pulvinar, with a right hemispheric asymmetry (see source reconstruction displayed in Fig. 4).

Table 1. Talairach coordinates (in mm) corresponding to intracranial generators explaining the N400 surface voltage recorded in response to congruent stimuli in the 380–460 ms time window, according to swLORETA. Magn. = magnitude in nAm; Hem. = hemisphere.

Magn. T − x [mm] T − y [mm] T − z [mm] Hem. Lobe Gyrus BA


CONGRUENT power RMS = 160.7

13.87 21.2 −46.8 −2.1 R O Lingual gyrus 19

13.77 40.9 −55.9 −10.2 R T Fusiform gyrus

37

13.65 31 −70 22.5 R O Precuneus 31

13.64 50.8 −48.7 15.3 R T Superior temporal gyrus

22

13.49 40.9 −75.2 −19.1 R T Middle

temporal gyrus

37

12.68 −18.5 −98.5 2.1 L O Cuneus 18

12.64 −38.5 −69 13.6 L T Middle temporal gyrus

39

12.44 50.8 −0.6 −28.2 R T Middle temporal

gyrus

21

12.41 −8.5 −0.6 −28.2 L Limbic Uncus 28

12.18 −38.5 −50.7 33.1 L P Inferior parietal lobule

40

12.12 −38.5 −28.5 17.1 L T Superior temporal gyrus

41

11.76 −38.5 43.4 23.9 L F Middle frontal gyrus

10

11.42 −8.5 57.3 −9 L F Superior frontal gyrus

10

11.27 −38.5 −21 35.7 L P Postcentral gyrus

3

Table options

Fig. 4. Left and right sagittal views of significant intracranial sources of activation for the processing of Incongruent gestures in the latency range 380–460 ms corresponding to the peak of N400 response. The inverse solution was applied to the grand average signals (N = 14). The different colors represent differences in the magnitudes of the electromagnetic signals (in nAm). The electromagnetic dipoles are shown as arrows and indicate the position, orientation and magnitude of the dipole modeling solutions that were applied to the ERP waveforms in the specific time windows. The numbers refer to the displayed brain slice in the axial view: A = anterior, P = posterior.

Figure options The swLORETA revealed activation in the left superior (BA 10) and middle (BA 9) frontal gyri to congruent gestures, but a greater and bilateral activation of DLPFc for incongruent gestures (see Table 2 for a list of electromagnetic dipoles active during processing of incongruent gestures). The swLORETA also revealed an activation involving the fronto-parietal mirror system. Specifically, there was activity in the left

postcentral gyrus (BA 3) during processing of congruent and incongruent gestures, and in the left inferior parietal lobule (BA 40) only in response to congruent gestures.

Table 2. Talairach coordinates (in mm) corresponding to intracranial generators explaining the N400 surface voltage recorded in response to incongruent stimuli in the 380–460 ms time window, according to swLORETA. Magn. = magnitude in nAm; Hem. = hemisphere.


INCONGRUENT power RMS = 168.5

13.41 31 −70 22.5 R O Precuneus 31

13.40 50.8 −48.7 15.3 R T Superior temporal gyrus

22

12.90 11.3 −40.6 34 R Limbic Cingulate gyrus

31

12.71 11.3 −27.5 8.2 R Sub-lobar

Thalamus, pulvinar

12.70 −18.5 −98.5 2.1 L O Cuneus 18

12.60 −8.5 −27.5 8.2 L Sub-lobar

Thalamus, pulvinar

12.59 40.9 −86.4 −12.4 R O Inferior occipital gyrus

18

12.37 −28.5 −51.7 42 L P Superior parietal lobule

7

12.24 −38.5 −28.5 17.1 L T Superior temporal gyrus

41

12.23 −48.5 −36.6 −1.3 L T Middle temporal

22


gyrus


3

11.23 50.8 33.4 23.1 R F Middle frontal gyrus

46


9

7.73 11.3 65.3 7.9 R F Superior frontal gyrus

10

6.56 −8.5 57.3 −9 L F Superior frontal gyrus

10

6.38 11.3 57.3 −9 R F Superior frontal gyrus

10

Table options

A series of activations in the temporal area were common to the two conditions and

involved the left and right superior temporal gyri (BA 41, BA 22), which roughly correspond to the primary auditory cortex and to Wernicke’s area, in addition to the left middle temporal gyrus (BA 21) for incongruent gestures and bilaterally the middle temporal gyrus (BA 21, BA39) for congruent gestures. In addition, areas belonging to the occipital cortex were found to be active in both conditions (BA 17, BA18).

To better appreciate the difference between the two conditions, a further swLORETA was applied to the grand-average difference wave that was obtained by subtracting the

ERPs elicited by congruent images from those elicited by incongruent images (see Table 3 for a list of significant dipoles). The processing of incongruent gestures was associated with significant activities in the thalamus and linguistic brain areas such as the medial temporal gyrus (BA21), the fronto-parietal mirror system (BA6, BA40), involved in action representation and understanding of others’ intentions, and limbic areas (BA28, BA34) possibly involved in the emotional reaction to deviance.

Table 3.

Talairach coordinates (in mm) corresponding to intracranial generators explaining the N400 difference voltage obtained by subtracting ERPS to congruent from incongruent stimuli in the 380–460 ms time window, according to swLORETA.

Magn. = magnitude in nAm; Hem. = hemisphere.

Magn.

T − x [mm]

T − y [mm]

T − z [mm]

Hem. Lobe Gyrus

BA

INCONGRUENT–CONGRUENT power RMS = 47.7

11.19 11.3 −9.4 −14 R Limbic

Parahippocampal gyrus

34

11.18 −8.5 −26.5 −0.6 L Sub-lobar

Thalamus

11.16 11.3 −18.2 0.1 R Sub-lobar

Thalamus

11.16 −8.5 −0.6 −28.2 L Limbic

Uncus 28

11.11 −58.5 −25.5 −8.1 L T Middle temporal gyrus

21

9.63 1.5 −29.4 26 R Limbi

c

Cingulate gyrus 23

9.62 50.8 −0.6 −28.2 R T Middle temporal gyrus

21

9.47 40.9 −75.2 −19.1 R O Fusiform gyrus 19

7.24 −28.5 53.4 24.8 L F Superior frontal gyrus

10


3


6

5.06 40.9 −30.4 34.9 R P Inferior parietal lobule

40

4.80 40.9 2.4 29.4 R F Precentral gyrus 6

Table options

3.5. LP response (600–800 ms)

The ANOVA performed on the mean amplitudes of central LP responses revealed a significant effect of Condition (F(1,13) = 151.4; p < 0.00001) indicating much larger LP amplitudes to congruent (7.08 μV, SE = 0.89) than incongruent (2.49 μV, SE = 0.74) stimuli. The LP response was much larger over the right central area, as indicated by the significance of Electrode (F(1,13) = 5.07, p < 0. 0.4), Hemisphere (F(1,13) = 9.28, p < 0.09) and the interaction between Electrode × Hemisphere (F(1,13) = 12–02,p < 0.005), with relative post-hoc comparisons (C1 = 4.91 μV; C2 = 5.76 μV, p < 0.01; F1 = 4.12, F2 = 4.36, n.s.).

4. Discussion

In this study we compared brain processing of face/body gestures that were either

congruent or incongruent with a previous verbal description. The task required participants to determine whether the gesture and description were congruent, and behavioral data indicated that responses made with the right hand were much faster than those made with the left hand, most likely related to participants’ right-handedness (their mean score of lateral preference was 0.81 on a −1 to 1 scale). This result is consistent with a finding from our previous study on emotional body language (EBL) (Proverbio et al., 2014). In addition, RTs were faster to congruent vs. incongruent gestures, which again is consistent with previous results in the literature (Bernardis et al., 2008, Wu and Coulson, 2005 and Wu and Coulson, 2007). According to Wu and

Coulson, 2005 and Wu and Coulson, 2007 incongruent stimuli require additional processing times and computing resources compared to congruent ones. In their two experiments, they found that participants were approximately 165 ms faster to classify gestures that were congruent with associated words than the reverse condition. The effect of congruence on reaction times also emerged when subjects were not asked to classify the gesture but, instead, the word associated with it. It might be speculated that, in our study, accuracy data indicated a greater categorization certainty for incongruent stimuli. The mean error percentage for congruent stimuli was 9%, and only 3% for incongruent stimuli. A similar asymmetry in

favor of gestures judged as incongruent was observed by Wu and Coulson (2005), who obtained an error rate of 29% for congruent gestures and 15% for incongruent ones, andProverbio et al. (2014), who found error rates of 20.9% for congruent and 7.7% for

incongruent emotional body language (EBL). This effect most likely arises because verbal description creates a specific expectation that is clearly violated by an incongruent gesture, while it may be more time consuming to ascertain a perfect

correspondence for congruent gestures, presumably because of the large variability in descriptions that are compatible with a single gesture (for example, a picture of a freezing woman is compatible with: “Its’ so cold!” or “I am terribly cold!”, or also “Brr..I am freezing” and so on, but it is drastically incompatible with: “It’s really hot”. Electrophysiological results showed a posterior P300 component as the earliest recognition index of gesture congruence. The P300 was larger in response to congruent behavior in the time window between 350 and 450 ms. This congruence effect was more evident over the right visual area (i.e., PPO1 and PPO2), which possibly reflected the recognition activity of cortical body- and face-related areas. This

result is consistent with previous studies in the literature on body language recognition. For example, the P300 was larger to congruent EBL in the Proverbio et al. (2014) study, and a study by Shibata, Gyoba, and Suzuki (2009) found greater P300 responses to congruent actions when participants evaluated the appropriateness of cooperative actions between two people. The P300 response was followed a large late positive component (LP) which shared the same properties with the P300 response but had a central distribution, possibly indicating higher-order cognitive processes, such as decision making and awareness (Düzel, Yonelinas, Mangun, Heinze, & Tulving, 1997). It is very well known that P300 response is larger to match than mismatch trials in task

requiring a decision making (Duncan-Johnson & Donchin, 1982). P3 central distribution is thought to reflect its underlying generators located in the prefrontal and anterior cingulate cortex (Winterer et al., 2001). As in our study, the amplitude of central late P3 response is clearly reduced in incongruent trials compared to congruent trials (Mahé et al., 2014, Neuhaus et al., 2007 and Neuhaus et al., 2010), thus suggesting its role in stimulus categorization and cognitive updating. Similar to what we found in our previous ERP study on EBL comprehension (Proverbio et al., 2014), perception of incongruent gestures was associated with an anterior negativity (N400) in the 380–460 ms time window. A similar response (anterior N400) to incongruent actions has been found in studies showing actions followed by an

unexpected ending (Reid & Striano, 2008), an object/context incongruence (Sitnikova, Kuperberg, & Holcomb, 2003), meaningless hand postures (Gunter & Bach, 2004), incongruent gestures with respect to cartoons (Wu and Coulson, 2005 and Wu and

Coulson, 2007), sport-related incorrect actions (Proverbio, Crotti, Manfredi, Adorni, & Zani, 2012), purposeless and meaningless complex behavior (Proverbio, Riva, & Zani, 2010), and incongruent affective body language (Proverbio et al., 2014). According

to Proverbio and Riva (2009), the N400 reflects the activation of brain mechanisms that automatically process biological actions as meaningful units by means of visuomotor mirror neurons belonging to the fronto-parietal action/observation system (Fogassi et al., 2005 and Rizzolatti et al., 2001). In this context, an action-related N400 would reflect a difficulty in integrating the gesture significance with the semantic context and previous world knowledge (Hagoort, Hald, Bastiaansen, & Petersson, 2004), thus suggesting a functional similarity with a linguistic N400 response. In the present study, to identify the neural generators of surface potentials, distinct swLORETA inverse solutions were applied to the group average voltages recorded at

the peak of the N400 response to congruent and incongruent gestures, as well as to the difference wave resulting from the contrast between incongruent and congruent pairs. The regions more strongly activated in response to congruent gestures were the right occipito/temporal areas that are supposedly involved in face and body processing. These areas include BA 18/19 for body parts and face details (as supported by a large neuroimaging literature, OFA: eyes, hands, mouths; Pitcher, Duchaine, Walsh, Yovel, & Kanwisher, 2011), BA19/37 for faces, bodies (FFA and EBA, Downing, 2001, Downing et al., 2011, Grill-Spector et al., 2004, Schwarzlose et al., 2005 and Urgesi et al., 2004) and body-related actions (Astafiev, Stanley, Shulman, &

Corbetta, 2004), superior temporal gyrus (STG, BA22) for biological motion (Grossman & Blake, 2002), lips reading, sign language and affective emotional expression (Iacoboni et al., 2001 and Okada and Hickok, 2009). In the incongruent condition, the strongest activations involved the right cuneus and the STG, suggesting a crucial role for these multimodal regions in gesture understanding, besides audiovisual speech processing. Interestingly, Calvert and coworkers (Calvert et al., 1997) identified portions of the STS and superior temporal gyrus (STG) that were activated by heard speech and silent lip-reading, and hypothesized that portions of this activation overlapped with primary auditory cortex (Bernstein et al., 2002). Indeed it seems that these area (and especially STS) particularly responds to signs not merely to verbal

speech (Wright, Pelphrey, Allison, McKeown, & McCarthy, 2003). Holle, Gunter, Rüschemeyer, Hennenlotter, and Iacoboni (2008) found a greater activation of the left STS during perception of iconic gestures than meaningless grooming gestures, thus

suggesting a specific role of STS in gesture comprehension. Consistently, the fMRI study by Emmorey, Xu, and Braun (2011) performed in ASL signers provided evidence of an increased activation of left posterior superior temporal gyrus (STG) for

linguistically structured pseudosigns compared to baseline. Therefore it seems that, although STS has been show to process multimodal stimuli (such as audiomotor inputs) being involved, for example, in lips reading, the specific response to gestures in our and other studies had not to do with phonetic, but rather visuomotor processing. Indeed, a study contrasting the processing of meaningful emblems with meaningless gestures in deaf signers and hearing non-signers revealed that the processing of meaningful stimuli activated the left posterior STG (BA 42/22), to a greater extent in the deaf compared to hearing people, thus indicating a specific role of the left STS in the processing of meaningful semantic gestures.

Visual processing of gestures (both congruent and incongruent) also activated regions belonging to the action observation system, such as the left postcentral (BA3) and precentral (BA6) gyri and the inferior parietal lobule (BA40). These regions, thought to index the activities of neurons representing action meaning and goals, as well as the agent intentions (Enticott et al., 2010, Goldenberg and Spatt, 2009,Iacoboni et al., 2005, Proverbio et al., 2014, Rizzolatti et al., 1996 and Rizzolatti et al., 2001), were particularly active in detecting the gesture incongruence (as evident in the swLORETA relative to the incongruent – congruent contrast), thus further supporting their role in action meaning understanding.

In both conditions, but with a stronger amplitude in the incongruent condition, processing of gestures activated the left superior temporal gyrus (BA41), which roughly corresponds to the primary auditory cortex, and the left BA22, corresponding to Wernicke’s area, also important for phonological analysis and word understanding. This piece of data, which indicates a multimodal processing of linguistic information in these areas, is consistent with a recent fMRI study by Dick et al. (2009), which showed that a discourse with accompanying hand movements increased neural activity in regions typically associated with auditory comprehension of language including the left IFG (Broca area), left primary auditory cortex, and bilateral STGp, MTGp, STSp, and IPL.

Returning to the present ERP study, during processing of both stimulus types, common activations of the superior frontal gyrus (BA10) and the dorsolateral prefrontal cortex (DLPF, BA9) were found. The prefrontal activity showed a left hemispheric

lateralization during the processing of congruent gestures, but it was bilaterally and more strongly engaged during the processing of incongruent gestures. The frontal activation might be associated with task requirements (common to both conditions),

thus reflecting decision-making, working memory and cognitive processes that lead the subjects to respond. Similar activations were found by Proverbio et al. (2014) with EBL processing, by Proverbio et al. (2012)comparing correct and incorrect basketball actions, and by Proverbio et al. (2010), in which they compared processing of meaningful vs. purposeless behaviors. However, the stronger activation of DLPF in the incongruent than congruent condition suggests that the former task was more demanding in terms of information coding and retrieval because it engaged both cerebral hemispheres instead of only the left one (Habib, Nyberg, & Tulving, 2003). Indeed, several studies have provided evidence that DLPF cortex is modulated by task

demands, being more activated during complex than easier decision making (Paulus et al., 2002, Rushworth et al., 2004 and Zysset et al., 2006) and during processing of conflicting vs. non conflicting information (Oehrn et al., 2014 and Wittfoth et al., 2009). The major difference between the incongruent and congruent conditions, also highlighted by the swLORETA applied to the difference wave between the incongruent and congruent conditions, was represented by the activation of the limbic areas (potentially indexing an emotional reaction triggered by the detection of stimulus deviance or mismatch), the thalamus, and the medial temporal gyrus (BA21/22). As for the thalamic source, the ability of the swLORETA algorithm to localize subcortical

sources has been demonstrated by previous work. For example, in the combined EEG/neuroimaging study by Bocquillon et al. (2012) involving an attentional task with distracters, P300 sources were localized in the basal ganglia using standardized weighted low-resolution electromagnetic tomography (swLORETA). Subcortical sources for human EEG have also been found, for example in a PCA study (Gómez-Herrero, Atienza, Egiazarian, & Atienza, 2008) involving a novel methodology based on multivariate autoregressive (MVAR) modeling and Independent Component Analysis (ICA) that can determine the temporal activation of the intracerebral EEG sources as well as their approximate locations. In this study, EEG was recorded from 20 cognitively normal volunteers in a relaxed wakeful state with their eyes closed. The

single electrical dipoles most likely to be generating the resulting alpha rhythm were located in the caudal regions of the thalamus (x = 9, y = 25, z = 9), in the precuneus (x = 2, y = −60, z = 28), and in the middle occipital gyrus, within the limits of the cuneus

(x = 11, y = −97, z = 13). In the present study, one might hypothesize a role for the visual thalamus, and especially the pulvinar, in the automatic orienting of attention vs. incongruent information ( Rafal and Posner, 1987 and Yuri et al., 2012), which has also

been related to the development of alpha EEG rhythm. On a different interpretative line, Wahl et al. (2008) have provided evidence of the role of the thalamus in generating an N400 potential in response to linguistic syntactic and semantic violations. They collected monopolar thalamic recordings that were referenced to linked mastoids and found that syntactic phrase violations induced increased ERP deflections (with an early and a late negativity approximately 200 and 600 ms), while semantic phrase violations induced a sustained monophasic ERP negativity in the same thalamic sites. Other neuroimaging and electromagnetic studies have identified the neural generators of a linguistic N400 response to deviance (see a thorough review in Dien, Michelson, &

Franklin, 2010). For example, Nobre and McCarthy, 1995 and McCarthy et al., 1995 recorded field potentials from isolated non-words and anomalous sentence-ending words using intracranial recordings and found a large negative field potential with a peak latency near 400 ms (N400), which was focally distributed bilaterally in the anterior medial temporal lobe (AMTL). More recent intracranial electroencephalographic studies (Dietl et al., 2008, Fell et al., 2004 and Meyer et al., 2005) have identified potential N400 sources in the bilateral AMTL (using auditory and visual sentences). In contrast, magnetoencephalography (MEG) studies (Laine et al., 2000,Maess et al., 2006, Mäkelä et al., 2001, Service et al., 2007 and Simos et al.,

1997) have pointed toward the middle temporal gyrus (MTG)/superior temporal gyrus (STG) region, while some other MEG studiesHelenius et al., 1998 and Helenius et al., 1999) have indicated that both the AMTL and the MTG/STG are sources for the N400 semantic incongruence effect. The source reconstruction relative to the gestural N400 effect, in our study, was associated, on one hand with significant activations in the thalamus and linguistic brain areas such as the medial temporal gyrus (BA21), and on the other hand in fronto-parietal areas involved in action representation and understanding of others’ intentions (BA6, BA40), thus suggesting their conjoined role in extracting (the latter) and constructing (the former) precise semantic information from visuomotor inputs.

One possible limitation of the present study results from the limited spatial resolution of EEG that arises from the possible signal distortion while traveling through cerebral tissues. However, the source reconstruction capability of LORETA is considered

relatively good and can be extremely focused, especially in the swLORETA version (Palmero-Soler et al., 2007). In particular, since EEG can mostly detect activity coming from the cortical gyri (e.g. STG), and not sulci (e.g., STS), its low spatial resolution

does not allow a clear anatomical demarcation between the two areas, located a few mm apart. However, fMRI literature (e.g., Calvert et al., 1997) has provided evidences of shared STG/STS functional properties as far as gesture understanding is concerned.

5. Conclusions

This experiment investigated the neural mechanisms supporting the comprehension of a large variety of spontaneous gestures. Typical Italian gestures involving facial mimicry and characteristic hand and body articulations were associated with an incongruent description in half of the cases. An anterior N400 indexed the mismatch detection, and its generators included some linguistic syntactic and semantic related brain areas (such as the left medial temporal lobe for semantic processing, the superior temporal lobe for audiovisual speech and the thalamus, also for semantic processing). Activation also included areas included in the action observation system (such as the

premotor cortex and IPL) and areas involved in face and body processing (FFA, EBA, STS). The results indicate the existence of a complex neural system for understanding spontaneous sign language, which is supposedly halfway between a formal sign language and an emotional body language (EBL) system. This result allows us to speculate that there has been a phylogenetic transition between a solely gestural and a verbal language communication systems in humans, as advanced for example by Arbib and coworkers (Arbib, 2010 and Arbib et al., 2008). They argue that “it was the coupling of gestural communication with enhanced capacities for imitation that made possible the emergence of protosign to provide essential scaffolding for protospeech in

the evolution of protolanguage”.

Acknowledgments

We are very grateful to Roberta Adorni for her precious help with EEG recording. We also wish to thank our experimental subjects for their kind participation.

References

1. o Andric et al., 2013

o M. Andric, A. Solodkin, G. Buccino, S. Goldin-Meadow, G. Rizzolatti, S.L. Small o Brain function overlaps when people observe emblems, speech, and grasping

o Neuropsychologia, 51 (8) (2013), pp. 1619–1629

o Article | PDF (4316 K) | View Record in Scopus |

Citing articles (6) 2. o Arbib, 2010 o M.A. Arbib o Mirror system activity for action and language is embedded in the integration of

dorsal and ventral pathways

o Brain and Language, 112 (1) (2010), pp. 12–24

o Article | PDF (677 K) | View Record in Scopus | Citing articles (30)

3. o Arbib et al., 2008

o M.A. Arbib, K. Liebal, S. Pika o Primate vocalization, gesture, and the evolution of human language

o Current Anthropology, 49 (6) (2008), pp. 1053–1063

o 4. o Astafiev et al., 2004

o S.V. Astafiev, C.M. Stanley, G.L. Shulman, M. Corbetta

o Extrastriate body area in human occipital cortex responds to the performance of motor action

o Nature Neuroscience, 7 (5) (2004), pp. 542–548

o View Record in Scopus | Full Text via CrossRef | Citing articles (262)

5.

o Bach et al., 2009 o P. Bach, T.C. Gunter, G. Knoblich, W. Prinz, A.D. Friederici o 400-like negativities in action perception reflect the activation of two components of

an action representation

o Social Neuroscience, 4 (3) (2009), pp. 212–232

o View Record in Scopus

| Full Text via CrossRef | Citing articles (29)

6. o Beattie and Shovelton, 1999 o G. Beattie, H. Shovelton o Mapping the range of information contained in the iconic hand gestures that

accompany spontaneous speech

o Journal of Language and Social Psychology, 18 (4) (1999), pp. 438–462


7.

o Beattie and Shovelton, 2002 o G. Beattie, H. Shovelton

o An experimental investigation of some properties of individual iconic gestures that mediate their communicative power

o British Journal of Psychology, 93 (2) (2002), pp. 179–192


8.

o Bernardis et al., 2008 o P. Bernardis, E. Salillas, N. Caramelli o Behavioural and neurophysiological evidence of semantic interaction between iconic

gestures and words

o Cognitive Neuropsychology, 25 (7–8) (2008), pp. 1114–1128



9. o Bernstein et al., 2002 o L.E. Bernstein, E.T. Auer Jr., J.K. Moore, C.W. Ponton, M. Don, M. Singh o Visual speech perception without primary auditory cortex activation

o NeuroReport, 13 (2002), pp. 311–315


10. o Bocquillon et al., 2012

o P. Bocquillon, J.L. Bourriez, E. Palmero-Soler, A. Destée, L. Defebvre, et al.

o Role of basal ganglia circuits in resisting interference by distracters: A swLORETA study

o PLoS ONE, 7 (3) (2012), p. e34239

o Full Text via CrossRef 11. o Braun et al., 2001 o A.R. Braun, A. Guillemin, L. Hosey, M. Varga o The neural organization of discourse – An H215O-PET study of narrative production

in English and American sign language

o Brain, 124 (2001), pp. 2028–2044


12. o Calvert et al., 1997 o G.A. Calvert, E.T. Bullmore, M.J. Brammer, R. Campbell, S.C. Williams, P.K.

McGuire, et al. o Activation of auditory cortex during silent lipreading

o Science, 276 (1997), pp. 593–596


13. o De Gelder et al., 2009 o B. De Gelder, J. Van den Stock, H.K.M. Meeren, C.B.A. Sinke, M.E. Kret, M.

Tamietto o Standing up for the body. Recent progress in uncovering the networks involved in the

perception of bodies and bodily expressions

o Neuroscience and Biobehavioral Review, 34 (2009), pp. 513–527

o 14. o Dick et al., 2009

o A.S. Dick, S. Goldin-Meadow, U. Hasson, J.I. Skipper, S.L. Small o Co-speech gestures influence neural activity in brain regions associated with

processing semantic information

o Human Brain Mapping, 30 (2009), pp. 3509–3526

o View Record in Scopus |

Full Text via CrossRef | Citing articles (51)

15. o Dien et al., 2010 o J. Dien, C.A. Michelson, M.S. Franklin o Separating the visual sentence N400 effect from the P400 sequential expectancy

effect: Cognitive and neuroanatomical implications

o Brain Research, 8 (1355) (2010), pp. 126–140


16. o Dietl et al., 2008 o T. Dietl, M. Kurthen, D. Kirch, M. Staedtgen, C. Schaller, C.E. Elger, et al. o Limbic event-related potentials to words and pictures in the presurgical evaluation of

temporal lobe epilepsy

o Epilepsy Research, 78 (2–3) (2008), pp. 207–215

o Article |

PDF (507 K) | View Record in Scopus

| Citing articles (6)

17. o Downing, 2001 o P.E. Downing o A cortical area selective for visual processing of the human body

o Science, 293 (2001), p. 2470


18. o Downing et al., 2011 o P.E. Downing, M.V. Peelen, A.J. Wiggett, B.D. Tew

o The role of the extrastriate body area in action perception

o Social Neuroscience, 1 (1) (2011), pp. 52–62

o 19. o Duncan-Johnson and Donchin, 1982 o C.C. Duncan-Johnson, E. Donchin o The P300 component of the event-related brain potential as an index of information

processing

o Biological Psychology, 14 (1–2) (1982), pp. 1–52

o Article | PDF (3425 K) |

View Record in Scopus |

Citing articles (233) 20. o Düzel et al., 1997

o E. Düzel, A.P. Yonelinas, G.R. Mangun, H.J. Heinze, E. Tulving o Event-related brain potential correlates of two states of conscious awareness in

memory

o Proceedings of the National Academy of Sciences United States of America, 94 (11) (1997), pp. 5973–5978 (May 27)



1. o Ekman and Friesenm, 1969 o P. Ekman, W.V. Friesenm o The repertoire of nonverbal communication: Categories, origins, usage, and coding

o Semiotica, 1 (1969), pp. 49–98

o View Record in Scopus | Citing articles (1)

2. o Emmorey et al., 2011 o K. Emmorey, J. Xu, A. Braun o Neural responses to meaningless pseudosigns: Evidence for sign-based phonetic

processing in superior temporal cortex

o Brain and Language, 117 (1) (2011), pp. 34–38



Citing articles (6) 3. o Enticott et al., 2010

o P.G. Enticott, H.A. Kennedy, J.L. Bradshaw, N.J. Rinehart, P.B. Fitzgerald o Understanding mirror neurons: Evidence for enhanced corticospinal excitability

during the observation of transitive but not intransitive hand gestures

o Neuropsychologia, 48 (2010), pp. 2675–2680

o Article |

PDF (264 K) | View Record in Scopus | Citing articles (31)

4. o Fell et al., 2004 o J. Fell, T. Dietl, T. Grunwald, M. Kurthen, P. Klaver, P. Trautner, et al. o Neural bases of cognitive ERPs: More than phase reset

o Journal of Cognitive Neuroscience, 16 (9) (2004), pp. 1595–1604


5.

o Fogassi et al., 2005 o L. Fogassi, P.F. Ferrari, B. Gesierich, S. Rozzi, F. Chersi, G. Rizzolatti o Parietal lobe: From action organization to intention understanding

o Science, 308 (2005), p. 662


Full Text via CrossRef |

Citing articles (864) 6. o Goldenberg and Spatt, 2009

o G. Goldenberg, J. Spatt o The neural basis of tool use

o Brain, 132 (6) (2009), pp. 1645–1655

o View Record in Scopus | Full Text via CrossRef


7. o Goldin-Meadow, 1999 o S. Goldin-Meadow o The role of gesture in communication and thinking

o Trends in Cognitive Sciences, 3 (11) (1999), pp. 419–429


8. o Gómez-Herrero et al., 2008

o G. Gómez-Herrero, M. Atienza, K. Egiazarian, M. Atienza o Measuring directional coupling between EEG sources

o NeuroImage, 43 (3) (2008), pp. 497–508

o Article | PDF (1019 K)

| View Record in Scopus


9.

o Goodwin, 2003 o C. Goodwin o The body in action

o J. Coupland, R. Gwyn (Eds.), Discourse, the body and identity, Palgrave Macmillan, Basingstoke (2003), pp. 19–42



10. o Grill-Spector et al., 2004 o K. Grill-Spector, N. Knouf, N. Kanwisher o The fusiform face area subserves face perception, not generic within-category

identification

o Nature Neuroscience, 7 (2004), pp. 555–562


11. o Grossman and Blake, 2002 o E.D. Grossman, R. Blake

o Brain areas active during visual perception of biological motion

o Neuron, 35 (2002), pp. 1167–1175



Citing articles (369) 12. o Guellai et al., 2014

o B. Guellai, A. Langus, M. Nespor o Prosody in the hands of the speaker

o Frontiers in Psychology, 5 (2014), p. 700 http://dx.doi.org/10.3389/fpsyg.2014.00700 o

13. o Gunter and Bach, 2004 o T.C. Gunter, P. Bach o Communicating hands: ERPs elicited by meaningful symbolic hand postures

o Neuroscience Letters, 372 (2004), pp. 52–56

o Article | PDF (197 K) | View Record in Scopus


14. o Habib et al., 2003 o R. Habib, L. Nyberg, E. Tulving o Hemispheric asymmetries of memory: The HERA model revisited

o Trends in Cognitive Science, 7 (6) (2003), pp. 241–245


15.

o Hagoort et al., 2004

o P. Hagoort, L. Hald, M. Bastiaansen, K.M. Petersson o Integration of word meaning and world knowledge in language comprehension

o Science, 304 (2004), pp. 438–441


16.

o Hänel-Faulhaber et al., 2014 o Hänel-Faulhaber, B., Skotara, N., Kügow, M., Salden, U., Bottari, D., & Röder, B.

(2014). ERP correlates of German Sign Language processing in deaf native signers.BMC Neuroscience, 15:62 & #xA0. doi: 10.1186/1471-2202-15-62.

o 17. o Helenius et al., 1998 o P. Helenius, R. Salmelin, E. Service, J.F. Connolly o Distinct time courses of word and context comprehension in the left temporal cortex

o Brain, 121 (1998), pp. 1133–1142


18.

o Helenius et al., 1999 o P. Helenius, R. Salmelin, E. Service, J.F. Connolly o Semantic cortical activation in dyslexic readers



Full Text via CrossRef |

Citing articles (84) 19. o Hickok et al., 1996

o G. Hickok, U. Bellugi, E.S. Klima o The neurobiology of sign language and its implications for the neural basis of

language

o Nature, 381 (1996), pp. 699–702


Full Text via CrossRef | Citing articles (82)

20. o Holle and Gunter, 2007 o H. Holle, T.C. Gunter o The role of iconic gestures in speech disambiguation: ERP evidence



1. o Holle et al., 2008 o H. Holle, T.C. Gunter, S.A. Rüschemeyer, A. Hennenlotter, M. Iacoboni

o Neural correlates of the processing of co-speech gestures

o Neuroimage, 39 (4) (2008), pp. 2010–2024 o

2. o Holler and Beattie, 2003 o J. Holler, G. Beattie o How iconic gestures and speech interact in the representation of meaning: Are both

aspects really integral to the process?

o Semiotica, 146 (2003), pp. 81–116 o

3. o Husain et al., 2012 o F.T. Husain, D.J. Patkin, J. Kim, A.R. Braun, B. Horwitz

o Dissociating neural correlates of meaningful emblems from meaningless gestures in deaf signers and hearing non-signers

o Brain Research, 10 (1478) (2012), pp. 24–35 o

4. o Iacoboni et al., 2001 o M. Iacoboni, L.M. Koski, M. Brass, H. Bekkering, R.P. Woods, M.C. Dubeau, et al. o Reafferent copies of imitated actions in the right superior temporal cortex

o PNAS, 98 (2001), pp. 13995–13999 o

5. o Iacoboni et al., 2005 o M. Iacoboni, I. Molnar-Szakacs, V. Gallese, G. Buccino, J.C. Mazziotta, G. Rizzolatti o Grasping the intentions of others with one’s own mirror neuron system

o Plos Biology, 3 (3) (2005), p. e79 o

6.

o Kana and Travers, 2012 o R.K. Kana, B.G. Travers o Neural substrates of interpreting actions and emotions from body postures

o Social Cognitive and Affective Neuroscience, 7 (4) (2012), pp. 446–456 o

7. o Kirk et al., 2013 o E. Kirk, N. Howlett, K.J. Pine, B.C. Fletcher o To sign or not to sign? The impact of encouraging infants to gesture on infant

language and maternal mind-mindedness

o Child Development, 84 (2) (2013), pp. 574–590 o

8.

o Kutas and Federmeier, 2000 o M. Kutas, K.D. Federmeier o Electrophysiology reveals semantic memory use in language comprehension

o Trends in Cognitive Science, 207 (2000), pp. 203–205 o

9. o Kutas and Hillyard, 1980 o M. Kutas, S.A. Hillyard o Reading senseless sentences: Brain potentials reflect semantic incongruity

o Science, 207 (4427) (1980), pp. 203–205 o

10.

o Lachnit and Pieper, 1990 o H. Lachnit, W. Pieper o Speed and accuracy effects of fingers and dexterity in 5-choice reaction tasks

o Ergonomics, 33 (12) (1990), pp. 1443–1454 o

11. o Laine et al., 2000 o M. Laine, R. Salmelin, P. Helenius, R. Marttila o Brain activation during reading in deep dyslexia: An MEG study

o Journal of Cognitive Neuroscience, 12 (4) (2000), pp. 622–634 o

12. o Levänen et al., 2001 o S. Levänen, K. Uutela, S. Salenius, R. Hari o Cortical representation of sign language: Comparison of deaf signers and hearing

non-signers

o Cerebral Cortex, 11 (2001), pp. 506–512 o

13. o MacSweeney et al., 2002 o M. MacSweeney, B. Woll, R. Campbell, P.K. McGuire, A.S. David, S.C. Williams, et

al.

o Neural systems underlying British Sign Language and audio-visual English processing in native users

o Brain, 125 (Pt 7) (2002), pp. 1583–1593 o

14. o Maess et al., 2006 o B. Maess, C.S. Herrmann, A. Hahne, A. Nakamura, A.D. Friederici o Localizing the distributed language network responsible for the N400 measured by

MEG during auditory sentence processing


15.

o Mahé et al., 2014 o G. Mahé, N. Doignon-Camus, A. Dufour, A. Bonnefond o Conflict control processing in adults with developmental dyslexia: An event related

potentials study

o Clinical Neurophysiology, 125 (1) (2014), pp. 69–76 o

16. o Mäkelä et al., 2001 o A.M. Mäkelä, V. Mäkinen, M. Nikkilä, R.J. Ilmoniemi, H. Tiitinen

o Magnetoencephalographic (MEG) localization of the auditory N400m: Effects of stimulus duration

o NeuroReport, 12 (2) (2001), pp. 249–253 o

17. o McCarthy et al., 1995 o G. McCarthy, A.C. Nobre, S. Bentin, D.D. Spencer o Language-related field potentials in the anterior-medial temporal lobe: I. Intracranial

distribution and neural generators

o Journal of Neuroscience, 15 (2) (1995), pp. 1080–1089 o

18. o McNeill, 1992 o D. McNeill

o Hand and mind: What gestures reveal about thought

o University of Chicago Press, Chicago (1992) o

19. o McNeill, 2005 o D. McNeill o Gesture and thought

o University of Chicago Press, Chicago (2005) o

20. o Meyer et al., 2005 o P. Meyer, A. Mecklinger, T. Grunwald, J. Fell, C.E. Elger, A.D. Friederici

o Language processing within the human medial temporal lobe

o Hippocampus, 15 (2005), pp. 451–459 o

1. o Neuhaus et al., 2007 o A.H. Neuhaus, S. Koehler, C. Opgen-Rhein, C. Urbanek, E. Hahn, M. Dettling o Selective anterior cingulated cortex deficit during conflict solution in schizophrenia:

An event-related potential study

o Journal of Psychiatric Research, 41 (2007), pp. 635–644 o

2. o Neuhaus et al., 2010 o A.H. Neuhaus, C. Urbanek, C. Opgen-Rhein, E. Hahn, T.M. Ta, S. Koehler, et al. o Event-related potentials associated with the attention network test

o International Journal of Psychophysiology, 76 (2010), pp. 72–79 o

3. o Neville et al., 1997

o H.J. Neville, S.A. Coffey, D.S. Lawson, A. Fischer, K. Emmorey, U. Bellugi o Neural systems mediating American sign language: Effects of sensory experience

and age of acquisition

o Brain and Language, 57 (1997), pp. 285–308 o

4. o Nobre and McCarthy, 1995 o A.C. Nobre, G. McCarthy

o Language-related field potentials in the anterior-medial temporal lobe: II. Effects of word type and semantic priming


5. o Oehrn et al., 2014 o C.R. Oehrn, S. Hanslmayr, J. Fell, L. Deuker, N.A. Kremers, A.T. Do Lam, et al. o Neural communication patterns underlying conflict detection, resolution, and

adaptation


6. o Okada and Hickok, 2009 o K. Okada, G. Hickok o Two cortical mechanisms support the integration of visual and auditory speech: A

hypothesis and preliminary data

o Neuroscience Letters, 452 (3) (2009), pp. 219–223 o

7. o Oostenveld and Praamstra, 2001 o R. Oostenveld, P. Praamstra o The five percent electrode system for high-resolution EEG and ERP measurements

o Clinical Neurophysiology, 112 (2001), pp. 713–719 o

8. o Palmero-Soler et al., 2007 o E. Palmero-Soler, K. Dolan, V. Hadamschek, P.A. Tass

o swLORETA: A novel approach to robust source localization and synchronization tomography

o Physics in Medicine and Biology, 52 (7) (2007), pp. 1783–1800 o

9.

o Pascual-Marqui et al., 2002 o R.D. Pascual-Marqui, M. Esslen, K. Kochi, D. Lehmann o Functional imaging with low-resolution brain electromagnetic tomography (LORETA):

A review

o Methods and Findings in Experimental and Clinical Pharmacology, 24 (Suppl) (2002), pp. C91–C95

o 10. o Paulus et al., 2002 o M.P. Paulus, N. Hozack, L. Frank, G.G. Brown o Error rate and outcome predictability affect neural activation in prefrontal cortex and

anterior cingulate during decision-making

o Neuroimage, 15 (2002), pp. 836–846 o

11. o Petitto et al., 2000 o L.A. Petitto, R.J. Zatorre, K. Gauna, E.J. Nikelski, D. Dostie, A.C. Evans o Speech-like cerebral activity in profoundly deaf people processing signed languages:

Implications for the neural basis of human language

o Proceedings of the National academy of Sciences of the United States of America,

97 (2000), pp. 13961–13966 o

12. o Pitcher et al., 2011 o D. Pitcher, B. Duchaine, V. Walsh, G. Yovel, N. Kanwisher o The role of lateral occipital face and object areas in the face inversion effect

o Neuropsychologia, 49 (12) (2011), pp. 3448–3453 o

13. o Poizner et al., 1987

o H. Poizner, E.S. Klima, U. Bellugi o What the hands reveal about the brain

o MIT Press, Cambridge (MA) (1987) o

14.

o Proverbio and Riva, 2009 o A.M. Proverbio, F. Riva o RP and N400 ERP components reflect semantic violations in visual processing of

human actions


15. o Proverbio et al., 2014 o A.M. Proverbio, M. Calbi, M. Manfredi, A. Zani o Comprehending body language and mimics: An ERP and neuroimaging study on

Italian actors and viewers

o PLoS ONE, 9 (3) (2014), p. e91294 o

16. o Proverbio et al., 2012 o A.M. Proverbio, N. Crotti, M. Manfredi, R. Adorni, A. Zani o Who needs a referee? How incorrect basketball action are automatically detected by

basketball players’ brain

o Scientific Reports, 2 (2012), p. 883 o

17.

o Proverbio et al., 2010 o A.M. Proverbio, F. Riva, A. Zani o When neurons do not mirror the agent’s intentions: Sex difference in neural coding of

goal-directed actions

o Neuropsychologia, 48 (2010), pp. 1454–1463 o

18. o Rafal and Posner, 1987 o R.D. Rafal, M. Posner

o Deficits in human visual spatial attention following thalamic lesions (selective attention)

o PNAS, 84 (1987), pp. 7349–7353 o

19.

o Reid and Striano, 2008 o V.M. Reid, T. Striano o N400 involvement in the processing of action sequences


20. o Rizzolatti et al., 1996 o G. Rizzolatti, L. Fadiga, L. Fogassi, V. Gallese o Premotor cortex and the recognition of motor actions

o Cognitive Brain Research, 3 (2) (1996), pp. 131–141 o

1.

o Rizzolatti et al., 2001 o G. Rizzolatti, L. Fogassi, V. Gallese o Neurophysiological mechanisms underlying the understanding and imitation of action

o Neuroscience, 2 (2001), pp. 661–670 o

2. o Rushworth et al., 2004 o M.F. Rushworth, M.E. Walton, S.W. Kennerley, D.M. Bannerman o Action sets and decisions in the medial frontal cortex

o Trends in Cognitive Science, 8 (2004), pp. 410–417 o

3. o Schwarzlose et al., 2005 o R.F. Schwarzlose, C.I. Baker, N. Kanwisher o Separate face and body selectivity on the fusiform gyrus

o Journal of Neuroscience, 25 (2005), pp. 11055–11059 o

4.

o Service et al., 2007 o E. Service, P. Helenius, S. Maury, R. Salmelin o Localization of syntactic and semantic brain responses using

magnetoencephalography


o 5. o Shibata et al., 2009 o H. Shibata, J. Gyoba, Y. Suzuki o Event-related potentials during the evaluation of the appropriateness of cooperative

actions


6. o Simos et al., 1997 o P.G. Simos, L.F.H. Basile, A.C. Papanicolaou o Source localization of the N400 response in a sentence-reading paradigm using

evoked magnetic fields and magnetic resonance imaging


7. o Sitnikova et al., 2003 o T. Sitnikova, G. Kuperberg, P.J. Holcomb o Semantic integration in videos of real-world events: An electrophysiological

investigation

o Psychochysiology, 40 (2003), pp. 160–164 o

8. o Urgesi et al., 2004

o C. Urgesi, G. Berlucchi, S.M. Aglioti o Magnetic stimulation of extrastriate body area impairs visual processing of nonfacial

body parts

o Current Biology, 14 (2004), pp. 2130–2134 o

9. o Wahl et al., 2008 o M. Wahl, F. Marzinzik, A. Friederici, A. Hahne, A. Kupsch, et al. o The human thalamus processes syntactic and semantic language violations

o Neuron, 59 (5) (2008), pp. 695–707 o

10.

o Winterer et al., 2001 o G. Winterer, C. Mulert, S. Mientus, J. Gallinat, P. Schlattmann, H. Dorn, et al. o P300 and LORETA: Comparison of normal subjects and schizophrenic patients

o Brain Topography, 13 (4) (2001), pp. 299–313 o

11. o Wittfoth et al., 2009 o M. Wittfoth, D.M. Schardt, M. Fahle, M. Herrmann o How the brain resolves high conflict situations: Double conflict involvement of

dorsolateral prefrontal cortex

o Neuroimage, 44 (3) (2009), pp. 1201–1209 o

12. o Wright et al., 2003 o T.M. Wright, K.A. Pelphrey, T. Allison, M.J. McKeown, G. McCarthy o Polysensory interactions along lateral temporal regions evoked by audiovisual

speech

o Cerebral Cortex, 13 (10) (2003), pp. 1034–1043 o

13. o Wu and Coulson, 2005

o Y.C. Wu, S. Coulson o Meaningful gestures: Electrophysiological indices of iconic gesture comprehension

o Psychophysiology, 42 (6) (2005), pp. 654–667 o

14. o Wu and Coulson, 2007 o Y.C. Wu, S. Coulson o Iconic gestures prime related concepts: An ERP study

o Psychonomic Bulletin & Review, 14 (1) (2007), pp. 57–63 o

15. o Wurm et al., 2014 o M.F. Wurm, M. Hrkać, Y. Morikawa, R.I. Schubotz

o Predicting goals in action episodes attenuates BOLD response in inferior frontal and occipitotemporal cortex

o Behavioural Brain Research, 274 (2014), pp. 108–117 o

16. o Yuri et al., 2012 o B. Yuri, Y.B. Saalmann, M.A. Pinsk, L. Wang, X. Li, S. Kastner o The pulvinar regulates information transmission between cortical areas based on

attention demands

o Science, 337 (6095) (2012), pp. 753–756 o

17.

o Zysset et al., 2006 o S. Zysset, C.S. Wendt, K.G. Volz, J. Neumann, O. Huber, D.Y. von Cramon o The neural implementation of multi-attribute decision making: A parametric fMRI

study with human subjects

o Neuroimage, 31 (2006), pp. 1380–1388 o

Corresponding author at: NeuroMI-Milan Center for Neuroscience, Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, U6, 20126 Milan, Italy. Fax: +39 026448 3788.

Copyright © 2015 Elsevier Inc. All rights reserved.

• About ScienceDirect • Contact and support • Terms and conditions • Privacy policy

Copyright © 2015 Elsevier B.V. or its licensors or contributors. ScienceDirect® is a registered trademark of Elsevier B.V.

Cookies are used by this site. To decline or learn more, visit our Cookies page. Switch to Mobile Site

brain and language volume 147 , august 2015, pages 30–40 · 2015-05-25 · italian culture....

Documents