1/13/10

1

On the perception of allegedly word-like units in signed language

Tommi Jantunen University of Jyväskylä, Department of Languages/3BatS

tommi.j.jantunen@jyu.fi

VIPP Symposium 2010: Seeing Red Aalto University/School of Art and Design, 14 January 2010 3BatS

•  Postdoctoral Researcher’s project 2010-2012

•  Funded by the Academy of Finland and Department of Languages, JyU

•  Aim is to investigate critically the ontological basis of modern sign language (SL) research by testing certain well-established presuppositions governing the research into sign, syllable and sentence in (Finnish) SL

•  http://users.jyu.fi/~tojantun/3BatS

The sign

•  ...is assumed to be a unit that corresponds to spoken language word; for example:

“Throughout this account [---] the term ‘sign’ will be used as a rough equivalent to ‘word’ in spoken language” (Johnston & Schembri 1999: 115/fn. 1)

“[---] the word and the sign are situated at an equivalent level of linguistic organisation [---] (Zeshan 2002: 154)

“[---] as far as I can see, signs would be partly equivalent to semantic units in spoken language” (Blanche-Benveniste 2007: 86)

The aim

•  …of this presentation is to investigate the correspondence between the sign and the word from the perspective of two foundational word-defining characteristics and perceptual experiments

1: In the word

meaning always associates

with the form

ja sitte

ja

[ja]…[sitte]

1/13/10

2

2: The (form of the) word

includes the maximally

perceivable part(s) of the signal

ja sitte

The prototypical… Word

•  Has a conventionalised meaning (semantics)

•  Contains two syllables/is bimoraic (phonology)

•  Has a fixed order of meaningful elements, can form a sentence by itself, can change place in a sentence etc. (grammar)

Sign

•  Has a conventionalised meaning (semantics)

•  Is a monosyllable, includes only one set of selected fingers, is produced at one major place of articulation (phonology)

•  Can form a sentence by itself etc. (grammar)

(e.g. Sandler 1999, Zeshan 2002) (e.g. McCarthy & Prince 1993, Karlsson 1994)

The method of identifying signs

•  Look for “meaningful” sequences (semantics)

•  Observe changes in the posture and/or movement of the hands and nonmanual articulators (phonology)

•  Use any grammatical knowledge you have access to (grammar)

E.g., in the Corpus NGT A sign starts:

•  at the first frame in which the hand starts to move away from the initial location of the sign to the final location of the sign; or

•  (in case the hand does not move through space): at the first frame in which the handshape starts to change, e.g. closing the hand in the sign for “MAN”; or

•  (in case the hand does not move through space and the handshape does not change): at the first frame in which the orientation of the hand starts to change, e.g. turning the hand in the sign for “OVERLEDEN” (“PASS_AWAY”);

A sign ends:

•  at the first frame in which the handshape starts to change after the sign was finished; or

•  at the first frame in which the hand starts to move away from the final location of the sign.

(Crasborn & Zwitserlood 2008: 6)

ELAN annotation

http://www.lat-mpi.eu/tools/elan/

1/13/10

3

Demo 1 Comprehension & perception test

What does she sign?

(adapted from Suvi)

What does he sign?

(adapted from Suvi)

Explanation & the result •  Superficially signing-like video clips, edited,

however, to contain only the parts between signs (i.e. transitions); note that

in the first clip, the durations of sign sequences are relatively short whereas the durations of transitions are relatively long

in the second clip, the durations of sign sequences are relatively long whereas the durations of transitions are relatively short

•  The artificial deletion of signs from the video does not crucially impair comprehension of the intended message

Demo 1: �interim discussion & conclusion

•  The demo suggests that, in the sign, meaning is not crucially associated with the (alleged) form; the meaning can be deciphered, up to a certain point, on the basis of transitions (cf. information concerning Aktionsart, other semantic nyances, and near-homonym disambiguation)

The finding seems to go against the conclusion made, for example, by Grosjean (1981) who in his perception study of isolated signs claimed that the "recognition point" is always inside the sign (cf., however, Grosjean's sign recognition technique)

The main conclusion made by Clark & Grosjean (1982) is implicitly supported; they found that in contextual signing, only 1/3 of a sign is needed to recognise it (cf. 80% of the word in speech)

The existence of a strong modality effect is implied: in SL (unlike in spoken language), the articulators are completely visible during the production of the signal; this allows the meaning to be continuously "guessed" and disambiguated

•  In the word, meaning is always associated with the form and cannot be deciphered on the basis of transitions (i.e. the sequences of silence)

•  Ergo: signs and words cannot be compared to each other in this respect

Demo 2 Motion analysis & perception

1/13/10

4

Background •  A group of people from the Aalto University's School of Science and

Technology/Dept. of Information and Computer Science (Markus Koskela & Jorma Laaksonen), the Finnish Association of the Deaf (Päivi Rainò), and Jyväskylä University/Dept. of Languages (Tommi Jantunen)

•  A semi-automatic analysis of various characteristics of articulators' movements on digital SL video

•  The basis of the method is quantitative computer vision analysis

•  The method consists of three distinct phases: detection of the target articulators on the video, tracking of the motion of the selected articulators, and the representation of the motion of the articulators with statistical descriptors

•  Issues that still need addressing: movements away from or towards the body, finer detection of articulators

(excerpts from Jantunen, Koskela, Laaksonen & Rainò, submitted)

Filter (1) and mtip (2)

(Jantunen, Koskela, Laaksonen & Rainò, submitted)

Motion descriptors (3)

(analysis of Suvi's article 1038, example 3)

The importance of acceleration

•  “[A] visual beat is communicated by periods of acceleration […]” (Luck & Sloboda 2008: 237)

Visual beats = “those events that are felt to be more forcefully produced and around which the other events in the sequence are organized” (Allen & al. 1991: 197)

•  "[A]cceleration is also important for the parsing of one event from another" (Shaw & Cutting 1980: 77)

•  Current operationalisation: the peak of acceleration = the most perceivable moment

Signs & acceleration peaks (1)

(analysis of Suvi's article 1038, example 3)

Signs & acceleration peaks (2)

(analysis of the first 512 frames of Pessi ja Illusia)

1/13/10

5

Demo 2: �interim discussion & conclusion •  Given that the notion of perceptivity is validly operationalised and

the forms of signs are correctly determined, the distribution of acceleration peaks suggests that the most perceivable moments in a signed signal tend to associate with transitions, not with signs

Similar findings have been presented, from the perspective of sign production, for example, by Wilbur (1990) and Wilcox (1992) (concerning stressed sign production and fingerspelling, respectively)

•  The (form of the) word always includes the maximally perceivable moment(s) of the signal

•  Ergo: signs and words are not equivalent units in this respect

General discussion •  The results indicate a conflict between that what the sign is

argued to be in and by the linguistic theory (i.e. a unit equivalent to word) and that what the sign is according to the perceptual experiments (i.e. a unit not equivalent to word)

•  Note that crucial differences between the sign and the word have also been indicated by the more traditonal grammatical research; however, this has not led to the questioning of sign=word assumption

•  Spoken language bias in sign linguistics (cf. Per Linell's WLB)

…Cont'd

•  Perhaps only the applied method of identifying and indicating signs from the video is misguided and in the need of revision?

•  Indeed, the method contains a certain degree of vagueness

•  However, the method does not exist in the vacuum; it is ontologically grounded

•  Therefore, also the current ontological conception of the sign is suggested to be misguided and in the need of revision

Still cont'd & conclusion

•  In general, it is implied that signs and their borders are ontologically more gradient and fuzzier than words and their borders (cf. modality effect); note that

the notions of gradience and fuzziness have already gained ground in other areas of SL research

•  More work on signs, transitions and their borders is needed. For example, it is generally assumed that transitions in SL are non-linguistic in the sense similar to word-to-word transitions in spoken language. However, the present data implies that SL transitions have a linguistic role.

References (1) •  Allen, G. D.; Wilbur, R. B. & Schick, B. B. (1991). Aspects of rhythm in ASL. Sign Language Studies 72,

297-320.

•  Blanche-Benveniste, C. (2007). Linguistic units in signed and verbal languages. In E. Pizzuto, P. Pietrandrea & R. Simone (Eds.), Verbal and signed languages. Comparing structures, constructs and methodologies, 81-106. Mouton de Gruyter: Berlin

•  Crasborn, O. & Zwitserlood, I. (2008). Annotation of the video data in the "Corpus NGT". Department of Linguistics & Centre for Language Studies, Radboud University Nijmegen. Online publication: http://hdl.handle.net/1839/00-0000-0000-000A-3F63-4 .

•  Grosjean, F. (1981). Sign & word recognition: a first comparison. Sign Language Studies 32, 98-125.

•  Clark, L. E. & Grosjean, F. (1982). Sign recognition processes in American Sign Language: the effect of context. Language and Speech 25, 325-240.

•  Jantunen, T.; Koskela, M.; Laaksonen, J. & Raino, P. (submitted). Towards automated visualization and analysis of signed language prosody: method and linguistic issues. Manuscript, 10 September 2009.

•  Johnston, T. & Schembri, A. (1999). On defining lexeme in a signed language. Sign Language & Linguistics 2, 115-185.

•  Karlsson, F. (1994). Yleinen kielitiede. Yliopistopaino: Helsinki.

References (2) •  Luck & Sloboda (2008). Exploring the spatio-temporal properties of simple conducting gestures using a

synchronization task. Music Perception 25, 225-239.

•  McCarthy, J. & Prince, A. 1993. Prosodic morphology II: constraint interaction and satisfaction. Manuscript. University of Massachusetts, Amherst; Rutgers University, New Brunswick, N.J.

•  Sandler, W. (1999). Cliticization and prosodic words in a sign language. In T. Hall & U. Kleinhenz (Eds.), Studies on the Phonological Word, 223-255. Johns Benjamins: Amsterdam.

•  Shaw, R. E. & Cutting, J. E. (1980). Clues from an ecological theory of event perception. In U. Bellugi & M. Studdert-Kennedy (Eds.), Signed and spoken languages: biological constraints on linguistic form, 57-84. Weinheim: Verlag Chemie.

•  Suvi = Suvi – The online dictionary of Finnish Sign Language. [Helsinki:] The Finnish Association of the Deaf, 2003. Online publication: http://suvi.viittomat.net .

•  Wilbur, R. B. (1990). An experimental investigation of stressed sign production. International Journal of Sign Language 1, 41-59.

•  Wilcox, S. (1992). The phonetics of fingerspelling. Amsterdam: John Benjamins.

•  Zeshan, U. (2002). Towards a notion of 'word' in sign languages. In R. M. W. Dixon & A. Y. Aikhenvald (Eds.), Word. A cross-linguistic typology, 153-179. Cambridge University Press: Cambridge.