some first steps towards a theory of embodied...

Embodied communicative feedback in multimodal corpora

Full length article

The analysis of embodied communicative feedback in multimodal corpora – a

prerequisite for behavior simulation

Jens Allwood 1)

Stefan Kopp 2)

Karl Grammer 3)

Elisabeth Ahlsén 1)

Elisabeth Oberzaucher 3)

Markus Koppensteiner 3)

1) Department of Linguistics Göteborg University, Box 200 SE-40530 Göteborg, Sweden

2) Artificial Intelligence Group Bielefeld University, P.O. 100131, D-33501 Bielefeld,

Germany

3) Ludwig Boltzmann Institute for Urban Ethology - Department of Anthropology of the

University of Vienna, Althanstrasse 14, 1090 Vienna, Austria

ABSTRACT

Communicative feedback refers to unobtrusive (usually short) vocal or bodily expressions whereby a recipient of information can inform a contributor of information about whether he/she is able and willing to communicate, perceive the information, and understand the information. This paper provides a theory for embodied communicative feedback, describing the different dimensions and features involved. It also provides a corpus analysis part, describing a first data coding and analysis method geared to find the features postulated by the theory. The corpus analysis part describes different methods and statistical procedures and discusses their applicability and the possible insights gained with these methods.

1


Author keywords

Communicative embodied feedback, contact, perception, understanding, emotions,

multimodal, embodied communication

1. INTRODUCTION

The purpose of this paper is to present a theoretical model of communicative feedback,

which is to be used in a VR agent capable of multimodal communication. Another purpose is

to briefly present the coding categories which are being used to obtain data guiding the agent’s

behavior. Below, we first present the theory.

The function/purpose of communication is to share information. This usually takes place

by two or more communicators taking turns in contributing new information. In order to be

successful, this process requires a feedback system to make sure the contributed information is

really shared. Using the cybernetic notion of feedback of Wiener (1948) as a point of

departure, we may define a notion of communicative feedback in terms of four functions that

directly arise from basic requirements of human communication: Communicative feedback

refers to unobtrusive (usually short) vocal or bodily expressions whereby a recipient of

information can inform a contributor of information about whether he/she is able and willing

to (i) communicate (have contact), (ii) perceive the information (perception), and (iii)

understand the information (understanding). In addition, (iv) feedback information can be

given about emotions and attitudes triggered by the information, a special case here being an

evaluation of the main evocative function of the current and most recent contributions (cf.

Allwood, Nivre & Ahlsén 1992 and Allwood 2000, where the theory is described more in

detail). To meet these requirements, the sender continuously evokes (elicits) information from

the receiver, who, in turn, on the basis of internal appraisal processes connected with the

requirements, provides the information in a manner that is synchronized with the sender.

2


The central role of feedback in communication is underpinned already by the fact that

simple feedback words like yes, no and m are among the most frequent in spoken language. A

proper analysis of their semantic/pragmatic content, however, is fairly complex and involves

several different dimensions. One striking feature is that these words involve a high degree of

context dependence with regard to the features of the preceding communicative act, notably

the type of speech act (mood), its factual polarity, information status and evocative function

(cf. Allwood, Nivre & Ahlsén 1992). Moreover, when studying natural face-to-face interaction

it becomes apparent that the human feedback system comprises much more than words.

Interlocutors coordinate and exchange feedback information by nonverbal means like posture,

facial expression or prosody. In this paper, we extend the theoretical account developed earlier

to cover embodied communicative feedback and provide a framework for analyzing it in

multimodal corpora.

1.1. DIMENSIONS OF COMMUNICATIVE FEEDBACK

Communicative feedback can be characterized with respect to several different

dimensions. Some of the most relevant in this context are the following:

(i) Degrees of control (in production of and reaction to feedback)

(ii) Degrees of awareness (in production of and reaction to feedback)

(iii) Types of expression and modality used in feedback (e.g. audible speech, visible body

movements)

(iv) Types of function/content of the feedback expressions

(v) Types of reception preceding giving of feedback

(vi) Types of appraisal and evaluation occurring in listener to select feedback

(vii) Types of communicative intentionality associated with feedback by producer

(viii) Degrees of continuity in the feedback given

3


(ix) Semiotic information carrying relations of feedback expressions

These dimensions and others (cf. Allwood 2000) play a role in all normal human

communication. Below, we will describe their role for embodied communicative feedback.

Table 1 shows how different types of embodied feedback behavior can be differentiated

according to these dimensions. The table is discussed and explained in the eight following

sections (cf. also Allwood 2000, for a theoretical discussion).

Table 1. Types of linguistic and other communicative expressions of feedback.(C = Contact, P = Perception, U = Understanding, E = Emotion, A = Attitude)

Bodily coordination Facial expression,

posture, prosody

Head gestures Vocal verbal

Awareness and

control

Innate,

automatic

Innate, potentially

aware + controlled

Potentially/mostly

aware + controlled

Potentially/mostly

aware + controlled

Expression Visible Visible, audible Visible Audible

Type of function C, P, E C, P, E C, P, U, E, A C, P, U, E, A

Type of reception Reactive Reactive Responsive Responsive

Type of appraisal Appraisal, evaluation Appraisal, evaluation Appraisal, evaluation Appraisal, evaluation

Intentionality Indicate Indicate, display Signal Signal

Continuity Analogue Analogue, digital Digital Digital

Semiotic sign type Index Index, icon Symbol Symbol

1.1.1. Degrees of awareness and control and embodiment

Human communication involves multiple levels of organization involving physical,

biological, psychological and socio-cultural properties. As a basis, we assume that there are at

least two (human) biological organisms in a physical environment causally influencing each

other, through manipulation of their shared physical environment. Such causal influence might

to some extent be innately given, so that there are probably aspects of communication that

function independently of awareness and intentional control of the sender. Other types of

causal influence are learned and then automatized so that they are normally functioning

automatically, but potentially amenable to awareness and control. Still other forms of

4


influence are correlated with awareness and/or intentional control, on a scale ranging from a

very low to a very high degree of awareness/control. In this way, communication may involve

1) innately given causal influence

2) potentially aware and intentionally controllable causal influence

3) actually aware and intentionally controlled causal influence.

Human communication is thus “embodied” in two senses; (i) it always relies on and

exploits physical causation (physical embodiment), (ii) its physical actualization occurs

through processes in a biological body (biological embodiment). The feedback system as an

aspect human communication shares these general characteristics. The theory has a

perspective on communication and feedback, which implies processes occurring on different

levels of organization or put differently as can be seen in table 1 as implying processes that

occur with different levels of awareness and control (intentionality). In addition to this, the

theory also involves positing several qualitatively different parallel concurrent processes.

1.1.2. Types of expression and modality used in feedback

Like other kinds of human communication, the feedback system involves two primary

types of expression, (i) visible body movements and (ii) audible vocal sounds. Both of these

means of expression can occur on the different levels of awareness and control discussed

above. That is, there is feedback which is mostly aware and intentionally controllable, like the

words yes, no, m or the head gestures for affirmation and negation/ rejection. There is also

feedback that is only potentially controllable, like smiles or emotional prosody. Finally there is

feedback behavior which one is neither aware of nor able to control, but that is effective in

establishing coordination between interlocutors. For example, speakers tend to coordinate the

amount and energy of their body movements without being aware of it.

5


1.1.3. Types of function/content of the expressions

Communicative feedback concerns expressive behaviors that serve to give or elicit

information, enabling the communicators to share information more successfully. Every

expression, considered as a behavioral feedback unit, has thus two functional sides. On the one

hand it can evoke reactions from the interlocutor, on the other hand it can respond to the

evocative aspects of a previous contribution. Giving feedback is mainly responsive, while

eliciting feedback is mainly evocative. Each feedback behavior may thereby serve different

responsive functions. For example, vocal verbal signals (like m or yes) inform the interlocutor

that contact is established (C) that what has been contributed so far has been perceived (P) and

(usually also) understood (U). Additionally, the word yes often also expresses acceptance of or

agreement with the main evocative function of the preceding contribution (A). Thus, four basic

responsive feedback functions (C, P, U and A) can be attached to the word yes. In addition to

these functions, further emotional, attitudinal information (E) may be expressed concurrent to

the word yes. For example, the word may be articulated with enthusiastic prosody and a

friendly smile, which would give the interlocutor further information about the recipient’s

emotional state. Similarly, the willingness to continue (facilitating communication) might be

expressed by posture mirroring.

1.1.4. Types of reception

As explained above, feedback behavior is a more or less aware and controlled expression

of reactions and responses based on appraisal and evaluation of information contributed by

another communicator. We think of these reactions and responses as produced in two main

stages: First, an unconscious appraisal is tied to the occurrence of perception (attention),

emotions and other primary bodily reactions. If perception and emotion is connected to further

processing involving meaningful connections to memory, then understanding, empathy and

6


other emotive attitudes might occur. Secondly, this stage can lead to more aware appraisal (i.e.

evaluation) concerning the evocative functions (C, P, U) of the preceding contribution and

especially its main evocative function (A), which can trigger belief, disbelief, surprise,

skepticism or hope and be accepted, rejected or met with some form of intermediary reaction,

(often expressed by modal words like perhaps, maybe etc). We distinguish between these two

types of reception and use the term “reactive” when the behavior is more automatic and linked

to earlier stages in receptive processing, and the term “response” when the behavior is more

aware and linked to later stages. For example, vocal feedback words like yes, no and m as well

as head gestures are typically responses associated with evaluation, while posture adjustment

and facial gestures are more reactive and linked more directly to appraisal and perception.

1.1.5. Types of appraisal and evaluation

Responses and reactions with a certain feedback function occur as a result of continuous

appraisal and evaluation on the part of the communicators. We suggest that the notion of

“appraisal” be used for processes that are connected to low levels of awareness and control,

while “evaluation” is used when higher levels are involved. The functions C, P, U all pose

requirements that can be appraised or evaluated as to whether they are met or not (positive or

negative). Positive feedback in this sense can be explicitly given by the words yes and m or

head nod (or implicitly by making a next contribution), and negatively by words like no or

head shakes. The attitudinal and emotional function (E) of feedback is more complex and rests

upon both appraisal, i.e. processes with a lower degree of awareness and control, as well as

evaluation processes. What dimensions are relevant here is not clear. One possibility is the

dimensions suggested by Scherer (1999), where it is suggested that the appraisal dimensions

most relevant are (i) novelty (news value of stimulus), (ii) coping (ability to cope with a

stimulus), (iii) power (how powerful does the recipient feel in relation to the stimulus), (iv)

7


normative system (how much does the stimulus comply with norms the recipient conforms to),

(v) value (to what extent does the stimulus conform to values of the recipient). The effect of

appraisal that runs sequentially along these dimensions is a set of emotional reactions, which

may include a certain prosody or other behavioral reactions, but primarily is expressed through

prosody and facial display. Additionally, there will be a cognitive evaluation of whether or not

the recipient is able and/or willing to comply with the main evocative function of the

preceding contribution (A), i.e. can the statements made be believed, the questions answered

or the requests complied with.

1.1.6. Types of communicative intentionality

Like any other information communicated by verbal or bodily means, feedback

information concerning the basic functions (C, P, U, A, E) can be given on many levels of

awareness and intentionality. Although such levels almost certainly are a matter of degree, we,

in order to simplify matters somewhat, here distinguish three levels from the point of view of

the sender (cf. Allwood 1976): (i) “Indicated information” is information that the sender is not

aware of, or intending to convey. This information is mostly communicated by virtue of the

recipient's seeing it as an indexical (i.e., causal) sign. (ii) “Displayed information” is intended

by the sender to be “showed” to the recipient. The recipient does not, however, have to

recognize this intention. (iii) “Signaled information” is intended by the sender to “show” the

recipient that he is displaying and, thus, intends the recipient to recognize it as displayed.

Display and signaling of information can be achieved through any of the three main semiotic

types of signs (indices, icons and symbols, cf. Peirce 1955/1931). In particular, we will regard

ordinary linguistic expressions (verbal symbols) as being “signals” by convention. Thus, a

linguistic expression like It's raining, when used conventionally, is intended to evoke the

8


receiver's recognition not merely of the fact that “it's raining” but also of the fact that he/she is

“being shown that it's raining”.

1.1.7. Degree of continuity (i.e. analog vs. digital)

Feedback information can be expressed in analog ways, such as prosodic patterns in

speech, continuous body movements and facial expressions, which are used over stretches of

interaction. It may also be more digital and discrete, such as when feedback words, word

repetitions or head nods and shakes are used. Normally, analog and digital expressions are

used in combination.

1.1.8. Type of semiotic information carrying relation

Following Peirce’s semiotic taxonomy, where indices are based on contiguity, icons on

similarity and symbols on conventional, arbitrary relations between the sign and the signified,

we can find different types of semiotic information expressed by feedback, i.e. there is

indexical feedback (e.g. many body movements), iconic feedback (e.g. repetition or behavioral

echo (see below)) and symbolic feedback (e.g. feedback words).

1.2. FALSIFICATION AND EMPIRICAL CONTENT

A relevant question to ask in relation to all theories is the question of how the theory could

be falsified. Since the aspect of the theory that has been presented in this paper mainly consists

of a taxonomy of the theoretical dimensions of the theory, falsification in this case would

consist in showing that the taxonomy is ill-founded, i.e. that it is not homogeneous, that the

categories are not mutually exclusive, not perspicuous, not economical or not fruitful. Since

the question of whether the above criteria are met or not can be meaningfully asked, we

conclude that the theory has empirical content, ie can be falsified.

9


1.3. EMPIRICAL BASIS

To test our theoretical framework for its adequacy and usability in analyzing multimodal

corpora, we have started to gather and analyze data on 30 video-recorded dyadic interactions

with two subjects in standing position. The dyads were systematically varied with respect to

sex and mutual acquaintance. The subjects were university students and their task was to find

out as much as possible about each other within 3 minutes. For the multimodal corpora 30 (10

male-male, 10 male-female, 10 female-female) dyads of strangers (mean age 22.4) were

filmed with a video camera for 3 minutes. Subjects were told that they should try to find out as

much as possible about the other person, as the purpose of the experiment was to find out how

well communication works. Participation was voluntary and subjects did not receive

compensation.

10


Extractions of one minute from the video-recordings were transcribed and coded,

according to an abbreviated version of the MUMIN coding scheme for feedback (Allwood et

al. 2005) and the coding program Anvil. The coding schema identifies the feedback units

(either verbal or non-verbal), which are coded for function type (giving, eliciting) and attitudes

(continued contact, perception, understanding; acceptance of main evocative function;

emotional attitudes). It further captures the following non-vocal behaviors: posture shifts,

facial expressions, gaze, and head movements. As a special case of feedback giving, we have

studied “behavioral echo”. Finally, subjects were asked to fill in a questionnaire about their

socio-cognitive perception of the other (e.g. rapport). Fig. 1 shows a snapshort of the

annotation board during a data coding session.

11

Figure 1. Snapshot of the annotation board for analyzing embodied communicative feedback.


2. METHODS

After the experiment the participants filled out a questionnaire which covered basic

demographic data and the following main topics:

- social success: an assessment of the likelihood of the interaction partner accepting an

invitation or giving his/her telephone number

- pleasure: if the subject found the interaction pleasant

- compliance: if the subject him-/herself would give his/her telephone number to or accept

an invitation from the other subject

- mutual agreement: if the subject found that they both agreed on the discussed topics

- dominance: if the subject believed that he or she dominated the interaction

- -target desirability: if the subject found the other subject attractive or desirable as social

partner.

All questionnaire items were rated on 1-7 point Likert scales.

Only the second minute of the interaction was analyzed with the help of Anvil. In an ad lib

observation a behavior repertoire of 43 feedback related behavior categories was established,

including categories, sounds, facial expression, gaze, head movements, postures and arms

(gestures). After transcribing the verbalizations, the start and end points of the behaviors were

coded. Percentage agreement of reliability was 78%.

Statistical analysis was carried out in SPSS and lag sequential analysis was done with

GSEQ (Bakeman & Quera, 1995).

12


3. RESULTS

3.1. Evaluation of the situation

In order to find out how the subjects evaluated the situation we correlated the items from

the questionnaire. The results (Figure 2) show that the evaluation of mutual agreement is in the

center of the situation perception (n for all correlations is 60, Spearmans rho).

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Figure 2. Evaluation in the questionnaire.

When we calculate the correlations between the subjects, we find that they share the

perception of dominance (n=30 rho=-.39, p=0.04), mutual agreement (n=30, rho=0.38,

p=0.05), pleasure (n=30, rho=0.39, p=0.04) and compliance (n=30, rho =0.43, p=0.03). A

comparison of the correlations suggests that mutual agreement is central for the interactions –

and that most of the evaluation is shared between subjects.

13


3.2. Verbalizations

During one minute of interaction subjects produced on average 12.5 utterances (sd=3.7)

and talked for 27.3 seconds (sd=9.8) using 91.7 words (sd=30.92). Verbal feedback occurred

on average 4.3 times (sd=2.8).

We observed N=258 one word and two word utterances, among which “ja” (17.4 %),

“mhm” (15.9%), “jo” (4.3%), “hihi” (4.3) were most frequent, but these four amount to only

41 % of all feedback utterances, indicating the high variety of utterances that can be used.

The only relation between the evaluation of the situation and the verbal utterances that

could be found was for the variable give/elicit feedback with mutual agreement (n=60,

rho=.297, p=0.02). Thus feedback only partially demonstrates mutual agreement, as we would

also expect when looking at the different functions of feedback.

3.3. Behavior

Feedback behavior exhibits both intrapersonal and interpersonal patterns and

dependencies. Since these patterns are complex and depend on many factors, we are only able

to report on a few of them at the present time (cf. also Grammer et al. 1999).

3.3.1. Behavioral echo

For the analysis of behavioral echo (an iconic feedback expression, probably associated

with low degree of intentionality and awareness) we first applied a more or less crude

approximation. We calculated the time lag between the behaviors and then identified how

often two similar behaviors followed each other. This means if a smile follows exactly after a

smile this would be one incident of a behavioral echo (Grammer et al. 2000). We excluded

turn taking were an utterance is followed directly by another utterance, without time lag.

14


Behavioral echo occurs at a mean rate of 1.25 (std=1.17). As compared to an average of

20.51 (std=6.98) for change of main contributor (speaker change) in the dyads, echo is

generally rare, occurring only in 6.1% of all speaker changes. In addition we find no

correlations between the situation evaluation and echo. Thus simple echoing seems to be

peripheral for the implementation as a feedback device – and feedback itself seems to be more

complicated.

3.3.2. Time patterns

The behavior events were further processed by the use of Theme (Magnusson, 1996;

2000), which is a software specifically designed for the detection and analysis of repeated non-

random behavioral patterns hidden within complex real-time behavior records. Each time-

pattern is essentially a repeated chain of a particular set of behavioral event-types (A B C D

…) characterized by fixed event order (and/or co-occurrence), and time distances ( 0)

between the consecutive parts of the chain that are significantly similar in all occurrences of

the chain (Magnusson, 1996; 2000). In this context, an important aspect of this pattern type is

that its definition does not rely on cyclical organization, i.e. the full pattern and/or its

components may or may not occur in a cyclical fashion. In order to perform Theme-analyses

on the behavior performed by each individual, the number of interactants was two in this

study. A minimum of two repeated pattern occurrences throughout the one-minute sampling

period and a 0.05 significance level were specified.

We looked for one pattern type: Patterns were both interactants contributed (interactive

patterns) and verbal feedback was part of the pattern.

The organization of the detected patterns can be described by pattern frequency, the

number of behavior codes (pattern length) and their complexity (hierarchical organization).

The number of patterns and their organization thus will give us an estimate of the amount of

15


general rhythmic coupling (a repeated pattern of interactive behavior) in the dyad and

rhythmic coupling generated by feedback itself.

The following figure (Figure 3) gives an example for a complex feedback pattern.

Figure 3. An example of a complex feedback pattern

In (a) the hierarchical organization of the pattern is shown, which consists of 14 member

events in time. Y and X denote the interactants, b and c the start or end of a behavior. The

pattern starts with Y doing automanipulation, followed by a brow raise and an utterance. Then

X responds with a repeated head nod, one word verbal feedback, and Y and later X produces

an utterance. X then looks at Y, automanipulates and then speaks again. Finally Y looks at X.

This complex pattern is created twice (c) in the same time configuration (b). THEME thus can

reveal hidden rhythmic structures in interactions.

On average the subjects produced 2.26 (sd=0.26) patterns per one-minute interactions -

this means that there are at least 4 rhythmical patterns per interaction which are interactive and

16


where verbal feedback is involved. We can compare this to the average of verbal feedback of

8.0 (sd=3.9) per interaction. This means that every third instance of verbal feedback enters

rhythmical patterns. Thus feedback seems to be a part of a rhythmical time structure in

interactions.

The average length of the patterns was 5.1 items (sd=1.7) and the average level of nodes

reached was 2.9 (sd=0.76). As reported previously by Grammer et al (2000) only the

composite score of perceived pleasure correlates positively with pattern length (rho=0.35,

p=0.05) but negatively with the average number of patterns (rho=-0.45, p= p=0.01). This

suggests that a few long patterns contribute to the perception of the interaction as pleasant. For

the rest of the socio-cognitions no relation to patterns and pattern structure was found.

Patterns are based on the following behaviors: “Arms Cross” (0.19), “Head Down” (0.85),

“Look At” (0.41), “Head Jerk (0.35), “Move Forward (0.18), “Head Right” (0.17), ”Head Tilt”

(0.41), “Smile” (0.40) and “Head Up” (0.20). The numbers in brackets tell how often a

behavior formed a pattern with feedback relative to its frequency of occurrence.

3.3.3. Sequential organization

As a last approach we calculated the transition probabilities for feedback and utterances

which will answer the question what actually leads to verbal feedback and what follows

feedback and what follows an utterance or precedes it. This was done in GSEQ (Bakeman and

Quera, 1995) calculating conditional transition probability at lag one. Note that this analysis

differs from the THEME analysis considerably. The lag analysis describes what immediately

follows or precedes a behavior as a state – THEME in contrast reveals hidden rhythmic

patterns without a direct relation in time space between two behaviors., like A follows B. In

THEME any behavior can occur in between pattern members.

17


The following table shows the conditional probabilities that feed back follows a behaviour

and the conditional probabilities that a certain behaviour follows feedback. Again like in the

THEME analysis it becomes clear that verbal feedback is embedded in a row of behaviours

and not only utterance related.

Table 2. Conditional probabilities of Feedback and Related Behaviors

Behavior preceding Feedback

Conditional probability

Clear Throat 0,5Scowl 0,5Move Forward 0,2857Head Right 0,2727Single Nod 0,2222Head Up 0,2Point At 0,2Repeated Jerk 0,2Look at 0,1795Lean Away 0,1667Repeated Nod 0,16Lean Toward 0,1579Head Jerk 0,1429Shrug 0,1429Utterance 0,1313Head Left 0,1111

Behavior following feedback

Utterance 0,2902Head Tilt 0,0909Repeated No 0,0545Body Sway 0,0545Smile 0,0545Head Left 0,0364Automanipulation 0,0364Lean Towards 0,0364Look At 0,0364Head Down 0,0364Shrug 0,0364Lip Bite 0,0364

4. CONCLUSIONS

We have presented a theory for communicative feedback, describing the different

dimensions involved. This theory is supposed to provide the basis of a framework for

analyzing embodied feedback behavior in natural interactions. We have started to design a

18


coding scheme and a data analysis method suited to capture those features that are decisive in

this account (such as type of expression, relevant function, or time scale). Currently, we are

investigating how the resultant multimodal corpus can be analyzed for patterns and rules as

required for a predictive model of embodied feedback. Ultimately, such a model should afford

its simulation and testing in a state-of-the-art embodied conversational character.

The results from our empirical study suggest that feedback is a multimodal, complex, and

highly dynamic process—supporting the differentiating assumptions we made in our

theoretical account. In ongoing work, we are building a computational model of feedback

behavior that is informed by our theoretical and empirical work, and that can be simulated and

tested in the embodied agent „Max“. Max is a virtual human under development at the A.I.

Group at Bielefeld University. The system used here is applied as a conversational information

kiosk in the Heinz-Nixdorf-MuseumsForum (HNF), a public computer museum where Max

engages visitors in face-to-face small-talk conversations and provides them with information

about the museum, the exhibition, and other topics. Users can enter natural language input to

the system using a keyboard, whereas Max responds with synthetic German speech along with

nonverbal behaviors like manual gestures, facial expressions, gaze, or locomotion.

In extending this interactive settings to more natural feedback behavior by Max, we will

work along two major lines. First, we will implement a „shallow“ feedback model using

probabilistic transition networks that are directly derived form the transition probabilities

found in the data. This is largely akin to previous approaches which tried to identify

regularities at a mere behavioral level and cast them into rules to predict when a feedback

behavior would seem approriate. This approach proved viable so far only for very specific

feedback mechanisms, e.g. nodding after a pause of a certain duration or certain prosodic cues

(Ward and Tsukahara, 2000). In our empirical study we took a broader look and we found a

large number of rather low conditional probabilities on behavior-behavior combinations. We

19


will thus also explore in how far our theoretical model can inform a „deeper“ simulation

model. This model will account for processes of appraisal and evaluation of information that

give rise to the different responsive functions that feedback expressions need to fulfill and that

can be mapped onto different types of expressions and modalities to realize these functions.

ACKNOWLEDGEMENTS

We thank the Ludwig Bolzmann Institute for Urban Ethology in Vienna for help with data

collection and transcription, the Department of Linguistics and SSKKII Center for Cognitive

Science, Göteborg University, and the ZiF Center of Interdisciplinary Research in

Bielefeld/Germany.

REFERENCES

1. Allwood, J. (1976). Linguistic Communication as Action and Cooperation. Gothenburg

Monographs in Linguistics 2. Göteborg University, Department of Linguistics.

2. Allwood, J. (2000). Structure of Dialog. In Taylor, M., Bouwhuis, D. & Neel, F. (eds.)

The Structure of Multimodal Dialogue II, Amsterdam, Benjamins.pp. 3 - 24.

3. Allwood, J., Cerrato, L., Dybjær, L., Jokinen, K., Navaretta, C. & Paggio, P. (2005).

The MUMIN Multimodal Coding Scheme. NorFA Yearbook 2005.

4. Allwood, J, NivreJ, & Ahlsén, E. (1992). On the semantics and pragmatics of linguistic

feedback, Journal of Semantics, vol. 9, no. 1, 1-26.

5. Bakeman, R., & Quera, V. (1995). Analyzing Interaction: Sequential Analysis with SDIS

and GSEQ. New York: Cambridge University Press.

20


6. Grammer K., Honda R., Schmitt A., Jütte A. (1999): Fuzziness of Nonverbal Courtship

Communication. Unblurred by Motion Energy Detection, Journal of Personality and

Social Psychology. 77, 3, 487-508.

7. Grammer K., Kruck K., Juette A., Fink B. (2000): Non-verbal behavior as courtship

signals: The role of control and choice in selecting partners, Evolution and Human

Behavior 21, pp. 371-390.

8. Magnusson, M. S. (1996). Hidden real-time patterns in intra- and inter-individual

behavior:

description and detection. Eur. J. Psychol. Assessment 12, 112-123.

9. Magnusson, M. S. (2000). Discovering hidden time patterns in behavior: T-patterns and

their detection. Behav. Res. Meth. Instr. Comp. 32, 93-110.

10. Peirce, C. S. (1931). Collected Papers of Charles Sanders Peirce, 1931-1958, 8 vols.

Edited by Charles Hartshorne, Paul Weiss, and Arthur Burks. Cambridge, Mass., Harvard

University Press.

11. Scherer, K. T- (1999). Appraisal Theory. In T. Dalgleish & M. J. Power (Eds.)

Handbook of Emoiton and Cognition (pp.637-663). Chichester: New York.

12. Wiener, N. (1948). Cybernetics or Control and Communication in the Animal and the

Machine. MIT Press.

21

some first steps towards a theory of embodied...

Documents