alignment in multimodal dialogue corpora robin hill and ellen gurman bard edinburgh
TRANSCRIPT
![Page 1: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/1.jpg)
Alignment in multimodal dialogue corpora
Robin Hill and Ellen Gurman Bard
Edinburgh
![Page 2: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/2.jpg)
Joint Construction Task
• Collaborative and cooperative joint action in a dyad.
• Two eye-trackers linked in parallel.
• Manipulate communication/feedback channels available from trial to trial:– Speech permitted/denied– Partner’s mouse cursor visible/invisible– Partner’s gaze position visible/invisible
![Page 3: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/3.jpg)
JEL Lab
![Page 4: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/4.jpg)
JCT in action
![Page 5: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/5.jpg)
Cross-recurrence
• Distribution of visual alignment over time
• Contrast with random gaze alignment.
• Richardson, D. C., & Dale, R. (2005). Looking to understand: The coupling between speakers' and listeners' eye movements and its relationship to discourse comprehension. Cognitive Science, 29(6), 1045-1060.
• Richardson, D. C., Dale, R., & Kirkham, N. Z. (2007). The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue. Psychological Science, 18(5), 407-413.
![Page 6: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/6.jpg)
Cross-recurrent Gaze
Richardson & Dale, 2005
![Page 7: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/7.jpg)
In the JCT corpus
• Eye-movement records for the central 120 seconds of each trial used.
• Time-linked overlap = percentage of time with JA gaze-tracks in same Region of Interest for a series of 200ms time lags across a moving 12-second temporal window.
• Baseline = randomly re-ordered series of the same eye-movement record .
• If the two are identical, temporal overlap purely by chance.
![Page 8: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/8.jpg)
Global mean recurrence
24
25
26
27
28
29
30
31
Lag (ms)
Rec
urre
nce
(%)
Dyad recurrenceRandom baseline
•Gaze at dynamic objects well coordinated• Greatest overlap simultaneous•Strongest coincidence approximately in the range ±2000ms
![Page 9: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/9.jpg)
Global mean recurrence
24
25
26
27
28
29
30
31
Lag (ms)
Rec
urre
nce
(%)
Dyad recurrenceRandom baseline
•Symmetrical distribution indicates similar behaviour exhibited by both partners.•Similar to the R&D findings
![Page 10: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/10.jpg)
Joint attention and communication modality
• Coordination is non-random in all conditions including baseline: Joint task always results in some level of gaze coordination.
![Page 11: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/11.jpg)
SpeechMouseGaze
MouseGaze
SpeechGaze
Gaze
SpeechMouse
Mouse
Speech None
21
23
25
27
29
31
33
-6000
-5600
-5200
-4800
-4400
-4000
-3600
-3200
-2800
-2400
-2000
-1600
-1200
-800
-400 0
400
800
1200
1600
2000
2400
2800
3200
3600
4000
4400
4800
5200
5600
6000
Lag (ms)
% R
ecu
rren
ce
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)%
Re
cu
rre
nc
e
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)
% R
ecu
rre
nc
e
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)
% R
ecu
rre
nc
e
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)
% R
ecu
rre
nc
e
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)
% R
ecu
rre
nc
e
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)%
Re
cu
rre
nc
e
Dyad recurrence
Random baseline
21
23
25
27
29
31
33
Lag (ms)
% R
ec
urr
en
ce
Dyad recurrence
Random baseline
![Page 12: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/12.jpg)
Speech summary
• Speech appears to reduce the occurrence of visual alignment.
• Speech tends to increase task completion times.
BUT• People make fewer mistakes during the task when
they can speak.• Speech-improved strategies persist even if dyads are
later prevented from talking to each other.
![Page 13: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/13.jpg)
Next…
• Examine cross-recurrence pinned to specific linguistic elements/events.– E.g. referring expressions.
![Page 14: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/14.jpg)
Eye-voice-eye
• Synchronising of gaze can be broken into two components:Eye-voice span (speaker)+Voice-eye span (speaker-listener)
• Examine the relationship to determine if the communicative burden is slanted towards the speaker of the listener.– E.g. Are there longer production times and do
these lead to shorter comprehension times?
![Page 15: Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062404/5515d83a550346cf6f8b4974/html5/thumbnails/15.jpg)
<blink>
Thank you.