![Page 1: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/1.jpg)
Quality in the Eye of the Beholder?Quality measures and user satisfaction
in live subtitling.
Monika SZCZYGIELSKA, Łukasz DUTKA
![Page 2: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/2.jpg)
Imagine subtitles with disappearing letters
2
![Page 3: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/3.jpg)
3
Nawiedź dusz wiernych Tobie krąg, Niebieską łaskę zesłać racz, Sercom co dziełem są Twych rąk.
Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
![Page 4: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/4.jpg)
4
Nawiedź dusz wiernych Tobie krąg, Niebieską łaskę zesłać racz, Sercom co dziełem są Twych rąk.
Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
![Page 5: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/5.jpg)
5
Subtitles with no Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
Nawiedć dusz wiernych Tobie krąg, Niebieskąśaskę zesśał racz, Sercom co dzieśem są Twych rąk.
ź -> ć ł -> ś ć -> ł
S
Someone messed up with diacritics
![Page 6: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/6.jpg)
6
Subtitles with Polish diacritics. Dancing with the Stars, Polsat, Poland 2017.
On powiedział że opanowałeśpodstawy. Jakie podstawy?
![Page 7: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/7.jpg)
7
Subtitles with no Polish diacritics. Dancing with the Stars, Polsat, Poland 2017.
On powiedzial, ze opanowalespodstawy. Jakie podstawy? S
Diacriticsdisappearedaltogether
![Page 8: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/8.jpg)
8
Disaster!What happened?
![Page 9: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/9.jpg)
POSZEDŁ DO SĄDU / SADUHe went to .... the court / the orchard
9
![Page 10: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/10.jpg)
ZROBIĆ ŁASKĘ / LASKESomeone turned diacritics
Make .... grace / a stick
10
![Page 11: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/11.jpg)
11
Niebieską łaskę zesłać racz.(“Give us your heavenly grace”)
Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.
![Page 12: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/12.jpg)
12
Disaster, really ?What do users think about it?
![Page 13: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/13.jpg)
The errors went unnoticed
13
![Page 14: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/14.jpg)
How is it possible?
14
![Page 15: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/15.jpg)
Research---
objective & subjective measures
● NER value● USERS'
OPINIONS
15
![Page 16: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/16.jpg)
SPEECHES FROM THE POLISH PARLIAMENT [2016]
16
USERS' OPINION
N = 55 [Hearing 24, HoH 15, Deaf 16]
online questionnairerecorded speeches with live transcriptioninstructions in sign language or written instructions
![Page 17: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/17.jpg)
Live subtitling priorities according to users [Szczygielska & Dutka 2016] 17
1 LEGIBILITY
3 LATENCY2VERBATIM
4DISPLAY MODE
5SEGMEN- TATION
4
6GRAMMAR
& SPELLING
5
3
![Page 18: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/18.jpg)
evaluated the subtitles with the highest NER value as the worst
12%
MYTHNER value >99%
NER value & users’ opinions [Szczygielska, Dutka 2016] 18
said that subtitles were highly edited and simplified
15%
MYTHreduction=0.5%
NER value is in line with users' opinions
overall
NER > 99%users rating: 3.6 (1-5)
![Page 19: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/19.jpg)
Stereotypes among users?
● too much delay● too much text reduction● too slow / too fast
19
![Page 20: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/20.jpg)
DANCING WITH THE STAIRS TV POLSAT [2017]
20
USERS' OPINION
N = 203 [Hearing 8, HoH 68, Deaf 127]
Questionnaire & instruction in sign language
![Page 21: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/21.jpg)
NER value Dancing with the Stars [Szczygielska & Dutka 2017] 21
NER value in final episode
99.5%
> 98 OFCOM recommendation
average delay in final episode
<5-6 sec.
real average delay in the UK after 10 years
lowest average delay in an episode(episode 3)
3.4 sec.
3 sec. OFCOM recommendation
![Page 22: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/22.jpg)
of the users were satisfied or very satisfied with the quality of live subtitles for “Dancing...”
22
67.5%
![Page 23: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/23.jpg)
BUT….Users’ opinions. Dancing with the Stars [Szczygielska & Dutka 2017]
23
![Page 24: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/24.jpg)
Users’ opinions. Dancing with the Stars [Szczygielska & Dutka 2017] 24
the text reduction is very high with many utterances missing
22 %
MYTHreduction > 10%
the delay was high or very high
18.70 %
MYTHdelay > 6 sec.
subtitles are too fast, I can’t keep up with the pace
19.20 %
18,72%subtitles are too slow
![Page 25: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/25.jpg)
Conclusions 1/2
25
Users might not be rational in their opinions.
Users can be very critical.
Users’ needs can be so diverse as to be mutually exclusive.
![Page 26: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/26.jpg)
Our approach is user-focused and user-informed but not user-lead.
We should listen to users but at some points we have to make an informed decision.
26
![Page 27: Quality in the Eye of the Beholder? Quality measures and ...pagines.uab.cat/act/sites/pagines.uab.cat.act/... · Quality measures and user satisfaction in live subtitling. ... I can’t](https://reader034.vdocuments.site/reader034/viewer/2022042402/5f10e2a57e708231d44b47f6/html5/thumbnails/27.jpg)
Quality in the Eye of the Beholder. Quality measures and user satisfaction
in live subtitling.
Monika SZCZYGIELSKA, Łukasz DUTKA