Quality in the Eye of the Beholder?Quality measures and user satisfaction

in live subtitling.

Monika SZCZYGIELSKA, Łukasz DUTKA

Imagine subtitles with disappearing letters

2

3

Nawiedź dusz wiernych Tobie krąg, Niebieską łaskę zesłać racz, Sercom co dziełem są Twych rąk.

Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.

4

Nawiedź dusz wiernych Tobie krąg, Niebieską łaskę zesłać racz, Sercom co dziełem są Twych rąk.

Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.

5

Subtitles with no Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.

Nawiedć dusz wiernych Tobie krąg, Niebieskąśaskę zesśał racz, Sercom co dzieśem są Twych rąk.

ź -> ć ł -> ś ć -> ł

S

Someone messed up with diacritics

6

Subtitles with Polish diacritics. Dancing with the Stars, Polsat, Poland 2017.

On powiedział że opanowałeśpodstawy. Jakie podstawy?

7

Subtitles with no Polish diacritics. Dancing with the Stars, Polsat, Poland 2017.

On powiedzial, ze opanowalespodstawy. Jakie podstawy? S

Diacriticsdisappearedaltogether

8

Disaster!What happened?

POSZEDŁ DO SĄDU / SADUHe went to .... the court / the orchard

9

ZROBIĆ ŁASKĘ / LASKESomeone turned diacritics

Make .... grace / a stick

10

11

Niebieską łaskę zesłać racz.(“Give us your heavenly grace”)

Subtitles with Polish diacritics. Canonization of John Paul II, Vatican Television Center CTV, Italy 2014.

12

Disaster, really ?What do users think about it?

The errors went unnoticed

13

How is it possible?

14

Research---

objective & subjective measures

● NER value● USERS'

OPINIONS

15

SPEECHES FROM THE POLISH PARLIAMENT [2016]

16

USERS' OPINION

N = 55 [Hearing 24, HoH 15, Deaf 16]

online questionnairerecorded speeches with live transcriptioninstructions in sign language or written instructions

Live subtitling priorities according to users [Szczygielska & Dutka 2016] 17

1 LEGIBILITY

3 LATENCY2VERBATIM

4DISPLAY MODE

5SEGMEN- TATION

4

6GRAMMAR

& SPELLING

5

3

evaluated the subtitles with the highest NER value as the worst

12%

MYTHNER value >99%

NER value & users’ opinions [Szczygielska, Dutka 2016] 18

said that subtitles were highly edited and simplified

15%

MYTHreduction=0.5%

NER value is in line with users' opinions

overall

NER > 99%users rating: 3.6 (1-5)

Stereotypes among users?

● too much delay● too much text reduction● too slow / too fast

19

DANCING WITH THE STAIRS TV POLSAT [2017]

20

USERS' OPINION

N = 203 [Hearing 8, HoH 68, Deaf 127]

Questionnaire & instruction in sign language

NER value Dancing with the Stars [Szczygielska & Dutka 2017] 21

NER value in final episode

99.5%

> 98 OFCOM recommendation

average delay in final episode

<5-6 sec.

real average delay in the UK after 10 years

lowest average delay in an episode(episode 3)

3.4 sec.

3 sec. OFCOM recommendation

of the users were satisfied or very satisfied with the quality of live subtitles for “Dancing...”

22

67.5%

BUT….Users’ opinions. Dancing with the Stars [Szczygielska & Dutka 2017]

23

Users’ opinions. Dancing with the Stars [Szczygielska & Dutka 2017] 24

the text reduction is very high with many utterances missing

22 %

MYTHreduction > 10%

the delay was high or very high

18.70 %

MYTHdelay > 6 sec.

subtitles are too fast, I can’t keep up with the pace

19.20 %

18,72%subtitles are too slow

Conclusions 1/2

25

Users might not be rational in their opinions.

Users can be very critical.

Users’ needs can be so diverse as to be mutually exclusive.

Our approach is user-focused and user-informed but not user-lead.

We should listen to users but at some points we have to make an informed decision.

26

Quality in the Eye of the Beholder. Quality measures and user satisfaction

in live subtitling.

Monika SZCZYGIELSKA, Łukasz DUTKA