1. Analysing the Quality of Experience of Multisensory Media from Measurements of Physiological Responses PhD Candidate Jacob Donley, Christian Ritz, Muawiyath Shujau School of Electrical, Computer and Telecommunications Engineering ICT Research Institute & Global Challenges University of Wollongong

2. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli)

3. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli) ← Ambient Lighting → ↓ ↑ Vibration Wind ↙Wind ↘4. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli)

5. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli) • Subjective testing of multisensory media – Discrete method – Continuous method

6. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli) • Subjective testing of multisensory media – Discrete method – Continuous method • Discrete – MOS has previously shown advantageous use of multisensory media

7. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli) • Subjective testing of multisensory media – Discrete method – Continuous method • Discrete – MOS has previously shown advantageous use of multisensory media • Continuous – Does the MOS reflect the experience evenly?

8. INTRODUCTION 2 • Multisensory Media (Multiple Sensory Stimuli) • Subjective testing of multisensory media – Discrete method – Continuous method • Discrete – MOS has previously shown advantageous use of multisensory media • Continuous – Does the MOS reflect the experience evenly? • What affect does synchronicity of effects have?

9. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) 3

10. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) 3

11. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) 3

12. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) • Sub-band frequency powers (Alpha, Beta, etc.) 3

13. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) • Sub-band frequency powers (Alpha, Beta, etc.) • Standardised Low Resolution Brain Electromagnetic Tomography (sLORETA) 3

14. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) • Sub-band frequency powers (Alpha, Beta, etc.) • Standardised Low Resolution Brain Electromagnetic Tomography (sLORETA) • Eye Gaze Tracking – Sony PS3Eye and Dual Infrared Lights • Modified variable focus lens (No IR filter) 3

15. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) • Sub-band frequency powers (Alpha, Beta, etc.) • Standardised Low Resolution Brain Electromagnetic Tomography (sLORETA) • Eye Gaze Tracking – Sony PS3Eye and Dual Infrared Lights • Modified variable focus lens (No IR filter) • Large sensor designed for variable lighting 3

16. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) • Sub-band frequency powers (Alpha, Beta, etc.) • Standardised Low Resolution Brain Electromagnetic Tomography (sLORETA) • Eye Gaze Tracking – Sony PS3Eye and Dual Infrared Lights • Modified variable focus lens (No IR filter) • Large sensor designed for variable lighting • High frame rate (60fps) 3

17. EMOTION FROM BIOSIGNALS • Electroencephalography (EEG) – Emotiv EEG (14 Channels) – Analysis: • Emotiv suites (Proprietary algorithms) • Event-Related Potential (ERP) (e.g. P300 responses) • Sub-band frequency powers (Alpha, Beta, etc.) • Standardised Low Resolution Brain Electromagnetic Tomography (sLORETA) • Eye Gaze Tracking – Sony PS3Eye and Dual Infrared Lights • Modified variable focus lens (No IR filter) • Large sensor designed for variable lighting • High frame rate (60fps) – ITU Gaze Tracker (Open Source) • Iris, Pupil and Glint (Haar Features) 3

18. BIOSENSOR-BASED QOE EVALUATION SYSTEM 4

19. BIOSENSOR-BASED QOE EVALUATION SYSTEM • Supports Full HD Audio/Video 4

20. BIOSENSOR-BASED QOE EVALUATION SYSTEM • Supports Full HD Audio/Video • Multisensory Media via Philips amBX system – MPEG-V Capable (Standard for Multisensory Media) 4

21. BIOSENSOR-BASED QOE EVALUATION SYSTEM • Supports Full HD Audio/Video • Multisensory Media via Philips amBX system – MPEG-V Capable (Standard for Multisensory Media) • Emotiv EEG calibration & recording – Facial Expressions via Expressiv™ Suite – Subjective Emotions via Affectiv™ Suite – Raw EEG Potentials and Gyroscope Data 4

22. BIOSENSOR-BASED QOE EVALUATION SYSTEM • Supports Full HD Audio/Video • Multisensory Media via Philips amBX system – MPEG-V Capable (Standard for Multisensory Media) • Emotiv EEG calibration & recording – Facial Expressions via Expressiv™ Suite – Subjective Emotions via Affectiv™ Suite – Raw EEG Potentials and Gyroscope Data • ITU Gaze Tracker calibration & recording 4

23. BIOSENSOR-BASED QOE EVALUATION SYSTEM 5

24. SUBJECTIVE TESTS • Asynchronous => All effects preceding A/V by 500ms • 500ms => Average skew perceptually noticeable for media synchronisation [1, 2] [1] R. Steinmetz, "Human perception of jitter and media synchronization," Selected Areas in Communications, IEEE Journal on, vol. 14, pp. 61-72, 1996. [2] W. Yaodu, X. Xiang, K. Jingming, and H. Xinlu, "A speech-video synchrony quality metric using CoIA," in Packet Video Workshop (PV), 2010 18th International, 2010, pp. 173-177. 6 Effects None Asynchronous Synchronous Subjects 10 Videos per 15 Subject

25. SUBJECTIVE TESTS • Asynchronous => All effects preceding A/V by 500ms • 500ms => Average skew perceptually noticeable for media synchronisation [1, 2] [1] R. Steinmetz, "Human perception of jitter and media synchronization," Selected Areas in Communications, IEEE Journal on, vol. 14, pp. 61-72, 1996. [2] W. Yaodu, X. Xiang, K. Jingming, and H. Xinlu, "A speech-video synchrony quality metric using CoIA," in Packet Video Workshop (PV), 2010 18th International, 2010, pp. 173-177. 6 Effects None Asynchronous Synchronous Subjects 10 Videos per 15 Subject

26. SUBJECTIVE TESTS • Asynchronous => All effects preceding A/V by 500ms • 500ms => Average skew perceptually noticeable for media synchronisation [1, 2] [1] R. Steinmetz, "Human perception of jitter and media synchronization," Selected Areas in Communications, IEEE Journal on, vol. 14, pp. 61-72, 1996. [2] W. Yaodu, X. Xiang, K. Jingming, and H. Xinlu, "A speech-video synchrony quality metric using CoIA," in Packet Video Workshop (PV), 2010 18th International, 2010, pp. 173-177. 6 EEG Setup Eye Gaze Calibration Example Media Trial QoE Vote x 3 Effects None Asynchronous Synchronous Subjects 10 Videos per 15 Subject

27. RESULTS • QoE Votes agree with previous research 75 55 35 QoE Vote (%) QoE Votes and 95% Confidence Interval Without Effects With Asynchronous Effects With Synchronous Effects Shortened Video Titles 7

28. • The effect of synchronicity on QoE votes T-Test showing the probability that mean QoE would be observed the same 50% 40% 30% 20% 10% 0% Probability RESULTS (alpha=0.05) Without Effects & Async Without Effects & Sync Async & Sync Shortened Video Titles N.B. One-way ANOVA showed discernable differences 8

29. DISCUSSION • What if a subject’s vote is biased in favour of a single event? 9

30. DISCUSSION • What if a subject’s vote is biased in favour of a single event? • Is there any observable physiological difference between the asynchronous and synchronous events? 9

31. DISCUSSION • What if a subject’s vote is biased in favour of a single event? • Is there any observable physiological difference between the asynchronous and synchronous events? • Can we find a correlation with the votes from either EEG or Gaze data? 9

32. RESULTS • Emotiv’s algorithms are proprietary & values undefined. • Brain lobe functionality well documented. 10

33. RESULTS • Emotiv’s algorithms are proprietary & values undefined. • Brain lobe functionality well documented. • Using sLORETA algorithm: Most Active Brain Lobes bit.ly/EEG_3D_ Vid 10 Frontal Temporal Occipital No Sensory Effects Sync Sensory Effects Async Sensory Effects

34. RESULTS • Emotiv’s algorithms are proprietary & values undefined. • Brain lobe functionality well documented. • Using sLORETA algorithm: 100% 75% 50% 25% 0% No Effects Async Sync Most Active Lobe (%) Type of Effects Frontal Parietal Limbic Occipital Temporal QoE MOS Most Active Brain Lobes Frontal Temporal Occipital bit.ly/EEG_3D_ Vid 10 No Sensory Effects Sync Sensory Effects Async Sensory Effects

35. PRELIMINARY RESULTS • Possible EEG correlation with Eye Gaze deviation Frontal Frontal Frontal Frontal Frontal Occipital 500 400 300 200 100 0 No Effects Async Sync Standard Deviation (Pixels) Type of Effects Subject 1 - X Coord Subject 1 - Y Coord Subject 2 - X Coord Subject 2 - Y Coord 11

36. PRELIMINARY RESULTS • Possible EEG correlation with Eye Gaze deviation Frontal Frontal Frontal Frontal Frontal Occipital 500 400 300 200 100 0 No Effects Async Sync Standard Deviation (Pixels) Type of Effects Subject 1 - X Coord Subject 1 - Y Coord Subject 2 - X Coord Subject 2 - Y Coord Frontal Frontal Occipital No Sensory Effects Sync Sensory Effects 11 Async Sensory Effects

37. CONCLUSIONS • Multisensory effects enhance QoE 12

38. CONCLUSIONS • Multisensory effects enhance QoE • Synchronicity at this level & direction is indiscernible 12

39. CONCLUSIONS • Multisensory effects enhance QoE • Synchronicity at this level & direction is indiscernible • Correlating EEG lobe activity – 25% increase in temporal lobe for async – 20% increase in occipital lobe for sync 12

40. CONCLUSIONS • Multisensory effects enhance QoE • Synchronicity at this level & direction is indiscernible • Correlating EEG lobe activity – 25% increase in temporal lobe for async – 20% increase in occipital lobe for sync • Software package for physiological subjective testing and multisensory media playback 12

41. CONCLUSIONS • Multisensory effects enhance QoE • Synchronicity at this level & direction is indiscernible • Correlating EEG lobe activity – 25% increase in temporal lobe for async – 20% increase in occipital lobe for sync • Software package for physiological subjective testing and multisensory media playback • Large dataset of physiological responses for multisensory stimuli (Availability depending on ethics restrictions) 12

42. Questions?

•

•

•

•

•–

–

•

•–

–

•

•

•–

–

•

•

•

•–

–

•

•

•

•–

•–

–•

•–

–•

•

•–

–•

•

•

•–

–•

•

•

•

•–

–•

•

•

•

•–

•

•–

–•

•

•

•

•–

•

•

•–

–•

•

•

•

•–

•

•

•

•–

–•

•

•

•

•–

•

•

•

–•

•

•

•

–

•

•

–

•

–

–

–

•

•

–

•

–

–

–

•

•

•

•

•

•

•

● ● ●

•

35

55

75

Qo

E V

ote

(%

)

Shortened Video Titles

Without Effects With Asynchronous Effects With Synchronous Effects

0%

10%

20%

30%

40%

50%

Pro

bab

ility

Shortened Video Titles

Without Effects & Async Without Effects & Sync Async & Sync

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

0%

25%

50%

75%

100%

No Effects Async SyncMo

st A

ctiv

e Lo

be

(%)

Type of Effects

Frontal

Parietal

Limbic

Occipital

Temporal

QoE MOS

•

0

100

200

300

400

500

No Effects Async Sync

Stan

dar

d

Dev

iati

on

(P

ixe

ls)

Type of Effects

Subject 1 - X Coord

Subject 1 - Y Coord

Subject 2 - X Coord

Subject 2 - Y Coord

•

0

100

200

300

400

500

No Effects Async Sync

Stan

dar

d

Dev

iati

on

(P

ixe

ls)

Type of Effects

Subject 1 - X Coord

Subject 1 - Y Coord

Subject 2 - X Coord

Subject 2 - Y Coord

•

•

•

•

•

•–

–

•

•

•–

–

•

•

•

•–

–

•

•