Immersive Multimedia and

Virtual Crowds

Farzad Safaei

Farzad Safaei

Slide 2

Physical and Virtual

Mixed reality continuum

The boundary between physical and virtual is

being blurred

Augmented Reality Augmented Virtuality Real Virtual

Farzad Safaei

Slide 3

Augmented reality

Augmentation of physical with virtual

Farzad Safaei

Slide 4

Augmented virtuality

Augmentation of virtual with physical

Farzad Safaei

Slide 5

Augmented virtuality challenges

• Presentation of video and audio within a virtual

space

• New methods of rendering

• New modes of consumption

• Complexity of assessing quality

Farzad Safaei

Slide 6

Examples

• Scalable video collaboration

• Free viewpoint video

Scalable Video Collaboration

Farzad Safaei

Slide 8

iSee

• Combines virtual spaces with video conferencing

• Emulates a natural human gathering

• Can scale to very large meetings

Farzad Safaei

Slide 9

Why augmented virtuality?

Farzad Safaei

Slide 10

Technical scalability

The required capacity grows

as the square of number of

participants

Farzad Safaei

Slide 11

Current solution

Farzad Safaei

Slide 12

Current solution

Farzad Safaei

Slide 13

Cognitive scalability

Utility will diminish if there

are too many video and audio

streams

Farzad Safaei

Slide 14

iSee scalability

• By emulating a natural human gathering,

iSee has both technical and cognitive

scalability

• But also smaller meetings become more

usable

demo

Farzad Safaei

Slide 15

iSee scalability

• Uses much less bandwidth than conventional

tools – 70% - 90% less bandwidth

– The bandwidth saving will be more for larger meetings

Farzad Safaei

Slide 16

iSee scalability

• Allows easy interaction in large

gatherings

– Multiple simultaneous conversations

– Moving from one conversation to another

Farzad Safaei

Slide 17

Key methods for technical scalability

Visibility Culling

• Only receive those video streams that are

within the visual field of view of this user

• Distance

• View frustum

• Back face

• Occlusions

P. Pourashraf, F. Safaei, and D. R. Franklin, "Distributed Area of Interest Management for Large-

Scale Immersive Video Conferencing," in Multimedia and Expo Workshops (ICMEW), 2012 IEEE

International Conference on, 2012, pp. 139-144.

Farzad Safaei

Slide 18

Visibility Culling

• More than 70% reduction with 25 participants – The saving is more significant as the number of users increases

Farzad Safaei

Slide 19

Visibility Culling

• Dealing with movement

– Predicting future visibility based on motion vectors

Farzad Safaei

Slide 20

Key methods for technical scalability

Quality Culling

• Only receive the video quality that is needed based on the

perspective of the user

• Virtual distance

• Virtual orientation

• Our technique for this is called Perceptual Pruning

F. Safaei, P. Pourashraf, and D. Franklin, "Large-scale immersive video

conferencing by altering video quality and distribution based on the virtual context,"

Communications Magazine, IEEE, vol. 52, pp. 66-72, 2014.

Farzad Safaei

Slide 21

Quality Culling

• The required quality is dependent on orientation and

distance of the avatars

Farzad Safaei

Slide 22

Quality Culling

Texel to pixel mapping • Video is rendered as texture on the front surface of the avatar

• Will result in texture minification and magnification

Farzad Safaei

Slide 23

Quality Culling

Perceptual pruning

• DCT block down-sampling based on texture minification

Farzad Safaei

Slide 24

Quality Culling

• Significant bandwidth saving in addition to visibility culling

Farzad Safaei

Slide 25

Quality Culling

• 61% saving in worst-case scenario (lecture theatre)

Farzad Safaei

Slide 26

Key methods for technical scalability

Saliency ranking

• During congestion, reduce quality of less salient videos

• Saliency measured by a number of attributes

• Virtual distance

• Virtual orientation

• Voice activity

Farzad Safaei

Slide 27

Scalability tests

• Tests conducted during

2014

Farzad Safaei

Slide 28

Problems solved

• Scalability, and more …

• Easy for the students

– Natural interaction and socialisation

– Peer-based learning

• Easy for the instructors

– No need to adapt to a new mode of delivery

Farzad Safaei

Slide 29

Case studies

• DoE debating competition

• DoE Aurora College – https://drive.google.com/a/iseevc.com/file/d/0B9wPZZq-45gIM3FRclBsTWFEQkE/view?usp=sharing

• OTEN classes by invitation

• AIS art gallery – http://youtu.be/wfOfUWe58-4

• UOW Engineering Project Pitch – https://drive.google.com/file/d/0BzDp2xswG__IX0lKMTcySVctRTQ/view?usp=sharing

• UOW Alumni Fair

Farzad Safaei

Slide 30

OTEN Health Class

Farzad Safaei

Slide 31

UOW Engineering Pitch

Farzad Safaei

Slide 32

Some of the feedback … UOW pitch

• 90% of students and 100% of industry participants indicated they wanted

to participate in a similar activity again – 80% of industry participants said they would not have been able to participate in the event had it

been held on-camps

Lee, Mark, et al., “Engaging Industry in Student Projects Through a Video-Augmented 3D Virtual Trade Fair”,

Under review, IEEE Transactions on Education – contact me for a copy of current manuscript.

Farzad Safaei

Slide 33

Some of the feedback … UOW pitch

• Improvements in pitch presentation was significant

Lee, Mark, et al., “Engaging Industry in Student Projects Through a Video-Augmented 3D Virtual Trade Fair”,

Under review, IEEE Transactions on Education – contact me for a copy of current manuscript.

Farzad Safaei

Slide 34

Some of the feedback … UOW pitch

Nikolic S, et al., “2D versus 3D Collaborative Online Spaces for Student Team Meetings: Comparing a Web

Conferencing Environment and a Video-Augmented Virtual World”, To appear, Proceedings of Australasian

Association for Engineering Education Conference, AAEE2015.

Farzad Safaei

Slide 35

UOW Fair

Farzad Safaei

Slide 36

Some of the feedback … Alumni Fair

• Student responses:

Farzad Safaei

Slide 37

Some of the feedback … Alumni Fair

• Industry responses:

Farzad Safaei

Slide 38

Summary

• Intersection between virtual and physical will

be of significant social and commercial

importance

• Potential for new industries (and leaders) to

emerge

ICT Research Institute University of Wollongong