Multimedia Content(Creation & Transfer)

Rik Van de WalleFrank Van Reeth

2

Overview topics - main competences

Mobile multimedia applications reduction impact network losses transparent handover of multimedia sessions hardware/software co-design multimedia systems

3

Overview topics - main competences

Advanced multimedia applications (scalable) video compression multimedia content adaptation & transcoding video quality measurement and improvement (QoS) iDTV applications network-based virtual environments & applications 3D graphics & modelling animation-based applications speech synthesis speech enhancement & recognition optimized codec implementation (HW & SW)

4

Overview topics - main competences

Metadata technology development of new metadata models information retrieval (e.g., for key-shot detection) object recognition & feature extraction

5

Overview topics - main competences

Standardization leading role in standardization of

video coding - H.264/AVC & MPEG-4 SVCMPEG-21 technology image coding - JPEGW3C multimedia semantics incubator

active participation in other standardization organs:W3C, JVT, VQEG

follow-up of DICOM, DVB, IFTA and ITU activities

JPEG

6

Some examples

Selected results from IBBT projects in the domain ofMultimedia Content (Creation and Transfer) 3D networked virtual environment Free view point visual communication GPU-assisted video processing

7

(1) 3D Networked Virtual Environment

Goal: sustain communication of long-term or chronically illchildren (8-12) with classroom environment

Communication: synchronous vs. asynchrounous

attending lessons vs. reviewing lessons later on social vs. educational

friends at home vs. friends in school / during play-timevs. during teaching time

easy to use (i.e., image- and video-based) Proof-of-concept

video chat based 3D networked virtual environment

8

Existing solutions web based or video chat based:

(1) 3D Networked Virtual Environment

9

Targeted solution in ASCIT:

(1) 3D Networked Virtual Environment

10

(2) Free-View Point Visual Communication

Real-Time (on powerful PCs)Off-line,non-real-time,by simulation

Viewpoint interpolation

MPEG-4compression

Packetization

MPEG-4decoding

x2 for bi-directional communication

LAN /Pers. Netw.

Raw Data recording

Distributed Source Coding

Encryption

Decryption

Decoding

File File

3D Face to FaceVideo Conference Demo

11

(2) 3D Face to Face Video Conference:Face to Face – 2 persons: EYE GAZING PROBLEM

Webcam

ComputerDisplay

SOLUTION:

Have a virtual camera“behind” the display,looking straight in theeyes of the participant

EYE GAZING PROBLEM:

Your participant does notlook into your eyes becauseof mismatch betweenwebcam direction anddisplay direction

12

(2) 3D Face to Face Video Conference:“Virtual camera behind the display”

=

Viewpoint interpolation= Virtual camera

Real cameras

13

(3) GPU-assisted video processing

Graphics Processing Unit (GPU) originally designed for 3-D-graphics today a powerful and programmable co-processor

General Purpose Computations (GPGPU) data processing in parallel development is a challenge significant differences with CPU

conceptual and architectural Current research

high-definition video decoding and processing

14

(3) GPU-assisted video processing

Accelerating video decoding motion compensation in H.264/AVC color space conversion (YCoCg-R)

Design of proprietary video codec GPU-friendly lossless and scalable

Post-processing effects film grain reconstruction noise reduction error concealment