real-time networking: data, voice, and video

Real-Time Networking:Real-Time Networking:Data, Voice, and VideoData, Voice, and Video

Dan Schonfeld

Multimedia Communications Laboratory

ECE Department

University of Illinois

Chicago, Illinois

Multimedia Communications Multimedia Communications LaboratoryLaboratory

Rashid AnsariRashid AnsariAshfaq KhokharAshfaq KhokharDan SchonfeldDan Schonfeld

Oliver YuOliver Yu

A Vision To The Future!A Vision To The Future!Real-Time High-Quality Global Video Communications over High-Real-Time High-Quality Global Video Communications over High-

Speed NetworksSpeed Networks

Research AreasResearch Areas

Video NetworkingVideo TrackingVideo Retrieval

Video NetworkingVideo Networking

Collaborators:Collaborators:Rashid Ansari (UIC)Rashid Ansari (UIC)Bulent Cavusoglu (UIC)Bulent Cavusoglu (UIC)Tom DeFanti (UIC)Tom DeFanti (UIC)Jason Leigh (UIC)Jason Leigh (UIC)Emir Mulabegovic (UIC)Emir Mulabegovic (UIC)Oliver Yu (UIC)Oliver Yu (UIC)

Homeland Security Homeland Security Application TestbedApplication Testbed

Chicago Video Surveillance Monitoring of Biochemical Sensor Arrays Helmet Mounted Cameras and Portable Sensors

for Search/Rescue Operations Dedicated Emergency Telephony Network

– Public network lack of reliability during emergency

Real-Time Real-Time Surveillance/MonitoringSurveillance/Monitoring

Central Monitoring– Ok as long as links are

dedicated

Central Monitoring

Search and rescue

Remote sharing with agencies in the field

Mobile and Portable Emergency Response Center

Next Generation Internet Next Generation Internet ArchitectureArchitecture

CR-LDPBGPIGP

ICMPRSVP

Signaling Protocol

Out-of-Band Associated

Signaling Transport

Traffic Control

Fiber/DWDM

SONETATM/PPP/HDLC

IP Routing

Path SelectionMPLS Forwarding

IP QoS Control

HTTP/SMTP/FTP/TELNET

Real-time Multimedia

Middleware

RTPUDP

TCP

Best Effort Guaranteed & Controlled-Load

Non-Real-time Real-time

IP / MPLS

Topology Distribution

Transport

User Application

Current

Internet NGI

Extension

Net

wor

k &

Lay

er

Man

agem

ent

VIPER: FEC/UDP Over VIPER: FEC/UDP Over STARTAPSTARTAP

(Chicago-Amsterdam)

Latency of transmitting 100 packets underUDP, TCP, FEC/UDP with 3:1 redundancy.

0

50

100

150

200

250

300

350

400

0 500 1000 1500 2000 2500

Packet size in bytes

1-way latency in ms

UDP

TCP

FEC over UDP

APPLICATION

RTP

UDP

IP

DATA LINK

PHYSICAL

TRANSPORT

Picture Type

f_code

MBZ T TR AN N S B E P FBV BFC FFV FFC

MPEG-1 Header extension

X Ef(0,0)

f(0,1) f(1,0) f(1,1) DC PS T P C Q V A R H G D

MPEG-2 Header extension

Real-Time Real-Time Transport Transport Protocol Protocol

(RTP)(RTP)

AdaptiveAdaptiveFECFEC

RTP MEDIAPACKETS

ADAPTIVEFEC

ENCODER

RTP FECPACKETS

MPEG-2 RTP NETWORK RECEIVER

1 2 3 4 5 6 7 8 9 1030

32

34

36

38

40

42

44

46

48

50

Packet Loss Ratio Percentage

PSNR in dB

AFECstaticIPBstaticFECoptimalFEC

(c) AFEC

(b) Static IPB(a) Static FEC

(d) Original

FEC SimulationsFEC Simulations

DiffServDiffServ

HOST

EDGE ROUTER CORE ROUTER CORE ROUTER EDGE ROUTER

HOST

TOKEN BUCKET

WEIGHTED FAIRQUEUING

(WFQ)

CHECK IF THERE IS ANYTOKENS AVAILABLE FOR

THE ASSIGNED DSCP

YES

ASSIGN NEXT AVAILABLEDSCP FROM THE TABLE

TOKENS

NO DSCP TABLEAF31AF32AF33AF21AF22AF23AF11AF12AF13

BEST EFFORT

QUEUESAF3

AF2

BESTEFFORT

RANDOM EARLY DETECTION(RED)

TRANSMITTED PACKETS

DROPPED PACKETS

MARKERWEIGHTS

FOR PACKETS

HOSTHOST

HOST

OTHER SOURCES THATCONTRIBUTES TRAFFIC

VIDEOSOURCES

VIDEO SOURCE WITH THEPROPOSED MARKING

ALGORITHM

RANDOM EARLY DETECTION(RED)

RANDOM EARLY DETECTION(RED)

5.7 5.75 5.8 5.85 5.9 5.95 6 6.05 6.1 6.1518

19

20

21

22

23

24

25

26

27

28

Mbps

PSNR in dB

MotionIPBGreedyOptimum

(c) Motion

(b) IPB(a) Greedy

(d) Original

DiffServ SimulationsDiffServ Simulations

Rate-ControllerRate-Controller

NETWORK

DROPPEDPACKETS

HOST HOST

HOST

HOST

HOST

OTHER FLOWS

HOST

HOST

HOST

OTHER FLOWS

RATE CONTROLLED FLOW

( )ip μ

LINEAR STATEFEEDBACKREGULATOR

$

$[ ]

( )

( ),

min( )i

tot

D

E Dp µ

p µ

$( )p µ

Nack packetsor RTCP report

( )iμ

µ

Networkpk

KalmanPredictor

vk

zk

wk

z-1 inv()

1kμ +

( ) 1kμ

−

kα inv()

1kα−

1

−

+

1( )k kp μ+)

11 kα−−

uk

uk

11(1 ) ( )k k kpα μ−+− )

−

+kp

z-1( )k kp μ)

1( )kp + µ)1 1

1

( )

( )

dk k

k

choosep μ

μ+ +

+ ∈µ

)

1kμ +

( )k

Generate

u µ

5

1

2

7

2

5

2

2 4

4

4

7 5

4

6

(b) Choke-ORCA(a) Choke-Only

Rate-Control SimulationsRate-Control Simulations

Lightweight Streaming Lightweight Streaming Protocol (LSP)Protocol (LSP)

Videofile

Framer Discriminator Packetizer Packetbuffer

Sender

ReceiverControlShared

parametersand statistics

RTX

To client

Control messages from theclient

Nominal frame rate (NFR)

Actual frame rate (AFR)

Server architecture

SEQ1 SEQ2(lost) SEQ3 SEQ4(lost) SEQ5 SEQ2 (rtx) SEQ4(rtx) NACK2 (ignored)

NACK2 (NACK2 + NACK4)

Receiver Sender

PacketsTrans-mitted

Packets received

UDP LSP LSP-PMN

DSL 94% 99% 100%

Wi-Fi 95% 96% 98%

LSPLSP

PSNR = 18 dBms PSNR = 33 dBms

QoS Control QoS Control Over Wireless & Over Wireless &

Core NetworkCore Network

CDMA-Based CDMA-Based Admission & Admission & SchedulingScheduling

(Oliver Yu)(Oliver Yu)

Internet

Backbone

Gateway

Backbone

Wireless QoS Control

Core QoS Control

Internet QoS Control

Router

MSC

AP-CAC

Backbone

BS

FDWFQ-MAC

AMDG-CAC

Wireless MAC ProtocolsWireless MAC Protocols(Khokhar)(Khokhar)

Listen Idle

Zzzz

Zzzz

Zzzz

Zzzz

Zzzz

Zzzz

Zzzz

Zzzz

Zzz

z

Zzzz

Duty Cycle

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Event Arrival Rate

Normalized Power Consumption

TDMA-W, 50 nodesTDMA-W, 100 nodesTDMA-W, 200 nodes10% S-MAC, 50 nodes10% S-MAC, 100 nodes10% S-MAC,200 nodes

9.30%

8.16%

5.13%

0.56%

Channel

Call AdmissionController

Slot Level Scheduler

X

X

Center for Global Multimedia Center for Global Multimedia Mobile CommunicationsMobile Communications

Internet over CableInternet over Cable

Digital Subscriber LinesDigital Subscriber Lines

A typical ADSL equipment configuration.

Low-Earth Orbit SatellitesLow-Earth Orbit SatellitesIridiumIridium

(a) The Iridium satellites form six necklaces around the earth.

(b) 1628 moving cells cover the earth.

GlobalstarGlobalstar

(a) Relaying in space.(b) Relaying on the ground.

The 802.16 Physical LayerThe 802.16 Physical Layer

The 802.16 transmission environment.

Computer Network Computer Network InfrastructureInfrastructure

UICNU

NCSA

UCSD

Optical Networking Transport Encoding

& Protocols Wired and Wireless

Network Integration Circuit and Packet Switched

Network Deployment

Wireless CommunicationsWireless Communications

Router Wireless Access Pointoptical Switch Wireless Terminal

Optical Core

PacketData Network

Wireless Access Network:• IEEE 802.11 (Year 1)• IEEE 802.16 (Years 2 & 3)• IEEE 802.20 (Optional)

Wireless AccessNetwork

Opportunistic Resource Allocation & Admission Control

Channel Estimation Power-Efficient

Wireless Protocols High-Capacity

Wireless Networks

Applications & PrototypesApplications & Prototypes

Video Communications Tele-Education Natural Event Monitoring Geosciences Monitoring Environmental Assessment Emergency Management Elderly Care Medical Diagnosis Remote Robotic Surgery

Visualization & DevicesVisualization & Devices

High-Resolution Scalable Displays

High-Resolution Capture

Interactive Tools

Intelligence SharingIntelligence Sharing

Real Time Monitoring and Real Time Multimedia Retrieval and Sharing across the continent

Video TrackingVideo Tracking

Collaborators:Collaborators:Nidhal Bouaynaya (UIC)Nidhal Bouaynaya (UIC)Karthik Hariharakrishnan (Motorola Research)Karthik Hariharakrishnan (Motorola Research)Dan Lelescu (NTT DoCoMo Research)Dan Lelescu (NTT DoCoMo Research)Josh Meir (NeoMagic)Josh Meir (NeoMagic)Magdi Mohamed (Motorola Research)Magdi Mohamed (Motorola Research)Wei Qu (UIC)Wei Qu (UIC)Philippe Raffy (R2 Technology)Philippe Raffy (R2 Technology)Fathy Yassa (NeoMagic)Fathy Yassa (NeoMagic)

MotivationMotivation

Target Tracking Surveillance Retrieval Video Coding Video Communications Videoconferencing Virtual Reality Human-Computer Interaction Computer Animation

VORTEXVORTEX

Reference frame

Object cluster

VORTEX: Video Retrieval and Tracking from Compressed Multimedia Databases

Template Template Matching [sec] VORTEX [sec]

Object #1 45.22 0.0084

Object #2 39.36 0.0092

VORTEXVORTEX

Adaptive Block Matching Adaptive Block Matching (ABM)(ABM)

Method Time [sec]

ABM 10

Partition Projection 165

Partition Lattice Operators 193

MBPFCondensation filter

Motion-Based Motion-Based Particle FiltersParticle Filters

Multi-Object Particle FiltersMulti-Object Particle Filters

. . .. . .

. . .

......

...

......

...

21x

1

mx 2mx

22x

1

2x1

1x 1

tx

2tx

m

tx

1

1z

2

1z

1mz 2

mz

2

2z

1

2z 1

tz

2

tz

mtz

HMM

MHMM

x1 x2 xt. . . .

z1 ztz2

p(zt|xt)

p(xt|xt-1)

Experimental ResultsExperimental Results

The Dynamic Graphic Model for Multiple Interactive Objects In Two Frames

IDMOTIDMOT

Magnetic-Inertia ModelMagnetic-Inertia Model

Reward

Punish

Video Tracking and Video Tracking and FoveationFoveation

(Ansari & Khokhar)(Ansari & Khokhar)

Future Research:Future Research:Video TrackingVideo Tracking

Randomly Perturbed Active Surfaces Video Stabilization Auto-Focus Recovery Pose Estimation and Feature Tracking Video Animation Stereography from a Single Camera Multiple Camera Mosaics Multiple Camera Tracking Low-Power Particle Filters

Video RetrievalVideo Retrieval

Collaborators:Collaborators:Faisal Bashir (UIC)Faisal Bashir (UIC)Ashfaq Khokhar (UIC)Ashfaq Khokhar (UIC)Dan Lelescu (NTT DoCoMo Research)Dan Lelescu (NTT DoCoMo Research)Fatih Porikli (Mitsubishi Research)Fatih Porikli (Mitsubishi Research)

MotivationMotivation

• Video Surveillance• Sign Language Recognition• Sports Video Analysis• Animal Mobility Experiments• Moving Object Databases• Video and Sensor Databases

Spectral ClusteringSpectral Clustering

Trajectory RetrievalTrajectory Retrieval

0

500

1000

1500

2000

2500

3000

1 2 3 4 5 6

PCA-Global

PCA-Seg Euc

PCA-Seg Str

Lei Chen

Gaussian Mixture ModelsGaussian Mixture Models

Figure: 1-Sigma contours of GMM’s learnt from three classes.(a) ‘Norway’. (b) ‘Alive’. (c) ‘Crazy’.

1

cN

i i ii

P( y ) ( y; , )π μ=

Θ = ∑∑ ¥

HMM for Class NHMM for Class 1

ClassificationClassification

…

…

Gaussian Mixtures

Training Set Database

Classification:

[ ]( )1

1m i

i , ,Larg max p Y , ,Y λ

∈ LL

AccuracyAccuracy

Datasets

ASL

HJSL#Classes : # Trajectories

2:138 4:276 8:552 16:1104 29:2001 38:2622

HMM 0.9638 0.9167 0.8587 0.7790 0.6882 0.6609 0.9074

GMM 0.9855 0.8949 0.8514 0.7455 0.6672 0.6400 0.8981

Moghaddam 0.9420 0.9312 0.8297 0.7283 0.5592 0.6175 0.4537

Accuracy values for various class sizes from ASL data set and the HJSL dataset (last column). Column headings are shown as (number of classes:number of trajectories) for the ASL dataset at different sizes.

Shape RepresentationShape Representation

Curvature Scale SpaceCurvature Scale Space

CSS Images of a Trajectory and its 36-degree rotated versionFigure: An example high jump trajectory and its translated, rotated and non-uniformly scaled version, along with their CSS images.

PerformancePerformance

Indexing Time (sec.)

(408 Traj.)

Retrieval Time (sec.)

(15 Traj.)

PCA Centroid 178.9270 8.2920

Hybrid PCA 175.2020 27.5400

CSS 1508.3 28.0500

Future Research:Future Research:Video RetrievalVideo Retrieval

Trajectory OcclusionCamera MotionMultiple CamerasMultiple TrajectoriesVideo MiningJoint Retrieval, Recognition, & MiningMulti-Modality Feature Integration