Copyright © 2001 Intel Corporation.
On Multimedia Technology & On Multimedia Technology & Infrastructure for Emerging Applications:Infrastructure for Emerging Applications:
from Desktop to Wireless to Peer-to-Peerfrom Desktop to Wireless to Peer-to-Peer
Minerva M. Yeung, Ph.D.Minerva M. Yeung, Ph.D.<[email protected]><[email protected]>
Media Technology ResearchMedia Technology Research
Microprocessor Research LabsMicroprocessor Research LabsIntel Corporation.Intel Corporation.
Feb 20, 2001.Feb 20, 2001.Stanford SeminarStanford Seminar
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 2
IntelMedia Technology
MRL/TRL
Page 2
IntelMedia Technology
MRL/TRLAcknowledgements*Acknowledgements* Yen-Kuang Chen, Yen-Kuang Chen, Ph.D.Ph.D. Matthew HollimanMatthew Holliman Rainer Lienhart, Rainer Lienhart, Ph.D.Ph.D. Skip Macy, Skip Macy, Ph.D.Ph.D. Igor Kozintsev, Igor Kozintsev, Ph.DPh.D Andre Zaccarin, Andre Zaccarin, Ph.D.Ph.D. Valery Kuriakin and the iNNL-MPL teamValery Kuriakin and the iNNL-MPL team
(Russia)(Russia) And many othersAnd many others
* Key collaborators/contributors to the work mentioned in the talk. * Key collaborators/contributors to the work mentioned in the talk.
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 3
IntelMedia Technology
MRL/TRL
Page 3
IntelMedia Technology
MRL/TRLUbiquitous MultimediaUbiquitous MultimediaWired and WirelessWired and Wireless
Interactive, High Performance, Interactive, High Performance, Enriched MediaEnriched Media
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 4
IntelMedia Technology
MRL/TRL
Page 4
IntelMedia Technology
MRL/TRLDigital Media: from desktop, to Internet, to Digital Media: from desktop, to Internet, to hand-helds, to wireless, and to Peer-to-Peerhand-helds, to wireless, and to Peer-to-Peer
Edge server
Edge server
Data Farms / Storage
Web ServerApps/DB Server
E-Commerce Server
Media Server
Wireless Comm Server
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 5
IntelMedia Technology
MRL/TRL
Page 5
IntelMedia Technology
MRL/TRL
InequalityInequality: compute, bandwidth, : compute, bandwidth, storage and displaystorage and display
PlatformPlatform CPUCPU MemoryMemory StorageStorage B/WB/W ScreenScreen
ServerServer MultipleMultiple1GHz+1GHz+
2GB2GB 100G to 100G to TerabytesTerabytes
1+ Gbps1+ Gbps N/AN/A
PCPC SingleSingle
1GHz+1GHz+
256M256M 40G40G 100 Mbps100 Mbps 1600 x 12001600 x 1200
LaptopLaptop 600 MHz600 MHz 128M128M 10G10G 100 Mbps100 Mbps 1024 x 7681024 x 768
Media Media
PDAPDA
200MHz200MHz 16M16M 8M8M 19.219.2kbpskbps
320 x 240320 x 240
PDAPDA 70MHz70MHz 8M8M 8M8M 19.2 kbps19.2 kbps 160 x 160160 x 160
Implications:Implications:– Blurring of Core vs. EdgeBlurring of Core vs. Edge
– Migration from client-server to peer-to-peer frameworkMigration from client-server to peer-to-peer framework
– Where should data come from?Where should data come from?
– Where should computations be done?Where should computations be done?
– Optimal partitioning of computationsOptimal partitioning of computations
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 6
IntelMedia Technology
MRL/TRL
Page 6
IntelMedia Technology
MRL/TRLChallengesChallengesWhat is the optimal multimedia What is the optimal multimedia
content distribution mechanism?content distribution mechanism?What is the optimal multimedia What is the optimal multimedia
processing methodology?processing methodology?What are the technology building What are the technology building
blocks?blocks?
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 7
IntelMedia Technology
MRL/TRL
Page 7
IntelMedia Technology
MRL/TRLMultimedia dataMultimedia dataSame content can exist in different Same content can exist in different
resolutions, bit rate and qualityresolutions, bit rate and qualityCan be gracefully degradedCan be gracefully degradedCan be transformed to multiple Can be transformed to multiple
representationsrepresentations
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 8
IntelMedia Technology
MRL/TRL
Page 8
IntelMedia Technology
MRL/TRLP2P for MultimediaP2P for Multimedia Maintain key features and extendMaintain key features and extend
– Supports traditional Supports traditional (existing)(existing) file sharing systems file sharing systems– Facilitates sharing of resources like CPU cycles, storage, Facilitates sharing of resources like CPU cycles, storage,
computation process or arbitrary functionalities that can be computation process or arbitrary functionalities that can be abstracted by the user as resources.abstracted by the user as resources.
Facilitates exchange of data with real time Facilitates exchange of data with real time requirements, exemplified by video or audio requirements, exemplified by video or audio transmission.transmission.
– Choice and flexibility of transport mechanisms Choice and flexibility of transport mechanisms – Meeting diverse application requirementsMeeting diverse application requirements
Efficient multimedia servicesEfficient multimedia services– Integrates media transformation/transcodingIntegrates media transformation/transcoding
Flexible query interfaces to facilitate multimedia Flexible query interfaces to facilitate multimedia searchsearch
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 9
IntelMedia Technology
MRL/TRL
Page 9
IntelMedia Technology
MRL/TRL
MAPS MAPS MAPS
MAPS: Media Accelerating Peer ServicesMAPS: Media Accelerating Peer Services
Technology, platform architecture, and Technology, platform architecture, and software infrastructuresoftware infrastructure– facilitates facilitates efficientefficient, , transparenttransparent and and high-high-
performanceperformance media content delivery & processing media content delivery & processing
– can take advantage of, integrate into, and bring out can take advantage of, integrate into, and bring out the benefits of the P2P computing infrastructurethe benefits of the P2P computing infrastructure
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 10
IntelMedia Technology
MRL/TRL
Page 10
IntelMedia Technology
MRL/TRLSystem ArchitectureSystem Architecture
Network Transport Services (TCP socket, CORBA, DCOM, SOAP)
Peer-to-Peer Service Layer (general security, Directory service, data replication, flow control, etc.)
Multi-media applicationSupport modules (Transcoding, compression,security/watermarking, etc.)
Script language
Application
operating system
applicationTraditional
Support modules of other types of applications (Document editing, distributed processing, etc.)
(Multimedia)MediaAcceleratingPeerServiceLayer
MAPS API
PlatformOptimization
MAPS API
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 11
IntelMedia Technology
MRL/TRL
Page 11
IntelMedia Technology
MRL/TRLSystem ComponentsSystem ComponentsPlatform Optimization (Media Acceleration)Platform Optimization (Media Acceleration)
– Detect and analyze the optimal platform configuration and Detect and analyze the optimal platform configuration and operation at a particular time instantoperation at a particular time instant
– Dynamic optimization via cost-function analysis across a Dynamic optimization via cost-function analysis across a globalglobal or or local local set of peer nodes: the result determines a set of peer nodes: the result determines a bestbest operational mode for a particular media delivery or operational mode for a particular media delivery or processing operationprocessing operation
– OptimizationOptimization is built upon (and calling) a set of support is built upon (and calling) a set of support modules, like streaming, transcoding, etc. for multimedia modules, like streaming, transcoding, etc. for multimedia delivery operations, and other support modules like delivery operations, and other support modules like distributed image processing primitives for multimedia distributed image processing primitives for multimedia processing operations.processing operations.
– InvisibleInvisible and and seamlessseamless to the applications to the applications– Overall results cascaded into an enhanced service Overall results cascaded into an enhanced service
experience experience – fast delivery, higher satisfaction, lower abort rates, and better fast delivery, higher satisfaction, lower abort rates, and better
viewing/listening/interactive experiences, …viewing/listening/interactive experiences, …
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 12
IntelMedia Technology
MRL/TRL
Page 12
IntelMedia Technology
MRL/TRLMinimal Cost to Deliver a Minimal Cost to Deliver a Resource to a Peer NodeResource to a Peer NodeTo find the minimal ofTo find the minimal of
– cost of getting a copy locally and transcoding if cost of getting a copy locally and transcoding if necessary, necessary,
– cost of getting a copy from a peer node + cost of cost of getting a copy from a peer node + cost of transmissiontransmission
Minimal cost Minimal cost Minimal cost
Minimal cost Minimal cost
Minimal cost
i j k
m n
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 13
IntelMedia Technology
MRL/TRL
Page 13
IntelMedia Technology
MRL/TRLCost AnalysisCost Analysis Request to evaluate cost Request to evaluate cost
of Rof R If the file is not cached If the file is not cached
(local), then find the (local), then find the minimal cost from the set minimal cost from the set of neighbors (can extend of neighbors (can extend from local to global)from local to global)
Cost of transcoding Cost of transcoding computationcomputation
Cost of transmission Cost of transmission available network available network bandwidthbandwidth
Other cost factors can be Other cost factors can be formulatedformulated
Cost of retrieval,
transcoding, &transmission
Repeat for the set ofneighbors
Compare the minimal cost
Cache the result
Return
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 14
IntelMedia Technology
MRL/TRL
Page 14
IntelMedia Technology
MRL/TRLMinimal Cost to Perform Minimal Cost to Perform a computationa computationFor a computation task on data (resource R), For a computation task on data (resource R),
optimize with respect to several variables optimize with respect to several variables such as delay, power consumption, temp such as delay, power consumption, temp storage, etc. (constraint parameters P)storage, etc. (constraint parameters P)
To find the minimal ofTo find the minimal of– cost of performing the computation locally, cost of performing the computation locally, – cost of transmitting the data to a peer (or multiple cost of transmitting the data to a peer (or multiple
peers), perform the computation there and peers), perform the computation there and transmitting the result backtransmitting the result back
– cost of finding the same data on a peer, perform cost of finding the same data on a peer, perform the computation there and transmitting the result the computation there and transmitting the result backback
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 15
IntelMedia Technology
MRL/TRL
Page 15
IntelMedia Technology
MRL/TRL
Cost Computation & EstimationCost Computation & Estimation
What is the cost?What is the cost?–Cost should be dynamicCost should be dynamic
– Resource availability at “server”Resource availability at “server”– Resource availability at “client”Resource availability at “client”– Network availabilityNetwork availability
How to measure?How to measure?How to estimate?How to estimate?Methodology?Methodology?Tools?Tools?
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 16
IntelMedia Technology
MRL/TRL
Page 16
IntelMedia Technology
MRL/TRLMethodology for Estimating End-to-End Computing Performance and Analyzing Network Computing Models
Methodology for Estimating End-to-End Computing Performance and Analyzing Network Computing Models
In a networked & distributed computing environment, end-to-end performance of applications depend on capability of components in the system and their interaction.
Need methodology to estimate end-to-end system performance.
Results of performance estimation used to make computing decisions.
– Best distribution of computational tasks.– Best algorithm choice.– Most effective hardware improvement.
– Enhanced security
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 17
IntelMedia Technology
MRL/TRL
Page 17
IntelMedia Technology
MRL/TRLOverviewOverview
results
best distribution
bestalgorithm
lowestcost
models
queuingnetwork
calculations
operations
grayscale
scale size
compression
halftone
encryption
digitalsignatures
Methodology
inputs
processing rates
data size
network bandwidth
cost weights
…
performancemetrics
response time
maximumthroughput
cost
quality
security
Sample Operations
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 18
IntelMedia Technology
MRL/TRL
Page 18
IntelMedia Technology
MRL/TRLSample Results: Computation DistributionSample Results: Computation Distribution
Assumptions: Assumptions: – server processing rate >> client server processing rate >> client
processing rateprocessing rate
– Simplest model; other server/client Simplest model; other server/client disparities not includeddisparities not included
Figure 1Figure 1 Server: grayscale, scale, halftone Server: grayscale, scale, halftone Client: none Client: none Results:Results:
– Processing rate limited. Processing rate limited.
– Lowest response time.Lowest response time.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0 4 8 12
arrival rate (requests/sec)
resp
onse
tim
e (s
ec) server process
server network
client network
client process
total
Color convert 512x512 to gray, scale to 128x128, halftone to 1-bit/pixel
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 2 4 6 8 10
arrival rate (requests/sec)
resp
onse
tim
e (s
ec)
0
0.5
1
1.5
2
2.5
3
0 3 6 9 12 15
arrival rate (requests/sec)re
spon
se ti
me
(sec
)
Figure 2 (Left)Figure 2 (Left) Server: grayscale, scale Server: grayscale, scale Client: halftone Client: halftone
Results:Results:
– Highest request throughputHighest request throughput
– client process raises timeclient process raises time
Figure 3 (Right)Figure 3 (Right) Server: grayscale, Server: grayscale,
scale, compress scale, compress Client: decompress, Client: decompress,
halftone halftone
Results: Results: Bandwidth rate limited. Bandwidth rate limited.
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 19
IntelMedia Technology
MRL/TRL
Page 19
IntelMedia Technology
MRL/TRLSample Results: Quality as a Performance MetricSample Results: Quality as a Performance Metric Compare JPEG2000, SPIHT and JPEGCompare JPEG2000, SPIHT and JPEG
– Order of processing rate: JPEG, SPIHT, JPEG2000 Order of processing rate: JPEG, SPIHT, JPEG2000 – Order of compression ratio for given PSNR: JPEG2000, SPIHT, JPEG Order of compression ratio for given PSNR: JPEG2000, SPIHT, JPEG
ServerServer and and Client Client can have equal capability (measured on PIII 600 MHz systems for the following) can have equal capability (measured on PIII 600 MHz systems for the following)
0
10
20
30
40
50
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
PSNR
co
mp
res
sio
n
JPEG2000
JPEG
SPIHT
Compression vs PSNR 128x128 24-bit Lena
JPEG2000 2 3
JPEG l6 15
SPIHT 12 12
MethodCompression
(Mb/s) Decompression
(Mb/s) PSNR
30.9
27.8
29.0
Comparison of processing rates
0
0.5
1
1.5
2
2.5
3
26 28 30 32 34 36 38 40
PSNR
maxim
um
th
rou
gh
pu
t
JPEG2000
JPEG
SPIHT
Network limits throughput deceases with PSNR
Processing limited
Maximum throughput for 1.5 Mbps
0
2
4
6
8
26 28 30 32 34 36 38 40
PSNR
dela
y (
sec)
JPEG2000
JPEG
SPIHT
Delay for low arrival rate against PSNR
@1.5 Mbps. JPEG2000 closer
to JPEG and SPIHT for slower
network
Delay increases with PSNR
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 20
IntelMedia Technology
MRL/TRL
Page 20
IntelMedia Technology
MRL/TRLMultimedia SupportMultimedia Support ““Malleability”/“flexibility” of (bulky) MultimediaMalleability”/“flexibility” of (bulky) Multimedia
– Graceful degradationGraceful degradation– Various representationsVarious representations– Complexity Complexity b/w and compute intensive! b/w and compute intensive!– More flexibility More flexibility more opportunities more opportunities
Special technology components in MAPSSpecial technology components in MAPS– Universal across multiple platformsUniversal across multiple platforms– Media Adaptive TranscodingMedia Adaptive Transcoding
– e.g. MPEG2 (6Mbps) e.g. MPEG2 (6Mbps) MPEG4 (300kps) MPEG4 (300kps)
– Audio/video Streaming Audio/video Streaming – Robust, error resilient coding and transmissionRobust, error resilient coding and transmission– Enhanced Multimedia ProcessingEnhanced Multimedia Processing– Technology modules for media security & tracking Technology modules for media security & tracking
(encryption/watermarking)(encryption/watermarking)– Technology that facilitates multimedia browsing/search/managementTechnology that facilitates multimedia browsing/search/management
Intel® MPL can serve as a basic media software infrastructure that provides some core media technology components while facilitates the building of the other ones
Intel® MPL can serve as a basic media software infrastructure that provides some core media technology components while facilitates the building of the other ones
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 21
IntelMedia Technology
MRL/TRL
Page 21
IntelMedia Technology
MRL/TRLIntel MPL - Media Processing LibraryIntel MPL - Media Processing Library(presented at IDF Fall’99, Fall’00)(presented at IDF Fall’99, Fall’00) General purpose, high performance software infrastructure General purpose, high performance software infrastructure
with API’s for MPEG encode/decode and with API’s for MPEG encode/decode and processingprocessing BUILDING BLOCK enabling NEW rich-media apps: targeted for BUILDING BLOCK enabling NEW rich-media apps: targeted for
applications beyond standard encode/decode/display, e.g., applications beyond standard encode/decode/display, e.g., ease-of-use of video, allow interactivity with digital videoease-of-use of video, allow interactivity with digital video
Universal & Ubiquitous:Universal & Ubiquitous:– Common API’s across multiple OS: Windows, LinuxCommon API’s across multiple OS: Windows, Linux– Optimized for multiple architecture: IA, XScale™ Optimized for multiple architecture: IA, XScale™ (in progress)– Can extend to network applications (Can extend to network applications (Internet, wireless, etc.)Internet, wireless, etc.)
Scalable solutionScalable solution– Low bit rate to high bit rateLow bit rate to high bit rate– Low resolution to high resolutionLow resolution to high resolution– 1 processor to multiple processors1 processor to multiple processors– Small devices to high end serversSmall devices to high end servers
Rich features, high performance, ease-of-developmentRich features, high performance, ease-of-development
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 22
IntelMedia Technology
MRL/TRL
Page 22
IntelMedia Technology
MRL/TRLMPL – MPL – PC and Internet Multimedia S/W PC and Internet Multimedia S/W infrastructure for Diverse Platformsinfrastructure for Diverse Platforms
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 23
IntelMedia Technology
MRL/TRL
Page 23
IntelMedia Technology
MRL/TRLReference: MPL featuresReference: MPL features Support MPEG-1 and MPEG-2 up to HDTV, MPEG4Support MPEG-1 and MPEG-2 up to HDTV, MPEG4 Optimized with Optimized with MMX™ technology and SIMD Streaming Extensions, Pentium™ 4MMX™ technology and SIMD Streaming Extensions, Pentium™ 4 Advanced Features - the highlights:Advanced Features - the highlights:
– Random access to any frames with near constant-timeRandom access to any frames with near constant-time– Fast extraction of reduced framesFast extraction of reduced frames– Simultaneous decode of multiple MPEG sequences. Simultaneous decode of multiple MPEG sequences. – Access via callbacks (API) to non-frame level information in the MPEG bitstream, Access via callbacks (API) to non-frame level information in the MPEG bitstream,
such as raw bits, blocks, macroblocks, GOP and slice, etc. [structured access to such as raw bits, blocks, macroblocks, GOP and slice, etc. [structured access to compressed video/audio]compressed video/audio]
– Flexible input plugins: MPL - DTV, MPL – NetworkFlexible input plugins: MPL - DTV, MPL – Network– Audio support (AC3, audio level 1, 2 and 3 – MP3, AAC)Audio support (AC3, audio level 1, 2 and 3 – MP3, AAC)– SMP support; multithreaded designSMP support; multithreaded design– Transcoding capabilitiesTranscoding capabilities
Ease-of-development and deploymentEase-of-development and deployment– Windows*, Linux* Windows*, Linux* – MPL as Component: COM objects (for VB, VC++, DirectShow*)MPL as Component: COM objects (for VB, VC++, DirectShow*)– IPP support for audio & video (2001)IPP support for audio & video (2001)
– cross-platform optimization: Pentium™ processor families, Xscale™ core cross-platform optimization: Pentium™ processor families, Xscale™ core architecture based processors (in planning)architecture based processors (in planning)
* Other names and brands are property of their respective owners* Other names and brands are property of their respective owners
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 24
IntelMedia Technology
MRL/TRL
Page 24
IntelMedia Technology
MRL/TRL
Video processing
control
MixerMixerMixerMixer
Output systemOutput systemOutput systemOutput system
Inside the MPL pipelineInside the MPL pipeline
Encoded stream
Videooutput
Users may easily plug in their own algorithms to encoder’s pipeline and, also, fully control encoding and decoding processes.
Variousinput streams from various devices
PreconditioningPreconditioningPreconditioningPreconditioning
DCTDCTDCTDCT
Control algorithmControl algorithm
Motion estimationMotion estimationMotion estimationMotion estimation
Huffman codingHuffman codingHuffman codingHuffman coding
Headers generationHeaders generationHeaders generationHeaders generation
QuantizationQuantizationQuantizationQuantization
Motion vector search algorithmMotion vector
search algorithm
Bitrate controlBitrate control
Enc
oder
use
r ca
llba
ck f
unct
ion
Enc
oder
use
r ca
llba
ck f
unct
ion
Enc
oder
use
r ca
llba
ck f
unct
ion
Enc
oder
use
r ca
llba
ck f
unct
ion
MPEG encoderMPEG encoder
Video encoding control
Huffman decodingHuffman decodingHuffman decodingHuffman decoding
Headers processingHeaders processingHeaders processingHeaders processing
DequantizationDequantizationDequantizationDequantization
Inverse DCTInverse DCTInverse DCTInverse DCT
Motion CompensationMotion CompensationMotion CompensationMotion Compensation
Dec
oder
use
r ca
llba
ck f
unct
ion
Dec
oder
use
r ca
llba
ck f
unct
ion
Dec
oder
use
r ca
llba
ck f
unct
ion
Dec
oder
use
r ca
llba
ck f
unct
ion
MPEG decoderMPEG decoder
Media inputMedia inputMedia inputMedia input
SplitterSplitterSplitterSplitter
Input systemInput systemInput systemInput system
Comparison of MPL decoding (MPEG-2 704x480)
126 128,2 136,2
0
50
100
150
Tbird (1GHz) PIII (1GHz) P4 (1.1 GHz)Dec
odin
g ra
te
(FP
S)
High performance performance Media Infrastructure
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 25
IntelMedia Technology
MRL/TRL
Page 25
IntelMedia Technology
MRL/TRL
Decoder A
MPL Next StepsMPL Next Steps
NETWORK
Decoder B
Decoder C High bitrate, excellent quality
Average bitrate, good quality
Low bitrate, acceptable quality
Core BitstreamEnhancement Bitstream 1Enhancement Bitstream 2
Demultiplex
Elementary sreams Scenes
decoding
Objectsdecoding
Scene Graph Scene Graph
A/V object 1 A/V object 1 A/V object 2 A/V object 2 Person Person 2D background2D background FurnitureFurniture
Voice Voice SpriteSprite GlobeGlobe DeskDesk
Natural video
2D-Mesh 3D-Mesh Face&Body animation
Synthetic video
BIFS
MPEG-4 (simple to advanced profiles)MPEG-4 (simple to advanced profiles)
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 26
IntelMedia Technology
MRL/TRL
Page 26
IntelMedia Technology
MRL/TRLMPL Next StepsMPL Next Steps Extend to embrace & integrate more technology Extend to embrace & integrate more technology
building blocksbuilding blocks– wirelesswireless
– P2P – embrace technology catering to and take advantage of P2P – embrace technology catering to and take advantage of the “inequalities” inherent in the computing platformsthe “inequalities” inherent in the computing platforms
– low power considerationslow power considerations
– Merging of video & graphicsMerging of video & graphics
Strive toStrive to– Cater to diversity (universal)Cater to diversity (universal)
– Be among the best in Be among the best in performanceperformance
– Be flexible, and maximize utility/usabilityBe flexible, and maximize utility/usability
– Be a key building block infrastructure for emerging Be a key building block infrastructure for emerging applicationsapplications
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 27
IntelMedia Technology
MRL/TRL
Page 27
IntelMedia Technology
MRL/TRL
Facilitating Multimedia SearchFacilitating Multimedia SearchAutomatic media content analysis Automatic media content analysis
algorithms AND meta-data can improve algorithms AND meta-data can improve search efficiency.search efficiency.
New browsing tools and methodologies can New browsing tools and methodologies can be deployed for distributed media sharing, be deployed for distributed media sharing, storage and retrieval, and collaboration.storage and retrieval, and collaboration.
Flexible query interfaces to facilitate Flexible query interfaces to facilitate multimedia searchmultimedia search
Search can be based on BOTH the file Search can be based on BOTH the file names and content descriptions.names and content descriptions.– e.g. MPEG7 or XMLe.g. MPEG7 or XML
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 28
IntelMedia Technology
MRL/TRL
Page 28
IntelMedia Technology
MRL/TRLContent analysis & SearchContent analysis & SearchExample: Video SearchExample: Video Search
– Very compact representationsVery compact representations– Fast Matching on Compact Fast Matching on Compact
SignaturesSignatures
Matched ClipsMatched Clips
representation: 4.26 real numbers per frame of video*
250 min of video searched within 7 sec on a PII 400 against 50 frame query
* achieved using window size 30 and polynomial degree 1;histograms of Y, U, V channels quantized into 16, 8 and 8 bins
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 29
IntelMedia Technology
MRL/TRL
Page 29
IntelMedia Technology
MRL/TRLOn Compact Representations & Fast SearchOn Compact Representations & Fast Search(Naphade, Yeo & Yeung, earlier work)(Naphade, Yeo & Yeung, earlier work) Extract compressed low-resolution temporal signature Extract compressed low-resolution temporal signature
– Compute spatial low resolution signature - for MPEG video, extract/estimate Compute spatial low resolution signature - for MPEG video, extract/estimate DC images for I/P/B frames, on compressed dataDC images for I/P/B frames, on compressed data
– Fit polynomial of N degrees over M samples in timeFit polynomial of N degrees over M samples in time– Describe temporal trace using the N degree polynomial for each M samplesDescribe temporal trace using the N degree polynomial for each M samples
Achieved matching rate of 6000+ frames/sec on a low-end PCAchieved matching rate of 6000+ frames/sec on a low-end PC Reduce complex MM search/matching to a fast string-matchingReduce complex MM search/matching to a fast string-matching Same signature & compact size regardless of resolution of videoSame signature & compact size regardless of resolution of video
DC Sequence Histogram Sequence
Database of Polynomial Coefficients
Similarity MetricComputation
DC SequenceHistogram Sequence
Approximated Histogram Sequence
Incoming Query
Retrieved Clips
Incoming Video
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 30
IntelMedia Technology
MRL/TRL
Page 30
IntelMedia Technology
MRL/TRL
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 31
IntelMedia Technology
MRL/TRL
Page 31
IntelMedia Technology
MRL/TRLBuilding MAPS into a P2P Building MAPS into a P2P computing platformcomputing platform The enhanced multimedia peer service layer can be The enhanced multimedia peer service layer can be
integrated into existing/emerging P2P integrated into existing/emerging P2P infrastructureinfrastructure
– To make P2P media-aware and optimized for media delivery and media applications…
APIs can be defined on P2P platform to facilitate APIs can be defined on P2P platform to facilitate multimedia applications developmentmultimedia applications development
Prototype & experimentation at Intel MRLPrototype & experimentation at Intel MRL
Internet Service “edge” service
Peer-to-Peer Service Media Accelerating
Peer Service MAPS
Internet Service “edge” service
Peer-to-Peer Service Media Accelerating
Peer Service MAPS
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 32
IntelMedia Technology
MRL/TRL
Page 32
IntelMedia Technology
MRL/TRLPrototype systemPrototype system
DaemonDaemon– Low level Low level
primitivesprimitives
– High level High level controllerscontrollers
Client APIsClient APIs – Application Application
access to P2P access to P2P networknetwork
P2P Daemon
Client API
ApplicationsP2P Daemon
Client API
Applications
Peer
Peer
Other Peer
Other Peer
Other Peer
Internet
The Global NetworkUniverse
(Intel MRL 2001)
Peer -to-Peer -to- peer “super” daemon
Daemon -to -daemoncommunication
Cost evaluation module
Media transcodingmodule
Application -to-daemon APIs
Daemon -to -daemoncommunication
Cost evaluation module
Media transcodingmodule
Application -to-daemon APIs
User defined modules
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 33
IntelMedia Technology
MRL/TRL
Page 33
IntelMedia Technology
MRL/TRLSeamlessSeamless Multimedia for Dynamic and Multimedia for Dynamic and Heterogeneous EnvironmentHeterogeneous Environment
HUB802.11b/BT/??
802.11b/BT/??
802.11b/BT/?? 10/100baseT
10/100baseT
Tablets
PDA’s & devices
• Transcoding for bandwidth management• Robust Wireless Streaming/transmission• Multi-platform Encode/Decode• Resource Optimization
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 34
IntelMedia Technology
MRL/TRL
Page 34
IntelMedia Technology
MRL/TRLPeer Media ExplorerPeer Media Explorer
SnapshotSnapshot
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 35
IntelMedia Technology
MRL/TRL
Page 35
IntelMedia Technology
MRL/TRLExtending Multimedia Apps and Extending Multimedia Apps and Technology from Desktop to P2PTechnology from Desktop to P2P
Emerging media-centric applications:Emerging media-centric applications: Content Transaction, Tracking and MonitoringContent Transaction, Tracking and Monitoring Media content indexing, browsing & organizationMedia content indexing, browsing & organization Interactive video, and merged video and graphicsInteractive video, and merged video and graphics Enhanced virtual realityEnhanced virtual reality Enhanced video conferencingEnhanced video conferencing Networked multimedia gamesNetworked multimedia games
Goal: to enable multimedia apps development and deployment on P2P with enhanced user experience.Goal: to enable multimedia apps development and deployment on P2P with enhanced user experience.
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 36
IntelMedia Technology
MRL/TRL
Page 36
IntelMedia Technology
MRL/TRLStreaming, sharing and Streaming, sharing and collaborating on digital mediacollaborating on digital media
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 37
IntelMedia Technology
MRL/TRL
Page 37
IntelMedia Technology
MRL/TRLBuilding BlocksBuilding BlocksBuilding BlocksBuilding Blocks
MPL Call-back/hooks
Ease-of-use as a natural data typeEase-of-use as a natural data type
Media DeliveryMedia DeliveryInteractive VideoInteractive Video
Manipulation/processingfast, intuitive, on un/compressed media, “Word for Video”
Scalable streamingFGSScalable streamingFGS
“Rendering” of video, graphics (& audio)
“Rendering” of video, graphics (& audio)
Automatic content analysis & abstractionAutomatic content analysis & abstraction
Robust/resilient delivery theory and practiceRobust/resilient delivery theory and practice
Optimization of interactive streaming
Optimization of interactive streaming
Search, browse, and retrieveSearch, browse, and retrieve
TranscodingB/W-compute adaptabilitySmarter delivery schemes
TranscodingB/W-compute adaptabilitySmarter delivery schemes
Object recognition & segmentationObject recognition & segmentation
WatermarkingEncryption, SecurityWatermarkingEncryption, Security
MPEG4 Technology and Apps UtilizationMPEG4 Technology and Apps Utilization
Desktop Networked Wired to wireless ubiquity P2PDesktop Networked Wired to wireless ubiquity P2P
PerformancePerformance Resource InequalityResource InequalityPowerPower
Next-gen CODEC’sNext-gen CODEC’s
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 38
IntelMedia Technology
MRL/TRL
Page 38
IntelMedia Technology
MRL/TRLWireless Multimedia:Wireless Multimedia:Robust Video Delivery over Wireless Robust Video Delivery over Wireless Networks (IP, Bluetooth, and beyond)Networks (IP, Bluetooth, and beyond)
Send ACK
CanDecode?
CanDecode?
Packets
PacketizePacketize GenerateFEC
GenerateFEC
ACKGot ACK?Got ACK?
MPEGfile
MPEGfile
WirelessAccess Point
WirelessAccess Point
File Selection
Packets
DecodeMPEG
DecodeMPEG
DecodeFEC
DecodeFEC
WirelessAccess Point
WirelessAccess Point
BufferBuffer
BufferBuffer
TCP
New Results
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 39
IntelMedia Technology
MRL/TRL
Page 39
IntelMedia Technology
MRL/TRLLooking forward…Looking forward… What is the optimal multimedia content distribution What is the optimal multimedia content distribution
mechanism?mechanism? What is the optimal multimedia processing What is the optimal multimedia processing
methodology?methodology? What are the technology building blocks? What are the technology building blocks? Many more questions…Many more questions…
– What are the computing models of the future?What are the computing models of the future?– What are the best algorithms?What are the best algorithms?– What are the What are the performanceperformance criteria? criteria?
Many good problems for researchMany good problems for researchNew framework, new formulation, …New framework, new formulation, …
Copyright © 2001 Intel Corporation.
More on Intel MRL More on Intel MRL Media ResearchMedia Research
University collaborationsUniversity collaborations
Summer internshipsSummer internships
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 41
IntelMedia Technology
MRL/TRL
Page 41
IntelMedia Technology
MRL/TRLAugmenting Research via University Augmenting Research via University Collaboration & PartnershipsCollaboration & Partnerships
Collaborations (2000-01)Collaborations (2000-01)–UC Berkeley UC Berkeley –Polytech University (NY) Polytech University (NY) –othersothers
PartnershipsPartnerships– InternshipsInternships–Technical exchanges, hosting visitsTechnical exchanges, hosting visits–Courses, resource utilization (e.g. s/w h/w Courses, resource utilization (e.g. s/w h/w
research infrastructure)research infrastructure)
Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20. Page 42
IntelMedia Technology
MRL/TRL
Page 42
IntelMedia Technology
MRL/TRLSummer Internships @ Intel MRLSummer Internships @ Intel MRL May/June to August/September 2001 (12-13 May/June to August/September 2001 (12-13
weeks, can be renewed to 9 months)weeks, can be renewed to 9 months) Looking for outstanding candidates (MS, Looking for outstanding candidates (MS,
Ph.D., and very exceptional BS students)Ph.D., and very exceptional BS students) Areas:Areas:
1.1. Algorithms research and development (general, Algorithms research and development (general, in various fields) – video, audioin various fields) – video, audio
2.2. Low power researchLow power research
3.3. P2P System researchP2P System research
4.4. Wireless multimedia, communicationsWireless multimedia, communications– Email: Email: [email protected]@intel.com and Matt and Matt
will forward to appropriate researcherswill forward to appropriate researchers