the mpeg-7 standard - a brief tutorial - ali tabatabai sony us research laboratories february 27,...
TRANSCRIPT
The MPEG-7 Standard- A Brief Tutorial -
Ali Tabatabai
Sony US Research Laboratories
February 27, 2001
2
Outline
Objectives of the MPEG-7 Standard
Main Elements of MPEG-7
Scope of MPEG-7
MPEG-7 Application Areas
MPEG 7’s relation with other standards
3
Why do we need MPEG-7 ?
N e e d
• Content Management
• Fast & Accurate Access
• Personalized Content Production and Consumption
• Automation
+
Support for Advanced Query
•Visual•Audio•Sketch
4
MPEG: A Brief History (1)
MPEG: Moving Picture Experts Group
ISO / IEC/JTC1/SC29/WG11
A Working Group of ISO/IEC in charge of the Development of Standards for
Coded Representation of Digital Audio and Video
Established in 1988
5
MPEG: A Brief History (2)
MPEG-1: Interactive CD and MP3 11 / 1992
MPEG-2: DTV, STB, DVD 11 / 1994
MPEG-4: Web and Mobility ver1: 09 /1998
ver2: 11
/1999
MPEG-7: ??? 08 / 2001
MPEG-21: Multimedia Framework 11 / 2001
6
MPEG-7: What Is It ?
THE MPEG 7 THE MPEG 7 STANDARDSTANDARD THE MPEG 7 THE MPEG 7 STANDARDSTANDARD
Content Description of Various Audio Visual
Information
IS NOT a COMPRESSION Standardsimilar to MPEG-1/2/4 or their
Extension
IS NOT a STANDARD forFEATURE
EXTRACTION/MATCHING
Types of Audio Visual Information• Audio, Speech• Moving video, still pictures, graphics• Information on how objects are combined in scenes
Types of Audio Visual Information• Audio, Speech• Moving video, still pictures, graphics• Information on how objects are combined in scenes
MPEG-7: Application Areas Storage and retrieval of audiovisual databases (image, film,
radio archives) Broadcast media selection (radio, TV programs) Surveillance (traffic control, surface transportation, production
chains) E-commerce and Tele-shopping (searching for clothes /
patterns) Remote sensing (cartography, ecology, natural resources
management) Entertainment (searching for a game, for a karaoke) Cultural services (museums, art galleries) Journalism (searching for events, persons) Personalized news service on Internet (push media filtering) Intelligent multimedia presentations Educational applications Bio-medical applications
8
MPEG-7Description Scope for AV Content
Description Granularity Low-level
High-level
Form
Access
Classification
Link
Context
9
MPEG-7: Main Elements
Descriptors (D)
syntax and semantics of each feature representation
Description Schemes (DS)
structure and semantics of the relationships between
components
Description Definition Language (DDL)
creation of new DS’s
modification/extension of existing DS’s
10
MPEG-7: Major Functionalities
Systems (ISO / IEC 15938 -
1)
Description Definition Language (ISO / IEC 15938
- 2)
Visual (ISO / IEC 15938 -
3)
Audio (ISO / IEC 15938 -
4)
Multimedia Description Schemes (ISO / IEC 15938
- 5)
Reference Software (ISO / IEC 15938 -
6)
11
DS1
DS2 DS2
DS3 D1 D3 D2 Systems
0001100
<Object> <Label/> <Definition/>
. .
</Object>
DDL
Instantiation
MPEG-7: Main Elements (2)
12
MPEG-7: Systems
It defines tools to:
manage and protect intellectual property
synchronize between content and description
provide for efficient storage and transport
13
MPEG-7: DDL and its Components
Description Definition Language:
Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions
Instantiation of XML
XML Schema:
Data types
Simple and Complex types
Elements, attributes
Inheritance, Abstract types
MPEG-7 extensions:
Array and Matrix data type
14
MPEG-7: Audio
Sound Effects
Music Instrument Timbre
Spoken Content
Melody Contour
15
MPEG-7: Visual (1)
Color quantization, dominant, scalable, color-structure, layout,
GoF/GoP
Texture
Shape region-based, contour-based, 3D
Motion camera motion, motion trajectory, parametric motion,
motion activity
16
MPEG-7: Visual (2)
Localization spatio temporal
Others face recognition
17
MPEG-7: Basic Visual Structures
Grid Layout
2D-3D Multiple View
Time Series
Spatial 2D Coordinates
Temporal Interpolation
18
Low level Audio Visual descriptors
• Color • Camera motion• Motion activity• Mosaic
• Color • Motion trajectory• Parametric motion• Spatio-temporal
shape
• Color • Shape• Position• Texture
Video segments Still regions
Moving regions Audio segments
• Spoken content • Spectral
characterization• Music: timbre,
melody
19
MPEG-7: MMDS Basic Elements
Datatype &structures
Link & medialocalization Basic DSs
Basic elementsBasic elements
Schematools
Time, Duration, Medialocators Language
Annotation,Person, Place
Root, Top-level elements, Packages
20
MPEG 7: Content Management and Description
Content descriptionContent description
Content managementContent management
Creation &production
Media ContentUsage
Conceptualaspects
Structuralaspects
Title, Creator, Creation location & date, Purpose,
Classification, Genre, Review, Parental guidance,
etc. (Author generated)
Format, Coding, Instances, Identification, Transcoding
Hint, etc.(Several instances)
Rights holder, Access rights, Usage Record, Financial aspects,
etc. (Evolution)
Datatype &structures
Link & medialocalization Basic DSs
Basic elementsBasic elements
Schematools
Viewpoint of the structure: Segments• Spatial / temporal structure• Audio, video low-level Ds
• Elementary semantic information.
Viewpoint of conceptual notions• Events, objects, abstract concepts, and
their relation
21
Foreground
Example of Segment trees
SR1: Creation, Usage meta
information Media description Textual annotation Color histogram, Texture
SR2: Shape Color Histogram Textual annotation
SR6: Color Histogram Textual annotation
SR5: Shape Textual annotation
SR4: Shape Color Histogram Textual annotation
SR3: Shape Color Histogram Textual annotation
Background
22
Segment Tree
Shot1 Shot2 Shot3
Segment 1
Sub-segment 1
Sub-segment 2
Sub-segment 3
Sub-segment 4
segment 2
Segment 3
Segment 4
Segment 5
Segment 6
Segment 7
Semantic DS (Events)
• Introduction
• Summary
• Program logo
• Studio
• Overview
• News Presenter
• News Items
• International
• Clinton Case
• Pope in Cuba
• National
• Twins
• Sports
• Closing
TimeAxis
23
MPEG 7: Navigation and Access
Navigation &Navigation &AccessAccess
Summary
Variation
Content descriptionContent description
Content managementContent management
Creation &production
Media ContentUsage
Conceptualaspects
Structuralaspects
Datatype &structures
Link & medialocalization Basic DSs
Basic elementsBasic elements
Schematools
Efficient support of: discovery, browsing, navigation, visualization
Substitution of the original contentAdaptation to terminal, network, or
user preferences
24
MPEG 7: Hierarchical summary
A-VData
HierarchicalSummary
HighlightLevel HighlightLevel
HighlightSegment
HighlightSegment
HighlightSegment
HighlightSegment
HighlightSegment
HighlightSegment
HighlightSegment
25
MPEG 7: Sequential summary
SequentialSummary
A-VData
TextProperty
TextProperty
FrameProperty
FrameProperty
FrameProperty
SoundProperty
SoundProperty
SoundProperty
26
MPEG 7: Variation
VIDEO IMAGE TEXT AUDIO
Modality
Fidelity
Source
Variation
A
IH
GFE
DCB
Universal Multimedia Access Adapt delivery to network and terminal characteristics (QoS)
27
MPEG-7: Content Organization
Navigation &Navigation &AccessAccess
Summary
Variation
AnalyticModel
Content organizationContent organization
Content descriptionContent description
Content managementContent management
Creation &production
Media ContentUsage
Conceptualaspects
Structuralaspects
Description and organization of collection of documents
Collection &Classification
Datatype &structures
Link & medialocalization Basic DSs
Basic elementsBasic elements
Schematools
28
MPEG-7: Collection
29
MPEG 7: User Interaction
Navigation &Navigation &AccessAccess
Summary
Variation
AnalyticModel
Content organizationContent organization
Content descriptionContent description
Content managementContent management
Creation &production
Media ContentUsage
Conceptualaspects
Structuralaspects
Collection &Classification
UserUserInteractionInteraction
User preferences
Datatype &structures
Link & medialocalization Basic DSs
Basic elementsBasic elements
Schematools
User identification and preferences:Filtering, search and browsing
30
MPEG-7 Its Relation with other standards
AHG on “Metadata harmonization”:
SMPTE: Metadata dictionary, KLV encoding
Dublin Core Metadata Initiative
European Broadcast Union
AHG on TV AnyTime Application
Large number of Liaisons:
SMPTE
Dublin Core
W3C (XML Schema)
etc.
Competition:• Individual work• Definition scope and r
MPEG-7: TimeLine - The Work Plan
Call for proposalsCall for proposals
20012000199919981996
Working draftWorking draft Committee draftCommittee draft
Draft internationalstandard
Draft internationalstandard
Final committeedraft
Final committeedraft
InternationalstandardInternationalstandard
Divergence Convergence
32
Conclusions on AV Content Description and MPEG-7
MPEG-7:
AV content description for interoperable
application
Description Definition Language:
XML Schema (flexibility) + Binary version
(efficiency)
Description Schemes:
Library of description tools
Covers a wide range of generic needs