02/mar/2001 mpeg-7 applications: multimedia content retrieval, ima workshop 6b, university of...

02/Mar/2001 MPEG-7 Applications: Multimedia Content Retrieval, IMA Workshop 6B, University of Minnesota

Winter 2001: IMA Workshop 6BDigital Libraries: Classification, Retrieval and Visualization

MPEG-7 Applications: Multimedia Content Retrieval

Neil Day

Digital Garage Inc. Japan

ChairMPEG-7 Applications and Promotions to

Industry (MAPI)

[email protected]


MPEG-7 & Industry

MAPI MPEG-7 Applications & Promotion to Industry

MIFG MPEG-7 Industry Focus Group

www.mpeg-7.com

M7AE MPEG-7 Awareness Event

Paris Oct. 2000 Singapore March 2001


MAPI & MIFG & M7AE

Industry/Academia MPEG-7MAPI & IFG & M7AE


MPEG-7 Applications

About MPEG-7

Application Examples

Views on Next Steps


About MPEG•ISO/IEC SC29Wg11 Standards Group

•MPEG-1 began in 1988

•Today, over 300 developers attend working conferences

•Average 4 conferences per year

•Over 35 National Delegations Vote

•MPEG-7 Standard – September 2001


Content is Diversifying

20002000

20032003

TextText

AudioAudio

VideoVideo

GraphicsGraphics

TextText

AudioAudio

VideoVideo

GraphicsGraphics

TextText

GraphicsGraphics19961996

TextText

GraphicsGraphics

19971997

Changing Bandwidth Usage

Data: Inktomi, USA


MPEG-7 Scope

Feature Extraction:Content analysis (D, DS)

Feature extraction (D, DS)

Annotation tools (DS)

Authoring (DS)

MPEG-7 Scope:Description Schemes (DSs)

Descriptors (Ds)

Language (DDL)

Ref: MPEG-7 Concepts

Search Engine:Searching & filtering

Classification

Manipulation

Summarization Indexing

Feature SearchExtraction Engine

MPEG-7Description

standardization


Defining MPEG-7MPEG-7 is a standard

for describing features of multimedia content

• Comprehensive Set of AV descriptions based on:• Catalogue (e.g., title, creator, rights), • Semantic (e.g., the who, what, when, where information about objects and

events)• Structural (e.g., the colour histogram –measurement of the amount of colour

associated with an image or the timbre of an recorded instrument) features of the AV content

• Leverages on AV data representation defined by MPEG-1, 2 and 4

• Interoperability• MPEG-7 uses XML Schema as the language of choice for content description• MPEG-7 will be interoperable with other leading standards such as, SMPTE• Metadata Dictionary, Dublin Core, EBU P/Meta, and TV Anytime.


MPEG Standards

• MPEG-1: For the storage and retrieval of moving pictures and audio on storage media.• MPEG-2: For digital television, it’s the timely response for the satellite broadcasting and cable television industries in their transition from analog to digital formats.• MPEG-4: Codes content as objects and enables those objects to be manipulated individually or collectively on an audiovisual scene.MPEG-1, -2, and -4 make content available. MPEG-7 lets you to find the content you need.


MPEG-7 & XML

Over 100 XML Industry Standard GroupsXML Repository (www.xml.org)

What Makes MPEG-7 Different?

Survey of MPEG-7 Applications & Issues


Some Application Domains

Digital libraries, Education (image catalogue, musical dictionary, Bio-medical imaging catalogues…)l Multimedia editing (personalised electronic news service, media authoring)l Cultural services (history museums, art galleries, etc.),l Multimedia directory services (e.g. yellow pages, Tourist information, Geographical information systems)l Broadcast media selection (radio channel, TV channel,…)l Journalism (e.g. searching speeches of a certain politician using his name, his voice or his face),l E-Commerce (personalised advertising, on-line catalogues, directories of e-shops,…)l Surveillance (traffic control, surface transportation, non-destructive testing in hostile environments, etc.),l Investigation services (human characteristics recognition, forensics),l Home Entertainment (systems for the management of personal multimedia collections, including manipulation of content, e.g. home video editing, searching a game, karaoke,…)l Social (e.g. dating services),


Typical MPEG-7 Queries• Audio:

• Users want to search for songs by humming or whistling a tune.

• Graphics: • Sketch a few lines on a screen and get in return a set of images containing

similar graphics, logos, and ideograms.

• Image: • Define objects, including colour patches or textures and get in return• examples among which users select interesting objects to compose their

image.• Check if a company logo was advertised on a TV Channel as contracted

• Video: • Allow access by mobile phone to video clips of goals scored in a soccer

game. • Automatically search and retrieve any unusual movements from

surveillance videos


MPEG-7 Technical Activities• A set of descriptors:

• A Descriptor (D) is a representation of a Feature. A Descriptor defines the syntax and the semantics of the Feature representation.

• A set of description schemes: • A Description Scheme (DS) specifies the structure and semantics of the relationships

between its components, which may be both Ds and DSs.

• DDL: • A language to specify description schemes, the Description Definition Language

(DDL). It also allows the extension and modification of existing Description Schemes. MPEG-7 decided to adopt XML Schema Language as the MPEG-7 DDL. However the DDL will require some specific extensions to XML Schema Language to satisfy all of the requirements of MPEG-7. These extensions are currently being discussed through liaison activities between MPEG and W3C, the group standardizing XML.

• Binary Representation: • One or more ways (e.g., textual, binary) to encode descriptions. A Coded Description

is a Description that has been encoded to fulfill relevant requirements such as

compression efficiency, error resilience, random access, etc.


MPEG-7 Application Context

ContentManagement

Layer

ApplicationsLayer

ContentDescription

Layer

Norm

ativeN

on-norm

ative

D(s), DS(s), DDL

MM Content & Standards

DDLDescriptions

Internet Streaming Query-by-Sketch

Query-by-Humming

ApplicationDomain

MM Descriptions

MPEG-7Description Tools


Navigation &Access

Summary

Variation

Content management

Content description

Creation &Production

Media Usage

Conceptualaspects

Structuralaspects

Userinteraction

Userpreferences

ModelCollection &Classification

Content organization

Datatype &structures

Link & medialocalization

Basic DSs

Basic elements

SchemaTools

Organisation: MPEG-7 Descriptions


Description Schemes in MPEG-7

Creation and Production: Contains meta information that describes the creation and production of the content; typical features include title, creator, classification, and purpose of the creation.

Usage: Contains meta information that’s related to the usage of the content; typical features involve rights holders, access rights, publication, and financial information.

Media: Contains the description of the storage media; typical features include the storage format, the encoding of the AV content, and elements for the identification of the media.

Structural aspects: Contains the description of the AV content that represent physical, spatial, temporal, or spatio-temporal components of the AV content. (color, texture, shape, motion, audio).

Conceptual Aspects: Contains a description of the AV content from the viewpoint of its conceptual notions.

Basic Elements: deal with basic data types, mathematical structures, schema tools, linking and media localization tools, as well as basic DSs, which are elementary components of more complex DSs.


OtherFace Recognition

LocalizationBounding BoxRegion LocatorSpatio-Temporal Locator

MotionCamera MotionMotion TrajectoryParametric MotionMotion Activity

ShapeRegion ShapeContour Shape3D Shape2D-3D Multiple View

TextureHomogeneous Text.Texture BrowsingEdge histogram

ColorGoF/GoP ColorScalable ColorColor LayoutColor StructureDominant Color

Visual Description Tools


Image Features in MPEG-7

3. Spatial edge distribution

2. Spatial color distribution

4. Object shape

1. Color histogram

Image: Thanks to Ricoh Co. Ltd.


Video Segment Decomposition

Video Segment

Segment Decomposition

Moving Regions


• Time• Mosaic• Text Annotation

• Time• Color• Motion• Texture• Shape• Text Annotation

Segment Relation

Video Segments

Before

Segment Features

Segment Features


Video Segment


Moving Region


Moving Regions


• Time• Mosaic• Annotation

• Time• Color• Motion• Texture• Shape• Annotation

Relation Link

Video Segments

Above

Another Example


MPEG-7 Applications

Application Examples


MPEG-7 Visual Annotation Tool(IBM, USA)


Visual Annotation Example

1. Drag a Still Region element from the Definition Tree to the Description List area.

2. Click Add for Media Information,3. Click Add for Media Profile,4. Click Add for Media Format,5. Enter values for File Format, Medium, Color, …6. Click Add of Media Coding to Media Profile7. Enter values for Frame Width and Frame Height8. Click Add of Media Instance to Media Profile9. Enter value for Identifier10. Click Add of Creation Meta Information, …


Example Package-<Package title="Sports Program Package">

<Package title="Sports Events Basic Description Package"><scheme name="Title" type="element" schemeRecurse="false" /> <scheme name="TitleText" type="element" /> <scheme name="Place" type="element" /> <scheme name="Time" type="element" /> <scheme name="Annotation" type="element" />

</Package><Package title="Production and Copyright Package">

<scheme name="CreationMetaInformation" type="element" schemeRecurse="false" />

<scheme name="Creation" type="element" /> <scheme name="Classification" type="element" /> <scheme name="MediaInformation" type="element" /> <scheme name="MediaProfile" type="complexType"

schemeRecurse="false" /> <scheme name="MediaFormat" type="complexType" />

</Package> </Package>


Annotation Process

MPEG-7Schema / Package

<StateTransitionModel numberStates="3"><Initial Size="3"> 0.25 0.5 0.25 </Initial><Transitions Size1="3" Size2="3">

0.2 0.2 0.6 0.1 0.8 0.1 0.3 0.3 0.4</Transitions>

<State><Label> Pass </Label></State><State><Label> Shot on goal </Label></State><State><Label> Goal score </Label></State>

</StateTransitionModel>

<complexType name="StateTransitionModelType" base="mpeg7:FiniteStateModelType" derivedBy="extension">

<element name="Initial" type="mpeg7:ProbabilityVectorType" minOccurs="0" />

<element name="Transitions" type="mpeg7:ProbabilityMatrixType" /><sequence minOccurs="1" maxOccurs="unbounded"><element name="State" type="mpeg7:ModelStateType"/>

</sequence></complexType>

Parser

MPEG-7Visual Annotation

Tool Interface

MPEG-7 Schema Definition (DDL) MPEG-7 Package

MPEG-7 Elements

MPEG-7 Description

User Input

M enu

MP EG- 7 Sche ma

D ef ini t ion A rea

Descri pt ionEn try Area

M PE G- 7 D escri pti on Area:1. R eusabl e el em ents2. Elem ent s t r ee

3. Vali dat ed de scri pti on tr ee4. MP EG -7 descr ipt io n ( XM L)

4321

Descr ip ti on cop ying


MPEG-7 Camera ApplicationSmart Fire Alert

(Fastcom Technologies, Switzerland)


Alarm Event Management

XML log file (search & retrieve, conversion to HTML, WML)

Alarm transmission via GSM

Remote User Interface

Encrypted transmission


MPEG-7 Camera Features

MVS-135 based platform

Embedded image processing for motion detection and tracking

MPEG-7 encoder: XML formatted description of the motion

Serial transmission

MPEG-7 decoder: XML parser


Application of the MPEG-7 Camera

Broadcast Camera Controler

Initial Coarse Grain Metadata generator for:

• Automatic Document Analyser

• Digital Video from Broadcast camera

Use the technology to create MPEG-7 Broadcast Camera with MPEG-2 (4) bitstream output.


Highlight Generation Tool (ETRI, Korea)

FeaturesFeatures Video content analysis: multi-modal features extractionVideo content analysis: multi-modal features extraction

visual features (camera motion, embedded text, shot boundary, slow motion replay, face)

Textual information (natural language processing using closed-caption text)

speech recognition CC text DB population for news videoCC text DB population for news video Highlights detectionHighlights detection

Knowledge-based detection rules tuned based on genre such as sports, news and drama programs, etc.

MPEG-7 Summary DescriptionMPEG-7 Summary Description Summarization DS (Hierarchical Summary DS)


Highlight Generation ToolHighlight Generation Tool

Media Data Extractor Description EncoderMPEG-7 File Formatter

Summary Server

User Interface

DDLfeature index

Evaluator

User

Video Sequence

Video Data

MPEG-7 Browser

Feature IndexHS List

HS List

CC Text

Extractor

Key Sound

ExtractorShot BoundaryVisual FeaturesCC Text

SportsNews

Highlight Detection

M7Description Data


Using the Highlight Tool


Web-based MPEG-7 Browser (ETRI, Korea)


Applications: Highlight Tool

Applications: Highlight Generation ToolApplications: Highlight Generation Tool Advanced EPG generationAdvanced EPG generation Video content analysisVideo content analysis Video editing, authoringVideo editing, authoring CC text DB populationCC text DB population

Applications: BrowserApplications: Browser Agent for Agent for Storage-based STBStorage-based STB News video retrievalNews video retrieval Web-based video browserWeb-based video browser


Humming: Elements of Melody

(MIT Media Lab, USA)

Series of related pitches Invariant to transposition Invariant to instrumentation

Rhythm is important

“Bridal Chorus” from Lohengrin(“Here Comes the Bride”)

“O Tannenbaum”


Humming: Example

“Moon River”

Contour: 2 -1 -1 -1 -1 -1 1

Beat Number: 1 4 5 7 8 9 9 10


Humming: System Overview

Server

ScoresMIDIs

Melody

Representation

PlaybackClient

Pitch tracking

Construct query


Streaming over Internet

(Singingfish.com, USA)

Create XSL transformation

From: MSNBC "Partner XML Format"

To:MPEG-7 <Mpeg7Main> root element


Design

Stream Delivery Network

M SN BCPartner X M L Feed

Java FTP Servlet

W ork flowR D BM S

SearchIndex

Pro

mot

ion

Pro

mo

tio

n

Prom otionFTP

SchedulerX SL

Engine

Im porter

Prom otionJDBC

Singingfish.comSearch Engine

Prom otion


MSNBC <article>

<article storyorder="12" pubdate="10/23/2000 8:02:00 AM" source="Today show" topnews="12">

<filename>tdy_fletcher_mideast_001023</filename>

<duration>00:01:09</duration>

<headline>Peace hopes slip farther</headline>

<description>The slim hopes for peace in the Mideast are rapidly fading, NBCs Martin Fletcher reports Monday from the outskirts of Jerusalem.</description>

<keywords>Israel, palestinian, Yasser Arafat</keywords>

...</article>


Link to stream

<MediaProfile><MediaInstance>

<InstanceLocator><OnlineLocator>

<MediaURI>http://www.msnbc.com/news/asx/video/28/tdy_fletcher_mideast_001023.asx

</MediaURI> </OnlineLocator>

</InstanceLocator></MediaInstance>

<MediaProfile>


<headline>

<headline>Peace hopes slip farther</headline>

<Creation>

<Title>

<TitleText><xsl:value-of select="headline"/> </TitleText>

</Title>

</Creation>


< description>, <keywords><description>The slim hopes for peace in the Mideast are

rapidly fading, NBCs Martin Fletcher reports Monday from the outskirts of Jerusalem.</description>

<keywords>Israel, palestinian, Yasser Arafat</keywords>

<Abstract><FreeText>

<xsl:value-of select="description"/> </FreeText>

<WhatObject><FreeTerm><xsl:value-of select="keywords"/>

</FreeTerm></WhatObject></Abstract>


MPEG-7Application(DENSO, JAPAN)

Wireless Images Retrieval using Speech Dialogue Agent The agent in the client terminal recognizes user's utterance in

English/Japanese with rather dedicated sentences and send a query profile to the server using wireless transceiver channel (32kbps). The server PC will retrieve the requested images and deliver the compressed video bitstream (H.263) to the client PC. Then the client agent will reply with synthesized voice and display the images.This time we use MPEG-7 CD based format of metadata. Moreover, retrieval speed is several times higher than conventional methods by newly

incorporating dependency information.


How aboutYokohamaInterchange?

Wireless Image Retrieval by Speech Dialogue

32 - 64kbps

Client

PHS-PIAFS Transceiver Mode

This is the current statusof Yokohama Interchange.

PublicNetwork

Base StationServer

：：：


Mobile ApplicationDialogue & Contents Retrieval

System

1. Agent acquires user’s situations and demands.2. Agent automatically sends profiles to centers. 3. Situations are roughly transmitted to centers

before utterance. 4. User can receive safely and conveniently

desirable information.


InteractiveAgent

InteractiveAgent

InformationCenter

InformationCenter

Delivery

User’s Preferenceshobby, favorite place,dish, music,display, volume, color, etc.

How about Nagoya Interchange ?

One family restaurant found one kilometer to the southward.

Not crowded right now.

Environments & Situationsseason, current location, destination, crews, purpose, weather, traffic conditions, etc.

Demands & Statuswhat user wants to know, what user wants to do,physical conditions, etc.

ProfilesProfilesDataset


Query &Information


Are there any restaurants nearby the East Park ?

Mobile: Dialogue & Contents Retrieval System


Scene Retrieval: Sports Events

Ricoh Co. Ltd., Japan

•Aerobatic Movies

•Motegi, Japan 2000

•Search Queries

•Pilot, Figures,

•Aircraft

•URL

•http://ab-movie.ricoh.co.jp


Organisationsnewly interested in MPEG-7

(25th/January – 25th/February 2001)


ARKIVE

www.arkive.org.uk

An initiative of The Wildscreen Trust, ARKive will be the world's electronic archive of photographs, moving images and sounds of endangered species and habitats. It will relate closely to the work of the World Conservation Monitoring Centre (WCMC).


DLF

The Digital Library Federation (DLF) is a consortium of research libraries that are transforming themselves and their institutional roles by exploiting network and digital technologies.

http://www.clir.org/diglib/

•Developing and applying appropriate architectures, technologies, systems, and tools;

•Developing sustainable, scaleable, and useful digital collections and services;

•Developing support and other services that enable the digital library to respond to its users' information requirements;

•Gaining experience preserving digital information; and identifying and promoting those standards and practices that enable the digital library cost effectively to develop and maintain its collections and services;

•Gaining a better understanding of the digital library's institutional roles, responsibilities, and potential developmental trajectories


3D-MURALE3D Measurement & Virtual Reconstruction of

Ancient Lost Worlds of Europe

www.brunel.ac.uk/project/murale

EU: IST Project

Create MPEG-7 Database

Sagalassos, Turkey


MPEG-7: (Mitsubishi, Japan)

Visual Access Technologies with MPEG-4 and MPEG-7

"contents search and retrieval" employing MPEG-4 (for contents compression) and MPEG-7 (for contents access).

Kohtaro Asai ([email protected])


MPEG-7 Applications (NEC, Japan)

NEC has developed a real-time video identification system based on MPEG-7 descriptions. This system monitors broadcast TV programs and identifies its contents through more than 50,000 different video segments in real-time. The system requires only one PC and software. This system is especially useful for survey applications of video clips like TV commercial programs, music video clips, etc. Furthermore, it enables home video server applications to access to a desired program and information.

[email protected]


MPEG-7: NECReal-time Video Identification System


MPEG-7: Emblaze

MPEG-4 and MPEG-7

www.emblaze.com


MPEG-7: NTTDocomoCustomized Content Delivery for Mobile Users In the next-generation wireless communication services, multimedia content delivery should be one of the most attractive applications for both mobile users and the content providers. Due to the limited time to play the content at mobile client and huge amount of multimedia content data, it is essential to support the functionality of flexible content summarization depending on the user’s situation. In this application scenario, push technology is used for sending summary content to the mobile users to provide pull access trigger for rich content. Summary content contains an instance of MPEG-7(content structure or pre-defined summaries) and media data(e.g. key frames, annotation) which enables mobile user to look at the overview of the whole content quickly. The information for pull access can be generated or selected using delivered MPEG-7 instance and user preferences input to the client.

Contact: Shun-ichi Sekiguchi,

Email: [email protected]


MPEG-7 Applications - next

Views on next stepsDigital (Multimedia) LibrariansKey Partners Needed

Search EnginesContent Providers

Mobile Search & Retrieval


InktomiSearch 4.0CCE & Dublin Core

CCE: Content Classification Engine

Inktomi Search CCE (formerly known as Ultraseek Server CCE) helps administrators to establish a system of classification, and to then create a taxonomy.

Example:http://www.inktomi.com

University of Minnesota: Dept. of Natural Resources


Dublin Core & MPEG-71 Title A name given to the

resourceCreationInformation/Creation/Title[@type="original"]

2 Creator An entity primarily responsible for making the content of the resource

CreationInformation/Creation/Creator

3 Subject The topic of the content of the resource

CreationInformation/Classification/Subject

4 Description An account of the content of the resource

Creation/Abstract ContentDescription/Creation/Abstract

5 Publisher An entity responsible for making the resource available

CreationInformation/Creation/Creator [Role/@term="Publisher"]

6 Contributor An entity responsible for making contributions to the content of the resource

CreationInformation/Creation/Creator [Role/@term="Contributor"]

7 Date A date associated with an event in the life cycle of the resource

CreationInformation/Creation/CreationCoordinates/CreationDate

8 Type The nature or genre of the content of the resource

CreationInformation/Classification/Genre


Dublin Core & MPEG-79 Format File format or mime type

[MPEG-1, QuickTime, RealVideo…)

MediaInformation/MediaProfile/MediaFormat/FileFormat

10 Identifier An unambiguous reference to the resource within a given context

MediaInformation/MediaProfile/MediaInstance/InstanceIdentifier

11 Source A Reference to a resource from which the present resource is derived

MediaInformation/MediaIdentification/EntityIdentifier(the “reality”)

12 Language A language of the intellectual content of the resource

CreationInformation/Classifcation/Language

13 Relation A reference to a related resource

CreationInformation/RelatedMaterial/MediaLocator

14 Coverage The extent or scope of the content of the resource

CreationInformation/Creation/Abstract orCreationInformation/Creation/CreationCoordinates/CreationLocation or CreationInformation/Classification/TargetCountry

15 Rights Information about rights held in and over the resource

UsageInformation/Rights/RightsID (Currently, RightsID is simply a UniqueIDType. Further identification of the rights holder awaits MPEG-7 IPMP development.)


Content Provider ExamplesWhole web coverage

Super-fresh mp3 index

Worlds largest full length music video collection

6200+ Videos from leading artists

Broad range of video content

Rich video data

Web's premier source of legitimate MP3s

Only signed artists from leading labels


MPEG-7 Applications - Summary

MPEG-7 StandardMultimedia Content Description

Structural and Semantic

Key Partner Types NeededContent Providers, Search Engines

Key App - Mobile Search & Retrieval


Contacts in MPEG-7

For more information about MPEG-7 http://www.cselt.it/mpeg

MPEG-7 Industry Focus Group http://www.mpeg-7.com

MPEG-7 Awareness EventSingapore, March 10th, 2001


Acknowledgments

The author is extremely grateful to the MPEG-7 developers, teams and companies that provided him with information on their MPEG-7 application development and activities. The author is also thankful to the support of Digital Garage Inc., his employer, for allowing him the time to survey and compile the information about current MPEG-7 applications being created throughout the world.


Thank you!!

Dr. Willard Miller

Dr. B.S. Manjunath

Michelle Glubke

02/mar/2001 mpeg-7 applications: multimedia content retrieval, ima workshop 6b, university of...

Documents

ima workshop

content descriptionmpeg

codes content

content available

applicationsabout mpeg

survey of mpeg

applications promotion

content analysis d