enabling integrated cross-media entertainment services over heterogeneous networks

Enabling integrated cross-media entertainment services over heterogeneous networks

Introduction:Reasoning, business aspects, involved processes

Dr. Kostas KarpouzisDr. Kostas KarpouzisImage, Video and Multimedia Systems Lab, National Technical University of AthensImage, Video and Multimedia Systems Lab, National Technical University of AthensInstitute of Communication and Computer SystemsInstitute of Communication and Computer Systems

The current scene…

Digital Content A valuable asset nowadays Associated with a variety of emerging and novel distributed

multimedia services Big revenue bringers: entertainment services

Convergence of media and technologies From single, distinct ones to multi-disciplinary ones

Allowing advanced service provisioning and consumption

The current scene…

The plethora of terminal devices available for networking access lead to immense market opportunities for content and service providers

The emergence of advanced network infrastructures enables fast, efficient and reliable end-to-end transmission of huge multimedia content

Media providers faced with two choices: select the most appropriate device for their content and tailor

the consumer experience and business models to take advantage of this environment

invest in multiple devices, access networks and content formats

the current trend… enables transparent access to services, thus more subscribers

Why entertainment..?

Entertainment and leisure market is an attractive commercial target for many business groups Data providers, telecommunication companies,

broadcasters, online publishers, device producers It further allows increasing revenue streams

A never-ending consumer interest Online entertainment and leisure market is a

small percentage of the overall it is expected to become the major boost of this

business sector

The underlying concepts…

Cross-media world: what does it mean? Publish/consume content to a variety of terminals

fixed or mobile, accessing heterogeneous networks Actually refers to:

Create once, Publish everywhere (COPE paradigm) At content/service providers, mediators, network operators

Universal access to content (UMA paradigm) From content/service providers to end-users

Heterogeneous networks Diverse core and access networks

Satellite, wireless, mobile, cable, ISDN, … Different QoS management

User-oriented Services driven by users’ real needs and preferences

So…

How can the same content be transmitted over different channels and consumed at various terminals without costly re-processing and re-production under specific QoS requirements dictated both by content/service providers and end-users?

The business implications

Content/service providers, network operators, content/service mediators increase market competitiveness and advantage establish new user-centered business opportunities

definition of new business models increase subscribers open up new revenue streams be technologically advanced and thus competitive

Consumers Access to any type of entertainment service wherever they

are, using any type of terminal Services tailored to their needs and preferences

The underlying value chain

Digital Rights Management

Content / ServiceCreation

Content/ Service

Mediation & Management

ContentDistribution& Delivery

PersonalizedConsumption

The technological requirements

Enable Create once, publish everywhere Universal multimedia access Adapting technology to people & people being the

centre i.e., primarily

Context awareness, content adaptation scalable coding, automatic transcoding use of standardized metadata descriptions & profiling

(MPEG-7, MPEG-21) Personalization

The technological requirements

Other requirements involved At the service provider side

Service management, discovery, integration Smart, self-adaptive content authoring & aggregation Context modeling

At the mediation/network layer Monitoring End-to-end QoS provisioning

At the end user layer interactivity mobility transparent access to services

In all, digital rights management and protection

Abstract end-to-end system design

PC PDA Mobile Digital TV

Personalized

Consumption

Content Distribution & Delivery

Re-production WorkflowContent

Aggregation

Service Managemen

t

Identification

Registration

Discovery

Integration

Cross Media

Provisioning

Context Awareness

Content Adaptation

Context Modeling

Personalization

User/Terminal Profile

QoS Managemen

t

Content and ServiceMediation

Content description / representation -

metadata

Content Authorin

g

Scalable coding

Content and ServiceCreation

Dig

ital R

ights

Managem

ent

Service/Content

Mediators

Content Providers

Service Providers

Content Transcoding

Users

Transmission


The concepts…:Context Awareness, Content Adaptation

Context Awareness

Context includes any information that characterizes an entity’s situation

Entity: person, place, object relevant to an interaction between a user and an application

is a function of time and environment environment is a function of users, services, resources and other entities in

environment Context-aware system

when using contexts to provide relevant information or services to the user Relevance depends on user’s task

Context-aware applications must detect, interpret and respond to contexts

Contextualization enables efficient content adaptable services and systems important component of the ubiquitous/pervasive paradigm aids in understanding the environment sufficiently

Context Awareness

Context-aware computing involves collection of context information dynamic program behavior dictated by knowledge environment

This is enabled by: Use of profiles

Environment (network, terminal, service) User (preferences, location, …) represented in standardised format (MPEG-7, MPEG-21)

since a common representation format is required to allow for interoperability in a distributed world, over heterogeneous

networks and a variety of terminals Monitoring mechanisms gathering context information Decision engines that use context information as input to adapt

content based on environment status Usually processed at service mediation and adaptation layers, but

gathered stored and interpreted at different parts of the system such as end-user terminal devices or access networks

So… a necessary step before content adaptation

Context Awareness

A context profile should be Structured

Aids in effectively filtering relevant information Interchangeable

Among different system components Mechanisms for transferring sub-trees (and not entire profiles) desirable

To compensate for big network delays Composable/decomposable

To allow maintenance in a distributed way To grasp changes in environment

Uniform Eases interpretation

Extensible To allow for future requirements/advances

Standardized To allow interoperable exchange among different entities in a distributed

environment

Context Awareness

Context-aware infrastructure Physical and logical sensors (software entity, network

components, software agents) used to collect context information related to presence, location, identity and profile of users and services/content

Typical context use Locate services and users Call up services according to user behavior Provide information for service composition Facilitate ad hoc communication mechanism between users Adaptation of QoS to changes in environment as a result of user

and service mobility Passive

context gathering and representation Active

smart context information delivery

Content Adaptation

Why adapt contents? Most contents for viewing are for the larger screens Creating multiple versions a burden Even if you don’t mind, there are just too many

possible devices Different users want different things Having one, original version is easier to manage

Content adaptation is about generating any content version from one single original version

Create once, Publish everywhere (COPE) Universal Multimedia Access (UMA)

Content Adaptation

Pre-adaptation Keeping just the original version (any other version is runtime-

generated) could be slow Pre-adaptation

to create all possible versions (at creation/production phase), and do static “selection” at runtime, or

to create just a few essential versions, and do dynamic adaptation – hence the “balance”

Dynamic adaptation On-the-fly adaptation Where it happens

Either at end systems (end-to-end service) Or at intermediate network nodes (active service)

At specially designed mediators

Content Adaptation

Major processes involvedAccess to and management of context

information and content/service metadataDecision engine to negotiate further actionsAdaptation by

Layered coding (scalable bitstreams)Transcoding

Content Adaptation

A Content AdaptationSystem

Mediation Layer

Content Adaptation

Use of context profileInformation and contentInstance metadatato decide upon transcodingstrategies


The concepts…:

Scalable Coding/Transcoding

Transcoding introduced

Video transcoding: converting a previously compressed video signal into another onePossibly different bit rate, frame rate, frame

size or compression standardInitially proposed for compressed video

bit rate scalingSun et al., 1996

Transcoding introduced (2)

Also used for:Video frame size conversion

spatial resolutionVideo frame rate conversion

temporal resolutionBit rate adaptationMultipoint video combiningError resilience

Transcoding for heterogeneous nets

Diversity of channel capacitiesChannel capacity of the outgoing

channel may be less than that of the incoming channelOr may change over time

Distribution to users with different connections target transmission channel conditions are

generally unknown during encoding

Major transcoder architectures

Cascaded Pixel Domain TranscoderKeesman et al., 1996Close-loop transcoder connects a standard

decoder and a standard encoder togetherMost straightforward method

Major transcoder architectures (2)

Open-Loop TranscoderSun et al., 1996 Input video bitstream is first partially

decoded to the DCT coefficient levelBit rate is scaled down by cutting higher

frequency coefficients or by requantizing all coefficients with a larger quantization step size

Major transcoder architectures (3)

DCT domain transcoder (DDT)Zhu et al., 1999 requantization of the DCT coefficientserror is stored in a buffer and is fed back to

the requantizer to correct the requantization error introduced in previous frames

Simplifies architecture of OLT by reusing motion vectors

Merges two motion compensation loops in the CPDT into one

Video rate control

Accurate and dynamic control of the output bit rate according to the channel bandwidth

Video rate control (2)

Two different steps:Frame-Layer Rate Control: allocate target

bits for each frame according to image complexities, buffer occupancy, or a given channel bit rate

Macroblock-Layer Bit Allocation: derive the actual quantization parameter for each picture Macroblock and make the number of produced bits meet the bit target

Frame-Layer Rate Control

Bit budget for every frame has to be determined considering the channel bit rate and buffer occupancy

Assumption for constant bit rate (CBR) Internet transmission cannot provide a

guaranteed constant bit rate channelwireless channels: high bit error rate and

variable effective channel bit rate

Frame-Layer Rate Control (2)

Theoretically, compression and error protection can be performed separately and sequentiallyShannon’s separation theorem

In practical video communication systems source coding and channel coding are combined together

Wireless video communication

Wire line channels: signal strength is relatively constant reception errors mainly due to additive noise

Wireless channels: reception errors mostly due to time-varying signal strengthmulti-path propagation from local scatterserror bursts

Wireless video transmission

Delay constraintsChannel errors hurt perceptual quality of

video at decoderEvident in standard coders

variable length codingpredictive coding compressione.g., MPEG or H.263

Challenge transmission error

At the encoder side: layered codingmultiple description codingerror resilience entropy coding

Increase robustness of video stream against channel errors

Challenge transmission error (2)

At the decoder side: Error concealment techniques recover lost

information without relying on information from the encoder

Channel coding techniques FEC codes may cause unnecessary overhead and

bandwidth waste when channel is in good state ARQ error control requires retransmission only

during periods of poor channel conditions

Abstract architecture

Delay = Processing Delay + Transmission Delay + Buffer Delay

Adaptive transcoding

Frame layer rate control based on video content frame typesvideo source coding ratescene changes

Frame types and source video

MPEG-1, -2 and H.26x use motion compensation

Motion compensation reduces temporal redundancy between successive frames

MPEG-1 and MPEG-2

In MPEG, a GOP (Group of Pictures) contains one I-frame (intra) and several P- or B-frames (inter) in a certain pattern

I-frame: no motion compensation motion compensation of following frames depends

on I-frame quality P-frames use the previous I- B-frames use both the previous and

successive I- or P-frames as references for motion compensation

H.263

Focus on low bit rateMore popular for wireless applications

Fixed quantization parameterNear constant visual objective qualityVarying bit rate

scene content different frame types

Constant rate quality degradation

H.263 (2)

Source bit rate can be recorded during encoding and saved as side informationunified quantization parameter in same type

of frames generated frame size indicates scene variations and motion activities

Thus, video content can be measured by the source bit rateUsed to calculate bit budget for every frame

Scene changes

Scene changes represent distinctive differences between adjacent video framesRapid motion of moving objectsChanges to different visual content

Information obtained from previously coded frames is no longer useful

Scene changes (2)

When scene changes happen, the first frame after scene change should be transcoded in high quality to prevent quality degradation after scene change

Scene change detectionMost MBs in the anchor frame will be

encoded as INTRA blocks INTRA mode MBs percentage can be used

to detect scene changes

Adaptive frame layer rate control

Relaxed requirement for end-to-end delay

Allowed initial startup delaydecoder buffer smoothes delay jitters

H.263 determines frame budget by buffer occupancy and channel bandwidthnearly constant bit budget similar end-to-

end delay for each frame

Adaptive frame layer rate control (2)

Adaptive methods consider scene changes, frame types and source coding ratee.g., I-frames are transcoded as I-frames,

with unified quantization parameter used for every MB, while the bit budget for B-frames will be half of that for P-frames

Scene change detected anchor frame is transcoded as an I-frame

A practical scenario

Offline encoding Initial presentation is prepared according to

pre-defined scenariosUses pre- created static content

Online transcoding In charge of transmission of initial

presentationGeneration, encoding and transmission of

real-time interactions


The technologies…:Metadata & Profiling (environment, user, terminal)

The need for standardised metadata (MPEG-7)

Today people cannot easily create, find, edit, share, and reuse media

Computers don’t understand media content Media is opaque and data rich We lack structured representations

Without content representation (metadata), manipulating digital media will remain like word-processing with bitmaps

Need standardized metadata framework Designed for video and rich media data Human and machine readable and writable Standardized and scalable Integrated into media capture, archiving, editing, distribution, and

reuse

MPEG-7

MPEG-7 Describing the multimedia content data that supports some

degree of interpretation of the information’s meaning, which can be passed onto, or accessed by, a device or a computer code

Motivation Create standardized multimedia description framework Enable content-based access to and processing of

multimedia information on the basis of descriptions of multimedia content and structure (metadata)

Support range of abstraction levels for metadata from low-level signal characteristics to high-level semantic information

MPEG-7 Query Examples

Play a few notes on a keyboard and retrieve a list of musical pieces similar to the required tune, or images matching the notes in a certain way, e.g., in terms of emotions

Draw a few lines on a screen and find a set of images containing similar graphics, logos, ideograms,...

Define objects, including color patches or textures and retrieve examples among which you select the interesting objects to compose your design

On a given set of multimedia objects, describe movements and relations between objects and so search for animations fulfilling the described temporal and spatial relations

Describe actions and get a list of scenarios containing such actions

Using an excerpt of Pavarotti’s voice, obtaining a list of Pavarotti’s records, video clips where Pavarotti is singing and photographic material portraying Pavarotti

MPEG-7 Sample Application Areas

Architecture, real estate, and interior design (e.g., searching for ideas)

Broadcast media selection (e.g., radio channel, TV channel)

Cultural services (history museums, art galleries, etc.)

Digital libraries (e.g., image catalogue, musical dictionary,

bio-medical imaging catalogues, film, video and radio archives)

E-Commerce (e.g., personalized advertising, on-line

catalogues, directories of e-shops) Education

(e.g., repositories of multimedia courses, multimedia search for support material)

Home Entertainment (e.g., systems for the management of

personal multimedia collections, including manipulation of content, e.g. home video editing, searching a game, karaoke)

Investigation services (e.g., human characteristics recognition,

forensics) Journalism

(e.g. searching speeches of a certain politician using his name, his voice or his face)

Multimedia directory services (e.g. yellow pages, Tourist information,

Geographical information systems) Multimedia editing

(e.g., personalized electronic news service, media authoring)

Remote sensing (e.g., cartography, ecology, natural

resources management) Shopping

(e.g., searching for clothes that you like) Social

(e.g. dating services) Surveillance

(e.g., traffic control, surface transportation, non-destructive testing in hostile environments)

MPEG-7 Scope

To define the description framework

Scope of MPEGScope of MPEG--77Scope of MPEGScope of MPEG--77

Descriptionconsumption

DescriptionconsumptionDescriptionDescriptionDescription

generation

Descriptiongeneration

MPEG-7 Metadata Framework

Data “multimedia information that will be described using MPEG-7, regardless

of storage, coding, display, transmission, medium, or technology.” Feature

“a distinctive characteristic of the data [that] signifies something to somebody.”

Descriptor “A representation of a Feature. A Descriptor defines the syntax and the

semantics of the Feature representation.” Description Scheme

“The structure and semantics of the relationships between its components, which may be both Descriptors and Description Schemes.”

Description Definition Language (XML Schema) “A language that allows the creation of new Description Schemes, and,

possibly, new Descriptors. It also allows the extension and modification of existing Description Schemes.”

MPEG-7 Framework

TagsTags

<scene id=1><time> ....<camera>..

<annotation</scene>

InstantiationInstantiation

TagsTags

<scene id=1><time> ....<camera>..

<annotation</scene>

InstantiationInstantiation

Descriptors:Descriptors:(Syntax & semantic(Syntax & semanticof feature representation)of feature representation)

D7

D2

D5

D6D4

D1

D9

D8

D10

101011 0

Encoding&

Delivery

101011 0

Encoding&

Delivery

D3

LanguageDescription Definition extensionextension

DefinitionDefinitionLanguage

Description Definition extensionextension

DefinitionDefinition

Description SchemesDescription Schemes

D1

D3D2

D5D4D6

DS2

DS3

DS1

DS4StructuringStructuring


D1

D3D2

D5D4D6

DS2

DS3

DS1

DS4


D1

D3D2

D5D4D6

DS2

DS3

DS1

DS4StructuringStructuring

MPEG-7 Standard Parts

1. MPEG-7 Systems The binary format for encoding MPEG-7 descriptions and the terminal architecture

2. MPEG-7 Description Definition Language The language for defining the syntax of the MPEG-7 Description Tools and for

defining new Description Schemes3. MPEG-7 Visual

The Description Tools dealing with (only) Visual descriptions4. MPEG-7 Audio

The Description Tools dealing with (only) Audio descriptions5. MPEG-7 Multimedia Description Schemes

The Description Tools dealing with generic features and multimedia descriptions 6. MPEG-7 Reference Software

A software implementation of relevant parts of the MPEG-7 Standard with normative status

7. MPEG-7 Conformance Testing Guidelines and procedures for testing conformance of MPEG-7 implementations

8. MPEG-7 Extraction and Use of Descriptions Informative material (in the form of a Technical Report) about the extraction and use

of some of the Description Tools (under development)

MPEG-7 Description Tools

MPEG-7 Top Level Hierarchy

MPEG-7 Still Image Description

Referencing Temporal Media

MPEG-7 Video Segments Example

MPEG-7 Segment Relationship Graph

MPEG-7 Conceptual Description

MPEG-7 Summaries

MPEG-7 Collections

MPEG-7 Application Framework

MPEG-7 Applications Today

IBM MPEG-7 Annotation Tool Assists in annotating video sequences with MPEG-

7 metadata Ricoh MPEG-7 MovieTool

A tool for creating video content descriptions conforming to MPEG-7 syntax interactively

Canon MPEG-7 Speech Recognition engine Web site allows you to create an MPEG-7 Audio

“SpokenContent” description file from an audio file in “wav” format

IBM MPEG-7 Annotation Tool

IBM MPEG-7 Annotation Tool

The IBM MPEG-7 Annotation Tool assists in annotating video sequences with MPEG-7 metadata Each shot in the video sequence can be annotated with static

scene descriptions, key object descriptions, event descriptions, and other lexicon sets

The annotated descriptions are associated with each video shot and are stored as MPEG-7 descriptions in an XML file

Can also open MPEG-7 files in order to display the annotations for the corresponding video sequence

Customized lexicons can be created, saved, downloaded, and updated

Ricoh MovieTool Creates an MPEG-7 description by

loading video data Provides visual clues to aid the

user in creating the structure of the video

Automatically reflects the structure in the MPEG-7 descriptions

Visually shows the relationship between the structure and MPEG-7 descriptions

Presents candidate tags to help choose appropriate MPEG-7 tags

Checks the validation of the MPEG-7 descriptions in accordance with MPEG-7 schema

Can describe all metadata defined in MPEG-7

Is able to reflect any future changes and extensions made to MPEG-7 schema

Canon MPEG-7 ASR Tool

MPEG-7 Resources

http://www.chiariglione.org/mpeg/index.htm

http://www.mpegif.org/mpegif/index.php

http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0471486787.html

The need for standardized profiles (MPEG-21)

Dictated by the emerging demands on Universal multimedia access Ubiquitous and pervasive computing Heterogeneity of networks and end-user terminals Use of content and services anywhere, at any time

To allow for uniform, exchangeable and interoperable definition of context profiles

To allow for efficient negotiation mechanisms for context retrieval and content adaptation anywhere in the content production, transmission and consumption chain

MPEG-21

A normative open framework for multimedia delivery and consumption for use by all the players in the delivery and consumption chain

Based on two essential concepts: the definition of a fundamental unit of distribution and

transaction (the Digital Item) Digital Items: the “what” of the Multimedia Framework

a video collection, a music album the concept of Users interacting with Digital Items

Users: the “who” of the Multimedia Framework

Goal define the technology needed to support Users to exchange,

access, consume, trade and otherwise manipulate Digital Items in an efficient, transparent and interoperable way

MPEG-21 scope

User A User BTransaction/Use/Relationship

Digital ItemAuthorization/Value Exchange

Term inals &Networks

ContentManagem ent

and usage

I ntellectualProperty

m anagem entand

Protection

Digital ItemIdentification

and Description

MultimediaContent

Representation

Digital I temDeclaration

Exam ples:•U nique Identifiers•C ontent D escriptors

Exam ples:•Encryption•Authentication•W aterm arking

Exam ples:•R esource Abstraction•R esource M gt. (Q oS)

Exam ples:•“C onta iner”•“Item ”•“R esource”

Exam ples:•S torage M gt.•C ontent Personalization

Exam ples:•N atura l and Synthetic•Scalab ility

Event Reporting

MPEG-21 standard parts

MPEG-21 Digital Item Declaration To describe a set of abstract terms and concepts to form a useful model for

defining Digital Items MPEG-21 Digital Item Identification

To uniquely identify Digital Items and link with related information such as descriptive metadata

MPEG-21 Intellectual Property Management and Protection To define an interoperable framework for Intellectual Property Management

and Protection MPEG-21Rights Expression Language

To define machine-readable language that can declare rights and permissions using the terms of the Rights Data Dictionary

MPEG-21 Rights Data Dictionary MPEG-21 Digital Item Adaptation

To achieve interoperable transparent access to (distributed) advanced multimedia content by shielding Users from network and terminal installation, management and implementation issues

Definition of environmental profiles

MPEG-21 DIA

Conceptual architecture a Digital Item is subject to a resource adaptation engine and a description

adaptation engine producing the adapted Digital Item

Within scope Descriptions and format-independent mechanisms in terms of resource

adaptation, descriptor adaptation, and/or Quality of Service management Not adaptation engines

Digital ItemAdapted

Digital Item

DIA Tools

Digital ItemAdaptation Engine

ResourceAdaptation Engine

DescriptionAdaptation Engine

Scope of standardization

Digital ItemAdapted

Digital Item

DIA Tools






MPEG-21 DIA Tools

Three major categories

Usage Environment Description Tools

• User Characteristics• Terminal Capabilities• Network Characteristics• Natural Environment Characteristics

Digital Item Resource Adaptation Tools

• Bitstream Syntax Description• Terminal and Network QoS• Bitstream Syntax Description Link• Metadata Adaptability

Digital Item Declaration Adaptation Tools

• Session Mobility• DIA Configuration

Usage Environment Description Tools

• User Characteristics• Terminal Capabilities• Network Characteristics• Natural Environment Characteristics

Digital Item Resource Adaptation Tools

• Bitstream Syntax Description• Terminal and Network QoS• Bitstream Syntax Description Link• Metadata Adaptability


• Session Mobility• DIA Configuration

MPEG-21 DIA Tools & Other MPEG-21 Parts

MPEG-21 DID


MPEG-21 IPMP/REL

Resource

MPEG-21 DII



Descriptor

MPEG-21 DIA Tools

MPEG-21 DID’

MPEG-21 IPMP/REL’

MPEG-21 DII’

MPEG-21 DID

MPEG-21 IPMP/REL

MPEG-21 DII

MPEG-21 DIA Tools

Usage Environment Description ToolsDigital Item Resource Adaptation Tools


DI-1 DI-3

DI-2

Descriptor

Resource’

Descriptor’

MPEG-21 DIA Tools

MPEG-21 DID


MPEG-21 IPMP/REL

Resource

MPEG-21 DII



Descriptor

MPEG-21 DIA Tools

MPEG-21 DID’

MPEG-21 IPMP/REL’

MPEG-21 DII’

MPEG-21 DID

MPEG-21 IPMP/REL

MPEG-21 DII

MPEG-21 DIA Tools

Usage Environment Description ToolsDigital Item Resource Adaptation Tools


DI-1 DI-3

DI-2

Descriptor

Resource’

Descriptor’

MPEG-21 DIA Tools


The concepts…:Personalization

Why personalize?

Multitude of concurrent programsDuring the upcoming Olympic Games, more

than 5 events may be broadcast during prime-time hours

Athletics offers many events in parallelDuring weekends, many sports events and

movies are broadcast in parallel

Why personalize? (2)

Diversity in presentation optionsPeripheral information, e.g. sports statistics,

critics’ reviews for movies, other results, etc.Visual enhancements, e.g. arrows pointing

distance to goal line, lines showing WR time, etc.

Different viewing options, e.g. multiple viewing angles, virtual replays, etc.

So, personalization is…

Choosing what to view and whenChoice of a particular event or specific

portions, e.g. the final 20 minsChoice of a particular movie or movies

belonging to a specific genreChoice of live or recorded viewing

and…

Choosing how to view the programChoice of specific enhancementsChoice of available interaction, e.g. betting

opportunities during a football matchAlerts when other programs of interest are

available

Is that all?

Adapting TV to individual viewers is a topic in itselfO’Sullivan et al., 2004

Watching TV, especially sports and movies, tends to be a social activity

Strategies for combining individual user models to adapt to groupsSocial Choice Theory?

Group theory

In contrast to the use of PCs, television viewing is largely a family or social activity (Barwise and Ehrenberg, 1988)

Unfortunately, statistics do not include data on the average number of people watching TV together and who watches with whom

Culturally dependent?

Group theory (2)

Groups can be heterogeneousage, gender, and personality may influence

program choiceswatching TV together is a major activity

shared between parents and childrenyoung people like to watch TV with friends

Challenge: to adapt to a group of viewers, in such a way that each individual enjoys the broadcast

Strategies for Combining User Models

Modeling of user preferencesDirectly, via explicit profile definition Indirectly, via observation and inference

Group modelingMUSICFX, McCarty and Anagnost, 1998POLYLENS, O’Conner et al., 2001 INTRIGUE, Ardissono et al., 2002

Group modeling

MUSICFX is used in a fitness center to select background music to suit a group of people working out at any given time

POLYLENS recommends movies based on group tastes inferred from ratings and social filtering

INTRIGUE recommends places to visit for tourist groups taking into account characteristics of subgroups within that group, e.g. children and disabled

However…

Limited actual evaluation no indication of how effective these group

modeling strategies really areApplication domains differ from TV

viewingpeople usually see one movie per eveningmusic stations can play foreverno need to select a balanced groupgroup of

items

Social choice

Also called group decision makinggroup decision makingDeciding what is best for a group, given the

opinions of individualsStudied extensively in economics, politics,

sociology, and even mathConstruction of a social welfare functionsocial welfare function

Pattanaik, 1971Taylor, 1995

Social choice (2)

Meta-Search Results from multiple search engines are combined

into one list - rank aggregation (Dwork et al., 2001)

Database middleware Objects with numerical values for multiple fields

have to be ordered (Fagin et al., 2003)

Collaborative Filtering Preferences of a group of individuals have to be

aggregated to predict a third party preference (Wallace et al., 2003)

A hands-on scenario

Choose a set of items available in an EPG e.g. video clips, TV series, movies, sports events,

etc.

Group of viewers Receiver knows who the viewers are Has preference ratings for each of them

Problem: which items to show, given time for a certain number but not all of them?

Assumptions are fine, but…

How will the receiver know who is watching? IR card or login procedure registers viewers Probabilistic mechanism: known probability of a

viewer watching at a particular time of day

How will user preferences be determined? Social, and content-based filtering can be used Inferences from group viewing actions when

user watches alone? An individual’s ratings may depend on the group

they are in

Assumptions are fine, but… (2)

How to deal with differences in rating tendencies? Some people either “really hate” or “really love” a

program Others may never be very positive and never really

negative

How to deal with uncertainty? How to deal with changing groups?

Assumptions are fine, but… (3)

How to deal with users feeling strongly about their preferences? All users are equal? Or some are “more equal than others”?

How to present recommendations to the group? Receiver decides which items to show? Receiver gives viewers a choice? List of recommendations?

Example strategies

Plurality VotingEach voter votes for their most preferred

alternativeSequence of alternatives repetitive

method

Example strategies (2)

Utilitarian StrategyUtility values express expected happinessAdditive/Average strategy (Masthoff, 2002)

Weighted form in multi-agent systemsMultiplicative

Individual viewers might always lose out, because their opinion is a minority view

In a small group, individual opinions will have a large impact on the average/product


Borda CountBorda, 1781 (!)Points awarded according to position

preference listbottom alternative gets zero points, the next

one up one point, etc.Multiple alternatives with the same rating?


Copeland RuleForm of majority voting - orders alternatives

according to Copeland indexNumber of times an alternative beats other

alternatives minus the number of times it loses to other alternatives

Approval VotingVoters allowed to vote for as many

alternatives as they wishPromotes election of moderate alternatives


Least Misery StrategyNew list of ratings with minimum of

individual ratingsHigher ranked items get selected soonerA group is as happy as its least happy

memberMinority opinions dictate the group

If everybody wants to see something, but one person does not like it, then it will never be seen.


Most Pleasure StrategyNew list of ratings with maximum of

individual ratingsHigher ranked items get selected sooner

Average Without Misery StrategyNew list of ratings with average of individual

ratingsSkip items that score below a certain

threshold for individuals


Fairness StrategySelect top items from all individualsEqual rating take into account others’

opinions It’s not so bad to watch something you hate,

as long as you get to watch the things you really love as well


Most Respected Person StrategyAlso called “Dictatorship”Use ratings of most respected personSimilar ratings use others’ ratingsTV remote control is often operated by the

oldest male presentAdults may dominate children, visitors may have

more influence than inhabitantsBirthdays or illness can influence who is ‘the

most respected’ person

The issue of satisfaction

A major difference between strategies is the emphasis placed on individual satisfactionavoidance of miserysatisfaction of group majority

Another major difference: use only relative item positions in preference lists, or also strengths of these preferences?

A basic satisfaction function

Input: a set of ratings for any sequence of clips

Output: a real numbere.g. summation of an individual’s ratings of

concerned clipshow do these ratings compare to those of

unselected clips? normalizationLinearity, misery, order, group solidarity?

The holy grail

Is there a strategy that keeps everyone happy?

Multiplicative Utilitarian seems the most promising strategyLength of clips may be importantDiscussion on the basis of the clipsSpecify and definitely include favorite clips

Presenting a Sequence

The receiver will also need to decide in which order to show itemsGroup List RankingChanging OrderChanging Ratings

Open questions

Do people adjust ratings?Having watched the first item influences

ratings for the other items?Mood? Thematic relatedness?Viewer’s satisfaction Interaction between these factors?

enabling integrated cross-media entertainment services over heterogeneous networks

Documents

business groupsdata

business aspects

content andtailor

type of entertainment

publishconsume content

subscriberswhy entertainment

access networkssatellite

leisure market