context-aware middleware for multimedia services in heterogeneous networks

Context-Aware Middleware for Multimedia

Services in Heterogeneous Networks

Liang Zhou, Naixue Xiong, Lei Shu, Athanasios Vasilakos, Sang-Soo Yeo

Abstract

An important challenge for supporting multimedia applications in heterogeneous networks is

the heterogeneity of fixed and mobile access networks. In this work, we design a new and

efficient context-aware middleware for facilitating diverse multimedia services in heterogeneous

networks environment. Firstly, we present an adaptive service provisioning middleware for

handling the heterogeneity of diverse networks and enable service provisioning to mobile users

and professionals anywhere, anytime. Then, a context-aware multimedia middleware framework

is presented based on the proposed adaptive service provisioning framework to support diverse

multimedia services, including, multimedia content filtering, recommendation, adaptation,

aggregation, learning, reasoning, and delivery. To the best of knowledge, this study is the first

one to provide a general heterogeneous multimedia middleware by jointly considering the

characteristics of context-multimedia service and heterogeneous networks.

Index Terms context-aware; heterogeneous networks; middleware; multimedia

L. Zhou is with the UEI, ENSTA-ParisTech, France; N. Xiong is with the Department of Computer Science, Georgia

State University, USA; L. Shu is with the Digital Enterprise Research Institute, the National University of Ireland, Ireland;

A. Vasilakos is with the Department of Computer and Telecommunications Engineering, University of Western Macedonia,

Greece; S.-S. Yeo is with the Division of Computer Engineering, Mokwon University, Korea.

1

Digital Object Indentifier 10.1109/MIS.2009.129 0885-9000/$26.00 © 2009 IEEE

This article has been accepted for publication in IEEE Intelligent Systems but has not yet been fully edited.Some content may change prior to final publication.

I. INTRODUCTION

Recent years have witnessed the increasing efforts towards standardization of

architectures for convergence of heterogeneous access networks. The integration of

heterogeneous networks (e.g., WiMax, WiFi, sensor networks, etc.) has fully become a part

of the 4G network design paradigm [1]. Supporting multimedia applications over

heterogeneous networks has been one of the main fields of research in the networking and

multimedia communities. For example, the IMS (IP Multimedia Subsystems) platform [1]

has defined an overlay architecture for providing multimedia services based on

heterogeneous wireless networks as shown in Fig.1. In addition, the provisioning of

multimedia contents and operations of multimedia devices usually need to be customized

based on changing contexts (i.e., context-aware multimedia services), for example, recording

favorite TV programs of family members, showing suitable content based on a user's social

activities, and presenting content in an appropriate form according to the capabilities of the

display device and network connection.

Fig. 1 An example of multimedia streaming architecture in a heterogeneous wireless network environment.

Context-aware multimedia services have attracted much attention from researchers in

recent years, and several context-aware multimedia systems have been developed in specific

network conditions. However, building context-aware multimedia services in heterogeneous

networks is still complex and time consuming due to heterogeneity in context-aware media

contents and network conditions. Although Tseng et al. [2] propose multimedia middleware

for video transcoding and summarization, they acquire context in an ad hoc network not

heterogeneous network. Zhu et al. [3] develop a multimedia rate allocation framework in

2



heterogeneous networks; however, the context-aware multimedia middleware is not taken

into account. Yu et al. [10] considers context-aware multimedia services in wireless network

framework and proposes a corresponding middleware. To the best of our knowledge, the

current literatures consider the context-aware middleware and the heterogeneous networks

separately and independently. In order to provide satisfying and unified multimedia services

in the context of heterogeneous networks, the above two factors are jointly considered in this

paper.

The rest of this paper is organized as follows. Section II provides the design principles and

requirements of the system. In Section III, we present the middleware infrastructure for

heterogeneous networks. Section IV provides context-aware multimedia service framework

and specifies the interaction between the different services. At last, we conclude this paper in

Section V.

II. DESIGN PRINCIPLES AND REQUIREMENTS

Context awareness of heterogeneous applications has recently been attracting the attention

of many researchers as an interesting and perspective topic for research [5], [6]. Context

awareness is an essential feature of heterogeneous systems, because almost all the ubiquitous

applications utilize context information in their operation. Generally speaking, there are two

technological difficulties and challenges for context-aware multimedia services in

heterogeneous networks:

How to design an adaptive service provisioning middleware for handling the

heterogeneity of diverse networks and enable service provisioning to mobile users and

professionals anywhere, anytime.

How to present a context-aware multimedia middleware framework based on the

proposed adaptive service provisioning framework to support diverse multimedia

services, including, multimedia content filtering, recommendation, adaptation,

aggregation, learning, reasoning, and delivery.

We adopt Dey’s definition of context as “any information that can be used to characterize

the situation of an entity” [10]. An entity is a person, place, or object that is considered

relevant to the interaction between a user and an application, including the user and

application themselves. An example of context information can be a user’s location, time,

user’s profile, local resources of the mobile device, available services, etc.

Context awareness characterizes a system to use context information when it performs its

3



tasks. In the present paper, we use the task-oriented definition of context awareness: “a

context-aware system uses context to provide relevant information and/or services to the

user, where relevancy depends on the users task.” [10]. That is, the main stress is put to the

context that is relevant to the task.

Context awareness involves performing data acquisition from sensors, context recognition

and other tasks necessary to complete before the context can actually be used. Delegating the

data acquisition and context processing tasks to applications makes them almost impossible

to reuse. One solution to such a problem is to decouple the tasks from applications and move

desired functionality to the lower layers. A context-aware middleware has to provide the

applications with the following context-oriented functionality:

Support of a variety of sensor devices,

Support of the distributed nature of context information, because the data comes from

different sources,

Providing for transparent interpretation of applications and abstraction of context

data,

Maintenance of context storage, and

Control of the context data flow.

In addition, multimedia applications set requirements for communication and computation.

Multimedia algorithms need a lot of bandwidth and processing power. Hence, the

middleware has to be capable of:

Using the available bandwidth: the middleware has to use all the different, available

connections and switch to the connection that best fulfills the requirements at the

given moment;

Controlling the place of computation: to cope with limited resources the middleware

needs to decide where the computations should be performed based on the situation at

hand.

Furthermore, the middleware must support various multimedia devices such as video

cameras, microphones, etc. Finally, there are requirements that the middleware has to fulfill

to make the system adaptable:

Triggering of adaptation on a system-wide level,

Support for system-wide adaptation policies, and

Providing a common interface between devices and middleware.

Developing a generic adaptation mechanism suitable for context-aware multimedia

4



service in heterogeneous networks is an excessively demanding task. A summary of the

fundamental requirements would be:

Clear demarcation of the logic implementing the different phases of the entire

adaptation process.

Support for arbitrary complexity. The adaptation mechanism should support elements,

like profiles and algorithms, of arbitrary complexity.

Independence from particular types of profiles and algorithms. The algorithms used for

profile matching should be loaded at runtime and thus should not be a static part of the

adaptation mechanism. That is, any logic pertaining to specific types of profiles and

algorithms should not be hard-coded in the adaptation system.

Interoperability and portability. Ideally, the adaptation system should not only handle

disparate data and algorithms, but also enable its seamless integration in a variety of

environments. System modularity and decoupling from context are vital to the

accomplishment of this goal.

III. MIDDLEWARE INFRASTRUCTURE FOR HETEROGENEOUS NETWORKS

A unified heterogeneous networks platform should target all types of access networks,

varying from picocells, to WLANs, and to current 2G/3G. Various middleware solutions are

available over these networks. We believe that the increasing diversity of device (terminals,

network elements, and application servers) leads to the conclusion that, at least in the near

future, there will not be a single dominant middleware platform sufficient for all devices and

purposes. As a starting point, therefore, we assume that various platforms, e.g., JAIN (Java

APIs for integrated networks), OSA (open service access), Parlay, CAMEL (customized

applications for mobile network enhanced logic), Parlay, and APIs (application

programming interfaces), will be available over these networks.

5



Fig. 2 Middleware infrastructure for heterogeneous networks.

The proposed heterogeneous network middleware platform architecture is shown in Fig. 2.

In order to achieve interoperability in the most open manner, we adopt the Object

Management Group (OMG) Model Driven Architecture (MDA) [8]. MDA goes far beyond

Common Object Request Broker Architecture (CORBA)-based interoperability at the level

of standard component interfaces by placing formal system models at the core of the

interoperability problem. The distinct feature of this approach is that the system definition is

independent of any implementation model, and formal mappings to many possible

implementation technologies (e.g., Java, XML etc.) are provided. Following the MDA

approach, services are described using formal models, initially expressed in a

platform-independent modeling language, such as Unified Modeling Language (UML).

Through MDA tools, these can be instantly mapped onto specific platform technologies,

such as CAMEL or OSA/Parlay. In the context of the heterogeneous networks, services that

can be identified would provide capabilities such as file transfer, voice calls, multiparty

video conferences, and terminal positioning.

The adaptability of the services is based on Adaptive Service Components (ASCs). ASCs

are polymorphic self-adaptive components that are specialized for a particular functionality

or feature and that are able to adapt to external triggers. For example, whenever network

6



layer reservations are violated, the relevant ASCs will be triggered to adapt themselves to

the available network resources, based on rules, scenarios, and Service Level Agreements

(SLAs). In order to achieve this polymorphism, ASCs will follow the disciplines of

meta-modeling [10]. The meta-modeling strategy is ultimately achieved via shared metadata,

while understanding metadata consists of the automated development, publishing,

management, and interpretation of models. The technology provides dynamic system

behavior based on the runtime interpretation of such models. Based on this technology,

ASCs will be highly interoperable, easily extended at runtime, and completely dynamic in

terms of their overall behavioral specifications (i.e., their range of behavior will not be

bound by hard-coded logic). As shown in Fig. 2, ASCs of various adaptive service

provisioning middleware framework have been identified, including location calculation,

session control, mobility, QoS, security, profiling, personalization, and provisioning.

The platform-independent ASCs are subsequently translated to

Network/Platform-Specific Components (NPSCs) by mapping the ASCs models to some

implementation the language or platform (e.g., Java) using formal rules. Development and

integration may be facilitated through common platform services and programming models.

For example, J2EE enables implementation and deployment of component-based distributed

applications, and the Java community is developing pure Java programming models in the

form of J2EE standard APIs. Examples of NPSCs may include components to interface OSA,

Parlay, VHE (virtual home environment), GMLC (gateway mobile location center), HLR

(home location register), GPS, SIP, MPLS (multiprotocol label switching), DiffServ

(differentiated services), and RSVP (Resource Reservation Protocol). In many cases, the

NPSCs will just wrap the functionality when the platform offers an open API (e.g., OSA,

Parlay); while in other cases (e.g., SIP, GPS), full-fledged NPSCs must be implemented.

Clearly, a large number of composite services can be built from a given set of atomic

services and components, but only a subset of these will be useful from an individual end

users’ perspective. The utility of a given composite service is not only highly dependent on

users’ personal preferences and the tasks they wish to perform, but also on the context in

which the user will access the service. The proposed adaptive scheme will provide a method

of dynamic incorporation of context information, in particular the information on current

network conditions of the service selection and composition process. This will contribute

significantly to the realization of services that can automatically adapt and reconfigure

themselves to handle changes in network conditions resulting, for example, from the user

7



switching to a different device or roaming among different access networks. Moreover, new

services will be able to advertise themselves, and software components can be automatically

installed on the user terminal according to user preferences.

Fig. 3 Service provisioning interactions.

The service provision interactions are shown in Fig. 3. As a result of service deployment,

discovery, and composition (1), service descriptions are published at the service repository

(2). These descriptions allow the identification of services (both atomic and composite)

available to complete specific tasks given specific terminal and network conditions. Given

descriptions of:

User and terminal preferences, which are stored at the user/terminal profile.

The tasks to be carried out.

The network context in which the user will be accessing the service.

The service broker is able to select a preexisting (atomic or composite) service or, if no

suitable services exist, to automatically compose a new service that meets the user demands

(3), (4). The user is represented by a user agent, which negotiates (via the provisioning ASCs)

with the service broker (5). The user agent provides to the service broker the tasks to be

performed in terms of agreed ontological elements, as well as an indication of the users

preferences for the means by which the service is to be delivered. The profiling ASCs build

up the user profile over time as the user agent reports to it, via the service broker, the users

service usage patterns and changing preferences (6). Information relating to current network

conditions is reported to the service broker by the events ASCs and is used as a key

8



constraint in the selection of potential service compositions to carry out the required task.

Furthermore, once a service session is ongoing, the proposed adaptive middleware will

notify the service broker, if changes in network conditions result in the selected service

composition does not satisfy the required QoS levels any longer. The broker will then search

for alternative service compositions that can satisfy the required task in a different manner,

or if this is not possible, in another manner that closely approximates what the user requires.

IV. CONTEXT-AWARE MULTIMEDIA SERVICE

Since multimedia metadata and context information are often parsed and processed by

automated systems inter-operating with third-party services and applications, they need to be

represented with standard-oriented, flexible, and inter-operable models. We propose an

ontology-based context model for context representation. In the modeling approach,

Ontology Web Language (OWL) [4] is adopted as representation language to enable

expressive context description and data inter-operability of context. In the domain of

knowledge representation, the term ontology refers to the formal and explicit description of

domain concepts, which are often conceived as a set of entities, relations, instances,

functions, and axioms. In this section, we design context-aware multimedia service

framework (see Fig. 4), and the components are described in detail as follows.

Fig. 4 Context-aware multimedia service framework.

A. Context Aggregation

The context aggregation aggregates a diversity of context information from an array of

diverse information sources. Context aggregation helps to merge the required information

related to a particular entity (e.g., user) or relevant to a particular context-aware system (e.g.,

all contexts needed by smart TV service). It then asserts them into the context knowledge

9



base for further reasoning and learning.

We deployed various hardware sensors in our prototype system, including location

sensors, lighting sensors, microphones, and video cameras. We also developed some

software programs to capture context, such as GUIs for explicitly inputting user preferences

and daily schedules, observers for capturing user feedback to specific content, and monitors

for detecting terminal capabilities and network characteristics.

B. Context Reasoner

The context reasoner infers abstract high-level contexts from basic sensed contexts,

resolves context conflicts, and maintains knowledge base consistency. To support various

kinds of reasoning tasks, we can specify different inference rules, and preload them into the

appropriate logic reasoner. We adopt a rule-based approach based on first-order logic for

reasoning about contexts. It provides forward chaining, backward chaining, and a hybrid

execution model. The forward-chaining rule engine is based on the standard rate algorithm.

The backward-chaining rule engine uses a logic-programming engine similar to Prolog

engines. A hybrid execution mode performs reasoning by combining both forward and

backward chaining.

Our current system applies Jena2 generic rule engine [7] to support forward-chaining

reasoning over the OWL represented context. To perform context inference, an application

developer needs to provide horn-logic rules for a particular application based on its

requirements. The context reasoner is responsible for interpreting rules, connecting to

context KB, and evaluating rules against stored context. We have specified a rule set based

on the forward-chaining rule engine to infer high-level contexts (e.g., a user’s social activity

in the smart home).

C. Context Learner

Preference context plays an important role in multimedia personalization services. The

context learner deduces and updates user’s preference by compiling statistical analysis on

user’s viewing history aggregated by the context aggregation from all kinds of media

playing devices (e.g., PC, television, PDA). The details of centralized preference learning in

a pervasive environment based on a master-slave architecture and implicit learning

algorithm that applies relevance feedback and naive Bayes classifier approach are described

in [8].

The centralized learning approach has several advantages. First, it learns users’ preference

10



by utilizing overall feedback information as opposed to other traditional methods that just

use partial feedback information in one device. Second, the approach can relieve pervasive

devices with limited resources from computation- and storage consuming learning tasks.

These devices are merely responsible for observing users’ behavior and uploading feedback

information to the aggregation.

Intuitively, a multimedia content is often viewed by a group of users, e.g., a family,

friends in a party, etc. Therefore, sometimes the common interest of the group users is

needed for the purpose of recommendation. The context learner can also deduce the group

preference by merging individual users’ preferences into a common one [9].

D. Content Filter

The content filter evaluates between incoming multimedia content and preference context.

It compares features in a media item with terms that characterize users watching preferences

to determine whether the user likes it. Only media items that have a high degree of similarity

to the users preference would be recorded to the local storage or directly forwarded to the

recommender. The multimedia content filtering strategy by using Vector Space Model

(VSM) was presented in [10].

The content filter also records contents according to a user’s situation context. For

example, knowing the user is currently participating in a legal course, the filter will record

law-related documentaries (e.g., legal cases).

E. Content Recommender

The content recommender provides the right content, in the right form, to the right person

based on all categories of context. The recommendation output consists of two aspects for a

media item: score and form. The score implies the degree of interest that a user pose about

the media item, while the form means the presentation features (e.g., modality, format, and

frame size) on a particular device.

For the purpose of efficient context processing in content recommendation, we classify

context into three categories: preference context (users’ taste or interests for media contents),

situation context (users’ spatial-temporal and social situation, e.g., location and time), and

capability context (physical running infrastructure, e.g., terminal capability and network

condition). The content recommender first calculates the similarity between the media item

and the preference context by adopting Vector Space Model. Then, it evaluates the

probability of the media item belonging to the situation context. The score is obtained as the

11



weighted sum of the above calculated similarity and probability. The appropriate form is

determined by applying rule-based approach to infer presentation details from the capability

context.

F. Content Adapter

The content adapter performs multimedia content adaptation by using two techniques:

summarization and transcoding. Multimedia summarization means summarizing an

audio-video item into a short one that can be viewed within a time constraint. Multimedia

transcoding involves transforming the content from one media type to another so that the

content can be suitably processed by a particular device or efficiently delivered by a specific

network condition. For instance, most handheld computers are not capable of handling video

data due to their hardware and software constraints. Therefore, video information can be

accessed alternatively through sets of images captured from the video through the

transformation/transcoding process.

Multimedia adaptation can be statically done at authoring time prior to delivery or

dynamically done on the fly if needed. In our system both strategies are accommodated. If

the content comes directly from the filter, online adaptation is done. Otherwise, when the

content is recorded to the PDR, offline adaptation is performed to prepare content variations

for later consumption. To perform online transcoding of images, we utilize the open source

compression/decompression libraries from the Independent JPEG Group. The video

transcoding is implemented based on the public domain software for H.263. The Power

Video Converter is used for offline adaptation that covers rich functionalities.

V. CONCLUDING REMARKS

In this paper, we first propose an adaptive service provisioning middleware that enables

service provisioning to mobile users and professionals anywhere, anytime, and in any

context by interoperating with existing heterogeneous networks. Then, a context-aware

multimedia framework is presented, which supports multimedia content filtering,

recommendation, adaptation, context aggregation, reasoning, and learning. By jointly

considering the context-aware multimedia service and heterogeneous networks platform, we

propose an efficient context-aware middleware for multimedia service in heterogeneous

networks.

12



REFERENCES

[1] L. Zhou, B. Geller, A. Wei, B. Zheng, “Cross-Layer Rate Allocation for Multimedia Applications in Pervasive

Computing Environment,” Proc. of IEEE GLOBECOM 08, New Orleans, USA, pp. 1-5, Dec., 2008.

[2] B. L. Tseng, C. Y. Lin, and J. R. Smith, “Video Summarization and Personalization for Pervasive Mobile Devices,”

Proc. of SPIE Electronic Imaging, pp. 59-70, San Jose, CA, Jan. 2002.

[3] S. Karlich, T. Zahariadis, N.Zervos, N. Nikolaou, B. Jennings, V. Kollias, T. Magedanz, “A Self-Adaptive Service

Provisioning Framework for 3G+/4G Mobile Applications” IEEE Wireless Communications, vol. 11, no. 5, pp. 48-56,

2004.

[4] D. L. McGuinness and F. van Harmelen, “OWL Web Ontology Language Overview,” W3C Rec., 2004.

[5] Y. Isoda, S. Kurakake, and K. Imai, “Context-Aware Computing System for Heterogeneous Applications,” Proc. of

ubiPCMM , pp. 17-25, Sep. 2005.

[6] F. Lopes, F. Delicato, T. Batista, and P. F. Pires, “Context-based heterogeneous middleware integration,” Proc. of the

2009 Workshop on Middleware for Ubiquitous and Pervasive Systems, pp. 13-18, May 2009.

[7] T. Lukasiewicz, U. Straccia, “Managing uncertainty and vagueness in description logics for the Semantic Web,” Web

Semantics: Science, Services and Agents on the World Wide Web, vol. 6, no. 4, pp. 291-308, Nov. 2008.

[8] Z. Yu, Y. Nakamura, D. Zhang, S. Kajita, and K. Mase, “Content Provisioning for Ubiquitous Learning,” IEEE

Pervasive Computing, vol. 7, no. 4, pp. 62-70, October-December 2008.

[9] G. Tao, H. Pung and D. Zhang, “Information Retrieval in Schema-Based P2P Systems using One-Dimensional

Semantic Space,” Computer Networks, Special Issue on Innovations in Web Infrastructure, vol. 51, no. 16, pp.

4543-4560, November 2007.

[10] Z. Yu, D. Zhang, X. Zhou, C. Chin, and Z. Yu, “An OSGi-Based Infrastructure for Context-Aware Multimedia

Services,” IEEE Communications Magazine, vol. 44, no. 10, pp. 136-142, October 2006.

13



Liang Zhou received his B.S. degree and M.S. degree (with honors) both major at

Electronic Engineering from Nanjing University of Posts and Telecommunications,

Nanjing, China in 2003 and 2006 , respectively. In March 2009, he received his Ph.D.

degree major at Electronic Engineering both from Ecole Normale Superieure, Cachan,

France and Shanghai Jiao Tong University, Shanghai, China. From Feb. 2009, he is a

postdoctoral researcher in ENSTA-ParisTech, France. His research interests are in the

area of multimedia communications and networks. E-mail: [email protected].

Naixue Xiong received his both PhD degrees in Wuhan University (on software

engineering based on networks), and Japan Advanced Institute of Science and

Technology (on dependable networks), respectively. Both are on computer science.

From 2008, he is a research scientist in the Department of Computer Science, Georgia

State University, Atlanta, USA. His research interests include Communication

Architecture and Design, and Optimization Theory. E-mail: Protocols, Network [email protected].

Lei Shu is a research scientist in Digital Enterprise Research Institute (DERI), at

a member of ACM a

National University of Ireland, Galway (NUIG). He received the B.Sc. degree from

South Central University for Nationalities, China, 2002, and the M.Sc. degree from

Kyung Hee University, Korea, 2005, and the PhD degree from National university of

Ireland, 2010. His research interests include wireless multimedia sensor networks,

orks, context aware middleware, and sensor network middleware, and security. He is

nd IEEE. E-mail:

wireless sensor netw

[email protected].

Athanasios Vasilakos is a professor at the department of Computer and

Telecommunications Engineering, University of Western Macedonia, Greece, and a

visiting professor at the Graduate Programme of the department of Electrical and

Computer Engineering, National Technical University of Athens (NTUA). His

research interests include wireless networks, cooperative communications

technology, context aware middleware, cross-layer design, and communications

security. E-mail: [email protected].

Sang-Soo Yeo received his bachelor's, master's and Ph.D. degrees in Computer

Science from Chung-Ang University, Seoul, Korea. He has joined Kyushu University

in Japan as a visiting scholar at the Graduate School of Information Science and

Electrical Engineering (ISEE). And then he came back to Korea and he worked for

BTWorks, Inc. as a General Manager. Now he is a professor at Division of Computer

Engineering, Mokwon University, Korea. E-mail: [email protected].

14



context-aware middleware for multimedia services in heterogeneous networks

Documents