model-driven design of audiovisual indexing processes for search apps

Model-Driven Design of Audiovisual Indexing Processes

for Search-Based Applications

Alessandro Bozzon, Marco Brambilla, Piero Fraternali

http://dbgroup.como.polimi.it/brambilla/mdd-search-apps

CBMI 2009, June 5th, Chania, Crete.

2

Model-driven design of search applications

Agenda

• Introduction• Search-based applications (SBAs)• The Content Provision and Annotation (CPA) process

Definition Verification of properties

• Conclusions

3


3Introduction

• Search has become the prominent paradigm for information seeking across both online spaces and enterprises.

• Search-Based Applications (SBAs)

searching over heterogeneous data constitutes the predominant user interaction paradigm

Search Engine

•canned applications

•fixed behavior•homogeneous data source

•simple content processing flows

•basic query flows

SBA

• tailor-made solution, depending on varying data and user's needs nature

• search engines are part of a complex system

• data source integration

• content analysis

• querying

• Web-mediated social interactions

4


4Motivations

• SBAs are complex application integrating: complex front-end: query expression, result presentation, personalization,

adaptation complex back-end: content provisioning, annotation, indexing, distributed query

execution

• SBAs embody data-intensive and process-intensive tasks

• content items, queries, and result lists are first-class objects, subject to a life cycle

• Content Provisioning and Annotation process (CAI process)– indexation of contents coming from the application data sources (data

retrieval from external sources, transformation or aggregation, analysis, indexation)

• Query and Result Presentation process (QRP process)

– operations related to query analysis, execution, orchestration and result-set composition

• User Interaction process (UI process)

– the way users interact with the application's functionalities ( statement of information needs, result navigation, social interactions)

5


5Thesis

• Current practices lack complete tools and methodologies• rely either on very simple models or on the programming skills

of the developers low separation of concerns among involved actors low productivity in the implementation phase problems in managing and maintaining applications over time

• SBA development demands for the same evolution in methods and tools that has characterized in the recent past the progress of Software Engineering for Web applications clear separation of concerns among the involved actors central roles of models as key development artifacts automatic code generation, etc.

SBA Model Driven Development Framework

6


6The contribution

A modeling framework

• an intuitive Rich Process Model to specify the schema of the CPA process for a given application,

• use of formal methods to investigate properties of interest of his design

• code generation techniques to implement and deploy the process

7


7Background

• Information Retrieval ”... the process of finding contents (e.g., textual documents, multimedia items, etc.)

that satisfy an information need from within large collections (usually stored on computers)...”

• Business Process Design representing processes (of heterogeneous nature) in terms of related, structured

activities or tasks that produce a specific service or product several proposals for visual modeling languages (e.g., UML, YAML, BPMN)

• Model Driven Web Engineering raise of abstraction (separation of platform independent and platform dependent

concerns) in Web application design and development use of models (and model transformations) as the key artifacts for application

developments several proposals (e.g., UML, Hera, OOHDM, UWE, W2000, WebML)

• Formal methods Formal semantics to models Automatic verification

8


8Background Models

BPMN WebML

Business Process Model Notation•Activities

•FlowsConstraints (OR-XOR-AND gateways)

•Artifacts (Data Objects and data associations)

•Events

Web Modeling Language•Domain ModelsNavigation Models ( content publication and manipulation, link behavioral semantics)

•Service composition and orchestration Models ( Web service invocation and publication, XML management)

9


9Modeling Approach

Process Models: BPMN (CAI, QRP), Interaction Pattern Composition (UI)

Domain data and process metadata: ER/UML

Application Models: WebML

Model To Model Transformation: Java / ATL

Model To Code Transformation: Java

10


10CPA Process - Domain Model

Content model: the objects that relate to the Content Items indexed by a search application

Annotation model: structure of the annotations associated with searchable Content Items during the indexing process

Usage model: user, roles, permissions

Index model:

abstraction for the actual physical implementation of search engine indexes

11


11CPA Process - Process and Application design

Coarse CPA process model Content Registration Content Analysis Content Indexation

Fine-grained process model Analysis of audiovisual content

trough face recognition and identification technologies

Application model Invocation algorithm of annotation

technologies

Re

fin

em

en

tM

2M

Tra

ns

form

ati

on

M2

T T

ran

sfo

rma

tio

n

Running CPA process Console trace of the working

annotation technology Process advancement control UI

Full Size

Video

12


12CPA - Examples of complex processes

Analysis of audiovisual content

Analysis of audio-visual content with textual annotations

13


13Search-specific extensions to the process model (BPMN*)

• BPMN does not provide support for:

• process data• data flows and dependencies among activities

does not convey domain-specific information about the modeled process for activity’s business logic

• Formalized extensions to the BPMN language, introducing: typed attributes (content, annotation, process) typed activities activity properties

• type of produced and consumed data• type of enforced operations (classified w.r.t. their input/output)

finer-grained data flows• mapping of links with guard conditions and parameters

14


14BPMN* > WebML transformation

• More detailed transformation rules finer-grained Application Model, needing less (or no) refinement by the

designer

• typed activities enables reusable PIM models

• data dependencies are specified at an higher level less errors in Application Model design fastest SBA development

ATL Transformation

Example of ATL BPMD-WebML transformation rule for activities

15


15Verification of properties for CPA processes

• Assessment of CPA process outputs Algebraic representation

• Expression of an activity depend on the type (number and type of input/output) of the activity and on the schema (activity pre-condition, mutual exclusions or indetermination in setting activity inputs) of the process

The expression evaluation requires assigning the semantics to each activity• functions from input to output values, described extensively, as tabular expression (or

ground-truth)

Variation of activities’ business logics (or in the process schema) produces variations in the representativeness of annotations assessment of produced annotation quality, validation against expected outcomes, compare

outcomes of different processes

indexAnnotation(AggregateAnnotation(

AnalyzeAudio(TranscodeAudioVideo(RetrieveVideo()).audioOut),

AnalyzeFaces(TranscodeAudioVideo(RetrieveVideo()).videoOut)))

16


16Implementation

• Extensions to a commercial tool for the automatic generation of Web applications

• Creation of a Web-based BPMN editor for the specification of the CAI and QRP process models

• Set of model to model and model to text transformations

17


17Experiences

• PHAROS (Platform for searcH of Audiovisual Resources across Online Spaces) project First-hand experience on a large scale SBA application

• Adoption of models in the platform design

– Domain model

– Query and result presentation process

– User interaction process

– Experiments with Content Analysis and

Indexing process Developer evaluation

Pros Cons

Complete set of design dimensionsHigh-quality prototypesQuick prototyping cyclesFast development time

Lack of reverse transformations from application to process modelsAvailability of only simple guard condition expressions

18


18Conclusions and future work

A modeling framework

• an intuitive and visual design of the CPA process,

• Helps the initial design of the process

• Helps for its evolution when requirements change

• the extensibility of the model, based on a plugin approach

• the amenability to automatic verification

• for checking properties

• for estimating the characteristics of a specific CPA configuration

• the availability of well-established code generation technology

• great improvement in productivity

Future work

• extension of the validation capabilities

• industrial implementation

19


19

Thank You!

Marco [email protected]

www.dei.polimi.it

http://www.pharos-audiovisual-search.eu/

19

model-driven design of audiovisual indexing processes for search apps

Technology

modeling framework

based applications

marco brambilla

cpa process

application

cpa

process

models