Data marketplaces: models and concepts

Hong-Linh Truong

Distributed Systems Group,

Vienna University of Technology

truong@dsg.tuwien.ac.at http://www.infosys.tuwien.ac.at/staff/truong

1 ASE WS 2012

Advanced Services Engineering,

WS 2012, Lecture 6

Outline

Data marketplaces

Description models

Exchange data agreement

Data contract

ASE WS 2012 2

Data service unit

3

Recall – data service units in

clouds/internet

data

Internet/Cloud

Data service unit

People

data

Data service unit

Things

ASE WS 2012

data data

Data-as-a-Service – service models

Recall – data as a service

ASE WS 2012 4

Storage-as-a-Service

(Basic storage functions)

Database-as-a-Service

(Structured/non-structured

querying systems)

Data publish/subcription

middleware as a service

Sensor-as-a-Service

Private/Public/Hybrid/Community Clouds

deploy

Data marketplaces

More than just DaaS

DaaS focuses on data provisioning features

Data marketplaces

Multiple data providers and consumers

Multiple DaaS

Complex interactions among DaaS, data providers

and consumers

Complex billing and pricing models

Market dynamics

ASE WS 2012 5

WHAT ARE IMPORTANT ISSUES IN DATA

MARKETPLACES?

Discussion time

ASE WS 2012 6

DAAS DESCRIPTION MODEL

Some important issues

ASE WS 2012 7

DATA AGREEMENT EXCHANGE

DATA CONTRACT

Description Model for DaaS (1)

State of the art:

Providers have their own way to describe DaaS,

mainly in HTML

Existing service description techniques are not

adequate in supporting description for DaaS

Problems

Service and data discovery cannot be done

automatically

On-demand data integration, service integration, and

query optimization cannot be supported well.

Service/data information and DaaS engineering

cannot be tied. ASE WS 2012 8

Description Model for DaaS (2)

Which levels must be covered?

ASE WS 2012 9

Data

items

Data

items

Data

items

Data resource

Data

assets

Data resource Data resource

Data resource Data resource

Consumer

Consumer

DaaS

Here

Description Model for DaaS – types

of information

Which types of information must be covered?

ASE WS 2012 10

Quality of

data Ownership

Price License ....

Service

interface Service

license Quality of

service ....

DEMOS – a description model for

Data-as-a-Service

ASE WS 2012 11

See prototype:

http://www.infosys.tuwien.ac.at/

prototype/SOD1/demods/

Quang Hieu Vu, Tran Vu Pham, Hong

Linh Truong,, Schahram Dustdar,

Rasool Asal: DEMODS: A Description

Model for Data-as-a-Service. AINA

2012: 605-612

Description model and data

marketplaces

ASE WS 2012 12

DEMODS – prototype (1)

ASE WS 2012 13

DEMODS – prototype (2)

ASE WS 2012 14

Check: http://demodsmanagement.appspot.com/

WHICH TYPES OF DAAS INFORMATION

ARE DYNAMIC? AND THEIR IMPACT ON

DESCRIPTION MODELS?

Discussion time

ASE WS 2012 15

Exchange data agreement (1)

ASE WS 2012 16

DaaS

Consumer

DaaS

Sensor

DaaS

Consumer DaaS provider Data

provider

How they interact w.r.t. data concerns?

How their data agreements look like?

Exchange data agreement (2)

Lack of models and protocols for data

agreement in data marketplaces

Constraints for data usage are not clear

Inadequate data/service description → hindering data

selection and integration

Existing techniques are not adequate for

dynamic data agreement exchange in data

marketplaces

Need generic exchange models suitable for different

ways of data provisioning in data marketplaces

ASE WS 2012 17

Data Agreement Exchange as a

Service (DAES)

Metamodel for data agreement exchange

Techniques for enriching and associating data

assets with agreement terms

Interaction models for data agreement exchange

Hong Linh Truong, Schahram Dustdar, Joachim Götze, Tino Fleuren, Paul Müller, Salah-Eddine Tbahriti, Michael Mrissa,

Chirine Ghedira: Exchanging Data Agreements in the DaaS Model. APSCC 2011: 153-160

ASE WS 2012 18

Metamodel for data agreements

Different

category of

agreements

Licensing,

privacy, quality

of data

Extensions

Languages

Different types

of agreements

Different

specifications

ASE WS 2012 19

Associating data with data

agreements

Solutions

(a) directly inserting agreements into data assets

(b) providing two-step access to agreements and data

assets

(c) linking data agreements to the description of DaaS

(d) linking data agreements to the message sent by

DaaS

ASE WS 2012 20

Possible interaction models for data

enriched with data agreements

ASE WS 2012 21

DAES – conceptual architecture

Jersey, JAX-RS Restful WS Weblogic

Using URIs to identify agreements ASE WS 2012 22

DAES – managed information

Specific applications: agreement creation, agreement validation,

agreement compatibility analysis, agreement management

Implementation: Jersey, JAX-RS Restful WS Weblogic

ASE WS 2012 23

Illustrating examples – insert

agreement into data asset

A pay-per-use consumer uses dataAPI of DaaS

search for data

The consumer pays the use APIs

Each call can return different types of data

Example with

People Search in

Infochimps

But a strong consequence

for data service engineering

techniques: dealing with

elastic requirements!

ASE WS 2012 24

Illustrating examples – link

agreements to geospatial data Domain-specific DaaS: different agreements for different data requests

Vector data of geographic features via Web-Feature-Service (WFS)

Terrain elevation data via Web-Coverage Services (WCS)

ASE WS 2012 25

Illustrating examples – link

agreements to geospatial data

Consumers can interpret and

reason if the data can be

used for specific purposes

ASE WS 2012 26

Illustrative examples – develop an

app for policy compliance (1)

ASE WS 2012 27

Illustrative examples – develop an

app for policy compliance (2) Configuration

Results

ASE WS 2012 28

HOW NEAR-REALTIME DATA IMPACTS ON

DATA AGREEMENT EXCHANGE?

Discussion time

ASE WS 2012 29

Data contract

How to specific data contract?

ASE WS 2012 30

Data

items

Data

items

Data

items

Data resource

Data

assets

Data resource Data resource

Data resource Data resource

Consumer

Consumer

DaaS

Data contracts

Give a clear information about data usage

Have a remedy against the consumer where the

circumstances are such that the acts complained

of do not

Limit the liability of data providers in case of

failure of the provided data;

Specify information on data delivery,

acceptance, and payment

31 ASE WS 2012

32

Data contracts

Well-researched contracts for services but not

for DaaS and data marketplaces

But service APIs != data APIs =! data assets

Several open questions

Right to use data? Quality of data in the data

agreement? Search based on data contract? Etc.

➔ Require extensible models

➔ Capture contractual terms for data contracts

➔ Support (semi-)automatic data service/data selection

techniques.

Hong-Linh Truong, Marco Comerio, Flavio De Paoli, G.R. Gangadharan, Schahram Dustdar, "Data Contracts for

Cloud-based Data Marketplaces ", International Journal of Computational Science and Engineering, 2012 Vol.7, No.4,

pp.280 - 295

ASE WS 2012

Study of main data contract terms

Data rights

Derivation, Collection, Reproduction, Attribution

Quality of Data (QoD)

Not mentioned, Not clear how to establish QoD metrics

Regulatory Compliance

Sarbanes-Oxley, EU data protection directive, etc.

Pricing model

Different models, pricing for data APIs and for data assets

Control and Relationship

Evolution terms, support terms, limitation of liability, etc

33

Most information is in human-readable form

ASE WS 2012

34

Data contract study

ASE WS 2012

35

Developing data contracts in cloud-

based data marketplaces

Follow community-based approach for data

contract

Propose generic structures to represent data

contract terms and abstract data contracts

Develop frameworks for data contract applications

Incorporate data contracts into data-as-a-service

description

Develop data contract applications

ASE WS 2012

36

Community view on data contract

development

Community users can develop:

Term categories, term names, values, and units

Rules for data contracts

Common contract and contract fragments

Community users =!

novice users

ASE WS 2012

37

Representing data contract terms

Contract term: (termName,termValue)

Term name: common terms or user-specific terms

Term value: a single value, a set, or a range

ASE WS 2012

38

Structuring abstract data contracts

Concrete data contracts

can be in RDF, XML or

JSON

generates

Use Identifiers and

Tags for identifying

and searches

ASE WS 2012

39

Development of contract

applications

Main applications:

Data contract compatibility evaluation, data contract

composition

Some common steps

Extract DCTermType in TermCategoryType

Extact comprable terms from all contracts,

- e.g., dataRight: Derivation, Composition and Reproduction

Use evaluation rules associated with DCTermType

from from rule repositories

Execute rules by passing comparable terms to rules

Aggregate results

ASE WS 2012

Prototype

RDF for representing term categories, term

names, term values, units

Allegro Graph for storing contract knowledge

ASE WS 2012 40

41

Illustrating examples

A large sustainability monitoring data platform

shows how green buildings are

Real-time total and per capita of CO2 emission

of monitored building

Open government data about CO2 per capita at

national level

We created contracts from

Open Data Commons Attribution License

Open Government License

ASE WS 2012

42

Existing

common

knowledge

about Open

Data

Commons

ASE WS 2012

43

Step 2: provide OpenBuildingCO2

OpenBuildingCO2 by

modifying quality of

data and data right

OpenGov for

government data

Data contract for green building data

ASE WS 2012

44

Experiments – composing data

contract terms

ASE WS 2012

CAN WE AUTOMATICALLY GENERATE

DATA CONTRACTS FOR NEAR-REALTIME

DATA?

Discussion time

ASE WS 2012 45

Exercises

Read mentioned papers

Examine existing data marketplaces and write

DEMODS-based specification for some of them

Develop some specific data contracts for open

government data

Work on some algorithms for checking data

contract compatiblity

ASE WS 2012 46

47

Thanks for your attention

Hong-Linh Truong

Distributed Systems Group

Vienna University of Technology

truong@dsg.tuwien.ac.at

http://www.infosys.tuwien.ac.at/staff/truong

ASE WS 2012