invited talk @ dcc09 workshop

IDCC’09, London - P.Missier

Paolo Missier Information Management Group School of Computer Science, University of Manchester, UK

with additional material by Sean Bechhofer and Matthew Gamble, e-Labs design group, University of Manchester

Scientific Workflow Management System

Researchobjects,myExperiment,andOpenProvenanceforcollabora;veE‐science

REPRISEworkshop‐IDCC’09

JanusProvenance

Momentum on sharing and collaboration

2http://www.nature.com/news/specials/datasharing/index.html

Prepublication data sharing: Nature 461, 168-170 (10 September 2009) | doi:10.1038/461168a; Published online 9 September 2009

The Toronto group: Toronto International Data Release Workshop Authors, Nature 461, 168–169 (2009)

Special issue of Nature on Data Sharing (Sept. 2009)

• timeliness requires rapid sharing• repurposing• the Human Genome project use case

• Ongoing debate in several communities– Clinical trials [1]– Earth Sciences -- ESIP - data preservation / stewardship, 2009– Long established in some communities - Atmospheric sciences,

1998 [2]• Science Commons recommendations for Open Science

– Open Science recommendations from Science Commons (July 2008) [link]

Reference scenario

workflowexecution

workflow+

input datasetspecification

Reference scenario

workflowexecution

workflow+

Reference scenario

workflowexecution

workflow+

outcome(data)

outcome(provenance)

Reference scenario

workflowexecution

workflow+

outcome(data)

outcome(provenance)

ResearchObject

Packaging

Reference scenario

workflowexecution

workflow+

outcome(data)

outcome(provenance)

ResearchObject

Packaging

Reference scenario

workflowexecution

workflow+

outcome(data)

outcome(provenance)

ResearchObject

Packaging

browse query

unbundle reuse

Reference scenario

workflowexecution

workflow+

outcome(data)

outcome(provenance)

Data-mediatedimplicit

collaboration

ResearchObject

Packaging

browse query

unbundle reuse

Collaboration through data

1.Packaging:– standards for self-descriptive data + metadata bundles:

Research Objects

2.Content:– data format standardization efforts – metadata representation

• process provenance–workflow provenance

3.Container:– a repository for Research Objects 4

What is needed for B to make sense of A’s data?

Research Objects

Common pathways

QTLPaul’sPackPaul’sResearchObject

Results

Slides

Workflow 16

Workflow 13

Common pathways

Results

Slides

Workflow 16

Workflow 13

Common pathways

Representation

Results

Slides

Workflow 16

Workflow 13

Common pathways

Representation

Domain Relations

Results

SlidesFeeds intoproduces

Included in

produces

Published in

produces

Included in

Included in Included in

Published in

Workflow 16

Workflow 13

Common pathways

Representation

Domain Relations

Results

Included in

produces

Published in

produces

Included in

Published in

Workflow 16

Workflow 13

Common pathways

Representation

Domain Relations

Aggregation

Results

Metadata

Included in

produces

Published in

produces

Included in

Published in

Workflow 16

Workflow 13

Common pathways

Representation

Domain Relations

Aggregation

ORE: representing generic aggregations

Resource Map(descriptor)

Data structure

http://www.openarchives.org/ore/1.0/primer.html section 4

A. Pepe, M. Mayernik, C.L. Borgman, and H.V. Sompel, "From Artifacts to Aggregations: Modeling Scientific Life Cycles on the Semantic Web," Journal of the American Society for Information Science and Technology (JASIST), to appear, 2009.

Content: Workflow provenance

A detailed trace of workflow execution- tasks performed, data transformations

- inputs used, outputs produced

lister

gene_id

output

pathway_genes

get pathwaysby genes1

merge pathways

concat gene pathway ids

• To establish quality, relevance, trust

• To track information attribution through complex transformations

• To describe one’s experiment to others, for understanding / reuse

• To provide evidence in support of scientific claims

• To enable post hoc process analysis for improvement, re-design

Why provenance matters, if done right

The W3C Incubator on Provenance has been collecting numerous use cases:http://www.w3.org/2005/Incubator/prov/wiki/Use_Cases#

What users expect to learn

• Causal relations:- which pathways come from which genes?- which processes contributed to producing an

image?- which process(es) caused data to be incorrect?- which data caused a process to fail?

• Process and data analytics:– analyze variations in output vs an input

parameter sweep (multiple process runs)– how often has my favourite service been

executed? on what inputs?– who produced this data?– how often does this pathway turn up when the

input genes range over a certain set S?

lister

gene_id

output

pathway_genes

get pathwaysby genes1

merge pathways

concat gene pathway ids

Open Provenance Model• graph of causal dependencies involving data and processors• not necessarily generated by a workflow!• v1.0.1 currently open for comments

A PwasGeneratedBy (R)

AP used (R)

wgb(R1)

wgb(R2)

used(R3)

used(R4)

P1wgb(R5)

P2wgb(R6)

to enable provenance metadata exchange

standardize causal dependencies

The 3rd provenance challenge

• Chosen workflow from the Pan-STARRS project– Panoramic Survey Telescope & Rapid Response Syste

• http://twiki.ipaw.info/bin/view/Challenge/ThirdProvenanceChallenge

• Goal: – demonstrate “provenance interoperability” at query level

The 3rd provenance challenge workflow

read input file

load database

verify

The 3rd provenance challenge workflow

read input file

load database

verify

Team A

OPM and query-interoperability

encode W as WA

run WA

Q(prov(WA))exportprov(WA)

prov(WA) execute query Q

OPM(prov(WA))

Team A

encode W as WA

run WA

OPM(prov(WA))

Team B

import execute query Q

PWA = import(OPM(prov(WA)))

Q(PWA)

Team A

encode W as WA

run WA

OPM(prov(WA))

?Team B

import execute query Q

PWA = import(OPM(prov(WA)))

Q(PWA)

OPM in Taverna

skippable

OPM in Taverna

skippable

OPM in Taverna

➡ the answer to any TP query can be viewed as an OPM graph

➡ encoded as RDF/XML (using the Tupelo provenance API)

skippable

Additional requirements

• Artifact values require uniform common identifier scheme– each group used artifacts to refer to its own data results– but those results were expressed using proprietary

naming conventions– Linked Data in OPM?

• OPM accounts for structural causal relationships– additional domain-specific knowledge required– attaching semantic annotations to OPM graph nodes

• OPM graphs can grow very large– reduce size by exporting only query results

• Taverna approach– multiple levels of abstraction

• through OPM accounts (“points of view”)

Query results as OPM graphs

encode W as WA

run WA

prov(WA)execute query Q

OPM(prov(WA))

encode W as WA

run WA

OPM(prov(WA))

encode W as WA

run WA

Q(prov(WA))

OPM(prov(WA)) exportQ(prov(WA))

encode W as WA

run WA

Q(prov(WA))

exportQ(prov(WA))

OPM(Q(prov(WA)))

encode W as WA

run WA

Q(prov(WA))

exportQ(prov(WA))

OPM(Q(prov(WA)))

- Approach implemented in Taverna 2.1

- Internal provenance DB with ad hoc query language

- To be released soon

Full-fledged data-mediated collaborations

exp. A

resultdatasets

ResearchObject

Aresult

provenanceA

workflow A +input A

exp. A

resultdatasets

ResearchObject

Aresult

provenanceA

workflow A +input A

result A → input B

exp. A

resultdatasets

ResearchObject

Aresult

provenanceA

workflow A +input A

exp. A

exp. B

resultdatasets

ResearchObject

Aresult

provenanceA

workflow A +input A

resultdatasets

ResearchObject

Bresult

provenanceB

workflow B+input B

resultdatasets

ResearchObjectA+B

resultprovenance

workflow B +inputB

resultdatasets

workflow A +input A

resultdatasets

ResearchObjectA+B

resultprovenance

workflow B +inputB

resultdatasets

workflow A +input A

Provenance composition accounts for implicit

collaboration

resultdatasets

ResearchObjectA+B

resultprovenance

workflow B +inputB

resultdatasets

workflow A +input A

Provenance composition accounts for implicit

collaboration

Aligned with focus of upcoming Provenance Challenge 4:“connect my provenance to yours" into a whole OPM provenance graph.

Contacts

The myGrid Consortium (Manchester, Southampton)

JanusProvenance

http://www.myexperiment.org

http://mygrid.org.uk

Me: pmissier@acm.org

invited talk @ dcc09 workshop

descriptive data

prepublication data

results logs slides

london p

research objects4idcc09

rapid sharing

metadata bundles

linkthe toronto group

Technology

learn at work suncor invited talk

invited talk at alexander college

“cyberinfrastructure for ocean cabled observatories"...

pre-computed asynchronous scan invited talk

i a invited talk, for presentation at the advanced non...

gcai 2015 (invited talk) robert...

invited talk: second search computing workshop

raminate invited talk at nii

fm6 - mcmaster universityfm06.mcmaster.ca/program.pdf ·...

industry standards – invited talk

invited talk for eudat workshop in barcelona

nips 2016. bayesopt workshop invited talk

invited talk at the geoclouds workshop, indianapolis, 2009

sayantan sur, intel exacomm workshop held in conjunction...

cwa invited talk fei pittcon 2009

icicte invited talk

invited talk oagm workshop salzburg, may 2015

bives & budhat: invited talk @ ipk gatersleben

invited talk at edmedia09

invited talk: – 5