benchmarking for graph transformation
DESCRIPTION
Benchmarking for Graph Transformation. Gergely Varró gervarro @cs.bme.hu Andy Schürr [email protected] Dániel Varró [email protected]. Talk o verview. Paradigm & tool features. GT overview. Introduction. Benchmarking terminology. Measurements. Model transformation in MDD. - PowerPoint PPT PresentationTRANSCRIPT
Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Benchmarking for Graph Transformation
Gergely Varró [email protected]
Andy Schü[email protected]ániel Varró [email protected]
2Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
3Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Model transformation in MDD• Model Driven Development (MDD)
– thorough system modeling before implementation– precise and visual SW models
• on a high abstraction level• from different viewpoints
– automatic code generation• Model transformation
– automated translations within and between visual modeling languages
• MDD requirements on model transformation:– Specification: high-level specification language– Execution: efficient algorithms and techniques– Industrialization: extensive tool support
• Upcoming standard: QVT
4Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Specification techniques for MT
• QVT• Main features:
– object based patterns– precondition– unidirectional– bidirectional
• Problems:– immature tool support– lack of experience
• GT• Main features:
– similar to QVT– wide tool support
• Problems:– non-bidirectional
(except for TGGs)
• Pattern matching phase:– most critical issue
from performance viewpoint
– same as in QVT
5Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Benchmarking
• Aim:– systematic and reproducable measurements– on performance– under different and precisely defined
circumstances
• Overall goal: help system engineers in selecting tools
• Popular approach in different fields– AI– relational databases– rule-based expert systems
6Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Benchmarking in model transformation
• Specification examples for MT– Goal: assessing expressiveness
– UML-to-XMI, object-relational mapping, UML-to-EJB, etc.
• No performance benchmarks for MT
• In the paper:Benchmarks for graph transformation tools– terminology for benchmarking
– quantitative comparison of tools
• Ongoing research:– Benchmarks for general model transformation tools
7Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
8Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Metamodel
Metamodeling
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
next
blocked
held_by releasetoken
request*
1
*
1*
111
***
*
Class
Association
Multiplicity constraint
At most one
Arbitrary
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
Resource
Process
Object
Link
Slot
Inheritance
Attribute
Instance model
9Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Graph transformation rule
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
Negative applicationc
ondition
p1:Process
r:Resource
GiveRule
n:nextp2:Process
rel:release
p1:Process
r:Resource
n:nextp2:Process
t:token
LHS RHS
Left-HandSide
Right-HandSide
10Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Pattern matching phase
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
11Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Updating phase
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
rl:release
12Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
13Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Paradigm features
• Definition: problem characteristics
• Model size– number of nodes and edges in the model graph
• Pattern size– number of nodes and edges in the LHS graph
• Fan-out (maximum degree of model nodes)– number of edges adjacent to a certain node
• Number of matchings
• Transformation sequence length
14Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Tool features
• Definition: typical optimization strategies supported by different tools
• Multiplicity based optimization– More powerful strategy when traversing edges with ‘at
most one’ (0..1) multiplicity
• Parameter passing
• Parallel rule execution
• ‘As long as possible’ (ALAP) rule application
• Indexes (for key/index attributes)
• Incremental pattern matching
• ...
15Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parameter passing I.
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
rl1:release
•Between consecutive rule applications•No recalculation in later steps
16Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parameter passing II.
rl1:release
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
p1:Process
r:Resource
GiveRule
n:nextp2:Process
rel:release
p1:Process
r:Resource
n:nextp2:Process
t:token
LHS RHS
t1:token
17Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parallel rule execution (PM)
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
4
1 2
3
1
43
2
4
3
2
1
•Calculation of all matchings•Independent matchings
18Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Parallel rule execution (Update)
h1:held_by
n4:next
n1:next n2:next
n3:next
h2:held_by
h3:held_byh4:held_by
p1:Process p3:Process
p2:Process
p4:Processr4:Resource
r1:Resource r2:Resource
r3:Resource
r:Resource
p:Process
r:Resource
ReleaseRule
hb:held_by
reqn:requestrel:release
rn:Resourcep:Process
LHS RHS
4
1 2
3
1
43
2
4
3
2
1
rl1:release
rl3:release
rl2:release
rl4:release
•No re-evaluation of valid matchings•Transaction block
19Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
20Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology I.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –
– Broad application field of graph transformation• model analysis• model transformation• visual languages with dynamic operational
semantics
– Informal characteristics
21Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology II.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –
next
blocked
held_by releasetoken
request*
1
*
1*
111
**
*
*
Resource
Process
TakeRule
RequestRule
GiveRule
ReleaseRule
LHS RHS• Incarnation of a scenario– fulfills the informal characteristics
– e.g. Mutex for the simulation of visual languages
22Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology III.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –Test set + + PD PD PD –
h1:held_by
nN:next
n1:next
hN:held_by
p1:Process
p2:Process
pN:ProcessrN:Resource
r1:Resource
Instance model
1. N ReleaseRule
2. N GiveRule
3. N RequestRule
4. N TakeRule
• Parametric and deterministicspecification
23Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology IV.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –Test set + + PD PD PD –Test case + + PD PD PD +
• Multiplicity based optimization: OFF
• Parameter passing: OFF
• Parallel rule execution: ON
24Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Terminology V.
Me
tam
ode
l
GT
rule
s
Instance m
ode
l
Tra
nsforma
tion
sequ
ence
Pa
radigm
fea
ture
s
To
ol fea
ture
s
Scenario – – – – – –Benchmark example + + – – – –Test set + + PD PD PD –Test case + + PD PD PD +Run + + + + + +
• Complete characterization
• Reproducable measurements
N = 10
25Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Talk overview
IntroductionGT
overview
Paradigm & tool features
Benchmarking terminology
Measurements
26Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Categorization
• Style– Interpreted: AGG, GTDB
– Compiled: PROGRES, Fujaba
• Base algorithm– Constraint satisfaction: AGG, GTDB
• variables + constraints
– Local searches: Fujaba, PROGRES • step-by-step extension of the matching
27Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Environment
• Hardware and OS– 1500 MHz Pentium
– 768 MB RAM
– Linux kernel of version 2.6.7
• Tool related– Runs without GUI
– Rule applications guided by Java programs• PROGRES: guided by C program
– Standard services of the default distribution• fine-tuning based on developers’ suggestions
28Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Tool specific issues
• AGG– Manual implementation of parameter passing
• Progres– GRAS (default underlying graph-oriented database)– Prolog-style cuts (determinism)– User undo/redo mechanism being switched off
• Fujaba– Additional node for storing processes and resources– Slightly altered models to provide better performance
• GTDB– MySQL (version 4.1.7)– No additional optimization
29Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Tool feature support
Multiplicity based
optimization
Parameter passing
Parallel rule application
AGG – + –
PROGRES + + +
Fujaba + + +
GTDB – + +
30Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Quantitative comparison
Updating phase
0.01
0.1
1
10
100
10 100 1000
N
av
g. t
ime
(m
s)
AGG
PROGRES
FUJABA
DB
• Result set clean-up procedure• Constant overhead distributed over more rule applications
• Compilation step from graphsto categories in each GT step
31Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Effect of multiplicity based optimization
10 100 1000 10 100 1000 10 100 1000
0.1
1
10
100
1000
10 100 1000
GiveRule
10 100 1000 10 100 1000 10 100 1000
0.1
1
10
100
1000
10 100 1000
ReleaseRule
AGG PROGRES FUJABA GTDB
32Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Effect of parameter passing
10 100 1000 10 100 1000 10 100 1000
0.1
1
10
100
1000
10 100 1000
AGG PROGRES FUJABA GTDB
STS test set, Pattern matching phase, ReleaseRule
33Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Effect of parallel rule application
0.1
1
10
100
1000
10 100 1000 10 100 1000 10 100 1000 10 100 1000
AGG PROGRES FUJABA GTDB
ALAP test set, Pattern matching phase, ReleaseRule
34Budapest University of Technology and Economics
Technical University of DarmstadtReal-Time Systems Lab
Thanks for your kind attention
• Contributions to model transformation benchmarking are welcome– additional benchmark examples, test sets, test
cases
– participation of further tools
– new optimization strategies
• Benchmark example descriptions, specifications, and measurement results– http://www.cs.bme.hu/~gervarro/benchmark/