heiko koziolek, decrc/i1 ladenburg, germany, 2014-11-13 6

© ABB Group December 3, 2014 | Slide 1

6 Years of Performance Modelingat ABB Corporate Research

Heiko Koziolek, DECRC/I1 Ladenburg, Germany, 2014-11-13

My Story Today


2008: Performance Modeling with PalladioOverview

1) Measure

System instrumentation

& performance tests

using load drivers

(custom tooling)

2) Model

Component-based

model with annotated

flow charts entered into

Eclipse-based tooling

3) Predict

Running

analytic solvers /

simulators, varying

model parameters to

test different situations


Model derivation (manually)

Prediction (automatically)

2010: Q-ImPrESS & Industrial Control System


2010: Q-ImPrESSModels & Tools


SoMoX / Sissy for Reverse Engineering Component Models from C++

Windows Performance Monitor for Performance Measurement

Self-implemented C#-Client as load driver

Q-ImPreSS Workbench for Modelling (meta model similar to Palladio)

LQN solver / Palladio SimuCom for Performance Prediction

PerOpteryx for Design Space Exploration

Q-ImPress WorkbenchSoMoX

2010: Q-ImPrESSResults


Koziolek, Schlich et al.

An industrial case study on

quality impact prediction for

evolving service-oriented

software.

In Proc. ICSE 2011 SEIP,

pp. 776-785. ACM, May 2011.

2010: Q-ImPrESSLessons Learned

Successes

Large performance model build with Q-ImPreSS tooling

Models validated through measurements (<30% error)

First experiments with design space exploration

Challenges

Not enough inputs on new ABB system available,

had to fallback to model older version

predictions for older system are not really actionable

as the older version will not be changed

Modeling tools disconnected from the tools currently

used during development (e.g., Enterprise Architect)

creating models with the tools from scratch required

high effort

Static code analysis challenged by Microsoft C++ code© ABB Group December 3, 2014 | Slide 7

2012: Performance Modeling for ABB Robotics


2012: Performance Modeling for ABB RoboticsModels & Tools

Dynatrace for distributed performance profiling

Neoload as load driver

Palladio Workbench for modelling ‚

(all manual no static code analysis)

LQN/SimuCom for performance prediction

PerOpteryx for design space exploration© ABB Group December 3, 2014 | Slide 9

Palladio Workbench

LQN Solver

2012: Performance Modeling for ABB RoboticsResults


Thijmen de Gooijer, Anton Jansen, Heiko Koziolek, and Anne Koziolek. An industrial case study of performance and cost

design space exploration. In Proc. 3rd Int. Conf. on Performance Engineering (ICPE'12), pp 205-216. ACM, April 2012.

2012: Performance Modeling for ABB RoboticsLessons Learned

Successes

Due to performance fixes based on the measurements

the performance could be improved by 50%

Roadmap for extending the system cost-effectively was devised

based on the models

Large-scale industrial case study on design space exploration of

a distributed, component-based system

ABB Robotics integrated Dynatrace into their development

environment

Challenges

Information extraction for the models took a long time, lots of

calibration needed, several assumptions required for models

Expensive measurement & testing tools (>30K€)


2014: Automation Cloud


2014: Automation CloudModels & Tools

Amazon Web Services / Own Cloud Server as test

environment (up to 36 AWS m1.large instances)

KairosDB, OpenTSDB, Databus as time-series databases

Apache Cassandra and Hbase as distributed DBMS

Netflix Priam / Apache Whirr for quick deployment

Visual Studio Ultimate Web Load Test as load driver

[No modeling, only benchmarks!]© ABB Group December 3, 2014 | Slide 13

2014: Automation CloudResults


Limited overload WITH

AWS Autoscaling

Linear scalability

for KairosDB

Avg. Roundtrip Time: 193ms

for 15 customers

ABB Phasor Measurement Unit

used in Power Grids

ABB Smart Meter

Thomas Goldschmidt, Anton Jansen, Heiko Koziolek, Jens Doppelhamer, and Hongyu Pei-Breivold. Scalability

and Robustness of Time-Series Databases for Cloud-Native Monitoring of Industrial Processes. In Proceedings

7th IEEE Int. Conf. on Cloud Computing (IEEE CLOUD 2014) Industry Track. IEEE, July 2014.

2014: Automation CloudLessons Learned

Successes

Showed technical feasibility for several scenarios from

industrial automation in a cloud computing environment

Created benchmarks for time-series databases based

on realistic workloads from ABB products

Created elasticity metrics and benchmark

Challenges

Better testing needed to improve robustness

Component-based model did not well fit with databases

/ cloud platform (e.g., auto-scaling?)

Limited insights expected from modeling due to focus

on initial technical feasibility


2015 Outlook: Collaboration with Uni WürzburgAutomatic Construction of Architectural Perf. Models


Kieker + C# adapter / JNBridge for distributed profiling

LibReDe for resource demand estimation

.NET Bookstore / Pet Shop (C#) for testing, later ABB system

Palladio / Descartes / PerOpteryx for modeling / prediction

6 Years of Performance Modeling at ABBConclusions

Performance modeling has matured over the last 6 years

But: to get wider adoption

lower costs and higher benefits

are required.


6 Years of Performance Modeling at ABBFuture Work

Future work for lower costs

Better integration between measurement and modeling tools

Faster modeling via more convenient software tools

Faster modeling via reusable model libraries

Future work for higher benefits

More performance questions to be answered

Decision support and incorporation of heuristics

Better integration into existing development processes & tools


heiko koziolek, decrc/i1 ladenburg, germany, 2014-11-13 6

Documents