feedback driven development

08.09.16 | Dept. of CS – Software Technology Group | Technical University of Darmstadt | Harini Gunabalan |

Harini GunabalanMaster Thesis, Distributed Software systems

Feedback Driven Development ofCloud Applications

1

Execution Analytics Coordinated Adaptation Feedback


Motivation

2


Motivation

Cloud

3


Motivation

Cloud

Developer

Operator

End Users

4


Motivation

Cloud

Developer

Operator

End Users

App App

Deploy app

5


Motivation

Cloud

Developer

Operator

End Users

App App

Deploy app

VMVM VM VMProvision resources

6


Motivation

Cloud

Developer

Operator

End Users

App App

Deploy app


Use App

7


Cloud

Developer

Operator

End Users

App App

Deploy app


Use App

8

Not working!

Motivation


Motivation

Cloud

Developer

Operator

End Users

App App

Deploy app


Use App

Where is the problem?Not working!

9


Cloud

Developer

Operator

End Users

App App

Deploy app


Use App

Where is the problem?

Maybe, I should

improvecode?

Not working!

10

Motivation


Cloud


App

End Users

AppUse App Developer

Operator

Maybe, I should

improvecode?

Is theresource

provisioningnot sufficient?

Deploy app

Where is the problem?Not working!

11

Motivation


Problem Statement

12


Problem Statement – APM Information

13


Problem Statement - FDD

14

Automating Resource provisioning Using the Run-time information

Identifying Source Code Issues

What is Feedback Driven Development?


Problem Statement – Auto-scaling

15

Automating Resource provisioning Using the Run-time information

Problem:Automate Resource

Provisioning

Solution: Auto-scaling

What is Auto-scaling?On-demand scale out and

scale in depending on Load.


System Design

16

Architecture

17

• 3 componentsØ Cloud

monitoring

Ø Scaling-service

Ø Data modeling

• Platform level autoscaling

Component Interaction

18


19


20


21


Implementation

22


System Footprint

23


Component 1: Cloud Monitoring

24

Key metrics monitored• Average response time (CF Logs)• Number of incoming requests (CF

Logs)• Number of running app instances

(CF API)• CPU utilization (CF API)• Memory utilization (CF API)• Disk utilization (CF API)

Component 2: Auto-scaling

25


• Model data to identify correlation between metrics• Most real world data are Multi-dimensional.

• Multiple-input and Multiple-output (MIMO) models.ØState-space ModelØPolynomial ModelsØ ARX

Ø ARMAX

Component 3: Data Modeling

26


State Space Models

• Represented by differential equations and state variables.

• Output can be predicted for any future time provided the input, output, and a minimum set of state variables xi(t), are known.

dx/dt = Ax + Buy = C x + Du

27


Polynomial models - ARX & ARMAX

ARX:

• Auto regression - current output depends on the past input and output values.

• Considering auto regressionand the current inputs, ARX model can be mathematicallydescribed as:

A(z) y(t) = B(z)u(t - n) + e(t)

28

ARMAX:

• Unlike ARX, ARMAX considers stochastic dynamics.

• Better for systems with extra disturbances

• Includes both AR(p) and MA(q) models.

• ARMAX is represented mathematically as

A(z) y(t) = B(z)u(t - n) + c(z)e(t)


1. Set Output = [Response Time]

2. Set Input = [ Number of incoming requests, No of instances, CPU%, memory, and disk utilization]

Data Modeling in MATLAB System Identification Toolbox: Import Data

29


MATLAB System Identification Toolbox –Dataset splitting

30


• Model estimation• State-space Model• Polynomial Models

• ARX

• ARMAX

• Validate the model

Model Estimation and Validation

31


Estimating State Space model

32


Estimating ARX and ARMAX models

33


Evaluation

34

Image Source: https://www.usu.edu/ccampis/evaluation/


Deploy Guestbook Application to Cloud

35


Generate Load on Guestbook app

36


Scaling occurs

37

Req

uest

s pe

r sec

ond

No.

of i

nsta

nces


Configuration File Settings

38


Comparison with and without the Auto-scaler

39

Req

uest

s pe

r sec

ond

Aver

age

resp

onse

tim

eR

eque

sts

per s

econ

d

Aver

age

resp

onse

tim

e

Without Auto-scaler, Maximum response time is 750 ms

With Auto-scaler,Maximum response time is 100 ms


Model output

The Model estimation output for State space, ARX and ARMAX models.

40


Insert page title

Headline here

§ Lorem ipsum dolor sit amet, consecteturadipiscing elit. Maecenas pellentesque laoreeturna molestie placerat ipsum.

Headline here

§ Lorem ipsum dolor sit amet, consecteturadipiscing elit. Maecenas pellentesque laoreeturna molestie placerat ipsum.

Video Demo


Conculsion

43

Automated Resource provisioning Using the run-time metrics

Solved!ü Monitoring Service

ü Auto-scaler

ü Correlation Model to make the auto-scaler smart!


• Metric parameter tuning

• Metric combination & Custom Metrics

• Determining metric thresholds

• Testing with different applications types - memory or databaseintensive.

• Improved modeling, collection of larger sample datasets

Open Challenges and Future work


References

Ø M. Httermann, DevOps for developers. Apress, 2012.

Ø J. Cito, P. Leitner, H. C. Gall, A. Dadashi, A. Keller, and A. Roth, “Runtime metric meets developer: building better cloud applications using feedback,” in 2015 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pp. 14–27, ACM, 2015.

Ø “Amazon web services.” https://aws.amazon.com/, Accessed: 08-Aug-2016.

Ø “New relic.” https://newrelic.com/, Accessed: 08-Aug-2016.

Ø J. Cito, P. Leitner, T. Fritz, and H. C. Gall, “The making of cloud applications: An empirical study on software development for the cloud,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 393–403, ACM, 2015.

Ø L. Ljung, System identification. Springer, 1998.

Ø “Sap hana cloud platform.” https://hcp.sap.com/index.html, Accessed: 08-Aug-2016.

Ø S. R. Seelam, P. Dettori, P. Westerink, and B. B. Yang, “Polyglot application auto scaling service for platform as a service cloud,” in Cloud Engineering (IC2E), 2015 IEEE International Conference on, pp. 84–91, IEEE, 2015.

Ø J. Humble and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education, 2010.

Ø http://docs.pivotal.io/pivotalcf/1-7/customizing/autoscale-configuration.html

Ø Cloud wave project - http://cloudwave-fp7.eu/

45


Thank You

46

www.cloudwave-fp7.eu

Prof. Dr. –Ing. Mira Mezini

Dr. –Ing. Guido Salvaneschi

Dr. Gerald Junkermann

Aryan Dadashi

Jürgen Cito


Appendix: Polynomial models - ARX

• The ARX model to evaluate the output is based on auto-regression.

• Auto regressive model is a model whose current output depends on the past input and output values. The generic notion to denote auto-regressive model of order p, AR(p) for a variable X is:

where c and i are constants and e(t) is the noise. Considering auto regression andthe inputs, ARX model can be mathematically described as:

A(z) y(t) = B(z)u(t - n) + e(t)

where y(t) is the output, u(t) is the input, and e(t) is the noise/error measured in the output. A(z) and B(z) are polynomials of the specified order with respect to the backward shift operator z-1.

47


Appendix: Polynomial models - ARMAX

Unlike the ARX model, in ARMAX, the stochastic dynamics are considered. ARMAX models are better for systems with more disturbances. In general, the moving average model of order q, MA(q) is represented in the below notation:

where e(t-i) is the noise/error. The notation for the autoregressive moving average(ARMA) model is as below:

This model includes both AR(p) and MA(q) models. Based on these the following mathematical equation for the ARMAX model can be written as:

A(z) y(t) = B(z)u(t - n) + c(z)e(t)

where, y(t) is the output, u(t)is the input, and e(t) is the noise. A(z), B(z) and C(z) are polynomials of specified orders with respect to the backward shift operator z-1

48


Appendix: CASpott

Feedback Handler- Queries the raw logs- Prepares the raw logs for

further processing by mapping them to the API format of the Statistical Analysis Engine

Statistical Analysis Engine- Analyzes and aggregates

the collected information coming from Feedback Handler based on predefined metrics

Static Code Analysis Engine- Analyzes the code and

realizes the connection between the feedback and the code