monitor-based testing of elastic cloud computing applications

© AtlanMod ([email protected])© AtlanMod ([email protected])

Monitor-Based Testing of Elastic Cloud Computing

ApplicationsMichel Albonico

PhD Student - AtlanMod - EMN (Nantes, France)([email protected])

Jean-Marie MottuGerson Sunyé

1

5th

Int.

Wor

ksho

p on

Lar

ge S

cale

Tes

ting

Del

ft, N

ethe

rland

s - 2

016

© AtlanMod ([email protected])

● Cloud Computing Elasticity● Motivation● Test Procedure● Experiments● Conclusion and Future Work

Outline

2


Cloud Computing Elasticity

3

● Cloud computing elasticity:

The ability of a cloud infrastructure/system modifying its resource

configuration according to demand.



4

● Thresholds: ○ Scale-out threshold: maximum resource usage, e.g., 80% of CPU

usage;○ Scale-in threshold: minimum resource usage, e.g., 20% of CPU

usage;○ Used to decide when varying a resource.

Resource AllocationResource DemandScale-out Threshold

Scale-in ThresholdScale-out Threshold BreachingScale-in Threshold Breaching

Time (s)

Res

ourc

e (P

roce

ssor

s)

1

Legend

80% 0.8

20% 0.2



5

● Resource demand varies according to workload variations.○ Example:

■ number of users increases from 1 to 2, the resource

demand doubles.



Time (s)

Res

ourc

e (P

roce

ssor

s)

1

Legend

80% 0.8

20% 0.2



6

● Resource demand varies over time;● Scale-out threshold breaching.



Time (s)

Res

ourc

e (P

roce

ssor

s)

1

Legend

scale-out threshold breaching

80% 0.8

20% 0.2



7

● Resource demand varies over time;● Scale-out threshold breaching;● Scale-out reaction time;



Time (s)

Res

ourc

e (P

roce

ssor

s)

1

Legend

scale-out reaction time

80% 0.8

20% 0.2



8

● Resource demand varies over time;● Scale-out threshold breaching;● Scale-out reaction time;● Scale-out time, then the thresholds are updated.



Time (s)

Res

ourc

e (P

roce

ssor

s)

2

180% 0.8

Legend

scale-out time80% 1.6

20% 0.2

20% 0.4



9

● Scale-in:○ Scale-in threshold breaching;○ Scale-in reaction time (resource is no longer available);

■ Thresholds reconfiguration.○ Scale-in time.



Time (s)

Res

ourc

e (P

roce

ssor

s)

2

1

Legend

scale-in time

scale-in reaction time

80% 0.8

80% 1.6

20% 0.2

20% 0.4



10

● Elasticity states transition.

scale-out threshold breaching

scale-in threshold breaching


● Elasticity states transition.

● Related work only test during the ready state;● Scaling states:

○ Considerable time: in our experiments, scaling-out takes more than 90 seconds (Amazon EC2);

○ Great part of the adaptation tasks: replication data, leader election, etc.

Motivation

11


Test Procedure

● Test cloud systems during all the elasticity states;● Execute tests dynamically:

○ Associate test cases to a set of elasticity states;○ Execute the test according to the current elasticity state.

● Test execution:○ Periodically monitor the resource during the test execution;

■ Current elasticity state.○ (Re)-execute the associated test cases during the current

elasticity state.

12


Experiments

● Research questions (answered by the experiments):

1. Is it necessary to run the test during different elasticity states?a. Does a cloud system react distinctly depending on the

elasticity state?

2. Is it possible to execute the test during different elasticity states and to assign the test verdicts accordingly?

13


Experiments

Question 1: system behavior during different elasticity state.

● First experiment:Measure the performance of a cloud system during different elasticity states.

○ Manually executed;

○ Workload (50% read / 50% write);

○ 2500 operations per second (ops).

14


Experiments

● First experiment results:○ 2000 ops: covers all the performance drops;○ Elasticity states extracted from the log files;

RQ1: It is necessary to run the test during different elasticity states.

15

Per

form

ance

- O

pera

tions

per

Sec

ond

(ops

)

Minimal PerformanceMeasured Performance

R R R R R RSISISO SO SOSI

200

400

1000

120080

0

600 Time (s)


Question 2: test execution during different elasticity states + test verdicts assignment.

● Second experiment: (same workload)○ We use our test procedure;○ We monitor the elasticity states throughout the test

execution;○ Test Case:

■ answered operation >= 2000 ops -> pass■ otherwise -> fail

○ Same test case associated to every elasticity state.

■ Test case re-executed throughout the cloud system

execution.

Experiments

16


Experiments

● Result of the second experiment:○ Test through different elasticity states;○ Assign test verdicts to different elasticity states;

■ Proportional to the previous experiment (correct elasticity

states).RQ2:

It is possible to execute the test according to the elasticity state, and we are able to assign the test verdicts correctly.

17


Conclusion and Future Work

● Identify all the performance problems;● Assign the test verdicts to the correct elasticity states (at

runtime);● Address the scaling states, which are not addressed by related

work;● Future work:

○ Write functional test cases;○ Apply to other study cases;○ Generate test cases based on elasticity states.

18


Monitor-Based Testing of Elastic Cloud Computing

ApplicationsMichel Albonico

PhD Student - AtlanMod - EMN (Nantes, France)([email protected])

Jean-Marie MottuGerson Sunyé

19

5th

Int.

Wor

ksho

p on

Lar

ge S

cale

Tes

ting

Del

ft, N

ethe

rland

s - 2

016

monitor-based testing of elastic cloud computing applications

Software