mieke gevers - performance testing in 5 steps - a guideline to a successful load test

Performance Testing in 5 Steps: 'A Guideline to a Successful Load

Test'

Mieke Gevers

Mieke Gevers, AQIS - Belgium

© 2008-2009 AQIS bvba 3

Agenda

1. What is load testing?

2. Methodology

3. Conclusion

© 2008-2009 AQIS bvba 4

Agenda

1. What is load testing?a. Terminology

b. Challenges

2. Methodology

3. Conclusion

© 2008-2009 AQIS bvba 5

1. Terminology

• "load testing" usually is defined as the process of exercising the system under test by feeding it the largest tasks it can operate with.

• Load testing is sometimes called volume testing, or longevity/endurance testing.

© 2008-2009 AQIS bvba 6

1. Terminology

• “Stress testing” is testing by trying to break the system under test by using overwhelming resources or by taking resources away from it (in which case it is sometimes called negative testing).

© 2008-2009 AQIS bvba 7

• Failover Tests : verify of redundancy mechanisms while under load.

• Can the system-under-test recover &/or react quickly enough? Can the other web servers (load balancers) handle the sudden dumping of extra load?

• Advantage: Allowing technicians to address problems in advance, in the comfort of a testing situation, rather than in the heat of a production outage.

1. Terminology

© 2008-2009 AQIS bvba 8

• Soak Tests ~ Endurance Testing : running a system at high levels of load for prolonged periods of time.

• Ideal for finding memory leaks and other defects which make a system prone to crashing/breaking after a certain number of iterations

1. Terminology

© 2008-2009 AQIS bvba 9

1. Terminology

• Network Sensitivity Tests : are tests that set up scenarios of varying types of network activity (traffic, error rates...), and then measure the impact of that traffic on various applications that are bandwidth dependant.

• eg. Chat applications, Streaming Media…

© 2008-2009 AQIS bvba 10

1. Terminology

• Targeted Infrastructure Test : are Isolated tests of each layer and or component in an end to end application configuration. It includes all components of the AUT infrastructure and beyond.

• eg. includes communications infrastructure, Load Balancers, Web Servers, Application Servers, Crypto cards, Citrix Servers, Database…

• Goal: identifying any performance issues that would fundamentally limit the overall ability of a system to deliver at a given performance level.

© 2008-2009 AQIS bvba 11

1. Terminology

Time

Predictedpoint of failure

Upgraded Capacity

Existing Capacity

LeadTimeTrigger

Point

AnticipatedPeak Loads

Workload

UsableReserveCapacity

Point Response Timedegradation becomesnoticeable

Currentload

©copyright 2004-2005 by Wilson Mar. All rights reserved.

Performance testing is the overall process,

Stress Test

Load Test

Scalability

© 2008-2009 AQIS bvba 12

2. Challenges

© 2008-2009 AQIS bvba 13

2. Challenge 1 : Communication

The requirements specification was defined like this

The developers understood it in that way

This is how the problem was solved before.

This is how the problem is solved now

That is the program after debugging

This is how the program is described by

marketing department

This, in fact, is what the customer wanted … ;-)

© 2008-2009 AQIS bvba 14

2. Challenge 2 : HW & SW

© 2008-2009 AQIS bvba 15

2. Challenge 2 : HW & SW

Security

© 2008-2009 AQIS bvba 16

2. Challenge 3 : User Nature

© 2008-2009 AQIS bvba 19

Response Times: The Three Important Limits [Miller 1968; Card et al. 1991]: •0.1 second

limit for having the user feel that the system is reacting instantaneously•1.0 second

limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay.

•10 seconds limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done.


© 2008-2009 AQIS bvba 20

• Computer scientists identify future IT challenges Goals for IT using the power of quantum

physics, building systems that can't go wrong

“Ambitious goals include harnessing the power of quantum physics, building systems that can't go wrong, and simulating living creatures in every detail.” By Peter Sayer, IDG News Service January 25, 2005

2. Challenge 4 : Future

© 2008-2009 AQIS bvba 21

2. Challenge 5 : Resources

• HW

• Infrastructure

• Access

• People

• Time

• ....

© 2008-2009 AQIS bvba 22

Agenda

1. What is load testing?

2. Methodology

3. Conclusion

© 2008-2009 AQIS bvba 23

3. Methodology

Analyze

© 2008-2009 AQIS bvba 24

Documentation

© 2008-2009 AQIS bvba 25

1. Determine system performance, response time, and throughput under a specific load

2. Testing the ability to handle load (and stress) to identify bottlenecks

3. Evaluating whether system resources are being utilised efficiently

4. Testing system robustness and capability to recover from errors

5. Testing across different configurations or versions

6. Testing systems for scalability

1. Goals

© 2008-2009 AQIS bvba 26

Examine SUT

• Architecture Overviews

• Deployment Topology• 3rd party components

and SLA’s• Firewall capacity• Load balancing• Connectivity• Network

• Session management• All queues• Caching models• Security methods• Fail over mechanisms• Redundancy• Bandwidth• ……

© 2008-2009 AQIS bvba 28

5 steps

Analyze

Plan/Define/Design scenario's

© 2008-2009 AQIS bvba 29

Design and Model

1. Transactions and Workflows scenario's

2. User profiling and modeling

3. Workload scenario's

4. Test environment

5. Test data

© 2008-2009 AQIS bvba 30

1. Transition matrix

online shopping visit Make Flow-charts of Functional Paths through the

application-under-test. Break this down in Logical modules = transactions

© 2008-2009 AQIS bvba 31

2. User Profiling

• User activities, transactions and usage patterns

• Client platforms and preferences

• Client Internet access speeds and browser types

• User geographic locations

© 2008-2009 AQIS bvba 32

3. Workflow Modeling

• Size of customer base

• Growth factor

• Site arrival rates and site abandonment

• Think times and latency

• Background noise

© 2008-2009 AQIS bvba 33

4. Test environment

• Production replica

– Expensive, usually not possible

• Scaled down

– Extrapolation factor

• Actual production equipment

– For new applications

© 2008-2009 AQIS bvba 34

5. Test data

• Data• Randomized• Database

• Cookies, Session ID’s, Hidden ID’s, values• Certificates• Security settings

• IP addresses identifications• User specifics identifications, PKI’s

© 2008-2009 AQIS bvba 36

Load Test scenario's

• Program Perl, Java

• Stubs

• HW devices

• FreeWare tools

• Commercial Tools (capture & playback)

•...

© 2008-2009 AQIS bvba 37

How to reproduce correct web behavior?

• Correct environment simulation– Correct protocol (Business level)

– Speed– UI (Browser/MMS/..)– Security simulation/usage– Calculating the amount of virtual users and

the arrival rate– ...

© 2008-2009 AQIS bvba 39

What is a baseline?

• Baseline with one user

• Monitor the involved back-end systems

• Can the load test achieve the goals

• Test-runs

© 2008-2009 AQIS bvba 43

What to Measure

Maximum login requests (per min/per sec) Average session length (at peak) max. Concurrent sessions Average pages per session Average hits per page Average request distribution Average size of a page & specific size of a page Arrival ratesTip: Session length(in min)= Max_concurrent_sessions/

Login_requests_per_min

Metrics

© 2008-2009 AQIS bvba 47

Tips

• Be aware of constant changes

• Constantly training

• User behaviours

• Increase of transactions (seasonal – permanent)

• Use Common Sence

© 2008-2009 AQIS bvba 48

Capacity planning for “Web performance”, Menascé Building High-Scalability Server Farms, 1999 Microsoft Corporation Detecting System Bottlenecks in Sites Using Site Server 3.0 Commerce

Edition, 1999 Microsoft Corporation Design for scalability, IBM High-Volume Web site team, December 1999

http://www7b.boulder.ibm.com/wsdd/library/techarticles/hvws/scalability.html Performane Testing Guidance for Web applications, Scott Barber ISO 9126, ISO/IEC 12207:2008, International Organization for

Standardization

Resources

http://www7b.boulder.ibm.com/wsdd/library/techarticles/hvws/scalability.htm

http://www7b.boulder.ibm.com/wsdd/library/techarticles/hvws/scalability.html

© 2008-2009 AQIS bvba

49

Questions &Thank you !

Mieke Gevers

[email protected]

www.aqis.eu

Agile Quality in Information Systems

mieke gevers - performance testing in 5 steps - a guideline to a successful load test

Software