2006 squad annual conference february 16-17, 2006 ppa ... · copyright 2006 by cordell vail joe...
TRANSCRIPT
Presentation by
Joe Towns & Cordell Vail
2006 SQuAD Annual Conference
February 16-17, 2006 PPA Event CenterDenver, Colorado
1
2006 SQuAD Annual Conference
February 16-17, 2006 PPA Event CenterDenver, Colorado
1
Presentation byCordell Vail & Joe Towns
Testing Tools that Will Increase Your ROI
Copyright 2006 by Cordell Vail
Copyright 2006 by Cordell Vail
Some of the joke slides you saw in the actual presentation as transition slides, have been removed from this web version of the presentation due to copyright laws. They can only be displayed in a class room setting and not distributed to the public with out permission from the cartoonist. Therefore, the ones I do not have permission to distribute have been removed.
COPYRIGHT NOTICE:
Copyright 2006 by Cordell Vail
The reason most people never find a 4 leaf clover is thatthey never look for one.Life is a lot like that.
Cordell Vail
Making Things Happen
2
Copyright 2006 by Cordell Vail
OK! Now I am going to just sit here on thisstool and hang onand I want the cow to jump up and down
Is this your view of improving ROI?
Do you just expect it to happen?
c
INTRODUCTION
3
Copyright 2006 by Cordell Vail
Was this YOU, when they asked you to find a load testing tool?
Purpose of this Presentation
j4
Copyright 2006 by Cordell Vail
Help you learn how to find aninexpensive testing tool that will give you predictable results
This presentation is NOT intended to teach you how to do the testing
Purpose of this Presentation
5
Copyright 2006 by Cordell Vail
• Know Where To Start• Uniformity• Definition Of Terms
Stress TestingLoad TestingVolume TestingPerformance TestingBenchmark TestingBaseline Testing
6
The “Big Picture” (Overview)
Copyright 2006 by Cordell Vail
• Know the testing objective• Know the testing environment• Only buy what you need• Know how you are going to test• Trusting the test results• Possible solutions• Can we trust market share• Testing tool comparisons
7
The “Big Picture" (Overview)
Copyright 2006 by Cordell Vail
In testing we would all do well to follow Stephen Covey’s advice:
“Begin with the end in mind”
What is it you need to test?
Know Where to Start
8
Copyright 2006 by Cordell Vail
Uniformity
c9
Copyright 2006 by Cordell Vail
Defining terms eliminates confusion
• Stress • Load • Volume • Performance • Benchmark• Baseline
Uniformity
10
Copyright 2006 by Cordell Vail
It is not so important how you define testing terms as it is that everyone in your organization use the same definitions
Uniformity
11
Copyright 2006 by Cordell Vail
Stress Testing
Tests the server
Peak volume over a short span of time
12
Copyright 2006 by Cordell Vail
Load Testing
Tests the database
Largest load the database can handle at one time
13
Copyright 2006 by Cordell Vail
Volume Testing
Tests the server & the database
Heavy volumes of data over time
Combination of Stress Testing and Load Testing over time
14
Copyright 2006 by Cordell Vail
Performance Testing
Tests user response time
With web applications this is normally the main consideration
15
Copyright 2006 by Cordell Vail
Benchmark Testing
Compares your testing standards to the same testing standards in other similar organizations in the industry
16
Copyright 2006 by Cordell Vail
Baseline Testing
Setting testing standards to be used as a starting point for comparison later within your own organization
17
See the 2006 search engine test tool update on the CD
Copyright 2006 by Cordell Vail
Will the tool need to test:
Bandwidth MultithreadingConcurrent users Disk capacityMultiple platforms FaultsMultiple browsers MemoryUsers per server User response
Identify the Target of the Test
18
Copyright 2006 by Cordell Vail
• Number of client workstations• Connectivity to servers• Database availability• Production copy of application• Application and web servers• Bandwidth and LAN• Test tools to monitor results
Know the Testing Environment
19
Copyright 2006 by Cordell Vail
WSIPC Testing Environment
NOTE: The Client Server and the 2 Web Servers areHP Proliant BL 20 PG2 Dual 3GH Xeon with 2GB Ram
20
Copyright 2006 by Cordell Vail
To find an economical tool you must know what your testing needs are
Each tool is very different
You are not going to pay $200 and get a tool that will do what an $85,000 tool will do!
Only Buy What You Need
21
Copyright 2006 by Cordell Vail
Bandwidth RestrictionServer PerformanceMultithreading
DevelopersDevelopers System EngineersSystem
EngineersTest
EngineersTest
EngineersSource Code AnalysisCyclomatic ComplexityMemory Leaks
Error HandlingFault InjectionUser Response Time
Consider Team Composition
The members of your testing team will be able to help determine what tests need to be done withwhich testing tool
22
Copyright 2006 by Cordell Vail
targets causes Stress FailureStress Test Stress Attack Point(s)
Buffer, cache Resource leak
Data corruption
Unhandled exception
Race condition\deadlock
Async pattern
p\invokes
Load Limit Breakpoint Torture Duration Synchronization Fault Injection
This example taken froma presentation at the Seattle SASQAG meeting 21 Apr 2005 by Keith Stobie. Diagram created by Reji John. Both work at Microsoft.(Used with permission)
Know How You Test
23
Copyright 2006 by Cordell Vail
With a new testing tool:
Are results accurate?
Are the results Valid?
What are you comparing to?
Trusting The Test Results
24
Copyright 2006 by Cordell Vail
How big is this equipment?
25
Trusting The Test Results
c
Copyright 2006 by Cordell Vail
Now how big is it?
26
Trusting The Test Results
Copyright 2006 by Cordell Vail
Possible Solutions
Test the tool on an existing application and web server where you already know what the test results should be
27
Are there other solutions?
Copyright 2006 by Cordell Vail
Test with more than one testing tool and compare the results
Is that feasible with the cost of the tools on the market today?
28
Possible Solutions
c
Copyright 2006 by Cordell Vail
93% of the market shareis controlledby 7 vendorsall chargingvery high prices
Can We Trust Market Share
29c
Copyright 2006 by Cordell Vail
Source:http://www.mercury.com/us/pdf/company/newport_load2000.pdf
30
Can We Trust Market Share
Copyright 2006 by Cordell Vail
We decided to look at that “OTHER” 7%mostly because of price
Affordable Testing Tools
31
Copyright 2006 by Cordell Vail
http://www.vcaa.com/testengineer/links.htm
http://www.testingfaqs.org/t-load.html
http://hammerhead.sourceforge.net/
http://opensourcetesting.org/performance.php
http://www.grove.co.uk/Tool_Information/Choosing_Tools.html
http://www.softwaregatest.com/gatweb1.html#LOAD
http://www.sqa-test.com/toolpage.html
http://www.webservices.org/index.php/ws/content/view/full/102
http://opensourcetesting.org/performance.php
http://sourceforge.net/projects/dieseltest/
NOTE: See our comparison list on the session CD handout
32
Affordable Testing Tools
Copyright 2006 by Cordell Vail
Testing Tool Vendor Links
Here are links to those 7 larger vendors that are holding 93% of the market share
Check the Prices out for yourself!They are all over $50,000 for 1,000 virtual users
www-306.ibm.com/software/awdtools/tester/performance/index.htmlwww.segue.com/products/load-stress-performance-testing/index.aspwww.mercury.com/us/products/performance-center/loadrunner/www.radview.com/products/WebLOAD.aspwww.quotium.com/qpro_overview_load_testing.htmlwww.empirix.com/default.asp?action=article&ID=418www.compuware.com/products/qacenter/performance.htm
33
Copyright 2006 by Cordell Vail
TestMaker – PushToTest: FREE http://www.pushtotest.com/Downloads/
WAST – Microsoft: FREEhttp://www.microsoft.com/downloads/details.aspx?FamilyID=E2C0585A-062A-439E-A67D-75A89AA36495&displaylang=en
LoadTester – AppPerfect: FREEhttp://www.appperfect.com/products/devsuite/lt.html
Our Testing Tool Findings
34
Copyright 2006 by Cordell Vail
Site Tester 1 – Pilot: $29http://www.pilotltd.com/eng/index.html
Portent Supreme – $279Loadtesting.com:www.loadtesting.com
WAPT - Logasoft: $299http://www.loadtestingtool.com
35
Our Testing Tool Findings
Copyright 2006 by Cordell Vail
Webserver Stress $625Tool 7 – Paessler:www.paessler.com
HOLODECK - SISE: $1,500http://www.sisecure.com/holodeck/learn.shtml
NOTE: Holodeck is a fault injection tool-not a normal virtual user testing tool
36
Our Testing Tool Findings
Copyright 2006 by Cordell Vail
Other Options
Testing Onshore Outsourcinghttp://www.veritest.com/services/load_stress.asp
Online Hosted Serviceshttp://servers.aplus.net/loadbalance.htmlhttp://www.webpartner.com/products/st_main.htmlhttp://www.keynote.com/http://www.webmetrics.com/loadtesting.html
NOTE: Hosted or online testing may be available free from your ISP
37j
Copyright 2006 by Cordell Vail
QUESTIONS
38
Copyright 2006 by Cordell Vail
HOW THE TOOLS WORK
The Tool Demonstration HandoutOnly Shows The WAPT Tool
Seeing the Results
39
Copyright 2006 by Cordell Vail
The tool we selected for our testing was WAPT
40
Copyright 2006 by Cordell Vail
To use WAPT first you need to createa profile, which becomes the individuallyrecorded test scripts
41
Copyright 2006 by Cordell Vail
Name the PROFILE (this will by your test script)
42
Copyright 2006 by Cordell Vail
A profile named TEST 1 is now createdthe actual name in the example below will be605-303-01 - Monroe JH - log in - Allan Shell - SAVE Attendance af
43
Copyright 2006 by Cordell Vail
You will now have to record the test script
44
Copyright 2006 by Cordell Vail
When you record the profile you can use thename you want to use
45
Copyright 2006 by Cordell Vail
It will then ask you to enter the URL for the web page to be recorded from
46
Copyright 2006 by Cordell Vail
It then begins to record all of the Web page mouse clicks as you navigate through the application
47
Copyright 2006 by Cordell Vail
Now you have arecorded test scriptthat you can play back
You place the loginscript in the “INITIAL”folder and the logoffscript in the “FINAL”folder so they only runone time in the sequence
48
Copyright 2006 by Cordell Vail
Each test script (or as they call them - profiles) has set properties that control it and can be changed
49
Copyright 2006 by Cordell Vail
Now, after you have recorded a whole series of profiles,you can run them in different combinations in a scenario
Each scenario is just a shell that holds profiles
You can run one profile or many in each scenario
50
Copyright 2006 by Cordell Vail 51
Copyright 2006 by Cordell Vail 52
Copyright 2006 by Cordell Vail 53
Copyright 2006 by Cordell Vail 54
Copyright 2006 by Cordell Vail
You can then add the profiles that you have created that you want to have run in this particular scenario
55
Copyright 2006 by Cordell Vail 56
Copyright 2006 by Cordell Vail
The test scripts for each profile you put in the scenario is displayed
57
Copyright 2006 by Cordell Vail
Each profile has set properties that can be changed that control how the profiles are to be run
58
Copyright 2006 by Cordell Vail
With the properties in the scenario you can control the test run
59
Copyright 2006 by Cordell Vail
You can also control where the logs will be sent and set properties for them
60
Copyright 2006 by Cordell Vail
As the test runs you can watch the results being built
61
Copyright 2006 by Cordell Vail
The test results are then displayedWAPT has a wonderful reporting system
62
Copyright 2006 by Cordell Vail
There are numerous test results that you can then display, print or email to management
63
Copyright 2006 by Cordell Vail
Joe Towns, Applications Development Manager/ Project Manager. Joe grew up in the Seattle area, attained his degree in Computer Science and has worked in the field of software development for over 18 years. Joe has worked in various phases of software development, with focus on quality assurance, quality control and configuration management. Joe is currently the Applications Development Manager and Project Manager for the Financial Management development team at Washington School Information Processing Cooperative (WSIPC), where he is pursuing PMI certification. Joe brings a development perspective to the presentation.
The AuthorsCordell Vail, cste – Quality Assurance Analyst. Cordell grew up in Utah and graduated from the Brigham Young University and has completed two years of graduate school work at the University ofUtah in Interpersonal Communications. Cordell brings to the presentation a test engineer’s perspective. He is a Certified Software Test Engineer and Certified School Business Specialist with 9 years experience in manual and automated testing. Cordell has made several presentations on “Improving Testing Processes”at both local and national conferences.
65
Copyright 2006 by Cordell Vail
NOTE: The information contained in this presentation handout and on the handout CD is for use only by the participants who attend our seminar at the Software Quality Association of Denver Conference held 16th
and 17th of February, 2006 in Denver, Colorado.
Distribution of this information to anyone other than those attending the seminar is not authorized by the authors. It is for educational purposes of the seminar attendees only. The jokes in the slide presentation are not printed here in the conference handout in compliance with current copyright laws.
Credits
66
Copyright 2006 by Cordell Vail
Care enough to be the very best!
THIS IS NOT THE END IT IS JUST THE BEGINNING