secrets of successful java testing · java test tools run anywhere java apps java tool jvm java os...
TRANSCRIPT
P R E S E N T A T I O N
International Conference On
Software Testing, Analysis & ReviewNOV 8-12, 1999 • BARCELONA, SPAIN
Presentation
Bio
Return to Main Menu
PresentationPaperBio
Return to Main Menu W7
Wednesday, Nov 10, 1999
Secrets of Successful
Gary Guillon
Java Testing
Secrets of successful Java testing
Gary GuillonSun Microsystems
© 1999 Sun M icrosystemsSlide 2
JVM JVM JVM JVM
The Java Virtual Machine levels the playing field...
Different operating systems have different facilities for running programs
OS 1 OS 2! OS 4OS 3
APP
An application written for one OS, will not run on another...
JAVA
APP
JAVA
APP
JAVA
APP
JAVA
APP
The same identical Java application can run on any OS, without modification!
Refresher : What is Java?
It’s a language, and an environment!
© 1999 Sun M icrosystemsSlide 3
Cost of Software Defects
DefectCost
Design Implementation Release Deployment
• The cost of software defects increases rapidly.
Test Earlier and More Often
© 1999 Sun M icrosystemsSlide 4
The Testing Process
•Test Earlier and More Often
Design Implement
Tra
diti
onal
Ap
proa
ch
Release Deploy
Design
Rec
omm
end
edA
ppr
oach
Unit testing Regression
Release Implement Deploy
LoadStatic Analysis
© 1999 Sun M icrosystemsSlide 5
When and what to check
Design
Unit testing
Code coverage analysis
Regression
Load testing
Static Check
Dynamic Check
Implement Release Deployment
Load testing Load testing Load testing
© 1999 Sun M icrosystemsSlide 6
Why is it different to test?
Java Apps
JVM
Java OS
Java Apps
Java Chip
NativeApps
Java Apps
JVM
Native OS
© 1999 Sun M icrosystemsSlide 7
Platform Specific Problems
Java Apps
JVM
Java OS
Java Apps
Java Chip
NativeApps
Java Apps
JVM
Platform-Specific Test Tool
Unix test script
Win 32 tool Win 32 script
Mac Test tool Mac test script
OS/2 Test tool OS/2 test script
Win3.x Test tool Win3.x test script
AS/400 test tool AS/400 test script
JavaOS tool ? ? ?
Unix Test tool
Java Chip tool ? ? ?Is your OS supported?
? ?Platform-specific tools require one test for each platformPlatform-specific test tools are “blind” to most Java events
© 1999 Sun M icrosystemsSlide 8
Java Test Tools Run Anywhere
Java Apps
Java Tool
JVM
Java OS
Java Apps
Java Tool
Java Chip
Java Apps
Java Tool
JVM
NativeApps
Native OS
Pure Java testing tools work on all Java platformsThe Java test tool runs in the same process as the application100% portable tests, “write tests once, run tests anywhere”
Java Test tool Java Test script
© 1999 Sun M icrosystemsSlide 9
Can you spot the difference?
A GUI testing tool that cannotresolve objects and is forcedto resolve bitmap images..
W ill FAIL a test for the wrongreason.
There is no difference, theseGUIs are functionally identical!A test PASS!
© 1999 Sun M icrosystemsSlide 10
The Advantages
Java-based Platform-Specific
Return Value
Color.Grey
“Replace All”
100,120
Boolean.TRUE
Boolean.TRUE
(Bitmap image)
! Data, m ethod
! getBackground ( )
! getLabel ( )
! getLocation ( )
! isVisible ( )
! isEnabled ( )
Java Apps
Java Tool
JVM
Native OS
PlatformTool
Peerless components are fully visible to Java-based toolsPeerless components are invisible to platform specific tools
© 1999 Sun M icrosystemsSlide 11
Testing Applets
Applets can be tested usingSuns appletviewer technology,or in the browser itself. (Thismay require the installation ofthe java plug-in)
© 1999 Sun M icrosystemsSlide 12
Write tests once, run tests anywhere!
Script is reviewed once forthoroughness and coverage
Application
TestScript
Application is installed ontarget configurations
Solaris
Win NT/95
NC
COMPARE
Result FileExecution on NT/95
Result FileExecution on Solaris
Result FileExecution on NC
© 1999 Sun M icrosystemsSlide 13
What is the Highest Priority?
What getseffected , whenyou shiftpriorities?
10
1010
10Time to Market ? 3
Quality of Product ? 6
Cost Saving = 1 !!
Total Available Marks - 10-6 = 4-3 = 1
© 1999 Sun M icrosystemsSlide 14
Are you dealing with the real problem?
Ship #1: Please divert your course 15 degrees to theNorth to avoid a collision.
Unknown contact#2: Recommend you divert your course15 degrees to the south to avoid a collision.
Ship #1: This is the Captain of a US Navy ship. I sayagain, divert YOUR course.
Contact #2: This is the watch ensign. I say again, youdivert YOUR course.
Ship #1: THIS IS THE AIRCRAFT CARRIERENTERPRISE! WE ARE A LARGE WARSHIP OFTHE US NAVY! DIVERT YOUR COURSE NOW!
Contact #2: This is a lighthouse. It's your call!!!!
Actual radio conversation released by Chief of Naval Operations…
© 1999 Sun M icrosystemsSlide 15
Load Performance
Time (sec)
0 20 40 60 80 100 120
SystemCrash!
In the past Clientshave estimatedperformance based
on an existingsystem oravailable hardware
100
90
80
70
60
50
40
30
20
10
0
User Count
W hat is the response time at 100 users?
Time by ‘Finger in Air’ method
Time by Load testing!
© 1999 Sun M icrosystemsSlide 16
Asynchronous Load testing…. Ouch.
Same System, same end result, but different internal transaction organisation!
Traditional Client/ServerSynchronous arrangement.Client always waits forserver.
Query andresponse arealways in thesame order.
GUI client waitswhilst server isbusy.
Multiple request canbe sent, data canarrive in any order!
GUI can do usefulwork whilst serverbusy.
New Era ORB based ordistributed applicationprovides asynchronouscommunication for greaterefficiency!
© 1999 Sun M icrosystemsSlide 17
Asynchronous Load testing….cont.
Input 1
Response 1
Input 2
Response 2
Input3
Response3
Script captured from Scenario 1
Tries to play against Scenario 2…but the system under testresponds differently...
“Record, capture andencapsulate” load testingtechnologies are allsynchronous!
FORCES SYSTEM TO BE SYNCHRONOUS!NOT A TRUE REPRESENTATION OF THE SYSTEM !!!
Input 1
Waiting forresponse 1…
waiting…waiting…waiting...
Script doesn’t sendinput 2 until response1 is received, forcingsystem to behavedifferently,INVALIDATINGTHE TEST!
© 1999 Sun M icrosystemsSlide 18
Network Java Load Testing
��������������
App Server
.
.
.
�����������������
�����������������
����������������
����������������������������������������������������������������������������
��������������
��������������
����������������
Distributed 3-tierSAP / Baan / Oracle
Mainframe / Legacy
InternetInternet
���������������
20 GUI vu500 Net4 Telemetry
Central Console
Injector Network
© 1999 Sun M icrosystemsSlide 19
Defect TrackingDefect Tracking
CM SystemsCM Systems
JavaScope
Coverage ToolsCoverage Tools
The Testing Infrastructure
SoftwareUnder TestSoftware
Under TestSystem
Scripts &Results
Unit TestingUnit Testing
Manual
GUI TestingGUI Testing
JavaStar
JavaSpec
GUI
API
! Test M anagem ent is essential!!!
?TestManagement!
TestManagement!
© 1999 Sun M icrosystemsSlide 20
Need to know more?
W eb : http://www.sun.com /suntest
Em ail : suntest_saleseurope@ sun.com
Phone : 0800 962761 (UK)
Phone : +353 1 819 9000 (International)
That was… Secrets of successful Java testing
By... Gary GuillonSun [email protected]
Gary Guillon
European Manager - Java Testing Tools for Sun MicrosystemsGary was a Engineer for GEC Avionics working on Hardware DataConditioning for the Tornado Aircraft and moved into the software groupto test the Nimrod - Early Warning Submarine Detection equipment,which is still being used today. He has a number of humorous stories of'Testing Disasters' that appear during his testing presentation.
Java is rapidly becoming the language/environment of choice in theTimes Top 1000 companies world-wide. With it comes a whole host ofTesting concerns. Asynchronous Testing being one of the frequentlyoverlooked elements of testing software running on parallel processors. This talk addresses the uniqueness of Java and how AsynchronousTesting can be done effectively with technology that was invented by SunMicrosystems, the author of Java itself.