susan windsor soft test 16th november 2005
DESCRIPTION
Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.TRANSCRIPT
![Page 1: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/1.jpg)
1
Strategic Direction for Strategic Direction for Functional Test AutomationFunctional Test Automation
SoftSoftTest 2005Test 2005
Susan Windsor Susan Windsor
Insight Through IntelligenceInsight Through Intelligence
WMHL Consulting Limited, MDWMHL Consulting Limited, MD
![Page 2: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/2.jpg)
2
AGENDA
Future for Functional Testing
History of Functional Automation
Strategic Direction
Automation Frameworks in Action
Impact upon You?
![Page 3: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/3.jpg)
3
Future for Functional
Testing
![Page 4: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/4.jpg)
4
Increased Demand………..?
Global development, with large projects having multi site, multi geography and multi suppliers to contend with.
Corporate and regulatory requirements growing all the time
Business demands new applications, faster and cheaper to obtain competitive advantage
Closer alignment of IT and business to repair lack of business confidence
![Page 5: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/5.jpg)
5
..Or, increased pressures?
Automation frameworks and greater business knowledge requirements, mean reduced numbers of traditional functional testers
Agile development methods mean developers undertake more unit and component testing
Growth of outsourced testing to different geographies, leading to greater competition for the roles
Is the role of the functional tester (who is neither technical or business specialist) dead?
![Page 6: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/6.jpg)
6
History of Functional Automation
![Page 7: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/7.jpg)
7
Record and Playback
“all you need to do…….…!” Easy to record the scripts Extremely fragile Expensive to maintain On average, each test run requires at least 50% of
the scripts to be recorded again
Please tell me no one does this anymore!!!
![Page 8: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/8.jpg)
8
Scripting
Use the scripting language to write scripts that do what you want
Build in as much robustness as possible But, you’re building an application to test an application,
which also needs testing! Maintenance costs can be very high Needs programmer skills Cost of implementation not affordable at project level
![Page 9: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/9.jpg)
9
Table Driven
Like scripting but more flexible and greater re-use Remove items from the script that change, e.g. data Reduces maintenance costs a bit Implementation costs about the same Still requires specialist skills to implement
![Page 10: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/10.jpg)
10
Functional Test Automation is Broken!
Focus on technology rather than business needs 80% of functional testing still manual 60% to 70% of automation tools used for non-
functional testing Typically, traditional functional automation stops at
100 scripts, regardless of test coverage requirement Critical factors; cost of implementation and
maintenance, skills required and asset sharing over different technologies
![Page 11: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/11.jpg)
11
Strategic Direction
![Page 12: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/12.jpg)
12
Automation Frameworks
Wouldn’t it be good if……..– Tests could be documented in common format, regardless of
whether they are manual or automated– The format for the tests resulted in quicker preparation time than
traditional manual tests– Both developers and business testers could understand and use
the same test format– Script maintenance was no longer a requirement– Tests could use any of the test execution tools required by the
underlying technology
![Page 13: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/13.jpg)
13
Business Analysts already using them, and use will grow
Home grown frameworks built within organisations to meet business demands
Niche suppliers providing frameworks; try a Google search – 512,000 hits this week
– Seen a handful that appear mature– Latest review by Paul Herzlich (OVUM analyst)
Market Leaders such as Mercury developing Business Process Tester (BPT)
This is the industry direction now
![Page 14: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/14.jpg)
14
Automation Frameworks
in Action
![Page 15: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/15.jpg)
15
The project
Lloyds TSB Offshore in Channel Islands Migration from current banking platform IBIS to Temenos Globus
G13 Tight deadlines due to EU Savings Tax Directive coming into force
on 1st July 2005 Business processes required additional Globus modules Multiple releases leading up to deadline required substantial testing SQS-UK contacted to provide consultancy
![Page 16: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/16.jpg)
16
The Team
SQS-UK recognised the requirement for test automation Joint team from Odin Technology and SQS-UK assembled Working alongside Lloyds TSB staff Odin Technology provided testing technology SQS-UK provided implementation resource and testing know-how
![Page 17: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/17.jpg)
17
Project Approach
Split project roles for Automation Technicians and Testers– Three phase technical approach
Tool Evaluation & Technical Feasibility Technical Implementation Support
– Test Analysis and Design throughout
Tester
Technician
ToolEvaluation
TechnicalImplementation Support
Test Analysis & Design
Project Time
![Page 18: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/18.jpg)
18
Project Phase 1:Tool Evaluation and Technical
Feasibility
![Page 19: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/19.jpg)
19
Tool Evaluation
Automation isn’t always straightforward! GUI Tools don’t work “out of the box” for all environments They will almost certainly require some custom functionality to
interact with the SUT The Evaluation process:
– Assess what custom functionality is required – Assess what involvement is required from an Automation
Technician to provide the custom functionality– Assess if there are any other efficiency improvements that can be
made through application of technology This provides the scope of technical implementation phase
![Page 20: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/20.jpg)
20
Project Phase 2: What would normally happen...
![Page 21: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/21.jpg)
21
Typical Automation Project
Take existing Manual Scripts Rework by hand into automated scripts & function libraries
The bulk of the work…
Building the Object Map– Providing the Mapping between the business terms used to
describe the SUT Interface and the test tools requirements Coding Test Scripts
– Providing the instructions and associated data for the test tool to interact with the SUT and perform testing
![Page 22: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/22.jpg)
22
Object Mapping
Providing the Mapping between the business terms used to describe the SUT Interface and the test tools’ requirements
Application
![Page 23: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/23.jpg)
23
Example
Manual Tester
Test Tool
“The Login Window”
class: MSWDialoglabel: “Login.*”
![Page 24: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/24.jpg)
24
Example 2
Manual Tester
Test Tool
“Username”
class: HTMLEditid: Lgn_Uid
![Page 25: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/25.jpg)
25
Object Mapping in a typical Automation Tool
Usually implemented under different names in different tools– GUI Map– Object Repository– Object Map
Mapping an Object – Start the Object recognition tool– Point at the Object with the Mouse Pointer– Select the Object– Give the Object a meaningful business name– Make manual amendments to the properties if required
![Page 26: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/26.jpg)
26
Test Script Creation
Providing the instructions and associated data for the test tool to interact with the SUT and perform testing
AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1
AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1
![Page 27: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/27.jpg)
27
Scripting Approach
Use the Tools Scripting Language to construct business processes
Function Login(ByVal UID As String, ByVal PWD As String)
Dim rc rc = wait_window(“Login”,30) if rc <> 0 Then
log_msg(“Window Not Found”, FALSE)Return
End if focus_window(“Login”) click_on_text (“UserName”) type(UID & “{Return}”) click_on_text (“UserName”) type(PWD & “{Return}”)
End Function
![Page 28: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/28.jpg)
28
System Under Test
Test Automation Tool
AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1
AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1
1. Open Application2. Enter Userid for Authorised User3. Enter Password for User4. Click Login5. Select Create new user from menu6. Enter postcode and house number7. Tick “Secure” user authority8. Enter unique password9. Hit Generate userid10. Validate userid is 8chars long11. Write down userid for later use
Script 1.1
1. Open Application2. Enter Userid for Authorised User3. Enter Password for User4. Click Login5. Select Create new user from menu6. Enter postcode and house number7. Tick “Secure” user authority8. Enter unique password9. Hit Generate userid10. Validate userid is 8chars long11. Write down userid for later use
Script 1.1
1. Open Application2. Enter Userid for Authorised User3. Enter Password for User4. Click Login5. Select Create new user from menu6. Enter postcode and house number7. Tick “Secure” user authority8. Enter unique password9. Hit Generate userid10. Validate userid is 8chars long11. Write down userid for later use
Script 1.1
Reworking of ManualScripts
Application
ObjectMaps
![Page 29: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/29.jpg)
29
Project Phase 2: What was done differently…
Taught an old dog some new tricks…
![Page 30: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/30.jpg)
30
Test Design Model
A simple structured way of defining tests that is independent of the interface and the means of execution
![Page 31: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/31.jpg)
31
Features of the Test Design Model
Easy to design tests without technical knowledge Well structured Can be used to describe tests for any interface
– Graphical User Interfaces (GUI)– Non-UI Components (WebServices, APIs etc.)
Independent of Execution Mechanism– Manual testing– Automated testing
Not theory - proven 7 year history of implementations In use in UK and Europe at many organisations for defining
testing
![Page 32: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/32.jpg)
32
Test Design Model
Test
Test steps
A single test step
Object
Action
Data
Sub-tests
![Page 33: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/33.jpg)
33
Test Model – GUI Example
TestLogin – enter user credentialsProduct search – enter product IDProduct details – validate product infoLogout
Sub-testLogin as user 1
StepEnter Username “jsmith”
![Page 34: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/34.jpg)
34
A test step
Object + Action + Data Action
– SET enter data, navigation– GET retrieve data– VAL compare expected/actual data
![Page 35: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/35.jpg)
35
How tests are designed
Using Microsoft Excel Business processes are modelled Sequences of steps and data are built up in Sub-Test Tables Sequences of Sub-Tests model business processes
[]Login SignInaction action data action data actionset set jsmith set testing1 set
Username Password
![Page 36: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/36.jpg)
36
Interpreting the Model
There are products that can process the model– To generate Test Automation Scripts
For UI Testing using GUI Automation tools For Non-UI Testing using Harnesses
– To generate documentation for manual testing
![Page 37: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/37.jpg)
37
Interpreting the Model
Manual Scripts Non-UI AutomatedScripts
AutomatedScripts
![Page 38: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/38.jpg)
38
Globus A Self Descriptive Application
![Page 39: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/39.jpg)
39
What is a Self-Descriptive Application?
An application that, through an automated process, can provide a description of its interfaces
Application
![Page 40: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/40.jpg)
40
The Globus User Interface How it works
GlobusDesktop
Application
Application
A description of the interface for an application is
stored in the database
The Globus Desktop interprets the description and
dynamically creates the user
interface
Windows and objects are
presented to the user
![Page 41: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/41.jpg)
41
How Globus Describes its Interface
CustomInterpreter
GlobusDesktop
Application
API
API can be invoked to provide the
description without presenting it
![Page 42: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/42.jpg)
42
A Sample of Globus Information
APP: CUSTOMER_INST
FIELD CUST_SNAMETYPE INPUTLABEL SurnameLENGTH 20SEQ 1
FIELD CUST_GENTYPE OPTIONLABEL GenderLENGTH 1SEQ 7
![Page 43: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/43.jpg)
43
How this Information was used
The Description gives us 3 things:
– The business names used for the objects– The properties the test tool required for object identification– The sequence of the objects in the window
This could provide us with two things automatically:
– The Object Map– The sequence of steps for our TDM sub-tests
![Page 44: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/44.jpg)
44
More on Self-Descriptive Apps
Requirement on development to “name” objects in business terms
Microsoft .NET Applications Windows Forms and Web Applications can provide us
with– Object Maps– Sub-tests for the Test Design Model
This can drastically reduce the cost of automation
![Page 45: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/45.jpg)
45
Axe
Microsoft Excel
Globus
AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1
AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1AxeMainAPI.TestBegin "6"rc=AxeMainAPI.CheckDependency("5")If rc <> 0 Then
ExitRun(rc)End IfAxeMainAPI.BasestateBegin "Home"Browser("Browser").Basestate(AXEDIR & "samples\OdinPortal\html\index.html")AxeMainAPI.BasestateEnd rc,"",""if rc <> 0 then
rc=AxeMainAPI.TestAbort
Script 1.1
Test DesignModel & Object Maps
Automation Scripts
Overall - The Test System Design
![Page 46: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/46.jpg)
46
Project Phase 2:Implementation
![Page 47: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/47.jpg)
47
The Implementation
Technical Implementation– 9 days of Test Tool Technician
Using Automatic Interface Description in Globus– 1800+ Objects Mapped automatically– 90 Sub-test components generated
30-75 steps per Sub-test
– Time to Generate: 42 seconds
![Page 48: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/48.jpg)
48
What was left for the Testers?
Test Design!– Designing scenarios for testing the application– Providing Test Data– Sequencing Sub-Tests to form business processes
![Page 49: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/49.jpg)
49
Designing the Tests
4 Testers – Using the Test Design Model No previous automation experience
40 Days Elapsed time
1012 Complex Banking Scenarios Automated
![Page 50: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/50.jpg)
50
Raw Statistics – Test Execution
Single Script– Automated = 80secs– Manual = 480secs– Automated is 6x Faster
Full Regression Pack– Automated = 22 Hrs (1 Workstation)
= 5.5 Hrs (4 Workstations)– Manual = 132 Hrs
= 22 Man/Days (using 6hrs productivity per day)
22 Man/Days Testing in 5.5 Hrs.
![Page 51: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/51.jpg)
51
Impact Upon You
![Page 52: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/52.jpg)
52
Recommendations
Find out your organisations direction Decide upon your own future
1. Increased management skills
2. Increased non-functional skills
3. Increased business skills
Ensure you understand how Automation Frameworks support testing so you have a view when asked, because you will be….!
![Page 53: Susan windsor soft test 16th november 2005](https://reader035.vdocuments.site/reader035/viewer/2022070312/553acbdd4a7959842a8b45d8/html5/thumbnails/53.jpg)
53
Thank YouThank You
Susan Windsor Susan Windsor WMHL Consulting WMHL Consulting Limited, MDLimited, [email protected]
Case Study provided by Odin Technology Ltd Case Study provided by Odin Technology Ltd www.odin.co.uk www.odin.co.uk