1 susan bockhoff, mckesson igor gershovich, connected testing enterprise-wide test automation with...
TRANSCRIPT
1
Susan Bockhoff, McKessonIgor Gershovich, Connected Testing
Enterprise-wide Test Automation with HP Quality Center Suite for Agile development projects.
Agenda
McKesson and Project Background Project Goals and Test Automation ROI Why BPT was Selected Test Automation Approach and Results Behind the scenes: the BPT architecture Creating Business Components Setting up Parameters Creating automated tests using BPT
Components
2
McKesson: Who Are We?
Largest healthcare services company in the world
Fortune 15 – $106 billion in revenues (FY09)
More than 32,000 employees dedicated to healthcare
Oldest U.S. healthcare company
Established 1833
Only company offering solutions at every point of care
Deep clinical, IT and process expertise
3
Horizon Clinicals Software Products Manage information across the healthcare lifecycle
4
Patient& Family
Pharmacy
Local Hospital
Pediatrician
Primary Care Physician / Medical Home
Grandma’s Physician
Alternative Medicine Practitioner
Our Software Is Complex ….
We develop software in multiple locations─ Westminster, Colorado─ Alpharetta, Georgia─ Roseville, MN─ Bangalore, India
Using many development tools─ Java─ .Net─ GWT─ Standard HTML
New release contains 35 different applications and over 40,000 tests
5
Why Automate Our Tests?
Revenue: Faster time to adoption will speed license revenue recognition, market penetration
Cost: Reduced cost to execute tests Agility: Ability to run regression tests more
quickly to speed time to market of our products. Repeatability: Detect regression errors more
quickly without human errors or variability. Coverage: Increase platform coverage on each
release. Increased confidence in our application quality.
6
Status at the Start of the Project
2000 working automated tests using QTP These took 3 years to develop The team was using a keyword-driven
framework Tests were scripted using VB script (no record
and playback) Tests required a lot of maintenance and many
had to be abandoned There were many false failures
7
The Challenge!
5000 new working automated tests in one year─ Must be in production and executed successfully─ Broad product coverage
Be able to execute the tests in 3 days Tests must be re-usable over multiple releases
with a minimum of maintenance Automate first – SME Centric approach Use automated testing as part of agile
development process
8
9
Business Process Testing
HP’s Business Process Testing (BPT) product was selected for this project
The QTP framework had some problems we wanted to resolve:─ Tests still required a lot of maintenance─ Tests required automation engineers to write, and
they did not know the applications well enough to know what the tests should do
─ When tests failed the failures could not be resolved by the automation engineers (did not know enough about the app to know if it was a bug)
Advantages of BPT
Components are created by automation engineers who can create them without really knowing everything the application needs to do
Tests themselves are created by subject matter experts who know what the tests need to do and how best to prioritize them.
Test failures are analyzed by subject matter experts who know if it is a software defect or a problem with the automated script itself.
10
How to Select Good Candidates for Automated Tests
App must be in a technology that is automatable (no apps requiring reading of bitmaps).
Frequently run tests are automated first since they will give the greatest payback
This requires that a full risk-based analysis be performed on your regression suite to prioritize all regression tests
High-priority regression tests (wide and shallow) are the ones that should be automated for highest ROI.
11
Frequency of Automated Tests
Environmental Health Check – 1 hr. Can you log in to all apps? Executed on every build.
Smoke tests – 4 hrs. Can all main pages be accessed? Executed every two weeks.
Critical automated regression – 6 hrs. can all major transactions be performed? Executed at start of regression testing and also last thing before release.
High priority regression - executed several times on a release.
12
Test Automation Architecture
Application Under Test
Business Process Testing (BPT)
Component Component Component
TestTest Test TestTestResources:- Functions- Object Rep.- Environment
HP Quality Center (QC)
HP QuickTest Pro (QTP)
SME-centricTest Automation Process
Aut
omat
ion
Eng
inee
rAutomation Architecture Design
Create Function Libraries
Create Object Repositories
Create Business Components
Drag Components to create tests in QC
Configure Input/Output parameters
Debug tests in QC Test Plan module
Add tests to test set in Test Lab module
Sub
ject
Mat
ter
Exp
ert
orTe
st E
ngin
eer
SME-centric Process Advantages
Responsibilities are clearly defined:─ Automation developers define architecture and create
components─ Subject matter experts create automated test cases
SME doesn’t need to know/use QTP. Uses very flexible Keyword/Data-Driven approach. Reduced learning curve due to same approach across
different applications/technologies Applicable to all areas of automated testing, like
regression, smoke, acceptance, etc. Designed for Maintainability
Automated Testing with Agile
Automation Engineers SME Testers
Shared between Agile teams
Create BPT components during each sprint
Modify components as needed
Embedded in Agile teams
Write automated tests using BPT components during each sprint
Execute automated tests at the end of each sprint
Use BPT tests as regression tests during later sprints
Benefits of New Approach
Leverage product knowledge of SMEs to write more meaningful and reliable tests
Increase test coverage -> fewer product escapes
Ability to keep up with increased product workload without increase in staff
Reduce time spent analyzing automation failures Faster execution of automated tests
The Results
The team created 5500 new, working BPT tests in one year using this approach.
With the same budget, the team increased the number of automated tests created by 5 times using this approach.
Automated tests now comprise 25% - 40% of the tests executed depending on the size of the release.
These tests are being maintained and executed consistently.
The team is now working on maintenance, speed of execution, unattended execution, and new projects.
18
Automated Testing Process
Quality Center
QTP AUTBPT Components,Automation Resources
Results
AUT Testing
Test case
Data
SETTING UP THE BPT ARCHITECTURE
Step 1
20
Test Automation Elements
IDE: Quality Center with BPT/Mercury QuickTest Professional (QTP)
Environment Files Application Areas Function Libraries Object Repositories Business Components and Test Scripts
Flexible Test Automation Lab – over 30 VMs and Desktops
22
Test Sets Environments Under TestLab Machines
QA1
QA2
QA3
Environment (BPT Test) for Computer Name–Test Environment Mapping
Computer name – Test Environment mapping
Environment.xls
Function Libraries
Technology libraries─ Web─ GWT─ .Net─ Win32
Common libraries─ Utility─ DB─ CommProtocols
Project Specific libraries
CREATING REUSABLE BPT COMPONENTS
Step 2
26
Business Components
A component is a reusable module to perform tasks on logically independent application parts (Page or Window).
Components use Input/Output parameters for data input/verification and navigation within Page or Window.
Input/Output parameters also are used to pass data between components.
Components can use iterations.
Login Component Steps(Example for Web Application)
Component initialization─ Load environment variables like URL, login
credentials, DB connection strings, etc.
Close all browsers except QC Invoke an application Login Verification that login is successful
Functional Component Steps (Example for Web Application)
Component initialization – ComponentInit() function─ Load environment variables like URL, login
credentials, DB connection strings from Environment.xls file using QC OTA
Verify that Expected page is displayed Side or Top menu navigation Web Page internal flow
BPT Component Script and AUT
AUTQTP
CREATING AUTOMATED TESTS BY CONFIGURING PARAMETERS WITHIN COMPONENTS
Step 3
31
BPT Test Case in Quality Center
Data
Parameter Naming Conventions
Prefix Object Example
lst List lst_User
txt Edit txt_User
chk Checkbox chk_User
rad Radiobutton/RadioGroup rad_User
tab Tab tab_User
tbl Table tbl_User
cell Cell in Table cell_User
file WebFile file_User
ele WebElement ele_User
cmb Combobox cmb_User
st Static Text (Read Only) st_User
.Apply Method
Object.Apply Parameter(“Parameter_Name”).WebEdit(“Name”).Apply Parameter(“txt_Name”)
Apply method functionality:─ Performs data substitution─ Verifies if object exists and visible─ Makes decision what to do with “Name” edit object
based on Parameter data
Input Constraints for Edit (Text Box) Object
Value Action Example
No action
<CLEAR> Clear edit field <CLEAR>
text Enter text Hello
[text] Verify text [Hello]
[<ENABLED>], [<DISABLED>] Verify if object is Enabled/Disabled
[<EXIST>],[<NOEXIST>] Verify if object exist/no-exist
[<REGEX>Pattern] Verify that Pattern matches text
[<REGEX>TestCase.*]
Working with Objects in Grids
row=WebTable().GetRowIndex(Parameter(“search_”),clmn)
WebTable().CellApply row, clmn, objType, Parameter(“cell_”)iRow = WebTable().GetRowIndex(Parameter(“search_Name”),2)
WebTable().CellApply iRow,3,“WebElement”,”Parameter(cell_Age”)
Parameter prefixed with the word “search_” indicates which row to work with.
─ Row can be selected by using row number: #n ─ the text contained in the specified Column─ Pattern
Parameters for objects in table cells are prefixed by “cell_”
Example: Working with Grids
Data Substitution
Special tags for Dates, Times, SSNs, Unique values, DB Queries, Random, etc.<SSN> - get substituted with unique SSN
<TODAY> - get substituted with today date (03/19/10)
<RNDDIG 3> get substituted with 3 random digits
<DateTime YYYYMMDDHHmmss>
<DB DBEnvironment::SQLStr>
Tags can be combined[<REGEX><TODAY> at .* AM]
Summary
This method provides a way for test automation engineers to create re-usable components without having to know all the ways the application will be used.
Subject matter experts can use the components to easily create tests that can do anything that the application can do without having to be software engineers.
This method allowed us to create a great number of working automated tests in a relatively short time.
The project has been a major success and has helped us achieve greater confidence in our release quality.
39
Q&A
40