test automation infrastructure - step-in forum · q&a. step-in summit 2008 ... an intuit case...
TRANSCRIPT
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 1 of 42
1
- Sudhir Patnaik, Narendran Bhojan
Test Automation Infrastructure
2
Agenda
Test Automation Infrastructure – Introduction
The Need
Setting up a Test Automation Infrastructure (Test Center of Excellence – TCoE)
TCoE OfferingsTCoE Architecture, other technical aspectsThe Silk frame
Benefits/Metrics
Challenges
Next steps
Conclusion
Q&A
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 2 of 42
3
Test Automation Infrastructure
It is NOT:The only solutionA radically different approach to development/ QA
It is:A process that enables the team members to take firm control of the automated testing and product qualityA framework that allows execution of automated test suitesA vehicle to collect “automation” metrics to provide insight and opportunities for improvement
4
Test Automation Infrastructure
GUI Applications
Non-GUI Applications
Web-based Applications
Multiple platformsUnix/Linux/Windows
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 3 of 42
5
Need for Automation Infrastructure
1. Building Quality vs. testing qualityCurrent Methods
All types of testing
Testing Quality
Emerging TrendsBuilding Quality instead of testing quality
2. Lack of process DisciplineStructured vs. unstructured
Test Execution
Test Metrics
Test Standardization (scripts)
3. Quality compromise due to environment dependencies
6
Requirements for the Infrastructure
Improve Quality -> Higher Productivity
Predictability
Visibility
Efficiency
Continual Improvement
Self-directed teams
Mindset change
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 4 of 42
7
An Intuit Case Study
8
What the TCoE team offers?The TCoE team builds and maintains an infrastructure that supports:
Automatic execution of tests (BATs/FSTs) for daily builds
On-Demand execution of tests
Centralized test set creation/test execution
Centralized results management
Developer build tests
Efficient usage of hardware resources
User Permissions
Administration
Ability to add any new tools into the infrastructure
An infrastructure that supports multiple geographic sites
A standard that can be followed across the company for all automation needs
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 5 of 42
9
Some definitions
BATs – Basic Acceptance Tests:A set of basic tests that is run on all builds. Only when these tests pass, other tests are run on the build (Approximately 2-3 tests)
FSTs – Feature Sanity Tests:A set of more detailed tests specific to any area/feature (Approximately around 20 tests)
Smokes:A set of more detailed tests than the FSTs
Regression:The complete set of tests
10
Automatic execution of BATs
Runs automatically when a new build is available
Runs daily on a predefined set of branches
Tests need to be run is completely configurable
Machines are locked for triaging purposes
Configurable E-mail notifications
Results are viewable on the web
Results can be updated based on triage
Supports multiple products/branches
Configurable OS and sku’s
Note: All the test files are part of the build
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 6 of 42
11
BATs processing
1. Request Generated
-Build started by SCM (.FIN)
2. Allocate resources
-Machines allocated-Copy build to lab
3. Execute request-configure machine
- install build-Test configuration (e.g.
install Office)-Execute tests
4. Generate Results
5. Post process-If pass, then other
scheduled requests are kicked off
-If fail, machine is locked for triage
6. Cleanup-Image OS
-Free machine
Request
Process
Post-Process
12
BATs/FSTs Web center– results page
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 7 of 42
13
Automatic execution – tests configuration
Tests needed to be run after the BATs pass for every build can be configured through a simple text file
Every site and every branch has a separate config file
Task preferences can be set as to what is a primary test. This is used to update the SCM database
Developers can sync to builds that have only passed the primary test for further development
Config files are all under source control.
taskPrefs file for different sites:
14
On Demand test executionAll automated tests written by QA are integrated to the system (made available through the web pages)
These tests are available as test sets for all the engineers
They are grouped under three categories - FSTs, Smokes and regression
Only builds that have passed primary tests are made available inthis page
Request confirmation e-mail is sent to the requestor along with a link to the test status page:
Follows the same process as BATs except that the request comes from the web
We can abort the task in between if required
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 8 of 42
15
On Demand test execution contd…On Demand Page
16
On Demand test execution contd…Request Status Page
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 9 of 42
17
On Demand test execution contd…
Once the test is complete an email notification is sent to the requestor
If the test has passed the machine is freed after copying the result files to the network/file server
If the test has failed the machine is kept locked for a period of 6 hours for the engineers to triage
The lock time may be extended if required through the web
18
On Demand test execution contd…On demand ID and task ID
On Demand ID uniquely identifies the request you submitted
Task ID is assigned to each task or test suite that is executed
A single On-Demand ID may have multiple Task ID’s
Each task can run on multiple machines in parallel
If the test is taking very long, we can convert it to run on two machines in parallel and decrease the test execution time depending on the machine resources available
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 10 of 42
19
Centralized test set creation/test execution
20
Creating new test set
1. Enter test set name:
2. Select the applicable projects/products for the test:
3. Choose test type, and other options:
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 11 of 42
21
Creating new test set contd…
Edit the XML template and save it:
22
Centralized test execution
Once a test is added to the system it is accessible for all the engineers across all sites
Engineers have the option to execute tests On-demand, use the Task Builder or through the Developer Build Test (DBT) requests
QA Engineers use On-Demand for their automated testing
QA Engineers use Task Builder for their manual testing needs and cloning the task
Developers use the DBT’s for verifying their changes before checking-in their code
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 12 of 42
23
Task Builder
Two types:
QA task Builder – Used by the QA engineers for setting up the systems for performing some manual testing
Automation Task builder – Used to clone tasks that had failed already in order to triage again
24
QA task builderSimilar to the On-demand systemUses a separate pool of machines called the QA poolNo automated tests are available hereMachines can be locked for longer periods of time (weeks together)The important usage is to have a clean machine and install the product in it automatically so that they can perform manual testing
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 13 of 42
25
Automation task Builder
Similar to the QA Task builder
But the test sets are available here to be run
Is mainly used to clone tasks for triage purpose
26
Centralized results management
Results of all the tests are available on the web
Ability to search for results using task ID or Request ID (On-Demand ID)
Results filtering options
Separate results (called BATs/FSTs Web center) page for tests run automatically on the builds
Results show details of tests and sub-tests run and their status
Also the results link has a link to the directory on the file server where the result files are copied
Also has a link to open the machine using VNC viewer
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 14 of 42
27
Results Page
28
Developer Build testsDeveloper Build Test (DBT) is a tool that allows a developer to run automated tests against their local QuickBooks build. This way an engineer can verify that their changes do not adversely affect the build prior to checking in their changes Helps to verify bug fixes to BATS, FST or other test areasVerify changes on different Operating SystemsVerify changes in a non-developer environmentGive Tools/QA a heads up at coming UI changes which may adversely affect the existing tests, allowing time for automation engineers to update effected areas prior to the change appearing in the build Multiple DBT tests can be run in parallel
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 15 of 42
29
How DBT works
30
The DBT Request web request form
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 16 of 42
31
Tracking the DBT request
32
Results page
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 17 of 42
33
Efficient usage of hardware resources
The automation lab consists of both physical machines and virtual machines
Currently there are 15 physical machines and 65 Virtual machines
5 ESX servers are installed each with 13 virtual machines
Each virtual machine has 1 GB RAM and 40GB hard disk space
Few machines are also configured with 512MB RAM
34
Automation lab setup
The lab is split into QA pool and automation pool of machines
QA pool of machines is used by QA engineers for manual testing purposes
Automation pool is used for all the automated runs
Automation Pool QA Pool
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 18 of 42
35
Machine Usage
Machines are re-imaged after every test. The entire process is automated
The Operating systems that every machine can install can be configured through the “machine operating system Configuration page”:
36
Machine Usage contd…Adding a new machine:
Purpose can be QA or GeneralMachines marked as QA will be available in the QA poolMachines marked as General will be available in the automation poolMachines “permalock”ed are permanently locked and not usable
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 19 of 42
37
Machine usage contd..
Machines in automation pool cannot be locked for more than 6 hours
In order to lock for more time engineers contact the TCoE team through the ticketing system
An admin user can lock the machines for any number of days
Machines in QA pool can be locked for longer periods of time
38
Admin pages
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 20 of 42
39
Tools integration
Currently we have integrated the following tools in to the infrastructure:
Silk TestPowershellNunitAnd few in house tools like Dart, Qite and automator
All GUI tests are written in Silk Test using a Silk Frame built by the TCoE teamA coding standard is followed in the Silk tests so as to make the integration effort easierAny tool integration is done by the TCoE team who wrap the tools around scripts( mostly Perl) and execute tests so that the infrastructure understands the test failures and reporting back on the web pagesAny tool can be integrated to the system with ease
40
Support for multiple sites
Web server is the same for all the sites
Each site can have it’s own automation lab setup
Test sets are common and available in all sites
Products and Projects are common and available in all sites
Admin pages are common for all sites
Depending on which site we are in, tests are scheduled in that sites automation lab using the sites local builds and other test related files from the local sites local file server
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 21 of 42
41
A single point of access for all automation across the company
All Automation efforts across a company can be ultimately integrated into the infrastructure
A common coding standard
Better utilization of hardware resources in the automation lab
Common point of access for all historic results
Better management automation suites
A centralized way to share tests across multiple sites
A framework setup for continual improvement
Scope for integration into other test case management tools like Quality Centre or Test Link
42
Architecture and other technicalAspects
Distributed system architecture
The application server
The web server
The database server
The backend server scripts
Client side scripts
Machine cleaning/re-imaging
The test execution flow – Automatic BATs
The test execution flow – On-Demand
The test execution flow – DBT
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 22 of 42
43
Architecture
43
QA/Dev Application ServerAcquires Resourcesand Executes Test
UserEnters
Request
Display Results & Notify User
Record Results
Acquire Physical Machine
Resources
Execute Teston AcquiredResources
Acquire Physical Machine
Resources
Execute Teston AcquiredResources
Mountain View Bangalore
Build Sys
44
Architecture
DBserver
Web server
Appserver
MTV Autolab
Bangalore app server
Bangalore autolab
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 23 of 42
45
Application server
All the backend scripts run on the application server
There is one application server in every site
The application server takes care of the following things:
Adding new test requests to the DBChecking for new requests and allocating machinesStarting the tests on the clientsChecking for new builds and copying them to file serverProcessing automatic execution of BATs based on task preferencesCleaning machines/re-imaging services
46
Web server
There is a single web server (Apache) residing in MountainView
All pages are built using PHP
Provides the user interface for the following:Triggering tests On-DemandTracking requestsViewing the resultsRunning DBTsGeneral AdministrationViewing metrics Locking machines for manual testingTest sets managementRelated documents
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 24 of 42
47
DB Server
A common DB server is used located in MountainView
MS SQL Server 2000 is used
Stores all the related data such as:TestsResultsTasksMachine infoUser infoProducts and projects infoBuild infoGroups and permissionsOS info
48
Client Side Scripts
Listener.vbs
Host_run.vbs
Clean_image.vbs
Mount.vbs
Master.wsf
WinPEStartup.vbs
WinPEImage.vbs
TestExecutionEngine
TestResultWriter
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 25 of 42
49
Listener
Start
Look for command files in the commands share on app server
NewCommand File
available
Run the command on local machine
No
Yes
Delete the command on the commands share
50
Machine Cleaning/Re-imaging
Ghost images are taken with the following configurations:
All drivers installedNecessary user accounts created and passwords setNecessary folders created and permissions setScreen resolutions set and screen savers disabledRequired software’s like Perl, Silk Test, ultra VNC and WinZipStartup scripts added and enabledSysprep is done: Sysprep is the process of creating hardware independent images provided by MicrosoftWinPE boot images are created on this machine which is used by clean_image.vbs during re-imaging
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 26 of 42
51
Machine cleaning/Re-imaging contd…
StartUpdate the DB with the OS to re-image
for this machine
Copy the ISO image to local drive;Modify boot.ini to boot using the ISO image file
Restart the machine
Check IP address and map network driveTrigger WinPEImage
Check OS to image from the DBMap drive to Images directory on file server
Invoke Ghost (XP) or ImageX (Vista) with appropriate arguments to re-image
and restart the machine
End
Clean_Image.vbs
WinPEStartup.vbs
WinPEIMage.vbs
52
Test Execution Engine (TEA)
This is the execution engine that will run on the client machine while executing the tests
It is a C-Sharp application
The processTask.pl starts this application on the client machines
It executes the tasks specified in the Tasks file created by processTask.pl
It takes care of fetching the test details like the actual path to test script files to execute and calls the appropriate applications to run the tests
It passes on the test results to the Test Results Writer
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 27 of 42
53
Test Results Writer (TRW)
This is another application written in C-Sharp
It communicates with the TEA through sockets
It gets the results of the tests being currently run
The results are written in an XML format and stored in the fileserver
ProcessResultQueue.pl puts these results in the database
The web parses this data and displays in the correct format when the results page is accessed for any task or request
54
The test execution flow – Automatic BATs
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 28 of 42
55
The test execution flow – AutomaticBATs contd…
56
The test execution flow – On-Demand
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 29 of 42
57
The test execution flow – On-Demand contd…
58
Remote On-Demand User
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 30 of 42
59
The test execution flow – DBT
60
The test execution flow – DBT contd…
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 31 of 42
61
Remote DBT User
62
The development environment setup
There is another replicated setup of the infrastructure with a separate app server, web server, file server, DB server and few test clients
All the infrastructure scripts are first tested in this Dev environment before they are “integrated” to Live
For web code we can setup the development environment on any machine and then integrate the changes to Live
Both the Dev and Live code are under source control and are kept in sync across all sites
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 32 of 42
63
Silk Frame
“Building test cases around a powerful test frame is the heart of what we do with SilkTest.”
The test frame components are:Product windows and Child object definitionsReusable functions for common features in the productGlobal variables for use by all the test cases
E.g.: product install path, version, build etc
RecoveryCleanup after tests
Results reporting
Application states handling
Unexpected windows handling
Functions to let the infrastructure know about test results and report in the web pages
64
Recovery System
What the recovery system does?
• Launch the Application if not running
• Close extra windows; Report which ones were closed
• Set the application active. Report which other window was active
• Execute user-defined BaseState
Report errors to the results file at conclusion of each test
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 33 of 42
65
Benefits…
66
Benefits…
Test Automation is finding more defects earlier and more cost effectively
Enabled by higher test coverage, reduced execution effort and increased test execution frequency
4 key automation metrics are captured:Automated Test coverage – The percentage of tests automatedEffort spent on automation execution -Test throughput – Number of tests executed over a period of timeDefects found through automation
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 34 of 42
67
Automated Test Coverage
0%
10%
20%
30%
40%
50% Automated Coverage Level
Lava Kona
Automated Code Coverage increased from 19% to 42%
68
Effort spent on Automation execution
0
20
40
60
Overall Main St Payroll Platform AIIR
Full Regression Test Execution Effort (FTE Months)
Lava Kona
Reduced full regression testing effort required by 44%
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 35 of 42
69
Test Throughput
050000
100000150000200000250000300000350000
Throughput
Automated Test Throughput
Lava Kona
Testing throughput has doubled
70
Defects found through Automation
9% 15%
0%10%20%30%40%50%
% of Defects Found by Automated Tests
Lava June 2006 Kona June 2007
More defects are being found by Automated Testing
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 36 of 42
71
Lessons Learnt
Problems faced when going distributed:Single central database server or multiple database servers in each siteSingle web server or multiple web serversIf going for single centralized database and web server, the WAN link needs to be really fastIf going for multiple database servers, the method to sync data between the two databases needs to be identifiedArchitecture was rigid and it was difficult to extend it to multiple sites. This was mainly because the infrastructure was not planned to be used in distributed environment.The web forms needs to be designed in such a way that they do not do lot of database query which returns large amount of data at a single stretch. These pages take extremely long time to load over the WAN All the back-end scripts which connect to the database server remotely, loose the connection often
72
Lessons LearntDatabase :
We decided to stick with a single centralized database as we found that keeping multiple databases and synching data selectively was a maintenance overhead
Web server:Decision was made to have a single centralized web server after figuring out that having a local web server which will be talking to the remote Database server will even worsen the performance of the web server to a greater extent
Architecture:Necessary integration points defined and scripts written to handle multiple sites
Web pages:Moving from PHP towards Adobe Flex
Database connection problems:Changing scripts to use Restful services
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 37 of 42
73
Next Steps…
74
Automation Lab - Virtualization
Use VMWare Lab manager/ VMLogix LabManager to effectively use the Virtual machines in the ESX server
Lab manager allows saving machine states and takes care of allocating machine resources dynamically
Machines will be created On-demand and resources allocated on the fly
Many OS configurations will be available as part of the images for tests to chose from
This will ensure better utilization of hardware resources
Also machines wil not be kept locked for triaging; When triaging is required the machine state will be saved which can be later used for triaging
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 38 of 42
75
Reduce time spent on setup
Through Virtualization there can be many number of images with different configurations
The tools allow us to store the base image and only the difference in a separate snapshot
That way the image size also remains less
When ever a new build is available the product to be tested will be installed automatically on the required OS’s and an image will be made available with the installed setup
This will reduce the time required to re-image to an OS and also the time required to install the product for all tests
76
Better Results ManagementCurrently the results for a requested task are only available after the task has completedThis is because we store the results as a single XML in the database and parse this when displaying the results for the taskWith this approach it is only possible to show results after the test is completed as the results will not be available till the xml is loaded in to the database by the TestResultsWriterSo we are working on replacing the XML approach for results and storing results in different database tablesThe results would be updated in the database when every subtest has completedThe web pages would query the different results tables to display the results rather than just parsing a single XML file
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 39 of 42
77
Reduce Triaging Effort
Currently when a test fails the reason to figure out the actual failure is sometimes difficult
The failure may be caused because of product failure, product change, script failure or infrastructure issues
The time spent on triaging is sometimes high because of insufficient logging, non-availability of resources, triage engineers knowledge on the infrastructure etc
This can be improved through better logging, automated mechanisms to narrow down the failure and better training for the triage engineers
The next step would be to have the system automatically take action like logging a defect in to the defect tracking database or even fix the issues if possible
78
Integration of test case management and automation
Currently there is no tracking between test cases that are automated and tests that are not automated.
Ideally we should track all the tests in a test case management repository like Mercury Quality Centre or Test Link
Also integrate the current automation infrastructure to the test case management tool to enable the following:
Be able to select group of tests from the Test case management tool and run it in the labBe able to view results of the tests in the test case management toolAutomatically schedule the automated tests on builds based on the test plan created on the test case management tool on different milestones an publish the results
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 40 of 42
79
Non-Windows Support
Currently the test execution engine supports only Windows
In order to support for non-Windows clients only this engine needs to be re-written to work on different Operating Systems
Remaining mechanisms will remain the same
80
Next Steps - Summary
Improving the automation lab resource usage
Reduce time spent on test set-up
Better results management
Reduce triaging effort
Integration of test case management and automation
Support for tests running on non-Windows
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 41 of 42
81
Conclusion
Key points:In order to achieve a successful automation infrastructure, think in long termBe confident and be patient to get the ROIInvest enough on resources and get the design and architecture rightHave a dedicated Automation development teamKeep on evaluating new tools which would improve the infrastructureThink in terms of company’s future roadmaps and the products that you will need to supportThese steps will ensure that we are on track for the next generation of test automation requirements
82
Q & A
STeP-IN SUMMIT 2008 Test Automation Infrastructure
STeP-IN Forum & QSIT Copyright Page 42 of 42
83
Thank You..
Sudhir PatnaikGroup Manager-QA
Narendran BhojanSoftware Engineer - TCoE