selenium detox for achieving the right testing pyramid
Post on 17-Jan-2015
521 Views
Preview:
DESCRIPTION
TRANSCRIPT
Manual Checking
End- to End Selenium Tests 70%
Integration Tests20%
Unit Tests10% Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests
4%
UI 1%
Naresh Jain @nashjain
Selenium Detox for Achieving the Right Test Pyramid
Selenium Detox for Achieving the Right Test Pyramid
Naresh Jain@nashjain
Achieving the Right Test Pyramid
Aditya Saigaonkar
Kirtesh Wani
Background
IDeaS is capable of applying 25 years of experience,
expertise and client dedication toward an expanding suite of automated revenue management solutions.
Background
Background
Background - 2011
5%
95% No Automation
Automation Coverage
Background
1 Months2 Months
0 0.5 1 1.5 2 2.5 3 3.5
Months
Typical Release Cycle
Development Regression
Background
Critical Issues reported from Production
0
2
4
6
8
10
12
Critical Data Issues Cosmetic Localization
Issues
Background
After Automation Introduced - 2013
Selenium Tests 45%
Unit & Integration Tests 20%
No Automation 35%
Automation Coverage
After Automation Introduced
2.25 Months 3 Weeks
0 0.5 1 1.5 2 2.5 3 3.5
Months
Typical Release Cycle
Development Regression
Critical Issues reported from Production
0
2
4
6
8
10
12
Critical Data Issues Cosmetic Localization
Duration…
0 0.5 1 1.5 2 2.5
Years
Duration of Implementation
2 Years
Demo
• Selenium WebDriver Demo
State of the Product
Tightly coupled with UI
Browser Dependent Data tests
False fails
Difficult to pinpoint real culprit
Difficult to maintain and time consuming as test are fragile
Lagging with current release development
Image Courtesy: Alex Yakyma
Analysis of Bugs Report
10%
90% Data Failures
Bug Report Analysis
UI Navigation Failures
State of the Product
Manual Checking
End- to End Selenium Tests 70%
Integration Tests20%
Unit Tests10%
Manual Checking
End- to End Selenium Tests 70%
Integration Tests20%
Unit Tests10% Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests
4%
UI 1%
Typical test strategies lead to an inverted testing pyramid
This is the need of the hour….Courtesy: Naresh Jain @AgileFAQs.com
The Test Pyramid
Our transition story…
• We decided to "Test things right , at right place on right time" which would result in "right test pyramid“
Unit Tests 70%
Domain Logic Acceptance Tests 10%
Integration Tests 9%
Workflow API Tests 6%
End to End Flow Tests
4%
UI 1%
Our transition story…
Application UI
Service Layer
DAO Layer
Automated UI Tests
Tightly Coupled
Fragile
Lag with current development
Automate One Level Below UI
UI Independent
Robust & Stable
With current dev cycle
Fast Execution
Introduced BDD with Cucumber
Dev & QA Pairing
Developer QA
Quick Demo
Current Status
Selenium Tests 10%
Workflow Tests: 40%
No Automation 20%
Unit & Integration Tests 30%
Test Automation Status
After Right Testing Pyramid
1 Week2.75 Months
0 0.05 0.1 0.15 0.2 0.25 0.3
0 0.5 1 1.5 2 2.5 3 3.5
Months
Typical Release Cycle
Regression Time Development
Critical Issues reported from Production
0
2
4
6
8
10
12
Critical Data Issues Cosmetic Localization
Duration…
0 0.5 1 1.5 2 2.5 3 3.5
Months
Duration of Implementation
3 Months
Pain points
• Learning curve for the team
• Difficult to implement in case where no API is exposed or tight coupling with UI
• Heavy dependency on pre-populated database
Looking ahead
• Expose RESTful APIs for all upcoming features
• Move BDD specs to lower layers
• Moving away from baseline database
Courtesy: http://3.bp.blogspot.com/
© Copyright Integrated Decisions and Systems, Inc. (IDeaS – A SAS COMPANY)
Visit IDeaS online at www.ideas.com
Questions?
Aditya.Saigaonkar@ideas.com
Kirtesh.Wani@ideas.com
Naresh Jain @nashjain
top related