QUALITY ENGINEERING CHALLENGES IN E-COMMERCE
Yotam Sharan,
Quality Engineering ManagerMay 2013
Ver. 1.3 for publish
WHAT CHALLENGE?
WHAT’S THE BIG DEAL?
WHAT CAN YOU TELL US?
QA documentation
Run sanity tests
Full-regression time
COTS test tools
Test Automation
Mocks and tools
No
Developers
Max 1 day
No
X-org F.W.
QE does
Quality Engineering Challenges 4
Enable merchants to succeed in new commerce env’t
Enable consumers to buy anything, anywhere, anyhow 110M
ACTIVE USERS
25M SELLERS
$175B COMMERCE VOLUME
EBAY MARKETPLACES
Quality Engineering Challenges 5
DATA CHALLENGE
• Big Data
• Various languages
• Numerous processes
• Subsystems dependencies
• QA env. is partial & detached from live data
• Daily ETL / deployments
• Manual is too long
Quality Engineering Challenges 6
AUTOMATED GOLDEN SET
• Copy Production data once, as baseline
• Use/reuse static dataset & manipulate variables (e.g. ID, timestamps)
• Sample specific data, evaluate large sets/volumes
• Automate predefined data input – check against expected output
• Mocking
• Monitors
• Reporting: test results, logs, asserts, notifications
• Smoke test and monitor Production
Tools we use: Selenium/WD, SoapUI, Jbehave, Jmeter
• Exploratory testing
• Regression testing
• Compatibility testing
• Automation testing
• Scenario testing
• Unit testing
• Code review
• Paired testing
• Test-driven development
• Bug bashes
• Capacity planning
• Load and Performance testing
• Accessibility testing
• Mocks and stubs
• Analytic testing tools (FindBugs, HTTPWatch, CAL, FireBug, Sonar)
TEST INFECTION • Integration testing
• E2E testing
• Smoke testing
• PRD review
• Test case review
• Dogfood testing
• Sprint demo
• Security testing
• Usability testing
• Definition of Done
• CI monitors
• Production monitors
• Live Site Testing
• Crowdsourcing
Automation testing
Automate everything, regression first. Plan the right automation approach.
Business Value
Providing thin vertical slices of functionality with business value, every iteration.
Continuous Integration
Constant integration testing to assure build integrity and quality, every drop.
Design for Testability
Ensure that it is easier to develop and apply tests for the designed product.
Early Start
The whole-team approach engages test engineers at early design phases.
AGILE DEV & TESTING PRINCIPLES
Everyone needs to let go of titles and self-image and engage in a constant
feedback, open, transparent, punishment-free work environment.
Developer test and Testers develop
Analysts, programmers and PM write test scenarios
All engineers help define requirements
Quality is everyone’s responsibility
QUALITY DRIVEN APPROACH
Quality Engineering Challenges 10
Sequential Execution
Test execution: 7000 minutes
WEBDRIVER GRID
Parallel Execution
Test execution: 60 minutes
150 VMs/Browsers on WD Grid
Quality Engineering Challenges 11
WEBDRIVER GRID
Selenium Grid Hub
Log
Analyzer:
test logs,
stack trace,
server logs,
configurat’n
WD
Listeners
Quality Engineering Challenges 12
INTEGRATED QE PROCESSES
• Automation Framework for component reuse • Generic test cases for data-driven testing• Fully-integrated env. for anyone to run tests easily
“In God we trust; all others must bring data” – W. Edwards Deming
Gut feelings…
Engineering clockspeed
Adopting/Adjusting best practices
Instant feedback: CI, Pair-x, RT dashboards, Quality indicators (bugs, code QI)
Constant feedback: Stories, Planning, Test results, Reviews, Retrospective, Surveys
FEEDBACK & FINE-TUNE
CONTINUOUS INTEGRATION & CODE QUALITY
Quality Engineering Challenges 15
I’d love to hear from you www.ebay-m.com/yotam
Win the draw!
Work @eBay:
Questions?