(ism318) measuring software quality w/ rainforest qa & mechanical turk
TRANSCRIPT
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Russell Smith and Fred Stevens Smith, Rainforest QA
October 2015
ISM318
Continuous QAMeasuring Software Quality with Rainforest
QA and Amazon Mechanical Turk
RainforestContinuous QA as a Service
Rainforest QA and Mechanical Turk
rainforest
Who we are
○Founded in 2012, HQ in San Francisco
○Team of 23
○$4m seed backing from YC, Storm, a16z
○50k testers, 5m jobs pw, 20% MoM $ growth
Fred Stevens-
Smith
CEO, co-
founder
Russell Smith
CTO, co-
founder
rainforest
Quality Assurance -
the maintenance of a chosen level of quality in
a service or product
tldr: making sure your product or service
“works” as expected
What is QA?
rainforest
Continuous Integration -
Check things build and pass tests
Continuous Delivery -
If it builds and passes, ship it
Traditional QA does not work with CD
What is CI / CD?
rainforest
Pick one:
Move Fast and Break things
or
Move Slow and Don’t Break Things
You shouldn’t have to make this
compromise:
Continuous QA is the answer.
The Continuous Delivery Paradox
rainforest
Why Continuous QA?
○Faster releases
○Better quality
○Catch x-browser regression bugs before
production
○Happier product teams & happier customers!
○QA: Cost center -> competitive advantage
○Bottleneck -> Seamless part of CI -> CD
rainforest
Continuous QA wishlist
○Test case management, reporting and
execution
○24/7 availability
○Super fast
○X-browser
○Comprehensive API
○Integration with existing CI / CD process
rainforest
Rainforest
○Usable by all - no code
○Results in < 30m on average
○Powered by 50,000 testers from Mechanical
Turk
○24 x 7 availability - run anytime, no waiting
○Fully cross-browser
rainforest
Where does Rainforest fit?
rainforest
Mechanical Turk
●API for microtasks
●Powered purely by humans
rainforest
Rainforest & Mechanical Turk
○Rainforest proprietary systems for ● Automated on-going training and management
● Algorithms for pairing testers together smartly
● Years of worker history and behavior data
● Reputation calculations
● Tooling for testers
● Tester lifecycle
rainforest
Rainforest Tooling
○ All run in custom VM infrastructure
○ Connected direct to the testers browser
○ Fully controlled and recorded
○ Automated screenshots, network recording
○ Desktop
● Support for all major browsers
● Enterprise support for custom builds
○ Phones, tablets
● Android 4.4 support
● Android 5 and iOS coming soon
rainforest
Traditional Workflow
1. Write code
2.A QE or QA engineer writes tests
3.Push to CI
4.Push merge to a branch
5.Schedule QA (1h+ ?)
6.Wait for QA (1-72h+ ?)
7.Deploy if QA passes (go to #1 or #2 if fail)
rainforest
Rainforest Workflow
1. Write code
2.Developer / PM writes tests in Rainforest
3.Push both to CI
4.Rainforest runs in CI (< 30m)
5.Deploy if Rainforest QA passes
rainforest
Who we do it for
rainforest
ROI from Continuous QA
Average speed of Rainforest to run 1000 tests in 5 browsers = 2.5
hrs (2.5 weeks with human QA)
Very low cost of ongoing maintenance (vs automation)
Zenefits has 1 QA person, ~50 engineers. Would have ~16 QA
otherwise. Save $2.5m.
Demo
rainforest
Customer Experience: creating tests
rainforest
Did my tests pass or fail?
rainforest
Which tests passed or failed?
rainforest
...in which browsers?
rainforest
..and what was the tester feedback?
Questions?