(ism318) measuring software quality w/ rainforest qa & mechanical turk

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Russell Smith and Fred Stevens Smith, Rainforest QA

October 2015

ISM318

Continuous QAMeasuring Software Quality with Rainforest

QA and Amazon Mechanical Turk

RainforestContinuous QA as a Service

Rainforest QA and Mechanical Turk

rainforest

Who we are

○Founded in 2012, HQ in San Francisco

○Team of 23

○$4m seed backing from YC, Storm, a16z

○50k testers, 5m jobs pw, 20% MoM $ growth

Fred Stevens-

Smith

CEO, co-

founder

Russell Smith

CTO, co-

founder

rainforest

Quality Assurance -

the maintenance of a chosen level of quality in

a service or product

tldr: making sure your product or service

“works” as expected

What is QA?

rainforest

Continuous Integration -

Check things build and pass tests

Continuous Delivery -

If it builds and passes, ship it

Traditional QA does not work with CD

What is CI / CD?

rainforest

Pick one:

Move Fast and Break things

or

Move Slow and Don’t Break Things

You shouldn’t have to make this

compromise:

Continuous QA is the answer.

The Continuous Delivery Paradox

rainforest

Why Continuous QA?

○Faster releases

○Better quality

○Catch x-browser regression bugs before

production

○Happier product teams & happier customers!

○QA: Cost center -> competitive advantage

○Bottleneck -> Seamless part of CI -> CD

rainforest

Continuous QA wishlist

○Test case management, reporting and

execution

○24/7 availability

○Super fast

○X-browser

○Comprehensive API

○Integration with existing CI / CD process

rainforest

Rainforest

○Usable by all - no code

○Results in < 30m on average

○Powered by 50,000 testers from Mechanical

Turk

○24 x 7 availability - run anytime, no waiting

○Fully cross-browser

rainforest

Where does Rainforest fit?

rainforest

Mechanical Turk

●API for microtasks

●Powered purely by humans

rainforest

Rainforest & Mechanical Turk

○Rainforest proprietary systems for ● Automated on-going training and management

● Algorithms for pairing testers together smartly

● Years of worker history and behavior data

● Reputation calculations

● Tooling for testers

● Tester lifecycle

rainforest

Rainforest Tooling

○ All run in custom VM infrastructure

○ Connected direct to the testers browser

○ Fully controlled and recorded

○ Automated screenshots, network recording

○ Desktop

● Support for all major browsers

● Enterprise support for custom builds

○ Phones, tablets

● Android 4.4 support

● Android 5 and iOS coming soon

rainforest

Traditional Workflow

1. Write code

2.A QE or QA engineer writes tests

3.Push to CI

4.Push merge to a branch

5.Schedule QA (1h+ ?)

6.Wait for QA (1-72h+ ?)

7.Deploy if QA passes (go to #1 or #2 if fail)

rainforest

Rainforest Workflow

1. Write code

2.Developer / PM writes tests in Rainforest

3.Push both to CI

4.Rainforest runs in CI (< 30m)

5.Deploy if Rainforest QA passes

rainforest

Who we do it for

rainforest

ROI from Continuous QA

Average speed of Rainforest to run 1000 tests in 5 browsers = 2.5

hrs (2.5 weeks with human QA)

Very low cost of ongoing maintenance (vs automation)

Zenefits has 1 QA person, ~50 engineers. Would have ~16 QA

otherwise. Save $2.5m.

Demo

rainforest

Customer Experience: creating tests

rainforest

Did my tests pass or fail?

rainforest

Which tests passed or failed?

rainforest

...in which browsers?

rainforest

..and what was the tester feedback?

Questions?