|

INTRO TO CROWDTESTINGC O M B I N I N G ‘ I N - T H E - L A B ’ & ‘ I N - T H E - W I L D ’ A P P R O A C H E S

Peter Shih | VP of Channel Development | pshih@applause.com

|

• Formerly known as uTest, the leader of in-the-wild software testing

– Founded by pair of Israeli entrepreneurs

– Founded in 2007; GA Launch in 2008

• Offices around the globe

– Boston, San Francisco, NYC, Seattle, Tel Aviv, Berlin & Warsaw

• World’s largest & most mature crowdtesting company

– Raised 5 rounds of VC funding, totaling $80 million

- Latest round led by Goldman Sachs in 2014

– Larger than all other crowdtesting companies combined

2

A L I T T L E A B O U T T H E C O M P A N Y

|

3,000+CUSTOMERS

3

B Y T H E N U M B E R S

70,000+B U I L D S T E S T E D

175,000+P R O F E S S I O N A L

T E S T E R S

500 Million+A P P S T O R E

R E V I E W S A N A LY Z E D

OVER 1 MILLIONI N - T H E - W I L D I S S U E S R E P O R T E D

| 4

C O M P A N I E S T H A T U S E C R O W D T E S T I N G

Retail Travel SocialMedia

| 5

C O M P A N I E S T H A T U S E C R O W D T E S T I N G

Health/FitnessFinancialSoftware Telco

|

The Challenge

PROBLEM WITH

THE STATUS QUO

•••

|

Test Labs Are Here Users Are Here

Modern apps & sites require real-world testing that mirrors the many

different combinations of devices, use cases and locations of users

7

T H E P R O B L E M W I T H T H E S T A T U S Q U O

|

• ‘Inside-out’ approach is still valuable, but it’s merely the starting point

• 360ₒ app quality requires an ‘outside-in’ view, which comes in many forms

– Functional defect reports

– Crash reports

– Security vulnerabilities

– Load and performance issues

– Localization issues

– Usability feedback

– User sentiment analytics

• And across all phases of the SDLC

– Research

– Pre-launch

– Post-launch

8

D E T A I L S O F ‘ O U T S I D E - I N ’ A P P R O A C H

|

360ₒ

APP QUALITYQ U A L I T Y T H A T M A T C H E S Y O U R U S E R S ’ E X P E C T A T I O N S

9

A P P L A U S E E N A B L E S …

|

DESIGN & DEVELOP

1

TEST &RELEASE

2

LISTEN &LEARN

3

ANALYZE& PLAN

4

360º APP QUALITY

5

EG: IBM

EG: JIRA EG: CRITTERCISM

EG: FLURRY

10

“Should I launch?”

“What should I do next?”

S O L V I N G T W O M A S S I V E C I O P R O B L E M S

|

• Modern apps have different recipe for success

– Infinitely more variables make up the test coverage matrix

- Devices, OSes, browsers, carriers, locations, use cases

– Traditional test playbooks & methodologies are merely starting point

• Crowdtesting augments traditional QA efforts

– True for web, but even more so for mobile and smart TVs/cars/watches

– Move beyond the QA lab, closer to where users work, live & play

– Close the loop between user POV and internal definition of quality

• 360°app quality isn’t just about testing or dev ops

– Requires modern tools and analytics that focus on users-centric quality

– Necessity due to outsized user expectations + voice in app stores

3 K E Y T A K E A W A Y S

11

|

The Challenge

A 360ₒ

APPROACH

TO APP QUALITY

•••

|

I N - T H E - W I L D T E S T I N G S E R V I C E S

F U N C T I O N A L

T E S T I N G

U S A B I L I T Y

T E S T I N G

L O C A L I Z AT I O N

T E S T I N G

L O A D

T E S T I N G

S E C U R I T Y

T E S T I N G

Do your apps function flawlessly

in the hands of your users?

Are your apps as intuitive and

easy-to-use as you think?

Are your apps primed to thrive in

different markets & cultures?

Can your apps handle the peaks

& pressures of the real world?

Are your apps protecting your

users data & privacy?

|

• Real-world functional testing solutions

– Ensuring apps work well in the hands of users

• Compliments in-house testing

– Team of testers provide you with scale, coverage & expertise

• Testing on real devices, in real locations

– Across OS versions, mobile browsers, wireless carriers

– Exploratory testing & test case execution

– Test around the clock or after major releases

• Integrates with your bug tracking systems

– JIRA, Bugzilla, Mantis, Pivotal, IBM Rational, Microsoft TFS and more

• Fully-managed service to reduce your overhead

– Integrated with your test services from QualiTest

– Handle all tester communications and bug triaging

– Professional testers from any region, on any device or OS

– Integrated with your internal team, processes and systems

F U N C T I O N A L T E S T I N G A P P R O A C H

14

| 15

I N - T H E - W I L D F U N C T I O N A L D E F E C T

Learn About The Tester

Tiered

Approval

Chat Directly

With The Tester

Reproducible

Action Steps

U I N O T R E N D E R I N G P R O P E R LY

|

• Real-world usability testing solutions

– Ensuring your apps are designed intuitively for intended audience

• Expert-led usability studies

– Remote unmoderated testing performed by industry experts with an

average of 12 years of expert usability experience

• Audience matching

– Survey 25+ respondents that match the demographics of your users

• Video and audio experience captured

– Watch and analyze your respondents use your app for the first time

• Comprehensive report & consulting

– Receive a complete report with findings, benchmarks and

recommendations on how to improve user experience

– Leverage your usability expert for advice on how to minimize

potential confusion & frustration for first-time users

U S A B I L I T Y T E S T I N G A P P R O A C H

16

|

• Real-world localization testing solutions

– Validate content and app localizations to ensure correct context

across languages and cultures

• Go beyond content

– In addition to translations, verify formatting, currency conversions, taxes,

shipping options and more

• In-market localization testers

– Understand the true implications of your localization choices by leveraging

cultural natives

• Double verification

– All issues and recommendations are verified by an additional cultural native

• Available string reporting

– Output a string document to facilitate easier updating of your apps

17

L O C A L I Z A T I O N T E S T I N G A P P R O A C H

|

• Real-world security testing solutions

– Protect your users privacy and data

• Expert-led and managed security testing

– A white hat security expert will work with you to develop a

comprehensive test plan

• White hat security testers

– Trusted team of security testers will attack your site through

a combination of manual penetration and automated tools,

looking for common vulnerabilities and creative new attacks

• Comprehensive report

– Receive a complete report with findings and

recommendations on how to improve your site security

• Remediation consulting

– Leverage your security expert for advice on how to fix

specific vulnerabilities and mitigate different types of risks

S E C U R I T Y T E S T I N G A P P R O A C H

18

|

• Real-world load testing solutions

– Understand how your app performs under real world pressures

• Expert-led and managed load testing

– A Performance Engineer will assess current infrastructure,

develop JMeter scripts, execute and analyze tests

• Scalable and flexible load scenarios

– A range of options to test your apps and sites and APIs

- Large number of concurrent users

- Varying amount of time on the app or site

- With various bandwidth constraints to mirror real-world conditions

• Comprehensive report

– Receive a complete report with findings and recommendations on

how to improve your apps’ performance

19

L O A D T E S T I N G A P P R O A C H

| 20

A P P Q U A L I T Y T O O L S

A P P L A U S E

S D K

Discover crashes in real time

and distribute builds

A P P L A U S E

A N A LY T I C S

Listen to what your users are

saying in the app stores

3 6 0°A P P Q U A L I T Y

D A S H B O A R DAccess all of your app quality

results in a single place

M O B I L E B E TA

M A N A G E M E N T

Run & manage beta programs

at a global scale

|

3 6 0°A P P Q U A L I T Y D A S H B O A R D

|

M O B I L E S D K ( I N C L U D E D I N F U N C T I O N A L )

| 23

U S E R S E N T I M E N T A N A L Y T I C S

| 24

U S E R S E N T I M E N T A N A L Y T I C S

| 25

C O M P E T I T I V E A N A L Y S I S

| 26

U S E R R E V I E W S E N T I M E N T

|

The Challenge

MATURING THE

CROWDTESTING MODEL

•••

|

• Anyone can join, but many do not qualify for paid projects

– ‘Test the Tester’ – initial round of validations

- Hard skill vetting: technical testing skills; device knowledge; OS expertise

- Soft skill betting: following policies and processes, communication skills

– Ongoing validations from every project:

- Tester Ratings, based on the quality of reports and impact of defects found

› Value determines how much tester gets paid for their work

› Quality affects tester’s performance ratings

› Judged by customers, project managers, and team leads

- Training and Mentoring programs to sharpen tester skills

› Crowd Teams Community of Experts

• Protecting customer IP

– NDA in our standard Terms of Use for every tester

– Custom NDA supported for each customer

– Proof: 70,000+ test cycles with zero breaches of IP

Q U A L I T Y Q U A L I T Y Q U A L I T Y

28

|

• Project Managers are highly aligned with internal Dev & QA teams

– Support your expectations, timelines, and priorities

– Team Leads in various time zones to provide around-the-clock coverage

– Recommend new ideas based on experience with many international companies

• Integration with your systems and test methodologies

– Bug Tracking Systems: JIRA, HP Quality Center, IBM Rational, Microsoft TFS,

Github, Bugzilla, Pivotal Tracker, Asana, Mantis, FogBugz, Rally, and more

– Methodologies such as continuous integration, test-driven development, agile,

waterfall, lean software development, behavior driven development, etc.

• Continuous improvement and optimization

– Assess results from every project and deliver monthly management reports

– Identify favorite testers and invite new members to optimize team performance

F U L L Y - M A N A G E D S E R V I C E

29

|

The Challenge

FINAL POINTS

•••

|

• App quality was traditionally conveyed by companies to end users

– Testing in the lab was sufficient due to smaller coverage/environments

• App quality is increasingly being overruled by end users

– Through app store ratings & reviews, social media, blogs, and more

– Across multiple dimensions: usability, security, interoperability, etc.

– And most importantly, the cost and publicity of failure is higher than ever

• Applause delivers tools & services that address both views

– ‘Inside-out’ point of view

- Collaborating with internal development & QA on strategy and methodology

– ‘Outside-in’ point of view

- Complementary ‘in-the-wild’ testing on real devices across location & OS

- Reveal important feedback from end users before launch

31

K E Y T A K E A W A Y S

|

The Challenge

Q&A

•••