utest whitepaper community driven qa

7/28/2019 uTest Whitepaper Community Driven QA

1/8

COMMUNITY-DRIVEN QASix Ways a Virtual Testing Team Can ComplimentYour In-House QA Efforts and Raise App Quality

WHITEPAPER: March, 2009


2/8

WHITEPAPER: Community-Driven QA 1

Table of Contents

The Current State of Managing Software QA...... 2

TodaysQA Landscape: Trying to Do More with Less..... 2

The Evolution of Quality Assurance: A Quick Look Back. 2

Crowdsourcing: A Complimentary Option for In-House QA 3

Ways in Which Crowdsourcing Compliments In-House QA Teams .. 4

1. Maintain Control . 4

2. Testing Coverage.. 4

3. Increase App Quality............ 5

4. Tester Diversity .. 5

5. Efficiency and Ease of Use..... 56. Cost Containment... 6

Dig Deeper & Take Action 7

About uTest. 7

As soon as we started programming, we found to oursurprise

that it wasnt as easy to get programs right as we had thought.

Debugging had to be discovered. I can remember the exact

instant when I realized that a large part of my life from then on

was going to be spent finding mistakes in my own programs.

Maurice Wilkes,Author, Memoirs of a Computer Pioneer


3/8


The Current State of Managing Software QA

Todays QA Landscape: Trying to Do More with Less

More than ever before, testing teams are being stretched thin in just about every organization. QA

leaders are being asked to keep their applications bug-free despite shrinking budgets and teams; despite

tighter launch deadlines; and despite increased app and testing complexity.

In many companies, top-down pressure from management to slash anything that isnt fully perceived as

mission-critical (or fully understood) has brought a sharp eye on the QA department. And while testing

managers and directors are trying to get their jobs done with smaller budgets and teams, theyre also

trying to achieve testing coverage to match the complexity of their users and their apps. This applies to:

Operating systems

Browsers Plug-ins and anti-virus programs Geographic locations Languages And in the case of mobile applications, handset makers and wireless carriers

All of this is enough to make even the best QA leaders feel like the deck is stacked against them that

all their efforts are merely delaying the inevitable crisis.

The Evolution of Quality Assurance: A Quick Look Back

Of course, things werent always this tough for QA managers (not that its ever been easy). As desktopapplications began migrating to the web as SaaS apps in the early

2000s, there was optimism in many QA departments. This was

based on the premature assumption that web apps would

dramatically reduce the complexity of the testing compliance

matrix. If it works in one browser, it will work in any browser

was the hope of many testing managers.

And for many, there was a temporary lessening of the testing

efforts required due to cross-browser, cross-platform

consistency of the web apps of those days: Simple HTML

content, generated entirely server side, with very little client

complexity and light usage of rich content or dynamic content. Yep, those were the days.

That euphoria was short-lived, however. As application complexity has increased exponentially, the

dimensions of the testing matrix would ultimately burden traditional QA departments. Customers

continue to expect more; and so executives and sales & marketing teams continue to demand more,

better and richer features. And after the product and engineering teams have managed to build these

magical apps, its often the QA team thats left to hold things together.

Testing is complex, but effective

use of exploratory techniques can

help tame that complexity and

contribute to the production of

high quality software.

- James WhitakerTesting Guru, Microsoft


4/8


The traditional solution to this has been to grow the QA department in a never-ending attempt to

provide proper coverage for an inherently unmanageable testing matrix. However, given the tightening

of budgets everywhere, testing coverage (and long-term app quality) often suffers.

Crowdsourcing: A Complimentary Option for In-House QA

In response to these mounting challenges, there is a rapidly growing trend of using community-based

testing (aka: crowdsourcing). By leveraging a global community of professional testers, QA teams are

doing the impossible: maintaining app quality, achieving broad testing coverage, meeting launch dates

and working within tighter budgets. Heres how it works:

1. QA teams specify their testing requirements (by location, OS, browser, app type, etc.)2. They upload their testing scripts and other documentation through a secure platform3. Bugs are reported and classified by the community, and monitored by the in-house QA team

(who can approve, reject or request more information from testers)

4. The QA team pays for each completed test cycleThe net effect is that QA teams get instant access to a global community of professional testers via a

platform that enables management of full QA cycles (or connects to

their in-house bug tracking system) all without having to commit to

long-term contracts. This testing community can function as an on-

demand extension to your in-house QA team.

If this sounds familiar, thats because the crowdsourcing concept

was inspired by the open source movement. But unlike open

source development, the QA team that uses crowdsourced testershas direct control of test plans, scripts, prioritization and timelines.

Also, while the open source model of distributed activity often

depends upon altruism and volunteer work, this is not the case with crowdsourced testing. These

professional testers are paid for their active and effective participation.

With this in mind, let us now look at six ways in which crowdsourcing can provide an ideal compliment

to your internal QA department.

The next logical step in the

evolution of testing is for

vendors to provide testers and

this is exactly the era weve

entered with crowdsourcing.

- James WhitakerTesting Guru, Microsoft


5/8


Ways in Which Crowdsourcing Compliments In-House QA Teams

1. Maintain Control:

One of the greatest concerns with traditional outsourcing is a loss of control, and thus, quality. But withcrowdsourcing, the in-house QA leader maintains complete control

and oversight of the entire test cycle.

In fact, the companies that use crowdsourcing understand that

effective, detailed communication and project management are

critical to successfully managing the crowd. Said differently, you

are the brains calling the shots; the community serves as your

hands, feet and eyes, reporting their legwork back to you for

review and approval.

And so while you dont have to micro-manage the crowd in terms

of tactical execution and idea generation, strong management enables the processes, plans and

deadlines to remain firm. The crowdsourcing model also provides the in-house QA team with the tools

they need to oversee the entire testing cycle:

Share your application via a secure platform Specify your unique testing requirements to custom-build your virtual QA team Load test scripts or QA scope requirements Specify timelines and other requirements Review bug reports Integrate with in-house bug-tracking systems likeJira,RallyandBugzilla

Its worth noting that there are many sites that connect employers with freelance workers, but suchsites do not provide structured community-building they are simply an online employment agency that

helps companies to find the right individual freelancer. Crowdsourcing, therefore, is unique in that it

encourages strong management and participation from the companys in-house QA team, rather than

replacing it.

2. Testing Coverage:

Unless your app is designed for a simple and homogenous audience thats identical to your in-house QA

team, its likely that you have gaps in your testing coverage. Yet to fill these voids - whether they be

browser, OS, location, language or device would be impractical, expensive and a huge waste of time.

This is no longer the case. Crowdsourcing, having already assembled a global testing community,

enables QA teams of all sizes to target very specific users (like Mac-users who live in Singapore) for

specific assignments.

Previously, even the largest software companies could not afford to have their products universally

tested. Now, even early-phase start-ups cant afford not to.

With the uTest community, we

get a complete picture of how our

software works, on different

environments, systems and

locations.

- Cosma DincaQA Manager, IMinent Software
http://www.atlassian.com/software/jira/http://www.atlassian.com/software/jira/http://www.atlassian.com/software/jira/http://www.rallydev.com/http://www.rallydev.com/http://www.rallydev.com/http://www.bugzilla.org/http://www.bugzilla.org/http://www.bugzilla.org/http://www.bugzilla.org/http://www.rallydev.com/http://www.atlassian.com/software/jira/


6/8


3. Increase App Quality:

For some strange reason, customers expect applications to be error-free and fully functional from day

one. Good enough is no longer an option, nor is beta testing as a substitute for thorough QA .

Whereas beta used to mean An early release of a software version, before the bugs have been worked

out, it is now understood quite differently. The meaning now is almost entirely business-driven: A

stable, first release of an application, before the precise business model has been set in stone . Simply

put: web and mobile app companies often do not have the luxury of making two releases to customers.

And so from a quality perspective, those considering using the community as a QA supplement must

also consider that there are two types of currency in an online

community: money and reputation. Of course, money talks,

but the transparency of crowdsourcing motivates people to

build strong reputations. For these testers, it is the only way

to earn the respect of their peers. For a QA manager, this

helps to ensure that the people testing their applications have

something at stake in the relationship. As with your internalQA team, motivation is the key to success.

4. Tester Diversity:

Not only does the internal QA leader maintain authority over

release cycles, they also exercise crowd control (pardon the pun) in regards to which community

members will test their applications. As the scope of your applications evolves, your virtual testing team

can change with it. An ideal testing community will be comprised of a wide range of users across:

Geographic locations Languages spoken Technical platforms (hardware, operating system, browser, anti-virus, etc.) App type expertise (mobile vs. web vs. desktop, specific industry knowledge, etc.) QA experience levels

For most companies, it is prohibitively expensive to simulate solely within an internal QA department.

Also, no matter how intelligent, driven and talented your in-house QA team may be (and weve met

some great ones), they are often less effective at generating product ideas and objective feedback than

you may think. By crowdsourcing, you can avoid (or at least off-set) the group-think that often plagues

so many internal QA teams.

5. Efficiency and Ease of Use:

Since it is an on-demand solution, crowdsourcing helps alleviate the pains associated with peak release

times (this is especially true of those adhering to Agile). Leveraging a community of testers exactly whenyou need them can ensure that resources are neither over-worked - nor underutilized - at any point in

the product development lifecycle.

When it comes to real-world

application testing, the crowds in

Camden Market are preferable to the

elite residents of Buckingham Palace.

Diversity yields world-class QA

coverage.

- Doron ReuveniCEO, uTest


7/8


The efficiency of any QA department is often a good indication of the efficiency of the business as a

whole. And so by smoothing over peak release times to achieve faster time-to-market, everyone reaps

the benefits. Vendors are free to pursue new revenue opportunities; the sales team enjoys a higher-

quality product to pitch and technical support is no longer overburdened with a faulty product. In short,

an efficient QA process lends itself to success in other areas. Just dont expect a trophy anytime soon.

6. Cost Containment:

The net impact of a user discovering a bug in your software is high for obvious reasons, including

customer dissatisfaction, impact on follow-up sales and support costs, to name a few.

This impact has been studied greatly, most notably by Basili

Boehm as published by IEEE; and by Littlewood as published in

his Software Reliability Achievement and Assessment. It turns

out the overall cost of fixing bugs that have been discovered by

users after launch can be 1,000 times greater than the cost of

fixing bugs that are discovered earlier in the process! Dontbelieve us, do you? Click here for a TechTarget article that

highlights these studies.

And in todays world of pay-as-you-go products such as mobile

apps or any SaaS web app any software bugs that make it in

front of your users will immediately decrease usage, dragging

revenue down with it. Even for standard license-based

applications, long evaluation cycles mean bugs often are found

before the purchase decision, and can hamper your reputation

and sales.

But when in-house QA teams leverage a global testing community, both parties benefit. The QA team

gets on-demand access to testers with the right skills, location and technical profile. Testers get to earn

extra money for doing what theyre good at, and build a reputation that helps them win more future

testing projects. In short, crowdsourcing is a resource that progressive QA leaders are utilizing to

achieve maximum testing coverage, which decreases the likelihood of bugs making it to launch, and

helps them get their apps to market faster.

What Would It Cost To Add An

Additional QA Analyst?

Base Salary: $65,455

Social Security: $5,192

401k: $2,443Disability: $679

Healthcare: $5,722

Time Off: $8,876

Total: $88,368

Source: Salary.com

March, 2009
http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1303496,00.htmlhttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1303496,00.htmlhttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1303496,00.html


8/8


Dig Deeper & Take Action

Want to capture the power of crowds to launch higher quality apps faster?

Design: Need help with UI or graphic design?o Guruwww.guru.como CrowdSpringwww.crowdspring.com

Development: Want to get your app built with the help of the crowd?o TopCoderwww.topcoder.como Rent-A-Coderwww.rentacoder.com

Testing & Usability: Want to make sure your new app or version is ready for primetime?o uTestwww.utest.com

Other inspirations for this paper and valuable resources about crowdsourcing:

Wired MagazineCrowdsourcing articleby Jeff Howe Jeff HoweCrowdsourcing blog ReadWriteWebCrowdsourcing review BNetCrowdsourcing review Crowdsourcing LinksCrowd-driven companies & news

About uTest

Headquartered near Boston, uTest is the world's largest marketplace for software testing services. Thecompany provides real-world QA services through its community of 25,000+ professional testers from

over 150 countries around the world. Hundreds of companies - from web start-ups to enterprise

software firms - have signed up to get their apps tested by the uTest community.

uTest enables companies to launch higher quality products; get their desktop, web and mobile

applications to market faster; and control the cost of testing. Customers specify their QA requirements

for tester experience, location, language, OS and browser, and uTest selects the right testers for each

project. And because uTest is on-demand, companies pay for completed test cycles.

A brief online demo is available atwww.utest.com/demo . uTest can be contacted at:

uTest, Inc.

153 Cordaville Road

Southborough, MA 01772

p: 1.800.445.3914

e:[email protected]

w:www.utest.com
http://www.guru.com/http://www.guru.com/http://www.guru.com/http://www.crowdspring.com/http://www.crowdspring.com/http://www.crowdspring.com/http://www.topcoder.com/http://www.topcoder.com/http://www.topcoder.com/http://www.rentacoder.com/http://www.rentacoder.com/http://www.rentacoder.com/http://www.utest.com/http://www.utest.com/http://www.utest.com/http://tinyurl.com/5zm9wdhttp://tinyurl.com/5zm9wdhttp://tinyurl.com/5zm9wdhttp://crowdsourcing.typepad.com/http://crowdsourcing.typepad.com/http://crowdsourcing.typepad.com/http://tinyurl.com/38lgz2http://tinyurl.com/38lgz2http://tinyurl.com/38lgz2http://tinyurl.com/5mh3tshttp://tinyurl.com/5mh3tshttp://tinyurl.com/5mh3tshttp://www.crowdsourcinglinks.com/http://www.crowdsourcinglinks.com/http://www.crowdsourcinglinks.com/http://www.utest.com/demohttp://www.utest.com/demohttp://www.utest.com/demomailto:[email protected]:[email protected]:[email protected]://www.utest.com/http://www.utest.com/http://www.utest.com/http://www.utest.com/mailto:[email protected]://www.utest.com/demohttp://www.crowdsourcinglinks.com/http://tinyurl.com/5mh3tshttp://tinyurl.com/38lgz2http://crowdsourcing.typepad.com/http://tinyurl.com/5zm9wdhttp://www.utest.com/http://www.rentacoder.com/http://www.topcoder.com/http://www.crowdspring.com/http://www.guru.com/

utest whitepaper community driven qa

Documents