Crowd testing framework: Trends, Challenges and Solutions for Mobile application Testing

1

Siddharth ShindeSr.QA Specialist

Agenda

Software testing evolution Mobile Testing Challenge Why mobile testing is difficult? Crowd Testing as a solution Principles & Features of crowd testing Success factor Summary

IT- Testing

Goals- Speed- Cost

-Quality

More customer driven

More cloud based

Shorter production cycle

More HW and

SW choices

More users, more

places

More open

source

Evolving IT landscape presents more challenges than ever!

What it does?

For achieving high Quality, Speed and Costs have gets compromised.

Software Testing Evolution

SOFTWARE TESTING

No dedicatedTesting team

Dedicated testing team (in house)

Outsourced testing

SOFTWARE DEVELOPMENT

(In house) resources Resources in the

cloud

Mobile Testing : Challenge

Only 31% of firms surveyed currently test mobile applications. Why?

Many report they do not have appropriate tools or expertise and limited access to the devices to test effectively mobile applications

Mobile Trends

Transactions in both B2C and B2B Products and services Account management

Internal processes (B2E) Business intelligence Process management Social collaboration

Business as usual in mobile testing

Functionality

Performance/Load

Security

System

... Mobile Testing additional hot spots

User experience

Functionality

Screen Size

Platform

Performance/Load

Security

System

Device breakup creates a need to performdevice level testing & UAT for mobile apps

Why Mobile Testing is difficult? (1/2)

Touching the device

Touch, swipe, pinch, rotate, move

Diversity in platforms, OSs &devices

Android is the most fragmented, with 5 major versions corresponding to 9 API sets. In typical

markets for feature phones, there can be more than 100 different device types in the installed base

Automationchallenges

Sophisticated user experiences involve touch, gestures, GPS location, audio, sensors(accelerometers) and physical actions (touching the handset to NFC readers). Such interactionscan't be fully scripted or simulated, and may involve manual testing on real devices

Network performance

Cellular networks are nondeterministic, and exhibit wide variations in bandwidth and latency.Users in different locations accessing different operators can experience vastly differentapplication performance

Applicationcomplexity andsophistication

Mobile devices and applications are becoming more sophisticated, using techniques such as

context, 3D graphics and gaming. Greater sophistication implies more complex testing

New OS versionsoften breakapplications

Developers have no control over when new OS versions will appear, and when or whether users

will upgrade. Thus, it's common for new OS releases to break existing native applications

Why Mobile Testing is difficult? (2/2)

New Technology risk

Mobile is at the leading edge of new technologies, such as HTML5, that are not yet well-understood in terms of testing

Performancevariations

There is a performance difference of greater than one order of magnitude across devices in the smartphone installed base. An app or HTML5 website that runs well on a top-end device may not be acceptable on a low-performance handset

Operatorintervention

Operators may modify mobile Web content to optimize network performance so that desk-based testing may not show what a real world handset user experiences

Contextual issues

Mobile applications and websites are used in a wide range of contexts, which raises many new testing challenges. Applications dealing with critical or regulated data may demand much more rigorous testing

Peripherals

Smartphones are acquiring a growing range of add-on devices, such as Bluetooth peripherals, that are generally not accessible to testing tools

Testing useropinions

Many applications will be distributed via public app stores, where user reviews and attitudes play a large part in determining whether an app is downloaded.Understanding user reactions to an app and its store collateral before it's published may be valuable

Even <<for free>> app

Must be <<right first time>>

Must get into the market<<first>>

CROWD TESTING

QA Need

s

Mobile and Web

Outsourcing

CROWD – Dictionary meaning

“The mass or multitude of ordinary people”

Crowdsourcing is the process of getting work or funding, usually online, from a crowd of people.

CrowdSourci

ng

Outsourcing

Crowd

Testing

Crowd

What is crowd testing?

“If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it.” (Paul Herzlich)

“ Crowtesting puts software to test under real-world conditions by real users, allowing companies to gather real insight and feedback and to identify defects quickly and cheaply” [CSTE2013]

Principles

Diversity/coverage/realistic

Independent and test as a user

Payment system

Immediate results

Crowd Testing Features

Players and Process

Crowdtesting ensures that a software produces expected output across all desired platforms & user groups

Who are the crowd testers?

The testing crowd can be a pool of: Inter-company testers Intra/inter-project testers in a company Strategic partners Domain knowledge experts Technology experts Tool experts Network of end users Social network users

Certified

Specialist

Recreational

Crowd Testers expertise and infrastructure = Crowd Testing types

Crowd Testing

Exploratory crowd testing

Enterprise crowd testing

Exploratory Crowd Testing Effective to cover all the many configurations Good for non critical systems Simulating real load and testing

- Outcome not predictable- Discovering basic failures

To help making it more effective: Show them the big picture of what they are doing Formal process A clear report

Enterprise Crowd Testing

• Secured private cloud infrastructure

• Testing ManagersOrganized

• Crowd testers screening• Project management,

tracking, effort planning

Professionalized

• Standardized process and reportingStandardized

Enterprise

Cloud

Infrastructure

Large enterprises utilizing their in-house excess capacity for on-demand crowd sourced testing

Enterprise («Managed») Crowd Testing «Enterprise crowd testing, although still in its nascent and

emerging stage, promises to be the next wave in the social, mobility and localization era.» [28+_TestExp14]

Crowd Testing

Platform

Test Scheduling

Testers Profile

Knowledge center

Comm. And collaborati

on

Test manageme

nt

Reporting

Principles of crowd-testing

Preparation Initiation Execution Evaluation Completion

Identify customers needsFix price

Define report templatePrepare crowd testers Prepare test environment

Initiate testingSupervise and monitor

Validate resultsPrepare client deliverablesTesters Payment

Present results to clientCrowd testers ratings

Crowd Testing … Glossary

Crowd Crowd Sourcing

Managed Crowd

Testing

Outsourcing

Crowd

TestingBeta

Testing

Exploratory

Crowd Testing

Enterprise

Cloud Testing

Opportunities for SE research in crowdtesting: COMMUNITIES Crowdsourcing in SE Global Software Engineering Social Software Engineering Empirical studies Software Testing

Effectiveness of different testing methods

Crowd

Testing

CloudTesting

LabTesting

«When» shall we test with the crowd, when not to?

Which software domain

Which type of tests

Complementing different testing methods

Crowd Testing

CloudTesting

LabTesting

How crowd and Lab testing can be used together?

Need: clients’ want to get the best

from the combination of both

Efficiency

Complementing different testing methods..cont.

Systematic Mobile

Testing

Crowd Testing

Solution to

?

Dimensions to account for

• Multi OS (Apple, Android, BBMS, MS…)

• Multi Technology (Native, Hybrid, Browser…)

• Multi manufacturer (Samsung, Micromax, Acer…)

• Network Quality (Intermittent, disconnected, bandwidth..)

• Device resource (Memory , CPU, Battery..)

• Device capability (GPS, Camera…)

• System of record (Don’t forget the backend)

• Does the user like it? (UX)

Complementing different testing methods..cont.

Test SelectionSystemat

ic Structur

al Testing

Crowd Testing

Test Executio

n

Solution to

?

Coverage issues

Complementing different testing methods..cont.

Model Based

Testing

Mobile Testing

Model Based

Mobile

Crowd Testing

Social Software Engineering How to keep Crowd Testers

motivated

How to keep them involved

Appropriate reward mechanisms

Types of Testing that can be Crowdsourced

ALL Functional Load/Performance Usability Security Localization

Testing Modes Exploratory Test Case Based Automated

Forms of crowd

Fully Private Crowd

• Utilizing internal infrastructure

• Leverages organizational

bench.

Public Crowd

• Utilizing external freelance capacity

• Community and partnering

Hybrid Crowd

• Optimizing internal and

freelance capacity

Application in Industry….Summary

Challenges in Crowd Testing

Securit

y

Breac

hes

Lack of standards

Duplicate

defect

managementCoverage

Measurement No control

Critical feature

coverage

Lack of ownership

Reward mechanism

Continuous improvement

Communication and

management

Concerns & Mitigations …..

Concerns : Security and Confidentiality

Mitigation: Background check NDA

Benefits

Diversity

Few questions you should ask from your provider

Can I specify the criteria for selecting testers? How do I know which functionality they covered? Will I know which configurations were covered? Do the testers know what they need to know? Who is accountable for quality and schedule?

Delivery Assurance?

Success Factor

Process Governan

ce

Managed and

controlled

delivery

Structurally

developed

community

Efficient technolo

gy backbon

e

Organization can opt…

Build their own

crowd testing solution

Outsourcing to crowd testing

Organizations/vendors

Summary

The reason behind the success of this framework is that this method is essentially user-centric. This means it the user who determines the adoption and success quotient of particular software.

In ConclusionCrowd testing is growing exponentiallyCrowd testing may bring big benefits to QAStill, there are number of challenges to be managed

Contact

If you are interested to this topic, please

Stop by after the presentation

Contact me at siddharth.shinde@ishisystems.com

References / Acknowledgements

[TestExp14] «Testing Experience, 25/2014» issue on Crowd testing [RV_TestExp14] Rama Murari & Venkat Ramesh Atigadda, “Panorama of Crowd

Testing”, in Testing Experience, 25/2014 [Stein_TestExp14] Markus Steinhauser “Crowd Testing – An Addition to Traditional

Software Testing”, in Testing Experience, 25/2014 https://www.passbrains.com [34+_TestExp14] Dieter Speidel, Mayank Mittal & Mithun Sridharan “Enterprise

Crowd Testing”, in Testing Experience, 25/2014, page 34+ [28+_TestExp14] Saisubramanian Sivasailem & Dilipkarthik Jayamohan, “The Next

Wave in the Social, Mobile and Localization Era Enterprise Crowd Testing”, in Testing Experience, 25/2014, page 28+ https://99tests.com/ Henry Muccini, DISIM, University of L’Aquila www.bugfinders.com