creating killer mobile apps · twists and turns of testing your mobile application. •...
TRANSCRIPT
Creating Killer
Mobile Apps The keys to developing, testing
and launching mobile applications
that your users love
A Free Book from uTest
Creating Killer Mobile Apps Introduction……………………………..………………………………………...………….……….2
The Mobile Boom………………………….……..……………………............................……......4 Mobile User Growth Rate………………..…........................................…….……………………..4 Why Mobile? ……………………………………………………..…......…….………….....…….…..5 What‘s at Stake? …..…..…..........................................…….…………...……….………………..6 Mobile App Strategy……………………………………………………………………………...….7 Mobile Web vs. Native Apps….……………………………………………….………….…………..9 What the Users Say …………………………………….……………….….………… ……………11 Alternatives: WAC and Hybrid Apps …………………………………………….…………………12 Operating Systems, Devices and Carriers………………………………..……………………….13 Mobile Testing Types, Tools and Methods………………………..…………….…….….…....16 Functional Testing…………………………………….……………………...….……..…………….17 Usability Testing…………………………………….………….……………………..………………18 Performance Testing…………………………………….………….…….……….……….……..…19 In-House Testing …………………………………….………….………………….……….…….…20 Outsourced Testing …………………………………….………….…………….………….…....…21 Emulators & Simulators………….…………………….………………………..…………………..22 Beta Testing…………………………….……….…………..........….………………….……..……23 Key Testing Challenges………………………….…………………………….……....……........24 Crowdsourced Testing………………………….………………………………………...…….....27 How It Works.……………..…………………………………………………………………………..29 Challenges …………………………………….………….……………………….…………..……..30 Benefits……………………………………………………………..…………..……………………..33 Appendix: Key Players in the Mobile Space…………………………….……………………..35
2 Table of Contents
The Challenge of Launching Great Mobile Apps
Launching a killer app is like catching lightning in a bottle – and this is especially
true of mobile applications. In an industry where there is fierce competition for
media attention, VC investment and/or consumer dollars, even the smallest flaw
can doom a mobile app (and the business behind it). With so much at stake, how
can so many mobile app companies ignore the benefits of usability testing,
functional testing and load testing? They want to launch apps that delight
their users, right?
This eBook will walk you through the challenges of creating a high-quality mobile
application, and will show you how to overcome them. We will demonstrate how
companies of all sizes – from startups to enterprises – can overcome these
obstacles to build apps that their users love. In doing so, we‘ll also cover:
• The Mobile Boom: Learn what‘s at stake and also what‘s driving the
worldwide mobile application frenzy.
• Mobile App Strategy: We‘ll explain how building great apps begins long
before code is written or tested.
• Mobile Testing Types, Tools & Methods: Emulators vs. real testers; the
differences between mobile development platforms and other critical,
strategic decisions that will need to be made.
• Key Testing Challenges: How to navigate through the peaks, valleys,
twists and turns of testing your mobile application.
• Crowdsourced Testing: Discover how tapping into the crowd can help
you launch a killer app, while getting you to market faster and on-budget.
• Key Players: If you‘re new to this mobile apps game, this appendix will
highlight the companies, technologies and players that you need to be
familiar with before getting started.
3 Mobile App Challenges
THE MOBILE APP BOOM
4
The Mobile Boom
To say that the use of mobile applications is booming would be an obvious understatement. In 2010, mobile app stores will recorded an
estimated $6.2 billion in overall sales, including 4.5 billion app downloads. By 2013, some analysts expect mobile app revenues to exceed
$21 billion. With charts like this, we wouldn‘t bet against them. As they say in the investing world, the trend is your friend.
So if you think that mobile apps are for kids or that it won‘t affect your industry/company/team, think again!
5 The Mobile Boom
Why Is The Mobile Market Booming?
The accelerating shift towards mobile applications (and the smart phones they run
on) is really quite simple, but frequently overlooked. Here are just a few of the
reasons why users are flocking toward this technology:
•Accessibility: With mobile devices, the web is always at your side. This
includes email, social media, video, games, news, stocks, sports, shopping and
other actions. The ability to stay connected 24/7 has perhaps the most
widespread appeal of all, and is the number one reason why the use of mobile
applications and devices show no signs of slowing down.
•Adaptability: Mobile applications are sometimes thought of as being used
primarily by kids and consumers only, but this couldn‘t be further from the truth.
Business professionals were, in fact, the earliest adopters of this technology,
and continue to make up a large segment of the overall user base. This can be
seen in the widespread use of business & productivity apps for CRM, ERP,
documents, presentations, email and messaging tools. Of course, you can also
play DoodleJump, too.
•Efficiency: Mobile apps (and to a lesser extent, the mobile web) have greatly
enhanced the efficiency of our everyday online activities. As we will explain
shortly, the goal of any mobile app should be to reduce each feature to its most
basic component. For example, users who want to see the latest market news
will generally download an app that focuses on this exclusively. They do not
want to enter a search term and click through five pages to see this information.
Mobile applications, in other words, cut out the middleman and give users direct
access to the things they want most.
6 The Mobile Boom
The Mobile Mantra
“Much as we love freedom and choice, we also love things that just work, reliably and seamlessly. And if we have to pay for what we love, well, that increasingly seems OK. Have you looked at your cell phone or cable bill lately?”
- Chris Anderson, Wired
What‟s At Stake?
Still think that mobile apps are for games, music and ―entertainment purposes
only‖? Do you really think that mobile applications won‘t be relevant to your
company, your team and your job a year from now?
Think again. As the adoption rate increases, mobile applications will become ever
more critical to the success of your business (and your career). Those who insist
on waiting for further confirmation will have already been left behind. Companies
that recognize this trend for what it is – a technological revolution – and take
preemptive action to improve the quality and usability of their mobile applications,
will find themselves with a tremendous advantage.
Most companies will simply build a watered-down version of their website. But
those who understand the implications of the mobile boom – and who understand
all of the feats this technology is capable of – will build applications with enduring
popularity. Which side will YOU be on?
This mindset is essential. In order to take the steps required for developing and
launching a quality mobile app, you will need to believe in what you‘re doing. If
you think mobile apps are simply a passing fad, it will be reflected in your
businesses and products.
Your users are counting on you. Don‘t let them down.
7 Mobile Testing Challenges
The Mobile Future
According to a Morgan Stanley analyst, the world is currently in the midst of the fifth major technology cycle of the past half a century. The previous four were the mainframe era of the 1950s and 60s, the mini-computer era of the 1970s and the desktop Internet era of the 80s. The current cycle is the era of the mobile Internet, she says — predicting that within the next five years “more users will connect to the Internet over mobile devices than desktop PCs.”
- GigaOm
STUFF YOU NEED TO KNOW
FIRST
8
Your Mobile App Strategy
One of the first – and most critical – decisions a mobile company will make is whether to develop a native application or spend their
resources optimizing for mobile web browsers.
9 Mobile Web vs. Native Apps
vs.
Mobile Web Native Apps
The Case For (and Against) the Mobile Web
Pros:
• Single platform (the web)
• One app to build, launch and maintain
• All that‘s needed for some companies
Cons:
• Less controlled user experience
• Slower user experience
• No app store distribution
• Lack of standards across mobile
browsers
10 Mobile Web
The Case For (and Against) Native Apps
Pros:
• Rich media functionality
• Controlled user experience
• Faster experience for users
• App store distribution
Cons:
• Multi-platform
• Must build, test & maintain multiple apps
• Not necessary for some companies
11 Mobile App Challenges
Native Apps or Mobile Web? What the Users Say
―There are particular native apps that I use on a regular basis, mainly
due to ease-of-use and simplicity. These include weather apps, travel
planning apps (kayak.com), music apps (Pandora). However, if I‘m
accessing news and searching for information, I use mobile sites that
automatically render as mobile versions of their web sites (cnn.com,
google.com)…
…I also foresee the future in which mobile web will be much easier
to use, and displace most (not all) of the need for native apps. Instead
of downloading the native app itself, there may be shortcuts on your
mobile device for easy access. Think about the recent progression away
from desktop apps and into purely web apps; mobile is simply a few
years behind.‖
**********
―I prefer native apps as they are more focused on your information
needs. I agree that the mobile web is more accessible and affordable,
but it takes longer to wade through all the sites to get to what you want.
Mobile web apps do solve the problem to certain extent, but they‘re not
on par with the native apps in my opinion.‖
12 Mobile App Challenges
vs.
Alternatives to Native Apps and Mobile Web
Of course, deciding what form of mobile app you plan to offer isn‘t an
either/or decision. There are alternatives, including:
•WAC: The emergence of the Wholesale Applications Community (WAC)
may end up providing software companies with their most flexible solution.
WAC is a consortium of mobile leaders, which has come to include
carriers like AT&T, Verizon, Sprint, China Mobile, Orange and others, as
well as manufacturers like Sony, Samsung and LG. This community
intends to enable developers to develop one mobile app that will work
across ALL carriers, devices and operating systems.
•Hybrid Apps: Similar to the ideas behind developing for the mobile web,
hybrid apps would allow developers to create mobile apps that share their
backend with their web application. They would create a native app ―shell‖
or ―wrapper‖ which would utilize the same features, functionality, data and
other characteristics.
*****
However promising in terms of simplicity and cost, these two options are
not without some major drawbacks, notably that of competing interests.
Will these communities treat all devices, carriers, etc. as equals? So don‘t
rule out these alternatives immediately, but proceed with caution.
13 Mobile Alternatives
Wholesale App Community
“Though many doubted the long-term viability of the WAC, it looks as though the group is gaining the momentum it needs to make a difference. There’s a long way to go, and the concepts the WAC are working towards are ambitious to say the least, but the steps they’ve taken so far seem to be the right ones.”
- MobileMarketing.com
Other Considerations
Mobile devices are evolving into increasingly sophisticated general purpose computers. This has led to the development of a variety of
platforms and operating systems. Today, the operating system inside your mobile device is as equally important as the device
manufacturer (OEM). The various features, GUI, processing speed, and most importantly, the applications available for your device will
depend greatly on the underlying OS, as well as the carrier and OEM.
14 Mobile Decisions
Carriers Operating Systems Device Manufacturers
Mobile OS: Past Present and Future
The mobile wars are heating up. This recent study by Quantcast shows that Android is gaining on Apple‘s web iOS, although the former
still maintains a huge share of the overall market. Keep this trend in mind when deciding which operating systems you intend to build an
app for first.
15 Mobile App Marketshare
Mobile Constants
No matter which route you choose – native apps or the mobile web – certain tasks
and backend functionality will remain a constant, including:
•API / web services
•Architecture
•Database
The key lesson here is that app quality begins before coding or testing. You
should start by reviewing the makeup of your users. What features and
functionality do they want? What types of information are they trying to access?
And most importantly, how can you reach them on their terms?
Other key decisions that every mobile company will have to make include:
•Static content vs. rich content
•Commerce vs. functionality
16 Mobile Constants
Mobile Fact Consumer conversion (from surf to purchase) on iPhone or Android native apps is 3X - 10X times that of mobile web sites Why? Because users are more comfortable with the device. In most cases, the usability of the native app, including page load speed, is much greater than the mobile web.
MOBILE TESTING TYPES AND METHODS
17
Mobile Testing Types: Functional Testing
The most essential testing procedure will be to verify that the basic functionality of your
application. Early on, much of this attention should focus on the download activities of
the prospective user (i.e. that the download process is seamless and intuitive). From
there, you can begin to verify more specific functionality. As we‘ll explain shortly, this
can be done either via a mobile simulator (desktop browser) or by using real-world
testers.
Finding device specific bugs: If, for example, you discover links that are not
functional on certain devices, then you should first determine whether it works on the
desktop browser. If it is reproducible on the desktop browser, then it is obviously not a
device-specific bug, but rather a bug in the basic application. If a link is functional in the
desktop browser and it doesn‘t work in the device, then it should be identified as a
device specific bug.
Such functional testing should occur when the application is in the early development
phase. At that point, testers should perform a quick test on the devices you deem as
the having the highest priority, in order to find any rendering or any functional issues as
soon as possible. This way, development can have enough time to fix them or arrive at
a proper decision as to whether to support these devices or not. If fixing the bug
involves high dev cost, they probably won‘t.
If this sort of functional testing doesn‘t happen at the an early stage (and most of the
time, it doesn‘t) there is the risk that the application might not work for certain devices.
This can lead dev/PM to changes the design, which can further escalate costs.
18 Testing Types: Functional Testing
Mobile Stats
• 51% of iPhone owners have five or more games installed on their devices
•37% of iPhone owners play games at least once per day – puzzle games are the most popular
•46% of Blackberry owners have NO games on their device at all
- Source: Mobile Entertainment
Mobile Testing Types: Usability Testing
Usability testing plays an important role for launching a quality mobile application.
Usability testing should occur as soon as the basic stable for the application is
ready. This is also true in order to test whether the application UI is user-friendly
or not, or whether some part/functionality of the application is easily discoverable
or not. Also, usability studies should be done for the different types of devices,
such as touch screen devices, since the user has different experience for the
same application on these devices.
A user may have a good experience on the iPhone, while another might have bad
experience on a non–touch screen device. In this instance, changes to the UI
should be considered to improve the experience on the non-touch screen device.
For example, let‘s say there is a form with various fields and it is saved by clicking
on the save button located at bottom of the screen. For this scenario, someone
using iPhone will have no issues to scroll to the bottom to click on the save button,
since the iPhone has a good scrolling mechanism. However, the same scenario
will give the bad user experience for the non-touch devices users, where scrolling
to the bottom of the page is not as easy.
Needless to say, usability testing is an extremely critical step towards launching a
high-quality mobile application.
19 Testing Types: Usability Testing
User Loyalty
“While most will tolerate a minor glitch, studies show that many won’t hang around if they are disappointed with the performance of a mobile app or website. And the damage doesn’t stop there. Word-of-mouth will ensure any bad experiences are soon broadcast.”
- Helen Leggatt, BizReport.com
Mobile Testing Types: Performance Testing
Performance testing is another critical step in the successful launch of a mobile
application. The key factor to be aware of here is mobile carriers, which can effect
the speed and ease-of-use of the application. If the application performance is
poor for the end user, they will be likely to divert to other apps and services. The
application must therefore be tested with different devices and carriers depending
on the county and region.
Most likely, you didn‘t build your application for automated users. You built it for
real users. Why then do so many companies rely solely on automated testing
results to verify functionality under stress? Until recently, it was because there
was no such mechanism by which a company could utilize real testers while a
system was under load.
But as we‘ll outline shortly, that obstacle has largely been overcome. By
leveraging real testers with automated tools, companies now have access to an
entirely new layer of data.
Better yet, they are able to identify and resolve issues that would have been
reported later in the development process.
20 Testing Types: Performance Testing
Mobile Testing Tip If the application being developed will be dynamic or interactive, then the browser must support the following: - JavaScript support: Mobile browsers must support JavaScript in order build an interactive application UI.
- XML HTTP Request Object: XML HTTP request object support is required to communicate with the back-end server and to update the page with new data without reloading the page. This will give the user a smooth browsing experience for the site. - CSS support: CSS defines how page elements are to be displayed and enables you to change appearance and layout of all of the pages by editing a single file.
Mobile Testing Methods: In-House Testing
Pros: In an ideal world, mobile testing could be performed under the watchful eye
of product managers, developers and QA managers, who could observe firsthand
how the application performs under every possible user scenario. The assembled
testing team – consisting of perhaps a dozen or so of the most talented testers on
the market – would have hundreds of handset models at their disposal: iPhone,
Blackberry, Nokia, Samsung, Sony - you name it, they test it. It would be a testing
lab that even the largest software company would envy.
With this type of control over the testing process, what could possibly go wrong?
Cons: The truth is, for mobile apps, building a comprehensive in-house testing
lab is prohibitively time-consuming and expensive. This is true for almost every
mobile app company, with very few exceptions.
Imagine the expense of building an in-house team and lab capable of assuring the
functionality for iPhone, Blackberry and Android handsets (of all makes and
models) across wireless carriers in the U.S., U.K, Spain and France. Now imagine
this with 3 apps… 5 apps… Will you send testers around the globe to assure
quality? Of course not. Thus, it‘s no surprise that mobile app companies – from
start-ups to large enterprises – don‘t rely solely on in-house testing resources.
To be fair, most firms don‘t intend to create an in-house team like the one
described above (although some have tried). Worse, they spend their money
trying to get whatever they can from this method. That is, they have 3-5 in-house
testers, with a dozen devices (at most). But their testing coverage is confined to
their own office, and the whole point of comprehensive testing is lost.
21 Testing Methods: In-House Teams
Just how important is global testing coverage? “By the end of this year, the mobile ad market is expected to grow to $913.5 million. And by 2013, Gartner expects mobile ad spending to surpass $13 billion, with the Asia-Pacific region bringing in the most revenue, followed by North America and Europe.”
- Meghan Keane, www.econsultancy.com
Mobile Testing Methods: Outsourcing
Pros: Of course, when mobile application developers discovered that the in-house
route couldn‘t provide the testing they needed, they logically said to themselves:
―I‘ll just outsource it.‖ This was not an isolated decision; it was made by thousands
of software companies, and few could fault them for their logic at the time. If
comprehensive mobile testing was to be achieved, then surely an offshore firm
(with their low-cost labor) would be the only entity capable of getting the job done
at a reasonable price.
It‘s this line of thinking that continues to define the way in which many mobile
companies deal with their testing objectives.
Cons: While outsourcing has certainly proven to be less expensive than hiring an
internal team, developers who contracted with such firms quickly ran into the
same limits as that of their in-house teams – while losing what little control they
had over the QA process. And for what? With no insight into the day-to-day testing
activities, how could these developers expect to improve their application in a
timely manner? Developers must remain close to their application at all times,
which is antithetical to the outsourcing mentality.
And as it turns out, even the largest of traditional outsourcing firms don‘t have the
resources needed to provide adequate testing coverage required by today‘s
mobile environment. Just like their clients, they cannot easily (or affordably) test
an app across dozens of handset makers, models, carriers, languages and
locations. Plus, the testers employed by these offshore firms are unlikely to be
among the target audience for a company‘s mobile apps.
22 Testing Methods: Outsourcing
Common User Complaints:
•Failed or stalled downloads •Credit card verification delays •No screenshots •App performance when there is slow connectivity •Inconsistent buttons, fonts, etc. •Missing or broken links •App not supported by device •Outdated versions; bugs still in app
Mobile Testing Methods: Simulators/Emulators
Pros: In many ways, the use of mobile emulators and simulators is essential for
developing and testing a quality application. Such technology enables developers
to verify certain functionality that is not specific to any device, carrier or operating
system. They are also very useful in terms of usability, and especially design,
including data input, screen size, button use, etc. – all from the convenience of
their own laptop.
Although many mobile app companies rely on these tools to identify bugs at an
early stage in development , they often fail to catch bugs at later stages. As we‘ve
seen, these type of bugs are always the most expensive to correct.
Cons: One of the biggest challenges for mobile developers is that the testing of
applications is occurring in an environment far removed from the real world, where
actual users run and interact with those applications ON their devices. Said
differently, the gap between ―in-the-lab‖ simulation and ―in-the-wild‖ usage is far
too great to ignore.
Consider, for instance, an application that is data entry intensive. To assume that
it can be fully tested on a simulator (i.e. with full keyboard and mouse access) and
be ready for release is a mistake mobile app developers have continually made
for the past five years. The convenience of simulators and emulators has made it
easier than ever to be lured into a false sense of security. But the advantages of
such tools are limited in scope, and should never be considered a substitute for
real-world, on-device testing.
23 Testing Methods: Simulators and Emulators
Mobile App Purgatory
Bugs in your mobile app cause unintended consequences. Take this story for example: A mobile app developer is alerted to a iPhone bug, and instead of being able to fix it right away, he spends weeks pleading with Apple to allow him to proceed with the update. If bugs in your app appear, getting them fixed quickly can be very problematic. Of course, if the bug had been discovered BEFORE launch, this would be a non-issue.
Mobile Testing Methods: Beta Communities
Pros: Mobile companies that can attract a large beta community have a terrific
advantage over their peers, as thousands of users submit bugs and feedback
around the clock…for free.
Needless to say, the allure of an active, engaged group of beta users is too great
for many developers to pass up. Not only do they gauge your application‘s
performance before the big release, they act as advocates for your product,
adopting a strong sense of ownership and helping you to establish a customer
base for your product. And so if you‘re at all uncomfortable with your app‘s
performance prior to launch - and don‘t want to go to great lengths running
professional usability tests – beta programs seem like the easiest, safest route.
Cons: In practice, it is rare for a software company to attract such a large group of
beta testers. After all, not every company can be Google or Mozilla, with its
various (and wildly popular) beta versions, where people line up to test drive their
software. Chances are, your app either lacks enough mass appeal, or is relatively
unknown (for now), and is hence not that attractive to the beta testing types.
But even if you are able to assemble a sizeable beta group, the method still falls
short of providing adequate testing coverage. And the reason for this is simple:
beta testers are more often like users, in that they will try to get your app to
function properly, whereas a real tester would attempt to ―break‖ your app.
In other words: Users use. Testers test.
24 Testing Methods: Beta Communities
KEY TESTING CHALLENGES
25
The mobile app testing matrix is infinitely complex – and getting worse
26 Mobile App Challenges
Device Models
Wireless Carriers
Location
OS & Browser
Features
The Challenges of Mobile Application Testing
The true cost of bugs: According to a study highlighted on TechCrunch, the average shelf life of an iPhone app is roughly 30 days. It
turns out that, for free apps, less than 20% of users return to an app even one day after downloading it. And by day 30, less than 5% of
users are still utilizing the app. And for paid apps, the drop-off is even slightly steeper.
Why the drop-off? For starters: bugs - especially ones that appear within minutes of downloading - are likely to cause users to abandon
the application after a short period of use. With the variety of apps now available, users have an extremely low tolerance for buggy
applications. In a never-ending attempt to appease their user base, developers must therefore test their products across:
Handset Maker/Model
Operating System
Browser
Wireless Carrier
Location
Language
No matter what type of mobile applications you develop – whether they‘re chat tools, social networking, games, business apps or others
– you face the same fundamental problems encountered by all mobile app developers.
Let‘s now take a quick look at how mobile app developers have attempted to deal with these problems…
27 Mobile Testing Challenges
0
20
40
60
80
100
Specification Design Coding Unit Test Integration Test
Release Test Post-Release
(Relative cost of defect, by time of discovery)
CROWDSOURCED TESTING FOR MOBILE
28
Crowdsourced Mobile Testing
Combining the benefits of the aforementioned methods, the use of community-
based testing – or crowdsourcing – has altered the way companies test their
mobile applications. This shift has empowered progressive mobile app companies
to make the best decisions, on a per-project basis, without sacrificing control,
costs or accountability. By leveraging a global community of professional testers,
QA teams are doing the impossible: maintaining app quality, achieving broad
testing coverage, meeting launch dates and staying within tighter budgets.
The net effect is that mobile app companies get instant access to a global
community of testers via a platform that enables detailed management of the
entire test cycle (or even connects to their in-house bug tracking system).
This community can serve as a virtual testing team (for small companies and
start-ups), or as an on-demand extension for in-house QA departments.
While the thought of having a virtual QA team from around the globe might seem
like a complicated free-for-all, the reality is just the opposite. With the right online
platform, the process fits neatly into your existing teams and processes. Here‘s a
quick look at how it works:
29 Mobile App Challenges
What is Crowdsourcing?
"Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to a large group of people in the form of an open call.”
- Jeff Howe Author of Crowdsourcing
Crowdsourced Testing: How It Works
30 Crowdsourced Testing: How It Works
Customer Tester Community
Specify Testing Needs - Handset Maker, Model & OS - Carrier, Location & Language
Selected testers invited to project
Testers report in real-time
Customer approves
Testers get paid
Crowdsourcing Challenges: Quality and Control
Crowdsourcing has a reputation for being noisy and chaotic, without much actual
productive work. Thus‘ it‘s critical to choose a reputable community that can point
to its customers‘ successes, and shares the past performance and reputation of
community members. In short, make sure you‘re choosing a community and not a
mob. Other common challenges include:
Communication: As with any successful project, communication is key. While
crowdsourcing doesn't enable face-to-face conversations, it does enable decision-
makers to converse with their virtual QA team in real-time through an easy-to-use
online platform. Also, make sure the testing company you choose provides a
dedicated project manager to help you manage the community and the process.
Intellectual Property: IP protection is sometimes a concern for customers who
are new to crowdsourcing. While it might seem like your IP could be exposed to a
community of thousands, top crowdsourcing companies allow customers to
carefully select their testers and establish non-disclosure agreements (NDA) with
their community members.
Quality: With crowdsourcing, it‘s easy to believe that anyone can test your
application. Fortunately, the crowdsourcing model enables customers to be as
selective as they want – in terms of testers‘ skill sets, experience, performance
ratings – in determining who gets invited to test their app.
As with other methods, unexpected problems do arise, but what makes
crowdsourcing different is its inherent ability to adapt to changes on-the-fly.
31 Mobile App Challenges
Mobile Challenges
“Mobile applications come with their own unique set of challenges. Between Wi-Fi, 3G, Edge and all their different behaviors, there are so many angles you have to cover. Until recently, testing for this criteria would have been impossible, but crowdsourcing has changed all that.”
-Travis McElfresh, -VP of Technology, 1Cast
Benefits of Crowdsourced Mobile App Testing
Although the advantages of such an approach are numerous, here‘s a quick look
at what to expect when implementing crowdsourced QA for your mobile app:
Cost Control: Crowdsourcing allows managers to utilize lower-cost outside
support without being tied down by long-term commitments. The fundamental
structure of online communities creates competition, making it more accountable,
efficient and cost-effective than traditional outsourcing.
Dealing with Increased Complexity: Significant quality improvement is achieved
when development and testing is done across a wider set of handsets, carriers,
locations and languages. Crowdsourced mobile app testers can be easily
recruited based on a wide variety of criteria.
Faster Time-to-Market: Crowdsourcing enables you to ramp up or down to meet
your needs at any given time - eliminating the delays and staffing constraints that
often stifle mobile app companies around peak release times.
Higher Quality: Customers expect their mobile apps to be fully functional and
bug-free from day one. Companies that believe ―good enough‖ is good enough
when it comes to quality and usability will learn a valuable lesson (the hard way).
Retaining Users: If your product has problems, or doesn‘t meet expectations, it‘s
easier than ever for your customers (and prospective customers) to find your
competition.
32 Benefits of Crowdsourcing
Why It Works
“Because crowdsourcing is built from the open source template, any company looking to leverage computer professionals has the advantage of an audience already familiar with the basic idea of community production.”
-Jeff Howe Journalist, Wired.com
Benefits of “In the Wild” Testing
„Hands On‟ Device Interaction
On a simulator, you still use a mouse to ‗touch‘ the screen and simulate gestures.
You also have a full-sized keyboard for data entry. Of course, this is very different
from using a mobile device.
First, a mobile device sits in your hand. Each of us likely has slightly different
ways of holding and operating the device. For some, it‘s done with one-hand using
your thumb or a finger. For others, it might be two hands using both thumbs.
Second, there‘s the act of touching various screen elements like buttons and
controls. This is much easier to do with a mouse pointer than a pudgy finger.
Based on the prior experience of many mobile testers, this difference is a critical
one, and the biggest, for testing application design and function. Using a mouse
with the simulator, you do not get the full effect of having to scroll through a large
list view of items or having to play ‗whack-a-mole‘ on the screen with your thumb
because button placement for navigating multiple screens is inconsistent.
Mobile developers are strongly encouraged to ensure that application testing
begins early, and happens often, on the mobile device itself rather than on a
simulator. The same holds true for tablet devices.
33 In-the-Wild-Testing
Mobile Testing Tip
“Even on the same mobile platform, screen sizes and resolutions can vary based on device type. For instance, the screen size and resolution on the HTC Incredible is different than that on the HTC EVO 4G. Consequently, for an application to have a consistent look and feel across both devices and across a variety of other devices, user interface elements and graphics need to be scalable.”
-Web Trends, Mobile Analytics
Benefits of “In the Wild” Testing
Screen Size and Viewing:
Even though the simulator application mimics the size of the iPhone screen, it‘s
still a different experience viewing an application on the device rather than a
monitor. A screen that is readable on a simulator may not be – and usually isn‘t –
as readable on the device.
Data Entry:
No contest here. It would be a terrible mistake to develop an application that‘s
―data entry intensive‖ and test it on a simulator, with full keyboard and mouse
access, and believe that it‘s ready to be released. Trying to type even a sentence
is a challenge on a touch screen keyboard.
Functionality:
In the world of mobile, there are some things you just can‘t test on the simulator.,
including the use of accelerometers and location functions to name a few.
Today‘s mobile devices are pushing the boundaries of traditional application
design and testing methodologies. While fundamentally still the same, those
methodologies need to be applied in new ways, especially with regard to testing
application usability.
34 Screen Size, Data Entry and Functionality
Mobile Testing Tips
Interrupts: Test how the device and application behaves when it is interrupted by another action, e.g. incoming/outgoing call, SMS, MMS, video-call, low battery, low/no connectivity.
Device Settings: Change the device settings: Date and Time, Languages (Advanced regional settings), Connections, Shortcuts – anything you have control over – and learn how each and combined settings impact the device.
- Bernard Lelchuk
Conclusion
Testing no longer has to be a neglected phase of mobile application
development. There was a time when the testing matrix was indeed too complex,
too burdensome to be completed using standard means. But thanks to the rapid
evolution of crowdsourced testing, all that has now changed.
The companies that are now leveraging crowdsourced testing - particularly in the
mobile space - are gaining more of a competitive advantage than they perhaps
even realize. As the mobile market doubles and triples in size over the next
decade, those that have made testing coverage a priority will enjoy their ROI in
terms of increased market share, profitability and above all, user loyalty. Those
who continue to neglect the testing phase will struggle at best, and at worst, cease
to exist. It‘s that simple.
Either way, the future of mobile applications remains bright. Once reserved for a
tech-savvy niche in the business world, the use of mobile applications is now
firmly entrenched in the mainstream. There‘s no turning back when it comes to
mobile applications, and that includes methods for testing them.
For more on how crowdsourcing can complete your mobile testing efforts,
chat with one of our QA gurus by clicking here or by calling 800.445.3914.
35 Summary
Crowdsourced Capacity
“Small development firms have limited means today to beta test and peer review their applications with a cross section of representative users. Given the hundreds of thousands of mobile apps, we believe that efficient (crowd-sourced) testing of apps in a global market of users is considerably under-utilized.”
- VisionMobile
APPENDIX: MOBILE PLAYERS
36
Mobile Operating Systems
iPhone OS from Apple
The iPhone uses its well known operating system, the iPhone OS, which is
derived from Mac OS X. The iPhone OS has four abstraction layers: the Core OS
layer, the Core Services layer, the Media layer, and the Cocoa Touch layer. The
operating system uses roughly 500 megabytes of the device‘s storage. The
iPhone app store is the online market for downloading these applications. Third
party applications were not officially supported until the release of iPhone OS 2.0
on July 11th 2008. Presently, XCode 3.1 is the development environment and the
applications are written in Objective C (a variant of language C). There is no doubt
that the iOs will continue to be among the preferred mobile operating systems
worldwide.
Android from Google
Developed by Google, Android is an open source, Linux-derived platform backed
by Google, along with major hardware and software developers (such as Intel,
HTC, ARM, and eBay, to name a few). Together, they form the Open Handset
Alliance. This OS, though very new, already has a cult following among
programmers eager to develop apps for its flexible, Open Source, back end.
Android promises to give developers access to every aspect of the phone‘s
operation. This lends many to foresee the promise of further growth for the
Android platform. Like the iOS, Android figures to remain a prominent figure in the
future of mobile application development.
37 Appendix
Mobile Operating Systems
RIM Blackberry OS
The BlackBerry OS is a proprietary software platform, created by Research In
Motion, for their BlackBerry phones. The OS is focused on easy operation and was
originally designed for business professionals. The Blackberry operating system
provides multi-tasking and supports specialized input devices that have been
adopted by RIM for use in its handhelds, particularly the trackwheel, trackball, and
most recently, the trackpad and touchscreen.
The BlackBerry platform is perhaps best known for its native support for corporate
email, through MIDP 1.0 and, more recently, a subset of MIDP 2.0, which and allows
complete wireless activation and synchronization with Microsoft Exchange, Lotus
Domino, or Novell GroupWise email, calendar, tasks, notes, and contacts, when
used in conjunction with BlackBerry Enterprise Server. The operating system also
supports WAP 1.2. Recently it has seen a surge in third-party applications and has
been improved to offer full multimedia support.
Linux Operating System
The Linux operating system is strongest in China, where it is used by Motorola, and
in Japan, where it is used by DoCoMo. Rather than being a platform in its own right,
Linux is used as a basis for a number of different platforms developed by several
vendors, including Android, LiMo, Maemo, Openmoko and Qt Extended, which are
mostly incompatible. PalmSource (now Access) is moving towards an interface
running on Linux. Another platform based on Linux is being developed by Motorola,
NEC, NTT DoCoMo, Panasonic, Samsung, and Vodafone.
38 Appendix
Mobile Operating Systems
HP/Palm webOS
The HP/Palm webOS is Palm‘s next generation operating system. PalmSource
traditionally used its own platform developed by Palm Inc. Underneath the graphical
user interface sits a software stack that has much in common with mainstream Linux
distributions for desktop computers. webOS uses, among other components, the
Linux kernel, Upstart, GStreamer as multimedia framework, libpurple for instant
messaging, Simple DirectMedia Layer (SDL) for game programming, PulseAudio for
audio management, and WebKit Applications for webOS are written in JavaScript,
HTML, AJAX, and some other common ‗web‘ technologies. It supports streaming
video in RTSP, H.263, and H.264 formats.
Bada from Samsung
The Bada operating system has a kernel configurable architecture, which allows the
use of either the Linux kernel, or another Real-time operating system (RTOS) kernel.
The Linux kernel is the preferred choice for smartphones, while RTOS is used for
cheaper feature-phones, due to its smaller memory footprint. It has a web browser
based on the open-source WebKit, and also features Adobe Flash. Both WebKit and
Flash can be embedded inside native Bada applications. Bada offers interactive
mapping with Point of interest (POI) features, which can also be embedded inside
native applications.
Bada supports various sensors, such as motion sensing, vibration control, face
detection, accelerometer, magnetometer, tilt, and GPS, which can be incorporated
into applications.
39 Appendix
Mobile Operating Systems
Windows Phone 7 from Microsoft
On February 15th, 2010 Microsoft unveiled its next-generation mobile OS,
Windows Phone 7. The new mobile OS includes a completely new over-hauled UI
called ―Metro‖. It includes full integration of Microsoft services such as Zune, Xbox
Live and Bing.
Windows mobile is based on the Windows CE 5.2 kernel, and features a suite of
basic applications developed using the Microsoft Windows API. It is designed to
be somewhat similar to desktop versions of Windows, feature-wise and
aesthetically.
Third-party software development is available for this OS, with several options for
developers to use when deploying a mobile application. This includes writing
native code with Visual C++, writing managed code that works with the .NET
Compact Framework, or Server-side code that can be deployed using Internet
Explorer Mobile or a mobile client on the user‘s device. The .NET Compact
Framework is actually a subset of the .NET Framework and hence shares many
components with software development on desktop clients, application servers,
and web servers which have the .NET Framework installed, thus integrating
networked computing space called the cloud.
It has been criticized for having a user interface which is not optimized for touch
input by fingers; instead, it is more usable with a stylus. However, unlike iPhone
OS, it does support both touch screen and physical keyboard configurations.
40 Appendix
Mobile Development Platforms
BREW (Binary Runtime Environment for Wireless) from Qualcomm
Brew is an application development platform that was originally developed for
CDMA mobile phones, but it also supports GSM for some devices. It debuted in
September 2001 as a software platform. It can download and run small programs
for playing games, sending messages, sharing photos, and the like. The main
advantage of BREW platforms is that the application developers can easily port
their applications between all Qualcomm devices. BREW acts between the
application and the wireless device on-chip operating system in order to allow
programmers to develop applications without needing to code for system interface
or understand wireless applications.
J2me by Sun
Java 2 Micro Edition (J2ME) is Sun‘s version of Java aimed at machines with
limited hardware resources such as PDAs, cell phones, and other consumer
electronic and embedded devices. J2ME is aimed at machines with as little as
128KB of RAM and with processors a lot less powerful than those used on typical
desktop and server machines. J2ME actually consists of a set of profiles. Each
profile is defined for a particular type of device — cell phones, PDAs, microwave
ovens, etc. — and consists of a minimum set of class libraries required for the
particular type of device and a specification of a Java virtual machine required to
support the device.
There are now more than 2 billion Java ME enabled mobile phones and PDAs.
41 Appendix
Mobile Development Platforms
LiMo
The LiMo Platform is developed by the LiMo Foundation (founded by a group of
cellular handset makers and network operators) for handheld devices, and it uses
Linux as its operating system. It has a modular plug-in architecture, and supports
DRM. LiMo application developers will be able to use SDKs to write managed code
running in a Java virtual machine, browser apps for WebKit, and native code.
Qtopia
Qtopia is a platform for embedded Linux devices that provides the end-user with a
set of core applications, such as PIM applications (Address Book, Calendar and
Todo) and a Media Player. Qtopia provides the developer with a clean object-
oriented API to write applications for an embedded device. Most of the API
presented to the developer is the same API offered by Trolltech for cross-platform
development between Windows, Mac and the UNIX/Linux. The same code can be
shared and reused on all of these desktop platforms and now Qtopia is just another
platform to run Qt applications.
The Qtopia platform encompasses both the software that runs on an embedded
device and the software that runs on an end-user desktop. The software that runs on
the desktop, called Qtopia Desktop, allows the user to synchronize their data
between the embedded device and the desktop. Qtopia Desktop has built-in
versions of the PIM application found in Qtopia. Qtopia Desktop allows developers
to write plug-ins to synchronize their information.
42 Appendix
Acknowledgements: Special thanks to Bernard Lelchuk, Brad Sellick, Jigar
Patel and Anuj Gupta for their contributions to this eBook.
About uTest uTest provides in-the-wild testing services that span the entire software
development lifecycle – including functional, security, load, localization and
usability testing. The company‘s community of 45,000+ professional testers from
180 countries put web, mobile and desktop applications through their paces by
testing on real devices under real-world conditions.
Thousands of companies -- from startups to industry-leading brands – rely on
uTest as a critical component of their testing processes for fast, reliable, and cost-
effective testing results.
More info is available at www.utest.com or blog.utest.com, or you can watch a
brief online demo at www.utest.com/demo.
uTest, Inc.
153 Cordaville Road
Southborough, MA 01772
p: 1.800.445.3914
w: www.utest.com
43 About uTest