mobiletesting sample

Hands-On Mobile AppTestingA guide for mobile testers andanyone involved in the mobile appbusiness.

Daniel Knott

This book is for sale at http://leanpub.com/mobiletesting

This version was published on 2015-01-10

This is a Leanpub book. Leanpub empowers authors andpublishers with the Lean Publishing process. LeanPublishing is the act of publishing an in-progress ebookusing lightweight tools and many iterations to get readerfeedback, pivot until you have the right book and buildtraction once you do.

2014 - 2015 Daniel Knott

http://leanpub.com/mobiletestinghttp://leanpub.comhttp://leanpub.com/manifestohttp://leanpub.com/manifesto

Tweet This Book!Please help Daniel Knott by spreading the word about thisbook on Twitter!

The suggested tweet for this book is:

Check out the book Hands-On Mobile App Testing by@dnlkntt

The suggested hashtag for this book is #mobiletesting.

Find out what other people are saying about the book byclicking on this link to search for this hashtag on Twitter:

https://twitter.com/search?q=#mobiletesting

http://twitter.comhttps://twitter.com/search?q=%23mobiletestinghttps://twitter.com/search?q=%23mobiletesting

For my wife Sarah. Thank you very much for your supportand encouragement in writing this book.

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . 1

Chapter 1 Whats special about Mobile Testing? . 6

About the Author . . . . . . . . . . . . . . . . . . 21

IntroductionMobile phones have been around since the middle of the1970s. The devices have of course changed profoundly sincethen, but the biggest change came in 2007 when Applepresented its first iPhone. From that moment on, themobilesmartphone market has only known one direction - UP!Eight years later and touch devices such as smartphonesand tablets have become ubiquitous. More than twomillion(iOS Store numbers, Android Play Store numbers, num-bers from June 2014) apps are available for download inthe stores of the biggest vendors and this number is stillrising. There are apps for every aspect of our lives, rangingfrom photos and music to office applications and gamesand on to fitness and health. But what about the qualityof those apps? Are they reliable, trustworthy, easy to use,well developed and tested?

This book is a practical guide on mobile testing for anyonewho works in the mobile business, but is especially aimedat mobile testers.

http://www.engadget.com/2014/06/02/apples-wwdc-2014-in-numbers-40-million-on-mavericks-and-more/

http://www.appbrain.com/stats/number-of-android-apps

1

http://www.engadget.com/2014/06/02/apples-wwdc-2014-in-numbers-40-million-on-mavericks-and-more/http://www.appbrain.com/stats/number-of-android-appshttp://www.engadget.com/2014/06/02/apples-wwdc-2014-in-numbers-40-million-on-mavericks-and-more/http://www.engadget.com/2014/06/02/apples-wwdc-2014-in-numbers-40-million-on-mavericks-and-more/http://www.appbrain.com/stats/number-of-android-apps

Introduction 2

Why I wrote this Book

It all started in 2010 when I had the opportunity to workon my first mobile project. The mobile team I worked inwas responsible for developing a mobile web app, a nativeAndroid app and a native iOS app. This was the companysfirst mobile project and a completely new testing envi-ronment for the quality assurance department. Togetherwith a colleague, I had the chance to build up a mobiletesting strategy from scratch. We evaluated several testautomation tools to see which one fits best in our softwaredevelopment lifecycle. At that time, mobile testing toolswere few and far between, and at an very early develop-ment stage. We then tried several testing approaches andtools. Of course we failed with some of them, but in theend the whole team, the company and our customers werehappy.

Another reason why I wrote this book was because of myblog http://www.adventuresinqa.com. I started blogging in2011 after giving a presentation at the Agile Testing Daysin Potsdam, Germany. This was my first talk at a majortesting conference and I was the only speaker on the agendawho spoke about mobile testing. After my presentation Iwas very busy for the rest of the conference as a lot ofpeople approached me to ask about mobile testing, theapproaches I use, what kind of tools I use and so forth. Thehuge interest and the lack of knowledge in mobile testingconvinced me to start writing a blog. The goal was to share

http://www.adventuresinqa.com

Introduction 3

my knowledge of mobile testing and to exchange viewsand ideas with other mobile testers, while also improvingmy written English skills. So far Ive written about 90 postscovering mobile apps and testing, and I never expected somany people from around the world to take an interest inmy blog. The feedback I got so far has been great and itconvinced me to take the next step.

That step is what youre reading: a book about mobiletesting where my practical experience and knowledge foranyone involved in the mobile business is written down. Ichose Leanpub to publish this book because writing a bookabout fast-changing technology in a conventional way issimply not able to keep the pace with developments. I hopeyou enjoy reading this book and can learn something newabout the mobile business.

Who should read this Book?

This book is aimed at anyone interested in mobile apps andmobile testing, ranging from junior to expert mobile testerswho are already involved in mobile development teams.

This book is also ideal for software test managers who needto manage mobile testing teams or to select a mobile teststrategy. Its also great for software testers who are new tothis topic and want to switch to mobile technologies.

Software developers who want to knowmore about mobiletesting and testing their mobile apps have also come to the

Introduction 4

right place.

This book is also intended for product managers lookingto gain further insights into the challenging job of mobiletesting.

Topics covered in this Book

This book covers the following topics:

The specialties of mobile testing. Introduction to mobile devices and apps. Challenges in mobile testing. Different techniques to test mobile apps. Mobile test automation and tools. Additional mobile testing methods. Mobile test and launch strategy. Important skills for mobile testers.

Each chapter focuses on the practical side of mobile testing.Sure, there will be some theoretical parts, but most of thecontent is based on real-life experience as a mobile tester.

How to use this Book

This book is a practical guide on mobile testing. You canread it from front to back to get an overview of mobile

Introduction 5

testing, or you jump straight to the chapters youre mostinterested in. Theres one important piece of advice youshould bear in mind while reading this book: make sureyou have at least one mobile device next to you so you cantry out the things you will read. If you want to get startedwith the test automation tools mentioned in this book, nowwould be a good time to get your computer.

Feedback

By using Leanpub to publish this book, I can swiftly reactto technological changes within the mobile testing businessupdate the contents of this book accordingly. Besides that,Im really looking forward to receiving your feedback so Ican keep on improving this book. Please send any feedbackyou may have to [email protected].

mailto:[email protected]

Chapter 1 Whatsspecial about MobileTesting?Before I start writing about the specialties of mobile testing,Id like to share a true story with you.

Whats special about mobile testing? Someone asked methis exact question several years agowhile on a testing con-ference. I started talking about mobile technologies, apps,how to test them and whats special about mobile testing.The guy simply smiled at me and said: but its softwarejust on a smaller screen. Theres nothing special aboutit. He was really arrogant and didnt see the challengespresented by mobile testing. No matter which arguments Iused to convince him, he didnt believe in the importanceof mobile technologies, apps and testing.

I met the same guy again in 2014 while on a testingconference where he talked about mobile testing. He wastalking about the importance of apps, and how importantit is to test them.

As you can see its very easy to underestimate new tech-nologies. Especially as a software tester its helpful to be

6

Chapter 1 Whats special about Mobile Testing? 7

curious about learning something new and exploring newtechnologies to broaden your skills.

So lets come back to the initial question. Whats specialabout mobile testing? I think I can assume you have atleast one mobile device, namely a smartphone. Or maybe atablet, or even both. If you look at your device(s), what doyou see? Just a small computer with little shiny icons on itsscreen? Or do you see a very personal computer with lotsof sensors and input options that contains all your privatedata? Please take a minute to think about that.

For me, my smartphone and tablet are very personal com-puters that hold almost all of my data. Be it e-mails,SMS, photos, music, videos and the like. I can access mydata no matter where I am and use my smartphone as anavigation and information system to find out more aboutmy surroundings. For that reason I expect my apps to bereliable, fast and easy to use.

In these four sentences, I described my personal expec-tations of mobile devices and apps. But you may haveentirely different expectations, as does the next person.And this brings me to my first specialty about mobiletesting.

User Expectations

In my opinion, the user of an app is the main specialty andmain challenge for mobile teams. The fact that every user


has unique expectations makes it so difficult to develop anddeliver the right app to customers. As several reports andsurveys show, mobile users have far higher expectations ofmobile apps when compared to other software like browserapplications. The majority of reports and surveys state thatnearly 80% of users delete an app after using it for thefirst time! The top four reasons for deletion are always baddesign and usability, loading time and crashes immediatelyafter installation. Nearly 60% of users will delete an appthat requires registration, while more than half of usersexpect an app to launch in under two seconds. If the appneeds more time it gets deleted. Again more than half ofusers experience crashes the very first time they start anapp. An average user checks his mobile device every sixminutes and has around 40 apps installed. Based on thosenumbers you can deduce that mobile users have reallyhigh expectations when it comes to usability, performanceand reliability. Those three points are by far the biggestcomplaints when mobile users were asked about theirexperience with mobile apps.

Currently there are more than two million apps (2014)available in the app stores of the biggest vendors. A lot ofapps perform the same task, meaning that theres alwaysat least one competitor app, which makes it very easy forconsumers to download a different app as its just a singletap away. Here are some points you should keep in mind

http://offers2.compuware.com/rs/compuware/images/Mobile_App_Survey_Report.pdf

http://offers2.compuware.com/rs/compuware/images/Mobile_App_Survey_Report.pdfhttp://offers2.compuware.com/rs/compuware/images/Mobile_App_Survey_Report.pdfhttp://offers2.compuware.com/rs/compuware/images/Mobile_App_Survey_Report.pdf


when developing and testing a mobile app:

Gather information about your possible target group. Ask your customers about their needs. Your app needs to solve a problem for the user. Usability is really important. Your app needs to be reliable and robust. App performance is really important. Apps need to be beautiful.

There is of course a plethora of other things you should takeinto account, but if you pay attention to the above points,your users are likely to be happy.

Youve probably already heard of the KISS principle. KISSis an acronym for Keep It Simple, Stupid and is always auseful reminder especially for software projects to notinflate the software with just another function or option.Keeping it small, easy and simple is the best way to go inmost cases and is likely to make your customers happy.Inspired by KISS, I came up with my own principle formobile apps: KIFSU.

http://people.apache.org/~fhanik/kiss.html

http://people.apache.org/~fhanik/kiss.htmlhttp://people.apache.org/~fhanik/kiss.html


KIFSU

This abbreviation is a good mnemonic to help you covercustomer needs and a constant reminder so that you dontinflate apps with useless functions.

Mobility and Data Networks

Another challengemobile apps have to deal withmore thansoftware running on computers is the fact that users aremoving around while they use apps, which requires aninternet connection to fetch data from the backend andserve the user with updates and information.

Mobile apps need to be tested in real life, in real environ-ments where the potential user will use them. For example,if youre testing an app for snowboarders and skiers whereslope information can be accessed, where the app is ableto record the speed of the current downhill run and whereusers are able to share records directly with their friends,you need to perform such tests on a slope. Otherwise youcant guarantee that every feature will work as expected.

Of course there are parts of an app that you can test in alab situation, e.g. slope information availability or whether


or not the app can be installed, but what about recordinga persons speed, the weather conditions or the internetconnection at the top of a mountain?

The weather conditions on a mountain can in particularbe very difficult to handle as they can of course rangefrom sunshine to a snowstorm. In such scenarios you willprobably find lots of bugs regarding the usability anddesign of an app. Maybe youll also find some functionalbugs due to the temperature that may have an impact onyour hardware and, in turn, your app.

As already mentioned, the internet connection could differin such regions. You will probably have a good networkconnection with high speed at the top of the mountainand a really poor one down in the valley. What happensif you have a bad or no internet connection while usingthe app? Will it crash or will it still work? What happensif the mobile device changes network provider while usingthe app? (This is a common scenario when using apps closeto an international border, e.g. when snowboarding in theAlps.)

All of these questions are very hard to answer when testingan app in a lab. You as a mobile tester need to be mobileand connected to data networks while testing apps.

As you can see, its important to test your app in real-life environments and to carry out tests in data networkswith different bandwidths as the bandwidth can have ahuge impact on your app, e.g. a low bandwidth can cause


unexpected error messages while the switch between afast and a low bandwidth can cause performance issues orfreezes.

Heres an exercise for you: Take any app youwant and note down three usage scenarioswhere the environment and/or network con-nection could cause it problems.

Mobile Devices

Before you continue reading, I want you to pick up yourmobile device and look at it. Take your device in your handand look at every side of it without turning it on. What doyou see?

You will most likely see a device with a touch-sensitivescreen, a device with several hardware buttons with acharger, a headphone connection and a camera. Thatsprobably it youre not likely to have more than five hard-ware buttons (except smartphones with a physical key-board).

In an era where the words cell phone have become syn-onymous with smartphone, its important to rememberthat there used to be other types of cell phones, so-calleddumb phones and feature phones which have lots morehardware buttons for making a call or typing a message.


Conventional dumb phones are only able to make a call,type a message or store a contact list, theyre not usuallyconnected to the internet. The more advanced ones, thefeature phones, have games, a calendar or a very basic webbrowser with the option to connect to the internet. Butall these phones are really basic in terms of functionalityand expandability as users arent able to install apps oreasily update the software to a newer version, if it allpossible. Both types of phone are still available, especiallyon emerging markets, but since 2013 more smartphoneswere sold worldwide than dumb phones or feature phones,and this trend is set to continue as time goes on. In fact, inthe next couple of years dumb phones and feature phoneswill be a thing of the past.

The phones we use nowadays are completely differentfrom the old ones. Current smartphones are mini super-computers with lots of functionality in terms of hardwareand software. Theyre packed with various sensors suchas brightness, proximity, acceleration, tilt and many more.Besides that, all modern smartphones have a front and rearfacing camera, various communication interfaces like Blue-tooth, NFC, GPS as well as Wi-Fi and cellular networks toconnect to the internet. Depending on the mobile platformand mobile manufacturer, you may find an array of otherhardware features.

From a software point of view, smartphones offer lots of ap-plication programming interfaces (APIs) for manufactur-

http://www.gartner.com/newsroom/id/2665715

http://www.gartner.com/newsroom/id/2665715http://www.gartner.com/newsroom/id/2665715


ers, developers and users to extend smartphone capabilitieswith apps.

If you just focus on the major mobile platforms iOS andAndroid, there are plenty of hardware and software com-binations mobile testers have to deal with. The fact thatthere are so many combinations is also known as fragmen-tation. Mobile device fragmentation is a huge topic and yetanother challenge when it comes to mobile testing.

You cant test your app with every possible hardware andsoftware combination. And if you bear in mind that youshould test your app in a real environment, this makes itevenmore impossible.Mobile testers need to find a strategyto downsize the effort of testing on different devices and tofind a way to test on the right devices.

But how can that be accomplished? By just testing on onemobile platform? By just testing on the latest device? Byjust testing with the latest software version?

Before you define a strategy, you should keep in mindthat every app is unique, has unique requirements, otherproblems to solve and that each app has a unique userbase. With these points in mind, you can ask yourself thefollowing questions to find the right mobile devices fortesting:

Who is my user base? How old is the average user? How many men or women are in my target group?


Which platform is used most among that user base? Which device is used most? Which software version is installed on most of thephones?

What kind of sensors does my app use? How does the app communicate with the outsideworld?

What is my apps main use case?

Of course there are lots more questions to ask, but ifyou answer most of the ones above, the list of possibledevices you should consider testing is much shorter. In laterchapters, I will describe other ways and techniques to selectthe right devices for mobile testing.

Mobile Release Cycles

Now you know how to find the right devices for testingyour app, it doesnt mean that this process is over. To behonest, its never going to end!

The main mobile manufacturers release a new flagshipphone with more features every year. In and around thosereleases they bring out other phones for different userscenarios and user groups. This is especially true in theAndroid world where every new phone comes with anew version of the operating system packed with newfeatures, designs or APIs. There are multiple software


releases within the course of a year, ranging from bug fixesto feature releases. You as a mobile tester need to be surethat your app will run on the latest hardware and software.

But how should you handle these situations? By buyingevery phone that appears on the market? By constantlyupdating to the latest operating system version?

Again, the most important point is your target group andthe app youre testing. When you know that your targetgroup always uses your app on the latest and fastest phoneson the market, you need to buy those phones as soon asthey appear. Irrespective of whether or not your targetgroup is up to date, you should always monitor the mobilemarket.

You need to knowwhen themain vendors are due to releasenew flagship phones that are likely to be bought by a lot ofpeople. You also need to know when the operating systemsreceive patches, new features or new design patterns.

So the answer is yes and no. Of course you dont needto buy every phone thats on the market but you shouldconsider updating to the latest operating system version.When doing so, keep inmind that not every user will installthe update. There are many people who dont know howto do that or they dont care about new versions. You needat least some phones that are still running older versionsof the operating system to see how the app reacts in thatenvironment. Older versions of the operating system arealso needed to reproduce reported problems and bugs.


A good way to manage all this is to stick with the operatingsystem version on the phone and buy new phones withthe latest software version. This of course leads to anotherproblem, its really expensive! Not every manager wants tospend so much money on mobile devices when a phone isonly going to be used for a couple of months. A solutionfor that is to rent devices. There are several providers andopen device labs where you can rent a device for a certaintime (a list of providers can be found in chapter 3). Anotherway to rent devices is the mobile device cloud as there area number of providers who build up mobile device cloudswhere mobile testers have exclusive access to the physicaldevices. Just use your search engine and check them out.

In the mobile projects Ive worked on, we always had thetop 10 to 15 devices used by our target group in differentvariations for developing and testing. This was a goodnumber of devices that covered nearly 90% of our targetgroup. With those 10 to 15 devices we were able to findmost of the critical bugs while the remaining 10% of deviceswe didnt havewere of nomajor consequence to the projector user expectations.

In order to handle the fast pace of mobile release cycles,you should keep the following things in mind:

Monitor the mobile device and software market. Know when new phones will be rolled out. Find out about the new features of the operatingsystems.


Keep an eye on your target group to see if newdevices are showing up in your statistics.

Think twice before updating a phone to the latestoperating system version.

Buy a new phone with the latest operating systemversion.

If buying is not an option, rent the devices.

Updating, buying and maintaining all of your devices isa challenging task and should not be underestimated! Atsome point, depending on the number of used test deviceswithin the project, this could be a full-time job.

Mobile Testing is SoftwareTesting

Lets come back to the story I told at the beginning of thischapterwhen the guy at the conference didnt believe in theimportance of mobile testing. The guy had the attitude thatmobile testing is not real software testing. In his opinion,mobile apps were only small programs with less function-ality and no real challenges when it comes to softwaretesting. But this is definitely not the case. If you look atthe topics I described in this chapter you should now havean initial impression about the challenging job of a mobiletester. Mobile testing is totally different to testing normalsoftware applications such as web or desktop applications.


Withmobile apps, physical devices have far more influenceover the software that is running on them when comparedto other software like web applications. Due to the factthat there are so many different smartphones available onthe market, mobile testers need to focus a lot more onhardware during the testing process. In addition to that,users moving around and using different data networksforce mobile testers to be on the move while testing.

Besides the hardware, user expectations play an importantpart in the daily business of a mobile tester and need to betaken seriously.

There are many more topics and issues mobile testers needto know about in order to help the whole team release asuccessful app. The following chapters will cover topicslike:

More challenges for mobile testers and solutions tothose challenges.

How to test mobile apps systematically. How to select the right mobile test automation tool. The different concepts of mobile test automationtools.

How to find the right mobile testing strategy. Additional mobile testing methods. Required skills for mobile testers.

Keep the topics from this chapter in mind as a good startingpoint. Keep your app simple and fast (remember KIFSU).


Test while youre on the move and test on different devicesbased on your target group.

About the Author

Daniel Knott

Daniel Knott has been working in the field of softwaredevelopment and software testing since 2003. He startedhis career as a trainee at IBM where he was involved inenterprise software development and testing. After his timeat IBM, Daniel studied computer science at the Universityof Applied Sciences in Wiesbaden, Germany. Softwaretesting soon became a passion during his time at universityand the reason why he chose a career in software test-ing. Daniel has worked at several companies in variousdifferent industries where he was responsible for testing

21

About the Author 22

web, desktop and mobile applications. During a number ofprojects he developed fully automated testing frameworksfor Android, iOS and web applications. Daniel is a well-knownmobile expert, speaker at various conferences in Eu-rope, and a blog author http://www.adventuresinqa.com.Furthermore, Daniel is the founder and organizer of twolocal software testing user groups in central Germany. Oneis the Software Test User Group Rhein Main and the otherone is the Rhein Main Mobile Quality Crew.

Get in touch with Daniel:

XING: Profile LinkedIn: Profile Twitter: @dnlkntt E-mail: [email protected]

http://www.adventuresinqa.comhttp://www.stugrm.dehttp://www.meetup.com/Rhein-Main-Mobile-Quality-Crewhttps://www.xing.com/profile/Daniel_Knotthttp://www.linkedin.com/pub/daniel-knott/1a/925/993https://twitter.com/dnlkntt

http://www.adventuresinqa.comhttp://www.stugrm.dehttp://www.meetup.com/Rhein-Main-Mobile-Quality-Crewhttps://www.xing.com/profile/Daniel_Knotthttp://www.linkedin.com/pub/daniel-knott/1a/925/993https://twitter.com/dnlknttmailto:[email protected]://www.adventuresinqa.comhttp://www.stugrm.dehttp://www.meetup.com/Rhein-Main-Mobile-Quality-Crewhttps://www.xing.com/profile/Daniel_Knotthttp://www.linkedin.com/pub/daniel-knott/1a/925/993https://twitter.com/dnlkntt

Table of ContentsIntroductionChapter 1 What's special about Mobile Testing?About the Author

mobiletesting sample

Documents

mobile testing

leanpub book

mobile project

mobile appbusiness

mobile business

mobile team

right book

book byclicking