testdroid: tips and tricks with calabash for mobile app testing
DESCRIPTION
Interested in a presentation video? Sign up at testdroid.com/webinars-archive Calabash is a cross-platform test automation framework for Android and iOS native and hybrid applications. Calabash's easy-to-understand language syntax enables - not only developers, testers and QA folks - but also people with no programming background or experience can write and execute automated acceptance tests for their mobile apps.TRANSCRIPT
webinar
Tips & Tricks with Calabash for Mobile App Testing
13 March 2014
Ville-Veikko HelppiTechnical Product [email protected]
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to
Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips & Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
2
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to
Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
3
Why Apps Fail?webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
4
Prerequisites for Mobile App Testing• Testing environment as authentic as possible– Real devices <-> emulators/simulators
• Test coverage needs to be adequate– Software is not identical in all Android/iOS devices– Hardware is even more fragmented landscape
• Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes• Test automation framework that works for you
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
5
What Framework Works You The Best?• High-level & declarative test automation
frameworks work well when app is under dev• Easy to integrate with continuous integration• Distance between use case and actual test• What do you build: Native, Hybrid or Web app?• The technical competence in your organization• Desired outcome from test automation
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
6
What Improves the App Quality?• Automate generic things as much as you can• During the development your app changes –
the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to
get you quickly covered in the global markets
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
7
Why Real Devices are Must-to-Have?• Emulators/simulators won’t help you to test…– User Experience and Usability– Hardware– Software– Infrastructure
webinar
0 % = the percentage of your app users that use emulator to run your app!
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
8
Basics: Different Ways of Doing Test Automation
webinar
Automatic test exercisers Record and PlaybackHand written test scripts
Benefits:
Accurate, specific to your testing needs, plenty of options with frameworks, tools
Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty
Fastest & extremely automated, excellent for smoke testing/quick testing, availability
Tradeoffs:
Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)
Compelling Recorder+Playback tools available for only few test automation frameworks
Not accurate as real test cases
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
9
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares
to Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
10
Introduction to Calabashwebinar
• Behavior driven test framework for native Android & iOS, and for hybrid web apps (embedded webviews)
• Tests are described in Cucumber and then converted to Robotium or “similar-to-Frank“ in run time
• Supports currently about 80 different natural language commands (controllers). New controllers can be implemented in Ruby or Java
• If your app uses custom UI Widgets you have to create controllers for those in order to test your app
• Calabash tools include a command line inspector for finding right UI element names/ids
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
11
Android Example: The Family Tree of Test Automation Frameworks
webinar
JUnit
Android Instrumentation Framework
Robotium Espresso
UI Automator
Calabash
Appium
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
12
Why Calabash is A Good Choice? • Cross-platform test automation framework• Human-readable syntax/tests• Not much changes needed for tests when the
mobile app changes (e.g. UI components)• Pretty easy to integrate with any CI• Great choice for teams that have limited
resources for development and/or testing
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
13
Calabash-iOS: Architecturewebinar
• Calabash installs an HTTP server as an instrumentation package that listens commands from Calabash server
• Tests are executed on server side• Each test scenario is described in
Cucumber• Ruby Client library converts Cucumber
commands to ”Frank” method calls• Webview support is implemented with
Javascript injection to the Webview
iOS device
Your app
Calabash HTTP server
Test server
Step definitions, Ruby client
library
Features
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
14
Calabash-Android: Architecturewebinar
• Calabash uses Android instrumentation test server (a separate app)
• Based on ActivityInstrumentationTestCase2
• Tests are executed on server side• Predefined and custom steps• Ruby Client library converts Cucumber
commands to Robotium method calls• Webview support is implemented with
Javascript injection to the Webview
Android device
Your app
Instrumentation Test Server
Test server
Step definitions, Ruby client
library
Features
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
15
Calabash: Code Examplewebinar
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello" Then I press view with id "Sign in" Then I enter text "username" into "login_username" Then I enter text "password" into "login_password" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post"
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
16
How Calabash Compares to Others?webinar
Calabash Robotium uiautomator Espresso Appium
Android Yes Yes Yes Yes Yes
iOS Yes No No No Yes
Mobile web Hybrid(webviews)
Yes (Android)
Limited to x.y clicks
No Yes (Android & iOS)
Scripting Language
Ruby Java Java Java Almost any
Test creation tools
CLI Testdroid Recorder
UI Automator viewer
Hierarchy Viewer
Appium.app
Supported API levels
All All 16 => 8, 10, 15-19 All
Community Active Contributors Google Google Active
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
17
Example #1: Calabash with Cloudwebinar
Developers
Source code
Repository
Build failed
Polling
Notifying
Successful build
QA & Testers
Building
Error reporting
Test Cases
Application
Users / Consumers
Rele
ase
Com
mit
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
18
Example #2: Calabash with CIwebinar
Developers
Source code
Repository
Build failed
Polling
Notifying
Successful build
QA & Testers
Building
Error reporting
Test Cases
Application
Users / Consumers
Rele
ase
Com
mit
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
19
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to
Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
20
Step Definitions webinar
• Fast and easy to modify definitions and features• Predefined and Custom steps– Predefined steps offer the way to start instantly– Custom steps provide more options & can be easily
tailored for your app• Works for both, Android and iOS, when added:
– calabash.framework– ActivityInstrumentationTestCase2
• Custom steps extremely easy to modify & create
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
21
Test Script – called Featureswebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
22
Feature: Login feature
Scenario: As a valid user I can log into my app
I wait for text "Hello" Then I press view with id "Sign in" Then I enter text "username" into "login_username" Then I enter text "password" into "login_password" Then I wait for activity "HomeTabActivity" Then I press view with id "menu_compose_tweet" Then I enter text "Testdroid" into field with id "edit" Then I press view with id "composer_post"
Example: Predefined Stepswebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
23
Given /^I press the "([^\"]*)" button$/ do |buttonText| performAction('press_button_with_text', buttonText)end
Then /^I press button number (\d+)$/ do |buttonNumber| performAction('press_button_number', buttonNumber) end
Then /^I press image button number (\d+)$/ do |buttonNumber| performAction('press_image_button_number', buttonNumber) end
Then /^I press view with id "([^\"]*)"$/ do |view_id| performAction('click_on_view_by_id',view_id)end
From Definition to Featurewebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
24
Then /^I press view with id "([^\"]*)"$/ do |view_id| performAction('click_on_view_by_id',view_id) end
Then I press view with id "menu_compose_tweet"
Definition:
Feature:
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to
Other Test Automation Frameworks• Building the Calabash Scripts• Tips and Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
25
#1: Use Only Real Deviceswebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
26
#2: Calabash is Perfect for “Test Early, Test Often” Approach
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
27
#3: Integrate Dev & Testing -> CI
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
28
Com
mit
Developers
Source code
Repository
Build failed
Polling
Notifying
Successful build
QA & Testers
Building
Error reporting
Test Cases
Application
Rele
ase
Users / Consumers
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to
Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
29
Agenda• Prerequisites for Mobile App Testing• Calabash Introduction and How It Compares to
Other Test Automation Frameworks• Inside the Calabash – Building Those Scripts• Tips and Tricks with Calabash• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
30
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
31