calabash, an open-source automated testing technology for native mobile, by karl krukow
DESCRIPTION
Mobile is now facing problems that have been existed in web for many years: fragmentation. In mobile, it is supporting the many phone and tablet models, operating system and versions, and languages. Calabash is an open source technology for functional testing of Android and iOS apps. It combines advanced automation technology with the Cucumber tool. Tests are written in a natural language of the business domain, and the cucumber tool can execute tests on Android and iOS devices or simulators.TRANSCRIPT
![Page 1: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/1.jpg)
Introducing Calabash
automated functional testing for mobile native apps
Karl Krukow, [email protected],
LessPainfulCodeMotion 2012, Rome
1
fredag den 23. marts 12
![Page 2: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/2.jpg)
About me
2
fredag den 23. marts 12
![Page 3: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/3.jpg)
About me
• PhD, Computer Science, University of Aarhus, 2006.
2
fredag den 23. marts 12
![Page 4: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/4.jpg)
About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java enterprise, last two years on iOS.
2
fredag den 23. marts 12
![Page 5: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/5.jpg)
About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java enterprise, last two years on iOS.
• Co-owner, iOS responsible at
2
fredag den 23. marts 12
![Page 6: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/6.jpg)
About me
• PhD, Computer Science, University of Aarhus, 2006.
• Developer at Trifork for about 6 years mostly Java enterprise, last two years on iOS.
• Co-owner, iOS responsible at
• Spare-time Hickey & Clojure fan-boy!
2
fredag den 23. marts 12
![Page 7: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/7.jpg)
Agenda
3
fredag den 23. marts 12
![Page 8: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/8.jpg)
Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
3
fredag den 23. marts 12
![Page 9: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/9.jpg)
Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
• Introduce Calabash
• Focus on iOS only
3
fredag den 23. marts 12
![Page 10: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/10.jpg)
Agenda
• Automated functional testing for native mobile
• Some desirable properties for a functional testing tool
• Introduce Calabash
• Focus on iOS only
• Live Demo:
• Calabash iOS
• LessPainful: test service and device cloud
3
fredag den 23. marts 12
![Page 11: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/11.jpg)
Functional testing
4
fredag den 23. marts 12
![Page 12: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/12.jpg)
Functional testing
4
• Functional and Acceptance tests
• Actual app, as opposed to an isolated component
• Often based on use-cases written in natural language (for domain)
• Visual appearance of screens matter! (Design guidelines, branding,..)
• As realistic an environment as practically possible
fredag den 23. marts 12
![Page 13: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/13.jpg)
Functional testing
4
• Functional and Acceptance tests
• Actual app, as opposed to an isolated component
• Often based on use-cases written in natural language (for domain)
• Visual appearance of screens matter! (Design guidelines, branding,..)
• As realistic an environment as practically possible
• For mobile apps, in particular
• Many devices, screens, OS versions, languages
• often a manual process: repetitive, expensive
• regressions: why should I test that again?
fredag den 23. marts 12
![Page 14: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/14.jpg)
5
Functional testing desiderata
fredag den 23. marts 12
![Page 15: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/15.jpg)
• Minimize distance between use cases and actual test code (DSLs?).
• Expressive and efficient to write.
• Extensible
• High-level, declarative (robustness against “minor” UI changes).
• Support testing in realistic environments (multiple real devices, on multiple OS versions, languages).
• Support Continuous integration.
5
Functional testing desiderata
fredag den 23. marts 12
![Page 16: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/16.jpg)
Cucumber and Calabash
fredag den 23. marts 12
![Page 17: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/17.jpg)
• Cucumber provides
• a framework for writing software specifications
• a tool for executing those specifications
• Specifications are written in a business readable language that is close to natural language.
• Extremely popular tool for test and specs of web applications.
• http://cukes.info/
fredag den 23. marts 12
![Page 18: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/18.jpg)
Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,so I can control access to the application
Scenario: Add test user When I touch the Add User button And I fill in text fields as follows: | field | text | | Last Name | Knorr | | Username | knorr | And I touch "Save" Then I should be on the Users screen And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
![Page 19: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/19.jpg)
Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,so I can control access to the application
Scenario: Add test user When I touch the Add User button And I fill in text fields as follows: | field | text | | Last Name | Knorr | | Username | knorr | And I touch "Save" Then I should be on the Users screen And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
![Page 20: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/20.jpg)
Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,so I can control access to the application
Scenario: Add test user When I touch the Add User button And I fill in text fields as follows: | field | text | | Last Name | Knorr | | Username | knorr | And I touch "Save" Then I should be on the Users screen And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
![Page 21: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/21.jpg)
Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,so I can control access to the application
Scenario: Add test user When I touch the Add User button And I fill in text fields as follows: | field | text | | Last Name | Knorr | | Username | knorr | And I touch "Save" Then I should be on the Users screen And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
![Page 22: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/22.jpg)
Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,so I can control access to the application
Scenario: Add test user When I touch the Add User button And I fill in text fields as follows: | field | text | | Last Name | Knorr | | Username | knorr | And I touch "Save" Then I should be on the Users screen And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
![Page 23: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/23.jpg)
Cucumber Example
Feature: As an administrator. I want to be able to add and remove users,so I can control access to the application
Scenario: Add test user When I touch the Add User button And I fill in text fields as follows: | field | text | | Last Name | Knorr | | Username | knorr | And I touch "Save" Then I should be on the Users screen And I should see a table containing "Knorr"
Scenario: ...
fredag den 23. marts 12
![Page 24: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/24.jpg)
Step Definitions• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Feature Step definitions
fredag den 23. marts 12
![Page 25: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/25.jpg)
Step Definitions• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Scenario: Add test user When I touch the Add User button ...
Feature Step definitions
fredag den 23. marts 12
![Page 26: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/26.jpg)
Step Definitions• Make the cucumber tests “come alive”
• Written in ordinary programming languages
• Mostly Ruby (but cucumber-jvm: Java, Clojure,...)
Scenario: Add test user When I touch the Add User button ...
When /^I touch the Add User button$/ do btn_txt = 'Add user' touch("button text:#{btn_txt}") end
Feature Step definitions
fredag den 23. marts 12
![Page 27: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/27.jpg)
Calabash• One interface: Cucumber, for Android and iOS tests.
• Predefined and custom steps (Ruby + soon: JVM).
• Reuse of tests across platform possible.
• Run on physical devices and simulators.
• Support for hybrid apps (embedded webviews)
• Free, open source with commercial add-ons:
• Run tests in a device cloud using the LessPainful service,
• we can write and run your tests,
• setting up continuous integration, consulting, training, support.
10
fredag den 23. marts 12
![Page 28: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/28.jpg)
Architecture iOS
iOS App--------------------Calabash library
fredag den 23. marts 12
![Page 29: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/29.jpg)
Architecture iOS
iOS App--------------------Calabash libraryCucumber
(ruby api)
features
Test report(cucumber)
fredag den 23. marts 12
![Page 30: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/30.jpg)
LessPainfulTest Execution in the Cloud
• Execute Calabash tests concurrently on many devices, OS’es, languages.
• Authentic: Not jailbroken, iOS and Android devices, rotation.
• Visual test reports.
• Comparison across models and operating systems.
• Continuous integration.
fredag den 23. marts 12
![Page 31: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/31.jpg)
fredag den 23. marts 12
![Page 32: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/32.jpg)
fredag den 23. marts 12
![Page 33: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/33.jpg)
fredag den 23. marts 12
![Page 34: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/34.jpg)
Calabash iOS: more detail
15
fredag den 23. marts 12
![Page 35: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/35.jpg)
Calabash: more detail• Very easy to get started for iOS developers/QAs.
• Declarative query language for finding views.
• Based on UISpec, but simplified and extended. (New Implementation, EPL licensed).
• Touch synthesis, supports (multitouch) gestures. Extensible. (pan, swipe, pinch, your own)
• Full power of Ruby programming language for test logic
• Interactive, exploratory development experience.
• Can use device accessibility for identifying views.
16
fredag den 23. marts 12
![Page 36: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/36.jpg)
Queries
17
fredag den 23. marts 12
![Page 37: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/37.jpg)
Queries • Queries are like CSS selectors or XPath
• label text:‘Hello‘
• label index:2
• view marked:‘thepane’ label
• view:‘MyClassName’
• label {text LIKE ‘Hel*’}
• webView css:‘#header a.cssclass‘
• webView xpath:’//node()’
17
fredag den 23. marts 12
![Page 38: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/38.jpg)
Demo: - Calabash iOS
- LessPainful Device Cloud
18
fredag den 23. marts 12
![Page 39: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/39.jpg)
iOS Comparisons• Several options available. To my knowledge:
• Calabash
• UIAutomation, Apple
• Zucchini, iOS Testing Framework
• Frank, Pete Hodgson, ThoughtWorks
• UISpec, http://code.google.com/p/uispec/
• FoneMonkey => MonkeyTalk, GorillaLogic
• KIF, Square
• NativeDriver, http://code.google.com/p/nativedriver/
19
fredag den 23. marts 12
![Page 40: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/40.jpg)
References• https://github.com/calabash
• https://github.com/calabash/calabash-ios
• https://github.com/calabash/calabash-ios/wiki
• https://github.com/calabash/calabash-ios-server
• https://github.com/calabash/calabash-android
• http://blog.lesspainful.com/
• https://www.lesspainful.com/
20
fredag den 23. marts 12
![Page 41: Calabash, an open-source automated testing technology for native mobile, by Karl Krukow](https://reader036.vdocuments.site/reader036/viewer/2022081400/554f597eb4c905b9508b52c6/html5/thumbnails/41.jpg)
Making app testing less painful...Please contact us with any questions:
[email protected]@lesspainful.com - iOS
[email protected] - Android
http://www.lesspainful.com
Questions?
fredag den 23. marts 12