don't fear our new robot overlords – a new way to test on mobile
DESCRIPTION
A talk I gave at europython 2014 about testing mobile web sites and apps on real devices using CV and a robotTRANSCRIPT
Don’t fear our new Robot Overlords!A new way to test on mobile devices
!
!
Philip Brechler • europython 2014
Agenda
Agenda
• Problems when testing on mobile devices
Agenda
• Problems when testing on mobile devices
• Idea
Agenda
• Problems when testing on mobile devices
• Idea
• Concept
Agenda
• Problems when testing on mobile devices
• Idea
• Concept
• Demo
Agenda
• Problems when testing on mobile devices
• Idea
• Concept
• Demo
• Conclusion and Learnings
Problems in general
CC-BY-SA: Javier Enjuto
Problems in general• You need to check how the app
looks
CC-BY-SA: Javier Enjuto
Problems in general• You need to check how the app
looks
• You can check coordinates, fonts etc.
CC-BY-SA: Javier Enjuto
Problems in general• You need to check how the app
looks
• You can check coordinates, fonts etc.
• In the end you never know if it looks right
CC-BY-SA: Javier Enjuto
Problems in general• You need to check how the app
looks
• You can check coordinates, fonts etc.
• In the end you never know if it looks right
• The human need to double check everything
CC-BY-SA: Javier Enjuto
Problems with simulators
CC-BY: Steve Jurvetson
Problems with simulators• To much computation power
CC-BY: Steve Jurvetson
Problems with simulators• To much computation power
• No memory issues
CC-BY: Steve Jurvetson
Problems with simulators• To much computation power
• No memory issues
• No touch interface, mouse pointers are to exact
CC-BY: Steve Jurvetson
Problems with simulators• To much computation power
• No memory issues
• No touch interface, mouse pointers are to exact
• Different browser engines
CC-BY: Steve Jurvetson
Problems with simulators• To much computation power
• No memory issues
• No touch interface, mouse pointers are to exact
• Different browser engines
• Some APIs not available
CC-BY: Steve Jurvetson
Problems with emulators
CC-BY-SA: Marshall Astor
Problems with emulators• Incredibly slow
CC-BY-SA: Marshall Astor
Problems with emulators• Incredibly slow
• Hard to interact with
CC-BY-SA: Marshall Astor
Problems with emulators• Incredibly slow
• Hard to interact with
• Android: Every manufacturer makes their own interface and browser
CC-BY-SA: Marshall Astor
Problems with emulators• Incredibly slow
• Hard to interact with
• Android: Every manufacturer makes their own interface and browser
• Android: No Google Play Services
CC-BY-SA: Marshall Astor
Best solution: Testing on real devices with real
inputs and a checking eye
CC-BY: yeowatzup
Idea
Idea
• Saw a talk about Sikuli for automated testing of websites
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
Sikuli
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
• A very python like script language
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
• A very python like script language
• Invented at MIT, now developed at Sikuli Labs (University of Colorado)
Sikuli• Sikuli is a tool for automating
computer tasks with screenshots
• A very python like script language
• Invented at MIT, now developed at Sikuli Labs (University of Colorado)
• Can be used for CI testing with Robot Framework or similar
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
• In short: Make a screenshot, use CV for detecting icon, tap icon
Idea
• Saw a talk about Sikuli for automated testing of websites
• Idea: Sikuli for mobile device testing but with a CI server
• In short: Make a screenshot, use CV for detecting icon, tap icon
• With a robot
Making a screenshot
Making a screenshot
• Android: Use adb command and some bash magic
Making a screenshot
• Android: Use adb command and some bash magic
• iOS: Making screenshots of devices is possible using Xcode but not with the terminal
Making a screenshot
• Android: Use adb command and some bash magic
• iOS: Making screenshots of devices is possible using Xcode but not with the terminal
• But there is an open source project that reverse engineered some of iTunes’s and Xcode’s magic called libimobiledevice
Detecting an icon
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
• OpenCV is written in C/C++ but has great python wrappers
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
• OpenCV is written in C/C++ but has great python wrappers
• Quite simple API for such a complex task
Detecting an icon
• Use OpenCV to do a template recognition (finding an image in an image)
• OpenCV is written in C/C++ but has great python wrappers
• Quite simple API for such a complex task
• Get the coordinate in the screenshot of your searched icon
The robot
The robot• Tapsterbot
The robot• Tapsterbot
• Open Source delta robot
The robot• Tapsterbot
• Open Source delta robot
• Made with Arduino, 3 Servos and 3D printed parts
The robot• Tapsterbot
• Open Source delta robot
• Made with Arduino, 3 Servos and 3D printed parts
• Costs: ~250€ (With printing)
The robot• Tapsterbot
• Open Source delta robot
• Made with Arduino, 3 Servos and 3D printed parts
• Costs: ~250€ (With printing)
• Driver written in node.js
Talking to a robot
Talking to a robot
• The robot is connected via WebSocket
Talking to a robot
• The robot is connected via WebSocket
• The robot is driven by a node.js „app“
Talking to a robot
• The robot is connected via WebSocket
• The robot is driven by a node.js „app“
• Theoretically the robot could be exchanged quite easily
Talking to a robot
• The robot is connected via WebSocket
• The robot is driven by a node.js „app“
• Theoretically the robot could be exchanged quite easily
• More robots?
Tightening it all together
Tightening it all together• Project Goldeneye
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
• All the heavy lifting is abstracted away from you
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
• All the heavy lifting is abstracted away from you
• Just tell the test to tap and to check
Tightening it all together• Project Goldeneye
• To test an app or a mobile website just write a standard python unittest test
• All the heavy lifting is abstracted away from you
• Just tell the test to tap and to check
• Writing such tests is quite easy, even your QA can do this
Demo
Conclusion
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
• Testing of existing projects, especially large web projects could be easier
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
• Testing of existing projects, especially large web projects could be easier
• Robots never get tired
Conclusion
• Goldeneye could be useful for testing mobile apps and especially websites in a CI system
• Testing of existing projects, especially large web projects could be easier
• Robots never get tired
• The robot’s hardware is fine, the software isn’t
Learnings
Learnings
• A function in a function in a function isn’t normal, but in node.js it is
Learnings
• A function in a function in a function isn’t normal, but in node.js it is
• Never trust a github project with one maintainer
Learnings
• A function in a function in a function isn’t normal, but in node.js it is
• Never trust a github project with one maintainer
• OpenCV is fun and not that hard to learn
