reliable mobile test automation
TRANSCRIPT
![Page 1: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/1.jpg)
VishalSRE-SET Automation Night #1
Reliable Mobile Test Automation
![Page 2: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/2.jpg)
Software Engineer in Test
Case Study (Mobile Automation)
SWET (past)
Core Member
Vishal Banthia @vbanthia @vbanthia_
![Page 3: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/3.jpg)
Some Previous Talks, Blogs & Tutorials
• Android E2E Testing at Mercari @ Android Test Night
• Android Device Farm at Mercari
• Appium Docker Demo
• STF Appium Example
![Page 4: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/4.jpg)
• Test Automation Life Cycle
• How to achieve Reliable Automation
• Demo
Agenda
![Page 5: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/5.jpg)
Test Automation Life Cycle
![Page 6: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/6.jpg)
• Have you ever wrote any kind of UI Tests? Browser,
Mobile, Desktop, etc
• Do you think reliable mobile test automation is possible?
Questionnaire!
![Page 7: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/7.jpg)
Test Automation Life Cycle ...
Product Manager
QA Team
Engineering Team
![Page 8: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/8.jpg)
➢ Product manager comes up with new features
➢ Developers implement them
➢ QA team test new features and do regression testing
➢ Release
Development flow
![Page 9: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/9.jpg)
Whats up?
Development flow ...
Product ManagerQA Team Engineering Team
Where is the
feature?
Code is done.
Waiting for QAToo many features
Too many test cases
Too many devices
No Time
I am dying ...
![Page 10: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/10.jpg)
Retrospective Meeting ...
Product Manager
QA Team Engineering Team
![Page 11: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/11.jpg)
We Need Automation!
Retrospective Result ...
![Page 12: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/12.jpg)
Creation of Test Automation Team
Engineering Team
Test
AutomationTeamEngineering Team
![Page 13: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/13.jpg)
Test Automation Team
• SET @google, @mercari
• SWET @DeNA
• SDET @microsoft
![Page 14: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/14.jpg)
Journey of Automation Team
![Page 15: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/15.jpg)
Start looking for automation tools
detox
![Page 16: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/16.jpg)
Initial Learning
• Initial learning such as setup, writing simple scenario is not too difficult.
• With some programming experience one can start writing mobile e2e tests.
![Page 17: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/17.jpg)
General Approach
Try to automate tests as much as possible from manual QA test cases list.
![Page 18: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/18.jpg)
Writing tests is the easiest part in whole journey!
![Page 19: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/19.jpg)
Difficult part is to actually use them in RELEASE flow!
![Page 20: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/20.jpg)
What are the difficulties?
• Flaky tests
• Test execution environment
![Page 21: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/21.jpg)
Because of these problems, people start losing
trust in it!
![Page 22: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/22.jpg)
And this happens…
Sad Reality :(
![Page 23: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/23.jpg)
So, what’s next!
![Page 24: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/24.jpg)
Let’s go back to difficulties
• Flaky tests
• Test execution environment
![Page 25: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/25.jpg)
Flaky Tests
![Page 26: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/26.jpg)
● User Interface issues:○ UI is not working as per feature specification
○ Test code is not handling UI automation logic properly for scenarios
involving scroll, swipe etc and test is flaky
● Backend issues:○ Backend is temporary down
○ API is not working as per specification
When can a test fail?
![Page 27: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/27.jpg)
When can a test fail? ...
● 3rd party service issues:○ Service is temporary down and error handling is not being done properly
in test automation logic.
● Hardware issues:○ Test is failing on specific device because of hardware limitation such as
Memory, Processor.
○ Device wifi is unstable.
![Page 28: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/28.jpg)
Problem is not the flaky test!
Problem is we don’t know what is happening
during test execution
![Page 29: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/29.jpg)
So, what’s solution?
![Page 30: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/30.jpg)
Record Everything!
![Page 31: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/31.jpg)
Record ...
• Screenshots• Screenrecord (Video)• ADB logs• Timestamp of each UI action and test assertion• Test script logs• Appium logs
![Page 32: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/32.jpg)
And put them in a nice report!
![Page 33: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/33.jpg)
rspec_html_reporter
• https://github.com/vbanthia/rspec_html_reporter
• Sample Report
• Android Example
http://tech.mercari.com/entry/2017/12/12/103741
![Page 34: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/34.jpg)
rspec_html_reporter ...
• Extension of RSpec HTML Formatter with feature such as:• Pretty report• Can embed metadata such as screenshot and screenrecord
• ADB Logcat & other metadata are WIP.
![Page 35: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/35.jpg)
This report will not solve flaky test itself, but it will help in
distinguishing between a bug and flaky test!
Flaky test solution
![Page 36: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/36.jpg)
Let’s go back to difficulties
• Flaky tests
• Test execution environment
![Page 37: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/37.jpg)
Test Execution Environment
![Page 38: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/38.jpg)
Test Execution Environment
In general, try to use Cloud Services as much as possible. This provides quick feedback and they are easily scalable.
And anyone should be able to execute test cases anytime easily
Android E2E Testing at Mercari
Android Device Farm at Mercari
![Page 39: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/39.jpg)
Mercari Android CI (Tools & Service)
• Bitrise: To build application• Google Cloud Storage: To store built apks• Google Container Registry: To store docker images• OpenSTF: For Android Device Farm• CircleCI 2.0: Build docker image & run tests in parallel containers• CircleCI Artifacts: To store test reports• AWS Lambda: To trigger nightly builds• Slack: Notification & manual triggers
![Page 40: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/40.jpg)
Demo Video
![Page 41: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/41.jpg)
• Mobile test automation is challenging but not impossible.
• With correct approach and planning it is possible to create
reliable test automation and actually improve product
quality and release efficiency
Conclusion
![Page 42: Reliable mobile test automation](https://reader033.vdocuments.site/reader033/viewer/2022050614/5a6625507f8b9a99198b4a3d/html5/thumbnails/42.jpg)
Thank you for coming!