testdroid: which one works you the best: in-house or cloud-based development environment
TRANSCRIPT
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 1
8 January 2015
Ville-Veikko Helppi Technical Product Manager [email protected]
Which One Works You The Best: In-House or Cloud-Based Development Environment
W E B I N A R
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 2
Agenda
• Comparison of In-house and Cloud-based Setups • Investments in Mobile App Development and
Testing Environments • The Characteristics to Use for Comparison of Both
Development Environments • Calculation Examples • Testdroid Update • Q&A
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 3
Introduction to Creation of In-House Development Labs
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 4
Picture from #GTAC2014
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 5
Why Automation?
Smaller coverage, More money burnt & ?me wasted, Error-‐prone
Large coverage, quickly
completed, Less money & ?me wasted, Exact
results.
Manual Automa@on
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 6
• Why Do You Need an In-House Development and Test Lab? – Pre-market devices (OEMs, ODMs,
integrators ...) – Prototypes / parts of hardware – Corporate security policies – Availability and high usage level – Special setups (integration w/other
equipments) – Large set of devices available
In-house Dev&Test Labs
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 7
• What is Required for an In-House Development & Test Lab? – Server hardware / infrastructure – Own mobile devices – Device control servers
• Hard Disks, RAM, Energy management
– USB hubs – WiFi / Network infrastructure – Continuous Integration/Delivery/Testing – Build systems / other software tools – POWER!
In-house Dev&Test Labs
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 8
Continuous Integration
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 9
• Version control and build revision • Build automation, instant execution&testing • All types of testing & integrations enabled • Frequent commits, Code consolidation ->
Fast builds • CI integrated with development and testing
provides the best fit for mobile agile process
Core Benefits of CI
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 10
• How to Reach Enterprise-Grade Reliability for In-House Dev&Test Lab – Device management
• Always run on clean devices • Create intelligent retry mechanisms • Create fall-back mechanisms
– System management • Automatic reconnect of network and USB • Automatic configuration changes • Professional-level monitoring system
– Diskspace, network/connections, Power
In-house Dev&Test Labs
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 11
Issues?
29% of problems are tightly related to devops of an
in-house system
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 12
Introduction to Adoption of Cloud-based Development Labs
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 13
• Someone else is taking care of heavy-lifting • Cloud-based device cloud includes ALL
relevant devices (phones, tablets, cams...) • You get everything automatically delivered – Data, Results, Logs, Screenshots, Pass/Fail
Reports, Additional data-sets
• Diversity and Keeping Up-to-Date with SW – New OS versions and OEM updates
• Easiness of Adoption and Integration
Why Cloud?
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 14
Access via Comprehensive API
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 15
• Authentication • Project Management – Create, Delete, Configure, Managing Users
• Configuring Access and Assets – Configs, Upload/Download Files (App & Tests) – Device Access & Device Group Configuration
• Running Tests – Upload App + Tests <-> Download Results/Data – Notifications, Sharing, Pre-processing of Data
Access via Comprehensive API
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 16
Investments in Mobile App Development Environment
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 17
Investments Aspects
• What is the Most Cost- Efficient Approach?
• What is the Total Cost of Ownership?
• Capital Expenses (CapEx)
• Operation Expenses (OpEx)
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 18
• CapEx is a business expense incurred to create future benefits or to add to the value of an existing asset
• OpEx refers to the expenses that the mobile app developer spends in order to get their day-to-day activities going on and to get their infrastructure fully working
CapEx vs. OpEx
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 19
• CapEx: Devices, Servers, Infrastructure • OpEx: Running & Maintaining the System
Costs: In-House Dev&Test Lab
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 20
• Cost of Service includes all major expenses (both CapEx and OpEx)
Costs: Cloud-based Dev&Test Lab
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 21
10 Different Characteristics for Comparing Environments
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 22
• A
Different Characteristics & Benefits
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 23
• Corporate Policies • Regulation • Minimizing security
breaches • Security/Protection
mechanisms • Monitoring • Authentication &
User Access
Security 1
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 24
• Needs to be fully configurable
• Back-ups & Retention • Preventing non-
authorized use and access • Needs to available 24/7 • Transferrable nature of
data, source code, results
Data Ownership and Retention 2
In-house vs.
Cloud
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 25
• API access – no delays • Critical/Medium/Low • DevOps in crucial role • Various data center
locations • Hardware failures?
Monitoring? Scale up? • Average Cloud Up-time:
99.5%
Performance and Availability 3
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 26
• Support: Email, Phone, Online service systems
• Intuitive online help: Manuals, How-Tos, Instructions (self-service)
• Integration with bug-tracking systems (e.g. JIRA)
• Frequent product updates • Needs to be available 24/7
Support 4
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 27
• Clear policies on updates, maintenance, monitoring
• Terms of Service + EULAs • Manual vs. Automated
enhancements/updates • Customized features • Product roadmaps
Enhancements and Update Policy 5
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 28
• Agile process enables agile integrations
• Tools, Frameworks, Other complementary software
• API • Avoid lock-ins to proprietary
frameworks and other SW! • Can define cost-efficiency of
using either approach
Integrations 6
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 29
• Intuitiveness, Easy-to-Use, Good Help Materials
• Directly related to TCO • Live demos and free
trials with the system • Frequent webinars,
Q&A, "How-To" materials
Usability 7
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 30
• Lock-ins with in-house environments
• Term or Perpetual licensing model
• Return-on-Investment?
• Subscription models – optimal use of service
Contractual Flexibility 8
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 31
• Own devices vs. devices from cloud farm?
• Scalability with server infrastructure
• Device control servers – Linux with Android = 15 – iOS with Mac = 3
• Easiest scalability comes w/privately hosted cloud
Equipment Scalability 9
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 32
• Case Studies • Customer References • Reaching out to user
to ask any details
Recommendations & References 10
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 33
Calculation Examples
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 34
Diminishing Value Curve
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 35
• Different levels of control and abstraction
• Definitions aren't always clear – so make sure you understand what you are buying!
SaaS-PaaS-IaaS vs. In-house
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 36
• Assumption – You are building several apps
• No. of apps is not limited here, as all example infrastructures can handle large number of apps
– You use 50 different devices for testing – You need everything up and running and
someone maintaining the system 24/5 – OR – just business hours
Basic Example
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 37
• In-house Setup – Devices: ($200 x 50 = $10,000) – Server infrastructure: ($1,000 per 5 devices -> $10,000) – Software: ($99/month x 50 = $4,950/mo -> $59,400/year) – Setup/deployment: 3 biz days ($50/hour x 8 x 3 = $1,200) – DevOps to manage the system 24/5: ($50/hour x 24 x 5 =
$6,000/week x 50 weeks = $300,000) OR ALTERNATIVELY – DevOps to manage the system on business hours: ($50 x 8 x
5 = $2,000/week x 50 weeks = $100,000)
• Majority of costs comes from OpEX
In-House Dev & Test Lab
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 38
• Public Cloud Setup – Deployment: 1 business day ($50 x 8 = $400) – Integration: 2 business days ($50 x 8 x 2 = $800) – License for 50 devices: ($1499/mo -> $17,988/year)
• Private Cloud Setup – Deployment: 1 business day ($50 x 8 = $400) – Integration: 2 business days ($50 x 8 x 2 = $800) – License for 50 devices: ($199 x 50 -> $9,950/month ->
$119,400/year)
• Majority of costs comes from CapEx
Cloud-Based Services
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 39
• Head-to-Head: – In-house (24/5)
costs $380KUSD vs. Private Cloud $120KUSD
• Privately hosted cloud is 31.5% of in-house system costs
Cost Comparison
Comparable environments:
216%
Significant difference in costs between in-house and cloud
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 40
Testdroid Update
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 41
Testdroid PrivateCloud - Running user-selected devices at any of our hosting facilities, more cost-efficiently than your own devices!
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 42
Option 1:
Public Device Cloud on-demand devices
(multitenant)
Mobile app testing on over 400+ real
Android and iOS devices hosted by
Bitbar
Option 2:
Private Device Cloud reserved devices
Hosted by Bitbar in the
US and/or Europe
Devices chosen by and reserved only for the
Customer
Option 3:
On-premise Device Cloud
Automated mobile app testing on real
Android and iOS de- vices hosted by the customer, usually 30-500 devices
Testdroid Deployment Options
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 43
• Get to know the essential facts in terms of selecting cloud-based or in-house solution for your needs: – Characteristics – Costs vs. Benefits – Calculation examples
• Available at www.testdroid.com/library
New EBOOK Available!
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 44
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 45
Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Visit at www.testdroid.com for more information.
THANK YOU!
W E B I N A R
© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 46
We operate the largest global device cloud with over 400 devices
instantly available for developers
Devices from the US, Europe, China, Japan and Korea – global devices for global customers