Download - SaaS Testing Overview - Foundation
-
SaaS Testing Overview
Software Performance Assurance Confidential
Ram Garg
CEO
Software Performance Assurance (SPA)
-
Software Performance Assurance Confidential
Outline
SaaS Introduction What is it?
Why SaaS?
SaaS Challenges Key Challenges
SaaS Testing How to approach SaaS testing?
How SaaS Testing is different?
SaaS Test Tools
-
Software Performance Assurance Confidential
SaaS Introduction
What is SaaS?
-
Software Performance Assurance Confidential
SaaS: What is it?
Software as a Service (SaaS)
Platform as a Service (PaaS)
Infrastructure as a Service (IaaS)
Software-as-a-Service (SaaS) is the application layer of the cloud computing model.
Application
Platform
Infrastructure
-
Software Performance Assurance Confidential
SaaS Definition
Software as a service (SaaS), sometimes
referred to as "software on demand," is
software that is 100% deployed over the
internet . With SaaS, a provider licenses
an application to customers as a service
on demand, through a subscription or a
"pay-as-you-go" model.
Wikipedia definition
-
Software Performance Assurance Confidential
Architecture: Three Key Differentiators
Key Architecture
Differentiators
Scalable
Multi-tenant Efficient
Configurable
-
Software Performance Assurance Confidential
SaaS Introduction
Why SaaS?
-
Upgrades
Platform
Installation
Infrastructure
management Responsibility
Software Ownership
Software Performance Assurance Confidential
The Software Business Model Changes
non -SaaS
Customer
Customer
Client - Server
Larger, less frequent
Multi-version
SaaS
Provider
Provider
Central
Shorter, frequent
Single-version
-
Maintenance
Backward compatibility
Multi-tenancy
Configurability
Requirement
Software Performance Assurance Confidential
Changing the Business Model cont
non -SaaS
Low
No
No
Customer
SaaS
High, Critical
Yes
Yes
Provider
-
Profits
Sales focus / Success
Cost, Allocation
Software Performance Assurance Confidential
Changing the Business Model cont
non -SaaS
High, CapEx
Close the deal,
License Revenue
Initial sale + maintenance
SaaS
Low, OpEx
Provide value,
Lack of Churn
Ongoing Subscription
-
Ecosystem
Starting point to every service within
cloud computing ecosystem
Highest on the maturity level and
growth curve amongst XaaS
Highest revenue generating service
within cloud model for next several
years
SaaS (Software)
PaaS (Platform)
IaaS (Infrastructure)
-
Software Performance Assurance Confidential
SaaS Catalysts
Intrigue of access to unlimited Computing and storage resources, made available and paid for only as needed (Pay as you go)
Decreasing cost of bandwidth
Customer frustration with traditional software buying cycle (license, maintenance contract, upgrades)
Perceived increase in control over the vendor relationship
Early SaaS success stories Salesforce.com, WebEx, etc. Demonstrate broader value proposition
You focus on your application and not on infrastructure
-
Software Performance Assurance Confidential
SaaS Introduction
SaaS Early Adoption
-
Software Performance Assurance Confidential
SaaS Adoption: Verticals
In-Stat research on
vertical market adoption
shows the healthcare
market is currently most
satisfied with hosted
applications, with
education identified as an
important potential target
market also.
AMI-Partners report -"Software-as-Service Hot
Spots : Education, Healthcare, and Financial"
Adoption rate of
SaaS in SMB is
higher than large
enterprises
All research points out that SaaS has been adopted heavily in :
HR applications (ERP, Salesforce automation, eRecruiting)
Online Collaboration (Web conferencing)
CRM
Healthcare
-
Software Performance Assurance Confidential
SaaS Adoption Factors(source: Forrester)
Availability & Reliability (71%) Fast Deployment (45%)
Reduced dependencies on internal IT (24%)
Flexible consumption (35%)
Simplified pricing model (32%)
-
Software Performance Assurance Confidential
The SaaS Market is and will
Continue to Grow
-
Software Performance Assurance Confidential
Today and tomorrow
-
Software Performance Assurance Confidential
Today and tomorrow
-
Software Performance Assurance Confidential
Today and tomorrow
-
Software Performance Assurance Confidential
SaaS Challenges
Challenges
-
Software Performance Assurance Confidential
Primary Concerns with the SaaS Model
Security concerns
Concerns over compliance with standards
Concerns over integration with legacy applications
Concerns over reliability and robustness of service
Not comfortable with putting mission-critical application on shared infrastructure and have access over internet
SaaS can not be configured to meet my corporations all specific needs
-
Software Performance Assurance Confidential
Numerous Challenges
Security Performance Availability SLA
Data Integrity PrivacyCompliance
with Standards
Regulatory
Integration Multi-tenancyLack of
confidence
-
Software Performance Assurance Confidential
SaaS Testing
-
Software Performance Assurance Confidential
SaaS Testing
Why SaaS Testing?
-
Software Performance Assurance Confidential
Why SaaS Testing?The average enterprise IT
department devotes up to 50
percent of its entire technology
infrastructure to development
and test, with up to 90 percent
of the available test
infrastructure remaining idle.
IBM Research
Those using cloud facilities to
run applications, 49.8% said
they were doing so
experimentally or for
prototyping; 28.6% for non-
critical business applications
and 21.6% for business
critical applications
Evans Data Survey
The average online
business had 8 days of
downtime a year due to
performance issues
leading to lost revenues
Aberdeen Group
A delay in website page
loading by even one second
can impact conversions by
7% and customer satisfaction
by 16%.
Aberdeen Group
Cloud
Testing
-
Trend leads to trend
Overall growth in Cloud Computing (investment,
revenue, ROI, market cap, etc.) is leading to a
marked increase in adoption of SaaS Testing
-
Software Performance Assurance Confidential
SaaS Testing
What is SaaS Testing?
-
Software Performance Assurance Confidential
Classifying SaaS Testing
Testing The CloudTesting applications which are in
cloud or migrating to cloud.
Test the performance, reliability &
security of applications
Testing Using CloudTesting applications using the
cloud-based hardware, software,
infrastructure and computing
resources.
Tests performed like performance,
load & stress, security and
compatibility
Testing The SaaSTesting applications which are in
cloud or migrating to cloud.
Test the performance, reliability &
security of applications
Testing SaaS Using CloudTesting applications using the
cloud-based hardware, software,
infrastructure and computing
resources.
Tests performed like performance,
load & stress, security and
compatibility
Testing the SaaS
Testing SaaS Using Cloud
Cloud / SaaS
Testing
-
Software Performance Assurance Confidential
How SaaS Testing is Different
Easier
No client-server installation
No multiplatform support
No multiple version upgrades
No backward compatibility
Challenges
Frequent live upgrades
Too many pages
Multiple privileges
Immature SaaS-specific testing tools
Access partitions critical
-
Software Performance Assurance Confidential
SaaS Testing
Approach
-
Software Performance Assurance Confidential
SaaS Testing Preparation
Assess Testing Needs
Understand Usage
Patterns
Prepare Test Plan and Test
Suite
Prepare Test Data
Prepare Test
Environment
-
Software Performance Assurance Confidential
SaaS Testing Areas
Secured SaaS
Application
Network testing
Infrastructure Testing
Application testing
Ensure SaaS testing does not damage the shared infrastructure
-
Software Performance Assurance Confidential
SaaS Functional Testing
Perform Manual and Exploratory test (similar to traditional applications)
Check Browser compatibility
Perform regression SaaS regression must cover significant more functionality as single
defect will impact all the customers
Functional and regression test automation. This is important to comprehensive testing cost-effectively as above
point indicates that, in case of SaaS, regression should cover more functionality.
Test SaaS on cloud-based staging environment Testing on in-house environment and then directly deploying on cloud-
based production environment may result in chaos
Test in a multi-user environment where different users are performing different activities.
-
Software Performance Assurance Confidential
SaaS Security Testing
Test application security Cross Site scripting, SQL Injection, HTTP header injection
Test network security Secured communication, Backup and storage policies, Controlled
access to sites and servers
Test for possible attacks
Test roles and privileges in multi-tenant environment Many roles and privilege levels
Test all the upgrades to detect broken security, Ensure security patches are not missed
Test data security / integrity among multiple tenants
-
Software Performance Assurance Confidential
SaaS Security Testing cont
Identify scenarios where application might be vulnerable
Test PCI compliance
Test cookie setting, SSL configuration, information leak, etc.
Log errors, warnings and suspicious requests
Tests tampering with the licensing mechanism; gaining unauthorized access to application data, editing registry entities or configuration files
-
Software Performance Assurance Confidential
SaaS Performance Testing
Test performance extensively first time and for each upgrade SaaS is revenue generating and business impacting applications.
Single performance issue will impact all your customers
Test performance extensively as SaaS need to handle large number of users and we dont have luxury of re-booting or going down every once in a while
Test load characteristics, ramp up rate, protocols, load origination,
Establish ability to test performance of your SaaS isolated fashion so that network related issues can be identified and passed to network infrastructure and internet providers
Test key metrics and SLAs
Test performance for mobile users
-
Software Performance Assurance Confidential
In-Service Upgrade Testing
Ability to upgrade (full release, patches) system with zero-to-minimum down time is essential
Global users make this complicated as somewhere in the world someone might be accessing your application
Thoroughly test in staging environment before perform upgrade ass this is one of the most complex features
Remote Accessibility and Usage
Global access with acceptable response time
24 hours availability
-
Software Performance Assurance Confidential
Disaster Recovery
Shutdown to bring up application quickly: server configuration; install and deploy software ; restore data
Roll back to last known stable data
Localization
Make sure it supports international characters as SaaS application can be accessed from anywhere in the world
Optimize by supporting target market language
-
Software Performance Assurance Confidential
SaaS Testing
Test Tools
-
Performance
Functional
Software Performance Assurance Confidential
Traditional and Cloud-based Test Tools
Traditional Test Tools
QTP, WinRunner
Rational Robot
Rational Functional Tester
SilkTest
Selenium, STAFF, Fitness, Watir
LoadRunner, Rational Robot,
Rational Performance Tester
Silk Performer
Jmeter, Grinder, WebLoad,
OpenSTA
Cloud-based Test Tools
Gomez
PushToTest,
SOASTA
CloudTesting
uTest
Many others
-
Software Performance Assurance Confidential
Traditional Vs. Cloud-based Tools
Testing SaaS Using Traditional Tools
Many providers who provide SaaS testing using traditional
testing tools
Many open source matured tools
Many options for professional services by independent
service providers
Professional service options allow cheaper cost
Testing SaaS Using Cloud-based Tools
A few providers who provide SaaS testing using their own
cloud-based test tools
Limited open source test tools options
Captive professional service by test tools provider which is not even core for their business
Captive profession service is sold at very high rate
-
Software Performance Assurance Confidential
Conclusion
-
Software Performance Assurance Confidential
Conclusion
Customers buy each month keep them happy
SaaS is only a delivering model, customers will not compromise on functionality, stability, security, and performance. They will not switch to SaaS it if not better than traditional systems.
Save time on support, maintenance, and upgrade, etc.
You need to pay more attention to stuff like accessibility, security, deployments. The way you earn reputation
Dont buy cloud-based test solution just because you are testing SaaS. In most cases, traditional test solutions work just fine and save your 30% 50% cost.