saas testing overview - foundation

SaaS Testing Overview

Software Performance Assurance Confidential

Ram Garg

CEO

Software Performance Assurance (SPA)


Outline

SaaS Introduction What is it?

Why SaaS?

SaaS Challenges Key Challenges

SaaS Testing How to approach SaaS testing?

How SaaS Testing is different?

SaaS Test Tools


SaaS Introduction

What is SaaS?


SaaS: What is it?

Software as a Service (SaaS)

Platform as a Service (PaaS)

Infrastructure as a Service (IaaS)

Software-as-a-Service (SaaS) is the application layer of the cloud computing model.

Application

Platform

Infrastructure


SaaS Definition

“Software as a service (SaaS), sometimes

referred to as "software on demand," is

software that is 100% deployed over the

internet … . With SaaS, a provider licenses

an application to customers as a service

on demand, through a subscription or a

"pay-as-you-go" model.”

– Wikipedia definition


Architecture: Three Key Differentiators

Key Architecture

Differentiators

Scalable

Multi-tenant Efficient

Configurable


SaaS Introduction

Why SaaS?

Upgrades

Platform

Installation

Infrastructure

management Responsibility

Software Ownership


The Software Business Model Changes …

non -SaaS

Customer

Customer

Client - Server

Larger, less frequent

Multi-version

SaaS

Provider

Provider

Central

Shorter, frequent

Single-version

Maintenance

Backward compatibility

Multi-tenancy

Configurability

Requirement


Changing the Business Model cont …

non -SaaS

Low

No

No

Customer

SaaS

High, Critical

Yes

Yes

Provider

Profits

Sales focus / Success

Cost, Allocation


Changing the Business Model cont …

non -SaaS

High, CapEx

Close the deal,

License Revenue

Initial sale + maintenance

SaaS

Low, OpEx

Provide value,

Lack of Churn

Ongoing Subscription

Ecosystem

Starting point to every service within

cloud computing ecosystem

Highest on the maturity level and

growth curve amongst XaaS

Highest revenue generating service

within cloud model for next several

years

SaaS (Software)

PaaS (Platform)

IaaS (Infrastructure)


SaaS Catalysts

Intrigue of access to unlimited Computing and storage resources, made available and paid for only as needed (Pay as you go)

Decreasing cost of bandwidth

Customer frustration with traditional software buying cycle (license, maintenance contract, upgrades)

Perceived increase in control over the vendor relationship

Early SaaS success stories – Salesforce.com, WebEx, etc. Demonstrate broader value proposition

You focus on your application and not on infrastructure


SaaS Introduction

SaaS Early Adoption


SaaS Adoption: Verticals

In-Stat research on

vertical market adoption

shows the healthcare

market is currently most

satisfied with hosted

applications, with

education identified as an

important potential target

market also.

AMI-Partners„ report -

"Software-as-Service Hot

Spots” : Education,

Healthcare, and Financial"

Adoption rate of

SaaS in SMB is

higher than large

enterprises

All research points out that SaaS has been adopted heavily in :

HR applications (ERP, Salesforce automation, eRecruiting)

Online Collaboration (Web conferencing)

CRM

Healthcare


SaaS Adoption Factors(source: Forrester)

Availability & Reliability (71%) Fast Deployment (45%)

Reduced dependencies on internal IT (24%)

Flexible consumption (35%)

Simplified pricing model (32%)


The SaaS Market is and will

Continue to Grow…


Today and tomorrow …


SaaS Challenges

Challenges


Primary Concerns with the SaaS Model

Security concerns

Concerns over compliance with standards

Concerns over integration with legacy applications

Concerns over reliability and robustness of service

Not comfortable with putting mission-critical application on shared infrastructure and have access over internet

SaaS can not be configured to meet my corporation‟s all specific needs


Numerous Challenges

Security Performance Availability SLA

Data Integrity PrivacyCompliance

with Standards

Regulatory

Integration Multi-tenancyLack of

confidence


SaaS Testing


SaaS Testing

Why SaaS Testing?


Why SaaS Testing?The average enterprise IT

department devotes up to 50

percent of its entire technology

infrastructure to development

and test, with up to 90 percent

of the available test

infrastructure remaining idle.

IBM Research

Those using cloud facilities to

run applications, 49.8% said

they were doing so

experimentally or for

prototyping; 28.6% for non-

critical business applications

and 21.6% for business

critical applications

Evans Data Survey

The average online

business had 8 days of

downtime a year due to

performance issues

leading to lost revenues

Aberdeen Group

A delay in website page

loading by even one second

can impact conversions by

7% and customer satisfaction

by 16%.

Aberdeen Group

Cloud

Testing

Trend leads to trend …

Overall growth in Cloud Computing (investment,

revenue, ROI, market cap, etc.) is leading to a

marked increase in adoption of SaaS Testing


SaaS Testing

What is SaaS Testing?


Classifying SaaS Testing

“Testing The Cloud”

Testing applications which are in

cloud or migrating to cloud.

Test the performance, reliability &

security of applications

“Testing Using Cloud”

Testing applications using the

cloud-based hardware, software,

infrastructure and computing

resources.

Tests performed like performance,

load & stress, security and

compatibility

“Testing The SaaS”

Testing applications which are in

cloud or migrating to cloud.

Test the performance, reliability &

security of applications

“Testing SaaS Using Cloud”

Testing applications using the

cloud-based hardware, software,

infrastructure and computing

resources.

Tests performed like performance,

load & stress, security and

compatibility

Testing the SaaS

Testing SaaS Using Cloud

Cloud / SaaS

Testing


How SaaS Testing is Different

Easier

• No client-server installation

• No multiplatform support

• No multiple version upgrades

• No backward compatibility

Challenges

• Frequent live upgrades

• Too many pages

• Multiple privileges

• Immature SaaS-specific testing tools

• Access partitions critical


SaaS Testing

Approach


SaaS Testing Preparation

Assess Testing Needs

Understand Usage

Patterns

Prepare Test Plan and Test

Suite

Prepare Test Data

Prepare Test

Environment


SaaS Testing Areas

Secured SaaS

Application

Network testing

Infrastructure Testing

Application testing

Ensure SaaS testing does not damage the shared infrastructure


SaaS Functional Testing

• Perform Manual and Exploratory test (similar to traditional applications)

• Check Browser compatibility

• Perform regression

• SaaS regression must cover significant more functionality as single defect will impact all the customers

• Functional and regression test automation.

• This is important to comprehensive testing cost-effectively as above point indicates that, in case of SaaS, regression should cover more functionality.

• Test SaaS on cloud-based staging environment

• Testing on in-house environment and then directly deploying on cloud-based production environment may result in chaos

• Test in a multi-user environment where different users are performing different activities.


SaaS Security Testing

• Test application security• Cross Site scripting, SQL Injection, HTTP header injection

• Test network security• Secured communication, Backup and storage policies, Controlled

access to sites and servers

• Test for possible attacks

• Test roles and privileges in multi-tenant environment• Many roles and privilege levels

• Test all the upgrades to detect broken security, Ensure security patches are not missed

• Test data security / integrity among multiple tenants


SaaS Security Testing cont …

• Identify scenarios where application might be vulnerable

• Test PCI compliance

• Test cookie setting, SSL configuration, information leak, etc.

• Log errors, warnings and suspicious requests

• Tests tampering with the licensing mechanism; gaining unauthorized access to application data, editing registry entities or configuration files


SaaS Performance Testing

• Test performance extensively first time and for each upgrade

• SaaS is revenue generating and business impacting applications. Single performance issue will impact all your customers

• Test performance extensively as SaaS need to handle large number of users and we don‟t have luxury of re-booting or going down every once in a while

• Test load characteristics, ramp up rate, protocols, load origination,

• Establish ability to test performance of your SaaS isolated fashion so that network related issues can be identified and passed to network infrastructure and internet providers

• Test key metrics and SLAs

• Test performance for mobile users


In-Service Upgrade Testing

• Ability to upgrade (full release, patches) system with zero-to-minimum down time is essential

• Global users make this complicated as somewhere in the world someone might be accessing your application

• Thoroughly test in staging environment before perform upgrade ass this is one of the most complex features

Remote Accessibility and Usage

• Global access with acceptable response time

• 24 hours availability


Disaster Recovery

• Shutdown to bring up application quickly: server configuration; install and deploy software ; restore data

• Roll back to last known stable data

Localization

• Make sure it supports international characters as SaaS application can be accessed from anywhere in the world

• Optimize by supporting target market language


SaaS Testing

Test Tools

Performance

Functional


Traditional and Cloud-based Test Tools

Traditional Test Tools

QTP, WinRunner

Rational Robot

Rational Functional Tester

SilkTest

Selenium, STAFF, Fitness, Watir

LoadRunner, Rational Robot,

Rational Performance Tester

Silk Performer

Jmeter, Grinder, WebLoad,

OpenSTA

Cloud-based Test Tools

Gomez

PushToTest,

SOASTA

CloudTesting

uTest

Many others


Traditional Vs. Cloud-based Tools

Testing SaaS Using Traditional Tools

Many providers who provide SaaS testing using traditional

testing tools

Many open source matured tools

Many options for professional services by independent

service providers

Professional service options allow cheaper cost

Testing SaaS Using Cloud-based Tools

A few providers who provide SaaS testing using their own

cloud-based test tools

Limited open source test tools options

Captive professional service by test tools provider which is not even core for their business

Captive profession service is sold at very high rate


Conclusion


Conclusion

Customers buy each month – keep them happy

SaaS is only a delivering model, customers will not compromise on functionality, stability, security, and performance. They will not switch to SaaS it if not better than traditional systems.

Save time on support, maintenance, and upgrade, etc.

You need to pay more attention to stuff like accessibility, security, deployments. The way you earn reputation

Don‟t buy cloud-based test solution just because you are testing SaaS. In most cases, traditional test solutions work just fine and save your 30% –50% cost.

saas testing overview - foundation

Technology

ecosystem saas software

saas market

saas definitionsoftware

application software

service saas platformplatform

traditional software

saas adoption factorssource

cloud model