testing cloud services: saas, paas, and iaas

57
TF Half-day Tutorials 5/6/2014 8:30:00 AM Testing Cloud Services: SaaS, PaaS, and IaaS Presented by: Martin Pol Jeroen Mengerink Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Upload: techwellpresentations

Post on 13-May-2015

762 views

Category:

Technology


3 download

DESCRIPTION

Cloud computing has changed the environment of testing. Its use is increasing for hosting business applications (SaaS) and testing (TaaS). Martin Pol and Jeroen Mengerink focus on SaaS, describing the relevant infrastructure and platform services (IaaS and PaaS). How do we test performance of the cloud itself? How do we make sure that the continuity of services is guaranteed? How do we cope with elasticity and the philosophy of bring-your-own-device (BYOD)? Martin and Jeroen discuss the risks that arise when implementing cloud computing―some traditional, but others completely new. Learn how to mitigate these risks with current, modified, and new test techniques. As testers, we must be involved earlier in the cloud selection process. Testers should help to create and evaluate selection criteria to minimize risk. In addition, testers should be involved in the project longer as testing in production is needed to determine if the Service Level Agreements are being met.

TRANSCRIPT

Page 1: Testing Cloud Services: SaaS, PaaS, and IaaS

TF Half-day Tutorials

5/6/2014 8:30:00 AM

Testing Cloud Services:

SaaS, PaaS, and IaaS

Presented by:

Martin Pol

Jeroen Mengerink

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Testing Cloud Services: SaaS, PaaS, and IaaS

Martin Pol Polteq

Martin Pol has played a significant role in helping to raise the awareness and improve the performance of testing worldwide. Martin provides international testing consulting services through POLTEQ Test Services BV. He’s gained experience by managing testing processes and implementing and improving structured testing in many organizations around the world. A co-author of Test Process Improvement, a classic text on models for improving testing, Martin has developed approaches to successfully manage test outsourcing services. In 2010, Martin received the Knight in the Order of Orange-Nassau award from The Netherlands for his lifetime contributions to the IT and software industries.

Jeroen Mengerink Polteq

As a test consultant for the Netherlands-based Polteq Test Services B.V. Jeroen Mengerink has performed multiple TPI assessments worldwide. His technical skills allow him to team with developers in testing websites, APIs, and web services. Jeroen performs both functional testing and performance testing. In addition to his work for clients, he is involved within various test innovations in the area of agile. Jeroen teaches the Certified Agile Tester course and several test courses on agile, SOA, and cloud; coauthored Testing Cloud Services; and blogs at jmengerink.wordpress.com. Follow him on Twitter @AngusVB.

Page 3: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 1

Testing Cloud Services: SaaS, PaaS and IaaS

Martin Pol

Jeroen Mengerink

Agenda

• Introduction Cloud computing

• Challenges Risks

• Solutions Test measures

Page 4: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 2

ISBN 978-1-937538-38-5

In the cloud?

Page 5: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 3

searching, recording, accounting, paying, writing,

reviewing, tracking, calculating, developing, listening,

analyzing, transmitting, learning, controlling,

purchasing, testing, alarming, changing, updating,

deleting, accessing, rejecting, correcting, studying,

booking, receiving, tracing, protecting, deciding,

managing, teaching, facilitating, identifying, copying,

removing, demonstrating, checking, showing,

selecting, subscribing, unsubscribing, sharing,

mailing, communicating, reading, playing, working,

meeting, gambling, shopping, storing, cross

checking, retrieving, configuring, sketching, saving,

accelerating, enhancing, creating, growing, checking

in, checking out, finding out, reaching, denying,

talking, designing, making, verifying, measuring

Email

Surf

Transfer

Develop and Test

Operate and Manage Store

Page 6: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 4

storage claim

80% unused

redundancy limitations

environmentally unfriendly

management overheadcosts for innovation

standard software bandwidth

internet technologySOA

virtualization

Page 7: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 5

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

�On-demand service

� Self service provisioning, pay-per-use

� No human interaction

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

�On-demand service

�Broad network access

� Standard mechanisms over networks

� “Any” client

Page 8: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 6

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

�On-demand service

�Broad network access

�Resource pooling

� Multi-tenant

� Storage, processing, memory, virtual machines, …

� Location independent

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

�On-demand service

�Broad network access

�Resource pooling

�Rapid elasticity

� Rapid scale in and out

� “Any quantity” at any time

Page 9: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 7

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

�On-demand service

�Broad network access

�Resource pooling

�Rapid elasticity

�Measured service

� Controlled resource use

� Transparency, pay-per-use

US: National Institute of Standards and Technologyhttp://www.nist.gov

Essential characteristics

�On-demand service

�Broad network access

�Resource pooling

�Rapid elasticity

�Measured service

Deployment models

– private cloud

– community cloud

– public cloud

– hybrid cloud

Service Models

Software as a Service

Platform as a Service

Infrastructure as a Service

Page 10: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 8

Service models

• Nocloud

• Infrastructure as a Service

• Platform as a Service

• Software as a Service

Application

Platform

Virtualization

Hardware

CloudInternal

Implementation models

• Public

• Private

• Community

• Hybrid

Page 11: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 9

What is “done” in the cloud?

>500

PrivateHybrideCommunity

IaaS, PaaS, DaaS, SaaS

Taas

*aaS

Data CentreData Management

Business processes

Consumer

Public

SaaS

Surf and mailAppsSocial mediaDropboxGoogle servicesSpotifyPicasaGames……………

<500 employees

Public

*aaS

MailStorage

Infrastructure

CRM

Finance

Business processes

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards

143143

Page 12: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 10

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

StandardsPerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 13: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 11

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Other customers

YOUR

Operational Profile

YOUR

Operational Profile

YOUR

Operational Profile

PLUS

YOUR

Operational Profile

PLUS

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 14: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 12

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Everything over the web

The idea:

“it’s safe”

The idea:

“it’s safe”

Home ground for

hackers

Home ground for

hackers

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

ManageabilityManageability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 15: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 13

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Bring Your Own Device

No free choice of

device.

No free choice of

device.

Endless

possibilities.

Endless

possibilities.

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 16: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 14

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Internet connection lost

@ supplier

@ user

@ other systems

‘Off line” does not work

Information is lost

Page 17: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 15

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 18: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 16

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Mismatchservice <> business process

Functionality is changed

Insufficient usability

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 19: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 17

Backup and recovery

Taken care of.Taken care of.

Who will support

me?

Who will support

me?

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 20: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 18

Updates, patches, fixes, H

Planned and

controlled

Planned and

controlled

Do I have a

choice?

Do I have a

choice?

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

ManageabilityManageability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & regulationsLegislation & regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 21: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 19

Where is my data?

And is that OK?

In house.In house.

SomewhereHSomewhereH

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & regulationsLegislation & regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & regulationsLegislation & regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 22: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 20

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & regulationsLegislation & regulations

Suppliers & outsourcingSuppliers & outsourcingRisks

Risks

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & regulationsLegislation & regulations

Suppliers & outsourcingSuppliers & outsourcingRisks

Risks

Page 23: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 21

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & regulationsLegislation & regulations

Suppliers & outsourcingSuppliers & outsourcingRisks

Risks

Vendor lock in

No agreements

Supplier of the supplier of the supplier H

Supplier is taken over

Testing?

Check

Review

Monitor

Interview

Proof of concept

Page 24: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 22

Testing!

Check

Review

Monitor

Interview

Proof of conceptTestenProefIntake

InterviewProof of concept

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

TestenProefIntake

InterviewProof of concept

Page 25: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 23

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 26: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 24

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Architecture

From “individual” risks

to

“individual” test measures

Architecture

From “individual” risks

to

“individual” test measures

Page 27: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 25

Selection

Implementation

Production

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 28: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 26

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Selection Criteria

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Completeness

Controllable

For service

For supplier

Spec’s and terms

References

HH

Page 29: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 27

“Inspiration List”

CRITERION PRIOFunctionalDo the service and the specific business processes align?Does the service fit well in the E2E business process?Is the service sufficiently adaptable to specific requirements?Are many adjustments needed?Is customization possibleIs (a lot of) customization needed?Are the required platforms supported?Are “het nieuwe werken” and BYOD supported sufficiently?Is it possible to connect / integrate the service with the other systems?Are sufficient manuals and/or courses available?ImplementationIs the impact on current activities acceptable?Is a feasible route for migration towards the service available?

Page 30: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 28

“Inspiration List”

CRITERION PRIOSupportAre changes in the service announced beforehand?Are sufficient test facilities available around the service (test environment, test tooling, testware, access to infrastructure, …)?Are there sufficient support facilities?Is it clear how incidents can be reported?Are incidents resolved fast enough?PerformanceAre response times low enough?Is the number of possible simultaneous users high enough?Is bandwidth sufficient?Is sufficient potential for growth available?Is the actual use charged correctly?

“Inspiration List”

CRITERION PRIOSecurityAre adequate authorization and authentication possibilities in place?Is the physical security of the service locations sufficient?Is the support access security of the service sufficient?Is mutual access security between customers sufficient?Are data changes traceable?Is data storage for the service reliable?Is deleting data in the service reliable?Is security of the connection to the service sufficient?Are security options for the customer sufficient?Does the supplier have security certificates? (for example SAS 70 type II)?AvailabilityIs the level of availability for the service sufficient?Are back-up / fail-over / disaster-recovery provisions sufficient?

Page 31: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 29

“Inspiration List”

CRITERION PRIOLaw and regulationsDoes the data location comply to all legal requirements?Does the data processing comply to all legal requirements?Do the terms contain parts that are conflicting to the duties of the customer?SupplierIs clear what happens when the contract ends, or in case of bankruptcy or conflict?Is a good helpdesk available?Does the supplier have experience in:- Offering this particular service?- Offering services in general?- Developing services?- The customer’s field?- Developing, testing and supporting services (know how)?Do methods used by supplier align with those of the customer (if relevant)?

“Inspiration List”

CRITERION PRIOSupplierIs quality assurance arranged?Is the supplier ahead in its field?Is the size of the supplier in accordance with the expectations of the customer?Does the supplier have a good reputation (are there references)?Is providing services the core business of the supplier?Does the supplier have opportunities for future expansion?Does the supplier speak the same language?Is transparency and flexibility of the supplier sufficient?

Page 32: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 30

Proof of Concept

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Dynamic testing

More suppliers

Time boxing

Representative

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 33: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 31

Known measures

tuned and tweaked

New measures developed

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Load Testing

YOUR

Operational Profile

YOUR

Operational Profile

YOUR

Operational Profile

PLUS

ACTUAL MOMENT

YOUR

Operational Profile

PLUS

ACTUAL MOMENT

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 34: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 32

Operational profile

Performance testing

• Test cases aimed at specific bottlenecks

• Including cloud aspectsin test cases

• Test setup for a

performance test

• Representative?

Page 35: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 33

Stress Testing

Yes, you can!Yes, you can!

Definitely NOT!Definitely NOT!

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Elasticity

Load and stress.Load and stress.

Load and elasticity.Load and elasticity.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 36: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 34

load

load test – ‘up’

extend?

200

charged100

charged

no

yes

path test

99

100

101

boundary values

‘up’

tc 1: use=99, pay 100

tc 2: use=100, pay 100

tc 3: use=101, pay 200

‘down’

tc1: use=101, pay 200

tc2: use=100, pay 100

tc3: use=99, pay 100

boundary values

load test – ‘down’

load

load test – ‘up’

extend?

200

charged100

charged

no

yes

path test

99

100

101

boundary values

‘up’

tc 1: use=99, pay 100

tc 2: use=100, pay 100

tc 3: use=101, pay 200

‘down’

tc1: use=101, pay 200

tc2: use=100, pay 100

tc3: use=99, pay 100

boundary values

load test – ‘down’

• (Automatic) scaling up or down

does not perform as required

• At scaling moments functional

problems emerge

• Insight in use based costs is

not sufficient

Page 37: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 35

ISO 27001 aspects:

• Confidentiality of the data and the accompanying risk that unauthorized people can view the data

• Integrity of data and the accompanying risk that data is altered or lost unintentionally

• Availability of data and the accompanying risk that data (and services) is not available when it is required

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

ISO 27001 aspects:

• Confidentiality of the data and the accompanying risk that unauthorized people can view the data

• Integrity of data and the accompanying risk that data is altered or lost unintentionally

• Availability of data and the accompanying risk that data (and services) is not available when it is required

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

• Who has access to the data?

• Can the user trust that the data is

correct?

• Can the user gain access to the data at

all times?

Page 38: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 36

• Security at:

– Network

– Supplier

– User

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easuresTesting security robustness against Internet

attacks

- Directory traversal. Read and/or write in

directories other than those allowed.

- XML external entity attack. Include extra

(bad) data in an XML file.

- SQL injection. Request and/or change data

by manipulating SQL queries.

- Cross-site scripting (XSS). Transfer data to

other websites without the user knowing.

- Session manipulation. Skip steps or

validation in a session.

• Security at:

– Network

– Supplier

– User

• Encryption

• Authentication and authorisation

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

IDaaS

Page 39: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 37

• Security at:

– Network

– Supplier

– User

• Encryption

• Authentication and authorisation

• Test logs and audit trails

• Security Audits

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

IDaaS

Experts

Security patch routines

• Completeness and correctness of specifications and manuals

– Supplier

– User

• Availability of test environments

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Interface specifications

Supported platforms

Business process specs

User manuals

Page 40: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 38

Manageablity of test environments

• Everything in the cloud

Manageablity of test environments

• Link all current environments to the service

Page 41: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 39

Manageablity of test environments

• Link Production to the real service

• Link other environments to a MOCK SERVICE(or another instance of the service)

• Completeness and correctness of specifications and manuals

– Supplier

– User

• Availability of test environments

• Management of:

– Defects

– Changes

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 42: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 40

Defect Management

Incident

Supplier resolves it

Client resolves it

Incident not resolved

Test

Change work process

Change configuration

Custom solution

Service not selected

Terminate use of service

Workaround work instruction

Test

Test

Test

Test

Migrate

and test

• Completeness and correctness of specifications and manuals

– Supplier

– User

• Availability of test environments

• Management of:

– Defects

– Changes

• Maintainability of the software

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 43: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 41

• Role of system architecture

• Monitoring and Logging

• Guarantees and SLA’s

• Test fail-over mechanism

• Test online/offline

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Fail-over testing

A: disrupted

B: active

A: active

B: inactive

A is disrupted

B takes over service

A: inactive

B: active

dis

ruptio

nin

A e

nded

no c

hange

A is

dis

rupte

d

no c

hange

A: active

B: disruptedB is disrupted

A takes over service

B is d

isru

pte

dno c

hange

dis

ruption

in B

ended

no c

hange A: disrupted

B: disrupted

Page 44: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 42

Fail-over testing

A: disrupted

B: active

A: active

B: inactive

A is disrupted

B takes over service

A: inactive

B: active

dis

ruptio

nin

A e

nded

no c

hange

A is

dis

rupte

d

no c

hange

A: active

B: disruptedB is disrupted

A takes over service

B is d

isru

pte

dno c

hange

dis

ruption

in B

ended

no c

hange A: disrupted

B: disrupted

• Has the configuration been disturbed?

• Is the failure even noticed?

• Does the automatic failover start to work?

• Are there any transactions lost?

• Is there any data lost (counts, checksums)?

• If there is an audit trail, does it function properly?

• Is performance back to normal?

• Are there any incidents from the functional regression

test (perhaps a limited set, for instance aimed at the fifty

most used or most vital functions)?

Fail-over testing

A: disrupted

B: active

A: active

B: inactive

A is disrupted

B takes over service

A: inactive

B: active

dis

ruptio

nin

A e

nded

no c

hange

A is

dis

rupte

d

no c

hange

A: active

B: disruptedB is disrupted

A takes over service

B is d

isru

pte

dno c

hange

dis

ruption

in B

ended

no c

hange A: disrupted

B: disrupted

Test management aspects

• Sufficient technical support

• Sufficient functional knowledge of the E2E processes

• All planned service tests completed

• The right authorizations in the services

• A supplier willing to cooperate.

Page 45: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 43

Online – Offline

Use case testing.

Global testing.

Use case testing.

Global testing.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Online – Offline

Use case testing.

Global testing.

Use case testing.

Global testing.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Off line tests focussed on problems:

• Work continues, based on out-of-date information, and

this information could be changed in the cloud during the

offline period.

• The users are not aware that they are working (partly)

online (and are lead to believe differently).*

• Synchronization conflicts arise because data is changed

locally as well as in the cloud.

Page 46: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 44

Online – Offline

Use case testing.

Global testing.

Use case testing.

Global testing.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Off line test cases:

• End the connection and check whether the users can see

that they are working offline.

• Disrupt the connection (for instance, a port or a certain

type of IP traffic) and check whether problems arise.

• Check whether changes that are made offline find their

way to the cloud when online status is regained.

• Check whether conflicts between offline and cloud data

are handled robustly (which is in fact a functional

requirement).

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing caused by

Legislation & Regulations

Testing caused by

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Page 47: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 45

Functional test objectives

• Does the service fit the business processes and vv?

• Is the service quality sufficient (number of bugs)?

• Is the service sufficiently user friendly?

• Is the service configuration done correctly?

• Does supplier customization function properly?

• Does customer customization function properly?

• Do interfaces work properly?

• Are platforms properly supported?

• Does everything work after changes (is there no regression)?

Functional test objectives

• Does the service fit the business processes and vv?

• Is the service quality sufficient (number of bugs)?

• Is the service sufficiently user friendly?

• Is the service configuration done correctly?

• Does supplier customization function properly?

• Does customer customization function properly?

• Do interfaces work properly?

• Are platforms properly supported?

• Does everything work after changes (is there no regression)?

PCT UCT E2E

ET

User documentation

Technique – syntax – semantics – non functional

Page 48: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 46

Any device – any platform

Multiplatform

testing.

Multiplatform

testing.

Multiplatform

testing.

Multiplatform

testing.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing caused by

Legislation & Regulations

Testing caused by

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

3997 distinct Android devices

http://opensignal.com/reports/fragmentation.php

Page 49: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 47

Internet Explorer 6

Internet Explorer 7

Internet Explorer 8

Firefox 3.5

Firefox 3.6

Firefox 4

Safari 4

Safari 5

Chrome11

Opera11

Windows XP

Windows Vista

Windows 7

Windows 2003 Server

Windows 8

Windows CE

Linux

Unix

Mac OS Lion

Mac OS Snow Leopard

iOS

Android

Operating systems

Browsers

Multi-platform testing

Devices

Computer

Mobile phones

Tablet

PC

Macintosh

SUN

NOKIA H

Samsung HWindows Mobile

iPhone ...

H

MOTOROLA H

Blackberry H

ASUS ...

H

Internet Explorer 6

Internet Explorer 7

Internet Explorer 8

Firefox 3.5

Firefox 3.6

Firefox 4

Safari 4

Safari 5

Chrome11

Opera11

Windows XP

Windows Vista

Windows 7

Windows 2003 Server

Windows 8

Windows CE

Linux

Unix

Mac OS Lion

Mac OS Snow Leopard

iOS

Android

Operating systems

Browsers

Multi-platform testing

Devices

Computer

Mobile phones

Tablet

PC

Macintosh

SUN

NOKIA H

Samsung HWindows Mobile

iPhone ...

H

MOTOROLA H

Blackberry H

ASUS ...

H

Page 50: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 48

Any device – any platform

Multiplatform

testing.

Multiplatform

testing.

Multiplatform

testing.

Multiplatform

testing.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing caused by

Legislation & Regulations

Testing caused by

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Off line

Apps

Web services

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing caused by

Legislation & Regulations

Testing caused by

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Testing in SOA

environments

Testing mobile

apps

Page 51: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 49

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Scenarios

• Transfer into the cloud, applications remain the same

– data moved to another location

• Transfer to SaaS

– data migrated to new service

• Transfer from one to another SaaS

– similar

• Transfer out of the cloud.

– similar

Data conversion

• Testing conversion rules

• Testing conversion on input data

• Testing if any data is lost

• Testing ongoing transactions

Existing

systems

Extraction Conversion Import

Conversion

softwareService

• Rounding (totals incorrect)

• Field lengths (truncation)

• Totals (information lost)

• Date and time conversions

� what means 08-09-11?

• Audit trail, check sums

• E2E business scenario’s

Page 52: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 50

Other aspects

• Cleaning data defects

– solved before migration

– no problems during migration

• Testing security aspects

– during and after migration

– not TOO much data migrated

• Testing performance

– speed (how long does it take?)

– volume (capacity sufficient?)

– stability at full volume

Example: email to the cloud

• Tools migrate existing emails to the cloud

• Low risk:

– migrating one or some mailboxes and executing a limited testing

– if successful: implementation for all mail boxes

• High risk:

– no emails lost in migration?

– formatting of the emails still correct?

– all attachments still there?

– all attributes migrated (priorities, timestamps, flags, …)?

Legal importance of email

reading, forwarding, replying,

check on contents

Page 53: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 51

Legislation + Regulations

=

Test basis

Incidental testing.Incidental testing.

Compliancy testing.Compliancy testing.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Sarbanes Oxley

Where is my data stored?

– nothing, or hardly anything, to be found on this subject

– service stores data outside the borders of permitted countries � additional measures?

– service stores data within the borders of permitted counties � okay

data owner is responsible for ensuring

that the protection of personal data is at

the required level wherever it is held

Page 54: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 52

Checking for legislation and regulations

• List where data that is stored in the cloud

• Find the requirements that are applicable to this data

• Check supplier terms with customer’s requirements

• Perform (external) audit for high risk

• Test manager provides advice, management decides

Legal support needed for high risk

Example. A supplier of a storage service claims to be the owner of the

intellectual capital of all data stored at their facilities. It is highly unlikely that

this is compatible with the interests of the organization that is the actual

owner of the data.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

Legal issues – threats

Page 55: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 53

Example: Dropbox

Compliance with Laws and Law Enforcement Requests; Protection of Dropbox's Rights.

• We may disclose to parties outside Dropbox files stored in your Dropbox and information about you that we collect when we have a good faith belief that disclosure is reasonably necessary to (a) comply with a law, regulation or compulsory legal request; (b) protect the safety of any person from death or serious bodily injury; (c) prevent fraud or abuse of Dropbox or its users; or (d) to protect Dropbox’s property rights. If we provide your Dropboxfiles to a law enforcement agency as set forth above, we will remove Dropbox’s encryption from the files before providing them to law enforcement. However, Dropbox will not be able to decrypt any files that you encrypted prior to storing them on Dropbox.

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Page 56: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 54

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

ContinuousEnd-to-End Testing

Continuous Change

Continuity

Privacy

Multi platform

Legislation

Cyber crime

Impact organisation

Standards

Check

Intake

Monitor

Interview

Proof of concept

Page 57: Testing Cloud Services: SaaS, PaaS, and IaaS

10-4-2014

© Polteq 55

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

MaintainabilityMaintainability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Testing starts early: in selection

Scope of testing is widened

Testing continues in production

Testing starts early: in selection

Scope of testing is widened

Testing continues in production

Performance TestingPerformance Testing

Security TestingSecurity Testing

Manageability TestingManageability Testing

Availability & Continuity

Testing

Availability & Continuity

Testing

Functional TestingFunctional Testing

Migration TestingMigration Testing

Testing due to

Legislation & Regulations

Testing due to

Legislation & Regulations

Testing in ProductionTesting in Production

Testing during SelectionTesting during Selection

Test M

easures

Test M

easures

PerformancePerformance

SecuritySecurity

Availability & ContinuityAvailability & Continuity

FunctionalityFunctionality

ManageabilityManageability

Legislation & RegulationsLegislation & Regulations

Suppliers & OutsourcingSuppliers & OutsourcingRisks

Risks

Thank you!Thank you!