cloud standards and virtualization

Cloud Standards and Virtualization

Dr. Peter Tröger, Senior Researcher Operating Systems and Middleware Group Hasso-Plattner-Institute Universität Potsdam

Dr. Peter Tröger | SDPS 2012

Cloud

-„...computing paradigm where the boundaries of computing will be determined by economic rationale rather than technical limits."(R.K. Chellappa 1997)

-Three independent (!) basic models of service provisioning

2

Servers Storage

Racks HVAC Power

Virtual Compute Virtual Machine

Virtual Storage Key-value Store

Block Store Infrastructure “Infrastructure as a Service” ,

“Utility Computing”

Cloud Data Store

Managed Container

Comm- unications

Platforms “Platform as a

Service”

Business Applications

Analytics Applications

Productivity Applications

Applications “Software as a Service”, “on-demand” apps


Cloud Role Model

-The customer needs ...

-Predictable scalability for minimal costs (think HPC).

-Application-driven cost optimization (think spot market).

-In many cases at least the reliability of local data centers.

-The customer gets ...

-... some provider-specific interface to a black box.

3 $Cloud ProviderCloud Customer

$Cloud ProviderCustomer‘s Client Cloud CustomerPractice

Theory

$

Hello,

A few days ago we sent you an email letting you know that we were working on recovering an inconsistent data snapshot of one or more of your Amazon EBS volumes. We are very sorry, but ultimately our efforts to manually recover your volume were unsuccessful. The hardware failed in such a way that we could not forensically restore the data.

What we were able to recover has been made available via a snapshot, although the data is in such a state that it may have little to no utility…

If you have no need for this snapshot, please delete it to avoid incurring storage charges.

We apologize for this volume loss and any impact to your business.

Sincerely,Amazon Web Services, EBS Support


Dark Clouds

-Amazon Elastic Cloud

-2006: S3 request volumes are monitored, but cryptographic overhead was not considered

-2008: Single-bit error in transmitted system state lead to global S3 storage outage, took 6 hours for repair, including complete ,re-boot‘

-2009: Bitbucket.org (Amazon-hosted), 19 hours outage

-2011: Outage of S3, Web 2.0 companies affected for days

-Google Apps (last case in September 2011)

-Microsoft Office 365 (cases in 2011, lasting more than a week)

-T-Mobile Sidekick: One week data outage (2009), permanent data loss for customers

-... an even larger set of unpublished issues ...

5


Why Clouds (May) Fail

-Traditional system fault models no longer fit -Memory with increased density and data rates

-Group of ,simple‘ cores instead of monolithic processor

-Interconnect as crucial component, fault isolation issues

-Reactive fault tolerance gets inappropriate -Recovery time correlates with system size

-24/7 business availability demands pro-active fault tolerance

-Reactive FT does not scale (Examples: HPC, clouds)

-Virtualization as new system layer

-Dependability of (hardware-supported) hypervisors, distributed load management

-Imprecise system knowledge -Information about reliability properties ranges from imprecise to missing

6


Solution on Provider Side

-Proactive failover: „Move load away before bad things happen“

-Migration object moved between failover units at one system layer

-System layer as containment barrier

-Coverage of the layer

-Fault model from available data

-Monitoring granularity may prevent fault detection for lower levels

-Overhead of the layer

-Prediction quality (from data) influences false migration percentage

7


Solution on Provider Side

8

!"#$

%&'()*(+&,$-**$

!.'($!.'($

!.'($!.'($

*&/01

.&'2$

3(4/5(6$

78$

9::

,/5&;

.0$8('4('$

78$

<.'=,.&2$

9::

,/5&;

.0$8('4('$

<.'=,.&2$

-/'+>&,/?&;.0$!,>6+('$*&0&@(A(0+$

Phy

sica

l Mac

hine

Sta

tus

Virtu

al M

achi

ne S

tatu

s

B(&,+C$D02/5&+.'$E&'@(+$*&5C/0($85C(2>,('$*/@'&;.0$!.0+'.,,('$

"'()2/5+.'6$

"'()2/5+.'6$

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

B&'2F&'(G$!"#$%"&'%&()*+,-.$%/&!/%

"'()2/5+.'6$

"'()2/5+.'6$

"'()2/5+.'6$

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%-/'+>&,$*&5C/0($*.0/+.'G$-:'.1(H$0123)4$%4()5%

"'()2/5+.'6$

"'()2/5+.'6$

"'()2/5+.'6$

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

7:('&;0@$8I6+(AG$63(750$%8,-6)91%!)-,3)(,-.%:0(-0+%

"'()2/5+.'6$

"'()2/5+.'6$

"'()2/5+.'6$

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

B&'2F&'($,(4(,G$!"#$%"&'%&()*+,-.$%/&!/%

9::,/5&;.0$J$*/22,(F&'(G$#44+,57;)-$%#44<0(=0($%><?@AA%

"'()2/5+.'6$

"'()2/5+.'6$


On Customer Side ?

-Allow customer to realize error mitigation

-Avoidance of vendor lock-in

-Functional replication

-Meta-scheduling, adaptive application reconfiguration

-Information dispersal, smart data replication-> Demands standardized status monitoring and control

-Support for Offline Operation -> Demands standardized status monitoring

9

Cloud ProviderCloud ProviderCloud Provider

Cloud Customer

Client


Cloud Standards

-API for lifecycle management of

-Customer virtual machine (IAAS)

-Customer application (PAAS)

-Customer service instance / tenant / job (SAAS)

-Wide area of functionality

-Deployment, installation, status change, configuration

-Monitoring - Access latency and data rates, availability

-Audit / SLAs - Data removal and locality, isolation

-Development - Tracing and Debugging

10


Classification of standards (adopted from Don Box, 2004)

11 - „Desert Island“ specifications - ,must have‘ standards for operations

- „Island Resort“ specifications - the next layer of important specs

- „New Zealand“ specifications - specs you'd probably need once in a lifetime

- „Island Of Doctor Moreau“ specifications - the ugly step children of the spec family

- „Fantasy Island“ specifications - specs everbody would love to see but never gets


Cloud Standards

-Prescriptive standards

-Cloud provider <-> provider remote interoperability

-If needed, ask Grid people (OGSI WSRF, Unicore, EMI)

-Cloud customer <-> provider remote interoperability

-Functional access: OCCI, OVF

-SaaS / PaaS data access: SNIA CDMI

-Security: CSA specifications, IETF CloudAudit

-Cloud-based applications (e.g. OGF DRMAA)

-Evaluative standards (ISO 9000, FIPS 140-2)

12


Distributed Management Task Force (DMTF)

-Open Virtualization Format (OVF)

-XenSource, IBM, Sun, Microsoft, VMWare, Intel, ...

-Portable virtual machine packaging, extensible

-Virtual disc format, virtual hardware description

-Lifecycle management information

-Specific resource description linked to DMTF CIM model

-Widely accepted in products (e.g. VMWare)

-Cloud Infrastructure Management Interface (CIMI)

-HTTP / REST based cloud management

-Sole IaaS focus

13


Open Grid Forum

-Open Cloud Computing Interface (OCCI)

-Runtime management API, ReST / HTTP - based

-Infrastructure profile for IaaS, relies on OVF

-Other groups: Monitoring, billing, SLA‘s

14


Example: OGF OCCI

15


Data Cloud

-Storage Networking Industry Association (SNIA)

-Cloud storage initiative (CSI) for on-demand storage

-Cisco, HP, IBM, Hitachi, NetApp, Oracle, Symantec, EMC, ...

-From ,manage your storage‘ to ,manage your data‘

-Cloud Data Management Interface (CDMI)

-Allows to tag data with special system metadata

-Tells the cloud storage provider about services requested

-Backup, Archiving, Encryption, ...

16

PAS xxxxx-yyy © ISO/IEC:2009(E) - 27 -

CDMI 1.0.1 (September 15, 2011) Technical Position

5.6 Reference Model for Cloud Storage Interfaces

The Cloud Storage Reference Model is shown in Figure 4.

This model shows multiple types of cloud data storage interfaces that are able to support both legacy and new applications. All of the interfaces allow storage to be provided on demand, drawn from a pool of resources. The storage capacity is drawn from a pool of storage capacity provided by storage services. The data services are applied to individual data elements, as determined by the data system metadata. Metadata specifies the data requirements on the basis of individual data elements or on groups of data elements (containers).

5.7 Cloud Data Management Interface

The Cloud Data Management Interface (CDMI™) shown in Figure 4 may be used to create, retrieve, update, and delete objects in a cloud. The features of the CDMI include functions that:

• allow clients to discover the capabilities available in the cloud storage offering,• manage containers and the data that is placed in them, and

Figure 4 - Cloud Storage Reference Model

Data Storage Cloud

Storage Services

Data Services

Storage Services

Data Services

Storage Services

Data Services

Storage Services

Data Services

Storage Services

Data Services

Storage Services

Data Services

SNIA Cloud Data Management Interface (CDMI)

Cloud Data Management

Table

Table

Table

Table

Table

Draws resources on demand

Container

POSIX (NFS, CIFS, WebDAV)

iSCSI, FC, FCoE LUNs, Targets

XAM VIM for CDMI Database/Table

Client

XAM ClientObject Storage Client

Block Storage Client Filesystem Client SNIA Cloud Data Management Interface (CDMI)

Multiple, vendor-specific interfaces

Container Container

Container

Data/Storage Management Client

Management of the cloud storage can be standalone or part of the overall cloud computing management.

Clients acting in the role of using a data storage interface

Clients acting in the role of managing data/storage

Clients can be inside the storage cloud (i.e., providing storage resources to the cloud as well as consuming them) or outside the storage cloud (i.e., only consuming resources).

Information Services (future)



Exports to cloud computing


Cloud Security Alliance

-Widely supported industry initiative

-Best practices, consistent measurements, cloud controls matrix, cloud trust protocol, assurance maturity model, incident management

-Top threats to Cloud Computing 1.Abuse and Nefarious Use of Cloud Computing

2.Insecure Interfaces and APIs

3.Malicious Insiders

4.Shared Technology Issues

5.Data Loss or Leakage

6.Account or Service Hijacking

7.Unknown Risk Profile

18


More ...

-Open Cloud Consortium (OCC)

-US-based effort for coordinated usage of clouds in research

-Open Science Data Cloud, Project Matsu, OpenFlow

-ETSI TC CLOUD - Continuation of Grid TC

-NIST - Meta standards (vocabulary, use cases, collections)

-OASIS - SAML, IDCloud, WS-*

-Open Group Cloud Work Group - business understanding

-TeleManagement Forum - Cloud marketplace-> IaaS is nicely covered, Paas / SaaS still missing ...

19


The End: Some Eco-SystemIn

tero

pera

bilit

y

XML, Schema

Messaging Metadata

Reso

urce

s

Tran

sact

ions

Security

Relia

bilit

y

Service Composition / Business Process

Transport (HTTP, MQ, TCP, IIOP, ...)

Agr

eem

ent

Man

agem

ent

20


The Quick Check: CSI

21 -Participating Companies ?

-Either agreed by competitors, or concurrent specifications for the same thing

-Status in standardization organizations ?

-Maturity of the document

-Implementations ?

-More than one implementation is anindicator for real-world adoption

-Look for implementations by competitors

-Moving target !!!

C

S

I


The Good, The Bad, And The Ugly

22 - The Good

- Competitors agree on something

- Backed by a true standardization body

- Multiple independent implementations

- The Bad

- Superseded specifications

- Specs without participation from the providers

- The Ugly

- Company or university proposals with a single (institutional) author

C

S

I


Summary

-Cloud dependability: Customer vs. provider perspective

-On customer side, standards would help with vendor lock-in

-IaaS management is covered, data models are hard

-Motivation for uptake of standards

-Innovation (re-use intellectual work)

-Competivness (invite new customers)

-Certification (market advantage for provider)

-Customer demand for interoperability (e.g. X.509)

-More research challenges with billing, PaaS, and SaaS

23