cloud computing lec2 - arvutiteaduse instituut · amazon elastic compute cloud (ec2) • one of the...

Basics of Cloud Computing – Lecture 2

Cloud Providers, SciCloud and

Research on Cloud at UTResearch on Cloud at UT

Satish Srirama

Outline

• Cloud computing services – recap

• Amazon cloud services

– Elastic Compute Cloud (EC2)

– Storage services - Amazon S3 and EBS– Storage services - Amazon S3 and EBS

• Cloud managers

• Eucalyptus

• SciCloud and Research at UT

10.04.2013 2/45Satish Srirama

Cloud Computing – Services - Recap

• Software as a Service – SaaS– A way to access applications

hosted on the web through your web browser

• Platform as a Service – PaaS– A pay-as-you-go model for IT

resources accessed over the

SaaS

Facebook, Flikr, Myspace.com,

Google maps API, Gmail

Level of

Abstraction

resources accessed over the Internet

• Infrastructure as a Service –IaaS– Use of commodity computers,

distributed across Internet, to perform parallel processing, distributed storage, indexing and mining of data

– Virtualization

PaaS

Google App Engine,

Force.com, Hadoop, Azure, Amazon S3, etc

IaaSAmazon EC2, SciCloud,

Joyent Accelerators, Nirvanix

Storage Delivery Network, etc.


Cloud Infrastructure

• Provisioning of computing resources

– CPU, Memory, Processing

– Basically an “Operating System” on demand

• Usually billed on a per-hour usage model• Usually billed on a per-hour usage model

• Players in this space

– Amazon EC2, Rackspace, GoGrid, Eucalyptus based SciCloud

• Management providers: RightScale, ElasticFox, Amazon Management Console


Cloud Storage

• Provisioning of data storage

– Either file/object based or Database like functionality

• Billed on bandwidth and storage consumed

• Players in the space• Players in the space

– Amazon S3, Amazon EBS

– Amazon’s SimpleDB, Google’s BigTable, Apache

Cassandra

• Management Providers: , Amazon Management

Console, Jungle Disk, Elephant Disk, PutPlace.com


Cloud Platforms

• Provides a complete software stack

– An IDE for the cloud

• Takes care of: Runtimes, Load balancing, Resource provisioningResource provisioning

• Players in the space

– Google AppEngine - Python (initially, now also Java)

– Force.com (SalesForce)

– Hadoop


Cloud Applications

• Applications that are completely ‘online’

• Operate on data that is stored in the ‘cloud’ or ‘ether’

• No client software generally required

• Billing: Ad. Revenue, Premium Services• Billing: Ad. Revenue, Premium Services

• Players in this space– Google Apps - Gmail / Google Docs

– Apple’s MobileMe

– Microsoft’s Live - Hotmail, Live Spaces

– SalesForce.com


Other cloud services

• Provides services, which other applications

can utilize

• Usually free for non-commercial use

• Players in the space• Players in the space

– Google/Yahoo Maps

– Google / Yahoo Web Services

– Amazon Merchant Services

– Amazon Simple Queue Service


Providers we focus at

• Amazon Web Services– Amazon EC2

– Amazon S3

– Amazon EBS

• Private cloud enabling technologies• Private cloud enabling technologies– Eucalyptus

– OpenStack

– SciCloud

• Management providers– ElasticFox

– RightScale


Amazon Elastic Compute Cloud (EC2)

• One of the very early pioneers of cloud computing

• In a nutshell:

– On Demand “Operating System”

• Complete virtual computer with CPU, Memory and disk spacespace

– Based on the XEN virtual image platform

• Variety of operating systems available

– Linux (Fedora, Ubuntu, CentOS, etc)

– Open Solaris

– Microsoft Windows


EC2 continued…

• Very simple pricing structure– CPU hours

– Machine size

– Bandwidth in and out of cloud

• Extremely FAST start up• Extremely FAST start up– 2-3 minutes from start to finish

• Instance snapshotting

• Very large/generous disk space provisioning– 160GB minimum for the standard instances

• Flexible API to control everything

• Wide range of virtual machine types


EC2 Instance types*

Instance CPU Memory Storag

e

Platform API Name Price (per h.)

Small

(default)

1 EC2

computing

unit

1.7 GB 160 GB 32/64 bit m1.small $0.06 (*nix)

$0.091 (win)

Medium 2 EC2 units 3.75 GB 410 GB 32/64 bit m1.medium $0.12 (*nix)

$0.182 (win)$0.182 (win)

Large 4 EC2 units 7.5 GB 850 GB 64 bit m1.large $0.24 (*nix)

$0.364 (win)

Extra Large 8 EC2 units 15 GB 1 690

GB

64 bit m1.xlarge $0.48 (*nix)

$0.728 (win)

Micro Up to 2 EC2

units

613 MB EBS

storage

only

32/64 bit t1.micro $0.02 (*nix)

$0.02 (win)

*Data taken on 08.04.2013 http://aws.amazon.com/ec2/instance-types/


EC2 advanced Instance types*

Instance CPU Memor

y

Storag

e

Platfor

m

API Name Price (per h.)

High-Memory

Instances

6.5 - 26

EC2

units

17.1 –

68.4 GB

420 -

1690

GB

64 bit m2.xlarge

m2.2xlarge

m2.4xlarge

Available at:

http://aws.a

mazon.com/

ec2/pricing/

High-CPU 5, 20 1.7, 7 350, 64 bit c1.medium Same as

• Other types– High I/O Instances– High Storage Instances

High-CPU

Instances

5, 20

EC2

units

1.7, 7

GB

350,

1690

GB

64 bit c1.medium

c1.xlarge

Same as

above

Cluster

Compute

Instances

88 EC2

units

60.5 GB 3370

GB

64 bit cc2.8xlarge Same as

above

*Data taken on 08.04.2013 http://aws.amazon.com/ec2/instance-types/10.04.2013 13/45Satish Srirama

EC2 Terminology

• Having an account– Access Key, Secret Key, Security group

• Availability Zone

• Amazon Machine Image (AMI)– A Virtual Machine File

– ami-XXXX – ami-XXXX

– Stored in a special bucket in Amazon’s S3

• Public and Private instances available– Private instances incur only S3 storage costs

• A Running Machine– Amazon Instance ( i-XXXX )

– Booted/Created from an Amazon Image

• Elastic IP addresses


Troubles with EC2

• On power-off all hard disk data is lost

• IP addresses are assigned at random

• Can’t turn off public IP address

• Do not forget to terminate the instances• Do not forget to terminate the instances


Simple Storage Service (S3)

• Enables you to upload, download, and store data across the Internet

• Buckets store data

– Buckets are the fundamental container in Amazon S3 for data storagedata storage

– 100 buckets for account

– No limit on no of objects that can be stored in a bucket

– Can store up to 5 TB of data in one object

– Object stores Data and Metadata

– Objects stored in a Region never leave the Region

– You cannot modify or append data to an existing object


Simple Storage Service (S3) - continued

• Objects are retrieved via a unique, developer-assigned key

• Keys– Example: http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.wsdl

"doc" is the bucket name and "2006-03-01/AmazonS3.wsdl" is the key.

• Prices– Storage: $0.095 per GB/Month–

– Reduced Redundancy Storage : $0.076 per GB/Month

– Data Transfer 0ut from S3 to AWS different zone: $0.02/GB

– PUT, COPY, POST, or LIST Requests: $0.005 per 1,000 Requests

– GET requests: $0.004 per 10,000 Requests

• Download Data– Downloading is possible from any where

– You can enable others to download and can charge them (Using Amazon DevPay)

http://aws.amazon.com/s3/


Elastic Block Storage (EBS)

• The answer to the persistence problem

• Raw unformatted external block devices

• Allocate 1GB to 1TB volumes

• 20 volumes per account

• Format with your own choice of file system • Format with your own choice of file system

• Attach to running instances in the same availability zone

• Automatically replicated to prevent data loss

• Create snapshots for backup, or to create new volumes from


Basic ec2-api-tools

• ec2-describe-images

• ec2-describe-instances

• ec2-run-instances

• ec2-create-volume• ec2-create-volume

• ec2-attach-volume

• ec2-allocate-address

• ec2-associate-address

• ec2-terminate-instances


Other cloud services from Amazon

• AWS management console

• Amazon Simple Queue Service (SQS)

• Amazon DynamoDB

• Amazon Relational Database Service• Amazon Relational Database Service

• Amazon CloudWatch

• Amazon Elastic MapReduce

• Elastic Load Balancing

• etc.


Getting started

• Signup for an AWS account

– Use your existing Amazon.com account if you

want

• Register a credit card• Register a credit card

– Billed on the 1st of every month for previous

month usage

• http://aws.amazon.com/ec2


Getting started – for free

• http://aws.amazon.com/free/ - free tier

• Valid for 12 months

• 750 hours of EC2 Linux Micro Instance usage

• 5 GB of S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests2,000 Put Requests

• 30 GB of Amazon EBS

• 25 Amazon SimpleDB Machine Hours and 1 GB of Storage

• Be careful

– You cross the limits, you are charged without any notice


ElasticFox

• Mozilla Firefox extension for managing your Amazon EC2 account

• Supported services

– List available AMIs – List available AMIs

– List your running instances

– Launch new instances of an AMI

– Manage security groups and launch permissions associated with your instances

– Manage Amazon EBS volumes


ElasticFox - GUI


Independent software vendors (ISV)

for S3

S3 Explorer filicio.us Jungle Disk

S3 Firefox Organizer MyOwnDB


RightScale

• RightScale cloud management platforms

• Automated configuration– From boot to production on auto-pilot

– Server templates & Right scripts

– Cloning/ replication of set-ups– Cloning/ replication of set-ups

• Automated management– Monitoring, alerts, SLAs -> actions

• Have several examples

• Can login as a developer https://my.rightscale.com


PRIVATE CLOUD ENABLING

TECHNOLOGIES

10.04.2013 28Satish Srirama

Eucalyptus

• Are you OK with using your credit card?

• Open source project

• Elastic Utility Computing Architecture Linking Your Programs To Useful Systems

• Web services based implementation of EC2 • Web services based implementation of EC2 infrastructure

• For establishing private clouds

• Functions as a software overlay– Existing installation should not be violated (too much)

• Focus on installation and maintenance


Eucalyptus Usage

• Experimentation vehicle prior to buying commercial services– Provide development, debugging, and “tech preview”

platform for Public Clouds

• Homogenize local IT environment with Public Clouds– AWS functionality locally makes moving & using Amazon – AWS functionality locally makes moving & using Amazon

AWS easier, cheaper, and more sustainable

• Provide a basic software development platform for the open source community– E.g. the “Linux Experience”

• Not designed as a replacement technology for AWS or any other Public Cloud service


Eucalyptus - architecture

Amazon EC2 Interface


Basic euca2ools

• euca-describe-images

• euca-describe-instances

• euca-run-instances

• euca-create-volume• euca-create-volume

• euca-attach-volume

• euca-terminate-instances

• euca-describe-availability-zones


OpenStack

• Founded by NASA and Rackspace

• The open source cloud computing platform

• Feature-rich and massively scalable

• Powers cloud storage, compute, and • Powers cloud storage, compute, and

networking

• A world-wide open source collaboration


OpenStack has 105+ participating

companies

…and over 1,350 individual contributors. [source: Piston]


3 Major OpenStack Components

• OpenStack Compute/Nova: provision and manage large networks of virtual machines

• OpenStack Object Store/Swift: Create petabytes of reliable storage using standard petabytes of reliable storage using standard servers

• OpenStack Image Service/Glance: Catalog and manage large libraries of server images

• Other components: Dashboard, Load Balancing, Authentication...


Working with OpenStack

• Last week keys are created for you all

• This week you work with the OpenStack

private cloud

• Just like Eucalyptus euca2ools are sufficient to • Just like Eucalyptus euca2ools are sufficient to

invoke the OpenStack services


SCICLOUD & UT’S RESEARCH ON

CLOUD

10.04.2013 37Satish Srirama

Scientific Computing Cloud (SciCloud) [Srirama et al, CCGrid 2010]

• Mobile Cloud Lab owned private cloud infrastructure

• Collection of smaller clouds built using Eucalyptus/OpenStack platformsEucalyptus/OpenStack platforms

• Goal of the project

– To efficiently use the already existing resources of universities

– To address computationally intensive scientific, mathematical, and academic problems

http://mc.cs.ut.ee/mcsite/projects/scicloud


Things that we do in cloud

• Research the utilization of cloud computing platforms for HPC [Srirama et al, SPJ 2011]

• Work with different Cloud computing frameworks– MapReduce (Lecture 3)

– Adapting scientific computing problems to cloud using – Adapting scientific computing problems to cloud using MapReduce [Srirama et al, FGCS 2012]

• Desktop to cloud migration (D2CM) tool [Srirama et al, CCSA 2012]

– Seamless migration of desktop virtual machine images to the Cloud

– Transform local VM images into cloud VM images


Things that we do in cloud - continued

• Mobile Cloud

– Delegate resource-intensive processing to the cloud [Flores et al, IJPCC 2012]

• To enrich the functionality of mobile applications

• Building social network and mashup applications on • Building social network and mashup applications on Android/iOS devices [Srirama et al, SOCA 2012]

– Mobile Cloud Middleware [Flores et al, MoMM 2011]

• Cloud interoperability

• Hiding the complexity of cloud from mobiles

– Code offloading


Things that we do in cloud - continued

• Distributed troubleshooting [Shor & Srirama, CLOSER 2011]

– Finding memory leaks in distributed applications

– Adapting the tools to cloud solutions

• Migrating Enterprise Applications to the Cloud• Migrating Enterprise Applications to the Cloud

– Refactoring enterprise applications for cloud

• ESB based applications

– Framework for monitoring performance of cloud

based web applications [Vasar et al, NORDICLOUD 2012]

• Data mining and Bioinformatics on the cloud


How can you contribute…

• We are always open to new research ideas

• Several theses are open…

– Check at project home page or

http://mc.cs.ut.ee/mcsite/theseshttp://mc.cs.ut.ee/mcsite/theses

– Interested students can approach me

• Join and participate in our seminars

– Mobile and Cloud Computing Seminar -

MTAT.03.280


This week in lab

• You work with SciCloud

– Creating instances

– Connecting to instances through ssh

– Installing software– Installing software

– Preparing images and snapshots


Next Lecture

• Introduction to MapReduce


References

• Amazon Web (Cloud) Services – documentation http://aws.amazon.com/documentation/

• SciCloud – homepage http://mc.cs.ut.ee/mcsite/projects/scicloud

• Eucalyptus User's Guide http://www.eucalyptus.com/docs/3.1/ug-3.1.0.pdf

• S. N. Srirama, O. Batrashev, E. Vainikko: SciCloud: Scientific Computing on the Cloud, 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2010), May 17-20, 2010, pp. 579. IEEE CS.

• S. N. Srirama, O. Batrashev, P. Jakovits, E. Vainikko: Scalability of Parallel Scientific Applications on the Cloud, Scientific Programming Journal, Special Issue on Science-driven Cloud Computing, 19(2-3):91-105, 2011. IOS Press.

• S. N. Srirama, P. Jakovits, E. Vainikko: Adapting Scientific Computing Problems to Clouds using MapReduce, Future Generation Computer Systems Journal, 28(1):184-192, 2012. Elsevier press.

• S. N. Srirama, C. Willmore, V. Ivanistsev, P. Jakovits: Desktop to Cloud Migration of Scientific Experiments, 2nd • S. N. Srirama, C. Willmore, V. Ivanistsev, P. Jakovits: Desktop to Cloud Migration of Scientific Experiments, 2nd International Workshop on Cloud Computing and Scientific Applications (CCSA) @ 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2012), May 13-16, 2012.

• H. Flores, S. N. Srirama, C. Paniagua: Towards Mobile Cloud Applications: Offloading Resource-Intensive Tasks To Hybrid Clouds, International Journal of Pervasive Computing and Communications, ISSN: 1742-7371, 8(4):344-367, 2012. Emerald Group Publishing Limited.

• S. N. Srirama, C. Paniagua, H. Flores: Social Group Formation with Mobile Cloud Services, Service Oriented Computing and Applications Journal, ISSN: 1863-2386, 6(4):351-362, 2012. Springer.

• H. Flores, S. N. Srirama, C. Paniagua: A Generic Middleware Framework for Handling Process Intensive Hybrid Cloud Services from Mobiles, The 9th International Conference on Advances in Mobile Computing & Multimedia (MoMM-2011), December 5-7, 2011, pp. 87-95. ACM.

• V. Shor, S. N. Srirama: A Statistical Approach For Identifying Memory Leaks In Cloud Applications, First International Conference on Cloud Computing and Services Science (CLOSER 2011), May 7-9, 2011, pp. 623-628. SciTePress.

• M. Vasar, S. N. Srirama, M. Dumas: Framework for Monitoring and Testing Web Application Scalability on the Cloud, Nordic Symposium on Cloud Computing & Internet Technologies (NORDICLOUD), 2012, pp. 53-60. ACM.


cloud computing lec2 - arvutiteaduse instituut · amazon elastic compute cloud (ec2) • one of the...

Documents