living in the compute cloud
DESCRIPTION
Cloud Computing stacks at Web 2.0 expo Berlin 2008 by Jonathan Weiss. Overview of Amazon Web Services and Google App Engine and real use casesTRANSCRIPT
![Page 1: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/1.jpg)
Living in the Compute Cloud
Jonathan Weiss
Peritor GmbH, www.peritor.com
Web 2.0 Expo Berlin, 2008
![Page 2: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/2.jpg)
2
Peritor
• Consultancy from Berlin, Germany
• Specialized on Architecture, Scaling, and Ruby on Rails
• We help our clients adopt Cloud Computing
![Page 3: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/3.jpg)
3
Motivation
![Page 4: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/4.jpg)
4
Starting Point
One machine:
• Apache
• Ruby on Rails / PHP / Perl / Java / …
• MySQL
![Page 5: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/5.jpg)
5
Worst Case Populariy
![Page 6: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/6.jpg)
6
A Difficult Path
![Page 7: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/7.jpg)
7
Problem: Backup
• High availability
• Redundancy
• Very big data sets
![Page 8: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/8.jpg)
8
Problem: File System
• Important files have to be accessed by many servers
• NFS / Samba not practical
![Page 9: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/9.jpg)
9
Problem: Database
• Scaling the database is really hard
• Replication setups are error-prone, complicated, and slow
![Page 10: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/10.jpg)
10
Problem: Spontaneous Traffic
![Page 11: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/11.jpg)
11
Problem: Load Fluctuation
![Page 12: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/12.jpg)
12
Don‘t reinvent the wheel!
![Page 13: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/13.jpg)
13
Cloud Computing Stacks
![Page 14: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/14.jpg)
14
Cloud Computing Stacks
Amazon Web Services
• EC2
• S3
• SimpleDB
Google App Engine
• Application hosting
• BigTable
![Page 15: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/15.jpg)
15
Amazon Web Services
![Page 16: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/16.jpg)
16
Amazon Web Services
Simple Storage Service - S3
Redundant Data Store
$0,15 per GB data per month
$0,10 - $0,20 per GB traffic
Virtual server per hour
$0,10 - $0,80 per CPU hour
$0,10 - $0,20 per GB traffic
Elastic Compute Cloud - EC2
…….
E-Commerce..
Hash-like database
$0,14 per SDB machine hour
$0,10 - $0,17 per GB traffic
SimpleDB
![Page 17: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/17.jpg)
17
S3 - Simple Storage Service
• Redundant storage - as much as you like
• max. 5 GB per object
• Organized in „Buckets“
• Web Service API for uploads
• Downloads via
• Web Service
• HTTP / HTTPS
• BitTorrent
![Page 18: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/18.jpg)
18
S3 - Buckets
• Unique over all S3
• Contains many key-value-metadata tupel
• Cannot contain other buckets!
• Key can contain „/“
MyBucket_name
S3
foto_5.jpg
backups/januar.zip
fotos/2007/001.png
www.peritor.com
site/screen.css
site/script.js
FreeBSD-6.2.iso
![Page 19: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/19.jpg)
19
EC2 - Elastic Compute Cloud
Based on XEN virtualization
On demand virtual servers - controlled with Web Service API
Use your favorite Linux distro (Linux 2.6.16)
![Page 20: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/20.jpg)
20
EC2 - Extras
• ACLs for hosts/ports access control
• Choose your favorite datacenter (currently only US)
• Elastic IP for dynamic reconfiguration of your endpoints
• Elastic Block Store
• Persistent storage
• Mountable as a filesystem (by one host at a time)
• Snapshots to S3
![Page 21: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/21.jpg)
21
EC2 Tools
List availabe images
Start a new instance
Login with SSH
Shutdown instance
![Page 22: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/22.jpg)
22
SimpleDB
• Hash-like database
• Stores items with attribute-value pairs
• Very different to the relational model of SQL databases
• Meant for small items, use S3 for binaries/files and store the URL in SimpleDB
• Organized into ‘domains’
• Redundant and distributed
![Page 23: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/23.jpg)
23
SimpleDB Data
• No schema
• Everything is a string
• Pad numbers with zeros 000098.5
• Use ISO 8601 format for dates 2008-10-22T 11:10+02:00CET
• Allows list values – multiple values for one attribute
• No joins between domains
• SQL-like queries to retrieve data
![Page 24: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/24.jpg)
24
SimpleDB Example
Sample Domain ‘products’
![Page 25: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/25.jpg)
25
Google App Engine
![Page 26: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/26.jpg)
26
Google App Engine
• Run your application directly on Google’s infrastructure
• No concept of hardware – just deploy your application
• At the moment limited to Python
• Free for up to 500MB and CPU for ~ 5mio page impressions
![Page 27: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/27.jpg)
27
Python Runtime
• Python 2.5.2
• SDK for local development
• Restricted on some low-level modules
![Page 28: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/28.jpg)
28
Google APIs
Compensation for not having access to lower levels like sockets:
• Memcache
• Image
• URL fetch
• Google Auth & User
![Page 29: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/29.jpg)
29
BigTable
Very similar to SimpleDB
• No schema
• List values
But also very different
• Data type support
• References and multiple tables
• Blob fields (1MB)
![Page 30: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/30.jpg)
30
BigTable in App Engine
Python bindings and GQL
![Page 31: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/31.jpg)
31
App Engine Gotchas
• All requests are killed after a few seconds
• No background jobs
• No possibility to backup/snapshot data
• Emails can only be sent from Google accounts
• Restricted to pure-Python libraries and given APIs
![Page 32: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/32.jpg)
32
How does this solve my problems?
![Page 33: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/33.jpg)
33
S3 - Backup
• s3sync.rb
• Brackup
• Jungle Disk
• S3 FUSE
• s3DAV
• Duplicity
• S3Browser
• Firefox S3 Organizer extension
• …
![Page 34: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/34.jpg)
34
s3sync.rb
Backup
Restore
![Page 35: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/35.jpg)
35
S3 Asset Host
![Page 36: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/36.jpg)
36
S3 - Authenticated User Data
![Page 37: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/37.jpg)
37
On-Demand Computing with EC2
Time based, e.g. with cron
![Page 38: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/38.jpg)
38
On-Demand Computing with EC2
Load based, e.g. with Monit
![Page 39: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/39.jpg)
39
On-Demand Computing with EC2
![Page 40: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/40.jpg)
40
On-Demand Computing with EC2
![Page 41: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/41.jpg)
41
Using SimpleDB & S3
![Page 42: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/42.jpg)
42
EC2 for extra capacity
![Page 43: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/43.jpg)
43
App Engine
![Page 44: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/44.jpg)
44
Real Life Use Cases
![Page 45: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/45.jpg)
45
GoodBad.me
• Twitter-Mashup for RailsRumble 2008
• Needed test/development servers and performance benchmarking infrastructure
• Conventional hosters too inflexible and slow
![Page 46: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/46.jpg)
46
DaWanda.com
• Marketplace for handmade goods
• Large amount of uploaded user data
• Use of S3 to scale and cope with asset load
![Page 47: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/47.jpg)
47
G.ho.st
• Virtual desktop/computer in the browser
• Running completely on EC2 and S3
![Page 48: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/48.jpg)
48
Final Thoughs
![Page 49: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/49.jpg)
49
Final Thoughts
• Get accustomed to “Eventual Consistency”
• Be prepared to leave relational databases
• Many miss strong SLAs – most of the time you can live fine without
• Hardware is a commodity – only specialize in it if really necessary
![Page 50: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/50.jpg)
50
Q & A
![Page 51: Living In The Compute Cloud](https://reader034.vdocuments.site/reader034/viewer/2022051412/54b75fd94a7959a23c8b4628/html5/thumbnails/51.jpg)
51
Peritor GmbH
Teutonenstr. 16 14129Berlin
Internet: www.peritor.com Email: [email protected]
© Peritor GmbH - All Rights Reserved
Phone: +49 (0)30 69 20 09 84 0 Fax: +49 (0)30 69 20 09 84 9