amazon web services cse 490h this presentation incorporates content licensed under the creative...
Post on 21-Dec-2015
219 views
TRANSCRIPT
![Page 1: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/1.jpg)
Amazon Web Services
CSE 490H
This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License.
![Page 2: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/2.jpg)
Overview
Questions about Project 3? EC2 S3 Putting them together
![Page 3: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/3.jpg)
Brief Virtualization Review
Hardware Machine Platform
Operating System
End-User Applications
![Page 4: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/4.jpg)
Host and Guest Systems
Hardware Machine Platform
Operating System
Host-machine applications
Sandbox
Guest OS
Guest Apps
Hypervisor
![Page 5: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/5.jpg)
Fully Virtualized Machine
Hardware Machine Platform
OS
Apps
OS
Apps
Hypervisor
![Page 6: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/6.jpg)
Interacting with the Hypervisor
Hardware Machine Platform
OS
Apps
OS
Apps
HypervisorControl
interface
![Page 7: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/7.jpg)
“add machine”
Hardware Machine Platform
OS
Apps
OS
Apps
HypervisorControl
interface
“add”
![Page 8: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/8.jpg)
New machine added
Hardware Machine Platform
OS
Apps
OS
Apps
HypervisorControl
interface
OS
Apps
![Page 9: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/9.jpg)
Managing Large Deployments
Hardware Machine Platform
OS
Apps
OS
Apps
HypervisorControl
interface
Provisioning Node
Network connection
![Page 10: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/10.jpg)
How Web Servers Work
Interacting with a web servers has three stagesRequest – A URL (and some data) is sent to
the serverHandler – Some logic looks at the requestResponse – Some data is sent back to the
user
![Page 11: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/11.jpg)
Serving a Web Page
Request: “GET /index.html” Handler: The server itself reads the
$wwwroot/index.html file Response: The contents of the file are
sent back to the user
![Page 12: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/12.jpg)
Web Applications
Request: “GET /buyItem.php?itemId=414&customerId=2000”
Handler: The server invokes the buyItem.php script and runs the code
Response: Whatever output is sent back from the script gets sent back to the end user’s web browser
![Page 13: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/13.jpg)
CGI Scripts
This sort of “Web page that does something” is referred to as CGI (the Common Gateway Interface)
Typically a script that takes in parameters, does some processing, and returns a new web page to view in your browser
![Page 14: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/14.jpg)
REST Interfaces
… Buy why the focus on “pages?”
Request: “GET /launchMissiles.exe?authCode=12345”
Handler: launchMissiles program works Response: “Boom!”
…This is a “web service”
![Page 15: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/15.jpg)
REST Interfaces
Well-defined “URLs” perform operations Web server is connected to programs
specific to each of those operations Typically work with XML-formatted data Designed for connections to be self-
contained and non-persistent
![Page 16: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/16.jpg)
Web without the Web Browser
Any application can send/receive data with the HTTP protocol
Requests can be sent by command-line utilities, other GUI apps, etc
They then parse the XML response, display data as is appropriate
![Page 17: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/17.jpg)
Put them together…
Hardware Machine Platform
OS
Apps
OS
Apps
HypervisorControl
interface
Provisioning Node
Web ServerRequests from the Internet
![Page 18: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/18.jpg)
EC2 Terminology
Instance – A virtual machine Image, AMI – The initial state for a VM Security Group – A set of instances with
shared firewall settings
![Page 19: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/19.jpg)
Launching Instances
ec2-run-instancesRequires AMI id (e.g., ami-1a2b3c4d)User key, security group, instance type, count
Doesn’t run immediately – instances start in “pending” state; later transition to “running”
![Page 20: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/20.jpg)
Where’s my instance? ec2-describe-instances
RESERVATION r-b27edbdb 726089167552tom
INSTANCE i-90a413f9 ami-4715f12e
ec2-67-202-10-48.compute-1.amazonaws.com
ip-10-251-22-143.ec2.internal
running tom 0 m1.large
2008-11-11T17:23:39+0000
us-east-1c aki-b51cf9dc ari-b31cf9da
![Page 21: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/21.jpg)
Firewall rules
ec2-describe-group (groupname)
GROUP 726089167552 aaron aaronPERMISSION 726089167552 aaron ALLOWS
tcp 22 22 FROM CIDR 0.0.0.0/0PERMISSION 726089167552 aaron ALLOWS
tcp 80 80 FROM CIDR 0.0.0.0/0
Create a group with ec2-add-group Control permissions with ec2-(de)authorize
![Page 22: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/22.jpg)
A new instance, a blank slate
How do you log in to an instance? How does an instance know what it should
do?Per-instance metadata
![Page 23: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/23.jpg)
ssh keypairs
ssh lets you log in to a remote machine with a usernameAuthentication can be done by passwordAlso can be done with public/private keys
EC2 will let you register a key pair in db Injects public key into instance on bootYou have the private key, you can log in
![Page 24: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/24.jpg)
Shutting down instances
ec2-terminate-instance (instance id)
Terminates a running instance Use ec2-describe-instances to get the
instance id (i-XXXXXXXX)
![Page 25: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/25.jpg)
Using Instance Metadata
You can create an AMI to do anything you want
Very specific AMI may already have full application stack already loaded
More generic AMI may run a bootstrap scriptCan download more programs, data from
another source
![Page 26: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/26.jpg)
S3 – The Simple Storage Service
S3 is an infinitely-large, web-accessible storage service
Data is stored in “buckets” as (key, value) pairsEffectively (server, filename) file mapping
![Page 27: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/27.jpg)
S3 has a REST API too
PUT request to a URL with data uploads the data as the value bound to the key specified by the URL
GET request to the URL retrieves the value (file) or “404 Not Found”
![Page 28: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/28.jpg)
S3 Buckets
Names must be globally unique(Since they are addressable as DNS entries)
Can hold an unlimited number of keys Each key can have up to 5 GB of value
![Page 29: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/29.jpg)
Starting a Server
ec2-run-instances can specify metadata A new server is provisioned and boots Boot process runs a script that reads
metadata This specifies location of another programRetrieves the program, runs itRetrieves data, starts more services, etc…
![Page 30: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/30.jpg)
Project 4 And You
Project 3 will provide you with map tiles and an index from (address lat, lon)
In project 4, you will:Upload this into S3Write a web server handler applet to do
address lookupsWrite the bootstrap scripts to retrieve data
from S3 into your instance and launch your server
![Page 31: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/31.jpg)
More Web Services
Simple Queue Service (SQS)Reliable producer—consumer queues that
hold millions of queue entries, with hundreds of servers connecting…
Simple Database Service (SDB)A lot like BigTable
![Page 32: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/32.jpg)
Self-Scaling Applications
www www www
Load monitorTo EC2
provisioning system
Load-balancing DNS frontend
S3 backing store for common data vault
End-user requests
![Page 33: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/33.jpg)
Self-Scaling Backends
Hadoop master
Job launcherTo EC2
provisioning system
Work queue
S3 output bucket
(many worker nodes)
S3 input bucket
Data collection processes
Front-end nodes
![Page 34: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/34.jpg)
GrepTheWeb
Large web crawl data is stored in S3 Users can submit regular expression to
the GTW programGTW uses Hadoop to search for dataPuts your results in an output bucket and
notifies you when it’s ready
![Page 35: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/35.jpg)
![Page 36: Amazon Web Services CSE 490H This presentation incorporates content licensed under the Creative Commons Attribution 2.5 License](https://reader036.vdocuments.site/reader036/viewer/2022062320/56649d625503460f94a452ed/html5/thumbnails/36.jpg)
Conclusions
Web Services make for clean couplings between systems
Hardware as a Service (EC2/S3) allows applications to use physical resources dynamically
The two put together allow for very scalable application design