cis 191: linux and unix class 8 oct 28th, 2015 practical networking and cloud solutions
TRANSCRIPT
CIS 191: Linux and Unix
Class 8Oct 28th, 2015
Practical Networking and Cloud Solutions
Final Project
• Raspberry Pi!– Basically a mini computer
• Why?– RPi is fun!– You have to set up a Linux environment from scratch– You will be limited by available computing power– Many online tutorials and resources
Final Project
• Raspberry Pi!• Idea
– Anything you want
• Hardware– Pi 1 by default. If you need Pi 2, request it in advance with a
valid reason– SD card and Wifi adapter– LEDs, resistors, capacitors, breadboards, batteries, cobbler– Sensors (light, pressure, sound, accelerometer etc.) on demand
Final Project
• Proposal due next week• Pi set-up bootcamp
– Tentatively next Saturday + the Monday after that
• Demo in last 191 lecture• You can also demo in the shared 19x project demo
Outline
Cloud Computing Overview
Amazon AWS Services
A Very Brief HTTP Overview
REST APIs
JSON Format
What is Cloud Computing?
• Generally a system of programming on a computer that exists somewhere else
• Possibly (and often!) not even really a computer, but a virtual slot on an existing server
What is Cloud Computing?
• Examples:
What is Cloud Computing?
• Examples:– Amazon AWS Elastic Compute Cloud (EC2)– Google App Engine– Microsoft Azure– Heroku app hosting– Eniac!
Cloud Computing – An Analogy
• Power sources– Everybody used to have their own!– Inconvenient…
Cloud Computing – An Analogy
• Power sources– Everybody used to have their own!– Inconvenient…
• So we scaled and built centralized power plants– Metered usage!– Outsource the problem of building, running, maintaining
generators– Pay-as-you-go model
Why Put Computers in the Cloud?
• Cost effective• Quick prototyping• Little to zero maintenance• Redundancy• Device independence• Easier collaboration
Outline
Cloud Computing Overview
Amazon AWS Services
A Very Brief HTTP Overview
REST APIs
JSON Format
Amazon Web Services
• A number of services– Amazon Elactic Compute Cloud (EC2) – Virtual machines– Amazon Simple Storage Service (S3) – Key-value store– Amazon DynamoDB -- Fast and flexible NoSQL database– Amazon Relational Database System (RDS) – Full virtual
distributed database system of your choosing– Elastic Load Balancing – Balance incoming traffic across EC2
instances– And much much more!
• All of these are free for the 1st year, some are free forever
Amazon AWS Account
• To use Amazon web services, you have to first make an AWS account at aws.amazon.com
• When you do this, you have to input a credit card number they can charge to – don’t worry, you won’t be charged as long as you follow some basic guidelines– And even if you were charged it would probably be a dollar or
two at most
• Many AWS services offer a “free tier” for the first year– This tier tends to be super slow and limited, but good enough
for our purposes
Zooming in: Amazon EC2
• After making an account on Amazon AWS, you can set up your very own EC2 instance– You can even pick the size!
• To do this, you must select “EC2” from the AWS management console, and then select some preferences– I’d suggest selecting the free tier, which is a “micro” instance– Yes it’s as small and limited as it sounds – but that’s fine!– I’d also suggest selecting an ubuntu operating system, since
that’s what we’ve been working with in class
Zooming in: Amazon EC2
• You’ll be asked to download a .pem security key; put this in a safe place
• After downloading the key, you’ll be able to log in to your ec2 instance by running – ssh –i my_key.pem ubuntu@ec2_IP_address
• You can get the IP address by looking at the instance’s details
Setting up EC2
• sudo apt-get install ...• Exactly the same as working in the virtual machine
command line!
Security Groups
• By default, your EC2 will only be accessible by you, over ssh– By using the pem key file and ssh -i!
• If you want to host a website, you’ll need to explicitly allow this in security groups by allowing access by everyone to port 80– This is the default internet port
Aside: Ports
• A port on your computer is like a sub-IP address (sort of)• These allow requesters to target particular applications
or services running on a machine!• So, web server applications tend to be running on port
80 of a larger server
Pricing Issues
• Make sure to run the micro instance if you don’t want to be charged!
• Make sure to terminate your instance when you are finished with it!– Shutting down isn’t the same…– Access this by right clicking on the instance in the console!
• Double check you don’t have any more resources allocated when you are finished using EC2
Data Issues – Don’t Lose It!
• If you store data on the virtual hard disk of an instance and the instance stops running for some reason– You terminate it– It fails (rare but it happens!)
• You will LOSE YOUR DATA!!!• If you want more security, you can use Elastic Block Store
– This is basically data replication across EC2 instances, for an extra fee
– This option will ensure data safety
Where Exactly Is My Instance?
• Who knows?• All you know is the general region in which the instance
runs, which you can request• Generally, you should pick an instance near where your
target audience is (in this case, us-east)– Reduce network latency as much as possible
Outline
Cloud Computing Overview
Amazon AWS Services
A Very Brief HTTP Overview
REST APIs
JSON Format
What is HTTP, anyway?
• We use it every day, but it’s more than just the http:// at the beginning of a web address
• HTTP specifies a protocol for sending data• This is really just an agreement between two machines,
and it allows for a friendly discourse between them
HyperText Transfer Protocol (HTTP)
• First specified in 1990, and it runs on top of TCP/IP– This ensures that we are already assured that data will reach the
target machine, and the packets we send will arrive in-order
• Default machine port is 80 for unsecure HTTP and 443 for secure (with secure socket layer technology – “Secure” basically means RSA encrypted traffic
• The protocol is actually very simple!– Choice: 1.0 (old and dumb) or 1.1 (new and shiny)– Always go for 1.1
GET /index.html HTTP/1.1Host: google.com
Basic “Methods” for HTTP – To Server
• GET– Retrieve contents of a URI
• HEAD– Retrieve only metadata of a URI
• POST– Adds new information– Intended for newsgroup posts; today, used to implement dynamic
content through web forms
• PUT– Store information at a URI
• DELETE– Delete information at a URI
HTTP Parameters
• HTTP can carry key-value pairs to the server!– Very useful for communication between client and server
• This typically look like – key=value
• These pairs can either be in the request URI, or in the request body
• The request body follows all http headers and will generally carry key/value pairs– Sometimes carries more information than this, however! Up to
the server how to interpret it– Useful as a communication standard… More on this soon
Zooming In: GET
• This is what your browser performs when you visit a page!
• You can add GET parameters (which the server can read) right to your URI– http://mysite.com/?myname=Spencer&mypet=Mochi
Zooming in: POST
• This is what your browser sends when you submit a form• For example, when you place an order at Amazon.com• Information can be sent via GET requests as well, but
POST is generally more secure– GET variables are included directly in the URI!– POST information is not stored by the browser or cached by the
server – GET requests are, for faster loading and response time– POST also has no restrictions on data length!– In general, POST when you want to send information to be
stored at a website, especially sensitive data
Zooming in: POST
• Also used to add something new to a page• For example, one might add a blog entry using the POST
request format• Especially useful if you want to post a new blog entry,
and don’t care what the resulting identifier is– I want to add a NEW post
Zooming in: POST
POST /order.html HTTP/1.1Host: www.my-shopper.comContent-Length: 114Content-Type: text/html
item1=6173849&item2=7272384&item3=32904234&password=monkeyfarts&username=doodooface&creditcardno=5555555555555555
Zooming in: PUT
• Used to update information at a URI• Pretty much the same as POST, except that the user
names the resource to be saved– Since you are naming an existing resource, you are updating it!
• Can also do different things depending on context…– But this is what it was intended for!
Zooming in: PUT
PUT /blogentries?id=2012 HTTP/1.1Host: www.my-site.comContent-Length: 35Content-Type:Text.html
Today was a terrible day because…
HTTP Status Codes – From Server
• Reports how the request was processed• Some common codes (way too many to list here)
– 200 OK– 301 Moved Permanently– 401 Unauthorized– 403 Forbidden– 404 Not Found– 500 Internal Server Error
• This last one is just a catch-all
HTTP Headers
• Included underneath an HTTP request or response• Specify more information for your request• These can allow you to set cookies, specify your current
browser, the types of charsets you will accept…
GET /index.html HTTP/1.1Host: 158.130.53.72:80Accept: text/html, application/xhtmlConnection: keep-alive
HTTP Headers
• An important header to remember is the Content-Length header
• This is used when a request (typically post or put) has a body!
HTTP Responses• Sent back to you by the server after you request a page• Include the protocol version and the status code, along with
some headers (additional information) and the content you requested– If you sent a HEAD request, just the headers are returned
HTTP/1.1 200 OKDate: …Server: ……
<html><head><title>…
Media (MIME) Types
• Specify nature of data requested or transferred– Defined in RFC 2046– Included in ‘Content-Type’ header– Includes at least type and subtype
• Examples– text/html– application/xml– text/plain– image/png– image/jpeg– video/quicktime
Outline
Cloud Computing Overview
Amazon AWS Services
A Very Brief HTTP Overview
Networking Tools
JSON Format
Telnet
• You can connect to a service via telnet and play around
telnet www.google.com 80
GET /rotots.txt 80HOST: www.google.com
ssh
• Secure shell via RSA encryption• What if we don’t encrypt the communication?
ssh
• Secure shell via RSA encryption• What if we don’t encrypt the communication?
– Your password, credit card information can be exposed in plaintext on any part of the network!
sshfs: Mounting A Remote Filesystem
• sshfs allows us to treat a remote resource as if it were located locally!
$ mkdir html$ sshfs eniac:public_html/ html$ vim html/index.html…
sshfs: Unmounting a Remote Filesystem
• On Ubuntu– You’ll have to use fusermount –u my_mount_directory
• On OSX– You’ll have to user umount my_mount_directory– If the directory is being stubborn and won’t unmount, then run
umount –f my_mount_directory
• If things are being really stubborn, you can always break out the ban hammer with– killall sshfs
sshfs: Uses
• Why would you want to pretend a remote resource is actually a local resource?– You can use cp instead of the more cumbersome scp
• Everything will be copied over the invisible ssh connection maintained by sshfs!
– You can keep your local editor and bash settings– Graphical programs are run locally on the remote file, rather
than remotely with graphics forwarding (faster)– You can run programs installed on your machine, provided with
data located elsewhere (also faster)
rsync: A File Copying Tool
• Rather than just doing a straightforward copy, rsync is a more powerful and clever copying mechanism
• rsync connects via ssh, just as scp does• However, rsync is smarter about what it copies
– rsync copies file differences rather than files– When copying large folders over the net, this is much faster…
• rsync preserves permissions, symbolic links, and other operating system features
• See man rsync for more details (if you have rsync installed)
nmap: Network Exploration Made Easy!
• The “Network Mapper” nmap explores a network or a host, and looks for “interesting ports”– Based on how a host responds to edge cases in the IP
specification (among other related information), nmap can guess the operating system, which ports are open, and which services are offered
• nmap is useful for examining your own computer– And other people’s computers…
Outline
Cloud Computing Overview
Amazon AWS Services
A Very Brief HTTP Overview
Networking Tools
JSON Format
JavaScript Object Notation
• Used very commonly across the web because of its simplicity!
• Lots of REST APIs return JSON on GET requests– and expect it for POST and PUT requests…
Why JavaScript?
• This format is literally the same format used by JavaScript to represent objects
• It’s very compact and easily readable• Also has the benefit that it can be directly copied and
pasted into a javascript interpreter!
Return of the Dictionary
• JSON Objects are just dictionaries, really…• And they look like dictionaries, too!• Because they are just dictionaries at heart, they can be
interpreted easily by many languages and stored natively• Java, Python, and Ruby all have JSON interpreters, for
example• Some open-source bash projects as well!
– http://kmkeen.com/jshon/ parses JSON into native bash arrays
JSON - Example
GET /users?id=6504 HTTP/1.1Host: www.my-api.com
200 OK{ “name” : “Spencer Lee”, “id” : 6504, “interests” : { “programming”: [“Python”, “Bash”,”JavaScript”], “animals” : [“cats”,”dogs”,”meercats”], “Pet” : “Mochi” } …..}
Example of a useful REST-ful API
• PARSE! • A no-sql datastore that is easy to use, with a full REST API• Learn more at https://parse.com/docs/rest• May be useful for your projects!