designing the call of cthulhu app with google app engine
DESCRIPTION
These are slides from a talk I gave at UCSB to the Senior Capstone class on 02/10/10 on how I developed the Call of Cthulhu application using Google App Engine.TRANSCRIPT
Designing the Call of Cthulhu app with
Google App EnginePresented by Chris Bunch at UCSB
CS 189A / 172 : Capstone Project / Software EngineeringFebruary 10, 2010
http://cs.ucsb.edu/~cgb
Motivation
•I am what you would call a “nerd”
•Plays nerdy board games
•Would like to upload stats from games and analyze them
•Could do other cool things as well
Specifically
Arkham Horror
•Lots of variables that can be analyzed
•Co-op game: all v. the board
•Mix and match expansions
•Many different winning / losing conditions
•But fundamentally boils down to:
•Save the world from total destruction
App Requirements
•Must be able to upload game data
•Should be able to predict a win or loss given some preliminary info
•Can experiment with prediction algorithms
•Should be able to summarize game data
App Requirements•Allows users to log in to upload their
data
•Allows users to share data with each other
•Handles users in a sane fashion
•Harder than it sounds
•Display a top score list for users to view
•Game provides scoring metric
Enter the Cloud
•Why go the cloud route?
•Simple answer: You likely don’t have your own hardware to host this app on, so just use somebody else’s
•But cloud computing is confusing and not well defined yet, so...
Defining Cloud Computing
•Three layers, each building on the last
• Infrastructure:
•Programmer gets a virtual machine and / or reliable storage
•Most control of environment
•Must scale manually
•Amazon EC2, Eucalyptus
Defining Cloud Computing
•Three layers, each building on the last
•Platform:
•Programmer gets a scalable API
•Less control of environment
•Scaling done automatically
•Google App Engine, Microsoft Azure
Defining Cloud Computing
•Three layers, each building on the last
•Software:
•Programmer gets to “skin” an existing app
•No control of environment
•Scaling done automatically
•SalesForce, GMail
My Choice•Clearly I want the platform!
•Platform:
•Scalable API = great!
•Less control of environment = ok
•Programability depends on APIs offered
•Have personal exp. w/ Google App Engine
What is Google App Engine?
•Scalable web framework first offered in 2008
•Users program apps in:
•Python
•Java
•Unofficially: JRuby, Groovy, JavaScript
Sandboxed Environment
•Requests limited to 30 seconds
•Only white-listed libraries can be used
•Anything non-trivial is quota-ed
•No reading / writing to the filesystem
•Web communication over HTTP(S) only
Datastore API
•A giant hash table
•Primitives:
•Get, put, delete - same as regular HT
•Query, count - new operations
•Transactions supported (row-level only)
Datastore API
•GQL queries offered - subset of SQL Select statement, no JOINs
•Tip: Queries are the most expensive operation, use sparingly
•If possible, perform it asynchronously
Memcache API
•Equivalent to memcached
•Provides access to low latency LRU cache
•Roughly 100x faster than Datastore
•But unreliable - remember it’s a cache!
•Exposes get, set, delete operations
•Exposes atomic increment / decrement
URL Fetch API•Equivalent to curl
•Used to grab web content from other sites
•Only allows for HTTP(S) data to be read
•Can also be done asynchronously
•Typically grabs images or mashup data
Mail API
•Equivalent to sendmail
•Can send or receive mail asynchronously
•Can also add attachments
•Not all types allowed - no .doc, .xls, or .ppt in Python SDK
XMPP API
•Equivalent to jabber
•Send and receive instant messages to
•appname.appspotchat.com
•App can’t join group chat yet
•Use as a notifier or alternate UI to app
Images API
•Equivalent to python-imaging
•Exposes image rotation, flipping, cropping, and histogram
•Image is stored internally as a blob, then retrieved and manipulated
Users API
•Equivalent to Google Accounts
•Provides for simple authentication
•Allows app to force login or admin on certain pages
•Also exposes URL to optionally login to
Blobstore API
•Allows for large file uploads (<50 MB)
•Users can upload videos, datasets, or large music files for later retrieval
•Upload must be done via form post
Cron API
•Equivalent to cron
•User specifies how often a web page should be accessed (e.g., daily, every Monday)
•Task is then run in the background
•Useful for running expensive data analysis
Task Queue API
•Spawns a new thread in the background
•But still must visit your app
•Configuration file specifies queues
•Useful for short to medium length tasks
To summarize:
•Many, many APIs available for what we need to do!
•Use Datastore to store / retrieve user data
•Use Users to authenticate users
•Gets around a lot of boilerplate authentication code
To summarize:
•Use Mail to e-mail users if their score is beat
•Use XMPP to allow users to add new data
•Use Cron to run compute-intensive code in the background
To summarize:
•Think about paid functionality:
•Maybe use Blobstore to allow users to upload videos explaining gameplay
•Handle greater traffic if site becomes popular
Let’s Get to Business
•Data Modeling
•Upload Stats Page
•Game Stats Page
•Game Prediction Page
Data Modeling•class Game(db.Model):
•player = db.UserProperty()
•goo = db.StringProperty()
•comments = dbStringProperty(multiline=True)
•date = db.DateTimeProperty(auto_now_add=True)
Data Modeling
•num_of_players = db.IntegerProperty()
•won_game = db.BooleanProperty()
•doom_counters = db.IntegerProperty()
•terror_level = db.IntegerProperty()
•expansions_used = dbListProperty(str)
Uploading Stats• class AddGames(webapp.RequestHandler):
• def get(self):
• user = users.get_current_user()
• if user: # means they’re logged in
• # write html form for getting data
• else:
• self.redirect(users.create_login_url(self.request.uri))
Uploading Stats•def post(self):
•new_goo = self.request.get(‘goo’)
•# validate goo, make sure data type is right
•game = Game()
•game.goo = new_goo
•game.put()
Game Stats
•games = db.GqlQuery (“SELECT * FROM Game ORDER BY date desc”)
•for game in games:
•# find the goo w/ highest player win percentage
•# can also just display the info here
Game Prediction
•def get(self):
•# user specifies the game they will play
•def post(self):
•# calculate how likely they are to win
•# right now we only care about the goo
Routing
•application = webapp.WSGIApplication([
•(‘/upload’, AddGames),
•(‘/stats’, GameStats),
•(‘/predict’, PredictGame)])
Thankfully•Most requirements have very little
state shared between URLs (all saved in DB)
•Thus each class is one of two forms:
•get / post: Grab data from user and do something with it
•get: Just display or synthesize some data
Rendering Web Pages
•Two main options available:
•self.response.out.write(“Hello World!”)
•self.response.out.write(template.render(‘index.html’, {‘my_title: ‘baz’, ‘data’: some_variable}))
•Use the second for any non-trivial app
Rendering Web Pages
•Now in your index.html:
•<title> {{ my_title }} </title>
•<div> {{ data }} </div>
Tips for Your Programs
•Separate out your code into files as needed
•app.yaml specifies this:
•handlers:
• - url: /one
• script: one.py
• - url: /two/(.*)/(.*)
• script: /two/\2/\1.py
Tips for Your Programs
•Make pages with sensitive data ‘secure’
•All access done over HTTPS
•Optional - HTTP and HTTPS work
•Never - HTTPS redirects to HTTP
•Always - HTTP redirects to HTTPS
Tips for Your Programs
•Specify static files in app.yaml
•- url: /images
• static_dir: static/images
•Can also tag ‘secure’ if desired
Tips for Your Programs
•Be creative!
•The webapp framework supports:
•get, post, head, options, put, delete, trace
•Map user pages to ~name?
Summary
•Google App Engine is more than flexible enough to handle your web app desires
•And it’s only getting better!
•More than enough possibilities to create a web app: you just need a good idea!