Lecture IV: REST Web Service with Google App Engine

CS 4593Cloud-Oriented Big Data and Software Engineering

Outline• Course Project• Google App Engine• Google Development Console• Google Custom Search API

2

Course Project• Topic

– Keyword trends analysis services for a certain topic• Platform

– Google App Engine– Any programming language supported by GAE

• Collaboration– Work individually– Hopefully you are not working on the same idea, but it

is fine if you happen work on one by accident

3

Course Project• Requirements

– Develop a web service based on Google Search API to follow the weekly trends of keywords on a certain topic

– Search one or multiple websites each day with Google Search API free quota and record the query results during a period of time (hopefully more than a week)

– Perform statistics on query results to provide daily keywords frequency and the trends of the week

4

Course Project• Requirements

– Provide at least two web services: one for the keyword frequency of each day, and the other for trend of the week

– Provide output as either JSON or XML – The developed web service should be deployed on

Google App Engine

5

Course Project• Advanced Requirements

– You are going to choose the topic and the website to search, but the results should be meaningful (searching a static website for a week will not give you any different results)

– Provide HTML presentation for the web services

6

Course Project• Additional Information

– Feel free to use additional free web services to support your task

– Searching a more volatile website often provides you more interesting results, e.g., searching "Obama" in CNN or NY Times.

7

Course Project• Evaluation

– fetching and recording Google Search API results (2.5 points)

– extracting keywords (2.5 points)– presentation of daily results (2.5 points)– presentation of weekly results (2.5 points)– The advanced features will be evaluated by quality.

8

Course Project• Deliverables

– Demo– Code– Commends and User Guide– Blackboard

• Due date– Demo: Oct. 5th– Code: Oct. 4th

9

Outline• Course Project• Google App Engine• Google Development Console• Google Custom Search API

10

Google App Engine: Overview• Google App Engine (GAE) is a PaaS cloud

computing platform for developing and hosting web applications in Google-managed data centers.

• Google App Engine lets you run web applications on Google's infrastructure.– Easy to build. – Easy to maintain.– Easy to scale as the traffic and storage needs grow

11

infrastructure vs. platform - What is “The Platform”?

Platform: same for all applications

Libraries: shared by multiple applications

Application-specific code

infrastructure: hidden by platform

Scaling

• Low-usage apps: many apps per physical host• High-usage apps: multiple physical hosts per app• Stateless APIs are trivial to replicate• Datastore built on top of Bigtable; designed to

scale well– Abstraction on top of Bigtable– API influenced by scalability

• No joins• Recommendations: denormalize schema; precompute joins

13

Google App Engine: Overview• Free

– 1GB storage– 1 virtual machine instance– 5 million page views / month– 10 applications / Google account

14

Preserving Fairness Through Quotas• Everything an app does is limited by quotas, for

example:– request count, bandwidth used, CPU usage,

datastore call count, disk space used, emails sent• If you run out of quota that particular operation is

blocked (raising an exception) until replenished

15

Preserving Fairness Through Quotas• Free quotas are tuned so that a well-written app

(light CPU/datastore use) can survive • The point of quotas is to be able to support a very

large number of small apps (analogy: baggage limit in air travel)

• Large apps can request raised quotas• Three types of quotas

– Free Quota– Billed Limits– Safety Limits

16

Google App Engine: Programming Languages

• Java– App Engine runs JAVA apps on a JAVA 7 virtual

machine (currently supports JAVA 6 as well).– Uses JAVA Servlet standard for web applications:

• WAR (Web Applications ARchive) directory structure.• Servlet classes• Java Server Pages (JSP)• Static and data files• Deployment descriptor (web.xml)• Other configuration files

17

Google App Engine: Programming Languages

• Python– Uses WSGI (Web Server Gateway Interface)

standard.– Python applications can be written using:

• Webapp2 framework• Django framework• Any python code that uses the CGI (Common Gateway

Interface) standard

18

Google App Engine: Programming Languages

• PHP– Local development servers are available to

anyone for developing and testing local applications.

– Only whitelisted applications can be deployed on Google App Engine. (https://gaeforphp.appspot.com/).

19

Google App Engine: Programming Languages

• Go– Go is an Google’s open source programming

environment.– Tightly coupled with Google App Engine.– Applications can be written using App

Engine’s Go SDK.

20

Google App Engine: Data Storage• App Engine Data Store

– NoSQL schema-less object based data storage, with a query engine and atomic transactions.

– Data object is called a “Entity” that has a kind (~ table name) and a set of

– properties (~ column names).– JAVA JDO/ JPA interfaces and Python datastore

interfaces.

21

Google App Engine: Data Storage• Google Cloud Store

– RESTful service for storing and querying data.– Fast, scalable and highly available solution. – Provides Multiple layers of redundancy. All data is

replicated to multiple data centers.– Provides different levels of access control.– HTTP based APIs.

22

Google App Engine: Services• Google Services

– URL Fetch– Mail– Memcache

23

Google App Engine: Security• Sandboxes

– All hosted applications run in a secure environment that provides limited access to the underlying operating system.

– Sandbox isolates the application in its own secure, reliable environment that is independent of hardware, operating system and physical location of a web server.

24

Google App Engine: Security• Sandboxes: Limitations

– Application can only interact with other computers over internet using URL fetch, email, and HTTP/ HTTPS requests on the standard ports

– Applications cannot write to local file system in any of the runtime environments.

– Application code runs only in response to a web request, a queued task or a scheduled task and must return the response data within 60 seconds.

25

Google App Engine: Demo• Create a project on Google App Engine

– At https://console.developers.google.com/project • Initialize the environment of Restlet + eclipse +

GAE– Tutorial at– http://restlet.com/technical-resources/restlet-framewor

k/guide/2.3/introduction/first-steps/first-application– Download Restlet-GAE, GAE SDK, and Eclipse-GAE

plugin

26

Google App Engine: Demo• Deployment of Applications

– Follow https://console.developers.google.com/start/appengine?_ga=1.67402004.429901446.1441394923

• Google Search API– Generate your search engine– Manage API usages– https://cse.google.com/manage/all

27

Google Map API• Provide Map Services

– Http: http://maps.googleapis.com/maps/api– Android– IOS– JavaScript

• Parameters– Center– Zoom– Size– Language

28

Google Map API• Pricing

– Free Quota: 2500 / day– Bill Quota: 1000 / $0.5– Safety Quota: 100, 000 / day

29

Google Place API• Searching for places around certain map

location• Usage Example

– Finding all food places within 500 miles from (long,lat)

– https://maps.googleapis.com/maps/api/place/nearbysearch/json?location=long,lat&radius=500&types=food&name=cruise&key=API_KEY

• Place ID– A ID of a place that can be used later (e.g., Costco at

I 10 @ UTSA Boulevard)30

Google Place API• Parameters

– Type: food, airport, hospital, …– https://

developers.google.com/places/supported_types• Pricing

– Free quota: 1000/day– Free up to 150,000/day with credit card info

31

{         "geometry" : {            "location" : {               "lat" : -33.870775,               "lng" : 151.199025            }         }, …         "name" : "Rhythmboat Cruises",         "opening_hours" : {            "open_now" : true         },         "photos" : […],         "place_id" : "ChIJyWEHuEmuEmsRm9hTkapTCrk",         "scope" : "GOOGLE",         "types" : [ "travel_agency", "restaurant", "food", "establishment" ],         "vicinity" : "Pyrmont Bay Wharf Darling Dr, Sydney" },

Google Place API: response

Google+ API• Intro• Usages• Parameters• Pricing

33

Google+ API• A series of APIs working with Google+

– People: Get public profiles with query or user ID– Activities: List activities of a user or searching

activities with query– Comments: List comments of an activity or searching

comments with query– Moments: Retrieve, insert or delete moments

• Usage– https://www.googleapis.com/plus/v1/– 10,000/day free quota

34

Google Translation API• Translate strings to other languages• Parameters

– source– target– q(query)

• Pricing– $20 for 1M characters– 50M / day limit

35

Youtube Data API• Provide various supports to retrieve and manage

data from Youtube• Some important services

– Video: List, search, Insert, Delete, Rate, …– Playlist: Insert, delete, List, …– Captions: List, insert, update, download, delete, …

36

Youtube Data API• Parameters

– Part– videoCategoryId– myRating

• Pricing– Unit based quota calculation– Services with different parameters may cost different

quota– https://developers.google.com/youtube/v3/

determine_quota_cost– 50M unit / day, and 30K unit / second, always free

37

Facebook Graph API• A web API library for Facebook social network• Provide support for various platforms

– RESTful– IOS– Android– JavaScript– PHP

38

Facebook Graph API• Usages

– Provide similar functions to google+ APIs– List friends, List events, Fetch comments, …– Uploading videos, photos, …

• Pricing– Free– Need to contact Facebook for more than 100M API

calls per day

39

MediaWiki API• API to get information from Wikipedia

programmatically– Entry:  https://en.wikipedia.org/w/api.php

• Parameters– Action: query, modules– Titles: specify the query– Limit: max results– Section: retrieve from which section

• Pricing– Free: with unofficial limits

40

Bing API• Search API from Microsoft• Root:

– https://api.datamarket.azure.com/Bing/Search/v1/• Parameters

– Query– NewsCategory– Sources: web, image, video, news, …

41

Bing API• Pricing

– 5000 / month free– After it: 10K/month for $20

42

Summary• Google App Engine• Application Deployment on Google App Engine• Popular free Web Services

43