introduction to google appengine development in java philippe beaudoin (track sponsor)

Introduction to Google AppEngine

Development in Java

Philippe Beaudoin

(Track Sponsor)

Overview

What is Google AppEngineHit the ground running: Your first GAE appThe datastoreTask QueuesOverview of other services

?

What is Google AppEngine

Fly me to the cloud, baby!Fly me to the cloud, baby!

PaaSPaaS(Platform as a Service)(Platform as a Service)

?

?

• Platform features– Not Infrastructure as a Service (i.e. Amazon AWS)– Sandboxed– NoSQL distributed datastore + memcache– Many services (task queues, image manipulation…)

• APIs for…– Java (And other JVM languages)– Python– Go (beta)

?

• Drawbacks– Not all of Java is available (sandbox)– Only communicate via HTTP/HTTPS + email– Read-only file system– Must finish processing within 30 seconds• Except for new backends

• Advantages– It will scale!– Pay only for what you need– Super-easy and fast deployment

?

Billing model• Free!– 500Mb storage– 5M page views/month

• Billing enabled

?

Admin console

?

Hit the ground running:Your first GAE app

?

?

?

?

?

The datastore

• NoSQL (Roughly: a fancy hashmap)• Optimistic concurrency– No locking: try first, fail later if things go wrong

• Multitenancy• Memcache (Use it!)• Default Java API: JDO or JPA– Get cover!

• Much better: Objectify (OSS project) ?

?

?

Transactions

• Each entity has an entity group• By default, each entity is in a group of its own• Transactions are restricted to one entity group

By default, transactions are restricted to a

single entity!

?

Everything in one Entity Group?

• Bad idea!– AppEngine limits the number of requests per

second on a given entity group.– Contention: every request wants the same group

• So design your entity groups carefully– The default (each entity in its group) often works

?

Entity groups design

• Decide them when designing your data model– Address and phone have coupled transactions?

Same group!

• Grouping is a bit weird: choose one entity as the parent of the other.– Not necessarily a “real” parent-child relationship– No “cascading deletes”

?

The components

?

Persistable Entities

• Basic Java types (String, int, etc.)• Date• Blob (binary)• Email, GeoPt, PhoneNumber, etc.• Key<AnotherEntity>• List<>, Set<>, or arrays of the above• @Embedded AnyPersistableEntity• @Serialized AnySerializableEntity

?

?

Accessing entitiesput(), get(), query(), delete()

?

Optimization

• Get/put/delete multiple keys at once• Asynchronous fetches for concurrent queries• Fetch only keys when needed

?

Using the memcache

• Simply annotate the entity with @Cached!

• Optimized for read-intensive apps– Write-through

• Caches negative results to• Caches get(), put() and delete(), not query().• Can go out-of-sync in some situations– Not for entities requiring rigorous data integrity

?

Useful trick: counters

• Remember: AppEngine limits the number of requests per second on a given entity group.

• A counter many “read-increment-write”!• The trick: a sharded counter– Create many counters (not @Cached!)– Pick one randomly, increment it– At the end, collect them all and sum them up

• Why it works? AppEngine has very fast read!• Same idea: sharded lists

?

Queries

• NoSQL: only allows efficient queries– No table scans– No in-memory sorts– No joins (almost)

• Queries only allowed on indexed properties• Allow “merge-join” queries:

Horn

4 legsSpotted

Cows!

?

Indexes

• List of entities sorted by a given field– Allows fast dichotomic search

• By default all fields are indexed– @Indexed, @Unindexed control that– Use on fields or classes

• Manual indexes– Specify more than one field to sort• Ex: Sort by gender then by age

– Needed for some complex queries

?

Complex queries

• income > 100000 and age < 30– Not allowed! Inequalities on two fields

• gender = f and age > 18 and age < 25– Allowed! Two inequalities on the same field– Needs a manual index (equality + inequality)• Sort by gender, then by age

• all, sorted by decreasing age– Allowed, needs a manual index (reverse sort)

?

Optimization

• Index only the fields you need• Index only the values you need!– @Unindexed(IfFalse.class) boolean admin;

• Try to limit manual indexes• Avoid manual indexes on many List<>

fields!• Exploding indexes!

?

Useful trick: full text search

• Full text search not available out-of-the-box• Build a KeywordIndex table– KeywordIndex entity is a [keyword, entityId] pair– Normalize keywords (uppercase, no diacritics…)

• When searching for a prefix:– ofy().query(Customer.class).filter("keyword >=",

prefix).filter("keyword <=", prefix + "\ufffd").list();

?

Task queues

• Work outside of a user request– Still need to be initiated by a user request

• Organize works in small discrete tasks– Again: split it up to make it scalable

• 10 minutes deadline (instead of 30s)• Examples:– email notifications– schema migration

?

Push queue vs Pull queue

• Push queue– Meant to be consumed by your AppEngine app– Dispatching managed by AppEngine

• Pull queue new!

– Meant to be consumed externally (REST)• or on an AppEngine backend

– Consume the tasks when you’re ready– Your task consumer must scale!

?

Configuring a Pull QueueIn queue.xml

<bucket-size>

<queue>

<max-concurrent-requests>

<rate>

?

Enqueing a task

Or: getQueue(name)

?

Mapper API

• The “map” part of “mapreduce”• Run a task on all the entities of a given type in

the datastore• Just configure the MapReduceServlet

?

Configure a new mapper

?

The mapper class

?

Launching the mapper

?

Overview of other services

• URL Fetch• Mail / XMPP (Google Talk)• OAuth, OpenID, Google Accounts• Image manipulation• Channel API• Blobstore

?

URL Fetch

• Access other web resources (REST Apis, etc.)– Only http/https– Maximum deadline of 10 seconds– Synchronous or asynchronous

• Use standard java.net.URLConnection• Low-level AppEngine API has more features

?

Channel API

• Enables COMET on AppEngine– Only for javascript clients

?

Channel API

?

Blobstore

• Objects up to 2 Gb• Useful for videos or large images• User use a form to upload to the blobstore• Can serve back the blob later

?

Conclusion

• Platform-as-a-Service– Learn the platform– Constraints are often opportunities– Work with it, not against

• Still in active development– No https on custom domain

• A great way to learn what is needed to build a scalable app.– Try your hand at the free version!

Hands on?

Tomorrow, 11am, CamstasiaGoogle Web Toolkit and the Model View Google Web Toolkit and the Model View

Presenter Architecture Presenter Architecture

Questions?philippe.beaudoin@gmail.com

@philbeaudoinhttp://arcbees.com