introducing the app engine datastore
DESCRIPTION
Describes the App Engine datastore. Explains how indexing and queries work.TRANSCRIPT
![Page 1: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/1.jpg)
Thursday, May 26, 2011
![Page 2: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/2.jpg)
2
Hands on with the App Engine Datastore
Ikai LanMay 9th, 2011
Thursday, May 26, 2011
![Page 3: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/3.jpg)
About the speaker
• Ikai Lan - Developer Programs Engineer, Developer Relations• Twitter: @ikai• Google Profile: http://profiles.google.com/ikai.lan
3
Thursday, May 26, 2011
![Page 4: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/4.jpg)
Lab prerequisites
• JDK 1.5+• Apache Ant• Codelab package: http://code.google.com/p/2011-datastore-
bootcamp-codelab/downloads/detail?name=2011-datastore-bootcamp-codelab.zip
Shortlink: http://tinyurl.com/datastore-bootcamp
4
Thursday, May 26, 2011
![Page 5: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/5.jpg)
Goals of this talk
• Understand a bit of how the datastore works underneath the hood
• Have a conceptual background for the persistence codelab
5
Thursday, May 26, 2011
![Page 6: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/6.jpg)
Understanding the datastore
• The underlying Bigtable• Indexing and queries• Complex queries• Entity groups• Underlying infrastructure
6
Thursday, May 26, 2011
![Page 7: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/7.jpg)
Datastore layers
7
Complex queries
Entity Group Transactions
Queries on properties
Key range scan
Get and set by key
Datastore ✓ ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓ ✓Bigtable ✓ ✓
Thursday, May 26, 2011
![Page 8: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/8.jpg)
Datastore layers
8
Complex queries
Entity Group Transactions
Queries on properties
Get and set by key, key range scans
Datastore ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓Bigtable ✓
Complex queries
Entity Group Transactions
Queries on properties
Key range scan
Get and set by key
Datastore ✓ ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓ ✓Bigtable ✓ ✓
Thursday, May 26, 2011
![Page 9: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/9.jpg)
What does a Bigtable row look like?
9
Source: http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/bigtable-osdi06.pdf
Thursday, May 26, 2011
![Page 10: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/10.jpg)
Bigtable API
• “Give me the column ‘name’ at key 123”• “Set the column ‘name’ at key 123 to ‘ikai’”• “Give me all columns where the key is greater than 100 and less
than 200”
10
Thursday, May 26, 2011
![Page 11: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/11.jpg)
Datastore layers
11
Complex queries
Entity Group Transactions
Queries on properties
Get and set by key, key range scans
Datastore ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓Bigtable ✓
Complex queries
Entity Group Transactions
Queries on properties
Key range scan
Get and set by key
Datastore ✓ ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓ ✓Bigtable ✓ ✓
Thursday, May 26, 2011
![Page 12: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/12.jpg)
Megastore API
• “Give me all rows where the column ‘name’ equals ‘ikai’”• “Transactionally write an update to this group of entities”• “Do a cross datacenter write of this data such that reads will be
strongly consistent” (High Replication Datastore)• Megastore paper: http://www.cidrdb.org/cidr2011/Papers/
CIDR11_Paper32.pdf
12
Thursday, May 26, 2011
![Page 13: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/13.jpg)
Datastore layers
13
Complex queries
Entity Group Transactions
Queries on properties
Get and set by key, key range scans
Datastore ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓Bigtable ✓
Complex queries
Entity Group Transactions
Queries on properties
Key range scan
Get and set by key
Datastore ✓ ✓ ✓ ✓ ✓Megastore ✓ ✓ ✓ ✓Bigtable ✓ ✓
Thursday, May 26, 2011
![Page 14: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/14.jpg)
App Engine Datastore API
• “Give me all Users for my app where the name equals ‘ikai’, company equals ‘Google’, and sort them by the ‘awesome’ column, descending”
14
Thursday, May 26, 2011
![Page 15: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/15.jpg)
Thursday, May 26, 2011
![Page 16: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/16.jpg)
Queries
Thursday, May 26, 2011
![Page 17: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/17.jpg)
Let’s save an Entity with the low-level Java API DatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
16
Thursday, May 26, 2011
![Page 18: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/18.jpg)
Get an instance of the DatastoreServiceDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
17
Fetch a client instance
Thursday, May 26, 2011
![Page 19: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/19.jpg)
Instantiate a new EntityDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
18
Set the Entity Kind
Thursday, May 26, 2011
![Page 20: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/20.jpg)
Instantiate a new EntityDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
19
Set a unique key
Thursday, May 26, 2011
![Page 21: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/21.jpg)
Set indexed propertiesDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
20
First argument is the property name
Second argument is the property value
Thursday, May 26, 2011
![Page 22: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/22.jpg)
Set unindexed propertiesDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
21
This property will be saved, but we will not run queries against it
Thursday, May 26, 2011
![Page 23: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/23.jpg)
Commit the entity to the datastoreDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]");
ikai.setProperty("firstName", "ikai"); ikai.setProperty("company", "google");
ikai.setUnindexedProperty("biography", "Ikai is a great man, a great, great man."); datastore.put(ikai);
22
Save the thing!
Thursday, May 26, 2011
![Page 24: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/24.jpg)
What happens when we save?
23
Write the entity
Write the indexes
Make the write RPC Success!
Thursday, May 26, 2011
![Page 25: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/25.jpg)
What actually gets written?
24
Bigtable key Value
AppId:User:[email protected] ( Protobuf serialized entity - includes firstName, company and biography values )
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key Value
AppId:User:firstName:ikai:[email protected] ( Empty )
AppId:User:company:google:[email protected] ( Empty )
Entities table
Indexes table
Thursday, May 26, 2011
![Page 26: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/26.jpg)
Now let’s run a query
• If we have the key, we can fetch it right away by key• What if we don’t? We need indexes.
25
Thursday, May 26, 2011
![Page 27: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/27.jpg)
Let’s run a queryDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Query queryByName = new Query("User");
queryByName.addFilter("firstName", FilterOperator.EQUAL, "ikai");
List<Entity> results = datastore.prepare( queryByName).asList( FetchOptions.Builder.withDefaults());
// Roughly equivalent to: // SELECT * from User WHERE firstname = ‘ikai’;
26
Thursday, May 26, 2011
![Page 28: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/28.jpg)
Step 1: Query the indexes table
27
Bigtable key Value
AppId:User:[email protected] ( Protobuf serialized entity - includes firstName, company and biography values )
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key Value
AppId:User:firstName:ikai:[email protected] ( Empty )
AppId:User:company:google:[email protected] ( Empty )
Entities table
Indexes table
Scan the indexes table for values >= AppId:User:firstName:
Thursday, May 26, 2011
![Page 29: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/29.jpg)
Step 2: Start extracting keys
28
Bigtable key Value
AppId:User:[email protected] ( Protobuf serialized entity - includes firstName, company and biography values )
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key Value
AppId:User:firstName:ikai:[email protected] ( Empty )
AppId:User:company:google:[email protected] ( Empty )
Entities table
Indexes table
That gets us this row - extract the key [email protected]
Thursday, May 26, 2011
![Page 30: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/30.jpg)
Step 3: Batch get the entities themselves
29
Bigtable key Value
AppId:User:[email protected] ( Protobuf serialized entity - includes firstName, company and biography values )
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key Value
AppId:User:firstName:ikai:[email protected] ( Empty )
AppId:User:company:google:[email protected] ( Empty )
Entities table
Indexes tableNow let’s go back to the entities table and fetch that key. Success!
Thursday, May 26, 2011
![Page 31: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/31.jpg)
Key takeaways
• This isn’t a relational database– There are no full table scans– Indexes MUST exist for every property we want to query– Natively, we can only query on matches or startsWith queries– Don’t index what we never need to query on
• Get by key = one step. Query on property value = 2 steps
30
Thursday, May 26, 2011
![Page 32: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/32.jpg)
Let’s run a more complex query!DatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Query queryByName = new Query("User");
queryByName.addFilter("firstName", FilterOperator.EQUAL, "ikai");
queryByName.addFilter("company", FilterOperator.EQUAL, "google");
List<Entity> results = datastore.prepare( queryByName).asList( FetchOptions.Builder.withDefaults());
// Roughly equivalent to: // SELECT * from User WHERE firstname = ‘ikai’ // AND company = ‘google’;
31
Thursday, May 26, 2011
![Page 33: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/33.jpg)
Query resolution strategies
• This query can be resolved using built in indexes– Zig zag merge join - we’ll cover this example
• Can be optimized using composite indexes
32
Thursday, May 26, 2011
![Page 34: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/34.jpg)
Zig zag across multiple indexes
33
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Begin by scanning indexes >= AppId:User:company:google
Thursday, May 26, 2011
![Page 35: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/35.jpg)
Zig zag across multiple indexes
34
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
There’s at least a partial match, so we “jump” to the next index
Thursday, May 26, 2011
![Page 36: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/36.jpg)
Zig zag across multiple indexes
35
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected] to the next index. Start a scan for keys >= AppId:User:firstName:ikai:[email protected]
Thursday, May 26, 2011
![Page 37: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/37.jpg)
Zig zag across multiple indexes
36
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected], so that’s a twist. The first value that matches has key [email protected]! Does this value exist in the first index?
Thursday, May 26, 2011
![Page 38: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/38.jpg)
Zig zag across multiple indexes
37
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Let’s advance the original cursor to >= AppId:User:company:google:[email protected]
Thursday, May 26, 2011
![Page 39: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/39.jpg)
Zig zag across multiple indexes
38
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Alright! We found a match. Let’s add the key to our in memory list and go back to the first index
Thursday, May 26, 2011
![Page 40: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/40.jpg)
Zig zag across multiple indexes
39
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Let’s move on to see if there are any more matches. Let’s start at [email protected]
Thursday, May 26, 2011
![Page 41: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/41.jpg)
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Zig zag across multiple indexes
40
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Are there any keys >= AppId:User:firstName:ikai:[email protected]?
Thursday, May 26, 2011
![Page 42: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/42.jpg)
Zig zag across multiple indexes
41
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
No. We’re at the end of our index scans. Let’s do a batch key of our list of keys: [ ‘[email protected]’ ]
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Thursday, May 26, 2011
![Page 43: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/43.jpg)
Batch get the entities themselves
42
Bigtable key Value
AppId:User:[email protected] ( Protobuf serialized entity - includes firstName, company and biography values )
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Entities table
Now let’s go back to the entities table and fetch that key. Success!
Thursday, May 26, 2011
![Page 44: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/44.jpg)
Let’s change the shape of the data
• Zig zag performance is HIGHLY dependent on the shape of the data
• Let’s go ahead and muck with the data a bit
43
Thursday, May 26, 2011
![Page 45: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/45.jpg)
Same query, sparsely distributed matches
44
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Thursday, May 26, 2011
![Page 46: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/46.jpg)
Same query, sparsely distributed matches
45
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Begin by scanning indexes >= AppId:User:company:google
Thursday, May 26, 2011
![Page 47: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/47.jpg)
Same query, sparsely distributed matches
46
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Move to the next index. Start a scan for keys >= AppId:User:firstName:ikai:[email protected]
Thursday, May 26, 2011
![Page 48: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/48.jpg)
Same query, sparsely distributed matches
47
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Oh ... no matches. Let’s move back to the first index and move the cursor down
Thursday, May 26, 2011
![Page 49: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/49.jpg)
Same query, sparsely distributed matches
48
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Okay, we’ve got another Googler
Thursday, May 26, 2011
![Page 50: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/50.jpg)
Same query, sparsely distributed matches
49
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Move to the next index. Start a scan for keys >= AppId:User:firstName:ikai:[email protected]
Thursday, May 26, 2011
![Page 51: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/51.jpg)
Same query, sparsely distributed matches
50
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Oh ... no matches here either. Let’s go back to the first index.
Thursday, May 26, 2011
![Page 52: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/52.jpg)
Same query, sparsely distributed matches
51
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:firstName:alfred:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:igor:[email protected]
AppId:User:firstName:ikai:[email protected]
AppId:User:firstName:zed:[email protected]
Bigtable key
AppId:User:company:acme:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:google:[email protected]
AppId:User:company:megacorp:[email protected]
Oh ... no matches here either. Let’s go back to the first index.
... if these indexes were huge, we could be here for a while!
Thursday, May 26, 2011
![Page 53: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/53.jpg)
What happens in this case?
• If we traverse too many indexes, the datastore throws a NeedIndexException
• We’ll want to build a composite index
52
Thursday, May 26, 2011
![Page 54: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/54.jpg)
Composite index
53
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:company:acme:firstName:alfred:[email protected]
AppId:User:company:google:firstName:david:[email protected]
AppId:User:company:google:firstName:ikai:[email protected]
AppId:User:company:google:firstName:max:[email protected]
AppId:User:company:megacorp:firstName:zed:[email protected]
Thursday, May 26, 2011
![Page 55: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/55.jpg)
Composite index
54
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:company:acme:firstName:alfred:[email protected]
AppId:User:company:google:firstName:david:[email protected]
AppId:User:company:google:firstName:ikai:[email protected]
AppId:User:company:google:firstName:max:[email protected]
AppId:User:company:megacorp:firstName:zed:[email protected]
Search for all keys >= AppId:User:company:google:firstName:ikai
Thursday, May 26, 2011
![Page 56: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/56.jpg)
Composite index
55
Read more: http://code.google.com/appengine/articles/storage_breakdown.html
Bigtable key
AppId:User:company:acme:firstName:alfred:[email protected]
AppId:User:company:google:firstName:david:[email protected]
AppId:User:company:google:firstName:ikai:[email protected]
AppId:User:company:google:firstName:max:[email protected]
AppId:User:company:megacorp:firstName:zed:[email protected]
Well, that was much faster, wasn’t it?
Thursday, May 26, 2011
![Page 57: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/57.jpg)
Composite index tradeoffs
• Created at entity save time - incurs additional datastore CPU and storage quota
• You can only create 200 composite index• You need to know the possible queries ahead of time!
56
Thursday, May 26, 2011
![Page 58: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/58.jpg)
Complex Queries takeaways
• This isn’t a relational database – There are no full table scans– Indexes MUST exist for every property we want to query
• Performance depends on the shape of the data• Worse case scenario: if your query matches are highly sparse• Build composite indexes when you need them
57
Thursday, May 26, 2011
![Page 59: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/59.jpg)
Thursday, May 26, 2011
![Page 60: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/60.jpg)
Entity Groups
Thursday, May 26, 2011
![Page 61: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/61.jpg)
Why entity groups?
• We can perform transactions within this group - but not outside• Data locality - data are stored “near” each other• Strongly consistent queries when using High Replication
datastore within this entity group
59
Thursday, May 26, 2011
![Page 62: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/62.jpg)
Entity groups and transactions
• A hierarchical structuring of your data into Megastore’s unit of atomicity
• Allows for transactional behavior - but only within a single entity group
• Key unit of consistency when using High Replication datastore
60
Thursday, May 26, 2011
![Page 63: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/63.jpg)
Example: Data for a blog hosting service
61
Comment
Blog
Entry
User
Has manyHas many
Has many
Thursday, May 26, 2011
![Page 64: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/64.jpg)
Example: Data for a blog hosting service
62
Comment
Blog
Entry
User
Has manyHas many
Has many
This can be structured as an entity group (tree structure)!
Thursday, May 26, 2011
![Page 65: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/65.jpg)
Structure this data as an entity group
63
Blog
Entry
User
Blog
Entry Entry
CommentCommentComment
Entity group root
Thursday, May 26, 2011
![Page 66: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/66.jpg)
How are entity groups stored?
64
Bigtable key ValueAppId:User:[email protected] ( Protobuf serialized User )
AppId:User:[email protected]/Blog:123 ( Protobuf serialized Blog )
AppId:User:[email protected]/Blog:123/Entry:456 ( Protobuf serialized Entry )
AppId:User:[email protected]/Blog:123/Entry:789 ( Protobuf serialized Entry )
AppId:User:[email protected]/Blog:123/Entry:456/Comment:111
( Protobuf serialized Comment )
AppId:User:[email protected]/Blog:123/Entry:456/Comment:222
( Protobuf serialized Comment )
AppId:User:[email protected]/Blog:123/Entry:789/Comment:333
( Protobuf serialized Comment )
Read more: http://code.google.com/appengine/docs/python/datastore/entities.html
Entities table
Thursday, May 26, 2011
![Page 67: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/67.jpg)
How are entity groups stored?
65
Bigtable key ValueAppId:User:[email protected] ( Protobuf serialized User )
AppId:User:[email protected]/Blog:123 ( Protobuf serialized Blog )
AppId:User:[email protected]/Blog:123/Entry:456 ( Protobuf serialized Entry )
AppId:User:[email protected]/Blog:123/Entry:789 ( Protobuf serialized Entry )
AppId:User:[email protected]/Blog:123/Entry:456/Comment:111
( Protobuf serialized Comment )
AppId:User:[email protected]/Blog:123/Entry:456/Comment:222
( Protobuf serialized Comment )
AppId:User:[email protected]/Blog:123/Entry:789/Comment:333
( Protobuf serialized Comment )
Read more: http://code.google.com/appengine/docs/python/datastore/entities.html
Entities table Entity groups have a single root entity
Thursday, May 26, 2011
![Page 68: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/68.jpg)
How are entity groups stored?
66
Bigtable key ValueAppId:User:[email protected] ( Protobuf serialized User )
AppId:User:[email protected]/Blog:123 ( Protobuf serialized Blog )
AppId:User:[email protected]/Blog:123/Entry:456 ( Protobuf serialized Entry )
AppId:User:[email protected]/Blog:123/Entry:789 ( Protobuf serialized Entry )
AppId:User:[email protected]/Blog:123/Entry:456/Comment:111
( Protobuf serialized Comment )
AppId:User:[email protected]/Blog:123/Entry:456/Comment:222
( Protobuf serialized Comment )
AppId:User:[email protected]/Blog:123/Entry:789/Comment:333
( Protobuf serialized Comment )
Read more: http://code.google.com/appengine/docs/python/datastore/entities.html
Entities table
Child entities embed the entire ancestry in their keys
Thursday, May 26, 2011
![Page 69: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/69.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
67
Thursday, May 26, 2011
![Page 70: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/70.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
68
Create the root entity
Thursday, May 26, 2011
![Page 71: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/71.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
69
This is the first child entity - notice the third argument, which specifies the parent entity key
Thursday, May 26, 2011
![Page 72: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/72.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
70
The next deeper entity sets the blog as the parent
Thursday, May 26, 2011
![Page 73: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/73.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
71
We can also opt to not provide a key name and just use a parent key for a new entity
Thursday, May 26, 2011
![Page 74: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/74.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
72
Start a new transaction
Thursday, May 26, 2011
![Page 75: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/75.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
73
Put the entities in parallel
Thursday, May 26, 2011
![Page 76: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/76.jpg)
Let’s write an entity group transactionallyDatastoreService datastore = DatastoreServiceFactory
.getDatastoreService();
Entity ikai = new Entity("User", "[email protected]"); Entity blog = new Entity("Blog", "ikaisays.com", ikai.getKey()); Entity entry = new Entity("Entry", "datastore-intro", blog.getKey()); // Auto assign an ID Entity comment = new Entity("Comment", entry.getKey()); Transaction tx = datastore.beginTransaction(); // Helper function for clarity datastore.put(Arrays.asList(ikai, blog,entry, comment)); tx.commit();
74
Actually commit the changes
Thursday, May 26, 2011
![Page 77: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/77.jpg)
Step 1: Commit
75
Commit Changes to entities visible
Changes to entities and indexes visible
Roll the timestamp forward on the root entity
Thursday, May 26, 2011
![Page 78: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/78.jpg)
Step 2: Entity visible
76
Commit Changes to entities visible
Changes to entities and indexes visible
On read, check for the most recent timestamp on the root entity
This is the version we want since it represents a complete write
Thursday, May 26, 2011
![Page 79: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/79.jpg)
Step 3: Indexes updated
77
Commit Changes to entities visible
Changes to entities and indexes visible
Indexes are written - now we can query for this entity with the new properties
Thursday, May 26, 2011
![Page 80: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/80.jpg)
Entity group and transactions takeaways
• Structure data into hierarchical trees– Large enough to be useful, small enough to maximize
transactional throughput
• Transactions need an entity group root - roughly 1 transaction/second– If you write N entities that are all part of 1 entity group, it counts as
1 write
• Optimistic locking used - can be expensive with a lot of contention
78
Thursday, May 26, 2011
![Page 81: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/81.jpg)
General datastore tips
• Denormalize as much as possible– As much as possible, treat datastore as a key-value store
(Dictionary or Map like structure)– Move large reporting to offline processing. This lets you avoid
unnecessary indexes
• Use entity groups for your data• Build composite indexes where you need them - “need” depends
on shape of your data
79
Thursday, May 26, 2011
![Page 82: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/82.jpg)
Thursday, May 26, 2011
![Page 83: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/83.jpg)
Questions?
Thursday, May 26, 2011
![Page 84: Introducing the App Engine datastore](https://reader034.vdocuments.site/reader034/viewer/2022042623/54572a67af795994188b4fda/html5/thumbnails/84.jpg)
Thursday, May 26, 2011