app engine testing techniques

8/2/2019 App Engine Testing Techniques

1/44

Tuesday, May 25, 2010


2/44



3/44

Testing Techniques ForGoogle App EngineMax RossMay 20, 2010



4/44



5/44

http://bit.ly/GAETesting

View live notes and ask questions about thissession on Google Wave

http://bit.ly/9XLCAshttp://bit.ly/9XLCAs


6/44

Who Am I?

Max!

Software Engineer at Google

Member of the App Engine team

Datastore

JPA/JDO Implementation

Java Runtime

http://gae-java-persistence.blogspot.com/

I feel naked and sad without good test coverage

4

http://gae-java-persistence.blogspot.com/http://gae-java-persistence.blogspot.com/http://gae-java-persistence.blogspot.com/


7/44

Agenda

Developer Testing

Testing App Engine Apps Locally

Testing App Engine Apps In The Cloud

Wrap Up

Questions

5



8/44Tuesday, May 25, 2010


9/44


10/44

Terminology

Unit test

tests a single method/class

behavior of dependencies controlled by the test

Integration test

tests interaction between multiple methods, classes, orcomponents

behavior of dependencies mightbe controlled by the test

End to end test

tests behavior by driving requests through external accesspoints

for App Engine that means HTTP

7



11/44

Soapbox: Why Developer Testing Is Important

Demonstrates correctness of new code

important

Demonstrates continued correctness of old code

more important

Provides a safety net when making large or risky changes

code always evolves organically

refactoring can help restore order

Treat your tests as first class citizens in your code base

please Deploying to App Engine doesnt change any of this!

8



12/44

Testing @ Google

part of our engineering culture

guiceberry

googletest

googlemock

thread-weaver

googletesting.blogspot.com

9



13/44

App Engine Execution Environments

Cloud

serving HTTP traffic

does not have to be prod

Local

serving HTTP traffic

equivalent functionality

nonequivalent scale

Tests!

serving HTTP traffic and test harness requests local and cloud

do we need App Engine specific testing strategies?

10



14/44Tuesday, May 25, 2010


15/44

Testing App Engine Apps

Locally



16/44

Local Testing - Whats The Same?

Frameworks

JUnit, pyUnit, Selenium

Best practices

design

avoiding flakiness

isolation

End to end testing

This is great news!

12



17/44

Local Testing - Whats Different?

RPC layer assumes dev appserver sets up the environment

What if there is no dev appserver?

set up the environment by hand or... use the App Engine Testing APIs!

currently Java only

13

// from com.google.appengine.api.datastore.DatastoreApiHelper.java

static String getCurrentAppId() {

ApiProxy.Environment environment = ApiProxy.getCurrentEnvironment();

if (environment == null) {

throw new NullPointerException(

"No API environment is registered for this thread.");}

return environment.getAppId();

}



18/44

Local Testing - Demo!

Guestbook Persistence

Integration test with the datastore and task queue

high(er)-availability writes

14



19/44Tuesday, May 25, 2010


20/44

Testing App Engine Apps In

The Cloud



21/44

I Want To Run My Tests In The Cloud!

Why?

Fidelity

true behavior and performance

Efficiency

run large test suites faster

let someone else manage your test grid

16



22/44

Cloud Testing - Whats The Same?

Same similarities as local testing

frameworks

best practices

end to end testing

This is also great news!

17



23/44

Cloud Testing - Whats Different?

Each test must complete in 30 seconds

App code and test code must obey sandbox restrictions

threads, direct network access, etc

Must invoke tests via HTTP

no need to worry about setting up the environment

GAE Cloud Cover can help

18

Tests AppIn Process



24/44








18

Tests

XAppIn Process



25/44








18

T

ests AppCloudCover

CloudCover

UI

HTTP



26/44

Cloud Testing - Simple Demo!

GAE Cloud Cover is designed to run existing test suites

Guestbook Persistence tests running in the cloud:

install JAR

configure Servlet

replace LocalServiceTestHelper

extend JUnit3Config

deploy

run

curl up under a cozy warm blanket of cloud test coverage

19



27/44

How It Works

A Suite is a collection of Tests

GAE Cloud Cover creates one task queue task per Test

Tests execute in parallel

number of workers determined by queue config

GAE is good at fan-out!

20

Task Queue

Worker 1 Worker 2



28/44

How It Works






20

SUITE

Test 1

Test 2

Test 3 Task Queue

Worker 1 Worker 2



29/44

How It Works






20

SUITE

Test 1Test 2Test 3

Task Queue

Worker 1 Worker 2



30/44

How It Works






20

SUITE

Task Queue

Worker 1 Worker 2

Test 2

Test 3

Test 1



31/44

Isolation Problems

Tests must be thread-safe

Test data must be thread-safe

Non thread-safety yields flakiness Flaky tests are worse than consistently failing tests

21

Thread 1: testReadThruCache Thread 2: testQuery

1) populate cache 1) perform query

2) install ErrorDelegate 2) verify results

3) fetch val from cache

Thread 1: testCreateUser Thread 2: testDeleteUser

1) create User m 1) create User m

2) assert m exists 2) delete m

3) assert m does not exist



32/44

Isolation Problems

Tests must be thread-safe

Test data must be thread-safe

Non thread-safety yields flakiness Flaky tests are worse than consistently failing tests

21

Thread 1: testReadThruCache Thread 2: testQuery

1) populate cache 1) perform query

2) install ErrorDelegate 2) verify results

3) fetch val from cache

Thread 1: testCreateUser Thread 2: testDeleteUser

1) create User m 1) create User m

2) assert m exists 2) delete m

3) assert m does not exist



33/44


34/44

Isolation Solutions

Use ThreadsafeDelegate instead of ApiProxy.Delegate

Use Namespaces to enforce test data isolation

coming soon to an App Engine SDK near you

implemented as a way to stripe persistent data with arbitraryidentifiers

this is exactly what we need!

22

Kind Namespace Key

Greeting testCreateUser max

Greeting testDeleteUser max



35/44

Cloud Testing - Fancy Demo!

App Engine JDO/JPA implementation test suite

userland library

1300+ unit and integration tests

makes extensive use of the datastore

uses custom ApiProxy.Delegate implementations

23



36/44

Running Arbitrary Test Suites

Why limit ourselves to test suites for App Engine apps?

GAE Cloud Cover is framework agnostic

requires a small plugin to hook up a framework

com.google.appengine.testing.cloudcover.spi

implement 3 simple interfaces

If you have a large test suite, App Engine can be your grid!

24



37/44

Cloud Testing - Arbitrary Demo!

Google Collections Framework

http://code.google.com/p/google-collections/

44,776 tests

Has nothing to do with App Engine

25

http://code.google.com/p/google-collections/http://code.google.com/p/google-collections/


38/44



39/44

Wrap Up



40/44

Looking Ahead

Testing APIs

Python

running tests inside the local sandbox

launching dev appserver from tests

GAE Cloud Cover:

IDE-integration

support for dynamically generated tests

project is completely open source

http://code.google.com/p/cloudcover/

Help!

27

http://code.google.com/p/cloudcover/http://code.google.com/p/cloudcover/


41/44

Conclusions

Developer testing is Important!

Your testing skillz transfer

App Engine Testing API makes local testing easy

GAE Cloud Cover makes cloud testing easy

You dont have to build an App Engine app to use AppEngine

28



42/44



43/44

Questions

http://bit.ly/GAETesting

http://bit.ly/9XLCAshttp://bit.ly/9XLCAs


44/44

app engine testing techniques

Documents