Test Driven Elephants: ETL Validation with Cascading on Elastic MapReduce

The problem

Difficulty with TDD on Hadoop:

· Slow feedback cycle

· End-to-end smoke tests only

· Requires Hadoop install in your CI environment

Cascading’s solutionOrganize map/reduce jobs into flows, connecting:

· taps: source and sink - where your data is coming from, where it is going to (e.g., on HDFS or S3)

· pipe assemblies: what types of transformations/operations to run against it, specified independently of the data sources they process

from http://docs.cascading.org/impatient/

And this buys us?

Lots of stuff: · Topological scheduling

· Reusable/parameterizable components

· Debug facilities (traps, intermediate sinks, logging, etc)

· LocalFlowConnector platform to run in-memory flows

· and... Testable Components!

An example application

Verify* extract data from a variety of production sources (MySQL, MongoDB, etc)

*meaning data creation/update patterns fall within acceptable (experimentally determined) thresholds

“Medium” data:

· 60M+ users, 200M+ signatures, 1.7B+ emails

· 100M+ rows a week for more active sources

· Scale with number of sources & traffic patterns

Cascading Flow

Initial source standardization:

Pipe source = new Rename( new Retain( new Pipe(sourceName + "/continuity"), sourceField ), sourceField, new Fields("timestamp"));

Cascading Flow

Custom Pipe Assemblies:

@VisibleForTestingPipe timestampContinuityAggregate(Pipe source) { return new Every( new GroupBy( source, new Fields("time_bucket"), new Fields("timestamp") ), new Fields("timestamp", "time_bucket"), new TimestampContinuityAggregator(), new Fields("time_bucket", "average_delta", "max_delta") );}

Pipe assembly: timestampContinuityAggregate

Simple unit tests@Testpublic void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") );

Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate );

flow.complete();

validateLength(flow, 2, null);

List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); }}

Simple unit tests@Testpublic void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") );

Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate );

flow.complete();

validateLength(flow, 2, null);

List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); }}

@Testpublic void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") );

Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate );

Simple unit tests@Testpublic void testContinuityAggregate() throws Exception { Pipe continuityAggregate = controller.timestampContinuityAggregate( new Pipe("timestampContinuityAggregate") );

Flow flow = getPlatform().getFlowConnector().connect( source(new Fields("timestamp", "time_bucket"), "timeBucket"), sink(new Fields("time_bucket", "average_delta", "max_delta"), "timestampContinuityAggregate"), continuityAggregate );

flow.complete();

validateLength(flow, 2, null);

List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); }}

flow.complete();

validateLength(flow, 2, null);

List<Tuple> values = getSinkAsList(flow); for (Tuple value : values) { Assert.assertTrue( value.getString(0).equals("2007-02-23") || value.getString(0).equals("2007-02-24") ); Assert.assertTrue(value.getString(1).equals("2717.5925925925926")); Assert.assertTrue(value.getString(2).equals("19331")); }}

Integration testingpackage org.change.ml.extract.verification;

import ...

@PlatformRunner.Platform({LocalPlatform.class})public class ControllerIntegrationTest extends VerificationTestCase { ...

@Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...)

Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete();

List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink);

Assert.assertTrue(continuityTuples.get(0).toString("\t").equals("2013-02-25\t7.166666666666667\t37")); Assert.assertTrue(continuityTuples.get(1).toString("\t").equals("2013-02-26\t720.6\t3599"));

Assert.assertTrue(verifiedTuples.get(0).toString("\t").equals( "2013-02-25,2013-02-26\t2013-02-26\t2013-02-25,2013-02-26\t2013-02-26") ); }...

}

Integration testingpackage org.change.ml.extract.verification;

import ...

@PlatformRunner.Platform({LocalPlatform.class})public class ControllerIntegrationTest extends VerificationTestCase { ...

@Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...)

Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete();

List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink);

Assert.assertTrue(continuityTuples.get(0).toString("\t").equals("2013-02-25\t7.166666666666667\t37")); Assert.assertTrue(continuityTuples.get(1).toString("\t").equals("2013-02-26\t720.6\t3599"));

Assert.assertTrue(verifiedTuples.get(0).toString("\t").equals( "2013-02-25,2013-02-26\t2013-02-26\t2013-02-25,2013-02-26\t2013-02-26") ); }...

}

package org.change.ml.extract.verification;

import ...

@PlatformRunner.Platform({LocalPlatform.class})public class ControllerIntegrationTest extends VerificationTestCase { ...

@Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...)

Integration testing

Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink );

package org.change.ml.extract.verification;

import ...

@PlatformRunner.Platform({LocalPlatform.class})public class ControllerIntegrationTest extends VerificationTestCase { ...

@Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...)

Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete();

List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink);

Assert.assertTrue(continuityTuples.get(0).toString("\t").equals("2013-02-25\t7\t37")); Assert.assertTrue(continuityTuples.get(1).toString("\t").equals("2013-02-26\t720.6\t3599"));

Assert.assertTrue(verifiedTuples.get(0).toString("\t").equals( "2013-02-25,2013-02-26\t2013-02-26\t2013-02-25,2013-02-26\t2013-02-26") ); }...

}

Integration testingpackage org.change.ml.extract.verification;

import ...

@PlatformRunner.Platform({LocalPlatform.class})public class ControllerIntegrationTest extends VerificationTestCase { ...

@Test public void testVerifyTimestampContinuity() throws Exception { Tap signaturesSource = source(...) Tap continuitySink = sink(...) Tap verifiedSink = sink(...)

Flow verifyContinuity = controller.verifyTimestampContinuity( "signatures", new Fields("created_at"), new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2013-02-25 00:00:00"), signaturesSource, continuitySink, verifiedSink ); verifyContinuity.complete();

List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink);

Assert.assertTrue(continuityTuples.get(0).toString("\t").equals("2013-02-25\t7\t37")); Assert.assertTrue(continuityTuples.get(1).toString("\t").equals("2013-02-26\t720.6\t3599"));

Assert.assertTrue(verifiedTuples.get(0).toString("\t").equals( "2013-02-25,2013-02-26\t2013-02-26\t2013-02-25,2013-02-26\t2013-02-26") ); }...

}

verifyContinuity.complete(); List<Tuple> continuityTuples = asList(verifyContinuity, continuitySink); List<Tuple> verifiedTuples = asList(verifyContinuity, verifiedSink);

Assert.assertTrue(continuityTuples.get(0).toString("\t").equals("2013-02-25\t7\t37")); Assert.assertTrue(continuityTuples.get(1).toString("\t").equals("2013-02-26\t720.6\t3599"));

Assert.assertTrue(verifiedTuples.get(0).toString("\t").equals( "2013-02-25,2013-02-26\t2013-02-26\t2013-02-25,2013-02-26\t2013-02-26") );

Running tests locally

is awesome!

Check out https://github.com/Cascading/cascading.samples for examples using Gradle to drive Cascading

Now what?

· Continuous Deployment· Gradle on Circle CI build to S3

· Elastic MapReduce scaleable· DataPipeline provisioned clusters

· Production Monitoring · SNS to Pager Duty integration on failures

We’re hiring!

Thanks to my colleagues at Change.org for all their help with this presentation.

If you’re interested in building large-scale distributed systems for data processing and recommendations (along with lots of other cool stuff that helps empower people everywhere to create the change they want to see), we’re hiring!

Drop me an email (vijay@change.org) or visit our website (http://www.change.org/hiring)