SHARETHISInsights

Ishika Paul

Principle Engineer

The Team

Jack Zhang Neil Lobo Jeff Blanding Ishika Paul

SHARING TOOLS AT SCALE

2.4 MM SITES AND APPS

120 SOCIAL CHANNELS

1. comScore Media Matrix Report * Includes PC, Tablet, and Mobile sites.

210 MM US USERS1

95% REACH*

EVENT INSIGHTS

BRAND ACTIVATION INSIGHTS30 DAY ACTIVITY

TRENDSCHANNEL BREAKOUT

GENDER BREAKOUT AGE BREAKOUT

Challenge● Shut down 3 hadoop

clusters (~100 machines)● integrate 4 existing

products ● Support over 30 insights

reports● Release to clients by

June 1 (dev started in early March)

Challenge

● Build a new team

● Analyze 350TB of Data

● increase response time

by 100x

Challenge Accepted

● Relatively Clean Code

● Reuseable code

● Reactive and

Responsive UI

● Unit Testing

Reactiveness with Meteor

● Built in integration with Mongodb

● Publish - Subscribe for immediate

reactiveness

● No polling for up-to-date data

● Use of websockets is opaque to developers

Response Architecture Demo

Response Architecture

ServerMeteor.publish("job-status", function (jobId) { return JobCollection.find(jobId);});

ClientTracker.autorun(function () { var jobId =

Session.get("jobId"); Meteor.subscribe("job-status",

jobId); var item =

Downloads.findOne({"jobId":jobId}); if (!item) { return; } var job =

jobService.getJob(jobId); if (job && job.key) { downloadReport(job,

item.campaignGroup); }

}

Jasmine Integration

Meteor: Jasmine integration

What we loved

● writing test are easy!

● Peace of mind

Not so much

● mocking meteor mongo

● no code coverage

● have to remember to add

tests

Free Stuff!

What is a Insights Report?

Each type a client calls a report:● Create a valid job for that particular type of Insights

● Return cached report data if job already ran

● Create new job, send back jobId to client.

Spawn off Async job to:o Call Rest-Insights

o save BQ data into S3 and key into Mongo

o Update Job state (Success | Failed)

Building New Reports is Easy CampaignInsights Prototypes Example

CampaignInsights = function() {};

CampaignInsights.prototype = new baseInsights(new CampaignJobService());CampaignInsights.prototype._decorateData = function (results) {};CampaignInsights.prototype._callInsights = function (job, report, callback) {};

CampaignJobService = function() {};CampaignJobService.prototype = Object.create(JobService.prototype);

CampaignJobService.prototype._requiredFields = [campaign required fields];

CampaignJobService.prototype._generatedFields = [generated fields that will be produced];CampaignJobService.prototype.getFilter = function getFilter(aJob) {};CampaignJobService.prototype._enrichJob = function (job) {};CampaignJobService.prototype._generateS3FileName = function (job, partialName){};

Building New Reports is Easy CampaignJobService Prototype Example

Prototype Benefits

● No code duplication

● Faster development turn around time

● Common functionality guaranteed to work!

Response Architecture

● BigQuery supports SQL!● Common Inputs● Report Definition Loading into Mongo:

insertReportWithDefaults({ reportName: "Income Segments of Ad Clickers", insightsType: INSIGHT_TYPE.CAMPAIGN, funcName: 'getInsightsByCampaign', fileName: 'income_segments_clickers', schema: 'segment_name,dlx,count', tooltip: "Number of ad clickers by household income group." });

Demo

We are hiring :)www.sharethis.com/rocketship

▪ Software Engineers ▪ Site Reliability Engineer / DevOps ▪ Data Scientists▪ Big Data▪ Full Stack▪ Machine Learning / Optimizations

QA