PretioReal-time Mapping of API Requests

Rob O'Dwyer

Nathan Lambert

Cameron Rohani

We power reward-based advertising on mobile apps and FB games

About Pretio

Why this ProjectWe wanted to start making visualizations of our dataSee patterns in geographic locationShow traffic changes and shifts in popularity as they happenInvestors like pretty graphs!

Where's the Data From?Mobile Ad Unit APITraffic from one of our US-only publishers

JSON Event{ "@version": "1", "@timestamp": "2014-09-20T02:11:57.681Z", "type": "offers:served", "offer": "3585b8da3f5011e4a11a12313f063e60", "user_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "geoip": { "country_code2": "US", "country_name": "United States", "city_name": "Louisville", "latitude": 38.2542, "longitude": -85.7594, "timezone": "America/New_York" }, "tags": ["geo"]}

Check it out at

Quick Demogeo.pretio.in

The MapUses Custom icons

Leaflet.js

Server-Sent EventsHTML5Stream data in real-time to the browser

Why not Websockets??Bidirectional, more flexibilitySpecial server support neededSSE - lines of text from a long HTTP connectionAuto reconnects!

Server-Sent Event Protocolevent: notificationdata: hello world

event: locationdata: {data: "latitude": 38.2542,data: "longitude": -85.7594data: }

In the browser// Connect to servervar source = new EventSource('server/path/to/events');

source.addEventListener('open', function(e) { console.log('Connected!');});

// Listen for "location" eventssource.addEventListener('location', function(e) {

// We're sending data as JSON var data = JSON.parse(e.data);

displayOnMap(data.latitude, data.longitude);

});

node.jsSo hot right nowEvent-driven JavaScript on the serverGood for building "real-time" apps

The appUses for rendering web pagesSimple UDP server that receives events

Express

Listening for UDP packetsvar udpServer = udp.createSocket('udp4', function(msg, remote) {

console.log('Received message from ' + remote.address + ':' + remote.port);

try { var data = JSON.parse(msg);

// Send out as event events.emit('received', data);

} catch(err) { console.error('Error parsing message: ' + err); }});

// Listen for UDP messagesudpServer.bind(PORT, '0.0.0.0');

Event Streamapp.get('/events', function(req, res) {

console.log('New client connected!');

res.status(200) .set({ 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive' });

// Listen for events from UDP server events.on('received', function(data) { // Send new "location" event to browser res.write('event: location\n'); res.write('data: ' + JSON.stringify(data) + '\n\n'); });});

Logstash

A tool for parsing, rewriting and shipping logsContinuous pipeline of eventsConnects to almost everything (log files, queues, sockets, DBs, etc.)IP » Geo is built-in

Sample Logstash Configinput { redis { data_type => channel host => "redis.server" port => 6379 key => logs:offer_serving codec => json }}

filter { grok { match => [ "[ip]", "%{IP:client_ip}" ] } geoip { source => client_ip }}

output { udp { codec => json host => "geo.pretio.in" port => 3847 }}

Read your web server logs!input { file { path => "/var/log/apache/access.log" }}filter { grok { pattern => "%{COMBINEDAPACHELOG}" }}

IP GeolocationUses free MaxMind GeoIP databaseNot very accurate

Redis

Advanced key-value datastoreAPI Servers publish events to hereLogstash picks them up

Multi-layered Rendering

IdeasShow server logs from your appPlotting real-time tweet data

?