Building “sexy” real-time analytics systems

AdGear is full-stack ad platform for publishers and advertisers, with advanced analytics, attribution measurement, ad serving, and real-time bidding technology.

Real-time bidding (RTB)

• help clients to make informed decisions

• should I increase the bid price?

• should I bid on exchange X?

• inventory control (brand safety)

• debugging (bots detection, creatives audits)

Real-time reporting... why?

“Sexy” real-time analytics systems

“Sexy”?

• elegant backend

• beautiful user interface

• ssh

• node.js

• socket.io

Architecture #1 (3 years ago)

Problems

• no SMP support

• each process needs to be monitored

• requires load-balancing (nginx)

• duplicated state (per process)

• duplicated work (de-serialization)

• bad error handling (event loop explodes)

• callbacks...

* promise construct

• ssh_channel *

• gproc (pub sub)

• ETS counters

• bullet (cowboy)

* https://gist.github.com/lpgauth/6529807

Architecture #2 (1.5 years ago)

1. receive buffered events, split and de-serialize

2. each event is sent to a collector process (3) using gproc (pubsub) for filtering

3. collector (gen_server) aggregates message using ETS counters and flush every second

4. bullet handler serializes the aggregates (tab2list to json)

Architecture #2

Problems

• ssh_channel process and collector process are bottlenecks

• number of messages increases with the number of clients

• requires lots of bandwidth for large streams

• limited filtering (match specs)

Improvements... (6 months ago)

• optimize collector’s msg loop (gen_server to proc_lib)

• use ssh compression

• added support for openssh zlib compression *

• R16B02

* https://github.com/lpgauth/otp/tree/openssh_zlib

This worked for a while...

“Hey man, it would be very cool if you could show in real-time the number of bid requests per domain for

Friday’s demo... Can you do it?” - boss

Sure.

What did I just agree too...

• I only have 3 days to build this...

• bid requests stream is too large to aggregate in a central location (1+ Gbit/s - 80K+/s)

Strategy for demo

1. move aggregation upstream

2. use ETS match select to find table ids (filtering)

3. increment counters in process (no message!)

4. periodically flush aggregates via message to collector node

5. collector node increments local counters and periodically flush aggregates to bullet handler

Success!

Introducing swirl! “lightweight distributed stream processor”

Swirl components

• “dynamic” streams (swirl_stream)

• simple behavior that implements a map-reduce like interface (swirl_flow)

• powerful filtering language (swirl_ql)

• process registry (swirl_tracker)

Streams

Flows

* application:start(swirl).

swirl_flow behavior

Mapper Node1. process “emits” event

2. lookup in ETS if there’s a flow that matches the stream name and filter

3. if there’s a match, call flow_mod:map/4

4. if map returns counters, increment in ETS

5. swirl_mapper periodically flush aggregates to reducer node

Reducer Node

1. swirl_tracker receives mapper aggregates and forwards it to reducer

2. reducer increments counters in ets

3. reducer flushes counters to flow_mod:reduce/4

Swirl-ql

• sql where clause like syntax

• supported operators:

• AND / OR

• <, <=, =, >, <>

• IN (x, y) / NOT IN (x, y, z)

• IS NULL / IS NOT NULL (undefined)

* https://github.com/lpgauth/swirl-ql

Swirl-ql

• examples:

• “event IN (‘impression’, ‘click’)”!

• “buyer_id IS NOT NULL AND buyer_id <> 3”!

• “event = ‘impressions’ AND (buyer_id IN (3, 5) OR buyer_id IS NULL)

Swirl-ql

• leex / yecc for parsing (use lex / yacc doc)

• pattern match ftw!

• use hipe (~200% speed gain in micro benchmarks)

• 0.286 vs 0.097 microseconds *

• experimenting with dynamic compilation

* http://theory.stanford.edu/~sergei/papers/sigmod10-index.pdf

Swirl limitations

• best-effort (hard problem!)

• netsplits

• crash

• in-memory only

Todo

• node discovery

• code distribution

• resource limitation

• better documentation!

• swirl

• bullet (cowboy)

Architecture #3 (now!)

* https://github.com/lpgauth/swirl-demo

Demo!

pssst: we’re hiring!

Thank You!

twitter: lpgauth github: lpgauth