real time bidding with erlang · real time bidding with erlang . about me . 2012 erlang at aol 3...
TRANSCRIPT
![Page 1: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/1.jpg)
Real Time Bidding with Erlang
![Page 2: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/2.jpg)
About Me
![Page 3: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/3.jpg)
2012 Erlang at AOL
3
Erlang Here
![Page 4: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/4.jpg)
Agenda
What is RTB ? RTB Exchange Architecture Bottlenecks Tracing and Debugging
![Page 5: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/5.jpg)
What is RTB ?
![Page 6: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/6.jpg)
Online Advertising
Paid for by Advertising
Free Content on WWW
Upfront agreements between Publishers and Advertisers
Placement with Advertiser’s Banner
![Page 7: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/7.jpg)
What is Real Time Bidding • The buying a selling of online advertisements in real time while a page is loading
Exoclick, www.exoclick.com (2014)
PUBL
ISHER
S B
ette
r val
ue fo
r Inv
ento
ry ADVERTISERS B
etter audience
SSP DSP
![Page 8: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/8.jpg)
SSP
Adserving Workflow
Web Server
AdServer
RTB
DSP 1
DSP 2
DSP 3
DSP N Advertiser CDN
![Page 9: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/9.jpg)
SSP
Adserving Workflow
Web Server
AdServer
RTB
DSP 1
DSP 2
DSP 3
DSP N Advertiser CDN
![Page 10: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/10.jpg)
RTB Exchange Architecture
![Page 11: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/11.jpg)
AdServer RTB Exchange
ETS ETS
ETS
ZMQ Auction
RTB Exchange
![Page 12: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/12.jpg)
Retrieving Campaign Data
Config Service
MySql Cluster
RTB RTB RTB RTB
JSON
![Page 13: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/13.jpg)
Hashing Node Name to MySql Server 1 hash_name(Node, MySqlHosts) -‐> 2 Sum = lists:sum(atom_to_list(Node)), 3 Index = Sum rem length(MySqlHosts),
4 lists:nth(Index, MySqlHosts).
![Page 14: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/14.jpg)
DSP 1
AdServer RTB Exchange
ETS
DSP 2
DSP 3
DSP N
Web UI
SNMP
folsomite
Creative Scanning
DSP
DSP
DSP
DSP
ETS ETS
Couchbase
ZMQ Auction
erlmc
RTB Exchange
lhttpc
![Page 15: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/15.jpg)
How Much does it Scale ?
![Page 16: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/16.jpg)
8,000 The number of bid requests build and sent
to DSPs every second per host at peak
16
![Page 17: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/17.jpg)
1 Million The number of placements
17
![Page 18: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/18.jpg)
50 Billion The number of bid requests build and sent
to DSPs every day
18
![Page 19: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/19.jpg)
Independent Auctions
Independent Requests
Independent DSP Processes
A Highly Concurrent System
![Page 20: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/20.jpg)
ZMQ Interface
![Page 21: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/21.jpg)
AdServer RTB Exchange
![Page 22: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/22.jpg)
AdServer RTB Exchange
AdServer
AdServer
![Page 23: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/23.jpg)
AdServer RTB Exchange
AdServer
AdServer
RTB Exchange
RTB Exchange
![Page 24: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/24.jpg)
The Symptom Requests handled by RTB Exchange Drop off for 1.5 hours
![Page 25: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/25.jpg)
The Problem
§ 10,000s messages in ZMQ Mailbox
§ Single Process
§ Active Socket
§ Mailbox full of messages being sent and received
§ Selective receive when sending to adserver
§ Other applications running on same host
![Page 26: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/26.jpg)
escript for montoring High Priority ZMQ Process
Passive Socket
Multiple ZMQ Endpoints
• RPC call to check message queue length
• erlang:halt/1 to shut down the node
How we Fixed It
• process_flag(priority, high)
• Convert every request into a process ASAP
• Polling for requests
• ZMQ High Water Mark
• Gives us >1 ZMQ process • Enabled by configuration
• heart to Restart node
![Page 27: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/27.jpg)
Pooling Resources
![Page 28: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/28.jpg)
DSP 1
AdServer RTB Exchange
DSP 2
DSP 3
DSP N
Web UI
SNMP
folsomite
Creative Scanning
DSP
DSP
DSP
DSP
Couchbase
ZMQ Auction Auction
Auction Auction
Auction Auction
ETS
Couchbase
erlmc
![Page 29: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/29.jpg)
Creative Scanning
DSP 1
AdServer RTB Exchange
DSP 2
DSP 3
DSP N
Web UI
SNMP
folsomite
DSP
DSP
DSP
DSP
ETS
Couchbase
ZMQ Auction Auction
Auction Auction
Auction Auction
erlmc
![Page 30: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/30.jpg)
Storing Resources in ETS Index erlmc process
1 <0.100.0>
2 <0.101.0>
3 <0.102.0>
: :
N <0.123.0>
Record Value pool_size N
index 1
![Page 31: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/31.jpg)
Round Robin Lookup in ETS 1 get_worker() -‐> 2 PoolSize = ets:lookup_element(?TABLE, pool_size, 2), 3 Index = ets:update_counter(?TABLE, index, {2,1,PoolSize,1}), 4 ets:lookup_element(?TABLE, Index, 2).
![Page 32: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/32.jpg)
Random ETS Lookup 1 get_worker() -‐> 2 PoolSize = ets:lookup_element(?TABLE, pool_size, 2), 3 Index = erlang:phash2(self(), PoolSize) + 1, 4 ets:lookup_element(?TABLE, Index, 2).
![Page 33: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/33.jpg)
phash2 vs update_counter
update_counter useful for round_robin phash2 useful for random selection phash2 faster than update_counter (x1.5 - x2) phash2 does no writes to ETS hence no locking
![Page 34: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/34.jpg)
Tracing and Debugging
![Page 35: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/35.jpg)
Bid Collection Time • Spike in Timer every 20-30 mins • X5 magnitude • System not under load
x5
30 mins
![Page 36: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/36.jpg)
Tracing the Spike Trace all new
spawned processes
![Page 37: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/37.jpg)
1 dbg:tracer(process, {fun trace_handler/2, {IO, %% file to log to dict:new(), %% dict for recording info in Threshold, %% timer threshold Count}}), 2 {ok, Tracer} = dbg:get_tracer(), 3 erlang:trace(new, true, [call, timestamp, {tracer, Tracer}]).
callback function for tracer process
![Page 38: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/38.jpg)
Tracing the Spike Trace all new
spawned processes
Dynamically enable more detailed tracing
Process used for an auction ?
![Page 39: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/39.jpg)
1 dbg:tp(bid_processor, process_request, x),
![Page 40: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/40.jpg)
Tracing the Spike Trace all new
spawned processes
Dynamically enable more detailed tracing
Process used for an auction ?
![Page 41: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/41.jpg)
1 trace_handler({trace_ts, Pid, call, {bid_processor, process_request, _}, TS}, {IO, Dict, Threshold, Count}) -‐> 2 erlang:trace(Pid, true, [timestamp, %% trace with timestamps 'receive', %% messages received call, %% funcion calls running, %% when scheduled in/out {tracer, self()}]), 3 Dict1 = dict:append(Pid, {TS, process_request}, Dict), 4 {IO, Dict1, Threshold, Count};
![Page 42: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/42.jpg)
Tracing the Spike Trace all new
spawned processes
Dynamically enable more detailed tracing
Process used for an auction ?
Time logged > X ?
Dump dict into file
![Page 43: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/43.jpg)
Conclusions
Highly Concurrent System Single Processes are bottlenecks § High Priority
§ Process Pools
Erlang VM a Great Platform for Debugging Production Systems
![Page 44: Real Time Bidding with Erlang · Real Time Bidding with Erlang . About Me . 2012 Erlang at AOL 3 Erlang Here ... What is Real Time Bidding • The buying a selling of online advertisements](https://reader033.vdocuments.site/reader033/viewer/2022060311/5f0ae7087e708231d42de7db/html5/thumbnails/44.jpg)
The Future
Scale for more Publishers and Demand Partners Continue to Build a Great Erlang Team Support Growth Erlang Community § Internships § University Presentations
Erlang Factory - Dublin Sept 2015