leveraging kafka for big data in real time bidding, analytics, ml & campaign management for...

@helenaedelson #kafkasummit 1

Leveraging Kafka for Big Data in Real Time Bidding, Analytics, ML & Campaign Management for Globally Distributed Data Flows

Helena Edelson @helenaedelson Kafka Summit 2016

@helenaedelson #kafkasummit

VP of Engineering, Tuplejump

Previously: Sr Cloud / Big Data / Analytics Engineer: DataStax, CrowdStrike, VMware, SpringSource...

Event-Driven systems, Analytics, Machine Learning, Scala

Committer: Kafka Connect Cassandra, Spark Cassandra Connector

Contributor: Akka, previously: Spring Integration

Speaker: Kafka Summit, Spark Summit, Strata, QCon, Scala Days, Scala World, Philly ETE

twitter.com/helenaedelson github.com/helena

slideshare.net/helenaedelson

The Real Topic

http://www.slideshare.net/palvaro/ricon-keynote-outwards-from-the-middle-of-the-maze/42

Chaos Of Distribution

One of the more

fascinating problems is

that of solving the chaos

of distributed systems.

Regardless of the

domain.

Aproaching this within the use case of:

High-Level Landscape

Platform & Infrastructure

Strategies and Patterns

Four-Letter Acronyms

Can't Touch This

Architecture

The Landscape

The Digital Ad Industry

An RTB Drive-By

Real time auction for ad spaces, all devices High throughput, low-Latency (similar to FIN Tech but not quite) OpenRTB API Spec - but not everyone uses it

Open protocol for automated trading of digital media across

platforms, devices, and advertising solutions

Ad Delivered to User

In A Nutshell

User hits a

Publisher'spage

Advertiser

Advertisers send Bid Requests

Highest Bid

Accepted

Site: Ad supported

content

Real Time Exchange & Auction (SSP):

OpenRTB Server used to bid

Bidder Service (DSP):

OpenRTB client

Advertiser:Buyer wants ad

impressions. Uses bidders to bid on

behalf

Publisher:Seller has ad spaces to sell to highest

bidders

User Devices

ad request

winning ad

bid request

win notice & settlement price

insert orders

bid response

winning ad

RTB Auction for Impressions

Time Is Money

RTB: Maximum response latency of 100 ms

Time Is Money

Assume some network latency!

Sampling of RTB Events

Ad Request

Bid Request - JSON 100 bytes

Compute optimal bid for advertiser

Bid Response - JSON 1000 bytes (may include ad metadata)

Win Notification (may or may not exist) with settlement price

Ad Impression - when the ad is viewed

Ad Click

Ad Conversion

Event Streams

Auctions: auction data + bid requests

Ad Impressions: which ad ids were shown

Ad Clicks: which auction ids resulted in a click

Ad Conversions: streams joined on auction id

Analytics Aggregations & ML to derive hundreds of metrics and dimensions

Real TimeJust means Event-Driven or processing events as they arrive.

Does not automatically equal sub-second latency requirements.

Seen / Ingestion TimeWhen an event is ingested into the system

Event TimeWhen an event is created, e.g. on a device.

The Platform

Platform Requirements24 / 7 Uptime

Brokerage model: DSPs only make $ on successful ad deliveries, so uptime is critical

Security

Enable service across the globe

Handle thousands of concurrent requests per second

Scale to traffic of 700TB per day

Manage 700TB per day of data

Derive Metrics

Business RequirementsSupport SLAs for bid transactions

Legal constraints - user data crossing borders

The critical path must be fast to win

No data loss on ingestion path

Bid & Campaign Optimization

Frequency Capping

Management UI for Publishers & Advertisers

Questions To Answer% Writes on ingestion, analytics pre-aggregation, etc.

% Reads of raw data by analytics, aggregated views by customer management UI

How much in memory on RTB app nodes?

Dimensions of data in analytics queries

Optimization Algos

What needs real time feedback loops, what does not

Which data flows are low-lateny/high frequency, which not

Where are potential bottlenecks

ConstraintsResources - I need to build highly functioning teams that are psyched about the work and working together

Budget

Cloud Resources

JDK Version (What?!)

Existing infrastructure & technologies that will be replaced later but you have to deal with now :(

Pro Tip: Pay well,

Allow people to grow & be

creative

Strategies

To Avoid

Beware of the C word

Consistency?

Convergence?

http://www.slideshare.net/palvaro/ricon-keynote-outwards-from-the-middle-of-the-maze/39

he went there

@palvaro

Complexity

Can't Ops your way out of that

Occam's razor: Simpler theories are preferable to more complex

Strategies

ApproachesEventual/Tunable consistency

Time & Clocks in globally-distributed systems

Location Transparency

Asynchrony

Pub-Sub

Design for scale

Design for Failure

Kafka as Platform Fabric

From MVP to Scalable with KafkaMicroservices

Does One Thing, Knows One Thing Separate low-latency hot path Separate deploy artifacts

Separate data mgmt clusters by concern

analytics, timeseries, etc.

CQRS: Separate Read Write paths

Scalpel...

Separate The Monolith

Immutable events stream to Kafka, partitioned by event type, time, etc.

Subscribers & Publishers

RTB microservices - receives raw, receives

Analytics cluster - receives raw, publishes aggregates

Management / Reporting nodes

Services communicate indirectly via Kafka

CQRS: Command Query Responsibility Segregation

Decouple Write streams from Read streams

Different schemas / data structures

Writers (Publishers) publish without having awareness who needs to receive it or how to reach them (location, protocol...)

Readers (Subscribers) should be able to subscribe and asynchronously receive from topics of interest

Eventually Consistent Across DCs

US-East-1

MirrorMakerEU-west-1

RTB micro

services

RTB micro

services

RTB micro

services

Publishers

Subscribers

Publishers

Kafka Cluster Per Region

Mgmt micro

services

Mgmt micro

services

Mgmt micro

servicesQuery Layer

Analytics & ML Cluster

Timeseries Cluster

Spark Streaming

Cassandra

Cross DC Replication

Topology Aware

Spark Streaming

Cassandra

Spark Streaming

Cassandra

Topology Aware

Spark Streaming

Cassandra

Compute Layer

MirrorMaker

RTB micro

services

RTB micro

services

RTB micro

services

Publishers

Subscribers

Publishers

Mgmt micro

services

Mgmt micro

services

Mgmt micro

services

US-East-1

EU-west-1

Timeseries Cluster

Spark Streaming

Cassandra

Topology Aware

Spark Streaming

Cassandra

Spark Streaming

Cassandra

Topology Aware

Spark Streaming

Cassandra

Compute Layer

Query Layer

Kafka Cross Datacenter Mirroring

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config config/consumer_source_cluster.properties --producer.config config/producer_target_cluster.properties --whitelist bidrequests --num.producers 2 --num.streams 4

Publish messages from various datacenters around the world

Users in the US and UK connect DCs in their geo region for lower latency

Both DCs are part of the same cluster for X-DC Replication

Configure LB policies to prefer local DC

LOCAL_QUORUM reads

Data is available cluster-wide for backup, analytics, and to account for user travel across regions

Cassandra Cross DC ReplicationIt's out of the box. Multi-region live backups for free:

[ NetworkTopologyStrategy ]

Cassandra Cross DC ReplicationKeep EU User Data in the EU

CREATE KEYSPACE rtb WITH REPLICATION = {

‘class’: ‘NetworkTopologyStrategy’,

‘eu-east-dc’: ‘3’,‘eu-west-dc’: ‘3’

Cassandra Time Windowed Buckets with TTL

CREATE TABLE rtb.fu_events ( id int, seen_time timeuuid, event_time timestamp, PRIMARY KEY (id,date)

) WITH CLUSTERING ORDER BY (event_time DESC) AND compaction = { 'compaction_window_unit': 'DAY', 'compaction_window_size': '3', 'class':'com.jeffjirsa.cassandra.db.compaction.TimeWindowCompactionStrategy'

} AND compression = { 'crc_check_chance': '0.5', 'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor' } AND bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"100"}' AND dclocal_read_repair_chance = 0.0 AND default_time_to_live = 60 AND gc_grace_seconds = 0 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE';

3 DAY buckets -

larger SSTables on disk minimizes bootstrapping issues when adding nodes to a cluster

3 MINUTE buckets 1 HOUR buckets 1 DAY buckets

MICROSECOND resolution:

Want Can Or Currently Use Status ButKafka Security Kafka Security TLS, Kerberos, SASL, Auth,

Encryption, Authenticationv0.9.0

Thanks Jun!

Integrated Streaming Kafka Streams processing inside Kafka, no alternate cluster setup or ops.

v0.10 Thanks Guozhang!

It's java :( Iw

Cassandra CDC Cassandra CDC. Triggers? Tiggers are a pre-commit

hook :(

The Epic JIRA: https://issues.apache.org/jira/browse/CASSANDRA-8844

no comment

And... Kafka Streams & Kafka Connect Integration

..wait for it..no comment

Always on, X-DC Replication, Flexible Topologies

Kafka, Cassandra

Fault Tolerance Kafka, Spark, Mesos, Cassandra, Akka

Baked In

Location Transparency Kafka, Cassandra, Akka Check!

Asynchrony Kafka, Cassandra, Akka Check!

Decoupling Kafka, Akka Check!

Pub-Sub Kafka, Cassandra, Akka Check!

Immutability Kafka, Akka, Scala Check!

My Nerdy Chart v2.0

Kafka Streams in v 0.10

val builder = new KStreamBuilder()

val stream: KStream[K,V] = builder.stream(des, des, "raw.data.topic")

.flatMapValues(value -> Arrays.asList(value.toLowerCase.split(" ")

.map((k,v) -> new KeyValue(k,v))

.countByKey(ser, ser, des, des, "kTable")

.toStream

stream.to("results.topic", ...)

val streams = new KafkaStreams(builder, props)

streams.start()

Kafka Streams & Kafka Connect?

val builder = new KStreamBuilder()

val stream1: KStream[K,V] = builder.stream(new CassandraConnect(configs))

.flatMapValues(..)

.map((k,v) -> new KeyValue(k,v))

.countByKey(ser, ser, des, des, "kTable")

.toStream

stream.to("results.topic", ...)

val streams = new KafkaStreams(builder, props)

streams.start()

/** Writes records from Kafka to Cassandra asynchronously and non-blocking. */ override def put(records: JCollection[SinkRecord]): Unit

/** Returns a list of records when available by polling for new records. */ override def poll: JList[SourceRecord])

https://github.com/tuplejump/kafka-connect-cassandra

Frequency Capping

1. Count the number of times user X has seen ad Y from Advertiser A's Campaign C

2. Limit the max number of impressions of an ad within T1...T2

Use Case:

Continuously count impressions grouped by campaign across DCs

low-latency reads & writes

Must scale

Cross DC Counters

Translation: Distributed Counters

Redis? Broke under the load

Aerospike? Great candidate

Eventuate? Interesting, much lighter

Kafka streams when it's out? Interesting, already in the infra

Flink? Very interesting but...

Cassandra Counters - not applicable for this

Frequency Capping

As a distributed counting microservice

As a key-value store for in-memory caching

Fast reads - Very read heavy

99% reads are < 1 ms latency (sweet)

30,000 writes per second

350,000 reads per second on 7 nodes

Replication factor 2:

Cross datacenter replication (XDC), SSD-backed

Excellent few posts by Dag, Tapads CTO on in-memory infrastructure + Ad Tech: (see resources slide)

Aerospike

CRDT: Conflict Free Replicated Data TypeState-based: objects require only eventual communication between pairs of replicas

Operation-based: replication requires reliable broadcast communication with delivery in a well-defined delivery order

Both guaranteed to converge towards common, correct state

Keep replicas available for writes during a network partition requires resolution of conflicting writes when the partition heals

EventuateA toolkit for building distributed, HA & partition-tolerant event-sourced applications. Developed by Martin Krasser (@mrt1nz) for Red Bull Media (open source)

Interactive, automated conflict resolution (via op-based CRDTs)

Separates command side of an app from its query side (CQRS)

Primary Goals: preserving causality, idempotency & event ordering guarantees even under chaotic conditions

AP of CAP - conflicts cannot be prevented & must be resolved.

Causality - tracked with Vector Clocks

Adapters provide connectivity to other stream processing solutions

Can currently chose Cassandra if desired

Kafka coming soon!

Replication of application state through async event replication across locations

Locations consume replicated events to re-construct application state locally

Multiple locations concurrently update as multi-master

Eventuate as Distributed CRDT Microservice

Applications can continue writing to a local replica during

a network partition

-> To Cassandra-> To Kafka

(soon)

Pass To Pipeline:

import scala.concurrent.Futureimport akka.actor.{ActorRef, ActorSystem}import com.rbmhtechnology.eventuate.crdt.{CRDTServiceOps, Counter, CounterService}

class CappingService(val id: String, override val log: ActorRef) (implicit val system: ActorSystem, val integral: Integral[Int], override val ops: CRDTServiceOps[Counter[Int], Int]) extends CounterService[Int](id, log) { /** Increment only op: adds `delta` to the counter identified by `id` * and returns the updated counter value. */ def increment(id: String, delta: Int): Future[Int] = value(id) flatMap { case v if v >= 0 && (delta > 0 || delta > v) => update(id, delta) case v => Future.successful(v) } start()}

import scala.concurrent.Future import akka.actor.ActorSystem

val a = new CappingService(id1, eventLog)a.increment(id1, 3) // Future(3) 3 impressionsa.value(id1) // Future(3) 3 impressionsa.increment(id1, -2) // increments only, idempotent.

val b = new CappingService(id2, eventLog) b.value(id1) // Future(a.value(id1))

Knows the same count over n-instances, all geo-locations, for the same id

class CounterService[A : Integral](val replicaId: String, val log: ActorRef) {

def value(id: String): Future[A] = { ... }

def update(id: String, delta: A): Future[A] = { ... }

Eventuate

Eventuate TakeawayIt's just a jar!

OOTB async internal component messaging and fault tolerance

Integrate with relevant microservices

No store/cache cluster to deploy, just keep monitoring your apps Written in Scala Built on Akka - a toolkit for building highly concurrent, distributed, and resilient event-driven applications on the JVM

Analytics & ML

Refresher: Sampling of RTB Events

Ad Request

Bid Request - JSON 100 bytes

Compute optimal bid for advertiser

Bid Response - JSON 1000 bytes (may include ad metadata)

Win Notification (may or may not exist) with settlement price

Ad Impression - when the ad is viewed

Ad Click

Ad Conversion

OpenRTB: objects in the Bid Request model

TopK most high performing campaigns

Number of views served in the last 7 days, by country, by city

What determined successful ad conversions

Age distribution per campaign

Streaming Analytics

Spark Streaming Kafkaclass KafkaStreamingActor(ssc: StreamingContext) extends MyAggregationActor {

val stream = KafkaUtils.createDirectStream(...).map(RawData(_))

stream .foreachRDD(_.toDF.write.format("filodb.spark")

.option("dataset", "rawdata") .save())

/* Pre-Aggregate data in the stream for fast querying and aggregation later

stream.map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip)

).saveToCassandra(timeseriesKeyspace, dailyPrecipTable)

Can write to Cassandra, FiloDB...

Machine LearningTrain on 1+ week of data for

Recommendations

Bid Optimization

Campaign Optimization

Consumer Profiling

...and much more

Machine Learning

The probability of an ad, from a specific ISP, OS, website, demographic, etc. resulting in a conversion

Which attributes of impressions are good predictors of better ad performance?

Bid Optimization & Predictive Models

Which impressions should an Advertiser bid for?

Per campaign, per country it may run in..?

What is the best bid for each impression

Compute optimal bid

Train the model

Score bid requests

Determine value of bid reqest

Train on every bid req attribute

Based on Campaign Objectives

Against Budget Send bid decision to bidder

Machine Learning

Spark Streaming, MLLib & FiloDB

val ssc = new StreamingContext(sparkConf, Seconds(5))

val kafkaStream = KafkaUtils.createDirectStream[..](..)

.map(transformFunc) .map(LabeledPoint.parse)

kafkaStream.foreachRDD(_.toDF.write.format("filodb.spark")

.option("dataset", "training").save())

val model = new StreamingLinearRegressionWithSGD() .setInitialWeights(Vectors.dense(weights)) .trainOn(dataStream.join(historicalEvents)) model.predictOnValues(dataStream.map(lp => (lp.label, lp.features))) .insertIntoFilo("predictions")

700 Queries Per Second: Spark Streaming & FiloDB

Even for datasets with 15 million rows! Using FiloDB's

InMemoryColumnStore

Single host / MBP

5GB RAM

SQL to DataFrame caching

https://github.com/tuplejump/FiloDB

Evan Chan's (@velvia) blog post

NoLambda: A new architecture combining streaming, ad hoc, machine-learning, and batch analytics

US-East-1

MirrorMakerEU-west-1

RTB micro

services

RTB micro

services

RTB micro

services

Publishers

Subscribers

Publishers

Mgmt micro

services

Mgmt micro

services

Mgmt micro

servicesQuery Layer

Timeseries Cluster

Spark Streaming

Cassandra

Topology Aware

Spark Streaming

Cassandra

Spark Streaming

Cassandra

Topology Aware

Spark Streaming

Cassandra

Compute Layer

Self-Healing SystemsMassive event spikes & bursty traffic

Fast producers / slow consumers

Network partitioning & out of sync systems

DC down

Not DDOS'ing ourselves from fast streams No data loss when auto-scaling down

Byzantine Fault Tolerance?

Looks like I'll miss standup

Everything fails, all the time

Monitor Everything

Non-Monotonic Snapshot Isolation: scalable and strong consistency

for geo-replicated transactional systems

Conflict-free Replicated Data Types

Implementing operation-based CRDTs

http://codebetter.com/gregyoung/2010/02/16/cqrs-task-based-uis-event-sourcing-agh

http://martinfowler.com/bliki/CQRS.html

http://github.com/openrtb/OpenRTB

http://akka.io

http://rbmhtechnology.github.io/eventuate

https://github.com/RBMHTechnology/eventuate

http://rbmhtechnology.github.io/eventuate/user-guide.html#commutative-replicated-data-types

http://www.planetcassandra.org/data-replication-in-nosql-databases-explained

http://wikibon.org/wiki/v/Optimizing_Infrastructure_for_Analytics-Driven_Real-Time_Decision_Making

Resources

twitter.com/helenaedelson

github.com/helena

slideshare.net/helenaedelson

Thanks!

leveraging kafka for big data in real time bidding, analytics, ml & campaign management for...

Technology

bidding data and cotract data

apache kafka - · pdf fileoverview what is apache kafka?...

kafka & redis for big data solutions

data loss and duplication in kafka

streaming big data with spark streaming, kafka, cassandra...

internet of things (iot) and big data kafka to the rescue...

developing real-time data pipelines with apache kafka

data pipelines with kafka connect

kafka streams and ksql - itoug · •kafka is popular for...

evaluation of apache kafka in real-time big data pipeline...

solace kafka connectors for pubsub+€¦ · kafka excels at...

processing (big) data the microservice way...

real time data streaming using kafka & storm

reactive fast data & the data lake with akka, kafka, spark

data streaming with apache kafka & mongodb - emea

data pipelines with apache kafka

building streaming data applications using apache kafka

kafka streams: hands-on session - ce.uniroma2.it · kafka...

developing realtime data pipelines with apache kafka

programmatic bidding data streams & druid