growing into a proactive data platform
TRANSCRIPT
Yaar Reuveni & Nir Hedvat
Becoming a Proactive Data Platform
Yaar Reuveni
• 6 Years at Liveperson• 1 Reporting & BI• 3 Data Platform• 2 Data Platform team lead• I love to travel• And
Nir Hedvat
• Software Engineer B.Sc• 3 years as a C++ Developer
at IBM Rational Rhapsody™• 1.5 years at LivePerson• Cloud and Parallel Computing
Enthusiast • Love Math and Powerlifting
Agenda
• Our Scale & Operation• Evolution in becoming proactive
i. Hope & Low awarenessii. Storming & Troubleshootingiii. Fortifyingiv. Internalization & Comprehensionv. Being Proactive
• Showcases• Implementation
Our Scale
• 2 M Daily chats• 100 M Daily monitored visitor sessions• 20 B Events per day• 2 TB Raw data per day• 2 PB Total in Hadoop clusters
• Hundreds producers * event types * consumers
LivePerson technology stack
Stage 1: Hope & Low awarenessWe built it and it’s awesome
Online producer
Offline producer
local files
DSPT Jobs
Raw Data
* DSPT - Data single point of truth
Stage 1: Hope & Low awarenessWe’ve got customers
Dashboards
Data ScienceAppsReporting
Data ScienceData AccessAd-Hoc Queries
Stage 2: Storming & TroubleshootingYou’ve got NOC & SCS on speed dial
Issues arise:• Data loss• Data delays• Partial data out of frame• Missing/faulty calculations for consumers• One producer does not send for over a week
Stage 2: Storming & TroubleshootingYou’ve got NOC & SCS on speed dial
Common issues types and generators:• Hadoop ops• Production ops• Events schema• New data producers• High new features rate (LE2.0)• Data stuck in pipeline• Bugs
Stage 3: FortifyingEvery interruption derives a new protection
Stage 3: FortifyingEvery interruption derives a new protection
Stage 3: FortifyingEvery interruption derives a new protection
• Monitors on jobs, failures, success rate• Monitors on service status• Simple data freshness checks e.g. measure the
newest event• Measure latency of specific parts of the pipeline
Stage 4: Internalization & ComprehensionAuditing requirements
• Measure principles:– Loss
• How much?• Which customer?• What Type?• Where in the pipeline?
– Freshness• Percentiles• Trends
– Statistics• Event type count• Event per LP customer• Trends
Producer
Audit DB
Audit Aggregator
Audit Loader
Stage 4: Internalization & ComprehensionAuditing architecture
ProducerProducer
Events Audit Events
Control
Freshness
Stage 4: Internalization & ComprehensionMechanism
Data
Common Header
Audit Header1. Enrich events with audit metadata
Control Event - Audit aggregation
Common Header
Audit Header2. Send control events per x minutes
Stage 4: Internalization & ComprehensionMechanism
Data
Common Header
Data
Common Header
Data
Common Header
Data
Common Header
Data
Common Header
Data
Common Header
Audit Header
Control Event - Audit aggregation
Common HeaderAudit Header
Control Event - Audit aggregation
Common HeaderAudit Header
Data
Common Header
Audit Header
Data
Common Header
Audit Header
Data
Common Header
Audit Header
Data
Common Header
Audit Header
Data
Common Header
Old Data Flow
Audited Data Flow
Stage 4: Internalization & ComprehensionHow to measure loss?
• Tag all events going through our API with an auditing header:
<host_name>:<bulk_id>:<sequence_id>When:
• host_name - the logical identification of the producer server• bulk_id - an arbitrary unique number that should identify a bulk (changes every X
minutes)• sequence_id - auto incremented persistent number used to identify missing bulks
• Every X minutes send an audit control event:{eventType: AuditControlEvent, Bulks: [{bulk_id:“srv-xyz:111:97”, data_tier:”shark producer”, total_count:785}, {bulk_id:“srv-xyz:112:98”, data_tier:”shark producer”, total_count:1715}]}
Stage 4: Internalization & ComprehensionWhat’s next?
• Immediate gain: enables research loss straight on the raw data
Next:• Count events per auditing bulk• Load into some DB for dashboarding:
In this example, assuming you look at the table after 11:34, and we refer to more than 3 hours as loss, we can see that from server srv-xyz at bulk_id 1a2b3c we can see 750 events were created and only 405+250 = 655 events arrived within 3 hours this means we can detect a loss of 95 events from this server.
Audit metadata Data Tier Insertion time Events count
srv-xyz:1a2b3c:25 Producer 08:34 750
srv-xyz:1a2b3c:25 HDFS 09:05 405
srv-xyz:1a2b3c:25 HDFS 10:13 250
Stage 4: Internalization & ComprehensionHow to measure freshness?
• Run incremental on the raw data• Group events by
– Total– Event type– LP customer
• Per event calculate Insertion time - creation time• Per group:
– Total count– Min, max & average – Count into time buckets (0-30; 30-60; 60-120; 120-∞)
Stage 5: Being ProactiveTools - loss dashboard
Stage 5: Being ProactiveTools - loss detailed dashboard
Stage 5: Being ProactiveTools - loss trends
Stage 5: Being ProactiveTools - freshness
Stage 5: Being ProactiveTools - freshness
Stage 5: Being ProactiveTools - data statistics
Showcase IBug in a new producer
Showcase IIDeployment issue
• Constant loss• Only in one farm• Depends on traffic• Only a specific producer type • From all of its nodes
Showcase IIIConsumer jobs issues
• Our auditing detected a loss in Alpha• Data stuck in a job failure dir• Functional monitoring missed it• We streamed the stuck data
Showcase IVProducer issues
• Offline producer gets stuck• Functional monitoring misses
ImplementationAuditing architecture
Producer
Audit DB
Audit Aggregator
Audit Loader
ProducerProducer
Events Audit Events
Control
Freshness
ImplementationAuditing architecture
Producer
Audit DB
Audit Aggregator
Audit Loader
ProducerProducer
Events Audit Events
Control
Freshness
• Storm topology• Load audit events from Kafka to MySql
Bulk Tier TS Count
xyz:123 WRPA 08:34 750
xyz:123 DSPT 09:05 405
xyz:123 DSPT 10:13 250
ImplementationAudit Loader
Audit DB
Audit Loader
Audit Events
ImplementationAuditing architecture
Producer
Audit DB
Audit Aggregator
Audit Loader
ProducerProducer
Events Audit Events
Control
Freshness
• Load data from HDFS
• Aggregate events according to audit metadata
• Save aggregated audit data to MySql
• Spark implementation
ImplementationAudit Aggregator
HDFS
DBData
Aggregate
#1 #2 #3
∑ #1 = N1 ∑ #2 = N2 ∑ #3 = N3
Collect & Save ZooKeeperOffset
Audit Aggregator jobFirst Generation
• Our jobs work incrementally or manually • Offset management by ZooKeeper
• Failing during saving stage leads to lost offset
• Saving data and offset on same stream
Audit Aggregator jobOvercoming Pitfalls
Audit Aggregator jobRevised Design
HDFS
DBAggregate
#1 #2 #3
∑ #1 = N1 ∑ #2 = N2 ∑ #3 = N3
Collect & Save
Data
Offset
Bulk Tier TS Count
xyz:123 WRPA 08:34 750
xyz:123 DSPT 09:05 405
xyz:123 DSPT 10:13 250
• Precedent - Spark Streaming for online auditing• We see our future with Spark• Cluster utilization• Performance
– In-memory computation– Supports multiple shuffles– Unified data processing: batch/streaming
Audit Aggregator jobWhy Spark
ImplementationAuditing architecture
Producer
Audit DB
Audit Aggregator
Audit Loader
ProducerProducer
Events Audit Events
Control
Freshness
• End-to-end latency assessment
• Freshness per criteria
• Output - various stats
ImplementationData Freshness
Freshness jobDesign
Map
Reduce
HDFS
Total LP Customer Event Type
Min Max Avg BucketsCount
EventEventEventEvent
Freshness jobMechanism
• Driver– Collects LP events from HDFS
• Map– Compute freshness latencies– Segmentize events per criteria by generating
a composite kay
• Reduce– Compute count, min, max, avg and buckets– Write stats to HDFS
Freshness jobOutput usage
Hadoop PlatformOvercoming Pitfalls
• Our data model is built over Avro• Avro comes with schema evolution• Avro data is stored along with its schema• High model-modification rate• LOBs schema changes are synchronized
Producer → Consumer
Hadoop PlatformOvercoming Pitfalls
• MR/Spark job is revision-compiled when using SpecificRecord
• Using GenericRecord removes the burden of recompiling each time schema changes
ImplementationAuditing architecture
Producer
Audit DB
Audit Aggregator
Audit Loader
ProducerProducer
Events Audit Events
Control
Freshness
THANK YOU!
We are hiring
YouTube.com/LivePersonDev
Twitter.com/LivePersonDev
Facebook.com/LivePersonDev
Slideshare.net/LivePersonDev