how comcast turns big data into real-time operational insights raanan dagan, big data solutions,...
TRANSCRIPT
How Comcast Turns Big Data into Real-Time Operational InsightsRaanan Dagan, Big Data Solutions, Splunk
Patrick Shumate, CDN Engineering, Comcast
Copyright © 2012 Splunk Inc.
What We’ll Talk About
Supporting the Anytime, Anywhere Network
Splunk and Big Data
Comcast’s Universal Database Initiative
Going for Gold – the London Olympics
2
• Company– Founded 2004, first software release in
2006– HQ: San Francisco, CA– Regional HQs: Hong Kong, London– Over 600 employees, in 8 countries
• 4,400+ Enterprise Customers– Customers in over 80 countries– 54 of the Fortune 100
• One of nation's leading providers of entertainment, information & communications products and services
The Comcast Cable Team
4
ProductEngineering
ProductApplication
Services
VideoSystemServices
CDNEngineering
CDN Engineering: Software Development, Selection and Management Across Services
Search VSS: Centralized machine data collector for real-time monitoring, analytics, event correlation, reporting and dashboards
5
Supporting an Anytime, Anywhere Network
6
The Challenge
Comcast – UDB Before Splunk
7
Turning This
8
To These
Requirements for Universal Database
9
Universal Database(UDB)
• High volume of data from many systems along a complex workflow
• Developers expressing artistic prerogative on log formats
• Many different data sources and formats
• Drive operational intelligence• Improve user experience• Troubleshooting, root cause analysis• Track and measure success• Reports, alarms
Caller ID Metadata Distribution
STB Menus
MenuEntitlement
Input Requirements Output Requirements
Big Data Comes from MachinesVolume | Velocity | Variety | Variability
GPS,RFID,
Hypervisor,Web Servers,
Email, MessagingClickstreams, Mobile,
Telephony, IVR, Databases,Sensors, Telematics, Storage,
Servers, Security Devices, Desktops
Machine-generated data is one of the fastest growing, most complex
and most valuable segments of big data
10
What Does Machine Data Look Like?
11
Sources
Care IVR
Middleware Error
Order Processing
Machine Data Contains Critical Insights
12
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Sources
Care IVR
Middleware Error
Order Processing
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
Splunk: The Platform for Machine Data
13
Insight and Visualizations for Executives
Statistical Analysis
Proactive Monitoring
Search and Investigation
Machine Data Operational Intelligence
Splunk storage - Hadoop
Customer Facing Data
Outside the Datacenter
ApplicationsWeb logsLog4J, JMS, JMX.NET eventsCode and scripts
NetworkingConfigurationssyslogSNMPnetflow
DatabasesConfigurationsAudit/query logsTablesSchemas
Virtualization & Cloud
HypervisorGuest OS, AppsCloud
Linux/UnixConfigurationssyslogFile systemps, iostat, top
WindowsRegistryEvent logsFile systemsysinternals
Logfiles Configs Messages Traps Alerts
Metrics Scripts TicketsChanges
Click-stream dataShopping cart dataOnline transaction data
Manufacturing, logistics…CDRs & IPDRsPower consumptionRFID dataGPS data
Splunk Collects and Indexes Machine DataNo upfront schema. No RDBMS. No custom connectors.
14
• Refine transactions into readable logs
• 10s TBs of multi event, multi-line transactions
Universal Database Use Case
Forwarder
Splunk visualize and report on Hadoop dataUDB
15
Before Splunk• 100G of data - monitoring and responding to errors cumbersome
and prone to false positives• KPI extraction near impossible
16
UDB After Splunk
17
“Universal Database” Video back office
Pipe the access logs into SplunkFind the errorsBuild the alarmsDefine the KPIBuild the dashboards!
Splunk Has Four Primary Functions
Searching and Reporting (Search Head)
Indexing and Search Services (Indexer)
Local and Distributed Management (Deployment Server)
Data Collection and Forwarding (Forwarder)
A Splunk install can be one or all roles…
18
Splunk Components and Scalability
Send data from 1000s of servers using combination of Splunk Forwarders, syslog, WMI, message queues, or other remote protocols
Auto load-balanced forwarding to as many Splunk Indexers as you need to index terabytes/day
Offload search load to Splunk Search Heads
19
Analyzing Heterogeneous Data
No data normalizationAutomatically handles timestampsParsers not requiredIndex every term & pattern “blindly”No attempt to “understand” up front
Normalization as it’s neededFaster implementationEasy search languageMultiple views into the same data
Knowledge applied at search-timeNo brittle schema to work aroundMultiple views into the same dataFind transactions, patterns and trends
Universal Indexing
Late Structure Binding
Analysis and Visualization
Rapid time-to-deploy: hours or days
20
Real-time Analytics
DataPa
rsin
g Q
ueue
Parsing Pipeline• Source, event typing• Character set
normalization• Line breaking• Timestamp identification• Regex transforms
Indexing Pipeline
Real-time Buffer
Raw dataIndex Files
Real-time Search Process
Monitor Input
Inde
x Q
ueue
TCP/UDP Input
Scripted Input SplunkIndex
21
Splunk Search Processing LanguageLots of random “hypothetical examples” from our Mugs
22
Operational Intelligence for IT and Business UsersWeb Intelligence
Application Management Business Analytics
Security & Compliance
LOB Owners/Executives
CustomerSupport
SystemAdministrator
IT Operations Management
OperationsTeams
SecurityAnalysts
IT Executives
Development Teams
Auditors
Website/BusinessAnalysts
23
Better Interoperability Drives Time-to-value
24
Splunk Hadoop ConnectReliable Data ExportImport Hadoop Data
Splunk App for HadoopOpsEnd-to-end monitoring, troubleshooting , analysis of Hadoop environment
>>>>
Real-time Collection and
Analysis
Dashboards, Reports,
Access Controls
>>
25
Splunk Hadoop Connect
Delivers reliable integration between Splunk and Hadoop
Export events collected and aggregated in Splunk to HDFSExplore and browse HDFS directories and filesImport and index data from HDFS for secure searching, reporting, analysis and visualizations in Splunk
Splunk App for HadoopOps
26
End-to-end monitoring and troubleshooting for Hadoop
Monitoring of entire Hadoop environment (Network, Switch, Operating System and Database)
Integrated alerting to track and respond to activities from MapReduce to the individual node in the cluster
Centralized real-time view of Hadoop nodes using intuitive heatmap display
Splunk Big Data Solution
Product-basedSolution
Performance at scale
Integrated and End-to-end
Easy to download and deploy
Pre-integrated, end-to-end functionality
Enterprise-grade features
Proven at multi-terabyte scale per day
Upwards of PB under management
4,000+ customers
Collects data from tens of thousands of sources
Advanced real-time and historical analysis of data
Fast, custom visualizations for IT and business users
Developer APIs SDKs
27
Splunking NBC Olympics Coverage
28
24x7Coverage
1,700 Assets
245Event Replays
219M Americans watched NBC's Olympics coverage
27.5MVOD Views
Data Splunked 24 hours a day for 21 Days during Olympics
Search VSS: Primary fault detection, alarming and reporting console for all Olympic content
NBC Olympics - Results
29
Content Management Team
NBC Olympics - ResultsOn Demand-Online• Real-time watch lists for active content
– How many customers watching what– Impact of Editorial promotion– “viral” content
• CDN Management– Finding, reporting, monitoring vendor
bugs• CDN Capacity Planning
– Monitoring throughput– Cache capacity evaluation– Time-to-serve monitoring
30
Combine technologies to deliver better results – faster
Use Hadoop for batch processingUse Splunk for real-time processing
31
Comcast – Key Takeaways
Summary - Splunk Big Data Solution
Product-basedsolution
Performance at scale
Integratedend-to-endreal-time
32
Come to the Splunk booth to see a demo of new Splunk-Hadoop integrations
Copyright © 2012 Splunk Inc.
Thank You
splunk.com/bigdata