operational intelligence webinar | sqlstream | july 2013 series
DESCRIPTION
As the speed of business accelerates, and more data is created every second, gaining a competitive edge relies on transforming massive volumes of log file and other machine-generated data into actionable intelligence in real-time. Leading organizations are leveraging SQLstream's real-time operational intelligence to make smarter decisions faster, and at price they can afford.TRANSCRIPT
Copyright © SQLstream Inc.
Analytics, Predictive Analytics, Prescriptive Analytics:
The Anatomy of Operational Intelligence
The SQLstream July 2013 Webinar series
| 2Copyright © 2013 | +1 877 571 5775 | [email protected]
• Deliver webcasts that explain real-time Big Data and the world of Operational Intelligence in real terms
• Introduction to streaming data management for real-time, high velocity, low latency applications
• Share our thoughts, experiences, industry use cases and examples
MISSION
| 3Copyright © 2013 | +1 877 571 5775 | [email protected]
• July 9 2013 |10:00am PSTAnalytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence
• July 16 2013 |11:00am PSTListen to your Sensors: A Tale of Managing Large Scale Sensor Networks in Real-time
• July 23 2013 |11:00am PSTPredict and Avert: Using Log File Data to Prevent Cybersecurity and Fraud Attacks in Real-time
• July 30 2013 |10:00am PSTNo more CPR for your CDRs: Meet Real-time Traffic Utilization, Billing and Fraud Detection
The Operational Intelligence Series
| 4Copyright © 2013 | +1 877 571 5775 | [email protected]
Analytics, Predictive Analytics, Prescriptive Analytics: The Anatomy of Operational Intelligence
As the speed of business accelerates, more data is created every second.
Gaining a competitive edge relies on transforming massive volumes of log file and other machine-generated data into actionable intelligence in real-time.
Leading organizations are leveraging real-time operational intelligence to make smarter decisions faster, and at price they can afford.
| 5Copyright © 2013 | +1 877 571 5775 | [email protected]
SQLstream, Inc.
History Launched 2009
Over 1.5M lines of code
Multiple deployments across many industries, with top real-world benchmarks
Features Supports all forms of
unstructured and structured data
Accelerates and extends Hadoop & RDBMS
Not limited to SQL
Key innovations Only true streaming
data management platform
Only true standard SQL streaming engine
Covered by 5 broad patents for stream processing
A streaming data management platform for real-time Operational Intelligence from high-velocity Big Data.
Operational Intelligence
| 7Copyright © 2013 | +1 877 571 5775 | [email protected]
Bridging The Chasm
As we move toward a real-time business environment, the capability to process data flows swiftly and flexibly will become increasingly important. SQLstream leads the industry in this kind of capability.
“
”Robin Bloor
Chief Analyst for Bloor Group
Aberdeen’s research has shown that Best-in-Class organizations are demanding access to actionable intelligence faster than ever. This is precisely the growing demand that SQLstream is meeting with their Streaming Big Data Engine, while continuing to bring other attractive features like full Hadoop integration.
“
”Nathaniel Rowe
Leading Analyst for Aberdeen Group
Business Intelligence
Post-hoc Analysis
Data Warehousing
Strategic insights
Operations
Transaction
Processing
Machine Data
Everyday business
Operational Intelligence integrates Operations and BI
Operational IntelligenceOptimizes tactical decisions from real-time actionable
insights
Combines operations data with BI data continuously
Provides Real-time integrated view of the business and
operations
SecurityCompliance
FraudQuality
PromotionAdvertising
Cross-selling
SecurityCompliance
FraudQuality
PromotionAdvertising
Cross-selling
| 8Copyright © 2013 | +1 877 571 5775 | [email protected]
The Information Value Chain
What is happening?
What might happen?
What just happened?
Make stuff happen!
| 9Copyright © 2013 | +1 877 571 5775 | [email protected]
Drivers For Operational Intelligence*Source: Ventana Research Operational Intelligence Benchmark Research, 2013
Analyze complex relationships
“Most organizations do not have technologies that can analyze complex relationships across multiple source systems in real-time, but most are planning to deploy”
Low latency is essential“No latency can be tolerated for security
and network management for example”
Integration is the missing link
“Most Business Intelligence systems are not integrated with operational intelligence data”
Specialized tools perform better
“When organizations use specialized tools they are much more satisfied with the Operational Intelligence initiatives”
Manage riskManage risk
Manage performance
Manage performance
Comply with regulations
Comply with regulations
Identify opps. for improvement
Identify opps. for improvement
59%
59%
58%
58%
54%*Importance of goals in current or planned deployment of technology
| 10Copyright © 2013 | +1 877 571 5775 | [email protected]
Machine-generated Big Data ExplosionHigh volume, high velocity, structured and unstructured data from software platforms, applications and systems
GPSGPS
TelematicsTelematics
IP Networks, VideoIP Networks, Video
Servers, Social Media, SecurityServers, Social Media, Security
Servers, Applications, Storage Servers, Applications, Storage
NetworksNetworks
Machine-generated data will increase Machine-generated data will increase to 42% of all data to 42% of all data by 2020, up from by 2020, up from 11% in 2005.11% in 2005.““The Digital Universe in 2020”The Digital Universe in 2020” IDCIDC
| 11Copyright © 2013 | +1 877 571 5775 | [email protected]
Connecting Machine Data, Connecting The World
DATA SOURCESDATA SOURCESINDUSTRIESINDUSTRIES
| 12Copyright © 2013 | +1 877 571 5775 | [email protected]
Machine DataWhere is the intelligence?
TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342Transaction Log Details
Web Server Logs
CDR Records
Smartphone GPS Updates
{"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str:304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco, time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson
<id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon= -122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing>
<id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing>
<id>1597940125</id><deviceid>0198873465</deviceid><lat>lat=47.647381</lat><lon>lon=-122.326501</lon><time>2013-02-17T15:37:26Z</time><bearing>87.4357</bearing>
[Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down[Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations
TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005, IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60, 234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465
Timestamp
Timestamp
Timestamp
Timestamp
Timestamp
Timestamp
Timestamp
Timestamp
Timestamp
Timestamp
Mobile #Mobile #CustomerCustomer
Mobile #Mobile #Device IDDevice IDTerm
ReasonTerm
Reason
Device IDDevice ID LocationLocation
LocationLocation
Service ProviderService Provider
Fail CodeFail Code
ServerServer
| 13Copyright © 2013 | +1 877 571 5775 | [email protected]
Real-time Big Data Heat Map
Hot right now
Hot, growing fast (< 2 years)
Growing fast (2 – 5 years)
5 – 10 years
> 10 years
Total Cost of Performance
| 15Copyright © 2013 | +1 877 571 5775 | [email protected]
RECORDS PER SECOND
LATE
NCY
Total Cost Of Performance (total COP)The High-Velocity, Low-Latency Tipping Point for Big Data
Patterns Trends MiningConnections
Searches Inventory ReportsStatistics Billing
SOCIAL E-COMM SECURITYTELEMATI
CSTELECOM
TradingAdvertisin
gAlerts
Detection
Signal
Intelligence
TO
TA
L C
OST
| 16Copyright © 2013 | +1 877 571 5775 | [email protected]
Intelligence
TELECOM
Patterns Trends MiningConnections
Searches Inventory ReportsStatistics Billing
TradingAdvertisin
gAlerts
Detection
Signal
SOCIAL E-COMM SECURITYTELEMATI
CS
RECORDS PER SECOND
TO
TA
L C
OST
LATE
NCY
Total Cost Of Performance (total COP)The High-Velocity, Low-Latency Tipping Point for Big Data
Streaming Data Management
| 18Copyright © 2013 | +1 877 571 5775 | [email protected]
High-velocity Big Data Analytics
Historical queries and data
enrichment
Storing valuable derived streams for future access
Op
era
tion
al In
telli
gen
ce
Continuous Queries over Sliding Time Windows Analysis and Integration of Unstructured and Structured
data Prescriptive Analytics drives Automated Actions
| 19Copyright © 2013 | +1 877 571 5775 | [email protected]
Real-time ArchitectureStreaming Analysis and Integration for Infinite Flows of Unstructured Data in Real Time
Streaming Agent & Adapter Layer + JDBC APIHadoop Streaming
Query Planner & Optimizer for MPP ExecutionSQL
Developer Tools
Platform Administration
Streaming SQL Real-time Applications
Real-time Dashboards & Visualization
Impala SQL
HBase
HDFS / MR
Hadoop for Stream Persistence, Enrichment & Replay (Optional)
Any external data warehouse, operational system and enterprise
platform
| 20Copyright © 2013 | +1 877 571 5775 | [email protected]
CLEANING & CLEANING & FILTERINGFILTERING
CLEANING & CLEANING & FILTERINGFILTERING
STREAMING STREAMING ANALYTICSANALYTICS
STREAMING STREAMING AGGREGATIONAGGREGATION
CONTINUOUSCONTINUOUSINTEGRATIONINTEGRATION
Internet of Internet of EverythingEverythingFraud Fraud
PreventionPreventionNetwork Network
MonitoringMonitoring
CybersecurityCybersecurity QoS andQoS andQoEQoE
An Operational Intelligence Platform
| 21Copyright © 2013 | +1 877 571 5775 | [email protected]
SELECT STREAM ROWTIME, url, numErrorsLastMinute FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute, AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute, STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute FROM ServiceRequestsPerMinute WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) )
AS S WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute;
The Power Of Streaming SQL
BUSINESS NEED:Predicting run-away applications before
resource consumption becomes an issue.
BLAZING SPEED:Processing millions of
records per second on low-end servers.
| 22Copyright © 2013 | +1 877 571 5775 | [email protected]
The SQLstream s-Streaming Product Portfolio
s-ServerData Management Platform for Streaming Big Data
s-AnalyzerReal-Time Visualization for Streaming
Operational Intelligence
s-TransportGeo-Analytics for Location-based
Applications
s-VisualizerAdvanced
Visualization
s-Clouds-Server EC2 AMI Deployment
Industry Use Cases
| 24Copyright © 2013 | +1 877 571 5775 | [email protected]
Real-time Operations – IT And BeyondHigh Velocity: The Next Frontier For Big Data
INTERNET OF INTERNET OF THINGSTHINGS
NETWORKS & NETWORKS & SERVICESSERVICES
SECURITYSECURITY
Machine-to-MachineMachine-to-Machine
Cars as SensorsCars as Sensors
Web SecurityWeb Security
Banking & FinanceBanking & FinanceTelecommunicationsTelecommunications
Customer LoyaltyCustomer Loyalty
| 25Copyright © 2013 | +1 877 571 5775 | [email protected]
Log Analytics For Security Intelligence
Identity Theft Monitoring
» Real-time detection and prevention
» Real-time log and web feed analysis on a massive scale
Fraud prevention
» Identify and block suspicious account activity
» Monitor transaction and activity logs in real-time
Cybersecurity Attacks
» Identify patterns of activity for Advanced Persistent Threats
» Log file and security device monitoring, with geospatial analytics
| 26Copyright © 2013 | +1 877 571 5775 | [email protected]
Internet Of Everything – Use Case Examples
Intelligent Transportation
» Reduce congestion and travel times; improve traveller experience
» Real-time flow and congestion prediction from GPS data
Telematics (V2V Infrastructure)
» Reduce ‘walkaway’ events and warranty costs
» Real-time vehicle health monitoring every 10 ignition cycles
M2M
» Monetization of M2M data feeds
» Real-time wireless sensor analytics and aggregation
| 27Copyright © 2013 | +1 877 571 5775 | [email protected]
Telecommunications – Use Case Examples
CDR Analytics
» Real-time charging and QoS / QoE monitoring
» CDR collection, session reconstruction and analysis at scale
4G Wireless Network Performance
» Identify and prevent QoS exceptions
» 4G cell performance data analysis in real-time
| 28Copyright © 2013 | +1 877 571 5775 | [email protected]
DATA EXPLOSION
COMPLEXITY
BUSINESS AGILITY
Streaming Operational Intelligence
Eliminates the development risk•Simplifies development, rapid time to market
Lowest Cost of Performance for Real-time Apps•Efficient scale-out for high velocity data
Adding new applications on the fly•With dynamic sharing of data streams across Apps