real time big data framework
TRANSCRIPT
Real Time Big Data Framework
Sharath Hosagrahar
Lead Architect, Red Hat Global Partners & Alliances Organisation
8 October 2015
Source: http://blogs.ifsworld.com/2013/02/how-will-big-data-influence-your-finance-team/
Simple Definition
Big data is a collection of data from traditional and digital
sources inside and outside your company that represents a
source for ongoing discovery and analysis
• Collected from various sources
• Analyze and act on data separately
• Using past data to make strategic decisions
• Batch processing driven
BIG DATA AT REST
• Similar collection process as big data at rest
• Analytics occur in real time as the event happens
• Common in Telco e.g. analyzing mobile phone subscriber usage patterns
• Complex Event Processing with low latency
BIG DATA IN MOTION
• Real time data analysis and as a result, real time action
• Captured data can be stored in a big-data store for batch analysis
• Batch analysis to identify and model customer segments
• Real time analysis to act on an identified customer segment
• Combining batch and real time analysis provides maximum actionable business insights
COMBINING THE TWO
• Analyze large quantities of stream data in real time
• Scale out architecture
• A flexible analytical approach using JBoss BRMS
• Virtual integration with structured and unstructured data and connects to Internet of Things (IoT) devices
• Open Source & Standards Based
• Proven Success
HIGHLIGHTS Agile business insights and actions through real time analysis and integration of
data in motion and data at rest
• Real Time Marketing / Promotions
– Upsell/cross-sell
• Social media analysis
– Trends and News
• Location Based Services (LBS)
• Network Channel Analysis
ACTUAL CUSTOMER USE CASES
• Distributed execution of Map-Reduce workloads based on data from various sources performed within JBoss Data Grid – Faster distributed processing in memory and within network – More complex processing scenarios can be achieved by combining
this workload with the decision management capabilities within JBoss BRMS
FEATURE 1: REAL TIME MAP REDUCE
Real-Time Processing Short Cycle Execution
Keyword extraction
Keyword extraction
Keyword extraction
Sum
Sum
Sum
Relay
receive
receive
receive morphological analysis
morphological analysis
morphological analysis
Watch the word ranking
Rule matching
• Modify decision rules in real time • Accumulate decision knowledge as rules and evolve
into an intelligent operational decision engine • Short Cycle Execution
FEATURE 2: REAL TIME DECISIONING
Real-Time Processing Short Cycle Execution
Keyword extraction
Keyword extraction
Keyword extraction
Sum
Sum
Sum
Relay
receive
receive
receive morphological analysis
morphological analysis
morphological analysis
Watch the word ranking
Rule matching
• Build virtual data views using JBoss Data Virtualization
– Integrate real time analysis of JBoss Data Grid with master information from RDBMS via JBoss Data Virtualization
– Analyst can view information via dashboard or existing business intelligence or custom applications
– Developers/Administrators can build virtual tables with an IDE-based tool or standard SQL notations
FEATURE 3: REAL TIME BUSINESS INSIGHTS
FEATURE 3: REAL TIME BUSINESS INSIGHTS
Data Virtualization
RDBMS
JBoss Data Virtualization
RDBMS
JBoss Data Grid
JBoss Data Grid
Deploy
Data Integration/ Virtualization
CALL CENTER SYSTEM (AS-IS)
IVR
call
call
call
call
call
call
Tele Operator
End User
Call Log Call Log Call Log
Call Log
Call Log Call Log Call Log
Call Log
Call Log Call Log
Call Log Call Log
Call Log Call Log
Automatic
input
input
input
Write Program Hadoop
Batch Platform
Report
Manager
Analyze Report per 1 day
Call Center System
CALL CENTER SYSTEM (TO-BE)
IVR
call
call
call
call
call
call
Tele Operator
End User
Call Log Call Log Call Log
Call Log
Call Log Call Log Call Log
Call Log
Call Log Call Log
Call Log Call Log
Call Log Call Log
Automatic
input
input
input
Write Program Hadoop
Batch Platform
Report
Manager
Analyze Report per 1 day
Call Center System
JBoss Data Grid
JBoss BRMS
JBoss EAP
JBoss DV
Alert over threshold
Hive
Real Time Big Data Framework