open-bda hadoop summit 2014 - mr. krish krishnan (driving business value – big data platforms)
TRANSCRIPT
Driving Business Value – Big Data Platforms
Hadoop Summit
Karachi, Pakistan
Nov 18th 2014
Content and services via connected products Page 4 Hortonworks © 2014
New routes to market via intelligent objects
Everything
has a URL
Remote sensing of objects and environment
Augmented reality
Building and infrastructure management Cameras and
microphones widely deployed
Over 50% of Internet connections are things:
2011: 15+ billion permanent, 50+ billion intermittent
2020: 30+ billion permanent, >200 billion intermittent
Situational decision support Source: Gartner Keynote at Hadoop Summit 2013
Emerging Trends 2010-2015
Page 1Nick Jones and Alexander Linden26A, ESC17, 11/05, AE
© 2005 Gartner, Inc. and/or its affiliates. All rights reserved. Reproduction of this publication in any form without prior written
permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims
all warranties as to the accuracy, completeness or adequacy of such information. Gartner shall have no liability for errors, omissions
or inadequacies in the information contained herein or for interpretations thereof. The reader assumes sole responsibility for the
selection of these materials to achieve its intended results. The opinions expressed herein are subject to change without notice.
Disruption: It's Not Just About Technologies
Identity/AccessManagement
Speech Recognition
Computer-BrainInterface
TruthVerification
ObjectIdentification
Linux
Wikis
Grid ComputingTablet PC
InformationExtraction
Location-AwareServices
Semantic Web
Service-OrientedArchitecture
UnifiedCommunications
Ultra Wide Band
RFID
SocialNetworkAnalysis
Web-Services-Enabled BusinessModels
Electronic Ink/Digital Paper
WebServices
InstantMessaging
VoIP
Sensor Networks
Smartphone
Really SimpleSyndication
Augmented Reality
4G Wireless
Blogging
Podcasting
IP Television
LocationSensing
Design Innovation
ProactiveTransparency
PersonalizedPricing
Counterfeit Reality
CollectiveIntelligence Perfect Recall
Real-Time Enterprise
UbiquitousAccess
Smart Objectsand AmbientIntelligence
Privacy Redefined
Global Sourcing
BusinessProcessManagement
Voice/DataConvergence
Microcommerce
Self-Sufficiency
GreenfieldBusiness
Global Micro-Business
Seamless Serviceto Self-service
Feedback Society
SemanticConnectivity
EmergingTechnologies
EmergingCapabilities
EmergingBusiness Models
Innovation is no longer the province of a few leading-edge companies — it is the lifeblood required for the
survival of any enterprise. Business and technology planners must understand how to harness the disruptive
potential of technology advances through a realistic assessment of emerging capabilities and IT-enabled
trends, and their impact on business and society. Planners must also acknowledge that disruptions arise not
just from new technology, but also from new applications or convergence in existing technologies that
drive new capabilities and new business models.
This emerging trends presentation identifies the most disruptive trends arising from emerging information
technology, and assesses their potential impact on the workforce, consumers, business, government and
society in the five to 10-year time frame.
@copyright Sixth Sense Advisors 2014 3
From Transactional to Behavioral
@copyright Sixth Sense Advisors 2014
The challenge facing the business today is the ability to influence the buyer decisions in a
window of opportunity that does not last long. The analytics available at a personalization
level drives the buyer whether it is choosing a Doctor or buying a Donut.
To compete in this new era, businesses need to be driven by data and analytics, which
are largely different from traditional transactions and campaigns
The “Gen Z” and “Millinieal Generation” of buyers will not be swayed by traditional
engagement models of selling products and services
4
A Growing Trend
@copyright Sixth Sense Advisors 2014 5
Executive Ask
• Why is our competition performing better in the same markets as us?
• What is the real revenue impact to the organization?
• Why do our campaigns not predict the accurate revenue lift?
• How can we improve our brand and align to customer sentiment?
• Who can help us get to the bottom of the data abyss?
• When do we get the real insights from Analytics?
@copyright Sixth Sense Advisors 2014 6
Forces Shaping Business
@copyright Sixth Sense Advisors 2014 7
Business Evolution – Product to Customer Focus Shifts
@copyright Sixth Sense Advisors 2014
Image source - internet
8
Decision Support – Now & Then
@copyright Sixth Sense Advisors 2014
Customer Promotion
Social Presence
Behavior Analytics
Competition Market
Value
Transactions
Price Product Channel
Sales
Promotion
Market
Price
Transactions
Customers
9
Innovation as Way of Life
@copyright Sixth Sense Advisors 2014
Old Brick
New Brick
10
State of Data
@copyright Sixth Sense Advisors 2014 11
Big Data
• Data that cannot be processed using traditional data processing techniques due to size, scale and formats
• Its beyond just that
– Complexity
– Ambiguity
– Veracity
@copyright Sixth Sense Advisors 2014 12
Noise vs Value
@copyright Sixth Sense Advisors 2014
Is Big Data Noise or Value?
Is Big Data a passing hype?
Is there real value behind Big Data?
Is there any measurable and actionable
insight from Big Data?
Is there a need to invest in this now?
Image source - internet
13
@copyright Sixth Sense Advisors 2014 14
Example
@copyright Sixth Sense Advisors 2014
Tweet: @jdoe – very disappointed with @united @checkin lousy svc,
bad mgmt, long lines #fail.
20000 retweets. 4 hours ago
IT Perspective Text of 140 chars will be stored as string. The data model for this will be a table with source, content, datetime.
Business Perspective User – JDoe Brand – United Sentiment – Negative Process – Check-in Time – Waiting in long lines Impact – shared 20000 times in 4 hours
15
Why Does Big Data Differ
@copyright Sixth Sense Advisors 2014
IT does not know data
Business knows the intelligence to be applied to the data to derive value
Business knows how to discover data patterns (manual and automated)
Business understands the semantics better
Business can perform data interrogation in an experiment and associate rules of engagement early on for data usefulness
Business can sift the data to curate the context
Big Data needs to be curated to be useful
16
@copyright Sixth Sense Advisors 2014 17 Source: McKinsey Presentation on Marketing
Big Data Challenges
@copyright Sixth Sense Advisors 2014
Complex Ambiguous Formats /
Availability
Minimally Organized
Complex Data Quality Issues
Metadata / Semantics
Needs Intervention
Needs Discovery /
Contextualization Needs Analysis
18
Perspective
@copyright Sixth Sense Advisors 2014
Data
Insights
19
The Paradigm Shift
@copyright Sixth Sense Advisors 2014
IT
• Facilitate
• Maintain
• Support
• Manage
Business
• Driver
• Budget Sponsors
• Program Owner
• Define & Consume
IT
• Driver
• Program Owner
• Budget Sponsor
• Maintain
• Support
Business
• User
Data Warehouse Big Data
20
Analysis
@copyright Sixth Sense Advisors 2014 21
Processing – The Details
@copyright Sixth Sense Advisors 2014
Tag
Categorize
Classify
Cleanse (Data Quality Rules)
Semantic Integration
Measure
Visualize
22
Big Data Platforms
Future Visualization
@copyright Sixth Sense Advisors 2014 23
Big Data Architecture
@copyright Sixth Sense Advisors 2014
Big Data Sources
Hadoop (and / or NoSQL)
Traditional Data
Sources
ETL ELT CDC
Staging Or ODS
ETL ELT EDW
BI Analytics
Discovery
Data Mining Algorithms Acquire
Big Data Analytics
24
@copyright Sixth Sense Advisors 2014 25
Infrastructure In
fras
tru
ctu
re
Ch
alle
nge
s &
So
luti
on
s
Challenges To Address
Semantic Data Integration
Compression & Storage
High Capacity Warehouse
Security and Governance
Scripting and Development Tools
Complex Event Processing
Solutions Available Today
Columnar Databases
Workload Optimization
Analytic Accelerators
Hadoop / Map Reduce
No SQL Engines
Stream Computing
@copyright Sixth Sense Advisors 2014 26
Big Data Analytics
@copyright Sixth Sense Advisors 2014
Existing EDW
Machine Learning
Text Mining Analytics
Coding & Learning
Semantic Knowledge Base Metadata & Semantic
Layer
27
28
Dealing with Big “Data Problems”
Data silos in disjointed systems
Multiple data sources - overlapping, conflicting
Timely processing of large volumes of data
Partial, inaccurate, inconsistent.. data
28
Single Data Platform
Reports
Analytical DBMS
Analytics Cluster
Data Asset Catalog
Analytical DBMS
Dashboards
Data Discovery
Interactive Queries
Batch Queries
Web Applications
Activity Logs
NoSQL
Reference Data
Device Apps
Probes
3rd Party
Device
User Profile
POI, Map
Activity Sensor
Dat
a In
take
ETL,
dat
a cr
un
chin
g,
attr
ibu
tio
n, M
L A
lgo
rith
ms
Agg
rega
tio
n
HDFS
Analytical DBMS
Big Data Analytics Platform Data Flows
29 @copyright Sixth Sense Advisors 2014 29
Analytics
• Behavioral Analytics
• Cohort Analytics
• Collections Analytics
• Competitive Analytics
• Financial Services Analytics
• Fraud Analytics
• Marketing Analytics
• Pricing Analytics
• Sales Analytics
• Risk Analytics
• Supply Chain Analytics
• Talent Analytics
• Channel Analytics
• Logistics Analytics
@copyright Sixth Sense Advisors 2014 30
Trends 2015
• Business Users Drive Big Data Initiatives
• Analytics is the “new” center stage
• Data Discovery and Storyboard Patterns Emerge Stronger
• Hadoop-Based Data Lakes / Swamps Unite with Data Warehouses
• Predictive Analytics Lend Fresh Insight From Big Data Explorations
• Prescriptive Analytics Bring New Insight to Current Business Processes
@copyright Sixth Sense Advisors 2014 31
Measuring Value
Increasing Revenue /
Opportunity / Market
Lowering Costs / Risks
/ Maintenance
Value
@copyright Sixth Sense Advisors 2014 32
Critical Success Factors
@copyright Sixth Sense Advisors 2014
Business needs to own and execute the Big Data program
Data collection and discovery is the most critical step
Metadata is needed to process the data prior and post Data Warehouse
integration
Data quality can be processed by integrating Taxonomies
Data visualization is needed to discover data
Metrics and Metadata will be the bridge to integrate to the Data Warehouse
33
Questions
@copyright Sixth Sense Advisors 2014 34
Thank You
Contact
Twitter: @datagenius
Images Source - Internet
@copyright Sixth Sense Advisors 2014 35
References
• Gartner
• McKinsey
• MIT Innovation Labs
@copyright Sixth Sense Advisors 2014 36