ibm big data-platform

© 2014 IBM Corporation

Big Data Platform

Arild KristensenNordic Sales Manager, Big Data AnalyticsTlf.: +47 90532591Email: [email protected]


Welcome to the Big Data Opportunity

“The list of life's certainties has gotten longer.

Along with death and taxes we can now include information overload.”


We have for the first time an economy based on a key resource

[Information] that is not only renewable, but self-generating.

Running out of it is not a problem, but drowning in it is.– John Naisbitt

Source, Megatrends, Naisbitt, John, Grand Central Publishing 1988

We are not suffering from Information Overload. We are suffering from Filter Failure.

– Clay Shirky

Sourcehttp://www.ted.com/talks/view/lang/en//id/575


Welcome to the Big Data Opportunity

Research firm IDC expects Big Data to grow from $3.2 billion in 2010 to $16.9 billion in 2015?

?by 2015 we'll see 4.4 million jobs devoted to the global support of Big Data?

?each IT job created by Big Data will generate three more positions outside of IT.


Big Data Analytics And Natural LanguageCognitive: The Next Wave of Disruptive Technology


Understands natural language and human style communication

Adapts and learns from training, interaction, and outcomes

Generates and evaluates evidence-based hypothesis

1 2

3

• Understands me

• Engages me

• Learns and improves over time

• Helps me discover

• Establishes trust

• Has endless capacity for insight

• Operates in a timely fashion

Watson combines transformational capabilities to deliver a new world experience using cognitive computing

Watson:


IBM Watson

family

IBM Watson

Solutions

IBM Watson

Transformation

IBM Watson

Foundations

IBM Watson

InnovationsProvides the big data and analytics

capabilities that fuel Watson

Products based on Watson’s core attributes and

capabilities

APIs, tools, methodologies, SDKs, and infrastructure that

fuels Watson

Bespoke solutions designed to meet some of industries most demanding needs leveraging

cognitive capabilities

IBM Watson

Ecosystems

The Watson Developer Cloud, Watson Content Store and Watson Talent Hub driving innovation from partners

Introducing the IBM Watson family


How is Big Data transforming the way organizations analyze information and generate actionable insights?


Paradigm shifts enabled by big dataLeverage more of the data being captured

TRADITIONAL APPROACH BIG DATA APPROACH

Analyze small subsets of information

Analyze all information

Analyzedinformation

All available information

All available informationanalyzed


Paradigm shifts enabled by big dataReduce effort required to leverage data


Carefully cleanse information before any analysis

Analyze information as is, cleanse as needed

Small amount of carefully

organized information

Large amount of

messy information


Paradigm shifts enabled by big dataData leads the way—and sometimes correlations are good enough


Start with hypothesis andtest against selected data

Explore all data andidentify correlations

Hypothesis Question

DataAnswer

Data Exploration

CorrelationInsight


Paradigm shifts enabled by big dataLeverage data as it is captured


Analyze data after it’s been processed and landed in a warehouse

or mart

Analyze data in motion as it’s generated, in real-time

Repository InsightAnalysisData

Data

Insight

Analysis


Hadoop &

Streaming

Data

New

Sources

Unstructured

Exploratory

Iterative

StructuredRepeatable

Linear

Data

Warehouse

Traditional

Sources

Traditional Approach

Structured, analytical, logical

New Approach

Creative, holistic thought, intuition

Enterprise

Integration

Customer Data

Transaction Data

3rd Party Data

Core System Data

Web Logs, URLs

Social Data

Text Data: emails, chats

Log data

Analytics is expanding from enterprise data to big data, creating new opportunities for competitive advantage

Contact Center notes

Geolocation data


Addressing Client Challenges through Big Data Platform


A New Architectural Approach is Required

Information Integration & Governance

Systems Security

On premise, Cloud, As a service

Storage

New/Enhanced

ApplicationsAll Data

What action should I take?

Decision management

Landing, Exploration and Archive data zone

EDW and data mart zone

Operational data zone

Real-time Data Processing & Analytics What is happening?

Discovery and exploration

Why did it happen?

Reporting and analysis

What could happen?

Predictive analytics and modeling

Deep Analytics data zone What did

I learn, what’s best?

Cognitive


Information Integration & Governance

Actionable insight

Exploration, landing and

archive

Trusted data

Reporting & interactive analysis

Deep analytics & modeling

Data types Real-time processing & analytics

Transaction andapplication data

Machine andsensor data

Enterprise content

Social data

Image and video

Third-party data

Decision management

Predictive analytics and modeling

Reporting, analysis, content

analytics

Discovery and exploration

Operational systems

Information Integration

Data Matching & MDM

Security & Privacy

Lifecycle Management

Metadata & Lineage

IBM Big Data Analytics (Watson Foundations) - One architecture that fits together

BigInsights

Streams

PureData

for

Analytics

DB2 Blu

Watson

Explorer

Cognos

Cognos

SPSSPureData

for

Analytics

PureData

Operational

Analytics


InfoSphere

DataStage

Automatically push transformational processing close to where the data resides, both SQL for DBMS and MapReduce for Hadoop,

leveraging the same simple data flow design process and coordinate workflow across all platforms

“Big Data Expert”


IBM InfoSphere Streams:Get real-time insights from data in-motion


27

Current fact finding

Analyze data in motion – before it is stored

Low latency paradigm, push model

Data driven – bring data to the analytics

Historical fact finding

Find and analyze information stored on disk

Batch paradigm, pull model

Query-driven: submits queries to static data

Traditional Computing Stream Computing

Stream Computing Represents a Paradigm Shift

Real-time Analytics


28

ModifyFilter / Sample

Classify

Fuse

Annotate

Big Data in Real Time with InfoSphere Streams

Score

Windowed Aggregates

Analyze


29

Streams Analyzes All Variety of Data

Mining in Microseconds

(included with Streams)

Image & Video (Open Source)

Simple & Advanced Text

(included with Streams)Text(listen, verb), (radio, noun)

Acoustic

(IBM Research)

(Open Source)

Geospatial

(Included with

Streams)

Predictive

(Included with

Streams)

Advanced

Mathematical

Models

(Included with

Streams)

Statistics

(included with

Streams)

∑population

tt asR ),(

Blue = included with the product

Red = built for Streams and used in

projects but not yet part of the product


30

How is Streams Being Used?

Stock market� Impact of weather on

securities prices� Analyze market data at

ultra-low latencies� Momentum Calculator

Fraud prevention� Detecting multi-party fraud� Real time fraud prevention

e-Science� Space weather prediction

� Detection of transient events� Synchrotron atomic research

� Genomic Research

Transportation� Intelligent traffic

management� Automotive Telematics

Energy & Utilities� Transactive control

� Phasor Monitoring Unit� Down hole sensor monitoring

Natural Systems� Wildfire management� Water management

Other� Manufacturing� Text Analysis

� ERP for Commodities

� Real-time multimodal surveillance� Situational awareness� Cyber security detection

Law Enforcement,

Defense & Cyber Security

Health & Life

Sciences� ICU monitoring� Epidemic early

warning system� Remote healthcare

monitoring

Telephony� CDR processing� Social analysis� Churn prediction� Geomapping


Watson (Data) Explorer IBM Software GroupInformation ManagementBig Data


Watson Explorer solves #1 challenge customers face in Big Data:

Unlocking the value of information through a single interface

Create unified view of ALL information for real-time monitoring

Identify areas of information risk & ensure data

compliance

Analyze customer analytics & data to unlock true

customer value

Increase productivity & leverage past work

increasing speed to market

Improve customer service & reduce

call times

InfoSphereData Explorer

• Analyzes structured & unstructured data—in place

• Unique positional indexing• Unlimited scalability• Advanced data asset navigation• Pattern clustering

• Virtual documentsContextual intelligence

• Text analytics• Secure data integration • Query transformation• Easy-to-deploy big data applications • User-friendly customisable interface

Providing unified, real-time access and fusion of big

data unlocks greater insight and ROI

Zoom inZoom out

12/05/201432


Watson Explorer Application Architecture

User Profiles

360O View

Applications

Information

Discovery

Applications

Big Data

Applications

Discovery & navigation

applications

Web Results

FeedsSubscriptions

Federated Query RoutingApplication Framework

Authentication/AuthorizationQuery transformation

PersonalizationDisplay

Meta-Data

User ProfilesApplication layer managing user

interactions, apps, creating context, routing queries

Thesauri

Clustering

Ontology Support

Semantic Processing

Entity Extraction

Relevancy

Text AnalyticsSearch Engine Metadata Extraction

Faceting

BI

Tagging

Taxonomy

Collaboration

Processing layer for indexing, analysis & conversion

CM, RM, DM RDBMS Feeds Web 2.0 Email Web CRM, ERP File Systems

Connector

Framework

Framework for accessing data

sources

12/05/201433


Highly relevant, secure &

personalized results

Access all sources

or individual source

Refinements based

on metadata

Dynamic

categorization

Narrow down results set

Information Navigation, Discovery & Insight Through One InterfaceLive link here

Setup alert to

notify change

Identify topical experts

Tag results

Rate results

Comment results

Store &

share results


Big Data Use cases


Top sources of information used as part of initial big data efforts –typically start with data already being captured

Source: The real world use of Big Data, IBM

& University of Oxford

Big data sources

Respondents with active big data efforts were asked which data sources are currently being collected and analyzed as part of active big data efforts within their organization.

88%

73%

59%

57%

43%

42%

42%

41%

41%

40%

38%

34%

92%

81%

70%

65%

27%

19%

36%

47%

32%

0%

21%

22%

Transactions

Log Data

Events

Emails

Social Media

Sensors

External Feeds

RFID Scans or POS Data

Free-form Text

Geospatial

Audio

Still Images / Videos

Banking & Fin Mgmt respondents

Global respondents

3

6


Big Data ExplorationFind, visualize, and understand all big data for improved decision making

Enhanced 360o Viewof the CustomerView all internal and external information sources to know everything about your customers

Operations AnalysisAnalyze a variety of machine data for improved business results

Data Warehouse ModernizationModernize the data warehouse with new technology: in-memory, stream computing, Hadoop, appliances, while building confidence in all data

Security Intelligence ExtensionLower risk, detect fraud and monitor cyber security in real-time

Big Data Use Cases


Arild Kristensen IBM Norway

Nordic Sales Manager Forusbeen 10

Big Data Analytics 4033 Stavanger

IBM Software Group Mobile: +47 90 53 25 91

Information Management [email protected]

linkedin.com/pub/arild-kristensen/34/96b/184twitter.com/ArildWK

www.ibmbigdatahub.com

www.analyticszone.com

ibm big data-platform

Data & Analytics

big data data

big data leverage

ibm corporation14

ibm corporation16

ibm corporation5

ibm corporation3

big data analytics tlf

global support of big