moving targets: harnessing real-time value from data in motion

Grab some

coffee and

enjoy the

pre-show

banter before

the top of the

hour!

The Briefing Room

Moving Targets: Harnessing Real-Time Value from Data in Motion

Twitter Tag: #briefr The Briefing Room

Welcome

Host: Eric Kavanagh

[email protected] @eric_kavanagh


  Reveal the essential characteristics of enterprise software, good and bad

  Provide a forum for detailed analysis of today’s innovative technologies

 Give vendors a chance to explain their product to savvy analysts

  Allow audience members to pose serious questions... and get answers!

Mission


Topics

February: DATA IN MOTION

March: BI/ANALYTICS

April: BIG DATA


Parmenides and the Truth of Now

There is no tomorrow

There is no yesterday

There is only today

There is only now


Analyst: David Loshin

David Loshin, president of Knowledge Integrity, Inc, is a thought leader and expert consultant in the areas of data quality, master data management, and business intelligence. David is the author of numerous books and papers on data management, including the “Practitioner’s Guide to Data Quality Improvement.” David is a frequent speaker at conferences and in web seminars. His best-selling book, “Master Data Management,” has been endorsed by data management industry leaders. David can be reached at [email protected], or at (301) 754-6350.


Datawatch

Datawatch began as a BI tool and has developed into a visual analytics platform

  The platform provides visual data analytics and discovery on any type of data, including streaming data

  The suite of products are Datawatch Desktop, Datawatch Server, Datawatch Report Mining Server and Datawatch Modeler


Guest: Dan Potter

Dan Potter is the Vice President of Product Marketing at Datawatch Corporation. In this role, Dan leads the product marketing and go-to-market strategy for Datawatch. Prior to Datawatch, Dan held senior roles at IBM, Oracle, Progress Software and Attunity where he was responsible for identifying and launching solutions across a variety of emerging markets, including cloud computing, visual data discovery, real-time data streaming, federated data and e-commerce.

VISUAL DATA DISCOVERY & STREAMING DATA New Technologies for Real-Time Analytics

Dan Potter Vice President, Product Marketing

NASDAQ: DWCH Pioneer in real-time visual data discovery and self-service data preparation

Global operations and support §  US, UK, Germany, France, Australia, Singapore, Philippines

Extensive global customer base §  93 of the Fortune 100 §  12 of the 15 largest financial institutions

Embedded and resold by leading vendors

About Datawatch

DISCOVER

GOVERN

ACQUIRE

PREPARE

AUTOMATE

Visual Analytics Platform For Any Data at Any Speed

Where Do Real-Time Streams Come From?

•  Internet of Things •  Machine data / log files •  Web clickstreams •  Enterprise applications •  Human generated •  Commercial data

Streaming Visualization Examples

Capital Markets §  Transac'on Cost Analysis § Analyze market data at

ultra-‐low latencies § Momentum Calculator

Fraud preven2on § Detec'ng mul'-‐party fraud §  Real 'me fraud preven'on

e-‐Science §  Space weather predic'on

§ Detec'on of transient events §  Synchrotron atomic research

§ Genomic Research

Transporta2on §  Intelligent traffic

management §  Automo've Telema'cs

Energy & U2li2es §  Transac've control

§  Phasor Monitoring Unit § Down hole sensor monitoring

Natural Systems § Wildfire management § Water management

Other § Manufacturing

§  ERP for Commodi'es

§  Real-‐'me mul'modal surveillance §  Situa'onal awareness

§  Cyber security detec'on §  Emergency Evacua'on

Law Enforcement, Defense & Cyber Security

Health & Life Sciences

§  ICU monitoring §  Epidemic early warning §  Remote healthcare

monitoring

Telephony §  CDR processing §  Social analysis

§  Churn predic'on § Geomapping

Visual Data Discovery

•  Easy for users to author, customize and share

•  Interactive exploration & visually filter results

•  Quickly identify anomalies and outliers with large or in-motion datasets

•  Rich palette of visualizations for static and time series data

Visualize Any Data at Any Speed

Stream Rela2onal NoSQL OLAP Warehouse Hadoop Content

Connect, Federate, Visualize

Data Architectures Evolving

Database Distributed or Hybrid Database

In-‐Memory Database

Streaming Analy'cs

Faster Speed, Faster Insights

Data at Rest

Limitations of Traditional BI



Streaming Analy'cs

Data at Rest

Streaming Data Visualization



Streaming Analy'cs

Datawatch Streaming Data Visualization

•  Connect directly to data in motion •  CEP (IBM Streams, Informatica Rulepoint, Tibco Streambase) •  Hosted IoT platforms (Amazon Kinesis, PTC ThingWorx) •  Message Bus (Informatica UltraMessaging, WebSphere MQ) •  Operational Intelligence Systems (OSIsoft Pi)

•  Purpose built data model optimized for both caching and persistence

•  High density visuals with rendering in milliseconds

Monitor

Analyze

Take Ac2on

Time Series Data •  Traditional BI only looks at buckets of

time •  Day, week, month, year

•  Streaming data is a continuous and has different requirements

•  Second, millisecond, nanosecond •  Time windows •  Time slices •  Playback

•  Complete situational awareness •  Now (streaming) •  Intra-day •  Historic

Predictive & Advanced Analytics

•  Connect to R (Rserv) and Python (Pyro) servers

•  Transform using R and Python

•  Many use cases in IoT (e.g. predictive maintenance, smart logistics, clinical pattern detection etc.)

Modeled and transformed for analysis

Complex File Formats

•  Sensor and machine data often in multi-structured format •  Need to transform, enrich and prepare data

•  Almost no metadata •  For example, wave form visualization from JSON arrays

stored in MongoDB and streaming

23

Log Files

HTML, XML JSON

PDFs

Real-Time Geospatial & Location

•  Real-time (stream) plotting •  Street-level geo maps or

custom SVG files •  Time-series playback

Healthcare Retail

Logis'cs

U'li'es

Customer Challenge

Dozens of risk management systems generating data silos of operational information

Server based solution to visualize integrated risk information in real-time to identify trends and anomalies

Analyze patterns in physiological data that may detect and eventually to predict deadly clinical events

Visualize large volumes of streaming, unstructured data from multiple devices in real-time

Improve yield production and enhance machine reliability in contact lens manufacturing process

Flexible visualization solution highlighting production line yield, leading to a 2% yield increase and 750,000 additional units produced

Real-World, Real-Time Examples

Process and visualize billions of streaming trades per day for leading surveillance and compliance platform

Fully embedded visual data discovery solution that delivers a single consolidated real-time view of trading across venues


Perceptions & Questions

Analyst: David Loshin

Brie%ing Room 02-‐17-‐2015: Considerations for Streaming Analytics

2015-‐02-‐17 David Loshin

Knowledge Integrity, Inc. loshin@knowledge-‐integrity.com

© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350 28

Technology Convergence & Stream Analysis

•  Discovery & Streaming Analy'cs employs a number of key evolving technologies beyond the expected “repor'ng & analy'cs”: –  Data virtualiza'on and federa'on –  Text parsing and text analy'cs –  Seman'c models –  Real-‐'me data inges'on –  Event stream processing –  Embedded rules for monitoring, no'fica'on, and alerts –  In-‐memory processing –  Visualiza'on

•  Con'nued improvements in these technologies will automa'cally improve the quality and speed of real-‐'me stream analy'cs

© 2015 Knowledge Integrity, Inc loshin@knowledge-‐integrity.com (301) 754-‐6350

29

Future Direction of Connected Devices?

•  More “things” will be networked –  Who’d a thunk that thermostats

would be in the first wave of smart devices?

•  Networked “things” will be gegng “smarter” –  More & beher resources at the

device •  Increased open-‐source

standardiza'on –  Including the hardware!

•  Increased ease of programmability expands the community of developers –  A 12-‐year old can program this


30

Considerations

•  The volume and variety of human-‐generated content will con'nue to explode –  This will require increased analy7c intelligence for parsing and filtering

within the network

•  Par'al analy'c computa'ons can be pushed out to the devices –  Move the applica7on to the data, not the data to the applica7on

•  Alerts and no'fica'ons base on the results of intermediate analyses can provide advantage in mul'ple ways –  The same data streams can feed a wide variety of consumer communi7es

•  Streaming analy'cs will become relevant at the personal as well as the business level –  Enable personalized algorithmic stream blending, analysis, and

monitoring/no7fica7on at the mobile device


31

Questions to Explore

•  What has predicated the growth in demand for analyzing streaming data in recent years?

•  What are the types of streaming data that are most frequently subjected to analysis?

•  What are the features of your product that have been most valuable to your customer community, and why?

•  How does your product help business users dis'nguish relevant streaming content from the “noise”?

•  Can you share some insight into how your tool uses in-‐memory processing and manages data in memory?

•  What fundamental differences do you see between the ability to enable analysis of human-‐generated content vs. machine-‐generated streaming content?

•  Can you share thoughts about external constraints that prevent the best opportuni'es for using streaming analy'cs and discovery?

•  What do you see as the next hurdles in enabling business consumers in adop'ng discovery analy'cs for streaming data?

•  Who are the compe'tors and what do you see as the advantages your tool provides over your compe'tors?


32

Following up…

•  www.knowledge-‐integrity.com •  www.dataqualitybook.com •  www.decisionworx.com •  If you have ques'ons, comments,

or sugges'ons, please contact me David Loshin 301-‐754-‐6350 loshin@knowledge-‐integrity.com


33


Upcoming Topics

www.insideanalysis.com

February: DATA IN MOTION

March: BI/ANALYTICS

April: BIG DATA


THANK YOU for your

ATTENTION!

Some images provided courtesy of Wikimedia Commons and Wikipedia

moving targets: harnessing real-time value from data in motion

Technology

market data

type of data

federated data

visual data analytics

master data management

big data twitter tag

areas of data quality

data quality improvement