real-world applications of streaming analytics- streamanalytix webinar

© 2014 Impetus Technologies1

Recorded version available at http://bit.ly/1AeGdxM

WEBINAR

Real World Applications

of Streaming Analytics

http://bit.ly/1AeGdxM



Recent Webcast Recap– Archived on the

Website

Real-time Streaming Analytics for Enterprises based on Apache Storm

Real-time Streaming Analytics: Business Value, Use Cases, and Architectural Considerations




Agenda

Q&A

Why rapid growth and

demand for real-time

analytics

StreamAnalytix –

Product Overview

Real World Case Studies

Business Problem, Solution

Architecture and Outcomes




Brief Intro

• Big Data Solutions & Services company• Unique in depth, expertise – started implementing in 2008

• Proven with customer success

• IP and Products

• We deliver - Business Impact from Big Data Solutions• Technology expertise

• Data Science

• Business Analytics

• Serving Fortune 1000 companies since 1996• Large-scale and mission critical software platforms

• HQ: Los Gatos, CA; 1500 people • Offshore operations in 3 cities in India




Drivers for Real-time Streaming Analytics

Fleet Operations & Logistics Security

Mobile Devices and Apps Energy Industry IT Operations





You and I :

The ‘CUSTOMER’





Multi-channel engagement in

real-time

Context

Sensitive service

Happy customers,

Loyalty, Revenue,

Profits, Growth





Business Operations

Business Analytics

Real-time Streaming Analytics




Real-time Business Analytics – The “Batch Gap”

The batch workflow is too slow

Views are out of date

Not yet

absorbed.Data absorbed into Batch

Views

Now

Time

Just a few hours of data.




t

now

Hadoop works great back

here

Storm works

here

Blended View – Historical and NOW

Blended viewBlended viewBlended View




Big Data and Fast Data Combined

Batch Layer

All dataPre-computed

information

Batch re-compute

Speed Layer

All dataPre-computed

information

Real time

increment

Batch view

Serving Layer

Batch view

Me

rge

Real time view

Real time view

Incoming

Data Query




Poll

Where are you in the process of

implementing real-time streaming analytics?




Enterprise Class Real time Streaming Analytics Platform

A Product developed and offered by




At a Glance

StreamAnalytix is a software platform that enables enterprises to analyze and

respond to events in real-time at Big Data scale. It is designed to rapidly build

and deploy streaming analytics applications for any industry vertical, any data

format, and any use-case




StreamAnalytix Block Diagram




Case Studies - Real World Applications




Case Study 1 – Intelligence Solutions Company

Problem:

Basic Schematic Architecture

Numerous " non-voice "communications





• Classify streaming text in real-time based on topic

• Sentiment Analysis on the stream in real-time

• 250 million messages a day

• Variety: weblogs, chats, emails, tweets etc.

• Accuracy Classification - 99.99% Sentiment analysis - 80%

20 Predefined Categories

"Arts_culture_entertainment" "law_crime_justice" "disaster_accident" "economy_finance" "education" "environment_weather" "health" "lifestyle" "politics" "religion" "science" "society" "sports" "conflict_war" "literature" "computing" "labor" "travel" "governance_government" "human_interest"

Problem statement





• English and Arabic content

• Other languages = “other” (no metadata)

• Data very rawHad CSS and JavaScript filesTo be categorized as “scripts”

• Ingest, Store, Index, QueryMetadata and Raw binary dataPetabytes

• Query SLA – On any 4 hour window "cold data"4 to 5 secondsETSI compliant encryption

Problem statement





Content Extraction and Preprocessing

ClassificationSentiment

Analysis

Tokenization of words based on delimiters (space)

Elimination of all “Stop Words”, non-contributory words

Removal of non-ASCII and Non UTF-8

Models built offline and scoring online





• 20 categories; Multiple labels if applicable

• Semantic similarity approach based on matrix

decomposition

• Language independent (with caveats)

• Low Latency achieved by two step process-Pre-processing -Numerical computation

Real-time Classification





• Dictionary or Lexicon approach; Unsupervised model

• Prepared offline with matrix decomposition

• Polarities assigned to adjectives (+ - 0 )

-Surrounding words could negate, amplify etc.

-Clusters of words treated separately

-Feature extraction possible for distinct sentiment

Sentiment Analysis





• Language independent technique worked well

• 50-60 documents per topic was sufficient

• Arabic content

-Is not 100% tokenizable – no spaces

-Did not hamper accuracy significantly

-Needed language expert to test model (for any foreign language)

Learnings - Analytics





• Task lends well to parallelization and scale out

• StreamAnalytix is a good fit – linear scale out

• Flexible topology

• Event size/ throughput – trade off

• Unique sharding and indexing for query optimization

• Many more types of use-cases possible

Learnings - Architecture




Case Study 2 – Hosted Contact Center

Solution




IVR

QueueAgent


Solution




Problem Statement

• Reactive –

Customer service

complaints on “What

happened to my call ?”

Diagnostics

- Easier

- Faster

• Proactive –

Business teams

want to understand

dominant call paths

• Lower “Queue” time

• Proactive -

Abandoned call

analysis

Hang up on

IVR/hold


Solution




Real-Time Dashboard and Alerts

• Ability to show counters on the existing Log Monitoring dashboard. For eg. #of inbound calls per tenant

• SLA based alarms – ability to generate alarms based on SLA threshold values over a moving time window per

tenant.

Log Aggregation

• Stream raw log events from

multiple remote servers

• Filter incoming log events – before

storage

• Index/search of log events

Auto Correlate Logs in

Real-time• Correlate log events arriving at

different time intervals based on

System ID, Channel ID, Call ID

• Visualize the complete call path

for a particular id

IVR Dominant

Path


Solution

Technical Requirements





Solution




Outcome, Next steps

• Next steps

– Sentiment analysis in real-time

(chat)

– Audio to text: Sentiment

analytics on transcript

– Rich real-time dash-boarding

and live counters

• Successfully solved key problems

– Call log aggregation, indexing and

search

– Real-time call path picture

– Dominant path analytics


Solution




Case Study 3 – Digital Content Provider

• Scholarly journals, educational, research

content

• Institutional Subscribers – 1000s of users

each

• Business wants real-time visibility and

analytics of customer behavior patterns




Problem Statement

• 10s of millions of events per day

– Clickstream data – complex XML events

• Real-time ETL

– Complex XML events parsed, filtered in real-time

• Clickstream-Analytics:

– Double click detection

– BOT detection

• Recommendation engine

– Upsell/ cross-sell






Data Flow and Real-time Pipeline Design




Case Study 4 – Web Application SLA Monitoring

• Healthcare insurance exchange software platform

• Server response time to front end application is a key

metric

• Complaints from key customers (potential revenue impact)

• Triggered need for aggressive monitoring and alerting

system




• Alert if response

breaches 4

second threshold

• Real-time

counters/

dashboard for a

variety of metrics

• Monthly report


Problem Statement




Remote

Node

Syslog

Server

StreamAnalyti

x

Agent

Syslog

Kafka

Server

Kafka

via

TCP

StreamAnalytix Agent Features

• The agent can publish to multiple destinations

• The agent can send encrypted data (optional)

StreamAnalytix

Real-Time Pipeline

Index

Store

Down

Stream

System

SLA

Events

Report

generationSLA

Alerts

Real-Time

Counters


Data Flow and Real-time Pipeline Design




Successful outcomes with all early customers

• Tier1 Healthcare Insurance Carrier – variety of use-cases

• Major Credit Card Brand and Bank – variety of use-cases

• End-point Security Application – On-prem and SaaS

• Mobile Field Devices – Real-time monitoring, predictive analytics

A few others in process




Q&A

Email us at [email protected]

www.StreamAnalytix.com

?

Request: On-premise and Cloud based trial and/or Proof of concept




StreamAnalytix Product Highlights

An “App Server” for real-time apps – on-premise and

cloud

Focus on your business logic - leave infra to us

Handle all the 3V’s of Big Data on one platform

Significant time to market acceleration

Seamless integration with Hadoop and NoSQL




Key Features

High Speed

Data

Ingestion

Elastic

Scaling –

Volume,

Velocity

Data Parsing

- Variety

Pluggable

Persistence

Real-time

Index and

Search

Dynamic

Message

Routing

Rule Based

Alert

Pluggable

Workflow

Management

Fault

Tolerance

and Data

Integrity

Optimized for

High

Performance
