Patterns for Deploying Analytics in the Real World

Sriskandarajah Suhothayan (Suho)Technical LeadWSO2

What’s Analytics ?

Problems to think about

• Can it handle my load ?• How costly it is ?• Adaptability ?• Can it analyse 3rd party systems ?• etc ...

Where to start ?

Where to start ?

• Think Big !

Where to start ?

• Think Big !

But...

• Start simple !• Eat Your Own Dog Food • Analyse what you already have

Step 1 :

Find Data Inside Your Organisation...

Collect Data Internally

• Don’t worry about – Data formats – Data sources – Platforms – Protocols

Start with WSO2 DAS it has a unified data capturing framework !

Deployment for Data Collection

Step 2 :

Understand how things have been ...

Deployment for Data AnalyticsBatch & Interactive Analytics

• Enable Searchability – Full text data– Drill down search

• See what has happened – Summarise the Data – Understand patterns and behaviors

Deployment for Data AnalyticsBatch & Interactive Analytics

• Enable Searchability – Full text data– Drill down search

• See what has happened – Summarise the data – Understand patterns and behaviors

• Simple Deployment – 2 Nodes– Use RDBMS to store the data

Deployment for Data AnalyticsBatch & Interactive Analytics

2 Node Deployment

Step 3 :

Keep informed ...

Deployment for Data AnalyticsRealtime Analytics

• Keep informed – Dashboard – Alerts– Feedback loops

Deployment for Data AnalyticsRealtime Analytics

• Keep informed – Dashboard – Alerts– Feedback loops

• High Availability – Zero downtime– Zero data loss

Realtime High Availability Deployment

Minimum 2 nodesMax throughput == 1 Node throughput

Deployment for Data CommunicationAlerting & Communicating

Legacy & Internal Services

Realtime + Batch Analytics

• Filter Data before you store – Realtime → Store & Process

• Summarize and store – Realtime → Store & Process

• Cross check with history – Lambda Architecture– Graph with Batch & Realtime

• Alerts based on batch processing – Batch → Realtime

From Batch

From Realtime

Step 4 :

Think ahead ...

Deployment for Data AnalyticsPredictive Analytics

1 Node of WSO2 ML 1 Node of WSO2 ML

Minimum High Availability Deployment

All you need a 2 Node

Deployment

Step 5 :

Expanding as a Connected Business …

Deployment for Data Collection ...From 3rd Party Apps & Cloud

HTTP

Utilize API Analytics !

Analyse Business with API Analytics

• APIs involved • Who invokes the APIs• Extract business information from

– Payloads– Resources URIs

Monetize APIs !

Step 6 :

Scale with your Data ...

Scaling Analytics Deployment The Changes !

• Realtime – Supported by Apache Storm

• For High Memory Requirement or CPU Intensive Processing – No query changes

• Batch – Move from RDBMS to HBase/Cassandra

• WSO2 DAS have a Data Abstraction Layer • Independent of underlying Data Store

Seamless migration :)

Realtime Scalable Deployment ...

Event Processing offloaded to Siddhi Running on Apache Storm Seamlessly :)

Realtime Scalable Deployment ...

Handling Stateless & Stateful Queries

Realtime Scalable Deployment

Apache Storm Cluster + N CEP nodes

Deployment for Scalable Data Analytics

Minimum 8 Nodes Deployment

(+ Storm if needed)

Step 7 :

Sense the world around you ...

Deployment for Data CollectionFrom Sensors

Analytics on the Edgewith WSO2 Siddhi

Push

Deployment for Data CommunicationMobile & 3rd Party Apps

● Expose analytics results as API

○ Mobile Apps, Third Party

● Provides ○ Security, Billing, ○ Throttling, Quotas & SLA

● How ? ○ Write data to database from DAS ○ Build Services via WSO2 Data Services Server or use Analytics

REST API○ Expose them as APIs via WSO2 API Manager

Analytics Life Cycle

Predefined analytics

• Artifacts bundled as CApps to and moved

Dev → Test → Preprod → Prod

Analytics on Production Environment

• Interactive Analytics• Personalizing Dashboards • Customised Alerts

Summary

• Start small and scale as you grow• Minimum HA Deployment

– 2 Nodes

• Fully Distributed Deployment – 8+ Nodes – Scale based on need, horizontally and vertically

• Analyser, Indexer, Receiver, Realtime (With Apache Storm), Dashboard

In God we trust; all others must bring data

- William Edwards Deming -

Thank You