wso2con asia 2016: patterns for deploying analytics in the real world

Download WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World

Post on 13-Feb-2017

310 views

Category:

Technology

0 download

Embed Size (px)

TRANSCRIPT

  • Patterns for Deploying Analytics in the Real World

    Sriskandarajah Suhothayan (Suho)Technical LeadWSO2

  • Whats Analytics ?

  • Problems to think about

    Can it handle my load ? How costly it is ? Adaptability ? Can it analyse 3rd party systems ? etc ...

  • Where to start ?

  • Where to start ?

    Think Big !

  • Where to start ?

    Think Big !

    But...

    Start simple ! Eat Your Own Dog Food Analyse what you already have

  • Step 1 :

    Find Data Inside Your Organisation...

  • Collect Data Internally

    Dont worry about Data formats Data sources Platforms Protocols

    Start with WSO2 DAS it has a unified data capturing framework !

  • Deployment for Data Collection

  • Step 2 :

    Understand how things have been ...

  • Deployment for Data AnalyticsBatch & Interactive Analytics

    Enable Searchability Full text data Drill down search

    See what has happened Summarise the Data Understand patterns and behaviors

  • Deployment for Data AnalyticsBatch & Interactive Analytics

    Enable Searchability Full text data Drill down search

    See what has happened Summarise the data Understand patterns and behaviors

    Simple Deployment 2 Nodes Use RDBMS to store the data

  • Deployment for Data AnalyticsBatch & Interactive Analytics

    2 Node Deployment

  • Step 3 :

    Keep informed ...

  • Deployment for Data AnalyticsRealtime Analytics

    Keep informed Dashboard Alerts Feedback loops

  • Deployment for Data AnalyticsRealtime Analytics

    Keep informed Dashboard Alerts Feedback loops

    High Availability Zero downtime Zero data loss

  • Realtime High Availability Deployment

    Minimum 2 nodesMax throughput == 1 Node throughput

  • Deployment for Data CommunicationAlerting & Communicating

    Legacy & Internal Services

  • Realtime + Batch Analytics

    Filter Data before you store Realtime Store & Process

    Summarize and store Realtime Store & Process

    Cross check with history Lambda Architecture Graph with Batch & Realtime

    Alerts based on batch processing Batch Realtime

    From Batch

    From Realtime

  • Step 4 :

    Think ahead ...

  • Deployment for Data AnalyticsPredictive Analytics

    1 Node of WSO2 ML 1 Node of WSO2 ML

  • Minimum High Availability Deployment

    All you need a 2 Node

    Deployment

  • Step 5 :

    Expanding as a Connected Business

  • Deployment for Data Collection ...From 3rd Party Apps & Cloud

    HTTP

    Utilize API Analytics !

  • Analyse Business with API Analytics

    APIs involved Who invokes the APIs Extract business information from

    Payloads Resources URIs

    Monetize APIs !

  • Step 6 :

    Scale with your Data ...

  • Scaling Analytics Deployment The Changes !

    Realtime Supported by Apache Storm

    For High Memory Requirement or CPU Intensive Processing No query changes

    Batch Move from RDBMS to HBase/Cassandra

    WSO2 DAS have a Data Abstraction Layer Independent of underlying Data Store

    Seamless migration :)

  • Realtime Scalable Deployment ...

    Event Processing offloaded to Siddhi Running on Apache Storm Seamlessly :)

  • Realtime Scalable Deployment ...

    Handling Stateless & Stateful Queries

  • Realtime Scalable Deployment

    Apache Storm Cluster + N CEP nodes

  • Deployment for Scalable Data Analytics

    Minimum 8 Nodes Deployment

    (+ Storm if needed)

  • Step 7 :

    Sense the world around you ...

  • Deployment for Data CollectionFrom Sensors

    Analytics on the Edgewith WSO2 Siddhi

    Push

  • Deployment for Data CommunicationMobile & 3rd Party Apps

    Expose analytics results as API

    Mobile Apps, Third Party

    Provides Security, Billing, Throttling, Quotas & SLA

    How ? Write data to database from DAS Build Services via WSO2 Data Services Server or use Analytics

    REST API Expose them as APIs via WSO2 API Manager

  • Analytics Life Cycle

    Predefined analytics

    Artifacts bundled as CApps to and moved Dev Test Preprod Prod

    Analytics on Production Environment

    Interactive Analytics Personalizing Dashboards Customised Alerts

  • Summary

    Start small and scale as you grow Minimum HA Deployment

    2 Nodes Fully Distributed Deployment

    8+ Nodes Scale based on need, horizontally and vertically

    Analyser, Indexer, Receiver, Realtime (With Apache Storm), Dashboard

  • In God we trust; all others must bring data

    - William Edwards Deming -

  • Thank You