how to create a real-time bag data application

Upload: steffbold

Post on 14-Apr-2018

220 views

Category:

Documents


0 download

TRANSCRIPT

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    1/11

    HowtoCreateaReal-meBigDataApplicaon

    BigDataSiliconValley,April24th2013

    1

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    2/11

    AboutMe

    About4yearsintheSemiconductorspace NaonalSemiconductor(TexasInstruments) LTXCorporaon(LTXCadence) AppliedMaterials

    About8yearsintheMapping/TrafficDataspace TeleAtlas(TomTom) Map24(NAVTEQ) INRIX

    About5yearsintheStartupspace

    WindSpring Skydera

    (2)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    3/11

    CreangAReal-meBigDataApp

    WhatisReal-me? ExamplesofReal-meBigDataApps

    GoogleAnalycs,KissMetrics,mixpanel,Woopra WhatdoweneedtocreateaReal-meBigDataApp?

    CanaReal-meBigDataAppruninmydatacenter?

    WhattypeoftoolswouldIneedtocreate(#)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    4/11

    WhatdoIneed?

    (#)

    WebTierPython, Ruby, Node.js, .Net, Java

    DatabaseTier

    NoSQL (MongoDB, Cassandra, DynamoDB, etc..)

    OffsetAnalysis

    Real-meAnalysis

    Hadoop, Hive, Mahout, AWS-EMR

    Caching (Memcached, Redis, etc..)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    5/11

    WebTier

    YourfavoritelanguageandFramework CreateAPItoreceivedata CreateAPItodistributedata

    Thiswillenableyourapplicaontorequestsupdateswithoutrefreshingthewholescreen

    BuildUserinterfacetorequestdataelementsandnotjustpages.

    (#)

    API

    UserInterface

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    6/11

    WebTierRealTimeprocessing

    Per-Requestprocessing Pre-Definedmatrix's Pre-Definedmeframes Pre-Definedresulngtables

    (#)

    API

    UserInterface

    Real-meProcess

    Pre-Definedresulngtables

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    7/11

    DatabaseTier

    NoSQLMongoDB,Cassandra,DynamoDB,CouchDBImportanttohavesomethingDistributedStorepredefinedtablesHoldsrawdataforHadoopandoffsetprocessing

    (#)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    8/11

    DatabaseTier

    CachingMemcached,RedisImproveretrievalmesLowertheburdenonNoSQL/SQLforfrequently

    requesteddata

    Holdneeddatabetweenrequests

    (#)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    9/11

    DatabaseTier

    SQLYourfavoriteRDBMSSomeusecasesbenefitfromhavinganRDBMSintheStack(complexIndexing)SomeNoSQLDatabasesprovidescomplex

    Indexing

    (#)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    10/11

    OffsetAnalysis

    Currentprocessforprocessinglargeamountsofdata

    Veryusefulforaskingquesonsnotpredefinedinthereal-meprocessing NoneedtolosetheabilitytoprocesslargeamountsofdataeveryXXhours.

    (#)

  • 7/30/2019 How to Create a Real-Time Bag Data Application

    11/11

    Contact

    SteffanyBoldrini

    [email protected]

    @steold

    (#)

    LecoleCole

    SkyderaInc.

    lecole@skydera

    @lecole