realtime predictive analytics using rabbitmq & scikit-learn

RealtimePredictive AnalyticsUsing scikit-learn and RabbitMQ

Michael BeckerPyData Boston 2013

Who is this guy?Software Engineer @ AWeberFounder of the DataPhilly Meetup group@beckerfufflebeckerfuffle.com

These slides and more @ github.com/mdbecker

What I'll cover• Model Distribution

What I'll cover• Model Distribution• Data flow

What I'll cover• Model Distribution• Data flow• RabbitMQ

What I'll cover• Model Distribution• Data flow• RabbitMQ• Demo

What I'll cover• Model Distribution• Data flow• RabbitMQ• Demo• Scalability

What I'll cover• Model Distribution• Data flow• RabbitMQ• Demo• Scalability• Other considerations

Supervised Learning

38 top wikipediasArabic العربيةBulgarian БългарскиCatalan CatalàCzech ČeštinaDanish DanskGerman DeutschEnglish EnglishSpanish EspañolEstonian EestiBasque EuskaraPersian فارسیFinnish SuomiFrench FrançaisHebrew עבריתHindi हि�न्दी�Croatian HrvatskiHungarian MagyarIndonesian Bahasa IndonesiaItalian Italiano

Japanese 日本語Kazakh ҚазақшаKorean 한국어Lithuanian LietuviųMalay Bahasa MelayuDutch NederlandsNorwegian (Bokmål) Norsk (Bokmål)Polish PolskiPortuguese PortuguêsRomanian RomânăRussian РусскийSlovak SlovenčinaSlovenian SlovenščinaSerbian Српски / SrpskiSwedish SvenskaTurkish TürkçeUkrainian УкраїнськаVietnamese Tiếng ViệtWaray-Waray Winaray

The model

Distributing the model

Data input

The client

Message loss

Enter RabbitMQ

Reliability

Flexible RoutingClustering

HA Queues

Many clients

AMQP

Reliability

Flexible RoutingClustering

HA Queues

Many clients

Data processing

The worker

The design

Demo time!

Demo time!

Demo time!

Scaling

Realtime vs batch

Monitoring

Load

Verify

Thank youAPI & Worker: Kelly O’Brien (linkedin.com/in/kellyobie)UI: Matt Parke (ordinaryrobot.com)Classifier: Michael Becker (github.com/mdbecker) Images: Wikipedia

My infoTweet me @beckerfuffleFind me at beckerfuffle.com

These slides and more @ github.com/mdbecker