samoa: a platform for mining big data streams (apache bigdata europe 2015)

1

SAMOA: A Platform for Mining Big Data Streams

Nicolas KourtellisAssociate Researcher

Telefonica I+D, Barcelona

SAMOA: Scalable Advanced Machine Online Analysis

2

What is Big Data?Search queriesFacebook postsEmailsTweetsPhoto sharesClicks on ads…

SAMOA: Scalable Advanced Machine Online Analysis 30/09/15

3

How BIG is your data?Volume (+ Variety)

Too large for RAM of single commodity serverVelocity

Too fast for CPU of single commodity server


4

What is the Streaming Paradigm?High amount of data, high speed of arrivalUpdated models at “real” timePotentially infinite sequence of dataChange over time (concept drift)


5

Mining Big Data StreamsApproximation algorithms:

Single pass, one data item at a timeSub-linear space and time per data itemSmall error with high probability

A platform solution:Support different algorithms & processing enginesDistributedScalable


6

What is SAMOA?Scalable Advanced Massive Online AnalysisA platform for mining big data streams

Framework for developing new distributed stream mining algorithms

Framework for deploying algorithms on new distributed stream processing engines


7

Taxonomy


8

SAMOA ArchitectureMachine LearningAlgorithms

Distributed StreamProcessing Engines Flink


9

Why is SAMOA important?Program once, run everywhere

Reuse existing infrastructureAvoid deploy cycles

No system downtimeNo complex backup/update processNo need to select update frequency


10

ML Developer API


11

ML Developer API


12

Deployment


13

Easy to get!


14

Easy to get!


15

Easy to get!


16

Easy to test!bin/samoa storm target/SAMOA-Storm-0.3.0-SNAPSHOT.jar"PrequentialEvaluation-d /tmp/dump.csv-i 1000000 -f 100000-l (classifiers.trees.VerticalHoeffdingTree -p 4 -k)-s (generators.RandomTreeGenerator –r 1 -c 2 -o 10 -u 10)"


17

Case study: Decision TreesVHT: Vertical Hoeffding Tree*


Task parallelism

*VHT: Vertical Hoeffding Tree. N. Kourtellis, G. De Francisci Morales, A. Bifet, A. Mordupo. IEEE BigData 2016.

18

Case study: VHT


Horizontal Parallelism

19

Case study: VHT


Vertical Parallelism

20

Benefits of Vertical ParallelismHigh number of attributes:

high level parallelism (e.g., documents)vs. task parallelism:

obvious parallelism observedvs. horizontal parallelism:

reduced memory usage (no model replication)parallelized split computation


21

Vertical Hoeffding Tree


22

Preliminary results: TweetsZipf skew: 1.5Bag of words: 100, 1000, 10000 (attributes)Size of tweet: ~15 wordsInstances: 1,000,000Class: positive or negative (Gaussian random variable)

10 runsLocal vs. Storm virtual cluster


23

Results: Accuracy


4 8 16 local0

20

40

60

80

100

Classification Accuracy vs.Parallelism Level vs.Number of Attributes

100 words1000 words10000 words

Parallelism Level

Cor

rect

Cla

ssifi

catio

n %

24

Results: Speedup


4 8 16 local0

1

2

3

4

5

Speedup vs.Parallelism Level vs.Number of Attributes

100 words1000 words10000 words

Parallelism Level

Spee

dup

25

Is SAMOA for you?Are you dealing with:

Big fast data?Possibly endless streams of data?Evolving data?

Do you need updated models at real time?Do you want to test an algorithm on different DSPEs?


26

SAMOA Team

Albert Bifet

GianmarcoDe Francisci Morales

Nicolas Kourtellis

Matthieu Morel

Arinto Murdopo

Olivier Van Laere


27

StatusApache Incubator

Released version 0.3.0 in JulyExecution Engines

Input: Local FS HDFS Kafka [pending]

Heron?


28

Algorithms in SAMOAExisting:

Vertical Hoeffding Tree (classification) CluStream (clustering) Adaptive Model Rules (regression)

Pending: Distributed Naïve Bayes Stochastic Gradient Descent Adaptive + Boosting VHT Parallelized Gradient Boosted Decision Tree PARMA (frequent pattern mining) …

Check Samoa Roadmap for more

Looking for contributors!


29

SAMOA: A Platform for Mining Big Data Streams

@ApacheSAMOAhttp://samoa.incubator.apache.org/

https://github.com/apache/incubator-samoa

Nicolas Kourtellis@kourtellis

[email protected]


http://samoa.incubator.apache.org/

http://samoa.incubator.apache.org/



mailto:[email protected]

samoa: a platform for mining big data streams (apache bigdata europe 2015)

Science