A Stock Prediction System using open-source software
Fred Melo
@fredmelo_br
William Markito [email protected]
@william_markito
HDFS
Data Lake
Store Analytics
Hard to change Labor intensive
Inefficient
No real-time information ETL based Data-source specific
Traditional models are reactive and static
HDFS Data Lake Expert System /
Machine Learning
In-Memory Real-Time Data
Continuous Learning Continuous Improvement
Continuous Adapting
Data Stream Pipeline
Multiple Data Sources Real-Time Processing Store Everything
Stream-based, real-time closed-loop analytics are needed
Info
Analysis
Look at past trends (for similar input)
Evaluate current input
Score / Predict
Neural Network
How can it be addressed?
Ingest Transform Sink SpringXD
Store / Analyze
Fast Data
Distributed Computing
Predict / Machine Learning
Other Sources and Destinations
JMS
Streaming real-time analytics architecture
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
SpringXD
shell - R
Transformer
geode-json client
geode-json client
http-client
http-server
obj-to-json
splitter
splitter
Simulator
tap
SpringXD
INGEST / SINK PROCESS ANALYZE
• Little or no coding required
• Dozens of built-in connectors
• Seamless integration with Kafka, Sqoop
• Create new connectors easily using Spring
• Call Spark, Reactor or RxJava
• Built-in configurable filtering, splitting and transformation
• Out-of-box configurable jobs for batch processing
• Import and invoke PMML jobs easily
• Call Python, R, Madlib and other tools
• Built-in configurable counters and gauges
Data Stream Pipelining
SpringXD
XD Nodes XD Nodes XD Nodes XD Nodes
Ingest
SpringXD
Split Filter Transform Sink
XD admin
XD Nodes
Ingest Split Filter Transform Sink
Stream Deployment
Messaging
Scale-Out and HA Architecture
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
Transform Sink
SpringXD
Extensible Open-Source Fault-Tolerant Horizontally Scalable
HTTP
Machine Learning
Fast Data
Filter
Predict Sink HTTP
Split
Dashboard
Push
Demo Architecture
SpringXD
shell - R
Transformer
geode-json client
geode-json client
http-client
http-server
obj-to-json
splitter
splitter
Simulator
tap