advanced spark and tensorflow meetup 08-04-2016 one click spark ml pipeline deploy to production
TRANSCRIPT
ADVANCED SPARKAND TENSORFLOW MEETUP
Updates and AnnouncementsAugust 4, 2016
advancedspark.com
MEETUP METRICS
• 4,000 Members in 1 Year!?
• Github Repo: 600 Stars, 230 Forks
• DockerHub Repo: 5,400 Pulls!!
MEETUP AGENDA
• Deploying and Scaling ML Models with PipelineIO Open Source (Chris Fregly)
• Fundamental Algorithms of Neural Networks (Sam Abrahams)
PIPELINE.IO
Extending Your ML Pipelines into Production
100% Open Source!
http://github.com/fluxcapacitor/pipeline
BRAINSTORMING AND VALIDATING
• Major Gaming Company
• Large Ride Sharing Service
• Popular Q & A Site
• Online Clothing Retailer
• Dominant Video Streaming
PIPELINE.IO FOCUS
• Model Deploying and Testing
• Model Scaling and Serving
• Online Model Training
• Dynamic Model Optimizing
MODEL DEPLOYING AND TESTING
Continuously Test and Deploy Models
in Production!
DEMO!
Deploy Spark ML DecisionTree to Production
Deploy to Cloudor On-Premise!
MODEL SCALING AND SERVING
DEMO!
Circuit Breakers and Request Batching
ONLINE MODEL TRAINING
• Continuous, Incremental, and Partial Training
• Kafka + Spark Streaming + Spark ML
• Real-time, Dynamic Recommendations
DEMO!
Real-time, Dynamic Recommendations
DYNAMIC MODEL OPTIMIZING
Generate OptimizedCode from Spark ML!
DEMO!
Dynamic Code Generation of DecisionTree
BECOME A CONTRIBUTOR!
WE’RE HIRING!!
• Kafka, Spark ML, and TensorFlow Contributors
• Systems Engineers
• GPU/CUDA Engineers
• C++, Java, Scala, Python
ONLYNICE PEOPLE!!
SANTA CLARA WORKSHOP, AUG 6, 2016