cloud foundry for data science

1 © Copyright 2015 Pivotal. All rights reserved.

@ianhuston 1 © Copyright 2014 Pivotal. All rights reserved. 1 © Copyright 2015 Pivotal. All rights reserved. 1 © Copyright 2015 Pivotal. All rights reserved. 1 © Copyright 2014 Pivotal. All rights reserved.

Cloud Foundry for Data Science

Ian Huston, Data Scientist


@ianhuston

Who am I?

�  Ian Huston

� @ianhuston

� github.com/ihuston

� www.ianhuston.net

� Talk resources: http://tinyurl.com/cf4ds-talk

� Data Scientist

� Previously a theoretical physicist using Python for numerical simulations & HPC

� Use Python/R/SQL/… for predictive analytics and machine learning


@ianhuston

What is Cloud Foundry?

http://cloudfoundry.org

Open Source Cloud Platform

Simple App Deployment, Scaling & Availability

No Cloud Provider Lock In


@ianhuston

How can CF help data scientists? �  Jamie is a data scientist who has just finished some

analysis. They want to put up a simple internal web app with Javascript visualisations connected to internal data stores.

� Sam is a data engineer who wants to set up a REST API to expose a production machine learning model as a service.

� Alex is a data scientist who has an existing RShiny or Python app that they want to make available with multiple instances.


@ianhuston

Cloud Foundry is a Platform

You bring the apps, the rest is taken care of!

Source: Albert Barron (IBM), https://www.linkedin.com/pulse/20140730172610-9679881-pizza-as-a-service


@ianhuston

Cloud Foundry Foundation: Industry Standard

Gold

Silver


@ianhuston

CF for data scientists & developers Easily deploy your web app

cf push myapp Scale up and out quickly

cf scale myapp –i 5 –m 1G Create and bind services

cf bind-‐service myapp redis


@ianhuston

Simple Python Flask App Demo � Simple one page “Hello World” web app

� Video: https://www.youtube.com/watch?v=QOfD6tnoAB8

� Demonstrates: –  Installation of requirements –  Scaling properties

� Need to Provide: –  App files –  Dependencies listed in requirements.txt file –  Optional manifest.yml file with configuration for deployment


@ianhuston

Official Language Support on Cloud Foundry

Go

Ruby


@ianhuston

Buildpacks – How to Customise CF

Use Heroku-like buildpacks to add additional runtimes:

� Community Buildpacks for Clojure, Haskell, .NET, Erlang,…

� R Buildpack –  https://github.com/alexkago/cf-buildpack-r –  Works with RShiny apps

� PyData Buildpack –  https://github.com/ihuston/python-conda-buildpack –  Uses conda for hard to install dependencies


@ianhuston

Comparison with Docker

runtime layer

OS image

application layer

Docker

system brings fixed host OS Kernel

* Devs may bring a custom buildpack

runtime layer*

OS image

application layer

CF Buildpack

system brings fixed host OS Kernel

App container

System Provides

Dev Provides


@ianhuston

Services now available These are available on Pivotal CF (Pivotal’s packaged Cloud Foundry offering).

See http://run.pivotal.io for the services available on Pivotal Web Services.


@ianhuston

Data Services

� Easy to bind and scale services –  Databases, NoSQL, message queues etc. $ cf create-‐service rediscloud PLAN_NAME INSTANCE_NAME $ cf bind-‐service APP_NAME INSTANCE_NAME

� Details provided in VCAP_APP_SERVICES env variable: { "rediscloud": [ { "name": "rediscloud-‐42", "label": "rediscloud", "plan": "20mb", "credentials": { "port": "6379",

"hostname": "pub-‐redis-‐6379.us-‐east-‐1-‐2.3.ec2.redislabs.com", "password": "your_redis_password” } } ] }


@ianhuston

What can you build with CF?

Some ideas:

�  Interactive data exploration apps

� APIs for a machine learning model

�  Internet of Things data collection and modelling tools

� Microservices architecture for data analysis

�  Interface to a natural language system like IBM’s Watson


@ianhuston

Examples

� Simple HTML + JS app

� Uses data in S3

� Scale out when heavy load expected with auto load balancing

http://ds-demo-transport.cfapps.io


@ianhuston

Examples

� RShiny app

�  Interactive controls with very little additional coding.

� Runs RandomForest model in background

https://ak-insurance-demo.cfapps.io:4443/


@ianhuston

Send me your examples!

Show off your data science related Cloud Foundry apps:

Twitter: @dsoncf http://dsoncf.com

http://DataScienceOnCloudFoundry.com


@ianhuston

Try Cloud Foundry today!

Pivotal Web Services: http://run.pivotal.io IBM Bluemix: http://bluemix.net Anynines: http://anynines.com

HP Helion: http://horizon.hpcloud.com


@ianhuston

More info and resources

� Material for this talk: http://tinyurl.com/cf4ds-talk

� Cloud Foundry: http://cloudfoundry.org

� Python Flask Mega Meta Tutorial for DS: http://tinyurl.com/flaskDS

� Twelve Factor Apps: http://12factor.net/

� Meetups: http://cloud-foundry.meetup.com/

cloud foundry for data science

Technology