data science team collaboration: forget about meeting me halfway, take me the last mile |...

DATA SCIENCETEAM COLLABORATION

FORGET ABOUT MEETING ME HALFWAY,TAKE ME THE LAST MILE

#OpenDataScienceMeans #AnacondaCON Ian.Stokes-Rees @ijstokes

OGT molecular dynamics simulationProtein “mouth” opening, 1us

#OpenDataScienceMeans #AnacondaCON Ian.Stokes-Rees @ijstokesCERN computing facilityGeneva, Switzerland

SUCCESS COMES FROM TEAM WORK

http://bit.ly/ac17-collab

IAN: ENGINEER, PHYSICIST, BIOLOGIST?

• Ian Stokes-Rees, @ijstokes• Product Marketing Manager• Computational Scientist• Passionate advocate of

Open Data Science• Educator and evangelist for use of

Python and Anaconda

FIRST TASTE OF “BIG DATA” COMPUTING

• 100,000 acoustic tri-phone models• 100 parameters per model• 10 million parameters to estimate• adaptation = real-time adjustment• computation = tricky!

PhD on CERN LHCb COMPUTING TEAM

Distributed computing infrastructure• 1000s of concurrent users• 100s of federated computing centers

• no centralized control• 1M+ servers with software installed• 20+ year life span• 20 GB of data per second• 14 hours per day• 7 days a week• 7 months of the year

March 26, 2010 LHCb first physics at 3.5 TeV

HOW DO CERN PHYSICISTS DO THIS?

• Some smart people over there• Who brought us the Web, HTTP, and HTML?

• Big Data• Multi-PB per year

• Large collaborating teams• 1000s of people accessing systems

• Computation critical• Or there is no way to make sense of the data• And discover new physics December 2, 2016

LHCb proton-lead collisions

CERN ATLAS detectorCalorimeter end cap wiring harnessMillions of data feeds @ 40 MHz signal rate

HOW WOULD YOU DO IT?

Custom hardware:CMS L0 muon trigger ASIC

Giant compute and storage clusters

Wicked fast algorithmswritten in Fortran and C

Python: the Swiss army knife for computational physics

PYTHON: LINGUA FRANCA FOR DATA SCIENCE

• Human readable• Easy to learn• Object oriented• Cleanly wraps C and Fortran• Amazing foundation of high

quality data science libraries• Suitable for scripting,

algorithms, data processing and applications

THE CALCULUS OF NEWTON AND LEIBNIZ

SOMETIMES ESOTERIC IS OK

HERMITS AND HIGH PRIESTS

NPS, Richard Proenneke 1985

MOLECULAR BIOLOGY:FROM PROTONS TO PROTEINS

• It takes 3-9 months in the wet lab to prepare protein samples

• Once prepared it is only a few days to ”image” those samples and produce digitized representations

• However the “images” aren’t yet 3D atomic models

• That takes from weeks to months to complete, sitting behind a computer

• You may know it as protein folding

Nature, 2011 PMID: 21240259Lazarus, Nam, Jiang, Sliz, Walker

HOW DO WE ACCELERATETHE TIME TO INSIGHT?

WHAT DOES “HALF WAY” LOOK LIKE?

Today’s “good” data science environment:• Provide high performance computing resources

• For example, Hadoop infrastructure• Deploy a wide selection of the most popular analysis software

• Training and documentation• Technical support

FISH OUT OF WATER

• Why would we take an expert biochemist and force them to be

• A software engineer?• An IT system administrator?• A statistician?

• What can we do to let them focus on being a great biochemist?

FISH OUT OF WATER

• Why would we take an expert business analyst and force them to be

• A software engineer?• An IT system administrator?• A statistician?

• What can we do to let them focus on being a great business analyst?

TAKE ME THE LAST MILE

• DevOps engineer pre-configures scalable computation• Laptop to server to cluster• DevOps team is a partner, not a service provider

• Software engineer creates and customizes software for the task, project or individual

• Avoiding generic, static software setups

• Data scientist composes workflow• Analyst is provided simple high level interface

• With option to “drill down”

WHAT ABOUT THOSE PROTEINS?

• Normally it takes 10-200 hours of computing time to match a ”template” protein fragment to the imaging data

• There are 100k templates (known protein “folds”) to choose from• ”Be stupid” and just try them all – sometimes you’ll be surprised!• I spent 18 months working with biochemists and IT sys admins across

the country to create a sensible parallel & distributed workflow• 4-40 hours wall clock time to run 2k-20k hour parallel computation• Real-time updates of results• Web based interface to access summary and detailed data viz• Analysis performed in Jupyter Notebook, allowing customization• File-system based to enable “drill down” and direct access• 6M hours per year (~700 years), peak parallelism 20k cores

DATA SCIENCE PATTERN

• How is it done today?• What is the opportunity for improvement?• Prototype and evaluate – is it better? Rinse and repeat• Standardize and automate the workflow/model• Scale the workflow/model• Preprocess and distribute the data• Instrument execution and set quality metrics• Establish easy access interface• Create programmatic APIs

Remember the footnote?Collaborative cross-functional teams

BREAKING DATA SCIENCE OPEN

ANACONDA & COLLABORATION

STEP 1: ANACONDA

http://continuum.io/downloads

NOTEBOOKS FOR DATA SCIENCE COLLABORATION

Do you understand why notebooks are so popular?There are many angles to this, but my take:

• Visual record of the data science process• They tell a story, and support rich hyperlinked prose• Data can be embedded• Algorithms or analysis techniques are captured• Interactive visualizations are inline• Sharable• Reproducible*

STEP 2: ANACONDA CLOUD

http://anaconda.org

STEP 2: ANACONDA CLOUD

STEP 2: (MY) ANACONDA CLOUD

http://anaconda.org/ijstokes

STEP 3: ANACONDA ENTERPRISE (TODAY)

STEP 3: ANACONDA ENTERPRISE (COMING SOON)

ANACONDA:GIVING SUPERPOWERS TO THE PEOPLEWHO CHANGE THE WORLD

THANK YOU! QUESTIONS?

Ian Stokes-Rees @ijstokes

data science team collaboration: forget about meeting me halfway, take me the last mile |...

Data & Analytics

the future of data science in the city of boston |...

meet me halfway -...

cracking washington’s black box | anacondacon 2017

halverwege/halfway there

can data scientists save hedge funds? | anacondacon 2017

leveraging open technologies pragmatically within a...

meet me halfway: developers and designers pairing for the...

co-evolving with the open source eco-system | anacondacon...

colab meet me halfway 2017 - colabsbc.org · meet me...

meet me halfway: split maintenance of continuous viewsmeet...

halfway house rrc report

meeting halfway

meet me halfway

workshop "meet me halfway"

supercharging excel with anaconda fusion | anacondacon 2017

halfway through 2013

project halfway

halfway home model

rcjeks halfway open

operation halfway house hunters