data analysis and statistics in python using pandas and statsmodels

Statistics and Data Analysis in Python with pandas and statsmodels

Wes McKinney @wesmckinn

NYC Open Statistical Programming Meetup9/14/2011

Thursday, September 15,

Talk Overview

• Statistical Computing Big Picture

• Scientific Python Stack

• pandas

• statsmodels

• Ideas for the (near) future


Who am I?

MIT Math AQR: Quant Finance

Statistics

Back to NYC


The Big Picture

• Building the “next generation” statistical computing environment

• Making data analysis / statistics more intuitive, flexible, powerful

• Closing the “research-production” gap


Application areas

• General data munging, manipulation

• Financial modeling and analytics

• Statistical modeling and econometrics

• “Enterprise” / “Big Data” analytics?


R, the solution?

“R is the most powerful statistical computing language on the planet”

Hadley Wickham (ggplot2, plyr, reshape, ...)


Easy to miss the point


R, the solution?

“I have been worried for some time that R isn’t going to provide the base that we’re going to need for statistical computation in the future. (It may well be that the future is already upon us.) ... I have come to the conclusion that rather than ‘fixing’ R, it would be much more productive to simply start over and build something better”

Ross Ihaka (One of creators of R)


Some of my gripes about R

• Wonky, highly idiosyncratic programming language*

• Poor speed and memory usage

• General purpose libraries and software development tools lacking

• The GPL

* But yes, really great libraries


R: great libraries and deep connections to academia

Jeff Ryanxts, quantmod

Hadley Wickhamggplot2, plyr, reshape

Example R superstars


Uniting against common enemies


“Research-Production” Gap

• Best data analysis / statistics tools: often least well-suited for building production systems

• The “Black Box”: embedding or RPC

• High productivity <=> Low productivity


• Production: much more than crunching data and making pretty plots

• Code readability, debuggability, maintainability matter a lot in the long run

• Integration with other systems



My assertion

Python is the best (only?)viable solution to the

Research-Production gap


Scientific Python Stack

• Incredible growth in libraries and tools over the last 5 years

• NumPy: the cornerstone

• Killer app: IPython

• Cython: C speedups, 80+% less dev time

• Other exciting high-profile projects: scikit-learn, theano, sympy


Uniting the Python Community

• Fragmentation is a (big) problem / risk

• Statistical libraries need to be able to talk to each other easily

• R’s success: S-Plus legacy + quality CRAN packages built around cohesive base R / data structures


pandas

• Foundational rich data structures and data analysis tools

• Arrays with labeled axes and support for heterogeneous data

• Similar to R data.frame, but with many more built-in features

• Missing data, time series support


pandas

• Milestone: 0.4 release 9/12/2011

• Dozens of new features and enhancements

• Completely rewritten docs: pandas.sf.net

• Many more new features planned for the future


The sleeping dragon


Little did I know...


pandas: some key features

• Automatic and explicit data alignment

• Label-based (inc hierarchical) indexing

• GroupBy, pivoting, and reshaping

• Missing data support

• Time series functionality


Demo time


statsmodels

• Statistics and econometrics in Python

• Focused on estimation of statistical models

• Regression models (GLS, Robust LM, ...)

• Time series models (AR/ARMA, VAR, Kalman Filter, ...)

• Non-parametric models (e.g. KDE)


statsmodels

• Development has been largely focused on computation

• Correct, tested results

• In progress: better user interface

• Formula frameworks (e.g. similar to R)

• pandas integration


Demo time


Ideas for the future

• ggpy: ggplot2 for Python

• Statistical Python Distribution / Umbrella project

• Interactive GUI widgets to visualize / explore data and statsmodels results


Thanks

• pandas: http://pandas.sf.net

• statsmodels: http://statsmodels.sf.net

• Twitter: @wesmckinn

• E-mail: wesmckinn (at) gmail (dot) com

• Blog: http://blog.wesmckinney.com


http://pandas.sourceforge.net

http://pandas.sourceforge.net

http://statsmodels.sf.net

http://statsmodels.sf.net

http://blog.wesmckinney.com

http://blog.wesmckinney.com

data analysis and statistics in python using pandas and statsmodels

Technology