Creativity and CuriosityTHE TRIAL AND ERROR OF DATA SCIENCE

I love data, everything comes easy to me…

There are so many things to try and explore on a given problem, where to start?

• Language (Julia, Python,R, C++,etc)• Visualization (ggplot, Tableau, D3,etc)• Pre-process (standardize, variance scaling, feature encoding, etc) • Classifier (GLM, SVM, SGD, Knn, Random Forest, etc)• Post-process (Rule-truncatation, post-pruning, etc)• Ensemble (weighted average, min, max, probabilities, etc)

Where Many Individual Come To Die…

(Model Tuning Hell)

Structured Process Allows you to remove uncertainty and ensure outcomes in a methodical way.

Gives you an idea of what activities to do and when.

Details for each project varies, however the structure should stay the same.

The process is almost never linear, you should revisit each step again and again.

Knowledge Discovery Process1. Define the goal2. Explore the data3. Prepare the data4. Choosing and evaluating

models5. Ensemble

Define the Goal• Why do the sponsors want the project in the first place?

What do they lack, and what do they need?• What are they doing to solve the problem now, and why

isn’t that good enough?• What resources will you need: what kind of data? Do

you have domain experts to collaborate with, and what are the computational resources?

• How do the project sponsors plan to deploy your results? What are the constraints that have to be met for successful deployment?

• Is the data quality good enough?

Define the GoalModeling:

• Classification• Scoring• Ranking• Clustering• Finding relations• Characterization

Model Evaluation and critique• Is it accurate enough for your needs? Does it

generalize well?• Does it perform better than “the obvious guess”?

Better than whatever is currently in use?• Do the results of the model (coefficients, clusters,

rules) make sense in the context of the problem domain?

Explore the DataUse summary statistics to spot problems

• Missingness• Data ranges (too wide/too

narrow)• Invalid values• Outliers• Units

Explore the DataUse graphics and visualization to spot problems

Single-Variable First• Peak of distribution?• How many peaks?• How normal (or lognormal is the data?• How much data variation is there? Is it

concentrated in a certain interval or category?

• Use histograms, density plots, bar charts, scatter plots with smoothing curve.

Prepare the Data

Cleaning Data• Treating missing

values (NAs)• Data

Transformations

Sampling for Modeling and Validation• Test and training splits• Creating sample group column• Record grouping

Choosing and Evaluating ModelsMapping problems to machine learning tasks (use a problem-to-method mapping)

• Solving classification problems• Naïve Bayes• Decision Trees• Logistic Regression

• Solving scoring problems• Linear Regression• Logistic Regression

• Working without known targets• K-means clustering• Apriori algo to find association rules• Nearest neighbor

Choosing and Evaluating ModelsEvaluating models

• Evaluating classification models• Confusion matrix• Precision• Recall• Sensitivity • Specificity

• Evaluating scoring models• Root Mean Square Error• R-squared• Correlation• Absolute Error

Choosing and Evaluating ModelsEvaluating models

• Evaluating probability models• Area Under the Curve• Log Likelihood• Deviance• Akaike Information Criterion (AIC)• Entropy

• Evaluating ranking models• Intra-cluster distances• Cross-cluster distances

Choosing and Evaluating ModelsValidating models

• Identify common model problems• Bias – systematic error• Variance – oversensitivity of the model• Overfit – doesn’t generalize well• Nonsignficance – relation may not hold

• Ensuring model quality• Testing on Held-Out Data• K-Fold Cross Validation• Significance Testing• Confidence Intervals

Ensemble

How do I bring all my work together?• Weighted average• Min• Max• Voting• Stacking• Neural network

More IdeasLearn about ensemble methods, regularization, and principled dimension reduction

• Hastie, Tibshirani, and Friedman’s The Elements of Statistical Learning, Second Edition

• If you want to understand the consequences of a method, has a math bent

Keep your saw sharp Plug-in

Using your creativity and curiosity you can slay mighty data science problems.

@DamianMinglehttp://www.WPC-Services.com

http://www.DamianMingle.com