knowledge management for analytic teams jaime fitzgerald and alex hasha - presentation @info360...
TRANSCRIPT
Knowledge Management for
Analytic Professionals and Teams
Jaime Fitzgerald, President, Fitzgerald Analytics, Inc.
Alex Hasha, Chief Data Scientist, Bundle Corporation
March 2011
Architects of Fact-Based Decisions™
Introduction
Alex Hasha
Data Scientist @
Bundle Corp
Jaime Fitzgerald,
Founder @
Fitzgerald Analytics
Responsible
� Leading development of data products
� Designing statistical methods / algorithm � Transforming data into value for clients
2Knowledge Management for Analytics
Responsible
For…
At a
Company
That
Also
Working
On
� Designing statistical methods / algorithm
that transform data into insights for
consumers
� Helps consumers with financial mgt by
providing tools & spending behavior data
that are available nowhere else
� Is growing and hiring!
� Learning about Hadoop, Hive, Etc.
� Buying a home and renovating a bathroom
� Creating meaningful careers for employees
� Helps clients convert Data to Dollars™
� Brings a strategic perspective to improve
ROI on investments in technology, data,
people, and processes
� Writing a book about the role of “thought-
style” in the information-era
About Bundle Corporation and Bundle.Com
� Bridging the gap between personal financial management and the good
living that healthy finances makes possible.
� Helping people save well & spend well, combining free personal
financial mgt tools with data tools and recommendation engines built
from the anonymous spending behavior of over 20 million households.
A joint venture of Citi, Microsoft Money, & Morningstar, Bundle is:
3Knowledge Management for Analytics
from the anonymous spending behavior of over 20 million households.
� Cross-referencing this data with other public and private information,
we developed tools available nowhere else:
� "Everybody's Money”: learn how your peers spend and save
� Restaurant Recommender: based on card spending data
� Merchant Recommender: our newest recommendation tool
The restaurant recommender: predictive analytics meets dining!
4Knowledge Management for Analytics
1. Complex analysis is high stakes, risky, and hard to
manage informally
2. Success requires knowledge management
standards and tools, even within small teams
Executive Summary
5Knowledge Management for Analytics
standards and tools, even within small teams
3. “No Silver Bullet”: to empower analysts,
knowledge management methods must be tailored
to a team’s workflow
Table of Contents
1. Challenges of Analytic Thought-Work
2. The Role of Knowledge Management
6Knowledge Management for Analytics
3. Implications for all Thought-Workers
Challenges Analysts Face
In this section we will discuss…
1. A core set of universal challenges faced by analytic pros
2. Specific examples of these challenges at:
1. Challenges
7Knowledge Management for Analytics
1. Bundle Corporation
2. Other analytic teams that are working hard as we speak
It’s a Different World
While analytic teams are indispensible in today’s information economy, the nature of their work
makes teamwork, management, and coordination challenging.
1. Challenges
8Knowledge Management for Analytics
This creates a set of pitfalls…
source: www.xkcd.com
Problems Analysts Face
There are several pitfalls into which analysts fall.
Analysis is
often:The Embarrassing Facts
1. Hard to
understand� A top complaint regarding analysis is that it is confusing and unclear
2. Impossible to
verify, audit, � Few executives report they fully trust analysis they receive
1. Challenges
9Knowledge Management for Analytics
verify, audit,
or replicate
� Few executives report they fully trust analysis they receive
3. Flawed
� 90%+ of spreadsheets used in the field are estimated to have
material errors
� In 201- Aetna cancelled a 19% rate increase due to flawed analysis
4. Inefficient� Average analysts spend less than 10% of their time actually
performing core analysis (with most time going to data gathering,
troubleshooting, etc)
Analyst Pitfalls: Real Life Examples
Based on personal experience, but modified to protect the guilty…
Analysis is often: Examples (not @ Bundle) Impact
1. Hard to
understand
� An analysis involving both simple and
weighted averages of borrower credit
scores
� In various parts of the analysis, both
metrics were called “credit score”
� Misinterpretation
� Wrong inputs
� Unaware of risk
1. Challenges
10Knowledge Management for Analytics
2. Impossible to
verify, audit, or
replicate
� Documentation of key input variables was
not consolidated, hard to find, and
therefore not widely known
� Unaware of risk
� Hard to tell
whether correct
input was used
3. Flawed
� A chain of analysis mixed up simple vs.
weighted averages.
� Error persisted 5 years before it was
discovered.
� Consequences can
be significant
4. Inefficient� 2 PhDs took a week to solve a problem
that should never have happened.
� Expensive waste of
skilled time.
Data Science at Bundle: Lots of Prototyping
1. Goal: Solve a
New Problem
4. User Feedback 2. Design Solution
It is crucial for us to
document, share, and re-use
lessons learned from each
cycle of effort
1. Challenges
11Knowledge Management for Analytics
4. User Feedback 2. Design Solution
3. Testing +
Validation Data Product
on Website
Table of Contents
1. Challenges of Analytic Thought-Work
2. The Role of Knowledge Management
12Knowledge Management for Analytics
3. Implications for all Thought-Workers
Challenges Analysts Face
In this section we will discuss…
1. How KM helps analysts in general
2. Case Examples from Bundle
2. The Role of Knowledge
Management
13Knowledge Management for Analytics
Knowledge Management to Avoid Analyst Pitfalls…
Knowledge Management makes a big difference in outcomes for analysts
Pitfall How KM Helps
Lack of detailed specifications often leads to
useless results, yet writing detailed
specifications for another teammate can take
longer than doing it yourself.
Slow on boarding due to “knowledge
Consistent coding and analysis standards
prevent make specifications easier to create
and execute
Learning curve remains, but progress is faster.
Senior teammates spend less time training new
2. The Role of Knowledge
Management
14Knowledge Management for Analytics
Slow on boarding due to “knowledge
diffusion” and lack of access
Reinventing the wheel due to lack of
awareness of previous work-products
Misinterpreting data definitions, or
misunderstanding provenance of data leads to
incorrect analyses.
Senior teammates spend less time training new
teammates.
Teammates post documentation of solutions to
common problems, and are encouraged to
search this documentation as a first step.
Enforcement of standardized, descriptive, field
names and centrally available data dictionaries
make these mistakes harder to miss.
Workflow at Bundle
Inputs Process Outputs
Credit Card
Transaction Data
Merchant
Listing Data
Natural Language
Processing/
Categorization
Customer/
Merchant
Spending Survey
Insights &
Recommendations for
Consumers
� Everybody’s Money™
2. The Role of Knowledge
Management
15Knowledge Management for Analytics
Anonymous Customer
Demographic Data
Census Data &
Government Consumer
Spending Data
Geographic Data
Statistical
Sample
Rescaling
Consumer
Demographic
Profiles
� Restaurant / Merchant
Recommender
• Loyalty/Popularity
Scores
• “Web of Offline
Merchants”
� Consumer Segment
Analysis
KM at Bundle
Wiki-Based
Knowledge
� Metric Definition
� Algorithms
� Dialog
� Meta-data
2. The Role of Knowledge
Management
16Knowledge Management for Analytics
Work Flow
“Persistent” Code & Scripts
(vs. ad hoc data processes)
In-System
Knowledge
Table of Contents
1. Challenges of Analytic Thought-Work
2. The Role of Knowledge Management
17Knowledge Management for Analytics
3. Implications for all Thought-Workers
Key Implications for all Through-Workers3. Implications
Concept Basis
1. The more technical the work, the
more you need KM!
� Technical work draws upon more
dimensions of knowledge than teams can
manage informally
2. Good KM supports YOUR
workflow and thought-work
� Since the goal of KM is to improve Quality
and Efficiency of outcomes, it is essential
18Knowledge Management for Analytics
workflow and thought-workand Efficiency of outcomes, it is essential
to customize KM to worker processes
3. One size does not fit all
� Because workflow for technical workers is
so variable, they require flexible KM
solutions
4. At times technical KM must span
multiple platforms
� In contrast to less technical knowledge,
there are times when technical knowledge
is better managed in multiple places
Our Thought-Work Creates Value
Like a good chef, we need to have the ingredients in place when we need them…
“The Chef’s Shelf”
3. Implications
19Knowledge Management for Analytics
Individual Work Collaboration Communication
Let’s Stay in Touch
We look forward to learning from each other…contact us anytime.
Alex Hasha Jaime Fitzgerald
20Knowledge Management for Analytics
Twitter: www.twitter.com/alexhasha
LinkedIn: www.linkedin.com/pub/alexander-
hasha/8/26a/30a
Email: [email protected]
Twitter: www.twitter.com/jaimefitzgerald
LinkedIn: www.linkedin.com/in/jaimefitzgerald
Email: [email protected]