Data Analysis and Reporting

“...a wealth of information creates a poverty of attention...”

-Herbert A .Simon

Melissa Chapman Haynes, Ph.D.

Senior Evaluator, Professional Data Analysts, Inc.

MESI Spring Training 2017

2Prepared by Professional Data Analysts, Inc.

“PEOPLE TODAY ARE IN DANGER OF DROWNING IN INFORMATION; BUT, BECAUSE THEY HAVE BEEN TAUGHT THAT

INFORMATION IS USEFUL, THEY ARE MORE WILLING TO DROWN THAN THEY NEED BE.

IF THEY COULD HANDLE INFORMATION, THEY WOULD NOT HAVE TO DROWN AT ALL.”

― IDRIES SHAH, REFLECTIONS

3Prepared by Professional Data Analysts, Inc.

WHAT PART OF THE PROCESS DOES DATA ANALYSIS AND REPORTING COME INTO PLAY?

“Justify conclusions”

Analyze the evaluation data

Determine what the findings say about your

program

“Use and share lessons learned”

Ensure stakeholders are engaged in the

evaluation process, are aware of the results

Use of evaluation results in decision-making

Those who participate in the evaluation should

have a beneficial experience

Minnesota Evaluation Studies Institute Spring Training 2017

4Prepared by Professional Data Analysts, Inc.

WHERE DOES DATA ANALYSIS FIT INTO THE EVALUATION PROCESS?

Minnesota Evaluation Studies Institute Spring Training 2017

5Prepared by Professional Data Analysts, Inc.

EVALUATION PLAN MATRIX

Minnesota Evaluation Studies Institute Spring Training 2017

6Prepared by Professional Data Analysts, Inc.

7Prepared by Professional Data Analysts, Inc.

SCOPE – WHAT ARE WE DOING TODAY?

Developing Qualitative Data Analysis Categories 5

3

4

2Analyzing and Interpreting Qualitative Data

Quantitative Data Analysis

Innovations and Technologies

Planning for Data Analysis 1

Reporting6

Section 1 – Planning for Data Analysis

9Prepared by Professional Data Analysts, Inc.

Data Analysis is an art…

Context matters

Process matters

Asking questions is essential!!

…What can be abstracted across different

contexts about data analysis?

Minnesota Evaluation Studies Institute Spring Training 2017

10Prepared by Professional Data Analysts, Inc.

Epicycles of Analysis – The Model

11Prepared by Professional Data Analysts, Inc.

Mapping the Five Core Activities of Data Analysis (Quant) to the Epicycles

1. Ask questions

2. Explore the data

3. Build statistical

models

4. Interpret results

5. Communicate

results

1. Set expectations

2. Collect data and

compare to

expectations

3. Revise data or

expectations as

needed

12Prepared by Professional Data Analysts, Inc.

Mapping the Five Core Activities to the Epicycles of Analysis

13Prepared by Professional Data Analysts, Inc.

Epicycles of Analysis – Cancer Example

Set Expectations

Collect Information

Revise Expectation

Question

EDA

Formal modeling

Interpretation

Communication

14Prepared by Professional Data Analysts, Inc.

Quantitative

o Descriptive

o Correlational

o Inferential

o Multivariate

Qualitative

o Thematic coding

o Case studies

o Success case method

o Content Analysis

o Grounded Theory

Systems Methods

o Outcome Mapping

o Social Network Analysis

o Process Monitoring of

Impacts

o Causal Loop Diagrams

Types of Data Analysis

15Prepared by Professional Data Analysts, Inc.

Mixed Methods

16Prepared by Professional Data Analysts, Inc.

Data Preparation Process

17Prepared by Professional Data Analysts, Inc.

Basic Data Cleaning Troubleshooting

18Prepared by Professional Data Analysts, Inc.

Activity: Planning for Data Analysis

Survey Data Analysis

Question or item numbers

What data analysis methods might you use?

Why would you use this analysis? What would you hope to learn?

20Prepared by Professional Data Analysts, Inc.

Activity: Planning for Data Analysis Debrief

What insights into data analysis did you gain from this activity?

If you were going to administer a survey to two different groups,

what other analysis might you consider?

What might be the impact on this evaluation if there were no

data analysis plan?

What implications does planning for data analysis have on an

evaluation budget?

Section 2 – Developing Qualitative Data Analysis Categories

23Prepared by Professional Data Analysts, Inc.

How to Develop Categories and Themes

What challenges did you experience in trying to sort the candy according to pre-determined categories?

After sorting the candy to what extent did you reach different conclusions about the data even though the data were the same? What implications are there for evaluation practice?

When might you use a predetermined set of categories for analyzing qualitative data? Where might the predetermined categories come from? What implications might using predetermined categories have on the evaluation’s findings?

How would you go about deriving categories or themes form the data if there were no predetermined categories?

What are the advantages and disadvantages of these two approaches to analyzing qualitative data?

24Prepared by Professional Data Analysts, Inc.

Coding Examples Excerpt

What are the negative aspects

of having your service member

deployed?

25Prepared by Professional Data Analysts, Inc.

Coding Examples Excerpt – Simple Coding

What are the negative aspects

of having your service member

deployed?

26Prepared by Professional Data Analysts, Inc.

Coding Examples Excerpt – Thematic Coding

What are the negative aspects

of having your service member

deployed?

Section 3 – Analyzing and Interpreting Qualitative Data

28Prepared by Professional Data Analysts, Inc.

Group A = Predetermined

categories applied to analysis

Review categories and

agree on definitions

Code data

Complete handout

Summarize data and

develop recommendations

Group B = Emergent categories

developed and applied

Review data and develop

categories based on themes

and patterns

Apply coding to Question 1

Complete handout

Summarize data and

develop recommendations

Pre-determined versus emergent categories

29Prepared by Professional Data Analysts, Inc.

Pre-determined versus emergent categories: Debrief

How similar or different were the findings among those in

Group A? In Group B?

What were some of the similarities and differences

between the results from Group A and Group B?

How would you describe your experience in analyzing

these data?

What surprised you about this work?

What insights did you gain from this activity that will be

useful the next time you collect and analyze qual data?

Section 4 – Quantitative Data Analysis

31Prepared by Professional Data Analysts, Inc.

Descriptive analysis

Correlational analysis

Inferential analysis

Small Group Activity

Section 5 – Innovations and Technologies

34Prepared by Professional Data Analysts, Inc.

Technologies for Qualitative Analysis

35Prepared by Professional Data Analysts, Inc.

Technologies for Quantitative Analysis

SPSS

SAS

Stata

R

Excel

Dedoose (mixed methods)

Section 6 – Reporting and Data Viz

37Prepared by Professional Data Analysts, Inc.

“Standard” Report Format

38Prepared by Professional Data Analysts, Inc.

Other Qualitative Report Formats

39Prepared by Professional Data Analysts, Inc.

Report and Support Use – Better Evaluation

http://www.betterevaluation.org/plan/reportandsupportuse

40Prepared by Professional Data Analysts, Inc.

CHART CHOOSER

Minnesota Evaluation Studies Institute Spring Training 2017

41Prepared by Professional Data Analysts, Inc.

Pop Culture meets Data Viz…

42Prepared by Professional Data Analysts, Inc.

Data viz can excite…

43Prepared by Professional Data Analysts, Inc.

And one more…

45Prepared by Professional Data Analysts, Inc.

Questions?