slayter on planning quant design for flc projects - may 2011

On planning your quantitative research design for a faculty learning community project

Elspeth Slayter, Ph.D.School of Social WorkSalem State University

Before you begin…don’t read these slides unless you…

• …Have skimmed a basic program evaluation text• …Know what your “intervention” will be

– Example: Increasing student engagement in research courses

• …Have framed your research question– Example: Does the use of a new type of pedagogical framework

increase self-reported student engagement at mid-semester?

• …Are pretty sure you want to compare groups– Example: Pre/post tests on one class, comparison of sections

Slayter 2

When babies are on the way, we prepare a nursery, yes?

• You must prepare a data analysis plan before the data are collected – and eventually – arrive!

• Mistake #1: Create survey, analyze data later, can lead to many problems – don’t do it!

Slayter 3

Metaphor: Preparation of nursery – Preparation of data

analysis plan• A somewhat iterative process

in choosing the following:– Over-arching research question

– Project goals

– Project objectives

– Data collection plan

– Identification of analytic task

– Choice of statistical tests

Slayter 4

Presentation Overview:How to approach this task

5Slayter

Step 1: Identify project research question,

goals, objectives, intervention• Example research question: How can I foster

student engagement in required research courses?– Example goal: To improve student engagement by

changing my teaching framework as explicated to students

– Example objective: To improve class participation grades after infusing new teaching framework in lecture, discussion, assignments, activities

Slayter 6

Step 1: Identify project research question,

goals, objectives, intervention

Slayter 7

Your goal is LINKAGE!!!

Step 2: Design data collection approach that

best answers your question

• Methods of assessing outcome must be best possible/feasible way to answer question– Will numbers or words answer the over-arching

question best?• Maybe quantitative?

– Maybe pre-post-test? Maybe not?

• Maybe qualitative? (if so, skip this presentation!)

Slayter 8

Step 3: Define analytic task, choose

statistical test, example 1

Slayter 9

Step 3: Define analytic task, choose

statistical test, example 2

Slayter 10

Step 3: Define analytic task, choose

statistical test, example 3

Slayter 11

On choosing statistical tests for bivariate analyses

Slayter 12

Requirements for each bivariate statistical test differ

• Comparing groups is one analytic task

• What you are comparing them ON is what matters with this choice

Slayter 13

Link variable type to appropriate statistical test for analytic purpose

Slayter 14

Check assumptions that must be met to conduct statistical testing

• Each statistical test is based on assumptions

• Must make sure assumptions are met or test results are spurious

• Common assumptions: Sample size-related, distribution-related, structure of variable

Slayter 15

Cheat sheet for choosing bivariate statistics

Analytic purpose you are seeking help for:

Test name: Basic requirements: How to interpret:

Comparing 2 or more groups on a percentage/rate

Chi-square test Variable to be compared is nominal

Interpret p value for statistical significance only

Comparing 2 groups only on a percentage/rate

Odds ratio test Variable to be compared is nominal

Interpret odds (1.23 times more likely, 0.40 = 60% less likely)

Comparing mean difference between 2 groups

Independent samples t-test

Variable to be compared is continuous

Interpret p value for statistical significance only

Comparing mean difference between pre and post in one group

Paired samples t-test

Variable to be compared is continuous

Interpret p value for statistical significance only

Comparing mean difference between 2 or more groups

Analysis of Variance (ANOVA)

Variable to be compared is continuous

Interpret p value for statistical significance only, conduct post-hoc test to identify divergent group

Slayter 16

Step 4: Envision finished product

• Make sure your data analysis plan makes sense – is this what I want when I am done?

• Consider using a “table shell”– Tables with no data in them – and a process– Take each question and think about how you will

use it analytically– Fill in which test you will use so you know what

you are doing ahead of time (creating a recipe for yourself)

17Slayter

Step 4: Envision finished product

• Use table shells to clarify thinking pre-data collection

Table 1: Comparison of student engagement rates between class sections

Variable: Target section:N=33

Comparison section:N=31

Statistical test result:

Self-reported engagement score#

Mean =3.5 Mean =2.5 t=2.45* (Independent samples t-

test result)

% reporting increased engagement at end of semester

67% 32% OR=3.42** (Odds ratio test result)

Χ2=3.46*** (Chi-square test result)

# - Likert scale indicates 1=not engaged to 5=totally engaged*** p<.001; **p<.01; *p<.05

Slayter 18

Step 4: Envision finished product

• Checklist once table shells are complete:Do I ask all necessary questions to get my data

into the shape I envision here?Do I ask questions that give me answers in the

correct format?Nominal vs. continuous variable for appropriate

statistical test to meet analytic purpose

Slayter 19

You are good to go!

• Check out my mid-term FLC portfolio from AY 10-11 for table shell examples

• If all else fails, get in touch with questions – eslayter@salemstate.edu x7459

• Good luck with your projects and have fun!

Slayter 20