slayter on planning quant design for flc projects - may 2011
TRANSCRIPT
On planning your quantitative research design for a faculty learning community project
Elspeth Slayter, Ph.D.School of Social WorkSalem State University
Before you begin…don’t read these slides unless you…
• …Have skimmed a basic program evaluation text• …Know what your “intervention” will be
– Example: Increasing student engagement in research courses
• …Have framed your research question– Example: Does the use of a new type of pedagogical framework
increase self-reported student engagement at mid-semester?
• …Are pretty sure you want to compare groups– Example: Pre/post tests on one class, comparison of sections
Slayter 2
When babies are on the way, we prepare a nursery, yes?
• You must prepare a data analysis plan before the data are collected – and eventually – arrive!
• Mistake #1: Create survey, analyze data later, can lead to many problems – don’t do it!
Slayter 3
Metaphor: Preparation of nursery – Preparation of data
analysis plan• A somewhat iterative process
in choosing the following:– Over-arching research question
– Project goals
– Project objectives
– Data collection plan
– Identification of analytic task
– Choice of statistical tests
Slayter 4
Presentation Overview:How to approach this task
5Slayter
Step 1: Identify project research question,
goals, objectives, intervention• Example research question: How can I foster
student engagement in required research courses?– Example goal: To improve student engagement by
changing my teaching framework as explicated to students
– Example objective: To improve class participation grades after infusing new teaching framework in lecture, discussion, assignments, activities
Slayter 6
Step 1: Identify project research question,
goals, objectives, intervention
Slayter 7
Your goal is LINKAGE!!!
Step 2: Design data collection approach that
best answers your question
• Methods of assessing outcome must be best possible/feasible way to answer question– Will numbers or words answer the over-arching
question best?• Maybe quantitative?
– Maybe pre-post-test? Maybe not?
• Maybe qualitative? (if so, skip this presentation!)
Slayter 8
Step 3: Define analytic task, choose
statistical test, example 1
Slayter 9
Step 3: Define analytic task, choose
statistical test, example 2
Slayter 10
Step 3: Define analytic task, choose
statistical test, example 3
Slayter 11
On choosing statistical tests for bivariate analyses
Slayter 12
Requirements for each bivariate statistical test differ
• Comparing groups is one analytic task
• What you are comparing them ON is what matters with this choice
Slayter 13
Link variable type to appropriate statistical test for analytic purpose
Slayter 14
Check assumptions that must be met to conduct statistical testing
• Each statistical test is based on assumptions
• Must make sure assumptions are met or test results are spurious
• Common assumptions: Sample size-related, distribution-related, structure of variable
Slayter 15
Cheat sheet for choosing bivariate statistics
Analytic purpose you are seeking help for:
Test name: Basic requirements: How to interpret:
Comparing 2 or more groups on a percentage/rate
Chi-square test Variable to be compared is nominal
Interpret p value for statistical significance only
Comparing 2 groups only on a percentage/rate
Odds ratio test Variable to be compared is nominal
Interpret odds (1.23 times more likely, 0.40 = 60% less likely)
Comparing mean difference between 2 groups
Independent samples t-test
Variable to be compared is continuous
Interpret p value for statistical significance only
Comparing mean difference between pre and post in one group
Paired samples t-test
Variable to be compared is continuous
Interpret p value for statistical significance only
Comparing mean difference between 2 or more groups
Analysis of Variance (ANOVA)
Variable to be compared is continuous
Interpret p value for statistical significance only, conduct post-hoc test to identify divergent group
Slayter 16
Step 4: Envision finished product
• Make sure your data analysis plan makes sense – is this what I want when I am done?
• Consider using a “table shell”– Tables with no data in them – and a process– Take each question and think about how you will
use it analytically– Fill in which test you will use so you know what
you are doing ahead of time (creating a recipe for yourself)
17Slayter
Step 4: Envision finished product
• Use table shells to clarify thinking pre-data collection
Table 1: Comparison of student engagement rates between class sections
Variable: Target section:N=33
Comparison section:N=31
Statistical test result:
Self-reported engagement score#
Mean =3.5 Mean =2.5 t=2.45* (Independent samples t-
test result)
% reporting increased engagement at end of semester
67% 32% OR=3.42** (Odds ratio test result)
Χ2=3.46*** (Chi-square test result)
# - Likert scale indicates 1=not engaged to 5=totally engaged*** p<.001; **p<.01; *p<.05
Slayter 18
Step 4: Envision finished product
• Checklist once table shells are complete:Do I ask all necessary questions to get my data
into the shape I envision here?Do I ask questions that give me answers in the
correct format?Nominal vs. continuous variable for appropriate
statistical test to meet analytic purpose
Slayter 19
You are good to go!
• Check out my mid-term FLC portfolio from AY 10-11 for table shell examples
• If all else fails, get in touch with questions – [email protected] x7459
• Good luck with your projects and have fun!
Slayter 20