![Page 1: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/1.jpg)
Program Program EvaluationEvaluation
Krista S. SchumacherSchumacher Research Group
Ph.D. StudentEducational Psychology:
Research, Evaluation, Measurements & Statistics (REMS)Oklahoma State University
Prepared for the Oklahoma State Regents for Higher Education
2012 Summer Grant Writing Institute
![Page 2: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/2.jpg)
To Evaluate or to Assess? To Evaluate or to Assess? Technically speaking….
◦ Assessment Long-term outcomes, aggregated judgment
◦ Evaluation Short-term outcomes, “unique event” judgment
In education terms….◦ Assessment
Student learning outcomes
◦ Evaluation Research on programs/curricula to increase student
learning
[email protected] 918-284-7276 2
![Page 3: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/3.jpg)
Why Evaluate? Why Evaluate? How will you know your project is
progressing adequately to achieve objectives?
How will funders know your project was successful? ◦ Increasing emphasis placed on evaluation,
i.e., U.S. Department of Education National Science Foundation Substance Abuse and Mental Health Services
Administration (SAMHSA)
[email protected] 918-284-7276
![Page 4: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/4.jpg)
Why Evaluate?Why Evaluate?Improve the program –
◦ “Balancing the call to prove with the need to improve.” (W.K. Kellogg Foundation
Determine program effectiveness – ◦ Evaluation supports “accountability and quality
control” (Kellogg Foundation)
◦ Significant influence on program’s futureGenerate new knowledge –
◦ Not just research knowledge◦ Determines not just that a program works, but
analyzes how and why it works With whom is the program most successful? Under what circumstances?
[email protected] 918-284-7276
![Page 5: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/5.jpg)
Why Evaluate? Why Evaluate? WHAT WILL BE DONE WITH THE
RESULTS?????
“Evaluation results will be reviewed (quarterly, semi-
annually, annually) by the project advisory board and staff. Results
will be used to make program adjustments as needed.”
[email protected] 918-284-7276
![Page 6: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/6.jpg)
Federal Emphasis on Federal Emphasis on Scientifically Based Research Scientifically Based Research (SBR) in Evaluation(SBR) in EvaluationExperimental research design
◦Random assignmentQuasi-experimental research
design◦No random assignment
Program Evaluation Standards◦ Joint Committee on Standards for Educational Evaluation
1.Utility
2.Feasibility
3.Propriety
4.Accuracy
5.Evaluation [email protected] 918-284-7276 6
![Page 7: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/7.jpg)
Types of EvaluationTypes of Evaluation
Process evaluation: What processes are used and
how well do they work?
Outcome evaluation: Did the project achieve its stated
objectives?
[email protected] 918-284-7276
![Page 8: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/8.jpg)
Process EvaluationProcess Evaluation◦ What was provided and to whom?
services (modality, type, intensity, duration) recipients (individual demographics and
characteristics) gender, age, race/ethnicity, income level, first-generation status
context (institution, community, classroom) cost (did the project stay within budget?)
• Do processes match the proposed project plan?
◦What types of deviation from the plan occurred?◦What led to the deviations?◦What effect did the deviations have on the project
and evaluation?
[email protected] 918-284-7276
![Page 9: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/9.jpg)
Outcome EvaluationOutcome Evaluation◦ What effect did the program have on
participants? Activities / Objectives Achievement / Attitudes and beliefs
◦ What program/contextual factors were associated with outcomes?
◦ What individual factors were associated with outcomes?
◦ How durable were the effects? What correlations can be drawn between outcomes
and program? How do you know that the program was the cause of the
effect?
You can’t! Unless you use an experiment. [email protected] 918-284-7276 9
![Page 10: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/10.jpg)
Who will Evaluate?Who will Evaluate?External evaluators increasingly
required or strongly recommended◦Partners for effective and efficient
programs◦Methodological orientations◦Philosophical orientations◦Experience and qualifications
[email protected] 918-284-7276
![Page 11: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/11.jpg)
How much will it cost?How much will it cost?External evaluations cost
money…period.Standard recommendation:
5% to 10% of total budgetKellogg Foundation; U.S. Dept of Ed.; NSF
Check funder limits on evaluationEnsure cost is reasonable but
sufficient
[email protected] 918-284-7276
![Page 12: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/12.jpg)
Two Types of DataTwo Types of DataQuantitative
◦ Numbers based on objectives and activities◦ Types of data needed:
Number of participants (process) Grade point averages (outcome) Retention rates (outcome) Survey data (outcome and process)
Qualitative◦ Interviews◦ Focus groups ◦ Observation
[email protected] 918-284-7276
![Page 13: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/13.jpg)
Methods/InstrumentsMethods/InstrumentsHow are you going to get your
data? ◦ Establish baseline data◦ Institutional Research Office (I.R.)
GPA Retention Graduation
◦ Pre- and post-assessments (knowledge, skills)
◦ Pre- and post-surveys (attitudinal)◦ Enrollment rosters◦ Meeting minutes
[email protected] 918-284-7276
![Page 14: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/14.jpg)
Data AnalysisData AnalysisQualitative Data
◦ Data analysis programs NVivo , ATLAS.ti, etc…
◦ More than pithy anecdotes “May explain – and provide evidence of – those hard-to-
measure outcomes that cannot be defined quantitatively.” – W.K. Kellogg Foundation
Provides insight into how and why a program is successful
◦ Analyze for themes that support (or don’t) quantitative data
[email protected] 918-284-7276
![Page 15: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/15.jpg)
Data AnalysisData AnalysisQuantitative data
◦ Data analysis programs:◦ SPSS (Statistical Program for the Social Sciences), Stata, etc...
◦ Descriptive and inferential statistics: Descriptive:
Frequencies Means Standard deviation
Inferential (parametric, nonparametric) t-tests, Mann-Whitney U test (difference of means, two groups) ANOVA, Kruskal-Wallis, Friedman (difference among more than
two groups) Correlation (relationship between variables) Regression (explanation, prediction) Etc……
[email protected] 918-284-7276
![Page 16: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/16.jpg)
A Detour: Educational A Detour: Educational ResearchResearchFor education grants, two
approaches to evaluation: ◦Program Evaluation◦Educational Research
Include Educational Researcher as co-PI◦Look in Educational Psychology
programs◦OSU Center for Educational Research
and Evaluation (a.k.a. REMS Center)[email protected] 918-284-7276 16
![Page 17: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/17.jpg)
Educational Research Educational Research (cont.) (cont.) Educational research is NOT
laboratory science research Requires Institutional Review Board (IRB)
approval Oversight arm for Human Subjects Research
May require informed consent
How NOT to set up ed research◦One Biology section, split into two sections
One receives “treatment”; other traditional instruction
Students blindly enroll in one section WITHOUT knowledge this is an experiment
[email protected] 918-284-7276 17
![Page 18: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/18.jpg)
Educational Research Educational Research (cont.)(cont.)Example: NSF TUES
◦Incorporate interdisciplinary, semester-long research projects into Introductory Biology
◦Hypothesis:◦ “We hypothesize that semester-long,
interdisciplinary, collaborative research projects will increase student learning and interest in science more than the standard three-hour laboratory activity.”
[email protected] 918-284-7276 18
![Page 19: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/19.jpg)
Educational Research Educational Research (cont.)(cont.)Measurements follow from hypothesis
◦Increases in student learning: Establish baseline data
E.g., Performance on tests, course projects, course grades
Pre/post tests Course projects compared to previous courses
without intervention
◦Increases in student interest: Establish baseline data (if possible) Pre/post attitudinal surveys Focus groups
[email protected] 918-284-7276 19
![Page 20: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/20.jpg)
Data Collection & Reporting: Data Collection & Reporting: Two Types of TimeframesTwo Types of TimeframesFormative
◦Ongoing throughout life of grant◦Measures activities and objectives
Summative◦At conclusion of grant funding
NEED BOTH!
[email protected] 918-284-7276
![Page 21: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/21.jpg)
TimelinesTimelinesWhen will evaluation occur?
◦Monthly? ◦Quarterly? ◦Semi-annually? ◦Annually? ◦At the end of each training session? ◦At the end of each cycle?
[email protected] 918-284-7276
![Page 22: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/22.jpg)
Origin of the Evaluation: Origin of the Evaluation: Need and ObjectivesNeed and Objectives
Need: For 2005-06, the fall-to-fall retention rate of first-time degree-seeking students was 55% for the College’s full-time students, compared to national average retention rates of 65% for full-time students at comparable institutions (IPEDS, 2006).
Objective: The fall-to-fall retention rate of full-time undergraduate students will increase by 3% each year from a baseline of 55% to 61% by Sept. 30, 2010.
[email protected] 918-284-7276
![Page 23: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/23.jpg)
Evaluation Data Collection Evaluation Data Collection and Reporting Planand Reporting PlanObjectives
Data collected and timeline
Methods for data collection and timeline
Instruments to be developed and timeline
Reports/outcomes timeline
Increase fall-to-fall retention by 3% per year to 61%
Student enrollment in first fall and second fall within one month of start of second fall
Enrollment entered by gender and race/ethnicity into database within first four weeks of each semester
Enrollment rosters separated by gender and race/ethnicity by Jan. 15, 2009
At mid-point of each semester
[email protected] 918-284-7276
![Page 24: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/24.jpg)
BEWARE THE LAYERED BEWARE THE LAYERED OBJECTIVE!OBJECTIVE!
By the end of year five, five (5) full-time developmental education instructors will conduct 10 workshops on student retention strategies for 200 adjunct instructors.
[email protected] 918-284-7276
![Page 25: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/25.jpg)
Logic ModelsLogic Models
From: University of Wisconsin-Extension, Program Development and Evaluation
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
A Logic Model is……◦ A depiction of a program showing what the program will do and
what it is to accomplish.
◦ A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes
◦ The core of program planning and evaluation
25
Inputs Outputs Outcomes
Get food Eat food Feel betterHungry
Situation
[email protected] 918-284-7276
![Page 26: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/26.jpg)
Evaluation ResourcesEvaluation Resources Evaluation Resource Center for Advanced
Technological Education (NSF ATE) http://evalu-ate.org/ (directory of evaluators)
Western Michigan University, The Evaluation Center◦ http://ec.wmich.edu/evaldir/index.html (directory of evaluators)
American Evaluation Association◦ www.eval.org (directory of evaluators)
Joint Committee on Standards for Educational Evaluation
◦ http://www.jcsee.org/program-evaluation-standards/program-evaluation-standards-statements
W.K. Kellogg Foundation Evaluation Handbook◦ http://www.wkkf.org/knowledge-center/resources/2010/W-K-
Kellogg-Foundation-Evaluation-Handbook.aspx
[email protected] 918-284-7276
![Page 27: Program Evaluation Krista S. Schumacher Schumacher Research Group Ph.D. Student Educational Psychology: Research, Evaluation, Measurements & Statistics](https://reader036.vdocuments.site/reader036/viewer/2022062407/56649d885503460f94a6d58f/html5/thumbnails/27.jpg)
Evaluation & Statistics Evaluation & Statistics ResourcesResources
Statistics and Research Methods Resources◦ http://statsandmethods.weebly.com/
The Research Methods Knowledge Base◦ http://www.socialresearchmethods.net/
“Discovering Statistics Using SPSS,” by Andy Field◦ http://www.sagepub.com/field3e/
Planning an Effective Program Evaluation short course◦ http://www.the-aps.org/education/promote/pen.htm
“Evaluation for the Unevaluated” course◦ http://pathwayscourses.samhsa.gov/eval101/
eval101_toc.htm
OSRHE list of evaluators and other resources◦ http://www.okhighered.org/grant%2Dopps/writing.shtml
[email protected] [email protected] 918-284-7276