how do we begin? what are the different types of evaluation research? how do these different types...
TRANSCRIPT
Evaluation Research
How do we begin?
What are the different types of evaluation research?
How do these different types fit together?
What purpose do they serve?
Begin with a Logic Model
Step 1 – Start with Program Model
Step 2 – Identify
Resources
Step 3 – Identify Inputs
Step 4 Identify
Activities
Step 5 – Identify Outputs
Step 6 – Identify
Outcomes
Logic ModelStep 1 – Identify Program Model
Step 2 – Identify Inputs
Step 3 - Activities
Step 4 - Outputs
Step 5 - Outcomes
Step 1 – Identify Program Model
Step 2 – Identify
Resources
Step 3 – Identify Program Events
Step 4 – Identify
Resulting Products
Step 5 – Identify ChangesSuccess
Other Common Terms Used to Identify Steps
Logic Model
Example – Service Learning Project
Program Model Inputs Activities Outputs
Outcomes
Program Model
Teachers Grant Money
Service LearningActivities
Service Learning Projects
Completed
Measure Leadership
Skills
Example
Program Model
# of TeacherAmount of Money for Building Materials
Number of Service Learning Projects Planned
by Students
Number of Projects
Completed
Perception of Leadership Skills
Developed
Program Model
Teachers Grant Money
Service LearningActivities
Service Learning Projects
Completed
Leadership Skills
Another Conceptualization of the Logic Model
Resources
•Step One – Identify and count resources used to instigate program
Activities
•Step Two – Identify and count activities that took place
Outputs
•Step Three – Identify and count events or products that resulted
Short-Term
Outcomes
•Step Four – Identify and count short-term changes that took place (i.e., positive changes or accomplishments immediately after participation)
Long Term
Outcomes
•Step Five – Identify and count long-term impacts that occurred (i.e., positive changes or accomplishments at specified future date)
Service Learning Example Using This Conceptualization
Resources
•Count number of staff and amount of money used to instigate program. Count number of networks established.
Activities
•Count number of planning meetings and number of service learning activities planned by students. Count number of hours students spent in activities.
Outputs
•Count number of projects completed by students. This could be subdivided into things such as number of picnic tables built for a park etc.
Short-Term
Outcomes
•Use survey to assess positive changes or accomplishments as perceived by service learning participants. Compare to control group.
Long Term
Outcomes
•At a specified future time, use survey to assess positive changes in the lives of students (i.e., number of hours spent in volunteer work etc.) Compare to control group
Another Example Using This Conceptualization
Resources
•Count number of staff and amount of money used to instigate 4-H program. Count number of networks established.
Activities
•Number of 4-H activities that occurred, and/or number of students who participated in planning activities
Outputs
•Number of students who participated in workshops and projects. Number of educational packets distributed.
Short-Term
Outcomes
•Assessment of their sense of accomplishment using survey. Assessment of increase in knowledge using survey.
Long Term
Outcomes
•Number of 4-H students who attend college after attending educational workshops. Compare to control group.
Types of Evaluation Research In order of Use
Outcome Evaluation
Process Evaluation
Formative Evaluation
Needs Assessment
Example - Surveys, intensive interviews and focus groups are used to determine (1) Long-term impact on USU students and (2)Long term -
impact on high school
Outcome Evaluation
Process Evaluation
An ongoing evaluation is used to monitor the program
Example -Grantee comes from time to time to “see how things
are going”.Formative EvaluationInformation is gathered to
formulate a plan to establish a program.
Example-Planning meetings with high school administrators
etc.
Needs Assessment
A need for a program is identified
Example –need for mentoring program with public school
students
An Example of Each Type of Evaluation
Needs Assessment Purpose
Evaluate the need for a program/intervention/ project
Basic Steps Use secondary data (i.e., census data, community
reports, etc.) and/or surveys, etc. to create a profile of the community that assess existing resources and current needs
Determine if current resources are adequately meeting needs
Identify one or more specific needs that are not currently being met
Example - Needs Assessment Purpose
Determine the percent of local post secondary students that are from an underrepresented population. Compare this to the percent of students in post secondary educational institutions. Based on this and programs already available determine the need to provide additional opportunities for this group.
Basic Steps
Use secondary data to determine percent of population that is Latino. Use USU data to compute the percent of USU that is Latino. Compare differences in percentages. Note any discrepancy.
Utilize local webpages and phone books, identify
programs/organizations that focus on the Latino population and meeting their needs.
Formative Evaluation Purpose
Plan (form) a program that will meet the needs of the community, keeping in mind the need to develop indicators that can be used to evaluate the success of the program
Basic Steps Locate other similar programs to use as a model
Meet with “experts” in this area and get their advice
Organize a program and apply for a grant
Form an “ad hoc” evaluation committee to evaluate program
Formative Evaluation Purpose
Plan a mentoring program where USU students mentor high school students. Identify objectives (i.e., increase high school graduation rates, increase college retention rates of mentees).
Basic Steps Identify other mentoring programs
Meet with directors of programs, professors, etc. who have been/are involved in mentoring programs
Organize your mentoring program and apply for a grant to implement program
Select members of your committee to engage in evaluation research
Process Evaluation
Purpose To monitor the implementation of your program on an
ongoing basis. It is a type of program monitoring.
Steps Specifically describe your program
Decide what products to count Consider objectives (verb) (target)
(date)
Decide on valid ways to count products Identify process indicators
Quality assurance Identify process indicators
Example - Process Evaluation
Purpose Monitor the mentoring program and address two questions:
(1) Is the mentoring occurring as planned? (2) How are the mentees
responding?
Steps Describe program
The mentoring program connects high school mentees with USU mentors through a monitored blog
Decide what products to count The number of hours that mentors spent in contact with
their mentee, etc.
Decide on valid way to count products Have program administrator count hours spent on website
Quality assurance High school counselors will be contacted periodically to
determine how mentees are responding
Outcome Evaluation
Purpose To determine how well you meet the objectives of
your program
Steps Identify short-term and long-term outcomes
Construct indicators to measure outcomes
Decide on data collection and sampling techniques
Collect data
Analyze data
Write Report
Outcome Evaluation
Purpose To determine how well you meet the objectives of your program
Steps Short-term outcomes are to establish relationship between
mentors and mentees
Long-term outcomes are higher graduation and retention rates for mentees.
Mentees will be asked questions about their experience, and then about their graduation/retention
Surveys will be administered to all mentees after completion of program and then again in three years
Surveys are administered
Using SAS, data is entered and analyzed
Report is written
Quality Assurance vs. Program Outcome Evaluation
Inform Administrators and
Clients
Rely on “Expert” Opinion
Inform Administrators
Rely on Statistics
Quality Assurance
Legislative Mandate
Client Satisfaction is Focus
Outcome Evaluation
Not Mandated
Does Not Necessarily Focus on Client
Satisfaction
Questions or comments?Please contact:
Carol AlbrechtAssessment Specialist
USU Extension979-777-2421