program evaluation presentation adapted from one developed by emily rothman, scd boston university
TRANSCRIPT
Program Evaluation
Presentation adapted from one developed by
Emily Rothman, ScDBoston University
Process Evaluation
∗Are you already collecting process evaluation data? How? What types of information?
Questions
∗Look at your logic model∗Brainstorm the type of process evaluation
data you will need to collect for you program∗Jot down how you will track the information
Small Group Activity
Process Indicator How will you track it?
Who will track it?
Attendance Weekly sign-in sheet Teen Center Director
Participant satisfaction Monthly surveys Annual focus groups
Teen Center Director
Session Implementation
Implementation report forms & session outlines
Program staff
Designing Pre and Post-Tests
Concept / word Explanation
Survey This is the whole package—typically comprises multiple measures
Measure Bunch of questions (or “items”) that together assess a construct; may comprise scales
Questionnaire same thing as measure
Scale Sub-section of a measure or instrument
Inventory Same thing as measure; Measure of typical performance
Test Same thing as measure; Measure of best performance
Instrument Same thing as measure
Item Single question on a scale
Lingo
Operationalizes a concept
Gets people to answer in the best possible way
“Best” means most reliable and valid
Gives you something to compare with other research studies
Ensures your response options can be analyzed
What does a measure do?
∗Example: Fear of falling cannot be directly measured. It is a concept.
Operationalizing a concept
Are we truly trying to figure out…:❖How worried might fall?❖How likely might fall?❖How often think about falling?❖How often hold stair railings?❖How confident can perform without falling?
All slightly different takes on the same thing
Operationalizing a concept
∗“Bullying”
∗Being physically assaulted by a peer∗Being called names, or harassed repeatedly
by one or more peers∗Being intentionally excluded from activities
by peers∗Being harassed via email, text, or social
media repeatedly by one or more peers
Operationalize…
∗The set of questions do a better job that one single item would alone
∗Why? ∗ People try to outsmart surveys∗ People are inconsistent in answers∗ More is better
Why use a “measure” instead of a question?
∗Do you have high self-esteem?
Example: self-esteem
1. On the whole, I am satisfied with myself
2. At times I think I am no good at all. 3. I feel that I have a number of good qualities.
4. I am able to do things as well as most other people.
5. I feel I do not have much to be proud of.
6. I certainly feel useless at times.
7. I feel that I'm a person of worth, at least on an equal plane with others.
8. I wish I could have more respect for myself
9. All in all, I am inclined to feel that I am a failure
10. I take a positive attitude toward myself.
Example: self-esteem
∗Social scientists develop scales∗They test and re-test them with groups of
people∗They determine how reliable and valid the
measures are
Psychometrics of scales/measures
∗ Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year?∗ 65% said companies could avoid layoffs∗ 22% said couldn’t avoid layoffs ∗ 15% had no opinion
∗ Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year, or do you think layoffs are unavoidable?∗ 35% said companies could avoid layoffs∗ 41% said couldn’t avoid layoffs∗ 24% had no opinion
Wording changes responses
Payne, 1951
What is the connection to your logic model?
Name of Program: Prevention System Enhancement Strategy Problem Statement: Domestic violence is a significant public health concern; youth-serving, community-based agencies can make marked contributions to the creation of environments that disrupt IPV Goal Statement: Increase the number of youth-serving organizations in Newport, RI that use public health tools for primary prevention program planning, implementation and evaluation from 0 to 10 by March 2018.
Contextual and Influential FactorsStrategy-specific Influential Factors: Participating agencies are not domestic violence service providers yet have ongoing access to youth; we are operating on the assumption that they have the potential to influence youths’ knowledge, attitudes, beliefs, and behaviors. These organizations are already providing programs that aim to address issues like substance abuse, teen pregnancy, and youth violence that share risk and protective factors with IPV perpetration and victimization. The Prevention System Enhancement work relies on relationships with various community partners; changes in their leadership, structure or processes could impact strategy implementation and evaluation. Community Context: According to YRBS data, the prevalence of teen dating violence in Rhode Island is 8.2%. Newport has large transient/temporary populations due to: Naval War College, Salve Regina University, and being a tourist town. There are extreme socio-economic differences in a relatively small city; 25% of children under 18 live in poverty. Tourism creates a drinking/partying environment. There is a relatively high unemployment rate (10.4% annual average in 2012).
Inputs/Resources(What you have
now)
Needed Capacity(What you need to
build)
Strategies/Activities
Outputs
OutcomesShort-term Intermediate Long-term
Imp
act
Incr
ease
in y
outh
acc
ess
to p
reve
ntio
n pr
ogra
ms
that
dec
reas
e ris
k fa
ctor
s an
d in
crea
se p
rote
ctive
fa
ctor
s as
socia
ted
with
IPV
-Full-time community organizer-Full –time director of prevention- Curriculum from year’s one and two- website-organizations with two years’ experience in the program- Co-facilitation & evaluation support from the RI Coalition Against Domestic Violence- Long-standing partnerships with youth-serving organizations- Access to free meeting space at various community locations
-Funding for stipends & outside trainers -increase commitment of partner organizations to institutionalize primary prevention practices-additional partners to include in years 3-5
-4, 7 hour training sessions per year-10 learning pod meetings per year-at least 6 continuous quality improvement working group meetings per agency, per year-at least 15 one-on-one technical assistance sessions per year-organizational assessment of participating agencies
- updated curriculum-updated website with resource library-portfolio of completed assignments-Process evaluation & pre/post test data -organizational assessment data-evaluation report
Immediate -Increase in the # of staff members at partner organizations who have used public health tools to plan, implement, and evaluate at least one prevention program they facilitate -increase in the % of PPI participants who are comfortable using public health tools to plan and evaluate prevention programs
Intermediate -increase in the level of organizational commitment to institutionalizing primary prevention practices among partner organizations-increase in the % of PPI participants who intend to use public health tools for planning and evaluating future prevention programs
Long-term -increase in the # of theory-based primary prevention programs being offered at partner organizations- Increase in the # of youth-serving organizations that have integrated primary prevention concepts into organizational practice
∗Look at your logic model∗Identify an outcome∗How will you measure it?
∗ Using which existing measure? … OR∗ Write your own “measure”?∗ Write your own single-item question?
Right now…
Where to get measures
∗Lit search for measures∗Lit search for articles on same topic, see
what measures they used∗After you know the name of a measure, you
can sometimes Google it∗Usually you have to find it printed in a peer
review journal∗It is hard to find measures!
Where do you find measures?
Golden rule:
Do not make up survey questions out of your own head, ever, unless you really, really, really have to
Acceptable reasons for writing own question:∗Searched literature, none exist∗Searched literature, all measures are inappropriate
for some reason∗You are never going to publish your results or
compare with another existing source
Don’t make up measures unless…
∗Likert-type scale∗Yes/no questions∗Free response or qualitative sections
Types of Instruments
Creating an entire survey
∗Standard things:∗ Version (pre- or post-test)∗ Today’s date∗ Subject ID number?
∗Demographics∗ Race/ethnicity?∗ Sex?∗ Age?
∗Measures∗ Measure short-term, intermediate and long-term
outcomes?∗Nothing extra!
∗ Don’t ask any questions that you don’t have to—long surveys are bad.
What goes into a survey?
Respondent (R)…1. Doesn’t know the answer2. Cannot recall an accurate answer3. Doesn’t understand the question4. Doesn’t want to report an accurate answer
What makes unreliable and invalid questions?
∗Putting two questions in the same question:
Have you ever broken your arm or been too sick to go to class?
Question design pitfalls
∗Using technical jargon without defining it:
How frequently do you advocate for prophylactic use during sexual encounters?
Question design pitfalls?
∗Leading respondents:
Would you agree that this program is awful?
Question design pitfalls
∗Be wary of socially desirable or undesirable questions:
“Do you like pancakes?” vs.“Have you ever been diagnosed with a STI?”“How much money did you earn last year?”
Question design
∗Keep it simple
∗Place the burden on yourself to make things clear, not the respondents to figure things out∗ “What was your pattern of enrollment?” vs.∗ “Were you a part-time or full-time student?”
∗Keep questions short
Question design
∗No double negatives!
If you have not had sex in the past year, do not answer the next question
Question design
∗Neat, clean appearance∗Easy to read∗Large enough font∗No fancy serifs or font style∗No grammar or spelling mistakes
Formatting design
Begin designing your survey…
Work Session!