program evaluation presentation adapted from one developed by emily rothman, scd boston university

Program Evaluation

Presentation adapted from one developed by

Emily Rothman, ScDBoston University

Process Evaluation

∗Are you already collecting process evaluation data? How? What types of information?

Questions

∗Look at your logic model∗Brainstorm the type of process evaluation

data you will need to collect for you program∗Jot down how you will track the information

Small Group Activity

Process Indicator How will you track it?

Who will track it?

Attendance Weekly sign-in sheet Teen Center Director

Participant satisfaction Monthly surveys Annual focus groups

Teen Center Director

Session Implementation

Implementation report forms & session outlines

Program staff

Designing Pre and Post-Tests

Concept / word Explanation

Survey This is the whole package—typically comprises multiple measures

Measure Bunch of questions (or “items”) that together assess a construct; may comprise scales

Questionnaire same thing as measure

Scale Sub-section of a measure or instrument

Inventory Same thing as measure; Measure of typical performance

Test Same thing as measure; Measure of best performance

Instrument Same thing as measure

Item Single question on a scale

Lingo

Operationalizes a concept

Gets people to answer in the best possible way

“Best” means most reliable and valid

Gives you something to compare with other research studies

Ensures your response options can be analyzed

What does a measure do?

∗Example: Fear of falling cannot be directly measured. It is a concept.

Operationalizing a concept

Are we truly trying to figure out…:❖How worried might fall?❖How likely might fall?❖How often think about falling?❖How often hold stair railings?❖How confident can perform without falling?

All slightly different takes on the same thing

Operationalizing a concept

∗“Bullying”

∗Being physically assaulted by a peer∗Being called names, or harassed repeatedly

by one or more peers∗Being intentionally excluded from activities

by peers∗Being harassed via email, text, or social

media repeatedly by one or more peers

Operationalize…

∗The set of questions do a better job that one single item would alone

∗Why? ∗ People try to outsmart surveys∗ People are inconsistent in answers∗ More is better

Why use a “measure” instead of a question?

∗Do you have high self-esteem?

Example: self-esteem

1. On the whole, I am satisfied with myself

2. At times I think I am no good at all. 3. I feel that I have a number of good qualities.

4. I am able to do things as well as most other people.

5. I feel I do not have much to be proud of.

6. I certainly feel useless at times.

7. I feel that I'm a person of worth, at least on an equal plane with others.

8. I wish I could have more respect for myself

9. All in all, I am inclined to feel that I am a failure

10. I take a positive attitude toward myself.

Example: self-esteem

∗Social scientists develop scales∗They test and re-test them with groups of

people∗They determine how reliable and valid the

measures are

Psychometrics of scales/measures

∗ Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year?∗ 65% said companies could avoid layoffs∗ 22% said couldn’t avoid layoffs ∗ 15% had no opinion

∗ Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year, or do you think layoffs are unavoidable?∗ 35% said companies could avoid layoffs∗ 41% said couldn’t avoid layoffs∗ 24% had no opinion

Wording changes responses

Payne, 1951

What is the connection to your logic model?

Name of Program: Prevention System Enhancement Strategy Problem Statement: Domestic violence is a significant public health concern; youth-serving, community-based agencies can make marked contributions to the creation of environments that disrupt IPV Goal Statement: Increase the number of youth-serving organizations in Newport, RI that use public health tools for primary prevention program planning, implementation and evaluation from 0 to 10 by March 2018.

Contextual and Influential FactorsStrategy-specific Influential Factors: Participating agencies are not domestic violence service providers yet have ongoing access to youth; we are operating on the assumption that they have the potential to influence youths’ knowledge, attitudes, beliefs, and behaviors. These organizations are already providing programs that aim to address issues like substance abuse, teen pregnancy, and youth violence that share risk and protective factors with IPV perpetration and victimization. The Prevention System Enhancement work relies on relationships with various community partners; changes in their leadership, structure or processes could impact strategy implementation and evaluation. Community Context: According to YRBS data, the prevalence of teen dating violence in Rhode Island is 8.2%. Newport has large transient/temporary populations due to: Naval War College, Salve Regina University, and being a tourist town. There are extreme socio-economic differences in a relatively small city; 25% of children under 18 live in poverty. Tourism creates a drinking/partying environment. There is a relatively high unemployment rate (10.4% annual average in 2012).

Inputs/Resources(What you have

now)

Needed Capacity(What you need to

build)

Strategies/Activities

Outputs 

OutcomesShort-term Intermediate Long-term

Imp

act

Incr

ease

in y

outh

acc

ess

to p

reve

ntio

n pr

ogra

ms

that

dec

reas

e ris

k fa

ctor

s an

d in

crea

se p

rote

ctive

fa

ctor

s as

socia

ted

with

IPV

-Full-time community organizer-Full –time director of prevention- Curriculum from year’s one and two- website-organizations with two years’ experience in the program- Co-facilitation & evaluation support from the RI Coalition Against Domestic Violence- Long-standing partnerships with youth-serving organizations- Access to free meeting space at various community locations  

-Funding for stipends & outside trainers -increase commitment of partner organizations to institutionalize primary prevention practices-additional partners to include in years 3-5

-4, 7 hour training sessions per year-10 learning pod meetings per year-at least 6 continuous quality improvement working group meetings per agency, per year-at least 15 one-on-one technical assistance sessions per year-organizational assessment of participating agencies

- updated curriculum-updated website with resource library-portfolio of completed assignments-Process evaluation & pre/post test data -organizational assessment data-evaluation report

Immediate -Increase in the # of staff members at partner organizations who have used public health tools to plan, implement, and evaluate at least one prevention program they facilitate -increase in the % of PPI participants who are comfortable using public health tools to plan and evaluate prevention programs

Intermediate -increase in the level of organizational commitment to institutionalizing primary prevention practices among partner organizations-increase in the % of PPI participants who intend to use public health tools for planning and evaluating future prevention programs

Long-term -increase in the # of theory-based primary prevention programs being offered at partner organizations- Increase in the # of youth-serving organizations that have integrated primary prevention concepts into organizational practice  

∗Look at your logic model∗Identify an outcome∗How will you measure it?

∗ Using which existing measure? … OR∗ Write your own “measure”?∗ Write your own single-item question?

Right now…

Where to get measures

∗Lit search for measures∗Lit search for articles on same topic, see

what measures they used∗After you know the name of a measure, you

can sometimes Google it∗Usually you have to find it printed in a peer

review journal∗It is hard to find measures!

Where do you find measures?

Golden rule:

Do not make up survey questions out of your own head, ever, unless you really, really, really have to

Acceptable reasons for writing own question:∗Searched literature, none exist∗Searched literature, all measures are inappropriate

for some reason∗You are never going to publish your results or

compare with another existing source

Don’t make up measures unless…

∗Likert-type scale∗Yes/no questions∗Free response or qualitative sections

Types of Instruments

Creating an entire survey

∗Standard things:∗ Version (pre- or post-test)∗ Today’s date∗ Subject ID number?

∗Demographics∗ Race/ethnicity?∗ Sex?∗ Age?

∗Measures∗ Measure short-term, intermediate and long-term

outcomes?∗Nothing extra!

∗ Don’t ask any questions that you don’t have to—long surveys are bad.

What goes into a survey?

Respondent (R)…1. Doesn’t know the answer2. Cannot recall an accurate answer3. Doesn’t understand the question4. Doesn’t want to report an accurate answer

What makes unreliable and invalid questions?

∗Putting two questions in the same question:

Have you ever broken your arm or been too sick to go to class?

Question design pitfalls

∗Using technical jargon without defining it:

How frequently do you advocate for prophylactic use during sexual encounters?

Question design pitfalls?

∗Leading respondents:

Would you agree that this program is awful?

Question design pitfalls

∗Be wary of socially desirable or undesirable questions:

“Do you like pancakes?” vs.“Have you ever been diagnosed with a STI?”“How much money did you earn last year?”

Question design

∗Keep it simple

∗Place the burden on yourself to make things clear, not the respondents to figure things out∗ “What was your pattern of enrollment?” vs.∗ “Were you a part-time or full-time student?”

∗Keep questions short

Question design

∗No double negatives!

If you have not had sex in the past year, do not answer the next question

Question design

∗Neat, clean appearance∗Easy to read∗Large enough font∗No fancy serifs or font style∗No grammar or spelling mistakes

Formatting design

Begin designing your survey…

Work Session!