Introduction to EvaluationIntroduction to Evaluation

Tammy Horne, Ph.D. WellQuest Consulting Ltd. (780) 451-6145thorne@wellquestconsulting.comwww.wellquestconsulting.com

What is Evaluation?What is Evaluation?“Program evaluation is the

systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development…

04/19/23 2

What is Evaluation?What is Evaluation?…Program evaluation does not

occur in a vacuum and is influenced by real-world constraints. Evaluation should be practical and feasible and must be conducted within the confines of resources, time, and political context…

04/19/23 3

What is Evaluation?What is Evaluation?…Moreover, evaluation should

serve a useful purpose, be conducted in an ethical manner, and produce accurate findings. Evaluation findings should be used to make decisions about program implementation and to improve program effectiveness.” (Centers for Disease Control, 2001, p. 6)

www.cdc.gov/tobacco/evaluation_manual/Evaluation.pdf  

04/19/23 4

What Is Evaluation?What Is Evaluation?Formative evaluation – improve

existing program, ‘stabilize’ it for summative evaluation – focus is usually processes and sometimes early outcomes

Summative evaluation – judgment of merit or worth of program; make decisions about to keep, change, expand, cut – often emphasizes longer-term outcomes

04/19/23 5

Why Do We Evaluate?Why Do We Evaluate?To learn what is happening in our

programs (Processes - what we do and how people respond)

To learn what is changing, what differences we are making (Outcomes - desired or expected results; the changes we hope to see.)

To learn how processes and outcomes are related (Theory of change/program theory)

04/19/23 6

Why Do We Evaluate?Why Do We Evaluate?To learn about how our program is

influenced by other factors in our environments (Context)

The above kinds of learning help us to: Be accountable to funders Continue to improve our programs Share our knowledge with others doing similar work

04/19/23 7

Some Example Process Some Example Process QuestionsQuestionsWhat activities do we offer? How

often?For whom?

What types of people do we reach, and how?

Who expresses interest in our program?

04/19/23 8

Some Example Process Some Example Process QuestionsQuestions• How do we respect diversity – e.g.,

be inclusive of/responsive to people from particular cultural backgrounds?

How do people participate? (e.g., recipients of a service; partners in program planning and/or delivery, community builders or leaders, advocates for healthy policies/practices)

04/19/23 9

Some Example Process Some Example Process QuestionsQuestions

What makes it easy for people to participate? What makes it hard for people to participate? (barriers – if any)

Who stays? Who drops out and why?

What is running smoothly?

What ‘kinks’ still need to be worked out?

04/19/23 10

Some Example Process Some Example Process QuestionsQuestions

What do participants like best about the program?

What changes would they like to see?

What would they tell others about your program?

04/19/23 11

Some Example Outcome Some Example Outcome Questions: Individual LevelQuestions: Individual Level

•What new knowledge do participants learn?

• What new skills do they develop or strengthen?

• What changes do they make in their behaviour? (e.g., changes in personal lifestyle, relationships, community engagement)

04/19/23 12

Some Example Outcome Some Example Outcome Questions: Organizational Questions: Organizational LevelLevel

Do organizations you are working with get involved in new collaborations or partnerships,?

Do organizations develop more culturally competent practices that are responsive to the cultural groups with whom they work?

04/19/23 13

Some Example Outcome Some Example Outcome Questions: Community LevelQuestions: Community Level

How has community capacity been enhanced, such as:

People involved in community leadership?

New community structures/resources? (e.g., community gardens, walking trails)

People participating, with others, in community-building activities?

People sharing knowledge or skills?04/19/23 14

Some Example Outcome Some Example Outcome Questions: Community LevelQuestions: Community Level

How is the quality of community life improving? (e.g., positive relationships between youth & seniors, neighbours getting to know each other, better access to local food, people feel safe on streets or in parks)

04/19/23 15

Some Example Outcome Some Example Outcome Questions: Policy/System Questions: Policy/System LevelLevelWhat policy changes are happening in your school district in areas like: availability of healthy food, opportunities for physical activity, prevention of bullying and harassment?

How well are public non-smoking policies being enforced?

04/19/23 16

Some Example Outcome Some Example Outcome Questions: Policy/System Questions: Policy/System LevelLevelHow receptive are municipal councils to community members’ requests for recreation /parks opportunities, land for community gardens?

04/19/23 17