1 i n s t i t u t e of m u s e u m and l i b r a r y s e r v i c e s washington, dc, knowing what...

1

I N S T I T U T E I N S T I T U T E ofof M U S E U MM U S E U M and and L I B R A R Y S E R L I B R A R Y S E R V I C E SV I C E SWashington, DC , www.imls.govWashington, DC , www.imls.gov

Knowing What Audiences Learn:Knowing What Audiences Learn: Outcomes and Program PlanningOutcomes and Program Planning

Association of Children’sAssociation of Children’sMuseumsMuseums

20032003

2

OverviewOverview

We willWe will Distinguish Outcome-based Planning and Distinguish Outcome-based Planning and

Evaluation (“OBE”) from other kindsEvaluation (“OBE”) from other kinds

Talk about choosing outcomesTalk about choosing outcomes Talk about basic elements of a Logic Model (a Talk about basic elements of a Logic Model (a

project or program plan)project or program plan)

Talk about measuring outcomesTalk about measuring outcomes

Review and summarizeReview and summarize

3

Outcomes are achievements or changes inOutcomes are achievements or changes in

Skill –Skill – Painting, basketball Painting, basketball

Knowledge –Knowledge – Zoology, state capitols Zoology, state capitols

Behavior – Behavior – Visits museums, reads dailyVisits museums, reads daily

Attitude –Attitude – I like science, I love animalsI like science, I love animals

What are Outcomes?What are Outcomes?

4

Outcomes can be achievements or Outcomes can be achievements or changes inchanges in

Status –Status – In school, citizenIn school, citizen Life condition –Life condition – Overweight, healthyOverweight, healthy


5


Ella goes to the Zoo’s summer programElla goes to the Zoo’s summer programandand

(Immediate)(Immediate) Takes more interest in animalsTakes more interest in animals

(Medium)(Medium) Gets better biology gradesGets better biology grades

(Long term)(Long term) Becomes a veterinary technicianBecomes a veterinary technician

6

OutcomesOutcomes

Where do they come from?Where do they come from? Social Services and United WaySocial Services and United Way OMB: (GPRA) 1993 and PARTOMB: (GPRA) 1993 and PART Trends in funding Trends in funding The need to focus on audienceThe need to focus on audience The need to communicate museum valueThe need to communicate museum value

IMLSIMLS

7

Forms of EvaluationForms of Evaluation

Formative EvaluationFormative Evaluation

You want to know which brochure works You want to know which brochure works best to bring school tours to the zoo best to bring school tours to the zoo

Develop two or three prototypes and test Develop two or three prototypes and test on a sample of your target audience; on a sample of your target audience; decide which worked bestdecide which worked best

8

Process EvaluationProcess Evaluation

You want to see how efficiently your You want to see how efficiently your conservation education program is runconservation education program is run

You count the number of participants in You count the number of participants in program components, examine how the program components, examine how the components are delivered, how long they components are delivered, how long they take, and what it coststake, and what it costs


9


Summative EvaluationSummative Evaluation

You want to know if your exhibit program You want to know if your exhibit program made visitors aware of how they can made visitors aware of how they can protect their environmentprotect their environment

You run the program, then interview visitors You run the program, then interview visitors to learn how many might use “green” to learn how many might use “green” products more in futureproducts more in future

10


Impact Evaluation (aggregated Impact Evaluation (aggregated outcomes create impact)outcomes create impact)

You want to know if your programs helped You want to know if your programs helped increase recycling in your areaincrease recycling in your area

You find out the rate of recycling, run the You find out the rate of recycling, run the program, then see if the recycling rate has program, then see if the recycling rate has increasedincreased

11


Outcome-Based Program EvaluationOutcome-Based Program Evaluation

You want to know if your conservation You want to know if your conservation education program changes participants’ education program changes participants’ behaviors behaviors

You identify audience needs, plan services You identify audience needs, plan services to provide participant-oriented outcomes, to provide participant-oriented outcomes, and assess program results on a regular and assess program results on a regular basisbasis

12

What can OBE achieve?What can OBE achieve?

Increase program effectivenessIncrease program effectiveness

Provide a logical, focused framework Provide a logical, focused framework to guide program designto guide program design

Inform decision-making Inform decision-making

Document successes Document successes

Communicate program valueCommunicate program value

13

What are its limitations?What are its limitations?

This is a management toolThis is a management tool OBE may suggest cause and effect OBE may suggest cause and effect ––

it doesn't try to prove it it doesn't try to prove it

OBE settles for contribution, not OBE settles for contribution, not attributionattribution

OBE uses some of the same OBE uses some of the same methods as research, but methods as research, but ……

14

What are its limitations?What are its limitations?

OBE is not the same as RESEARCH:OBE is not the same as RESEARCH: Doesn’t try to compare your programDoesn’t try to compare your program

with another, similar programwith another, similar program

Doesn’t try to compare your methodsDoesn’t try to compare your methods with different methods to create with different methods to create a similar resulta similar result

Accepts “good enough” dataAccepts “good enough” data

15

How to develop an outcome-based How to develop an outcome-based program: program: ExampleExample

What need do you see?What need do you see? Many kids don’t understand basic science principles; Many kids don’t understand basic science principles;

many girls are intimidated by or uninterested in many girls are intimidated by or uninterested in sciencescience

Head Start is an opportunity for early science Head Start is an opportunity for early science experience, but many teachers have no science experience, but many teachers have no science education and don’t know how to make it funeducation and don’t know how to make it fun

Parents in at-risk families often have little science Parents in at-risk families often have little science knowledge, limited child-development skills, and knowledge, limited child-development skills, and limited reading skillslimited reading skills

16

Who has the need (target audience)?Who has the need (target audience)? Kids need science experienceKids need science experience

Head Start teachers could provide experience if Head Start teachers could provide experience if they had knowledge and skillsthey had knowledge and skills

Families might encourage science learning if we Families might encourage science learning if we made it easy and funmade it easy and fun

Who could you work with most easily Who could you work with most easily and effectively? and effectively? Head Start teachersHead Start teachers


17

What could your museum do?What could your museum do? Provide science education resources for Head Start Provide science education resources for Head Start

Teachers – learning kits and training to use themTeachers – learning kits and training to use them

Provide take-home science kitsProvide take-home science kits

Partner with public libraries to provide age- and Partner with public libraries to provide age- and education-appropriate books on science subjectseducation-appropriate books on science subjects

MESS – Marvelous Explorations MESS – Marvelous Explorations through Science and Stories!through Science and Stories!


18

Planning for OutcomesPlanning for Outcomes

Articulates process Articulates process andand outcomes of outcomes of your programyour program

Clarifies each element of your programClarifies each element of your program

Identifies indicators of change to be Identifies indicators of change to be measuredmeasured

Identifies targets for program impactIdentifies targets for program impact

The Logic ModelThe Logic Model

19

The amount of impact desired

When data is collected

The population to be measured

Sources of information about conditions being measured

Observable and measurable behaviors or conditions

Targets Targets (Goals)(Goals)

Data IntervalData Interval(When)(When)

Applied to Applied to (Who)(Who)

Data Data SourcesSourcesIndicatorsIndicators

What goes into outcomes planningWhat goes into outcomes planning

Mission + Influencers PROGRAM PURPOSE

OutputsActivities ServicesInputs Outcomes

Outcomes Logic ModelOutcomes Logic Model

20

InfluencersInfluencers

To change process To add partners

To change responsibilities

Is responsibility equal?

Which services produce outcomes?

Project Partners

Who does the program serve?

Is the program effective?

To fund program To increase funding To promote replication

Funders

Is the program meeting target audience needs?

To improve the program To end program To start another program

Your Organization

How they will How they will use informationuse information

What they wantWhat they wantto knowto know

InfluencerInfluencer

GoalWhenWhoData Source

IndicatorsOutcomesAudienceActivities Services

ProgramPurpose

Mission Influencers

Staff, participants, agencies, funding sources, Staff, participants, agencies, funding sources, competition, community groups, professional competition, community groups, professional associations associations

21

Logic Model: MissionLogic Model: Mission

How do the organization’s mission How do the organization’s mission and the program connect?and the program connect?

Is the link between mission and Is the link between mission and program purpose reasonable?program purpose reasonable?

What action words connect the What action words connect the mission to the program?mission to the program?

GoalWhenWhoData

Source IndicatorsOutcomesAudience

ActivitiesServices

ProgramPurposeMission Influencers

22

We do what, for whom, for what outcome or We do what, for whom, for what outcome or benefit?benefit?

Program PurposeProgram Purpose

The Science Museum, Public Libraries, and School District The Science Museum, Public Libraries, and School District will partner to provide science kits, training, reference will partner to provide science kits, training, reference materials, and take-home kits for Head Start teachers to materials, and take-home kits for Head Start teachers to increase teachers’ ability to make science learning fun, increase teachers’ ability to make science learning fun, increase kids’ science interest and knowledge, and engage increase kids’ science interest and knowledge, and engage parents in science playparents in science play

GoalWhenWhoData


ActivitiesServices

ProgramPurpose

Mission Influencers

23

Logic Model: Activities/ServicesLogic Model: Activities/Services

Program Program activitiesactivities make it possible to deliver make it possible to deliver services to program participants services to program participants

(Most “activities” are managerial or administrative)(Most “activities” are managerial or administrative)

Program Program servicesservices engage participants and engage participants and produce outcomesproduce outcomes

Program Program services and activitiesservices and activities are driven by are driven by audience characteristicsaudience characteristics

GoalWhenWhoData

Source IndicatorsOutcomesAudienceActivities

ServicesProgramPurposeMission Influencers

24

ActivitiesActivities

Create kitsCreate kits Design trainingDesign training Workshop LogisticsWorkshop Logistics

ServicesServices Teacher trainingTeacher training MESS kits and booksMESS kits and books Take home kits Take home kits

A Museum /Library/School CollaborationA Museum /Library/School Collaboration

25

Target AudienceTarget Audience Head Start teachers in Alachua Co.Head Start teachers in Alachua Co. 3 to 5 year-olds from at risk families in Alachua3 to 5 year-olds from at risk families in Alachua Families in Alachua Co.Families in Alachua Co.

Knowing if you reaching the intended audience is Knowing if you reaching the intended audience is critical–you must decidecritical–you must decide

What information is critical to know?What information is critical to know? How will I get the information?How will I get the information? What are the confidentiality issues?What are the confidentiality issues?

GoalWhenWhoData

Source IndicatorsOutcomesAudienceActivitiesServices


26

Logic Model: OutcomesLogic Model: Outcomes

OutcomesOutcomes

State how you expect people to benefit State how you expect people to benefit from your programfrom your program

State the intended results of your services State the intended results of your services

Describe changes in skills, attitudes, Describe changes in skills, attitudes, behaviors, knowledgebehaviors, knowledge

GoalWhenWhoData


ActivitiesServices


27

SamplesSamples

Outcome 1Outcome 1

Teachers will be more confident helping Teachers will be more confident helping students learn about sciencestudents learn about science

Outcome 2Outcome 2

Teachers will include science experiences Teachers will include science experiences in their classroomsin their classrooms

GoalWhenWhoData


ActivitiesServices


28

Logic Model: IndicatorsLogic Model: Indicators

IndicatorsIndicators

Are Are measurablemeasurable conditions or behaviors conditions or behaviors that can show an outcome was achievedthat can show an outcome was achieved

Say what you hope to see or knowSay what you hope to see or know

Are observable evidence of Are observable evidence of accomplishments, changes, or gainsaccomplishments, changes, or gains

““Stand for” the outcomeStand for” the outcome

GoalWhenWhoData



29

IndicatorsIndicators

The # and % of teachers who say they have at least “some” confidence on a 5-point scale (none, a little, some, a lot, complete confidence)

Teachers will be Teachers will be more confident more confident helping students helping students learn about sciencelearn about science

IndicatorsOutcome 1Outcome 1

GoalWhenWhoData



30

Logic Model: Data SourcesLogic Model: Data Sources

Data sources are tools and locations for Data sources are tools and locations for information that will show what happened information that will show what happened other forms of informationother forms of information

Pre/Post test scoresPre/Post test scores Program recordsProgram records Assessment reportsAssessment reports Records from other organizationsRecords from other organizations ObservationsObservations

GoalWhenWhoData


ActivitiesServices


31

Data SourcesData Sources

Teachersurveys

Data Sources

The # and % of teachers who say they have at least “some” confidence on a 5-point scale

Teachers will Teachers will be more be more confident confident helping helping students learn students learn about scienceabout science

IndicatorsIndicatorsOutcome 1Outcome 1

32

Logic Model: Applied to (Who)Logic Model: Applied to (Who)

Decide if you will measure all Decide if you will measure all participants, completers of the participants, completers of the program, or another subgroup program, or another subgroup

Special characteristics of the target Special characteristics of the target audience can further clarify the group audience can further clarify the group to be measuredto be measured

GoalWhenWhoData


ActivitiesServices


33

Applied toApplied to

All teachers N = 445

Applied to

Teacher surveys

Data SourcesData Sources


Teachers Teachers will be more will be more confident confident helping helping students students learn about learn about sciencescience


34

Logic Model: Data Intervals (When)Logic Model: Data Intervals (When)

Outcome information can be collected at specific Outcome information can be collected at specific intervals (end of program, every 6 months)intervals (end of program, every 6 months)

Data can be collected at the end of an activity or Data can be collected at the end of an activity or phase and as follow-up (after 3 workshops, after 2 phase and as follow-up (after 3 workshops, after 2 yearsyears

Data is usually collected at program start and end Data is usually collected at program start and end for comparison when increases in skill, behavior, for comparison when increases in skill, behavior, or knowledge are expectedor knowledge are expected

GoalWhenWhoData


ActivitiesServices


35

Data IntervalData Interval

At end of program year 1

Data Interval

All partici- pating teachers

N = 445

Applied toApplied to

Teacher survey

Data Data SourcesSources


Teachers will Teachers will be more be more confident confident helping helping students students learn about learn about sciencescience


36

Logic Model: Targets (Goals)Logic Model: Targets (Goals)

Targets (goals) are chosen expectations Targets (goals) are chosen expectations for outcomes a program hopes to achieve for outcomes a program hopes to achieve usually a percentage and/or numberusually a percentage and/or number

Influencer expectations affect targets, Influencer expectations affect targets, which can also be based on a program’s which can also be based on a program’s past performancepast performance

TargetWhenWhoData


ActivitiesServices


37

TargetTarget

At end of program year 1

Data Data IntervalInterval

All partici- pating teachers

N = 445

Applied Applied toto

Teacher surveys

Data Data SourcesSources


Teachers Teachers will be will be more more confident confident helping helping students students learn learn about about sciencescience


50%

Target

38

What should reports say?What should reports say?

We wanted to do what?We wanted to do what? We did what?We did what? So what?So what?

Above allAbove all Reports should meet influencer needs forReports should meet influencer needs for

information and program results information and program results

Reports should guide program staff toReports should guide program staff toimprove outcomes for program participantsimprove outcomes for program participants

39

What will it cost?What will it cost?

Assume 7-10% of program costs for Assume 7-10% of program costs for program evaluation (non-research)program evaluation (non-research)

What will you get?What will you get?

Low cost –Low cost – Know numbers, Know numbers, audience characteristics, and audience characteristics, and customer satisfactioncustomer satisfaction

Low to moderate cost –Low to moderate cost – Know Know changes in audience skills, changes in audience skills, knowledge, behaviors, and attitudesknowledge, behaviors, and attitudes

40

What will you get?What will you get?

Moderate to high cost –Moderate to high cost – Comparison groups can show Comparison groups can show attribution, short-term changes due attribution, short-term changes due to program to program

High cost –High cost – Long-term follow up, Long-term follow up, can attribute long-term changes to can attribute long-term changes to audience due to program services audience due to program services (research)(research)

41

Karen MotylewskiKaren MotylewskiInstitute of Museum and Library ServicesInstitute of Museum and Library Services1100 Pennsylvania Avenue, NW1100 Pennsylvania Avenue, NWWashington, DC 20506Washington, DC 20506

202-606-5551202-606-5551http://www.imls.govhttp://[email protected]@imls.gov

For more informationFor more information

1 i n s t i t u t e of m u s e u m and l i b r a r y s e r v i c e s washington, dc, knowing what...

Documents