evaluation capacity building sampler
DESCRIPTION
Evaluation Capacity Building Sampler. Michael Duttweiler , Assistant Director for Program Development and Accountability Monica Hargraves , Manager of Evaluation for Extension and Outreach Cornell Office for Research on Evaluation. Topics. A little context about CCE - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/1.jpg)
Michael Duttweiler, Assistant Director for Program Development and
Accountability
Monica Hargraves, Manager of Evaluation for Extension and Outreach
Cornell Office for Research on Evaluation
![Page 2: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/2.jpg)
TopicsA little context about CCESupporting and Sustaining Evaluation
PracticeConsiderationsTraditional ApproachesNot-so Traditional Approaches
Featured Evaluation: Family Economics and Resource Management
![Page 3: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/3.jpg)
Cornell Cooperative ExtensionLarge and decentralized systemLocal units are subordinate agencies of state
government in partnership with county government and Cornell
Local staff are not Cornell employeesStrong local determination of programmingLocal funding dominates at the local level
Statewide evaluation practice is glued together by common needs, good will, and strong relationships.
![Page 4: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/4.jpg)
“The biggest barrier to stronger program evaluation in extension is lack of practical evaluation approaches and tools.”
Quick Poll – Agree/Disagree
= Agree
= Disagree
![Page 5: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/5.jpg)
Supporting and Sustaining Evaluation PracticeConcept mapping process involving program
staff, county directors, extension administration, and evaluation consultants
“Concept mapping seeks the open contribution of participant stakeholders’ ideas on a specific issue, organizes the ideas, and portrays them in pictures or maps that are readily understood.”
Kane and Trochim, 2007“Concept Mapping for Planning and Evaluation”
Focus question: One specific thing an Extension organization can do to support the practice of evaluation is…"
![Page 6: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/6.jpg)
Conclusions …There is a strong human element inherent in
supporting and sustaining evaluation practices. People at all levels of an organization, including paid and volunteer staff, need to have their own good reasons to care about evaluation.
Effective evaluation is not just about the programs, it’s about the organization.
Evaluation should be integrated into organizations in all ways – into job descriptions and performance reviews, strategic planning, staff discussions, external reporting, proposal development, etc.
Evaluation Policies should include all phases of evaluation: planning, implementation, and utilization
For example, “good use” of evaluation is important in motivating staff
![Page 7: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/7.jpg)
Traditional Capacity Building in CCEIn-service Education
“Getting Started with Program Evaluation”“Exploring Public Value”
Web ResourcesCourse ModulesReferences and Tools
Technical AssistancePlan of Work Outcomes and ReportingTargeted Program Evaluation
![Page 8: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/8.jpg)
Webinar Wrap-Around
![Page 9: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/9.jpg)
Program Leadership CertificateWhat it is
Comprehensive professional development experience
At least 12 modules and an applied project over ~18 months
Evaluation ComponentsIntro to Program EvaluationAccountability and EvaluationEvaluation TopicsApplied Projects
![Page 10: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/10.jpg)
Evaluation Partnerships (EP): usingThe Systems Evaluation ProtocolEP Planning Phase entails …
Program recruitment and selection, MOU, cohort formation Extensive program modeling Evaluation Plan development Mix of in-person trainings, web-conferences, listserve etc.
EP Implementation support phase entails … Program and evaluation timelines clarified In-person “Kickoff” meeting with presentations, experts and
resources supporting hands-on work time Follow-up Q&A web-conferences on topics as needed End-of Year “Capstone” meeting for closure and
accountability
![Page 11: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/11.jpg)
Sequence of EP Cohorts in CCE
20062007
2007 and in 2009 cohort2009
Cohort Year (Planning Phase)
2010 ( indicates associations where 2010 cohort staff are located)
July 2010
2009 and in 2010 cohort
Rough counts: # associations … 34 with EP programs 27 with EP-trained staff in-house
# CCE staff participants … over 200 staff on Netway through EP 54 staff trained directly by CORE as EPMs or as working group members
# Programs … 102 programs listed in Netway (37 added since the initial EP work)
![Page 12: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/12.jpg)
Evaluation Partnership LessonsSustaining evaluation
Key is to integrate evaluation with existing work (program development, reporting, funding, etc.)
Higher levels of the “system” need to be aware, involved, supportive
Side-benefits matter Staff understanding of program; improved “ownership by
staff, volunteers, other stakeholders; Builds a network of educators with common tools and
language, and better opportunities for sharing resources and solutions
![Page 13: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/13.jpg)
More on Evaluation PartnershipsOctober 2009 Webinar “Systems Evaluation
Protocol: The Right Tools Through the Evaluation Cycle”
Cornell Office for Research on Evaluation: http://core.human.cornell.edu/
Questions?
![Page 14: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/14.jpg)
Targeted ProgramsWork with teams of educators and facultyDevelop an outcome frameworkDevelop an evaluation framework
DocumentationImmediate feedbackFollow-up feedback
![Page 15: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/15.jpg)
Financial Education Evaluation2006
Modified CCE POW2007
Intro NEFE Evaluation ToolbookWork group selected priority
outcomes2008
Piloted instruments for two programs
Wide use of revised instruments2009
Pilot data summariesWide use of standard forms
2010Pilot follow-up survey
![Page 16: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/16.jpg)
Educator’s PerspectiveAnn Gifford, Consumer & Financial
Management Program Coordinator, CCE Tompkins County
![Page 17: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/17.jpg)
“Making Ends Meet” Highlights89% rated program as “useful” or “very useful”98% would recommend program71-74% indicated increased confidence on five key
behaviors47-54% said they would take five key actions
(12-20% already were taking the actions)51% identified one or more new things learned31% identified at least one additional thing they
would do differently
![Page 18: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/18.jpg)
Follow Up Evaluation PilotWhat?
Small sample (target of 30) follow-up evaluation for Making Ends Meet
Why? To see if learning “sticks”Identify practice changesSolicit feedback
Where? When? Who? CCE TompkinsApril-May 2010Volunteer Interviewer
![Page 19: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/19.jpg)
The Questions (paraphrased)Things gained?Changes made as a result of attending the
workshop?Since the workshop, have you: set goals,
tracked spending, developed plan, made payments on time?
Learn anything from tracking expenses? Make any changes?
Use any new community resources?Confidence in managing money?Any other comments?
![Page 20: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/20.jpg)
Changes Made (93% made one or more changes)
Started tracking (8)Started saving money (6)Saving on unnecessary items (4)Started budgeting (3) Paying credit cards/bills on time (2)Reduced credit card debt (2)Went to financial advisor (2)Misc. comments (10)
![Page 21: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/21.jpg)
Quotable Quotes“It was very eye-opening to see how much and
where I spent my money.”“I realized I had lost control of my credit card
spending and did something about it.”“I got a piggy bank for my son and have him save
for things he wants.”“I’m able to save money even though I don’t have
much to work with.”“The workshop made finances less scary.”“I learned how to be responsible.”“This workshop helped me get my home loan. I
have a home!““It gave me hope.”
![Page 22: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/22.jpg)
Putting it All TogetherStatewide Needs Info
Program Description
Immediate Feedback
Optional Local Results
Follow-up Results
Follow-up Quotes
![Page 23: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/23.jpg)
Next Up: Field Crops ProgramsThis year: Basic program
documentation and immediate feedback pilot testing
Next year: Standard near-term evaluation
Third year: Follow-up impact documentation
![Page 24: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/24.jpg)
NEFE Evaluation Toolkithttp://www.nefe.org/eval/
![Page 25: Evaluation Capacity Building Sampler](https://reader033.vdocuments.site/reader033/viewer/2022051417/568148e5550346895db5ffcc/html5/thumbnails/25.jpg)
Questions/Comments?Presentation slides, documents, and links will
be available at the usual site:http://nc4-heval.wikispaces.com/
Webinars+for+2010