Evaluation Best Practices
Results Based & Logic Model ProgramsResults Based & Logic Model Programs
Bob Craven, Director – Digital CenterBob Craven, Director – Digital CenterBrad Lubben, Director – North Central RegionBrad Lubben, Director – North Central RegionJo Ann Warner – Assistant Director – Western RegionJo Ann Warner – Assistant Director – Western RegionKevin Klair – Digital CenterKevin Klair – Digital Center
With help from: With help from: Kynda Curtis, Utah State UniversityKynda Curtis, Utah State UniversityGary Hachfeld, University of Minnesota ExtensionGary Hachfeld, University of Minnesota Extension
The Extension Risk Management Education Grants Program Team
Our Mission
“Helping America’s farmers and ranchers manage risk”
Extension Risk Management Education
Program University of Delaware – Northeast Region
University of Arkansas – Southern Region
University of Nebraska - North Central Region
Washington State University Extension – Western Region
University of Minnesota – Digital Center
USDA, National Institute of Food and Agriculture
Regional Centers North Central Center
University of Nebraska
Northeast Center University of Delaware Cooperative Extension
Southern Center University of Arkansas Division of Agriculture
Western Center Washington State University Extension
Return on Investment
• The return on investment of Extension RME awarded grant dollars are the risk management improvements that agricultural producers make as result of a project.
• The risk management results/outcomes that agricultural producers achieve “are” the measure of success of any project we fund.
Process Focused on Results/Outcomes
• The results based process, including the grant selection, reporting, and management system,
– increases the probability that funded programs will improve the risk management skills of the farmers & ranchers who participate,
– is designed to improve, over time, the capacity and quality of the programs we fund, and
– increases the opportunities to improve the economic viability of the farm and ranch family participants.
Evaluation Best Practices Keep the focus on:
Producer Results/Outcomes
Project Team’s Accountability to Measuring/Verifying Producer Results
Commitment to helping producers reach their risk management goals
Ability to Measure Outcomes
• Improve effectiveness by looking at your project from start to finish in terms of results for participants.
• Demonstrate participant achievement of risk management tools and strategies by posting on the public reporting system (final reports on ExtensionRME.org & the Ag Risk Education Library)
• Move the bar higher for the effectiveness of risk management education
Beginning with the “end” in mind• Begin with the end in mind
– Determine firstfirst what the result/outcome is that the participants will achieve.
– Design the program to lead the participants to those results/outcomes.
• Do not wait till the end of the project to determine what the outcomes might be.
Outcomes- Will Depend Upon:
- Knowledge of your target audience
- Why ag producers will choose to participate
- Action and learning processes (project steps) that help producers achieve long term outcomes
- Program resources including the team, collaborators and expert presenters sufficient to lead participants to the desired outcomes.
Outcomes – A Systems Approach
The Proposal as a working document
i. Application Stage
ii. Program Delivery
iii. Reporting
Outcomes & the Logic Model
Identifying and Articulating Program Outcomes
Participant Outcomes
“Begin with the end in mind” Or
“plan backward, implement forward”
Jon Newkirk, Director, WSU Western Center Beverly Anderson Parsons, WKKF Cluster
Evaluator
An Outcome Defined….
Change in:KnowledgeActions orConditions
Outcomes Defined….
Changes in knowledge…When project participants 1. Learn or become aware of new
fundamental or applied knowledge which could include:a) Methods and techniquesb) Policy Knowledgec) Improved skills such as decision making, life
skills or quality of life choices
Outcomes Defined….Change in Actions…
When project participants 1. Act upon what they have learned such as:
a) Implementing methods and techniquesb) Changing a practicec) Adoption of improved skills such as decision
making, life skills or quality of life choicesd) Direct application of knowledge gained
Outcomes Defined….
Change in Conditions…A socio/economic condition is positively changed because of participant actions1. Information resources that improve
community and/or farming infrastructure2. Technology transfer3. Improved farm income for target audience4. Decrease in farm foreclosures
A Logic Model can help you if you begin with the end in mind….
A Logic Model can help to identify your A Logic Model can help to identify your outcomes (changes in knowledge, outcomes (changes in knowledge,
actions or conditions) and the actions or conditions) and the activities and partners that will activities and partners that will support the accomplishment of support the accomplishment of
outcomes for project participantsoutcomes for project participants
Logic Model as a Tool to Construct Logic Model as a Tool to Construct Realistic OutcomesRealistic Outcomes
FirstFirst – Identify the situation(s) that your project will help to address (farm foreclosures, high price of inputs, lack of record keeping skills etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer Demand).
1.1. OutcomesOutcomes – The identification of changes in knowledge, actions, or conditions for program participants
2.2. OutputsOutputs – (activities – curriculum development, workshops, trainings etc.) that will support and lead to the achievement of participant outcomes
3.3. InputsInputs – Resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc.
Logic Model Beginning With the End in MindLogic Model Beginning With the End in Mind
OUTCOMES
(Identify Measurable
Risk Management
Results)
OUTPUTS
(Lead producer participants to the
desired risk management
results)
INPUTS
(Resources that will strengthen
producer achievement of
risk management results)
INPUTSINPUTSOUTPUTSOUTPUTS
OUTCOMESOUTCOMES
Logic Model Implementing Logic Model Implementing ForwardForward
Producer Results Outcomes Impacts
(Learning – Action – Condition )
A Hierarchy of Actions and Decisions that Range from Improved Understanding to Incorporating Beneficial Risk Management Practices
Producer Results/Outcomes – an example of progressive steps…
↓ UnderstandUnderstand or have improved understanding of the risk or have improved understanding of the risk issue, tool or need to take action.issue, tool or need to take action.
↓ DecideDecide to take an action. Producer commitment to write a to take an action. Producer commitment to write a business or marketing plan, set up a record keeping system, business or marketing plan, set up a record keeping system, conduct a risk analysis, or initiate other steps that will lead to conduct a risk analysis, or initiate other steps that will lead to improved risk management strategiesimproved risk management strategies
↓ DevelopDevelop business tools to strengthen economic capacity, business tools to strengthen economic capacity, such as writing a marketing plan, business plan, farm transfer such as writing a marketing plan, business plan, farm transfer plan, or performing a risk assessment etc.plan, or performing a risk assessment etc.
ImplementImplement or take an action that improves or optimizes their or take an action that improves or optimizes their profitability– implementing a marketing plan, using a record profitability– implementing a marketing plan, using a record keeping system etc.keeping system etc.
In Summary…Results/OutcomesResults/Outcomes
– Measure of success for the participant• Ability to change/incorporate/implement/adopt risk
management practices as a result of participation in your project.
– Achievable and measureable within a reasonable timeframe during and following training.
– Sequential• Identifies changes in condition or practice you seek
for the participant• Specifies the degree of change required for
participant success• Stated in a measurable way knowing methods you
will use to verify changes in condition, practice etc.
Optimizing Producers’ Chances for Success
FirstFirst – Identify the situation(s) that your project will help to address (farm foreclosures, high price of inputs, lack of record keeping skills etc.) and consider if what you are proposing will be well received by the target audience you have identified (Producer Demand).
Then…Identify the Then…Identify the Results/OutcomesResults/Outcomes – changes in – changes in knowledge, actions, or conditions for program participantsknowledge, actions, or conditions for program participants..
Describe what outputsoutputs (activities – curriculum development, workshops, trainings etc.) will support and lead to the achievement of participant outcomes. Create a realistic timetable for your actions (Project Steps)Create a realistic timetable for your actions (Project Steps)
Describe inputsinputs – resources that go into your program in the way of project team, partners, collaborators, who provide expertise, research, money etc.
Logic Model Beginning With the End in MindLogic Model Beginning With the End in Mind
OUTCOMES
(Identify Measurable
Risk Management
Results)
OUTPUTS
(Lead producer participants to the
desired risk management
results)
INPUTS
(Resources that will strengthen
producer achievement of
risk management results)
INPUTSINPUTSOUTPUTSOUTPUTS
OUTCOMESOUTCOMES
Logic Model Implementing Logic Model Implementing ForwardForward
Using the Logic Model to Identify Results/Outcomes & Create a Plan
of Work (Project Steps)
Planning Backwards – Beginning with the End in Mind
Results/Outcomes Activities/Tasks Responsible Entities & Roles
Outcomes Outputs Inputs
Implementing Forward Responsible Entities & Roles Activities/Tasks Results/Outcomes Inputs Outputs Outcomes
Post/Short Term Follow-up
Long Term or 3-6 Month Follow-up
Project Steps
Production Risk ManagementOutputs or Outcomes?• A series of insurance workshops will
be delivered to producers• A crop/livestock insurance
spreadsheet will be developed• Producers will increase purchases of
crop/livestock insurance• Producers will evaluate crop/livestock
insurance decisions
Production Risk ManagementOutputs or Outcomes?• A series of insurance workshops will be
delivered to producers - output• A crop/livestock insurance spreadsheet
will be developed - output• Producers will increase purchases of
crop/livestock insurance – w. outcome• Producers will evaluate crop/livestock
insurance decisions – s. outcome
Marketing Risk ManagementOutputs or Outcomes?• Producers will attend a marketing
workshop• A marketing curriculum will be
written• Producers will implement a
marketing plan• Producers will write a marketing
plan
Marketing Risk ManagementOutputs or Outcomes?• Producers will attend a marketing
workshop – output• A marketing curriculum will be
written - output• Producers will implement a
marketing plan – s. outcome• Producers will write a marketing
plan – w. outcome
Financial Risk ManagementOutputs or Outcomes?
• Producers will calculate financial ratios for case farms/ranches
• Producers will understand and interpret financial ratios
• Producers will evaluate the financial ratios for their farm/ranch
Financial Risk ManagementOutputs or Outcomes?
• Producers will calculate financial ratios for case farms/ranches - output
• Producers will understand and interpret financial ratios – w. outcome
• Producers will evaluate the financial ratios for their farm/ranch – s. outcome
Legal Risk ManagementOutputs or Outcomes?
• Producers will attend a farm program educational meeting
• Producers will successfully analyze farm program sign-up decisions
• Farm program fact sheets and calculators will be produced
Legal Risk ManagementOutputs or Outcomes?
• Producers will attend a farm program educational meeting - output
• Producers will successfully analyze farm program sign-up decisions – s. outcome
• Farm program fact sheets and calculators will be produced - output
Human Resources Risk ManagementOutputs or Outcomes?
• Succession planning materials will be published
• A series of programs will discuss the components of a farm succession plan
• Producers will develop a succession plan
• Producers will implement a succession plan
Human Resources Risk ManagementOutputs or Outcomes?
• Succession planning materials will be published - output
• A series of programs will discuss the components of a farm succession plan - output
• Producers will develop a succession plan – w. outcome
• Producers will implement a succession plan – s. outcome
Exercise – Writing Effective Outcomes
1. Identify the risk situation(s) and the risk topic areas that you want your project to address
2.2. Build a set of Build a set of measurablemeasurable risk management risk management results/outcomesresults/outcomes
Include both short and long term outcomes (results that will happen after the project – within 3 months to a year)
3. Determine if the outcomes are a good fit for the target audience
*Handout
Target AudienceAre Outcomes suitable for audience being
served?Realistic goals that can be achieved within specified timeframe?Linguistically and Culturally Appropriate?
Learning Styles and Environment?
Reliability of InformationScience based content?Appropriate for outcomes being proposed?
How well do you know your target audience?Relationship Capital?
Evaluation MethodologyEvaluation Methodology
Measuring OutcomesMeasuring Outcomes
Evaluation Mechanics Protocols for Administering Evaluations
– Preparing Workshop Facilitators• Evaluation Procedures Audience Preparation
Methodologies– Pre/Post/Follow-Up– Ease of design and implementation
• Methods suitable to audience• Framing questions appropriately to know if outcomes have been
achieved. • Questions designed to measure outcomes should flow/follow from
the hierarchy of results/outcomes.• Qualitative and Quantitative
– Open and close ended questions
Gathering informal data- Producer Stories and Testimonials
Evaluation Mechanics Tabulating the results
Who will collect the data?How will data be collected?When will data be collected?What will be done with the compiled data?
These questions need to be addressed before administering your evaluations and are an integral part of program design!
Evaluation Mechanics
Suitable Question Formats– Multiple Choice – Yes/No – Rating scales– Open ended
Simple for respondent to understandEasier to analyze for project teams
with less evaluation experience
Evaluation Mechanics
Multiple Choice – Yes/No1. Assign a numeric value to each
question response For example code “yes” responses as “1”
and “no” responses as “2”)
2. Use the “count” function in Excel to analyze the responses Pie Charts can be used to graphically depict
responses
Evaluation MechanicsRating Scales
– Allow respondents to indicate level of agreement with a statement or to rate level of importance.
Straightforward and easy to understandEasier to analyze than other question formats (because responses are already in numeric form, it is not necessary to recode them)Shows how respondents feel on average about the question being asked.
Evaluation MechanicsOpen ended questions
– Invite more detail from the respondent– Require a little more effort from
respondentCan identify challenges/barriers to progressProvide a complete story or testimonial
– Analysis may require identifying common themes
Evaluation Mechanics– Add yes’s and no’s– Add up ranking #’s– Provide an average of answers,
rankings, or range– Organize comments into similar patterns – Report results should be structured to
show that participants goals/outcomes are being achieved.
• Numbers first, percentages second
Keep it simple!
NIFA Metrics for Measuring Outcomes
Testing your methods - Will your evaluation tool(s) clearly
demonstrate progress participants are making towards achieving the outcomes?- Changes in Knowledge?- Changes in Attitude?- Changes in Behavior/Approach?- Changes in Economic or Social
Circumstances?
Pre Course Evaluation
Purpose
1.Benchmark info on knowledge levels2.Information on participant background,
past experience, attitudes and needs for program managers and instructors
Combining Pre/Post into Post• “Please rate your knowledge PRIOR
to attending today’s workshop”• “Please rate your knowledge AFTER
attending this workshop”• “Please rate to what extent you have
the knowledge, tools and resources needed to …”
• “I better understand”Likert Scale or Similar Method
Post Course Evaluation
Purpose:
MeasuresKnowledge GainAttitude ChangeActions participants expected to take as a
result of attending course
Post/Short Term Follow-up
Follow-Up Evaluation
Purpose:
To see what actions participants have taken as a result of attending your course and how these actions have led to economic and other benefits.
Long Term or 3-6 Month Follow-up
3 – 6 Month Follow-Up
3 – 6 Month Follow-Up Cont.
Ease of Design
Do evaluation questions flow logically from sequentially stated outcomes?
Asked in a manner to obtain specific information on level of accomplishment or barriers encountered?
Build on progress/commitments made from short term or post evaluations?
Usability of information?
Ease of Design
Avoid “respondent fatigue”?
Use definitive language to obtain accurate responses? Avoid words like “generally” and “usually” – i.e. rather
than “Do you generally follow a business plan?”, “Do you follow a business plan?”
Keep it short and simple!
Exercise – Writing an Effective Evaluation
Post – Short TermFollow-Up – Long Term
Within 3 months to a year
Evaluation Tools
Variety of Methods:– Pre/Post Tests
• Audience Response Systems (Clickers)
– Follow-up Evaluations (3 Months – Year)• Mail or Internet Survey
– Telephone Interview– Face-to-face interviews (in-person survey)– Dot survey– Informal Interview
Evaluation Tools
Considerations:
Time FrameBudgetAudience characteristics
Results/Outcome Based Reporting
• Documents:– Producer Progress towards achieving
goals (short and long term)– Producer Stories and Testimonials– Unexpected results/outcomes
• Provides Access To:– RME Curriculum and resource materials
• Training methodologies• Evaluation Tools
Integrating Results/Outcomes into
Project Reporting Summarizing Outcomes
Project Overview Actual Results/Outcomes Comments/Stories/Examples of Success
Input into other reports
Impactful Reporting• Builds capacity for educators who work directly with farmers
and ranchers through… • Access to risk management curriculum and resources• Access to training methodologies• Access to successful evaluation methods and tools
• Creates transparency• Improves accountability• Documents in a database format risk management
outcomes/impacts for farmers and ranchers.• Available to federal and state agencies, government offices such as the GAO and
OMB, institutions and other funding partners.
• Keeps moving the bar higher for the effectiveness of risk management education.
•
Evaluation Best PracticesMeasuring results/outcomes Measuring results/outcomes
successfully:successfully:
• Choose methods that balance best practices with available resources – Combination of direct and indirect measures
• Direct measures (assignments, action plans etc.) yield stronger evidence of change in knowledge and beneficial practices, but are more time intensive and may take more expertise to evaluate.
• Indirect measures can give instant feedback but may not provide real evidence of knowledge gained.
• Qualitative measures (like phone interviews or focus groups) that may be more resource intensive could be accomplished through a representative sample.
Evaluation Best Practices
Measuring results/outcomes Measuring results/outcomes successfullysuccessfully
• A randomly sampled group, representing approximately 5 percent of participants submitting a post-meeting evaluation, will be surveyed by phone. Participants will be asked whether or not they have completed and implemented their farm/ranch transition and personal estate plan. They will be asked what challenges and obstacles they encountered during the process, how the program assisted them in the process and how useful the resource workbook was to them.
Evaluation Best Practices
Measuring results/outcomes successfully
– Direct Interactions with Producers can increase the chance that participants will implement beneficial risk management practices.
– Good follow-up can lead to collegial relationships that may bring results beyond the initial results/outcomes that were initially projected (Unexpected results).
– Good program design allows for learning and application to take place in linked stages within a timeframe that encourages the implementation/adoption of longer term results/outcomes.
Evaluation Best Practices
• Interactive discussion on best practices – Successes – Pitfalls
*Handout
In Summary
If the “Return on Investment” of grant dollars awarded are the risk management improvements that producers will make…
Then measuring producer results (outcomes) becomes an integral component of program planning and design, not an afterthought!
Thank You!Bob Craven, Director – Digital Center, University of MinnesotaBob Craven, Director – Digital Center, University of [email protected]@umn.eduBrad Lubben, Director – North Central Region, University of Nebraska-LincolnBrad Lubben, Director – North Central Region, University of [email protected]@unl.eduJo Ann Warner , Assistant Director – Western Region, Washington State UniversityJo Ann Warner , Assistant Director – Western Region, Washington State [email protected]@wsu.eduKevin Klair – Digital Center, University of MinnesotaKevin Klair – Digital Center, University of [email protected]@umn.edu
Kynda Curtis, Utah State UniversityKynda Curtis, Utah State [email protected]@usu.eduGary Hachfeld, University of Minnesota ExtensionGary Hachfeld, University of Minnesota [email protected]@umn.edu
Results/Outcome ResourcesThe Rensselaerville Institute:http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266
“Outcome Funding: A New Approach to Targeted Grantmaking”
Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key questions that the funder as investor should ask: What am I buying? What is the probability I will get it? Outcome Funding begins with an abrupt and timely challenge to the standard wisdom of the Request for Proposal process. The authors make the point that very little of the traditional proposal content addresses results, favoring explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical on how to achieve it.
Logic Model Templates:http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html
Developing a Basic Logic Model For Your Program:
http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf I have become convinced that it makes a considerable difference if you do the outcomes before planning the
activities. I definitely advocate doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, “plan backward, implement forward.” Beverly Anderson Parsons, WKKF Cluster Evaluator