if we build a partnership will they come? integrating needs assessment, process evaluation, and...
TRANSCRIPT
![Page 1: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/1.jpg)
If we build a partnership will they come?
Integrating Needs Assessment, Process Evaluation, and Impact Evaluation
Ronald Jester and Robert Wilson, University of Delaware
![Page 2: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/2.jpg)
A Program Evaluation may include
• Process Indicators
or• Impact Indicators
or• A combination of Process and Impact
Indicators
![Page 3: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/3.jpg)
A Process Evaluation Analyzes Program Inputs, such as
•Outreach Activity•Services Delivered•Adherence to Grant
Proposal•Day-by-day Staff Activity•Costs •Materials
![Page 4: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/4.jpg)
An Impact Evaluation Analyzes Program Outputs, such as
Productivity•Farm output•Individual farmer or waterman output•Efficiency•Benefits/Costs
Client Satisfaction with services
Improvements in Quality of Life
![Page 5: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/5.jpg)
AgrAbility evaluations may include both process and impact indicators
![Page 6: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/6.jpg)
The Delaware Maryland Program (DMAP) employs a systems model for program evaluation
This model starts with the target population - -2400 farmers with disabilities
The model tracks the target population through a service delivery system
The system includes a number of filters that regulate the flow of AgrAbility clients
The model includes a number of efficiency measures that identify impediments to efficient client flow
![Page 7: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/7.jpg)
The AgrAbility Process includes 5 steps:
1. Identify farmers with disabilities.
2. Diagnose their problems.
3. Identify the services that will alleviate the problems.
4. Find resources to support for the services.
5. Deliver the services.
![Page 8: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/8.jpg)
The Delaware-Maryland project relies on “partnerships” to deliver services.
![Page 9: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/9.jpg)
The efficiency of the project depends on
1. Locating the target population 2. Engaging the target population 3. Linking the target population with the
partners 4. Routing the clients to the partners in the
network 5. Delivering appropriate services6. Producing specific outcomes (impacts)
as a result of the services
![Page 10: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/10.jpg)
Overall efficiency is gauged by theproportion of the target population
• Located• Diagnosed • Referred to services • Provided services • Helped by services
![Page 11: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/11.jpg)
Areas of Evaluation Process Evaluation Impact Evaluation
Locating Clients Diagnosing Clients Referring Clients to
services Providing Services
Measuring the Impact of Services
![Page 12: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/12.jpg)
The Model (Figure 1) portrays the flow of a target population through a series of filters.
As the target population of 2400 farmers flowsthrough the service delivery system an increasing numberdrop out at each stage.
![Page 13: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/13.jpg)
How to interpret Figure 1 The target group is the estimated 2400 disabled farmers who reside in
Delaware and Maryland. This estimate was generated from a random survey.
A farmer may seek help the AgrAbility program (columns B and C) or a program can reach out to a farmer (columns D,E, and F).
Line 1 shows that 280 of the estimated 2400 disabled farmers were contacted either by AgrAbility or another program.
Line 2 shows that 168 farmers out of the 280 farmers who were contacted were subsequently evaluated.
Line 3 shows that 120 of the farmers who were evaluated were subsequently referred for service.
Line 4 shows that 105 of those who were referred for service subsequently received a service.
Line 5 shows that 90 out of the 105 farmers who received a service subsequently experienced an improvement in their quality of life.
![Page 14: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/14.jpg)
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
![Page 15: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/15.jpg)
Six Process Measures are employed
Each measure indicates the efficiency of a different part of the
system
![Page 16: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/16.jpg)
1. Market Penetration
The percentage of the target population that contacts social service agencies for help.
Include all agencies- AgrAbility and other agencies that provide services to farmers with
disabilities
![Page 17: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/17.jpg)
Market Penetration Measurementis based on Target Population: the estimated 2400 disabled farmers residing in MD and DE
![Page 18: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/18.jpg)
The farmers who were contacted are indicated in line 1 of Figure 1
![Page 19: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/19.jpg)
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
![Page 20: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/20.jpg)
1. Market Penetration(220)/(2400)
The percentage of the target population that contacts social
service agencies for help
![Page 21: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/21.jpg)
1. Market Penetration(B1+ C1)/(2400)=.09
The percentage of the target population that contacts social
service agencies for help
![Page 22: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/22.jpg)
Market Penetration is a crude measure of public service advertising reach.
![Page 23: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/23.jpg)
2. Outreach
(B1 + C1+DI)/TP
% of the target population (TP)
identified and contacted by
a specific program
![Page 24: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/24.jpg)
The target population that was reached is shown in cells B1, C1, and D1 of Figure 1.
![Page 25: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/25.jpg)
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
![Page 26: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/26.jpg)
3. Assessment Efficiency.
(B2+D2)/(B1+D1)
Assessment Efficiency is the percentage of potential clients eligible for assessment that was assessed
The clients who were assessed are show in cells B2 and D2 of Figure 1.
Clients eligible for assessment are shown in cells B1 and D1 of Figure 1
![Page 27: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/27.jpg)
3. Assessment Efficiency. (100 +58)/
(110 +60)=.93
The percentage of potential clients eligible for for
assessment that was assessed
![Page 28: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/28.jpg)
4. Referral Efficiency. (B3+D3)/(B2 with positive screen + D2 with positive
screen)
Referral efficiency is the percentage of clients with a positive screen that was referred for
service.
Clients who were referred for services are shown in cells B3 and D3.
Those with a positive screen are in cells B2 and D2.
![Page 29: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/29.jpg)
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
![Page 30: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/30.jpg)
5. Service Delivery Efficiency. (B4 + D4)/B3 +D3)=
(60+ 45)/(80 + 40)=.88
Service delivery efficiency is the percentage of clients referred for service
that received service.Clients who received a service are show
in cells B4 and D4.Clients who were referred for service are
shown in cells B3 and D3.
![Page 31: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/31.jpg)
The Delaware-Maryland Project employs the SF-36 Quality of Life Instrument as an Impact Measure
![Page 32: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/32.jpg)
Additional impact measures that are specific to farmers with disabilities
and assistive technology will be incorporated in the evaluation
These measures are being developed by a committee drawn
from AgrAbility programs and national program staff
![Page 33: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/33.jpg)
The SF-36 is the most widely used quality of life instrument in health and social services
![Page 34: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/34.jpg)
The SF-36 is calculated for 8 health scales (0-100)1. Physical limitations
2. Limitations in social activities
3. Limitations in work and home roles
4. Bodily pain
5. Psychological distress and wellbeing
6. Limitations because of emotional problems
7. Energy and fatigue
8. General health perceptions
![Page 35: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/35.jpg)
The SF-36 has been validated through a
numerous studies
Several thousand scientific articles employ the SF-36 as a
quality of life measure for measuring program impact
![Page 36: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/36.jpg)
All active Delaware and Maryland clients are currently completing SF-36 interviews
![Page 37: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/37.jpg)
The SF-36 serves two purposes
The SF-36 is used in the initial assessment to develop a service plan.
The SF-36 is also used in a 1-year follow-up evaluation to assess changes in the quality of life subsequent to the initial assessment.
![Page 38: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/38.jpg)
6. Improved quality of Life: (# of Clients with improved quality of life) /
(# of Clients that Received AgrAbility Service)
Clients with improved quality of life are shown in cells B4 and D4. Those who received a
service from AgrAbility are shown in cells B3 and D3.
(B4 + D4)/(B3 +D3)=
(50 + 40)/(60 + 45)=
90/105=
.86
![Page 39: If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University](https://reader031.vdocuments.site/reader031/viewer/2022032200/56649f555503460f94c78c61/html5/thumbnails/39.jpg)
Finally, 2 questions for the audience:
Which of the measures focus on the operation of partnerships in AgrAbility programs?
What other techniques should be used to evaluate partnerships?