using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program...
TRANSCRIPT
![Page 1: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/1.jpg)
Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions
Case studies from consultancy
Wendy Hodge, Principal Consultant
![Page 2: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/2.jpg)
This paper
2
1.Knowledge utilisation 2.The case studies
![Page 3: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/3.jpg)
A very brief potted history of knowledge utilisation
3
• “The results of research are worthless if the are not used” (Last 1989)
![Page 4: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/4.jpg)
Focus on evidence based policy
• Assumes that using knowledge will lead to better policy, programs
Initiatives:• Topic specific centres of excellence with structural
links to government• Systematic reviews of evidence e.g. Cochrane
Collaboration• Clearing Houses e.g. evidence and practice
guidelines and summaries
4
![Page 5: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/5.jpg)
Operating assumptions
• Knowledge is transferred from individual to individual and through organisational structures
• All knowledge is taken up subjectively• Not just users and researchers who influence use of
knowledge• Knowledge is refined and adapted by the user• Use and generation of knowledge are interdependent
and complicated to improve• Not all knowledge is intended to be directly
applicable to policy development or program design
5
![Page 6: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/6.jpg)
Ways evidence is used
6
![Page 7: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/7.jpg)
Instrumental or direct use
When findings/ data are used in specific or direct ways e.g. directly influence a program design or delivery, inform policy directions or professional practice
Page 7
![Page 8: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/8.jpg)
Conceptual use
Involves using research evidence for general enlightenment. Users are exposed to new information, ideas but may not use the information directly
Page 8
![Page 9: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/9.jpg)
Symbolic or strategic use
Using findings/ data to legitimise policy directions or to justify actions taken for other reasons.
Page 9
![Page 10: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/10.jpg)
Predictors of use
“Data are no use if the report on them is too late. They are precious little good if
the relevant audience does not comprehend them”
(Cronbach, 1977)
10
![Page 11: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/11.jpg)
Predictors of use (dissemination model)
• Decision-makers know about the research • Interdependence of policy makers and evaluators e.g.
organisational links exist or joint planning• Good personal relations between key players• The right people - credible source• The right evidence at the right time• The inherent quality of evidence• Whether the evidence conforms to commissioners beliefs and
previous knowledge• Whether data is interpreted in a way that suits the needs of the
user• Tells the story - clear, succinct formats, understood, user friendly
11
![Page 12: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/12.jpg)
“Context matters-values matter – politics matter.” Brewer, 1983
“The interplay between science and policy is commonly neither purely instrumental nor purely political”
Hertin et al 2007
12
![Page 13: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/13.jpg)
Organisational and political predictors of use
• structure, culture and politics of the user organisation including assumptions about a program or policy’s worth and service models
• rewards and incentives for dissemination activity in both the “user” and “researcher” context
• value placed on evaluation or research evidence in the user context
• Other inputs on policy development or program design; lobbying, negotiations
• Boundaries of policy assessment analysis
13
![Page 14: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/14.jpg)
ARTD cases considered
• Evaluation of drink driver education program
• Evaluation of a carers program• Evaluation of drug education program
14
![Page 15: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/15.jpg)
Case 1 – Evaluation of a drink driver education program
Predictor of use The evaluation
Findings known to decision makers High-level interagency senior officer committee + report tabled in parliament
Organisational links Contract+ interagency steering committee for project involved in planning discussion of findings
Good relations None at start but built over 2 years
Credible source Us + academic advisor + guru
The right evidence-the right time Quasi-experimental design + mixed methods. Timed to meet budget cycle.
User friendly report Evidence synthesised and report structured around evaluation questions
Assumptions of program worth Program valued; design based on best evidence
Value placed on evaluation Highly valued; direct client research background
15
![Page 16: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/16.jpg)
Case 2 – Evaluation of a carers program
Predictor of use The evaluation
Findings known to decision makers
Responsible officers commissioned evaluation. Able to drive changes to delivery models.
Organisational links Contract provided formal structure.
Good relations Fostered by regular informal reporting of progress and findings.
Credible source Sought evaluation specialists. Previous knowledge of area and experience in conducting large reviews.
The right evidence-the right time
Extensive consultation with carers and service delivery organisations. Findings delivered in time to inform renewal of 3-year contracts
User friendly report Executive summary identified deficiencies of service model and suggested changes. Report told the story of carers and what respite was needed.
Assumptions of program worth
High given vulnerable nature of the carers...fits with national priorities
Value placed on evaluation Moderate to high...previous bad experience
16
![Page 17: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/17.jpg)
Case 3 – Evaluation of an action enquiry as professional development
Predictor of use The evaluation
Findings known to decision makers Responsible offices commissioned evaluation. Able to drive changes to program structure.
Organisational links Contract provided formal structure and Premiers Panel.
Good relations Fostered by regular informal reporting of progress and findings.
Credible source Long standing clients.
The right evidence-the right time Qualitative methods. Findings reported verbally initially to inform stage 2 planning
User friendly report Answered evaluation questions; placed in context of adult learning principals
Assumptions of program worth New approach, testing
Value placed on evaluation High, value independence
17
![Page 18: Using knowledge utilisation theory to demonstrate how commissioned evaluations can influence program design and funding decisions Case studies from consultancy](https://reader030.vdocuments.site/reader030/viewer/2022032804/56649e495503460f94b3cd1b/html5/thumbnails/18.jpg)
In summary
• Evaluators generate knowledge • Our clients, policy officers and program designers are
users and disseminators of knowledge in their own sphere
• As evaluators, we need to pay attention to predictors of use under our control
• Policy officers transform evidence to meet their needs
• Policy officers could also actively pay attention to predictors of use within the agency context
18