a theory based assessment of evaluations intended to inform transitions … · 2017. 3. 22. ·...
TRANSCRIPT
A theory based assessment of evaluations
intended to inform transitions towards
sustainability
Per Mickwitz, Finnish Environment Institute SYKE
Lena Neij, Maria Johansson, Mats Benner, Sofie
Sandin, Lund University
The 5th European Environmental Evaluators Network
Forum (2016 EEEN Forum), 15.9.2016
Financed by
● The world is not sustainable
● A transition is needed
○ Consumption, production & lifestyles
○ Policies + research & development
● Knowledge is needed to facilitate the
transition
New types of evaluations
2
The starting point
Should we forget everything we know
about evaluation?
3
NO
We aim to build on the theory and
practice of evaluation
4
We need a “lens”
through which we
can assess the
practice of
evaluation from
the perspective of
transitions
I. To construct a framework for assessing
evaluations from a transformative
perspective
II. Assess the evaluations undertaken in the
area of energy efficiency of buildings in
Sweden
III. Develop an evaluation approach for
evaluations enhancing transformative
change
5
Our research strategy
I. To construct a framework for assessing
evaluations from a transformative
perspective
II. Assess the evaluations undertaken in the
area of energy efficiency of buildings in
Sweden
III. Develop an evaluation approach for
evaluations enhancing transformative
change
6
Our research strategy
Evaluation theory (and practice)methods, valuing, use
(context, focus)
Tran
siti
on
th
eo
ry
Po
licy
anal
ysis
Envi
ron
me
nta
l p
sych
olo
gy
Soci
olo
gy o
f sc
ien
ce
Synthesis: assessment framework
Evaluation theory: generations (Guba and
Lincoln 1989), waves (Vedung, 2010) and
roots (Alkin 2013).
8
1. The problem question: What is the fundamental
problem the evaluation aims to address?
2. The general purpose question: For what aims is
the evaluation done?
3. The organization question: Who should do the
evaluation?
4. The principal judgment question: Based on which
value criteria will the intervention be judged?
5. The object question: How is the evaluand
described?
6. The implementation question: What takes place
between decision and final outputs?
9
The practice of evaluation: 11 questions
Vedung 2009
7. The cause question in the implementation? What
might cause bottlenecks and barriers in the
implementation process, up to and including the
output?
8. The results question: What are the outcomes of the
evaluand (inside and outside the target area)?
9. The impact question: To which degree can the
outcomes be attributed to the evaluand?
10. The practice judgment question: What value has the
evaluand when value criteria and standards are used in
practice?
11. The utilization question: Who and how is the
evaluation to be used?
10
Th
eo
ry o
f E
valu
ati
on
The practice of evaluation: 11 questions
Vedung 2009
1. The problem question: What is the fundamental
problem the evaluation aims to address?
2. The general purpose question: For what aims is
the evaluation done?
3. The organization question: Who should do the
evaluation?
4. The principal judgment question: Based on which
value criteria will the intervention be judged?
5. The object question: How is the evaluand
described?
6. The implementation question: What takes place
between decision and final outputs?
11
Th
e c
on
text
of
an
evalu
ati
on
The practice of evaluation: 11 questions
Vedung 2009
Synthesizing key elements
12
Main aspect
The context of the evaluation (A)
The focus of the evaluation (B)
The design, methods and data used to assess impacts (C)
The criteria used for valuing (D)
The approaches to facilitate use (E)
Th
eo
ry o
f
Evalu
ati
on
Co
nte
xt
13
Main aspect Specific questionsThe context of the evaluation (A)
A.1 – A.4
The focus of the evaluation (B)
B.1 – B.3
The design, methods
and data used to assess impacts (C)
C.1 – C.7
The criteria used for valuing (D)
D.1 – D.5
The approaches to facilitate use (E)
E.1 – E.5
Synthesizing key elements
14
The context of the evaluation (A)When is the evaluation undertaken in relation to past and future decisions? (A.1.)What is the main purpose of the evaluation? (A.2.)Who is commissioning the evaluation? (A.3.)
Who is doing the evaluation? (A.4.)
Synthesizing key elements
Is the focus on just intended effect or are side-effects
(intended and un-intended) and rebound effects also
considered? (B.2.)
● The “side-effects evaluation” model (Vedung 1997)
● “the main task of the social sciences … is the task of
analyzing the unintended social repercussions of intentional
human actions.” Popper (2003, 105)
● A particular side-effect: the rebound-effect.
○ The rebound-effect refers to the fact that when energy-
efficiency is improved money is also saved, due to
reduced cost of energy. The saved money is either used
to expand the activity or it is used on something else,
which directly or indirectly uses energy.
15
Example on how theory is used to develop
the framework
Synthesizing key elements from theory
into an assessment framework
16
The approaches to facilitate use (E)Has key stakeholders been identified and involved in the evaluation process? (E.1.)Have there been any specific efforts to engage different groups, including those that are not well organized and powerless? (E.2.)What has been the time frame for the use of the results? (E.3.)
What particular activities have been undertaken to facilitate use? (E.4.)Have there been efforts to promote use beyond “intended use by intended users” by making the process open and transparent or by making the evaluation results/report freely available and easy to obtain? (E.5)
“evaluations should be judged by their utility and actual use;
therefore, evaluators should facilitate the evaluation process
and design any evaluation with careful consideration of how
everything that is done from beginning to end, will affect use.
The focus in utilization-focused evaluation is on intended use
by intended users.” Patton (2008)
Has key stakeholders been identified and involved in the
evaluation process? (E.1.)
What particular activities have been undertaken to facilitate
use? (E.4.)
17
The approaches to facilitate use (E):
Background
• The role of evaluation for the quest for social justice (e.g.
House, Green and Mertens)
• The need to involve stakeholders and especially
unorganized and least advantages ones in the dialogues
and deliberations through which value criteria and
legitimacy is determined.
Have there been any specific efforts to engage different
groups, including those that are not well organized and
powerless? (E.2.)
Have there been efforts to promote use beyond “intended
use by intended users” by making the process open and
transparent or by making the evaluation results/report freely
available and easy to obtain? (E.5)
18
The approaches to facilitate use (E):
Background
● Sofie Sandin and Lena Neij are assessing evaluations of
energy efficiency policies and programs
○ So far 40 evaluations are included in the sample
○ Most of the evaluations have ben commissioned by the
Swedish energy authority
● Maria Johansson is assessing evaluations of efforts that
have attempted to change individual behavior, by analyzing
how behavioral aspects have been considered
● Mats Benner is assessing evaluations of research efforts
as a knowledge base for transitions, by analyzing how
research has been incorporated
Develop an evaluation approach for evaluations enhancing
transformative change
19
The next step: Assess the evaluations
undertaken in the area of energy
efficiency of buildings in Sweden
● The world is not sustainable
● A transition is needed
○ Consumption, production & lifestyles
○ Policies + research & development
● Evaluations built on interdisciplinary
theory integration might produce the
needed knowledge
20
Linking evaluators and users
– are we exploring the
right questions?
Financed by