A theory based assessment of evaluations

intended to inform transitions towards

sustainability

Per Mickwitz, Finnish Environment Institute SYKE

Lena Neij, Maria Johansson, Mats Benner, Sofie

Sandin, Lund University

The 5th European Environmental Evaluators Network

Forum (2016 EEEN Forum), 15.9.2016

Financed by

● The world is not sustainable

● A transition is needed

○ Consumption, production & lifestyles

○ Policies + research & development

● Knowledge is needed to facilitate the

transition

New types of evaluations

2

The starting point

Should we forget everything we know

about evaluation?

3

NO

We aim to build on the theory and

practice of evaluation

4

We need a “lens”

through which we

can assess the

practice of

evaluation from

the perspective of

transitions

I. To construct a framework for assessing

evaluations from a transformative

perspective

II. Assess the evaluations undertaken in the

area of energy efficiency of buildings in

Sweden

III. Develop an evaluation approach for

evaluations enhancing transformative

change

5

Our research strategy

I. To construct a framework for assessing

evaluations from a transformative

perspective

II. Assess the evaluations undertaken in the

area of energy efficiency of buildings in

Sweden

III. Develop an evaluation approach for

evaluations enhancing transformative

change

6

Our research strategy

Evaluation theory (and practice)methods, valuing, use

(context, focus)

Tran

siti

on

th

eo

ry

Po

licy

anal

ysis

Envi

ron

me

nta

l p

sych

olo

gy

Soci

olo

gy o

f sc

ien

ce

Synthesis: assessment framework

Evaluation theory: generations (Guba and

Lincoln 1989), waves (Vedung, 2010) and

roots (Alkin 2013).

8

1. The problem question: What is the fundamental

problem the evaluation aims to address?

2. The general purpose question: For what aims is

the evaluation done?

3. The organization question: Who should do the

evaluation?

4. The principal judgment question: Based on which

value criteria will the intervention be judged?

5. The object question: How is the evaluand

described?

6. The implementation question: What takes place

between decision and final outputs?

9

The practice of evaluation: 11 questions

Vedung 2009

7. The cause question in the implementation? What

might cause bottlenecks and barriers in the

implementation process, up to and including the

output?

8. The results question: What are the outcomes of the

evaluand (inside and outside the target area)?

9. The impact question: To which degree can the

outcomes be attributed to the evaluand?

10. The practice judgment question: What value has the

evaluand when value criteria and standards are used in

practice?

11. The utilization question: Who and how is the

evaluation to be used?

10

Th

eo

ry o

f E

valu

ati

on

The practice of evaluation: 11 questions

Vedung 2009

1. The problem question: What is the fundamental

problem the evaluation aims to address?

2. The general purpose question: For what aims is

the evaluation done?

3. The organization question: Who should do the

evaluation?

4. The principal judgment question: Based on which

value criteria will the intervention be judged?

5. The object question: How is the evaluand

described?

6. The implementation question: What takes place

between decision and final outputs?

11

Th

e c

on

text

of

an

evalu

ati

on

The practice of evaluation: 11 questions

Vedung 2009

Synthesizing key elements

12

Main aspect

The context of the evaluation (A)

The focus of the evaluation (B)

The design, methods and data used to assess impacts (C)

The criteria used for valuing (D)

The approaches to facilitate use (E)

Th

eo

ry o

f

Evalu

ati

on

Co

nte

xt

13

Main aspect Specific questionsThe context of the evaluation (A)

A.1 – A.4

The focus of the evaluation (B)

B.1 – B.3

The design, methods

and data used to assess impacts (C)

C.1 – C.7

The criteria used for valuing (D)

D.1 – D.5

The approaches to facilitate use (E)

E.1 – E.5

Synthesizing key elements

14

The context of the evaluation (A)When is the evaluation undertaken in relation to past and future decisions? (A.1.)What is the main purpose of the evaluation? (A.2.)Who is commissioning the evaluation? (A.3.)

Who is doing the evaluation? (A.4.)

Synthesizing key elements

Is the focus on just intended effect or are side-effects

(intended and un-intended) and rebound effects also

considered? (B.2.)

● The “side-effects evaluation” model (Vedung 1997)

● “the main task of the social sciences … is the task of

analyzing the unintended social repercussions of intentional

human actions.” Popper (2003, 105)

● A particular side-effect: the rebound-effect.

○ The rebound-effect refers to the fact that when energy-

efficiency is improved money is also saved, due to

reduced cost of energy. The saved money is either used

to expand the activity or it is used on something else,

which directly or indirectly uses energy.

15

Example on how theory is used to develop

the framework

Synthesizing key elements from theory

into an assessment framework

16

The approaches to facilitate use (E)Has key stakeholders been identified and involved in the evaluation process? (E.1.)Have there been any specific efforts to engage different groups, including those that are not well organized and powerless? (E.2.)What has been the time frame for the use of the results? (E.3.)

What particular activities have been undertaken to facilitate use? (E.4.)Have there been efforts to promote use beyond “intended use by intended users” by making the process open and transparent or by making the evaluation results/report freely available and easy to obtain? (E.5)

“evaluations should be judged by their utility and actual use;

therefore, evaluators should facilitate the evaluation process

and design any evaluation with careful consideration of how

everything that is done from beginning to end, will affect use.

The focus in utilization-focused evaluation is on intended use

by intended users.” Patton (2008)

Has key stakeholders been identified and involved in the

evaluation process? (E.1.)

What particular activities have been undertaken to facilitate

use? (E.4.)

17

The approaches to facilitate use (E):

Background

• The role of evaluation for the quest for social justice (e.g.

House, Green and Mertens)

• The need to involve stakeholders and especially

unorganized and least advantages ones in the dialogues

and deliberations through which value criteria and

legitimacy is determined.

Have there been any specific efforts to engage different

groups, including those that are not well organized and

powerless? (E.2.)

Have there been efforts to promote use beyond “intended

use by intended users” by making the process open and

transparent or by making the evaluation results/report freely

available and easy to obtain? (E.5)

18

The approaches to facilitate use (E):

Background

● Sofie Sandin and Lena Neij are assessing evaluations of

energy efficiency policies and programs

○ So far 40 evaluations are included in the sample

○ Most of the evaluations have ben commissioned by the

Swedish energy authority

● Maria Johansson is assessing evaluations of efforts that

have attempted to change individual behavior, by analyzing

how behavioral aspects have been considered

● Mats Benner is assessing evaluations of research efforts

as a knowledge base for transitions, by analyzing how

research has been incorporated

Develop an evaluation approach for evaluations enhancing

transformative change

19

The next step: Assess the evaluations

undertaken in the area of energy

efficiency of buildings in Sweden

● The world is not sustainable

● A transition is needed

○ Consumption, production & lifestyles

○ Policies + research & development

● Evaluations built on interdisciplinary

theory integration might produce the

needed knowledge

20

Linking evaluators and users

– are we exploring the

right questions?

Financed by