Different approaches to the evaluation of effects of basic research – Results from a pilot project

Author: Bo Sandberg, Evaluation Unit, Swedish Research Council

Key words: Impact studies, Basic research, evaluation methods

Abstract

Introduction

The Swedish Research Council is the main government agency in Sweden that provides funding for

basic research of the highest scientific quality in all disciplinary domains. The Swedish Research

Council’s also evaluate research and assess its scientific quality and significance.

Historically, basic research has been about the pursuit of knowledge “for its own sake” (Calvert &

Martin 2001), but there is an increased focus on describing the impacts of research funding in

Sweden and other European countries (EMBO 2012, RAND 2006, TEKES 2011). The Swedish

government has urged the Swedish Research Council to put focus on analysing impacts of basic

research (Prop 2008/09:50, 21).

This paper presents findings from a pilot project aiming to develop methods valid to identify and

describe impact of basic research activities.1 The project focused on testing three different evaluation

approaches in order to trace the impact of basic research activities that was the result of a research

program that between 1994 and 2003 aimed at strengthening basic criminological research in

Sweden. The study was made by the Evaluation Unit at the Swedish Research Council.2

1 In the context of this paper, the term basic research refers to research that is meant to increase our scientific

knowledge base. Basic research can be purely theoretical and often with the intent of increasing our

understanding of certain phenomena or behavior but does not necessarily seek to solve or treat these

problems.

2 The Pilot study was conducted by Sten Söderberg, Eva Mineur, Susanna Bylin, Per Janson and Bo Sandberg

and will be published in the autumn of 2012 (Vetenskapsrådet 2012).

This paper presents a pilot study were three different

methodological approaches were used to study impacts of a

basic research program. I will shortly present the program and

the three sub-studies, then discuss what the contribution of the

program was and give recommendations for the study of impact

of basic research policy initiatives.

What is impact?

In evaluation research impact is often described as a change that has occurred due to an

intervention; and it is a change that would not have happened without the intervention

(Ekonomistyrningsverket 2006, Lindgren 2012, Rossi, Lipsey & Freeman 2004). Impact is the

difference between the outcome of an activity and the outcome that would have been without that

activity (Rossi, Lipsey & Freeman 2004).3

Academic impact refers to the impact of research within the academia, for example knowledge

production, citations and the organization of research (Vetenskapsrådet 2012). The study of

research’s relevance for the academia and knowledge production is how the Swedish Research

Council traditionally evaluates the outcome and impact of basic research.

Non-academic impact or societal impact (see for example Davies, Nutley & Walter 2005, EMBO 2012,

Spaapen & Drooge 2011) is about what value research provides to the community, for example

economic or commercial impacts that usually are associated with applied research and measured by

indicators such as new products, services, start-up companies, joint ventures etc. It can also refer to

social and cultural impacts that manifested as a result of applied or basic research influencing

government policy and practices etc. The societal impact of basic research is more complex to

evaluate than academic impact.

In the following, the term impact includes outcome changes both within the academia as well as

outside societal impact. We applied this wide approach in order to identify, categorize and compare

different types of outcomes of the research program. It is important to note that the purpose of our

study has been to try different methods of studying impact, not to systematically assess the program

effect of the specific research program used as a case.

Background

The pilot-study has been a method development project with the purpose to develop and try

different methods that can be used to assess the impacts of basic research. The “Criminology

Program” initiated by the Swedish government in 1992 had been proposed as a program suitable for

a pilot study about the impacts of basic research. This research program financed 34 research

projects to a total amount of 34 MSEK during 1994-2003 and was administrated by Swedish Council

for Research in the Humanities and Social Sciences (HSFR) between 1994 and 2000, and from 2001 by

the Scientific Council for Humanities and Social Sciences within the Swedish Research Council. The

purpose of the initiative was to strengthen Swedish criminological research.4 In three sub-studies we

tested different methodological approaches to the study of impacts of the Criminology Program.

3 In innovation policy evaluation this is often referred to as the additionality of an activity (Buisseret, Cameron

& Georghiou 1995, Georghiou 2002). 4 Criminology was defined its broadest interdisciplinary sense including various academic disciplines involving

crime science research such as for example psychology, sociology, criminology, history, law, social work, ethnology and theology.

Sub-study 1: The program theory perspective

A program theoretical approach was used to compare the logical framework of the initiative with the

implementation and outcomes of the program, as perceived by different stakeholders within the

criminal justice policy and research field. We studied policy documents related to the program and its

origination to map out the means and goals of the research program. Then we conducted interviews

with eleven different stakeholders representing researchers, civil servants, politicians and

governmental officials within the criminal policy and research field at the time of the research

program. Stakeholders were asked about their overall impression of the means and goals of the

program and its impact.

Identifying the means and goals of the research program

A program theory conceptualises the means and goals of an activity (Rossi, Lipsey & Freeman 2004).

In an evaluation the program theory can be analysed to what extent it relates in a reasonable way to

the situation the action intends to improve. Based on the documents studied, we created a simple

framework of the program that we used to structure interviews with the stakeholders (Figure 1,

below).

Figure 1: Overview - program theory of the Criminology Program

In the early 1990s Swedish criminological research was characterized by a fragmented governmental

ministry mandate ship, lack of a central and unifying function, lack of financial resources, an

unbalanced relationship between basic and applied research, lack of researchers, lack of specialized

university departments, etc. (SOU 1992:80). The government proposed that money should be

transferred from Swedish National Council for Crime Prevention to HSFR in order to strengthen and

build up basic criminological research in Sweden (prop. 1992/93:170).

Many factors affect a research field and its development (Sandberg & Faugert 2012) and in our

analysis we chose to include two other events (frame factors) that were implemented as a

consequence of the government proposal (Prop. 1992/93:170): The review of the Swedish National

Council for Crime Prevention, at that point the largest research environment in criminology (SOU

1992:80), and the appointment of new professorships and university positions within criminological

research.

We identified six different goals of the program: One was to better balance funding for university

research and research performed at governmental agencies. In the government report ”Criminal

justice research” (SOU 1992:80) basic research was described as research performed at universities.

Applied research was research activities performed within sectorial agencies. The other goals of the

program was to strengthen existing research environment and to stimulate new research

environments; to stimulate the quality of basic research; to widen the field; to create new networks

and to increase co-operation.

Our interviews with stakeholders focused on the rationale of the different goals of the criminological

program and whether or not the program had moved the research field towards the desired impact

of strengthening basic criminological research in Sweden.

Two main views on why the program had been initiated were presented in the interviews; some

focused on the need to strengthen criminological research in Sweden at the time of the research

program, while about half of the interviewed stakeholders choose to describe the initiative as a

political move made by the government to move basic research activities away from the Swedish

National Council for Crime Prevention in order to gear it towards more applied research that could

serve as an input for criminal policy.

The interviewees meant that the different goals of the program were unclear and contradicting. For

example, some meant that categorisation of criminological research into basic and applied research

was oversimplified. Several stakeholders questioned the assumption that a small research program

financing individual researchers would be able to both strengthen established research environments

and to stimulate the progression of new research environments. A closer study of the distribution of

grants showed that the program foremost had been a support to researchers representing a few well

established research environments.5

Some of the interviewees pointed out that the criminological research field expanded geographically

in Sweden at the end of the Criminology Program but that this development could be explained by

other events, for example the establishment of police training at the universities of Umeå and Växjö

in the early 2000s which led to new investments in police research there.

The majority of the interviewees did not think that the program had significantly increased the

quality or international status of Swedish criminological research. We concluded that the thought

that a small research program that funding individual research projects would be able to fulfil all the

goals that were attached to was not convincing. And that it was not likely that the program in would

strengthen the whole criminological research field, even though it was clear that the program along

with new professorships in criminology was an important contribution to strengthen certain research

environments.

5 18 out of 34 research projects had been conducted by researchers representing two university departments

in psychology and criminology. 16 projects had been distributed among researchers representing 15 different university departments.

Program theory to help understand the interaction between policy, outcomes and impact

Using a program theoretical approach can be an efficient way of understanding and tracing impact by

comparing program implementation with its theoretical framework. It can also help identifying

expected and unexpected outcomes of an initiative. If a research policy is well thought and well

documented, this approach can help explaining how a research initiative fulfil creates certain

outcomes (or not). This approach can be a good companion to quantitative outcome evaluations as it

provides in-depth understanding of the mechanisms behind research policy and how it interacts with

outcomes and possible impacts of initiatives. Results can generate knowledge that can be transferred

into new actions to develop initiatives and improve research policy.

Sub-study 2: Interviews with researchers and research users about individual research projects

In this study we followed a strategic sample of individual research projects funded by the

Criminology Program. An expert committee was asked to pick six projects of the highest research

quality and/or the highest societal relevance from the research program. We interviewed the

principal investigator (PI) of the projects and asked about the impacts of their research project. We

also asked them to name users of their research results outside of the academia. In total, we

interviewed five researchers and seven users of research results. The questions addressed in this

study were if it was possible to trace impact of individual research projects outside of the academia

and how research results were utilized.

Different types of impacts of individual research projects

Through the interviews we were able to map different types of outcomes generated by the research

projects (Table 1, below).

Table 1: Examples of impacts from research projects funded by the Criminology Program.

Impacts within the academic domain Societal impact

Increase in scientific publications Numerous examples of training and courses for

civil servants as a result of research projects

PhD-training and theses Research results used in public debates and in

media

Scientific presentations Research from at least one research project

served as a significant contribution to a criminal

court procedure reform

Strengthened research within a specific research

area or a university department

New research constellations and collaborations

The majority of examples of societal impact were about knowledge transfer through courses and

other types of training for professionals within the criminal justice system. The research projects had

resulted in courses for lawyers, law clerks, police academy students, custom officials, civil servants,

social workers etc.

Reported societal impacts of the research projects varied vastly: Some of the researchers could

provide several examples of agencies and organisations that had used their research results, while a

few said that their main interest was not on the possible impact that the research might have outside

of the academia. Consequently, they could not provide us with concrete information to track

potential non-academic use of their research. All research users interviewed expressed a need to

better bridge the gap between the research community and possible users of research results,

including that the universities need to develop their skills in disseminating research results to outside

of the academia.

Only one interviewee could give a specific example of that his research project had had a direct

impact on policy. This research had made a direct impact upon how courts in Sweden document and

assess witness reports in criminal cases.6

Interviews with researchers and users to identify predicted and unpredicted impacts

Studying individual research projects can be an efficient way to show impact of basic research

programs. Interviewing researchers and those who have been recipients of research results provides

information about research outcomes and their possible impacts and can identify both predicted and

unpredicted outcomes and impacts of basic research. While some researchers may not be able to

provide details about their research’s impact outside of the academia, different stakeholders in

6 This is one of the cases in the third sub-study presented below

society can provide information about possible further impact of basic research. If we had had the

possibility to expand this study to more potential users and other stakeholders, we would most likely

have identified more examples of research impacts.

The study of individual research projects provide for highly credible and convincing input to the study

of the impact of basic research. However, this approach is fairly challenging in that it is a time

consuming method to follow individual research projects. Also, for us to be able to draw more

general conclusions about the impact of the research financed by the Criminology Program, the study

would require a lot more projects than we were able to include in this study.

Sub-study 3: References to research in the policy making process

In this study we analyzed to what extent criminological research was cited or referred to in the policy

process regarding two criminal policy issues. We looked at the use of research references in the

official documentation from the Swedish parliament. We looked at a criminal court reform called “A

more modern trial” (Prop. 2004/05:131) and the creation of the Swedish anti-prostitution law (Prop.

1997/98:55). We wanted to know whether it was possible to trace references to research in the

official documents preceding government bills and if research had any (visible) impact on the political

process.

When studying the policy process of the trial reform, we used a so called down-stream approach

(Vedung 2009, VINNOVA 2007). We started with a specific research project chosen from the second

study (above) that we knew had made a direct impact on the trial reform. In the case of the anti-

prostitution law, we used an up-stream approach (Vedung 2009, VINNOVA 2007). This meant that we

went backwards form the law-change and studied what events and arguments that caused this

change. The anti-prostitution law criminalises the purchase of sexual services, while selling such

services is not illegal. We had reasons to expect to find references to research in the policy process

since this law was a major law change that had been debated during a long period among policy

makers, professionals and others. The law also dealt with a social problem that for a long time has

been a research subject within several disciplines, such as criminology, sociology and social work for

example.

In both cases we studied official background material (official government reports, motions from

members of the parliament, government bills and responses from different organizations on referrals

of government reports). In total, we analysed over 350 public documents in the search of references

to research.

The impact of one research project? - The trial reform

In the trial reform we found that references to the research project identified in the second sub-

study (above) could be found in several different types of public documents throughout the policy

process: The specific research project was mentioned in two out of four governmental reports (Ds

2001:36, SOU 2001:103) and in the government proposal (prop. 2004/2005:131). On the other hand,

this was as far as we could find the only research referenced in this part of the policy process.

In the responses from bodies that had been asked to consider the report (121 in total) we found a

handful of references to academic textbooks on legal principles. One court questioned the results of

the research project that the governmental report had cited, however without making any reference

to other research. One legal department at a university pointed out to the government that (better)

scientific proof about the efficiency of the reform was needed.

When looking at parliamentary motions regarding the reform, we found 84 documents. Research

was mentioned and referenced in three of these, but not in such a way that it was possible for us to

identify what kind of research it was or who had performed it.

In the trial reform, certain aspects of the reform where reinforced by referring to results from

research performed by one research group at a university in Sweden. To conclude, one single

research project made a clear impact on the policy decision making process. This was, however,

basically the only specific research findings that was clearly referred to in over 200 documents

studied.

Research with limited impact on the law making process? – The anti-prostitution law

The official government report that had been commissioned by the government in the process of

revising the prostitution law in Sweden contained more than 100 research citations. The report

suggested criminalising both the selling and buying of sexual services. We also found plenty of

research citations in the responses on the government report; however different research

departments used different kinds of research and draw different conclusions about how to deal with

prostitution. The government concluded that most of the bodies that had been asked to consider the

report had rejected the idea of criminalising the selling and buying of sexual services, and suggested

to criminalise the buying of sexual services. The government did not use any additional research

references to support this proposal, rather they concluded that crime research about violence

against women had been neglected and needed to be further developed (Prop. 1997/98:55).

In parliamentary motions regarding the law change we found research being mentioned in less than

3 % of about 350 documents. However, it was extremely rare that they had such detailed

information that it was possible for us to identify specifically what research they were referring to.

Studies of the policy process as an input to a systematic impact analysis

The use of research citations was extremely rare in both of the criminal policy cases studied.

References to a published or unpublished research source were almost exclusively made in official

government reports or in comments to referrals of such reports. These types of official documents in

the Swedish policy process are the kinds often written by non-politicians. While a single research

project turned out to have a direct impact in one criminal policy area (the trial system), there was no

clear indication of that research results had a direct impact in the other (the anti-prostitution law).

The fact that references are not made to research by policy makers does not necessarily mean that

policy makers do not use research results:

(…) it is now widely argued that evaluation research, and social science research in

general, is more often used in a conceptual rather than an instrumental way, reaching

decision makers in unsystematic and diffuse forms (Sanderson 2000, p.435)

It is, given this, difficult to draw clear conclusions about what to expect and how to assess when

studying the use of direct references to research in the policy making process. Most likely the use of

research results a basis for policy making also varies depending on what policy area we are studying.

Despite this, the conclusion of this sub-study is that studying the impact of research on the policy

making process can be an important step in a more developed impact analysis of basic research.

Studying the policy process in a specific field gives an idea of if the use of research references is

common or rare, clear or vague. While it seems reasonable to assume that a systematic use of

research results that are visible in the policy making process is a sign of high level of trust in research

within this field, it is probably wrong to assume that a limited use of research references necessarily

is a sign of low confidence in research. For this reason, it is clear that this approach to studying the

impact of basic research needs to be complemented with other methods, such as for example

interviews with politicians and other stakeholders in order to get an idea to what extent research has

an impact on their views, standpoints and opinions – and to what extent research has an actual

impact on the policy making process.

Discussion – contribution vs attribution

Contribution analysis (CA) is a relatively new approach to evaluation, and an important aspect of

impact evaluations.7 Contribution refers to if an action has “(…) made a noticeable contribution to an

observed result (…)” (Mayne 2012, p.273), while attribution is a quantitative estimation how much of

an impact is caused by a certain activity. Attribution is an important activity of evaluations within all

social activities as it raises the question of observed outcomes of activities can be directly attributed

to that specific activity (Patton 2012, Sandberg 2011). For agents financing research, it obviously

desired that certain research achievements are attributed to a certain agent, but impact the social

impact of basic research is seldom a direct result of the research. Impact is realized through a

complex chain of events and influences that can be difficult to trace and that can be stretched over a

long period of time (Sandberg & Faugert 2012). It is difficult to predict when in time impact of basic

research might occur, and the time from research to impact might vary largely from one research

field to another (Spaapen & van Drooge 2011).

To ask questions about contribution is important when trying to capture what difference an activity

has made and how it has contributed to a certain outcome. If we look at research impact as a desired

change caused by – in this case – the Criminology Program, an important question is if the

observations we made were unique and would not have occurred anyway without this specific

research initiative. While we have not been able to make any systematic attempt to analyse to what

7 For example, Evaluation has recently devoted a special issue to Contribution Analysis (Volume 18, Number 3,

July 2012).

extent the actual research program contributed to observed impacts; there are some observations

that can serve as input to the question of what difference the research program made:

1. In the program theoretical study of the research program it became clear that changes in the

criminological research during the period of the research program were also greatly affected

by many factors outside of the program’s control. For example, since the criminological

research field was quite small at the time of the Criminology Program (SOU 1992:80)8, it was

easily affected by structural changes such as re-organisations of R&D activities within

governmental criminal justice field agencies. Additionally, there were several other research

funders that also supported this kind of research during the same period and the Criminology

Program was not coordinated with these actions. The research program most likely

influenced the research field; however this contribution might have been modest in

comparison to other events during this period.

2. Our impression is that the six projects chosen for the study of individual projects would have

been financed through the general calls for research grants by the Swedish Research Council

even if the Criminology Program. Consequently, it would be problematic to attribute the high

quality research and its societal impact to the Criminology Program.

Research policy initiatives can make an important contribution to research and research impact, but

it requires rigorous studies of cause and effect to be able to make statements about attribution.

Research having an impact on official policies or public debates, or leading to the implementation of

new medical treatment or a new technology, may not be predictable at the stage when the research

itself is planned, and outcomes of the research may not necessarily occur when and where one might

necessarily expect them to happen. The question of attribution is further complicated by the fact

that the outcomes and impact of research depends on many different pieces of research funded by

diverse research organisations (Vetenskapsrådet 2010).

Conclusions

Finally, I would like to list some straightforward advice when it comes to the study of impact of basic

research. I propose five conditions that regard the evaluability of the impact of research policy

activities:

1. Define what is being assessed to make sure that the evaluation is made with methods that

are appropriate for generating reliable evaluation results (Sandberg & Faugert 2012). The

research field and research environments as well as the research policy initiative should be

8 As an illustrative example, between 1988 and 1997 there 3 individuals received a PhD-degree in Criminology

in Sweden, compared to 27 between 1999 and 2008.

clearly specified so that a representative sample of relevant research interventions can be

made.

2. Analyze at what point in time it is reasonable to expect that impact of the research can be

identified and of course that data to make a convincing analysis is available. How much time

is reasonable between research activities and a thoughtful attempt to identify impact? It is

reasonable to assume that a longer time frame between research and the search for impact

is better (see for example Spaapen & van Drooge 2011). At the same time, it is important to

be aware of that it might be a methodological problem identifying particulars about a

research initiative or project some 10-15 years after it was made (as in our pilot study). If

program activities are poorly documented and if key stakeholders have problems recollecting

details about the activity studied, it might jeopardise the reliability of data and assessment of

impact.

3. Describe and analyse the studied action’s role and function in relation to other relevant

interventions. This to enable that the possibility for an analysis of what the initiative has

contributed to in relation to other factors, and to avoid oversimplification. This also to avoid

making false assumptions about any causal claims between the action and identified

impacts; With a narrow and linear focus on an intervention and outcomes there is a risk that

the impact of contextual factors are ignored (Sanderson 2000, Simons 2004, Toulemonde

1995)

4. Encourage decision-makers and program owners to make an effort to thoroughly document

how their initiatives are thought to lead to certain (desired) outcomes and impacts.

Documentation that describes research policy initiative in detail makes it possible to assess if

the assumptions behind the initiative are reasonable, and detailed program documentation

is crucial since studies of the impact of research might not occur until many years after a

research initiative.

5. Study the impact of basic research with multiple methods used parallel. This to improve the

reliability of the study, and so that results and impacts are not overlooked. It is crucial to

choose data collection methods that can contribute to the likeliness of finding impact,

including unforeseen outcomes and effects of basic research. This could be impact inside of

the academia or societal impact. For instance, researchers can most often at length describe

the impact of their research within their academic field but they might have limited

knowledge about potential users and usage outside of the academia. Interviews with

researchers can be complemented with interviews with users of research results or other

stakeholders and with document studies of the policy making process (at least within policy

fields that can be assumed to be based on a reasonable amount of scientific evidence). It is

necessary to use data and method triangulation to make credible assessments of the impact

of basic research.

References:

Buiseret, T.J, Cameron H.M & Georghiou, L. (1995): What difference does it make? Additionality in

the public support of R&D in large firms. International Journal of Technology Management, Vol. 1,

No. 415/6, 587-600

Calvert, J. & Martin, B.R. (2001): Changing conceptions of basic research? Background Document for

the Workshop on Policy Relevance and Measurement of Basic Research, Oslo 29-30 October 2001.

SPRU, Brighton, United Kingdom.

Davies, H., Nutley, S. & Walter, I. (2005): Assessing the impact of social science research: conceptual,

methodological and practical issues. ESRC Symposium on Assessing Non-Academic Impact of

Research.

Ds 2001:36 Hovrättsprocessen i framtiden. Regeringskansliet, Stockholm, Sweden.

Ekonomistyrningsverket (2006): Effektutvärdering. Att välja upplägg. ESV 2006:8. Swedish National

Financial Management Authority, Stockholm, Sweden.

EMBO (2012): Measuring the societal impact of research. European molecular Biology Association,

Vol. 13, No. 8, 2012

Georghiou, L. (2002): Innovation policy and sustainable development: Can public innovation

incentives make a difference? Paper presented at the Six countries program on innovation. Spring

Conference 20002, 28 February – 1 March, 2002. Brussels, Luxemburg.

Lindgren, L. (2012): Terminologihandbok för utvärdering. Studentlitteratur, Lund, Sweden.

Mayne, J. (2012): Contribution analysis: Coming of age? Evaluation 18(3), 270-280

Patton, M.Q (2012): A utilization-focused approach to contribution analysis. Evaluation 18(3), 364-

377

Prop. 1992/93:170 Forskning för kunskap och framsteg. Regeringskansliet, Stockholm, Sweden.

Prop. 1997/98:55 Kvinnofrid. Regeringskansliet, Stockholm, Sweden.

Prop. 2004/05:131 En modernare rättegång – reformering av processen i allmän domstol.

Regeringskansliet, Stockholm, Sweden.

Prop. 2008/09 Ett lyft för forskning och innovation. Regeringskansliet, Stockholm, Sweden.

RAND (2006): Measuring the benefits from research. Rand Europe, Cambridge, United Kingdom.

Rossi, P.H., Lipsey, M.W & Freeman, H.E. (2004): Evaluation – A Systematic Approach. Sage

Publications, Thousand Oaks, U.S.A.

Sandberg, B. (2011): Alcohol prevention and evaluation in the era of evidence based practice – the

need for a systematic approach to evaluation. Nordic Studies on Alcohol and Drugs¸ Vol. 28, 2011, 3,

235-250.

Sandberg, B. & Faugert, S. (2012): Perspektiv på utvärdering. Studentlitteratur, Lund, Sweden.

Sanderson, I. (2000): Complexity, Evaluation and Evidence-Based Policy. Paper for European

Evaluation Society, October 12-14th, 2000. Lausanne, Switzerland.

Simons, H. (2004): Utilizing Evaluation Evidence to Enhance Professional Practice. Evaluation,

Vol.10(4), 410-429.

SOU 1992:80: Kriminologisk och kriminalpolitisk forskning. Betänkande av tillkallad särskild utredare.

Regeringskansliet, Stockholm, Sweden.

SOU 2001:103 En modernare rättegång. Regeringskansliet, Stockholm, Sweden.

Spaapen, J. & van Drooge, L. (2011): Introducing ’productive interactions’ in social impact

assessment. Research Evaluation, 20(3), September 2011, 211-218.

Tekes (2011): Better Results, more value. A framework for analyzing the societal impact of Research

and Innovation. Tekes, Helsinki, Finland.

Toulemonde, J. (1995): Should evaluation be freed from its causal links? Evaluation and Program

planning. Vol.18, No.2, 179-190.

Vedung, E. (2009): Utvärdering i politik och förvaltning. Studentlitteratur, Lund, Sweden.

Vetenskapsrådet (2010): Evaluation of Impacts of Medical Research. Swedish Research Council

9:2010. Swedish Research Council, Stockholm, Sweden.

Vetenskapsrådet (2012): Att utvärdera effekter av ett grundforskningsprogram – en pilotstudie.

Report in print. Swedish Research Council, Stockholm, Sweden.

VINNOVA (2007): VINNOVAs focus på effekter. En samlad ansats för effektlogikprövning, uppföljning,

utvärdering och effektanalys. VINNOVA VA 2007:14. Stockholm, Sweden.