lessons from Involving the People you Work With in Evaluation

draft workbook

Please help us develop this workbook by reading this document and using the provided sections to:

give us your comments

tell us about any experiences your organisation has of involving the people it works with in evaluation

NB: we plan to include some of the experiences you tell us about in the final version of the workbook

Please return your completed workbook by Monday 2nd June 2014.

e-mail to:martha@evaluationsupportscotland.org.uk

post to: Martha Lester-Cribb

Evaluation Support Scotland

Thorn House

5 Rose Street

Edinburgh

EH2 2PR

You can download the electronic (Word) version of this document from: www.evaluationsupportscotland.org.uk/resources/255/

March 2014 v7

contents

what this workbook is about3

how you can help3

terminology4

your details5

stages of evaluation5

the spectrum of involvement8

advantages11

principles14

approaches16

case studies23

links to further reading29

anything else to tell us?30

Norwood,2011

what this workbook is about

We want to produce a practical tool which will help organisations to plan why, when and how to involve the people they work with in evaluation.

Support in the Right Direction is a programme of evaluation support for Self Directed Support [SDS] Independent Support Organisations, funded by the Scottish Government. This support is provided by Evaluation Support Scotland [ESS].

As part of this programme, four of the funded projects have been taking part in a learning set to look at how the people they work with can be involved in evaluation. These projects are:

Advocacy Service Aberdeen

Highland SDS Consortium

Carers Trust

Scottish Consortium for Learning Disability

Each of these projects already had considerable experience of involving the people they work with in their projects, and were keen to extend their understanding of how to do this in relation to evaluation.

The initial aim was to share what we have learned with other Independent Support Organisations working with people eligible for SDS; however we believe that the workbook will also be relevant to voluntary sector organisations working in many other fields.

how you can help

We want to test our workbook and refine it. Please give us your comments and your experiences of involving the people you work with in evaluation by completing the blue sections of this document.

You can download the electronic (Word) version from: www.evaluationsupportscotland.org.uk/resources/255/

Please e-mail your completed workbook to martha@evaluationsupportscotland.org.uk by Monday 2nd June 2014.

The learning set will then take all your comments and experiences into account and produce a final version of this workbook with illustrative examples by November 2014. NB: we plan to include some of the experiences you tell us about in the final version of the workbook. Please make it clear if you want us to keep your experiences anonymous.

terminology

By People you Work With we mean your clients or service users. For Independent Support Organisations (the original intended audience for this workbook), this will probably mean people who are eligible for Self Directed Support and/or those who support / care for them. As another example, for a youth club it would be the young people you work with.

our experience

The learning set discussed the best terminology to use. One participant commented that she doesnt like the phrase service users (and particularly users) as it doesnt reflect the individuality and humanity of the people involved. One other possibility suggested was clients but this too isnt ideal. People you work with is a better phrase and is the one we are using in the meantime but it can be clumsy in use.

your experience

Please put an X beside the term(s) your project currently uses:

Clients

Service Users

People we Work With

Other (please specify)

your thoughts

Please put an X beside the term you think we should use in the final version of this workbook:

Clients

Service Users

People we Work With

Other (please specify)

your details

name:

organisation:

project (if applicable):

who are the people your project / organisation works with?

e-mail address:

stages of evaluation

Evaluation Support Scotland bases its work on a 4 stage evaluation pathway. You can read more and find free resources on the ESS website.

setting your outcomes

what is the difference you are trying to make for the people you work with?

collecting information

what kind of evidence do you need to help you find out if you are achieving your outcomes?

how will you gather it?

analysing and reporting

what does the evidence you have gathered tell you?

who do you need to tell about it and how?

learning from (and acting on) your findings

how are you going to use what you have learned about your work?

The Evaluation Pathway

our experience

In ESS experience, voluntary organisations predominantly involve the people they work with in evaluation at Stage 2: Collecting Information. This is most likely to be in terms of seeking feedback (rather than involving the people they work with in collecting the information or deciding to do it). For example, they ask the people they work with to complete questionnaires, take part in focus groups or participate in interviews.

It may well be that this is usually the most appropriate way for your organisation to involve the people you work with in evaluation, however it can also be worth considering involving people at other stages - when organisational and personal capacity allow.

Involving People at Stage 1 (setting outcomes)

Advocacy Service Aberdeens experience

In the past we have always approached this task [evaluation] by looking at how we can measure the outcomes weve identified in the funding application. It has been interesting to start the process of evaluation by looking at what our service users consider to be the most important things to evaluate. While they do consider outcomes significant and do want to measure these, they are more interested in exploring the relationship between advocacy worker and service user as their questions are more about trust and empathy.

read the full case study from Advocacy Service Aberdeen on p23

your experience

How does your project involve the people it works with in evaluation?

Evaluation Stage

Are the people you work with currently involved?

Please describe how the people you work with are involved

setting outcomes

collecting information

analysing and reporting

learning from (and acting on) your findings

your thoughts

We think the people voluntary organisations work with are mostly involved at Stage 2: Collecting Information. What do you think?

the spectrum of involvement

User-run evaluation

Asking users for feedback

our experience

There is a range of different degrees to which organisations might involve people they work with in evaluation.

ask people you work with for comments

ensure people involved are a representative sample

develop user forums which contribute regularly to evaluation

provide training / additional support to those involved

invite people to contribute their individual stories

involve people you work with in planning part of the process

involve people you work with in deciding the format of materials

involve people in doing some of the evaluation work

people you work with control the evaluation process

greater amount of commitment required lesser

This spectrum corresponds with the degree of commitment required from both the organisation and the people they work with.

The degree to which one project might involve the people they work with in evaluation might not be appropriate for another and this may vary over time.

Equally, a project might choose to involve the people they work with to different degrees at each stage of the evaluation pathway. For example, staff might ask the people they work with to comment on the outcomes they have set (ie at the top of the spectrum for stage 1: setting outcomes), but to be in control of collecting information (ie at the bottom of the spectrum for stage 2: collecting information).

your experience

In what ways does your organisation involve the people it works with in different aspects of evaluation?

Point on the Spectrum

Do you do this?

In which stage/s of evaluation?

Is it a priority for the future?

ask people you work with for comments

partially / fully

yes / no

invite people to contribute their individual stories

partially / fully

yes / no

ensure people involved are a representative sample

partially / fully

yes / no

develop user forums which contribute regularly to evaluation

partially / fully

yes / no

provide training / additional support to those involved

partially / fully

yes / no

involve people you work with in planning part of the process

partially / fully

yes / no

involve people you work with in deciding the