Out of the Box: Evaluation

for Neighbourhoods North West

dr brendon Harveyaspects associates

07973 266976

today....

explore evaluation and your context

why is a different view of evaluation needed?

put forward examples from practice of different ways of working with evaluation

encourage dialogue, critical reflection, a space of possibility

..having new eyes?

“the real voyage of discovery lies not in seeking new landscapes, but in having new eyes”

marcel Proust

A reminder...Creative A reminder...Creative ThinkingThinking

One of the main purposes for the use of

creative thinking is to find better ways of

doing things. It would be quite wrong to

assume that creative thinking means only

risk. Creativity also means insight and

new perceptions that at once make sense.

De Bono (1996)

Using Left Brain/Right BrainLeft Brain/Right Brain

Numbers

Sequences

Languages

Detail

Logic

Song Script

Analysis

Maths Formulae

Daydreaming

Spatial Awareness

Colour

Rhythm

Forms and Patterns

Big Picture

Imagination

Visualisation

Musical Appreciation

Duckett and Tatarkowski (2005)

...so what is the link with evaluation?..........

do you recognise this?

• evaluation an onerous chore put off when the project has ended?

• outside evaluators commissioned and then the project is told it has not accomplished its goals after much time and expense?

• just a ritual by project directors to justify to funders that use of resources been successful and now we need additional funding?

performance management

The performance management movement.

Is the current, nearly exclusive emphasis on the OUTCOMES or results of programmes (the performance management movement) leading to flawed social policy and misguided judgements of programmes ?

Is there a commitment to collect information about WHY and HOW those results occurred?

understanding evaluation

what do you understand evaluation to mean in terms of your role?

Use the ‘post-its’ supplied and write on three words that sum up your understanding

evaluating potential?… the most basic determinants of the extent to which one is likely to realize one’s potential are affective and conative (attitudinal, motivational) and not cognitive. Desire, internal motivation, and commitment are more important… than either domain-specific knowledge or knowledge of specific creativity-enhancing techniques… (nickerson in Sternberg, 1999)

what implications has this view for evaluation of individuals?

Why does this matter?

distinguishing between results that can be attributed solely to the unique intervention of these programmes, or be due to variety of influences WITHIN or OUTSIDE of the programme, or is it just CHANCE?

Have we sufficient information that has been researched to say that programme implementation may be influencing programme results? As performance management has grown in importance there has been a tendency to undervalue the critical role of evaluation research.

Great play has been made in recent years of empowering staff, citizens and communities. The message has been of empowerment, involvement, listening and engagement. Where evaluation has been concerned can we truly say that these principles have been upheld in the way we work with others to learn?

Being true to our principles

learning

Learning is paramount, especially when resources are tight. Such learning involves knowledge we can gain from the workings within a programme. but also, as important, the learning that can be transferred between programmes within an area, a region, a continent.

sound economic sense

Wasting valuable resource on a programme without any evaluation, continuing to support a programme that has been inadequately evaluated that seems to work but we don’t know why, or ignoring the knowledge and information locked up in the experiences of service users, practitioners and managers of programmes on the ground, does not make sound economic sense.

Planning and Managing

A Performance Management approach is a planning and management tool - it is there being driven by management accounting practices. THIS IS NOT WRONG in itself....

........but it cannot be our only evaluation process.

what are our projects like?

simple and straightforward?

we can be certain what is going to happen?

there is always agreement on goals and actions to accomplish them?

Take a recent project and discuss above

integrating knowledge and Experience

What point is there to have a solution to a problem if the elements of that solution are scattered in the knowledge and experiences of many participants (or non-participants) with no way to integrate them?

Action Evaluation:

“seeks to generate knowledge that is a useful, valid, descriptive of the work, and informative about how we might change it" (Argyris, 1985)

who is involved?

•at root we believe in high participation.

• viewing all stakeholders as legitimate participants in the evaluation process.

• Action Evaluation, like Action Research, focuses on creating conditions of collaborative inquiry.

• Evaluation is owned by participants

So what happens..?

•data gathering on establishing shared goals - what, why, how?

•Formative: During the project determining across stakeholders if these are being achieved

•summative -The final stage which is the more conventional “so have we achieved” phase of determining the project’s success.

be like Lara Croft• collects a range of seemingly disconnected

items, people and data as she travels, some of which prove indispensable and others surprisingly useless. But we don’t know which is which when she picks them up.

• She also shows us that progress is not necessarily linear – going straight on can lead to a dead end but trying a few side alleys may create unforeseen opportunities.

• Just when you think that the conundrum is unsolvable, you find yourself breaking through to the next level.

• Lara falls and gets back up over and over again, inspiring us to keep trying and keep learning.