EVALUATING, REVIEWING REFLECTING & REVISING

Ros Tennyson

Some definitions:

Evaluating Measuring

Reviewing Taking stock

Reflecting Understanding

Revising Changing

Essentially we seek to understand / describe the process of acquiring, analyzing, interpreting, adapting, disseminating and (ultimately) transforming experience and knowledge

Some further definitions:

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT

What you need / bring

What you do What you produce

Benefits of your work to those involved

Benefits of your work to wider society

Eg:

• Human resources

• Facilities / equipment

• Contacts / networks

• Specialist knowledge

• Funding

Eg:

• Specific development project(s)

• Comms

• Creating mechanisms

• Capacity bldg.

Eg:

• Physical products

• Events

• Services delivered

• Number of people trained

Eg:

• Changes in levels of skill / confidence / behaviour

• Improved systems or organisational efficiency / capacity

Eg:

• More stable communities

• Increased cross-sector respect / collaboration

• Influence on policy and strategy

In Phase 1:

• Set up parameters, baselines and performance indicators

• Creating review / evaluation systems

• Ensuring commitment (‘buy-in’) to evaluation and review from partners

In Phase 2:

• Collecting project data (outputs and outcomes)

• Tracking performance / Keeping records (decisions, events, changes)

• Maximising partner resource contributions / relationships and engagement

In Phase 3:

• Evaluating project outcomes and impacts

• Assessing the the partnership as an effective mechanism for sustainable development activity

• Reviewing partner relationships to assess and maximise value

In Phase 4:

• Undertaking and disseminating a final evaluation

Or: • Agreeing new parameters and performance

indicators• Revising systems• Creating new review and evaluation

procedures

Evaluating Partnerships:What do we most want toknow?1. That the partnership has

been effective in achieving its aims

2. That the partners have all benefited from their involvement

3. That the partnership approach was / is the best way to do it

We need to understand the bits that are not obvious and that we can’t see

Painting by Magritte

AN EMERGING EVALUATION APPROACH FOR PARTNERSHIPS:

1IMPACT ASSESSMENT

2. PARTNER RELATIONSHIP

REVIEW

3EVALUATING PARTNERING

AS A MECHANISM

To evaluate:

Tangible impacts: – Technical / Physical – Human – FinancialIntended / unintended outcomesValue (of investment)

1. Impact assessment

To understand:

Transaction costs / value added Sustainability of outcomes Strategic influence Systemic impact / reform

Whether a partnership approach was ultimately better than the next best alternative

3. Evaluating partnering as a mechanism

Formula for assessing ‘added value’ of a partnership

AV = (OP + SC) – (RC + NA + EC + OC + FC)

Key:• AV = Added Value of a Partnership

• OP = Outcomes of Partnership • SC = Social Capital• RC = Resources Contributed• NA = Net Benefit of the Next Most Likely Alternative• EC = Environmental Contributions• OC = Opportunity Costs (eg time spent)• FC = Facilitation Costs

To reveal: Value for partner organisations (& other

stakeholders) – expected – unexpected

– potential Degree of effectiveness / efficiency / impact Level of influence (sectoral / strategic)

2. Partner relationship review

This form of ‘evaluation’ links to:

• A review of partnering principles in practice (ie is the partnership equitable, transparent and mutually beneficial?)

• Whether the partnership is achieving individual goals / underlying interests of partner organisations or not

• Exploring whether the partners have made maximum use of the range of resources available

• Whether the partnership could work better – if so, how?

• Whether the partnership could do something quite different – if so, what?

Who are the audiences for a partner review?

Brainstorm…

This is a new ‘science’ and it raises questions about:

Validity?Reliability?Legitimate measurement?Ethical considerations?Added value?

A working hypothesis…

Any truly valid and effective review of a partnership always need to:

• Involve all partners & key stakeholders in design and data collection• Include a genuine feedback loop so that the process truly informs the development of the partnership• Find a good balance between external ‘objectivity’ and internal knowledge / experience of the partnership’s history

Sources of knowledge:

• Your own and others experiences andobservations

• Formal records (eg minutes, proposals, actionplans, agreements)

• Physical evidence• Other relevant materials giving contextual

information (eg newspapers, legislation, local events)

• Theories and hypotheses (to challenge your thinking and understanding)

• Guidelines / manuals• Case studies

Ways of Knowing…

• Sensory (sight, smell, touch, hearing, taste)

• Mental (memory, patterns, shapes)

• Intellectual (concepts, logic, assumptions)

• Intuitive (instinct, feelings)

• Imaginative (vision, empathy)

Capturing knowledge involves…

• Looking for tangible evidence• Empowering and inviting people to

reveal what they know• Active listening / Astute observation• Asking ‘open’ questions• Record keeping (endless note-taking;

keeping a ‘log book’)• Good filing systems (or a good

memory)

Being aware of the ‘filter’ of the reviewer…

• Preconceptions, assumptions and prejudices

• Cultural / sectoral / political ‘lens’• Personal values / belief system• Professional discipline / training• Experiences (good or bad) of partnering

to date• Capacity to make sense of

complex and multi-source data

Making sense of data involves…

• Active engagement and interest• Attention to detail• Willingness to explore contradictions• Sifting and selecting material• Interpretation and clarification• ‘Triangulation’ (confirmation)

of findings• and above all…

Time for quiet

reflection!

Painting by Salvador Dali

Tools and technologies

Tools: Questionnaires SWOT analysis Open questioning Observed activity Tangible evidence Document analysis Other?

Processes: Written responses 1:1 interviews Partner groups Partnership groups Third-party inputs Contextual evidence Other?

“As a reviewer, make no

assumptions

and always

remember to expect the

unexpected”

THE UNEXPECTED ANSWER by Rene Magritte