methodologies and approaches to evaluation in third sector organisations / jean ellis
DESCRIPTION
TRANSCRIPT
Evaluation conference, Barcelona, 3 November 20111
Methodologies and approaches to
evaluation in the third sector
International conference on evaluation, Barcelona
3 November 2011
Dr Jean Ellis
Charities Evaluation Services
Evaluation conference, Barcelona, 3 November 20112
The policy waves
• Early 1990s – more business-like sector and efficiency focus
• Mid 1990s – focus on results and effectiveness
• From 2004 – emphasis on performance and value for money
• Current situation – emphasis on savings and social value
Evaluation conference, Barcelona, 3 November 20113
MonitoringThe routine, systematic collection of information for the purpose of checking progress against project plans EvaluationUsing monitoring and other information to make judgements and give value
Some definitions
Evaluation conference, Barcelona, 3 November 20114
Evaluation
Evaluation aims to answer agreed questions and to make a judgement against specific criteria. Like other research, for a good evaluation, data must be collected and analysed systematically, and its interpretation considered carefully.
CES (2009) Practical Monitoring and Evaluation
Evaluation conference, Barcelona, 3 November 20115
The first policy wave
Early 1990s – drive for a more business-like sector and efficiency focus
Evaluation conference, Barcelona, 3 November 20116
The purposes of evaluation
Evaluation for accountability – to demonstrate achievement and compliance
Evaluation for learning – to share findings, improve, replicate and innovate
Evaluation conference, Barcelona, 3 November 20117
Monitoring and evaluation:
• is good for you
• is good for users
• meets external demands
• can make your organisation better
Why do it?
Evaluation conference, Barcelona, 3 November 20118
Steps to self-evaluation
1- Clarify aims and outcomes, objectives and outputs
2- Set performance indicators; choose tools
5-Use findings
3- Collect information
4- Analyse and interpret
Evaluation conference, Barcelona, 3 November 20119
CES planning triangle © Overall
aim
ObjectivesThis is what we do
Specific aimsThis is why we do it
Outputs
Outcomes
Evaluation conference, Barcelona, 3 November 201110
The Family Centre
To provide opportunities
for play, drama and
dancing
Toprovide
workshops, information
and advice on parenting
To organise outings for children and their parents
Toimprove
the lives of the children who use the centre
To increase children’s
self-esteem
To enable parents to give each other practical support
Objectives
Specific aims
Overall aim
To run a drop-in centre with a
play area
To improve the
parenting knowledge and
skills of the parents who
use the centre
Evaluation conference, Barcelona, 3 November 201111
Second policy wave
• Mid-1990s – focus on results and effectiveness
Evaluation conference, Barcelona, 3 November 201112
More definitions
Outputs are an organisation’s detailed activities, services and products – what you do
Outcomes are all the changes, benefits, learning or other effects that happen as a result of an organisation’s activities – what you change
Evaluation conference, Barcelona, 3 November 201113
Aims, outcomes and indicators
Aims – the broad changes you plan to achieve
Outcomes – the specific changes for individuals, families, communities, organisations and policy
Indicators – well-defined and measurable information that will show change
Evaluation conference, Barcelona, 3 November 201114
Families engage with befriending service
Greater awareness within the family
of later life conditions
Families feel supported, less isolated
Increased communication and coping skills
Improved family relationships and dynamics
Intermediate outcomes
There are often smaller changes that need to happen before the final, desired outcome can be reached.
These steps along the way are known as intermediate outcomes.
Evaluation conference, Barcelona, 3 November 201115
The Outcomes Star
© Triangle Consulting 2011
Evaluation conference, Barcelona, 3 November 201116
Underpinned by a model of change
© Triangle Consulting 2011
Evaluation conference, Barcelona, 3 November 201117
Third policy wave
• From 2004 – emphasis on performance and value for money
Evaluation conference, Barcelona, 3 November 201118
Drowning in data
• Too much time collecting and reporting information
• Data not always analysed or used• Costly in time and resources
But monitoring and evaluation IT systems could help greatly.
Evaluation conference, Barcelona, 3 November 201119
Current policy wave
The current situation – emphasis on savings to the public purse and social value.
Evaluation conference, Barcelona, 3 November 201120
Using Social Return on Investment
Fab Pad supports young homeless people to
sustain new tenancies. The SROI evaluation
carried out on Fab Pad revealed that for every £1
invested by the government in support, £8.38 of
social return was derived in reduced health care
costs, reduced welfare benefits expenditure and
reduced costs of repeat homelessness.
Evaluation conference, Barcelona, 3 November 201121
Evaluation for learning
• What worked?
• What parts worked?
• Who did they work for?
• In what contexts did they work?
• What else happened?
• Were there any negative consequences?
• Did the results last?
Evaluation conference, Barcelona, 3 November 201122
Accountability or Learning?
• Closed enquiry
• Proving success
• Showing cause and effect
• Used for reporting and marketing
• Open enquiry
• Valuing failure as well
• Understanding context
• Used for improvement and change.
Evaluation conference, Barcelona, 3 November 201123
Theory of change
It makes clear
all the
assumptions
about how
change will
happen.
Activities
Short-term change
Steps along the way
Longer-term change
Evaluation conference, Barcelona, 3 November 201124
Challenges and opportunities
Managing data Getting
skilled
Finding meaning
Method
s
Innovation
Using findings
New technolog
y
Evaluation conference, Barcelona, 3 November 201125
A winning charity
The Leeds Survivor Led Crisis Service:
We are passionate about monitoring and evaluation as we believe it is the absolute right of our visitors and callers to know that we scrutinise our work in order to ensure that we are providing an excellent service which meets their needs.
Evaluation conference, Barcelona, 3 November 201126
CES contacts:
www.ces-vol.org.uk020 7713 5722
Jean Ellis: [email protected]
02070789386