cta scotland conference prove it! effective evaluation for the community transport sector jonathan...
TRANSCRIPT
![Page 1: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/1.jpg)
CTA Scotland ConferenceProve It! Effective Evaluation for the Community Transport Sector
Jonathan CoburnAssociate Director, EKOS
![Page 2: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/2.jpg)
About the session
• An introduction to evaluation theory and practice• A wide-ranging overview
• Understanding what evaluation is and is not
• Giving an overview of the evaluation process
• Looking at some of the key concepts and terminology
• Exploring the main approaches and methods
• Some practical tips for evaluation
• ... all in the next 45 minutes!
![Page 3: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/3.jpg)
Understanding Evaluation
![Page 4: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/4.jpg)
What is evaluation?• Evaluation is a part of everyday life• It is about making judgements (criteria vs values)• It is comparative by nature• It is about using evidence• It is about cause and effect• It is about drawing on different perspectives• It is about critical reflection• It is about learning (and doing!)
![Page 5: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/5.jpg)
Whatever it is, it’s becoming more prominent in all of our work ...
Evaluation in an era of accountability
•A period of financial austerity•Increasing competition for public funds•Tougher choices for funders/investors•A time of reckoning for all
= must demonstrate efficient and effective service delivery and prove your worth
![Page 6: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/6.jpg)
Evaluation can help you to ...• Make your work more effective and efficient • Anticipate problems and deal with them early on• Identify gaps, improvements, and new directions• Celebrate progress and achievement• Motivate staff and volunteers• Develop your organisational memory• Tell the story more clearly or convincingly to others• Make the case for new/continuing resources
![Page 7: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/7.jpg)
Why then is evaluation treated with such dread? Some myths
•Its a negative experience
•Its only for funders
•Its just dwelling on the past
•Its ‘damn lies and statistics’
•Its about making your case
•Its a wasteful distraction
•Its something done to you
![Page 8: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/8.jpg)
The reality is, evaluation can either be ‘carrot or stick’
A ‘stick’ used to beat underperforming organisations with
A ‘carrot’ used to entice learning organisations committed to change
![Page 9: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/9.jpg)
Two main roles for evaluation• Formative (or process) evaluation
• Takes place during the life of the project• Provides feedback on process and outputs• Examines progress towards your objectives
• Summative (or impact) evaluation• Takes place at the end of the project• Makes a judgement about the success of the work• Typically looks at outcomes and impacts
• .... but the distinction is not always as rigid as the above suggests
![Page 10: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/10.jpg)
Evaluation is best viewed as part of a process
![Page 11: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/11.jpg)
The Basics of the Evaluation Framework
![Page 12: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/12.jpg)
Good evaluation depends on setting a clear direction
Mission (or overall aim). Why an organisation or project exists. The broad effect it wants to have. A summary of the difference it wants to make.
Specific aims. Statements about the differences that an organisation hopes to bring about, typically for its users. There will usually be several aims.
(SMART) Objectives. The areas of activity or practical steps that you will take to accomplish its aims.
![Page 13: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/13.jpg)
Linking what you expect with what happens ... an evaluation framework
Mission
Specific aims This is why we do it
ObjectivesThis is what we do
Outcomes. The changes, benefits, learning or other effects that result from what the organisation makes, offers, or provides
Outputs. The things that result from an organisation’s activities that are intended to achieve change (outcomes).
![Page 14: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/14.jpg)
Setting indicators ... going back to our basic evaluation framework ...
Mission
Specific aims This is why we do it
ObjectivesThis is what we do
Outcomes
Outputs
OutcomeIndicators
OutputIndicators
![Page 15: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/15.jpg)
Using indicators• The different types of indicator are often confused
• Output indicators – assess the things that you do or accomplish and measure progress against objectives
• Outcome indicators – assess the changes, learning, or other effects and measure progress against aims
• 3 easy steps to developing your indicators• Ask yourself what output/outcome would look like if
you achieved it – give potential indicators
• Narrow these down to 2-3 key indicators
• Check appropriateness with others
![Page 16: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/16.jpg)
Your output indicators
• Output indicators may be set for the following• Quantity – the things you deliver• Take up – the people/orgs that use them• Access – your ability to reach certain target groups• Quality – the extent to which standards are reached• Satisfaction – the ability to meet service user needs
• When selecting your output indicators• Be realistic about what you can measure
• Choose the most appropriate indicators
![Page 17: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/17.jpg)
Your outcome indicators
• How are you actually bringing about change?• Outcomes are usually expressed as: the extent of ...; the
level of ...; the degree of ...; the ability to ...; etc.• The challenge is to put measurable indicators to this ...
• Positive or negative• Quantitative(countable) or qualitative (views)• Intended or unintended• Hard (tangible) or soft (pointers/proxies)
• When selecting your outcome indicators must prioritise only significant changes and make them simple
![Page 18: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/18.jpg)
Everything is now about outcomes
• An increasing move in the public sector towards setting high level outcomes to drive performance
• These outcomes are now included in every Council’s strategy (e.g. SOAs) and the plans other public services
• Increasingly organisations are being asked how they fit and what they contribute to local area outcomes
• Funders and service commissioners are increasingly talking about outcome-based funding and contracts
• They are increasingly interested in the long-term outcome or impact of your work
![Page 19: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/19.jpg)
A growing number of frameworks are focused on outcome/impact
• There are many new outcome/impact measurement frameworks that are now coming to the fore
• ... some are holistic e.g.• Social Accounting and Audit• Social Return on Investment
• ... some are specific e.g.• Volunteering Impact Assessment• LM3 – Local Multipliers 3• Prove It
![Page 20: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/20.jpg)
SROI is the latest game in town
• Builds on established tools – CBA, Social Accounting, ROI• Outcome-based – all types of outcomes• About understanding and telling the story of change• A structured and systematic process – an Impact Map• Based on the perspective of stakeholders• Seeks to value the things that matter• Attempts to measure the unmeasurable - proxies• To be used with caution – not the only game in town
![Page 21: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/21.jpg)
The Evaluation Process and Tools
![Page 22: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/22.jpg)
The evaluation process
Manage the Evaluation
•Team•Timescale•Budget
1. Engage stakeholders
2. Establish focus 2. Collect
data 3. Analyse/ interpret 4. Use
findings
•Establish a Framework
•Determine use/users
•Define scope and purpose
•Select indicators
•Determine design
•Identify sources
•Select method(s)
•Determine sample
•Design instruments
•Pilot instruments
•Conduct fieldwork
•Process data•Analyse data•Interpret data•Develop main findings
•Explore implications
•Produce report•Share findings•Use in decision making
•Determine next steps
![Page 23: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/23.jpg)
Methods of data collection• There are many ways of collecting required data,
typically based on social and market research methods• The most commonly used techniques are
• Monitoring records• Observation• Survey questionnaires• Interviews (face-to-face and by telephone)• Focus groups
• Other participatory methods often provide a creative, visual , and rewarding alternative
![Page 24: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/24.jpg)
Monitoring data (records)•Existing sources of data or documentation routinely collected
•Examples include client database, timesheets, satisfaction forms, etc.
•Best if can provide a historical series of data otherwise unavailable
•Gives a first-level indication of progress and performance
•Often inflexible, incomplete, or inconsistently collected
•Usually a useful starting point
![Page 25: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/25.jpg)
Observation•A holistic method providing a vivid picture of the project
•Complex process, with different forms of participant observation
•In this case the evaluator is the research instrument (reliability)
•Can be more or less structured depending on the purpose
•Good at gaining insights into group interactions and behaviours
•Often overlooked and therefore underused
![Page 26: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/26.jpg)
Survey questionnaires •The most commonly used method in evaluation research
•Can be flexibly administered on-site, by post, by email, online, etc.
•Able to compile factual information across many respondents
•Can combine open and closed questions to good effect
•Less useful in exploring complex issues where views are important
•Questionnaires are not a straightforward option
![Page 27: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/27.jpg)
Interviews (face-to-face or telephone)•Provides opportunity for personal, detailed interaction with interviewee
•Can be more or less structured depending on needs
•Used typically with complex issues and where in-depth views required
•Either telephone or face-to-face can be used depending on needs
•The success of interviews depends on the skills of the interviewer
•There will be differing resource implications
![Page 28: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/28.jpg)
Focus groups•Collect data through group interaction on an agreed topic
•A tried and tested market research tool for gauging audience reaction
•Often used in combination with other methods
•Provides an opportunity to get a collective view and develop ideas
•The success of focus groups depends on good facilitation skills
•Choices in terms of focus, participation, and venue
![Page 29: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/29.jpg)
Other creative and visual methods•A set of techniques often referred to as participatory appraisal
•Designed to work with groups of varying abilities and backgrounds
•Examples include graffiti walls, time lines, body maps, H-Form, photography, video booth, mapping networks, arts and drama, etc.
•Inclusive, participative and democratic approach
•Flexible and appropriate in many group settings
![Page 30: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/30.jpg)
Choose the right tools ... and use them well
•The choice of tool(s) depends•On the questions to be answered•On the time available•On the resources available•On the technical skills of the team•On the research participants•On the intended audience
•Make sure that the mix of tools you use are appropriate
•Make sure that you have the skills, resources, and practice
![Page 31: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/31.jpg)
Carrying Out and Using the Evaluation
![Page 32: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/32.jpg)
Decide how much of the evaluation work to take on
yourself
Don’t underestimate the added workload
![Page 33: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/33.jpg)
In-house vs external evaluation• Evaluation always works best when you build it into your
everyday activities (= self-evaluation) but there are choices
• In-house evaluation• Knowledge of the organisation• Organisational learning• Commitment to implement findings
• External evaluation• Specialist skills• Independence• Fresh perspective• Wider insights
• In practice, your use of external evaluators is likely to be determined by resources rather than preference
![Page 34: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/34.jpg)
Make time for analysis•Summarise and organise your data in a logical, ordered way
•Analyse the info as you go along, don’t wait until the report is due
•Break the analysis down into manageable chunks
•Approach the analysis of quantitative, qualitative, and visual data in very different ways
•A systematic and rigorous process is required throughout
![Page 35: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/35.jpg)
Interpretation ... focus on the questions that really matter
• Structure your thinking around the key questions• ... the things that you really should know (learning)
• What happened and why?• What has worked well and not so well?• Did we achieve what we set out to do?• What can we learn from this?
• ... the questions others will want answered (accountability)• Was the project implemented as planned?• Has the money been spent in the right way?• Did the project achieve its aims and objectives?• What difference and contribution has it made?• What proof is there of this?
• Your interpretation must be based on the facts
![Page 36: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/36.jpg)
Make time to produce a compelling report ... some tips
Be well prepared • Collate the inputs, know the audience,
get a structure, leave time
Keep it simple• Focus on essential info only, tell a clear
story, and use headings, bullets, etc.
Make the meaning clear• Present findings in context, comment
on reliability, provide clear messages
Make it user friendly• Plain English, colour, graphics, charts,
case studies, quotes
![Page 37: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/37.jpg)
Sharing the findings in other ways• There will be different audiences interested in the
evidence, not all of whom want to read a weighty report • Think creatively about ways of sharing the findings
• Summary report for circulation• Annual report• ‘Open day’ type events• Conferences• Feedback meetings/presentations• Newsletters• Websites• DVD• Radio interviews• Correspondence to key stakeholders
• Remember to adapt your approach and communication channel to each audience
![Page 38: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/38.jpg)
Making use of the evidence•Share your findings to keep or get important decision-makers on board
•Begin dialogue with service users – ensure that their voice is important
•Use your findings to inform service design
•Don’t be afraid to make brave decisions where the evidence dictates
•Use the evidence as justification for funding applications or tenders
•Use the evaluation where you can to raise your profile
![Page 39: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/39.jpg)
Embedding self-evaluation and maintaining momentum
• Makes sure that everyone understands and buys in• There must be commitment from the top• Someone must lead and take responsibility• It works best when built into activity, meetings, job roles• Ongoing communication to update and refresh • Use any early evidence to achieve some quick wins• Ensure that you build in the resources to do the job
![Page 40: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/40.jpg)
Take the first steps today to improve your evaluation practice
![Page 41: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/41.jpg)
Some resources to help you along the way
Evaluation Support Scotlandwww.evaluationsupportscotland.org.uk
Charities Evaluation Servicewww.ces-vol.org.uk
Prove and Improvewww.proveandimprove.org
Monitoring & Evaluation Resource Guidehttp://www.ces-vol.org.uk/downloads/meresourceguidecompletetext-574-582.pdf
![Page 42: CTA Scotland Conference Prove It! Effective Evaluation for the Community Transport Sector Jonathan Coburn Associate Director, EKOS](https://reader036.vdocuments.site/reader036/viewer/2022062423/56649ea35503460f94ba70fc/html5/thumbnails/42.jpg)
Thank you!