Elizabeth Gray-KingMonitoring Soft Outcomes

1. Introductions2. Expectations of this session, using the Whiteboard3. Knowing the differences – Outputs and Outcomes4. Why Monitor?5. Finding outcomes and creating indicators6. Monitoring Tools7. Resources8. Mopping up Expectations

Monitoring Outcomes Webinar

There are two sorts of outcomes…

› Hard outcomes– The things which can be counted, almost completely

objective – the statistical kind of stuff

› Soft outcomes– The things which are difficult to count, but which

demonstrate growth – the touchy/feely kind of stuff

Hard outcomes

› courses booked

› courses completed

› qualifications earned

› progression made

› Any statistical kind of evidence able to be demonstrated by clear records

Soft Outcomes

› Interpersonal skills – social skills and coping with authority

› Organisational skills – personal organisation and the ability to order and prioritise

› Analytical skills– the ability to exercise judgement, managing time or problem

solving

› Personal skills – insight, motivation, confidence, reliability and health awareness

» (Dewson et. al. 2000a).

The fundamental question:How do you know you’ve made a difference?

Outcome

confident people

Output

3 Drop in Sessions

Outcomes and Outputs example….

Services Outcome Output

Dads n’kids Saturday 10-12

More confident fathers Stronger family relationships Less stressed mothers

10 fathers and 15 children at each session from a range of 30 families each Saturday in term.

Thursday Late Thursday nights theatre workshop

More confident young people

Increased courage to seek specific support on a range of issues

Maximum 25 young people at each session from a wider group of 50; 13 sessions each term

XYZ Information Service Provision of information on welfare benefits and advocacy

More healthy older people Reduced fear in accessing

information Increased confidence in self

management Increased visibility of

appropriate information

1000 leaflets distributed 2 Your Rights weeks in each

year 30 home visits each quarter 40 volunteers trained Advocacy information produced

for 100 leaflets and 2 web sites

Why monitor at all?

› Evidence – that activities were designed effectively– that learners needs were met– for fundraising– for regulatory bodies– for funders

› Ideas and evidence to – target new learners– market effectively– design future learning– design future monitoring tools

Where will we find our outcomes to measure?

› Business Plan, proposals and tenders (the difference intended to be made)

› Vision or Mission statement of organisation or project

› Publicity about the project/course/services – what are the ‘selling points?’ and therefore the outcomes on offer?

How do we translate outcomes to measures?

› Consider the features of a successful outcome…

› For instance, Confident people can be spotted because they:– Talk with other people easily– Listen actively– Ask for help clearly– Informally offer help and guidance to others– Etc (quickthink with others what success LOOKS like)

› Chart those descriptions to create– success indicators [what success looks like]– success measures [how you will count/observe/collect information]

› Often these are called performance indicators

success indicators

What can be seen to indicate – to show – that positive changes have been made?

› Run a Quick think or a participation exercise (or a simple mind map)

› What would success LOOK like? (more relaxed expressions, engagement with others, clear preparation)

› What would success SOUND like? (clear answers, effective questions, effective listening, compliments rather than complaints)

› Each kind of course or intervention will have similar indicators and a range of different ones. Each one has a comparative and/or short accurate description (more, less, clear, fewer, more frequent, etc)

success measures

how you will count/observe/collect information?

› Agree the tools to collect the success indicators

› Agree what could be set in place to see the changes– observations at course start and at intervals– reports from tutors/volunteers at start and at intervals– self perception scales at course start and at intervals– Web based self perception scales at course start and at intervals

› Ensure whatever is created is understood by all who will use it and asks information which is relevant (not merely interesting)

Tools for monitoring soft outcomes 1

› Learning Records– Records of learner self-, group- and peer- assessment– Tutor records of assessment activities and individual/group progress– and achievement– Learners’ files, journals, diaries, portfolios, artwork, videos, audiotapes, – performances, exhibitions and displays– Individual or group learner testimonies: artefacts, photographs and other

forms of evidence are also acceptable (LSDA/NIACE, 2003).

Tools for monitoring soft outcomes 2

Beyond recorded learning: Participatory approaches

http://www.peopleandparticipation.net/display/Methods/Participatory+Appraisal

“The term Participatory Appraisal describes a family of approaches that enable local people to identify their own priorities and make their own decisions about the future, with the organising agency facilitating, listening and learning.

Participatory Appraisal uses visual and flexible tools to ensure that everyone can join in regardless of background. It can be carried out in a place where people already meet in their everyday lives.”

› User friendly and user driven

Tools for monitoring soft outcomes 3

› Existing programmes and web based tools› Bespoke tools

– target charts

– scaled lines

– "race courses"

– visuals

Target charts for Distance measure and travel

T Chart to gain impressions and future direction

Pairwise chart to choose from a range of options

Sticky dot voting to agree a choice

Resources

› www.pms.ac.uk/mymop/ - health based personal measures and tools

› http://www.dwp.gov.uk/asd/asd5/WP8_annexe.pdf - examples of different tools used to record personal soft skills and outcomes

› http://communityplanning.net› http://www.peopleandparticipation.net/display/Methods/Home

[example - http://www.peopleandparticipation.net/display/Methods/Open+Space]

Mopping Up

› Unmet expectations?› Examples from UK online centres?› Other issues?