Operationalising Social Return on Investment in advice services

Michelle Farr¹, Nick Abercrombie¹, Beth Jaynes¹, Sue Milner² and Peter Cressey¹

Department of Social and Policy Sciences¹,Politics, Languages & International Studies (PoLIS)²

University of Bath

Operationalising SROI

How can social return on investment be used to value advice services?

1. Short overview of the process - Michelle• From client recruitment to interviewing to analysis to

SROI

2. Applying SROI - Nick• What are the issues and difficulties associated with

applying SROI?

3. How our research tools are currently being adapted and developed for a range of advice services – Beth• Current project developing these methods for a

range of advice organisations

What is social return on investment?

• Outcome based evaluation• It seeks to measure changes that are most

important to different stakeholders• Form of cost-benefit analysis

• takes into account economic, social and environmental benefits

• Intends to create a ratio of benefits to investment

SROI= Value of benefits

Investment

Data collection in a SROI• Focussing on outcomes

• Changes that have occurred as a result of CAB• Mapping the impact of advice through interviews –

who has been affected by the changes and how?

• Deadweight• What would have happened without the client getting

advice from the CAB?• Would they have gone to another advice service or

got the same support elsewhere?• Would the same outcomes have occurred?• Benchmarks often used – what would have

happened without the CABs intervention?

Data collection in a SROI

• Attribution• How much credit can the CAB take for the outcomes?

(other organisations, client action)• Expressed as a percentage – not exact science!

• Displacement• How much does the outcome displace other outcomes?

• Benefit period and drop-off• How long is the benefit of the outcome likely to last for?

• Valuation of outcomesHow do we attach a financial value to the outcomes?

Research tools • Speaking to clients about the outcomes of advice

• Recruitment sheet, topic guides, asking clients about attribution, deadweight (Appendix 1)

• Analysis, developing financial values and proxies (Appendix 2)

• Developing rules around attribution, deadweight, and drop-off (Appendix 4)

• Creating 80 impact maps (e.g. Table 3.1)• Range of different outcomes (e.g. Table 3.5)• Great sensitivity of SROI ratios, therefore did sensitivity

analysis (Appendix 5).

Problems, problems… (Nick)

1. The problem of casework; a problem of cost.

2. The problem of attribution, deadweight; a problem of acceptance.

3. The problem of stakeholders; a problem of point of view.

The problem of casework

1. CAB offers a wide range of advice to anyone.

2. A very large number of outcomes are possible. Petra has 700. We logged 240 outcomes from advice and outcomes will have different values.

3. Every client’s combination of outcomes is unique. So each client has to have an impact map all to themselves. Very demanding research problem.

4. Very costly. 3 to 4 times the actual grant. Only possible with a lot of input – often expert – from volunteers.

5. Solution is either for the whole advice sector to rely on a few studies or to develop modal impact maps.

6. True not only of advice but most casework agencies.

The problems of attribution and deadweight

1. Attribution, deadweight, displacement, drop-off.

2. Ratio very sensitive to variations in these. Attribution or deadweight as an example.

3. Tried very hard in interviews to mitigate.

4. RCTs the traditional solution. Government policy. A problem of acceptance. RCT+CB in education.

5. RCTs hard work in these circumstances. Why?

The problem of stakeholders

1. SROI=CB + stakeholders. CB usually an investment tool. But need to consider all who my be affected by an intervention not just the investors. Hence social ROI.

2. But how to restrict the idea of stakeholder. Pebble in the lake. Drawing the line may be arbitrary.

3. We treated the client as primary stakeholder. So largely from the client’s point of view. But family impacts to some extent plus state agencies.

4. The gain from one stakeholder’s POV may be a loss for another e.g. benefit or debt

5. Our solution is to present an SROI divided into different types of value for different stakeholders.

Adapting Research Tools (Beth)

Advice Partnership for Excellence (APEX)• Advice Services Transition Fund (ASTF) project, funded

by the Big Lottery• One of the objectives of the funding is to measure

impact• Using SROI methodology we will measure the impact of

advice organisations and one council department• 100 service users

• Baseline data collected using GHQ12• 2 interviews over a year

• Dependent on volunteer time

Anticipating Challenges

• Co-ordinating project over a number of organisations will be difficult

• Different clients journeys - CAB - often outcomes take a while but for others they are immediate

• Criteria of some organisations - only some people are allowed access and guaranteed success - how do we allow for this?

• Need to have slightly different topic guide for each organisation but keep most relevant questions the same

Ways forward

• Can a survey based on a range of quantitative and qualitative questions be developed to cut down on transcribing costs and time?

• Using more fine tuned research questionnaires• Moving from short Warwick and Edinburgh Mental Well Being

Scale (WEMWBS) to General Health Questionnaire (GHQ12)

• Developing modal impact maps from our existing data