assessing the contribution of policies to outcomes – logic modelling and contribution analysis...
TRANSCRIPT
Assessing the Contribution of Policies to Outcomes – Logic Modelling and Contribution Analysis
Jackie Horne, Alison Stout Strategic Research
Strategy and Performance DivisionThe Scottish Government
Heather DohertyResearch Manager
Museums Galleries Scotland
Learning Outcomes
• Distinguish between inputs, outputs and outcomes• Understand the theory and practice of logic modelling and
contribution analysis• Practice developing a results chain and a logic model to help
assess how and whether policies are delivering on outcomes• Practice developing indicators to measure progress against
outcomes
National Performance Framework
Outcomes approach
Integral to:
•Programme for Government
•Spending Review 2011
•Government Economic Strategy
•Renewing Scotland's Public Services - Priorities for reform in response to the Christie Commission
The outcomes approach
It is widespread……• Countries that have implemented an outcomes based
approach include Canada, New Zealand, Unites Stated (in various forms across different states), England, Singapore and Malaysia, almost all other OECD member countries, many developing countries and a number of international agencies and NGOs.
But the language differs………• Outcome governance, outcomes management, accountability
for outcomes, results based management, but, overall, the reform agendas are similar.
The outcomes approach
There are commonalities:• they generally involve greater devolution of power and
decision-making and commonly adopt some form of performance management and reporting system (e.g. Virginia Performs, New Zealand governments’ Statements of Intent, Canadian Government’s whole of government framework).
It takes time is a common message…. • Evidence suggests at least 4-5 years of consistent effort is
required to embed an outcomes approach —and many organisations have been at it much longer.
Exercise
Distinguishing between inputs, activities,
outputs and outcomes
Understanding the relationship between policies and outcomes – single
intervention
A simple results chain
Inputs
Activities (policies)
Outputs
Reach
Short term outcomes
Intermediate outcomes
High level outcomes
Over 65s, especially those not doing enough exercise
Staff, equipment, leisure centres, etc.
Invest in exercise facilities and classes for over 65s
More accessible exercise classes and facilities for over 65s.
Increased levels of physical activity in over 65s
Lower blood pressure, improved mobility among over 65s
Increase in healthy life expectancy
National outcome We live our lives safe from crime, disorder and danger (NO 9)
Our young people are successful learners, confident individuals, effective contributors and responsible citizens (NO 4)
Intermediate outcomes
• Less antisocial behaviour.• Less time spent bored and on
streets.• Constructive use of leisure
time.
• Aspirations and confidence increase.
• Young people recognise that they have more opportunities.
• Young people learn a new skill.
Reach 16 – 21 year old in areas hardest hit by crime.
Output Coaching, games, leagues, role models, education during time out.
Activities Twilight basket ball (funded through Cashback)
Exercise
Results chain
Multiple policies - logic models and contribution analysis • Move beyond results chains to consider inter-
linkages between policies and outcomes• Consider strengths (and weaknesses) of the
contribution of different policies to outcomes• Assess and measure progress towards
purpose targets and national outcomes• Inform cost-effectiveness and resource
allocation – decisions on priorities• Highlight strengths and weaknesses in
evidence base
Steps of contribution analysis
Identify outcome
Logic modelling
Evidence gathering
Performance story
Uses of logic modelling and contribution analysis• Evidence effectiveness of a defined policy area
(Population Purpose Target)• Work out public sector partner organisations’
contribution to policy area and high level outcomes (Alcohol – Health Scotland)
• Identify cross office and agency contributions to National Outcomes (National Outcome 12)
• Develop indicators (CYP)• Business planning and indicators (Justice)
Alcohol – Partner Contributions to Outcomes
Inputs
Activities
Outputs
Reach
Short-term outcomes
Intermediate outcomes
High level outcomes
NHS
Brief advice
Preventive services
Hazardous and harmful
drinkers
Service uptake &
engagement
Police
Enforcement of drink driving laws
Random breath testing
Drivers
Increased detection rate
Local authorities
Enforcement of planning
controls & licensing laws
Enforcement actions
Licensed trade
Compliance with laws
BehaviourReduced alcohol consumption levelsLess drunkenness; less drink-driving
Improved mental wellbeingReduced inequalities in healthy life expectancy
Reduced inequalities in alcohol-related deaths and hospital admissions
EnvironmentsPhysical: Reduced exposure to alcohol-related hazardsEconomic: Reduced availability/affordability of alcohol
Social: Drunkenness less attractive; sensible drinking the norm
SG, UK govts, EU
Industry regulation
Taxation,displays, promotions,
advertising
Alcohol industry
Increased priceReduced incentives
Scottish Govt
Media campaigns
Sensible drinking
messages
General public - targeted
Understanding risks, attitudes
to drinking
Vol orgs
Detox, Intensive support
Addiction services
Adults with alcohol
problems
Increased sobriety &
stability
NO 12 - We value and enjoy our built and natural environment and protect and enhance it for future generations
5 years 10 years 15 years 20 years 25 years +
Processes
Improved understanding (amongst Government, public bodies and landowners) of:- impacts of Climate Change and how to address them [deleted CC adaptation bullet as this is the same]- relative flood risk across Scotland- the role of land use in managing flooding and river basin quality- the marine environment and ecosystems- how to achieve 'good environmental status' in land and marine environments
Better partnership working with and between stakeholders and key agencies
Plans and strategies in place for:- embedding CSGN into development plans under NPF2- flood risk management (also co-ordinated with river basin planning)- surface water management- driving decisions on use of marine environment
Actions planned at ecosystem level
National planning policy context helps to ensure that development plans and management decisions protect and enhance natural and built environments and create opportunities for their enjoyment
Greater consensus on which climate change impacts to tackleMore co-ordinated approach to flood risk management
Planning is less adversarial Fully integrated approach to surface water management
Public Awareness
Greater public understanding and awareness of:- greenspace and protected areas and opportunities they provide- responsibilities towards the built and natural environment- flood risk and of actions people can take
Greater appreciation of the role and value of the environment (built and natural, including marine) for ecosystem health and our economy
Widespread public understanding of:- the impacts of climate change- how soils store carbon (especially amongst land managers)- flood risk (and people take responsibility for their own protection)Knowledge informs land use decisions (at community level)
People and communities value direct and indirect benefits and services provided by biodiversity and healthy functioning ecosystems
The whole of central Scotland is seen as place of choice to live and workThe green network is valued for its contribution to enabling life styles which rely on less resource usePublic take responsibilty for protecting themselves against flooding
Recreation, Enjoymentand Well-Being
Tourism and recreational use increasesHigher levels of satisfaction reported with tourist visitsHealth and well-being improves
Participation in outdoor recreation equals or exceeds national average with greater rate of increase in deprived areasHealth and well-being continues to improve
People and communities are strongly involved in enjoying and protecting biodiversity and healthy ecosystems
Protection and Enhancement
Land:- greening of derelict land increased- rate of woodland development increased- access widened for all groups- 72% of water bodies at Good Ecological Status- water abstracted in a more efficient way
Marine:- protected areas are designated- areas are managed to improve health and diversity of environment and species- businesses are economically and environmentally sustainable
Land:- green networks integrated with buildings, paths and transport infrastructure- increase in quality of greenspace- losses in biodiversity minimised- 95% of features in special conservation areas in favourable condition- 77% of water bodies are at Good Ecological Status- land management improved, leading to protection of river basins- places actively managed to be part of ecosystems
Marine:- species are healthy and diverse
Land:- new infrastructure developed to withstand effects of climate change- land based businesses are successful, using natural resources sustainably- 97% of water bodies are at Good Ecological Status- protected areas are part of healthy ecosystems- populations of rare and endangered species are maintained- reduced number of areas assessed as being at significant flood risk
Land:- loss of species and habitats halted- ecosystems in protected natural areas offer optimum levels of services- land is better used, and use is better integrated- planning developments more sustainable- Scottish Sustainable Communities are exemplary models for 21st century living- Scotland's communities are resilient to change- Scotland taking advantage of positive opportunities from climate change
Land:
National Outcomes- Justice
9 – We live our lives safe from crime, disorder and danger
11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others
15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs
Unpacking theNational Outcomes – Justice
National Outcome Intermediate Outcomes
9 – We live our lives safe from crime, disorder and danger
A – We experience low levels of crime
B – We experience low levels of fear, alarm and distress
C – We are at a low risk of unintentional harm
11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others
D – Our social and cultural values promote pro-social behaviours
E – We have high levels of public confidence in justice institutions and processes
F – Our public services are fair and accessible
15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs
G – Our institutions and processes are effective and efficient
H – Our public services respect the rights and voice of users
National Outcome Intermediate Outcomes
Key low level outcomes
9 – We live our lives safe from crime, disorder and danger
A – We experience low levels of crime
tackling the underlying causes of crime, deterring offending, active prevention of offending, promoting desistance for those who have already offended
B – We experience low levels of fear, alarm and distress
people valuing family and communities, good social and dispute resolution skills, communities that are integrated, good enforcement of the law, good public understanding of risk
C – We are at a low risk of unintentional harm
active management of physical risks, effective emergency planning, effective management of risk by justice organisations
11 – We have strong, resilient and supportive communities where people take responsibility for their own actions and how they affect others
D – Our social and cultural values promote pro-social behaviours
public understanding of rights and responsibilities, people valuing family and communities, active citizenship and strong community leadership, low levels of social inequality, public intolerance and social stigma of criminal behaviour, strong and consistent political messages
E – We have high levels of public confidence in justice institutions and processes
equality of access to justice, efficient processes and joined up information systems, a swift and visible criminal justice system, positive experiences of the justice system
F – Our public services are fair and accessible
strong community engagement and involvement, equality of access to justice, protection of the rights of all member of the public
15 – Our public services are high quality, continually improving, efficient and responsive to local people’s needs
G – Our institutions and processes are effective and efficient
good leadership and management, continuously improving services, efficient processes and joined up information systems, good governance and understanding of organisational roles and responsibilities, efficient and effective resource allocation
H – Our public services respect the rights and voice of users
effective user feedback mechanisms, protection of the rights of all member of the public, a “consumer-led” service ethos
Supporting Outcome Intermediate Outcomes Possible pathways for intervention Address the biological factors predisposing some individuals to offend Promote the development of self-control Promote the development of empathy and compassion Promote healthy self-esteem / Broaden horizons and recognise potential Effective response to (social and physical) trauma Encourage communities to set appropriate boundaries on behaviour Strengthen social attachments. Ensure young people receive a good basic education. Natural boundary challenging behaviour in adolescence is managed. Prevent the problematic use of drug and alcohol. Stable employment
Address the underlying causes of crime
Remove structural barriers to services Help potential offenders to recognise that their behaviour has wider social costs Foster a moral/ethical commitment to the rule of law Encourage communities to exercise informal guardianship over their own public spaces Encourage public condemnation of criminal behaviour. Increase the severity of criminal justice sanctions for offending Increase the celerity (swiftness) of punishment
Deter offending
Increase the certainty of punishment Improve the use of situational crime prevention techniques Restrict the movement of those at risk of offending Restrict access to weapons Restrict access to alcohol and drugs
Reducing the opportunities to commit crime
Provide diversionary recreational opportunities
Low levels of crime
Encourage desistence amongst those who have offended
<REDUCING REOFFENDING PROGRAMME>
KEY TO RECOMMENDATION: RED – No further action Amber – Evidence unclear or lacking Green – Pursue further
Piloting contribution analysis in Museums Galleries
Scotland
What did we pilot it on?
• Hands On - An existing project that had come to an end in its current form
Where have we used it since?
• Festival of Museums – A new programme that was about to start
• Choices for Change toolkit – a decision-making toolkit that was developed collaboratively
• Collaborating to Compete conference
What did we learn?
1. It was helpful to name our first attempt as a pilot
2. It’s additional to any existing evaluation activity – rather than a replacement
3. It was helpful to apply the tool retrospectively to familiarise ourselves with the logic model part of the proces
4. BUT, more value in applying from the beginning of a project
5. It gives an overview of a piece of work and its component parts in a way that we don’t get with any other method
6. It can be applied to a small programme as well as a large one
7. The logic model can be customised to suit our needs– To incorporate our evaluation framework– To incorporate other frameworks, for example
Curriculum for Excellence– From project to project
8. Using the logic model part of the process is valuable in itself, even if you don’t go on to further analyse each contribution
9. It’s helpful to populate the logic model as a group (even a group of 2 people)
10. Once you understand the logic model headings and what they mean in a contribution analysis context, populating the logic model becomes easier and quicker
11. The logic model headings are very useful for revisiting the reason for your project or programme
12. The type of work that fits well with this approach quickly becomes apparent
13. Introducing contribution analysis to people works best with conversation and coaching, being given the documents alone to read is not enough
Exercise
Logic modelling
Indicators
‘’Imagine a car dashboard: an indicator is a warning light flashing on the dashboard. It is fed by one of many streams of data – maybe oil level, temperature, etc...It flashes when all is not well, suggesting we stop the car. The indicator “alerts us to something worthy of further investigation.”
The Good Indicator Guide (NHS Institute for Innovation and Improvement)
Early Years Outcomes and Indicators Framework
• Core set of indicators to monitor progress at a national and local level www.scotland.gov.uk/Topics/People/Young-People/Early-Years-and-Family/Early-Years-Framework/Implementation/Measuring-Practice
• Developed using logic modelling • Criteria
– easily available at sub national level (local authority or NHS Board)
– collected regularly– strong evidence base– collected by a credible source– tell us more than the face value of the individual indicator– taken together, cover the whole pre-birth to 8 age range.
Early Years – Core Indicators1. Teenage pregnancy rate (pregnancies among under 16 year olds (3
year average per 1000 relevant population))2. Appropriate birth weight for gestational age (low weight live births)3. Breastfeeding at 6-8 weeks (% of newborn children exclusively
breastfed at 6-8 weeks)4. % parents who rate their neighbourhood as a good place to live (% of
adults stating their neighbourhood is a 'very good' place to live)5. Children in benefit dependent households (% of children living in
households that are dependent on out of work benefits or Child Tax Credit)
6. % of obese children in P1 (estimated % of obese children in P1)7. % of children with dental decay in P1 (% of children with dental decay
in P1)8. Children (pre birth to 8) referred to reporter on care and protection
grounds9. Number of children pre-birth to 8 looked after a) at home b)
accommodated10. % of young people in sustainable post school destinations (% of school
leavers in positive and sustained destinations)
Exercise - Developing indicators
1. How will the indicator be defined and measured?2. Where will the data come from?3. Will it measure absolute numbers or proportions? 4. How frequently will the data be updated?5. Will you set an associated target, to drive forward change?6. What is a reasonable length of time to achieve the desired
outcome?7. Is it clear what type of change in the data represents an
improvement in performance?8. Are there any caveats/warnings/problems? e.g. missing data,
perverse incentives
The outcomes approach: general findings
Key organisational challenges:• ensuring visible and clear political and senior level leadership• unrealistic expectations about the timescale for change • ensuring co-ordination across policy areas and tiers of
government • linking performance information and decision-making locally and
nationally• most jurisdictions have struggled to link outcomes with budgets• accepting that outcomes are only one factor in decision making
The outcomes approach: general findings
Key technical challenges
• defining outcomes in a measureable way can be difficult• avoiding performance indicator overload • attribution of outcomes to government action per se, or to a
given policy, programme or project is not straightforward • outcomes generally have a long-term focus, which can result
in reporting time-lags• outcome measurement, data quality and reporting problems
Contact details
• Jackie Horne ([email protected])
• Alison Stout ([email protected])
• HeatherDoherty ([email protected])