‘Pathfinder’: Maximising Results for Communities

Dr Roger Waite (Pathfinder Project / NZ Treasury)

io.ssc.govt.nz/pathfinder

Meta-messages

• All NZ Depts: enhanced outcome focus by 2003• Outcomes: ultimate goal (Holy Grail?) • Pathfinder (collective action, individual solutions)

• Outcomes driving strategy and output mix• Evidence-based approach to enhancing outcomes /

achieving Millenium Development Goals (MDGs)• Maximising outcomes from development proposals

(prioritise, assess & assist roles for development banks?)

An Incremental Approach to Capability Building

Focus on Outcomes

PATHFINDER

Strategic Plan;Statementof Intent

Value-for-Money

TOOLBOX; DECISIONMAKING SYSTEMS

STRATEGY,CAPABILITY,

PERFORMANCE

TRADE-OFFS,TRANSFERS(Within Vote)

(Next FY)

Goal: Build NZ Capability at Agency, Sector and ‘Joined Up’ Levels

Why Pathfinder?

After 10+ yrs, output funding has delivered

Quantum gains requires modified approach

Keep what works (output mgmt), but …

Complete public sector management modelLink inputs (through outputs) to outcomes

Meet requirements of legislation, parliament, etc

Ensure we deliver to stakeholders, taxpayers

Start: Limited capability & management tools

8 Agencies, Supported by CentreBuilding capability to enhance outcomesGoal: Better results for NZ communitiesAssumes performance feedback (outcomes) is a key factor in further improving performanceDefine outcomes; Prioritise; Measure; Manage– Performance of whole agencies / business units– Maximise outcomes from the output mix / $– Accentuate the positive; eliminate the negative

Outcomes / MDGs core to strategy, prioritising funding (macro) and proposals, assessing impact

Is the approach helpful in implementing Milennium Development Goals?

PATHFINDERBetter Results, Stronger Communities

Three Basic Approachesto Improving Results

• Top Down (Pathfinder emphasis / SOI / MDG focus?)

• Hierarchical (evolutionary direction for Pathfinder)

• Bottom Up, practitioner driven (opportunity)

SOI: Statement of Intent – a forward looking strategic plan deriving outputs from outcomes, and identifying capability

needs

Pathfinder ‘Building Blocks’1. Define & measure ‘mission critical’ outcomes

(using ‘state’ or ‘situation’ outcome indicators)

2. Map causal logic linking outcomes to outputs3. Assess impact of interventions4. Assess cost-effectiveness 5. Management to maximise outcomes6. Benchmarking with outcomes (business units / nations)

7. Focus strategic / annual plans on improving o/c8. Redesign planning & operations to maximise o/c

(incl. feedback & continuous improvement)

9. Improving outcomes across agencies / sectors

Building Blocks being documented on io.ssc.govt.nz/pathfinder

First Steps towards Real Results Mission, Measures, Management (M3)

• From aspirational outcomes (‘safer communities’)

• To the ‘Vital Few’ – focus on business actions• Tight Outcome Definitions (‘% re-offending in 24 months’)

• Outcome Indicators (‘28% re-imprisonment rate for Maori’)

• Impact Measures (‘12% reduction in reimprisonment rate’)

• Outcomes as Inputs into Management Systems

Delivery-based measures can substitute for Impact Measures where local factors can be managed (e.g. drugs work, but delivery factors critical)

Levers for Improving Outcomes

Status quo

outcome (client /service groups)

Risk, Need &

Likely Change (individuals / cases)

Improvement

in outcome (‘treated’ groups)

Situation Report

Benchmarking

Crude targeting

Priority setting

Demand analysis

Risk managemnt

Continue?

Modify? Grow?

Kill?

Measure Applications

ImpactAssessment

Assessment /AllocationDecisions

State /Environment

Indicators

Outcome Type

Screening for Risk: ValidationSocial Costs (Justice + Community)

Risk of Imprisonment (decile)

Societal

Cost

Top (Squares): Community-based Offenders

Bottom (Diamonds): Prison Inmates

$0

$100,000

$200,000

$300,000

$400,000

$500,000

$600,000

$700,000

0 0.2 0.4 0.6 0.8 1

Results Chains (After OAG Canada)Simple tools for reviewing funding proposals?

Inputs

Activities

Outputs

Immediate Results

Intermediate Outcomes

End/Final Outcomes

Outputs

Activities

Inputs

End/Final Outcomes

Intermediate Outcomes

Immediate Results

O2

A2

I2

O3

A3

I3

O4

A4

I4

IO

IR

Etc

Etc

Etc

Technical Efficiency Allocative Efficiency

Core Outcomesof AgencyDefined

Agency OutcomeIndicators

ImpactMeasures

Risk-basedTargeting

Tools

StrategicPriorities &

Planning

Benchmarking& Best Practice

MaximisingBenefits from

Intervention Mix

BusinessProcess Design

ManagementApplications

Linked OutcomeMeasures

After a business model used by the Department of Corrections

MANAGEMENT SYSTEMS & CAPABILITY

Identify CoreOutcomes

Define in Measurable

Terms

Measure as Outcome Indicators

Identify AreasFor Change

Define / RefineIntervention

Logic

IdentifyIntervention

Options

PrioritiseIntervention

Options

Design & Deliver

Interventions

ImpactMeasurementFramework

Cost Effectiveness

AnalysisOptimising Performance

Strategy Formulation

Ex Ante

Focus on Systems, Not Measures(lessons from the NZ Experience)

• Build management, not o/c measurement, systems• End game is outcomes – but output results important• Conceptual approaches give ‘one-off’ gains …

• Significant technical challenge to measure impact (management stuff is easy (?) with good measures)

• Ongoing gains in outcomes needs impact feedback?• Risk-based targeting an important lever for gains

• Stretching the capability of first world nation(s) • Capability development for 3rd world required?• Outcome-based allocative tools for the funder

‘Pathfinder’: Maximising Results for Communities

Dr Roger Waite (Pathfinder Project / NZ Treasury)

io.ssc.govt.nz/pathfinder

Success is more a function of consistent Success is more a function of consistent common sense than it is of genius -common sense than it is of genius -

Success is not a place at which one Success is not a place at which one arrivesarrives,, but the spirit with which one but the spirit with which one undertakes and continues the journeyundertakes and continues the journey

Most advanced NZ agencies have several things in common: Most advanced NZ agencies have several things in common: Committed senior management; Long-term focus; Clear Committed senior management; Long-term focus; Clear

outcomes; Strong analytical capability & MIS in support; outcomes; Strong analytical capability & MIS in support; Incremental development to vision rather than ‘big bang’Incremental development to vision rather than ‘big bang’

Will require cultural & capability changes of all managers; Will require cultural & capability changes of all managers; Outcomes subversive – managers will not like all answers; Outcomes subversive – managers will not like all answers;

No quick fixes – requires effort over multiple yearsNo quick fixes – requires effort over multiple years

Office of the

Controller and Auditor-GeneralTe Mana Arotake

Robust Impact Measures: Challenging

Structured Trial

Clear, EnforcedInterventionCriteria andThresholds

RetrospectiveDesign of

Structured Trial

RandomisedDesign

Non-randomisedDesign

RegressionDiscontinuity

Impact typicallymeasured in absolute

terms. Substantialcorrection for external

factors

Output / quality measures good surrogates where outcomes not strongly influenced by local factors (e.g. Drugs; Engineering)

Accountability: High Demands

Management (SOI)• Logical story of how

outputs enhance outcomes

• Selective use of indicator and impact measures to ‘tell story’ & set priorities

• Some discretion on external reporting

Accountability• Robust model linking

outputs to outcomes

• Precise, timely and comprehensive impact measures for each output

• Mandated external reporting of outcomes

In a ‘perfect world’ managers would have all the information required by an outcome-based accountability regime

Outputs and Outcomes

After: HM Treasury (UK)

Value for Money

Better healthMore treatmentsMore nursesMore money

External influences

Budget Inputs Output Outcome

Economy Efficiency Effectiveness

Which Outcome?

F ew er V ic tim s(Ju s tice )

P erce ived S a fe ty(Ju s tice )

R eso lved

Tim es from C h arg esF iled to P rosecu tion

(C ou rts )

C ou rt H ou rsb y C rim e Typ e

(C ou rts )

P rop ortion C on vic t ion sO vertu rn ed on A p p ea l

(C ou rts )

P rosecu ted C on vic ted

C rim es S o lved(P o lice )

[P ro po rtio n; T im eliness]

R ed u cedR e-o ffen d in g

(C orrec tion s , C Y F ,P o lice D ivers ion s )

L ess R ep orted C rim e(Ju s tice / P o lice )

CO M M UNITY S AFE TY(F rom C rim e)

Example

Rehabilitation QuotientsDetermined on NZ Offender

Groups / Population

BEST USE OF REHABILITATIVE FUNDING

Full Cost (Per Head) ofIntervention as Delivered in

New Zealand

Mean Future Cost perNZ Offender within thePrimary Target Group

Risk(s) of Reconviction

Cost, if Reconvicted

Cost-Benefit Ratio(Individual Interventions)

Rank Individual Interventionson the Basis of Cost-Benefit

Determine Budget for 'Core'Rehabilitative Interventions

Determine Mix of CoreProgrammes to be Delivered

in the Next Year

Determine Mix of CoreProgrammes to be Delivered

in the Next Year

Input Government andDepartmental Priorities

RI / RQ Framework

[A] CORE PROGRAMMESAND INTERVENTIONS

[B] PILOT / EVALUATIONOF NEW INTERVENTIONS

Cost and Benefit Datafrom Successful Pilots

Cost and Benefit Datafrom Successful Pilots

Managing for Better Outcomes• Broad focus (strategy, priorities, output mix, business systems)

• Decision making, better outcomes (evaluation just a tool)

• Measure outcomes (different outcome measures & groups)

• Work with (& test) clear intervention logic• Integrate with resource allocation systems• Focus business strategy on improving outcomes• Focus business processes on improving outcomes• Improve outcomes over multiple business cycles, and

across business units / agencies / sectors

Goal: maximise social benefit from finite resources