Concepts of Performance – Effectiveness and Equity

Effectiveness

Concepts of Performance

Effectiveness – represents the relationship between an agency’s outputs and the impacts or outcomes they achieve.

This criterion has received different levels of emphasis between different agencies and over time.

The tension between agency performance defined and managed in terms of the economy and efficiency with which specific outputs are delivered, and the broader and longer term concepts of the effectiveness of performance defined in terms of outcomes, remains an unresolved issue.

Concepts of Performance

The New Zealand Public Finance Act 1989 definition states that an outcome :

(a) means a state or condition of society, the economy, or the environment; and

– includes a change in that state or condition.

Outcomes are the impacts on, or consequences for, the community resulting from the existence and operations of the reporting entity. Desired outcomes provide the rationale for action and are the basis on which decisions should be made concerning the outputs as part of the range of possible interventions.

Definitions - Outcomes

• Market access for New Zealand’s animal and plant products is maintained and enhanced. (MAF)

• New Zealanders have a high-performing, trusted and accessible State sector, delivering the right services in the right way at the right prices. (SSC)

• A stable and sustainable macroeconomic environment. (Treasury)

• More people get into work and stay in work. (MSD)

Outcome Examples

Outcomes are the desired results or changes in the general state of well being in the community and are sometimes of a longer term nature. To take account of long-term outcomes and to report annual performance to show that society has moved towards desired outcomes as a result of policies and actions, intermediate outcomes may need to be identified and reported.

Intermediate outcomes are likely to be more specific with respect to targeted groups or in terms of their effects, and will be measurable with a greater degree of confidence.

Intermediate Outcome / Impacts

a) Is the outcome focused on a result of an action rather than the action itself (for example the impact that is sought)?

b) Does the outcome clearly describe the impact on the people of New Zealand or the people in a specific region, community or target group?

c) Is the outcome something that the elected representatives believe the community values?

d) Does the outcome provide a clear statement of what the purchaser is seeking to achieve?

Outcome specification checklist

e) Is the outcome part of a “hierarchy” of outcomes?

If so, is this hierarchy clear?

f) Where the outcome focuses on specific people or groups within the community, can these people or groups be clearly identified?

g) Is the outcome achievable within a specified time?

If so, is that timeframe specified?

Outcome specification checklist

Legislative requirement – PFA section 40

Information on future operating intentions must set out:

a)the nature and scope of the department’s functions and intended operations;

b)the specific impacts, outcomes or objectives that the department seeks to achieve or contribute to through its operations;

c)how the department intends to –

i. perform its functions and conduct its operations to achieve those impacts, outcomes or objectives; and

ii. effectively manage those functions and operations within a changeable operating environment;

Legislative requirement – PFA section 40

d) the main measures and standards that the department intends to use to assess and report on matters relating to the department’s future performance, including without limitation, the following matters:

i. the impacts, outcomes or objectives achieved or contributed to by the department (including possible unintended impacts or negative outcomes);

ii. the cost-effectiveness of the interventions that the department delivers or administers;

iii. the department’s organisational health and capability to perform its functions and conduct its operations effectively;

Legislative requirement – PFA section 40

d) any other matters that -

i. are reasonably necessary to achieve an understanding of the department’s operating intentions and direction; or

ii. may be specified by the Minister or the Responsible Minister for the purposes of subparagraph (i).

New Zealand Treasury guidance

The Crown Entities Act 2004 requires Crown entities to provide output performance information in their statement of service performance. However, to provide a coherent account of achievement, entities are encouraged to include information on their contribution to achievement of impacts, outcomes and objectives in their statement of service performance.”

[Guidance and Requirements for Crown Entities: Preparing the Annual Report, 2008, pp 15-16]

TPA 9 Service Performance Reporting

Concerned with the provision of information in order to strengthen external accountability for the delivery of specified outputs.

“(a) allowing external stakeholders to make informed judgments about an entity’s achievements with public resources; and

(b) reflecting the extent to which the entity has complied with significant legislative requirements for delivery of outputs. “

Outcomes are a less effective basis for accountability

• Cause and effect relationships between outputs and outcomes can be difficult to establish and measure.

• A number of entities may be involved in the delivery of contributing outputs.

• Other policy actions such as regulations, may also contribute to the outcome.

• The impact of an output on an outcome may take an unpredictable and/or extended period to emerge.

TPA 9 Service Performance Reporting

“This TPA encourages specification and reporting of the outcomes that purchasers are seeking to achieve from the delivery of outputs (among other means) and the link between those outcomes and outputs.

Information on outcomes should, however, be provided as contextual information, and care should be taken not to inappropriately imply that the entity is accountable for the outcomes.”

TPA 9 Service Performance Reporting

Although delivery entities, such as ministries, departments and crown entities in central government, and council management and staff in local government, are not accountable for outcomes, they nevertheless should have a strong interest in the extent to which their outputs contribute to the achievement of outcomes.

Such approaches and attitudes have in some instances been described as outcome-focused management.

Lines of Enquiry

Core business - what are the core functions, business and operational services - (core business) that the agency is responsible for delivering?

Outputs and measures - as the agency identified the right outputs and impact measures for the desired outcomes?

Achievement - does the agency deliver core business to the right people at the right time and in the right way?

Review and alternatives - does the agency review delivery? Does the agency consider alternative delivery options to achieve the intended impact?

Crown entities - how well does the agency undertake its Crown entity monitoring responsibilities including assisting the Crown entity to improve performance?

How effectively is the agency delivering its core business?

Performance Indicators

• The agency’s core business is delivering the intended results.

• The agency’s targets and indicators show the links between inputs, outputs, impacts and outcomes.

• Managers can articulate the appropriate balance between standards and cost.

• The monitoring agency has clearly defined its expectations to the Crown entity.

• The Minister is kept informed with good information.

• The agency can demonstrate its Crown entity monitoring has helped the monitored agency improve performance.

How effectively is the agency delivering its core business?

Effectiveness

All of which is based on one crucial assumption – that it is possible, and desirable, to arrive at one single set of evaluative criteria and thus a single statement of organisational (or programme) effectiveness.

And the measurement of effectiveness requires a means of predicting what would have occurred in the absence of an action.

Effectiveness

Effectiveness statements are typically not descriptive; they are evaluative and often normative.

Effectiveness

“… an answer to the question ‘how well is X performing?’ is inevitably contingent on whom one is asking. That is, the evaluative criteria required to transform a descriptive into an evaluative statement flow from the individuals or groups to whom we are referring to as constituencies, not from abstract, value-free theory of organisations or systems”

(Connolly, Conlon and Deutsch, 1980)

Effectiveness

What do you understand by the term ‘effectiveness’ in the context of the public sector?

Does it have any different meaning for an organisation in the private sector?

Effectiveness

For your organisation:

1.Who are the stakeholders (constituencies)?

2.What does effectiveness mean to each of them (i.e. what is their problem)?

3.What are the consequences of their assessments?

Effectiveness

“… effectiveness in organisations is not a thing, or a goal, or a characteristic of organisational outputs or behaviours, but rather a state of relations within and among relevant constituencies of the organisation. An effective organisation is one that is able to fashion accounts of itself and its activities in ways in which these constituencies find acceptable.”

(Gaertner and Ramnarayan, 1983)

Equity

Equity - is concerned with the way in which services are provided either in terms of the equal availability of services (as in the case of Police services) or the targeted provision of special education services to those most in need (e.g. those with learning difficulties or sensory impairment).

Concepts of Performance

v

Equity

No one would, hopefully, argue with the proposition that services in respect of water and sewerage should be consistently available to all citizens.

But we might argue that some services such as health or policing should be targeted more specifically at the differing requirements of differing social groupings.

Equity

Horizontal Equity – the equal treatment of equals, is exhibited when services are equally accessible to everyone in the community with a similar level of need.

Vertical Equity – the unequal but equitable (fair) treatment of unequals, is exhibits when the special needs of certain groups in the community are recognised and service delivery is adjusted to suit those needs.

Source: Australian Productivity Commission

Performance indicators for all schools

Equity

Unless we consider the impact of an action (intervention) on different groups in society, we cannot be sure that action is having its desired effect.

Note that an action may be specifically intended to redress an existing inequity.

Equity

Equity is also reflected in how services are provided.

•Are all those affected by a decision given the opportunity to have a say?

•Do consultative processes give due consideration to the views of those consulted (i.e. before decisions are made)?

•Does how and where services are delivered take account of citizens’ ability to access them?

Equity

Equity in:

•how decisions are made,

•what services are delivered,

•to whom they are delivered, and

•how they are delivered

may all occur at the expense of the economy and efficiency of service delivwery.

Equity

Equity is not mentioned in the New Zealand State Service Commission’s Performance Improvement Framework

•because equity is perceived as a function of Governmental policy, rather than agency implementation?

•and yet how that implementation occurs, at least, is a function of agency managers.

Equity

From a public sector organisation of your choice, what evidence can you find of the equity/fairness with which goods and services are provided?