m&e on the cutting edge - wageningen portals...world summit on sustainable development cricket...

The Presidency: Department of Performance Monitoring and Evaluation

M&E on the cutting edge

Improving the Use of M&E Processes and Findings –

Experiences from South Africa

Mr Ismail Akhalwaya

Programme Manager

Institutional Performance Monitoring and Evaluation

Department of Performance Monitoring and evaluation in the Presidency

20 March 2014

Wageningen, The Netherlands

1


1. COUNTRY BACKGROUND – Pre 1994

2


1. COUNTRY BACKGROUND - Post 1994

Geograhic

3


1. COUNTRY BACKGROUND (continue)

Semi-federal system

3 spheres – national, provincial and local government - 9 provinces (states)

with own governments – but unitary state

278 municipalities

Most development services provided by provinces (health, education,

agriculture, social development) with local governments providing water,

sanitation, electricity, planning (cities a bit wider)

4


1. COUNTRY BACKGROUND (continue)

5

Total population= 52.98 million

Population by race

Africans= 42 284 100 (79.8%)

Whites= 4 602 400 (8.7%)

Coloureds= 4 766 200 (9%)

Asians= 1 602 400 (2.5%)

Government structure

Federal

Number of Ministries

47 Ministries

9 Provinces

The Presidency: Department of Performance Monitoring and Evaluation 7

1994 1995 1996 1997

First democratic election

Rugby World Cup TRC hearings National Anthem adopted

National Flag adopted Land rights restitution

Constitution adopted

Robben Island Museum

Presidential inauguration

Motor Industry Development Programme (MIDP)

Africa Cup of Nations 50th ANC conference

Reconstruction and Development Programme (RDP)

1998 1999 2000 2001

TRC final report Elections Coat of arms adopted World Congress against Racism

Voortrekkerhoogte renamed Thaba Tshwane

iSimagaliso Wetland Park declared World Heritage Site

13th international AIDS conference

Timeline


2002 2003 2004 2005

Mark Shuttleworth first South African in space

Towards a 10 year review

Elections 50th Anniversary of the Freedom Charter

World Summit on Sustainable Development

Cricket World Cup Announcement of SA hosting 2010 World Cup

Southern African Large Telescope (Salt) in Sutherland inaugurated

Hector Pieterson Museum opens

TRC report released Expanded Public Works Programme launched

Charlize Theron wins Oscar

Gold for 2004 Olympic Games swimmers

2006 2007 2008 2009

Same-sex unions legalised

SA serves on United Nations Security Council

Towards a Fifteen Year Review

First Mandela Day

Tsotsi wins Oscar Freedom Park opens Natalie du Toit qualifies for Beijing Olympics

Elections

Rugby World Cup

8


2010 2011 2012 2013

FIFA World Cup Third Census since democracy

SKA bid winners announced

Smart ID cards

SA joins BRICS 17th Conference of the Parties (COP17)

African Union Commission chairperson

1913 Natives' Land Act Centenary

Gautrain starts running

Nelson Mandela bank notes released

Nelson Mandela remembered

Framework of the New Economic Growth Path

National Development Plan launched

Mandela movie 'Long Walk To Freedom'

50th anniversary of Human Rights Day

Announcement of Infrastructure plan

Development Indicators 2012 released

Team SA wins 6 medals at Olympics

2014

20 Years Review

MAIN FEATURES OF GOVERNMENT PERFORMANCE MONITORING


2. WHAT IS THE PROBLEM THAT M&E AIMS TO ADDRESS?

• Implementation

weaknesses

• Disappointing

results - lack of

progress in key

indicators, such

as rural

unemployment

• Poor quality of

service delivery

• Insufficient value

for money

• Poor

accountability

• Need for continuous improvement, but..

• Poor planning, weaknesses in setting of

indicators and targets, weak logic models /

theories of change

• Focus on activities not outcomes, without

assessing the results or impact of the

activities

• Monitoring and reporting for compliance

Sporadic evaluation – so no in-depth

learning

• Insufficient use of data to inform

improvements

• Evidence-based planning and decision-

making not sufficiently valued

• Lack of reengineering of plans and

business processes based on evidence


3. WHY M&E?

12

Improving policy or programme performance (for continuous improvement):

this aims to provide feedback to programme managers.

Improving decision-making: Should changes be made to the intervention? Should it be continued? Should increased budget be allocated?

Improving accountability: where is public spending going? Is this spending making a difference?

Generating knowledge (for learning): increasing knowledge about what works and what does not with regards to a public policy, programme, function or organization.


4. ‘CULTURE’-BASED BARRIERS

13%

13%

27%

29%

33%

39%

39%

40%

44%

45%

54%

0% 10% 20% 30% 40% 50% 60%

Problems are concealed

Resistance from officials to transparent decision-making processes

Little respect for evidence-based decision-making in the department

The hierarchy makes it difficult to openly and robustly discuss performance

Fear of admitting mistakes or problems

The M&E unit has little influence in the department

M&E is seen as policing and a way of controlling staff

There is not a strong culture of M&E in the department

M&E is regarded as the job of the M&E unit, not all managers

Senior management do not champion M&E and honesty about performance

Problems not treated as an opportunity for learning and improvement


5. ROLES AND RESPONSIBILITIES FOR PLANNING AND M&E IN SA

14

Auditor General

• Independent monitoring of compliance

• Auditing of performance information

• Reporting to Parliament

Public Service Commission

• Independent monitoring and evaluation of public service

• Focus on adherence to public service principles in Constitution

• Reporting to Parliament

National Treasury

• Regulate departmental 5 year and annual plans and reporting

• Receive quarterly performance information

• Expenditure reviews

Public Service Dept (DPSA)

• Monitor national and provincial public service

• Regulate service delivery improvement

Presidency

• National Planning Commission (NPC): o Produce long-term plan

(20 years) • Department of

Performance Monitoring and Evaluation (DPME) o Produce government-

wide M&E frameworks o Facilitate government 5

year plans for priorities o Monitor and evaluate

plans for priorities o Monitor management

practices of government o Monitor front-line

services

Cooperative Governance Dept (DCOG)

• Regulate local government planning

• Monitor performance of local government

• Intervention powers over local government

Constitutional power

Legal power

Executive power

Line depts (national/prov)

• Monitor sectors


6. TIMELINES

2010 DPME established. 12 outcomes agreed, Minister’s performance agreements, delivery agreements for each outcome, quarterly reports start, using traffic light system

2011 Systems for Management Performance Assessment (MPAT) created,

monitoring of front-line services, and Evaluation Policy Framework. MPAT assessment of 103/156 national and provincial departments 2012 Reports against all of these programmes started including the 1st

evaluation completed. MPAT assessment of all 156 national and provincial departments 2013-14 Systems for monitoring local government, citizen-based monitoring

and operations management being piloted. Currently consulting on an ‘Discussion Document on Performance M&E Principles and Approach’

15


7. FOCUS OF DEPT OF PERFORMANCE MONITORING AND EVALUATION (DPME)

M&E of national priorities (Outcomes)

National evaluation system

Management performance assessment

M&E of front-line service delivery

Unannounced visits

Presidential Hotline

Citizen based monitoring

Monitoring of local government

16


The Outcome System

Government has adopted 12 outcomes with a view to measurable performance and accountable delivery.

At the end of April 2010, the President signed performance agreements with all Cabinet Ministers. In these performance agreements, Ministers were requested to establish and participate in Implementation Forums for each of the twelve outcomes.

The Implementation Forums have developed delivery agreements for the Outcomes. All departments, agencies and spheres of government that are involved in the direct delivery required to achieve an outcome, are party to the agreement.

The PoA system monitors progress of the delivery agreements. It tracks and reports on the key aspects through indicators and targets for the outputs, sub-outputs and in some cases, activities). A dedicated website for the PoA

17


National Evaluation System

While monitoring tracks what you have planned to do, evaluation asks deeper questions of effectiveness, efficiency, causality, relevance and sustainability. Government is taking forward evaluation to improve government’s performance and development impact, accountability, decision-making and to widen the knowledge base around government’s work.

A National Evaluation System is being established to entrench the quality and the use of evaluations in government, led by the Evaluation and Research Unit in DPME, and supported by a cross-government Evaluation Technical Working Group.

The first National Evaluation Plan 2012/13 has the strategic priorities for evaluation for 2012/13 and was approved by Cabinet on 13 June 2012. The second National Evaluation Plan for 2013/14 to 2015/16 was approved in November 2012 and the evaluations are being started.

The 2014/15 evaluations have been selected, and approved by Cabinet

19


Management Performance Assessments (MPAT)

DPME has been mandated to regularly assess the quality of generic

management practices in departments. A methodology for doing this was developed, in collaboration with the Offices of the Premier, National Treasury and the DPSA, and in consultation with the Office of the Auditor General and the Office of the Public Service Commission.

The methodology has been informed by similar management performance assessments carried out in other countries . The methodology involves working with the management of national and provincial departments to carry out self-assessments which are then moderated by subject matter experts and by cross-referencing to data produced by bodies such as the Auditor General, Public Service Commission, National Treasury and DPSA.

21

Level Description

Level 1 Non-compliance with legal/regulatory requirements

Level 2 Partial compliance with legal/regulatory requirements

Level 3 Full compliance with legal/regulatory requirements

Level 4 Full compliance and doing things smartly

22

Levels of assessment


Frontline Service Delivery Monitoring

Programme of unannounced visits to service sites such as schools, health facilities, social grant facilities, police stations, and municipal customer walk-in centres.

The objectives are to collect evidence on the quality of services and to work with the relevant departments to demonstrate how to use such monitoring information to inform improvements in the quality of service delivery

24


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%U

ser

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Use

r

Sta

ff

Mon

itor

Location &Accessibility

Visibility &Signage

QueueManagement

& WaitingTimes

DignifiedTreatment

Cleanliness& Comfort

Safety Opening &ClosingTimes

ComplaintManagement

System

National overview of monitoring visits (April-December 2013)

Not scored(0)

NotApplicable(N/A)

Very Good(4)

Good(3)

Fair (2)

Poor (1)


Presidential Hotline

Set up in 2009 to allow citizens to log their complaints and queries regarding service delivery.

To date, more than 180 000 cases have been logged and assigned to the relevant departments and agencies for resolution, with 94% of cases having been resolved

On a monthly basis, departments are informed of their responsiveness to complaints in the form of a scorecard on progress in case resolution and they are informed of the outcome of satisfaction surveys conducted of the quality of their complaints resolution

27


Citizen-Based Monitoring (CBM)

CBM describes an approach to monitoring government performance that focuses on the experiences of ordinary citizens in order to strengthen public accountability and drive service delivery improvements.

After the approval by Cabinet in August 2012 of a “Framework for strengthening citizen-government partnerships for monitoring of frontline service delivery”

28


8. LESSONS OF EXPERIENCE

The importance of consistent communication with users, continuous training on the use of models, improves relations and buy in for the use of these models (Outcomes and MPAT)

Documenting experience that emerge from the application of tools, enhances shared learning and ease of improvement / adjustment of tools

Creating a culture of partnership and support, builds ownership and interest in the use of the tools

More discipline and commitment in the application of legislation and prescripts, and improved knowledge of users of these prescripts (MPAT)

Improved understanding of the need to develop more impact indicators and improve the lives of the citizens

29


THANK YOU

30

m&e on the cutting edge - wageningen portals...world summit on sustainable development cricket...

Documents