using monitoring and evaluation methods and ... - world...

Using Monitoring and Evaluation Methods and Tools Using Monitoring and Evaluation Methods and Tools for Effective Public Sector Management: Some for Effective Public Sector Management: Some

International ExperiencesInternational ExperiencesAndrew AsibeyThe World BankJune 20, 2006

ORGANIZED BY THE WORLD BANK AFRICA IMPACT EVALUATION INITIATIVEIN COLLABORATION WITH THE SOUTH AFRICA COUNTRY DEPARTMENT

HUMAN DEVELOPMENT NETWORK AND WORLD BANK INSTITUTE

REPUBLIC OF SOUTH AFRICAGOVERNMENT-WIDE MONITORING & IMPACT EVALUATION SEMINAR

Monitoring and Evaluation as a Core Function of Public Monitoring and Evaluation as a Core Function of Public Sector ManagementSector Management

““No regime can effectively govern or manage its fiscal No regime can effectively govern or manage its fiscal resources well if it does not invest in the evaluation of resources well if it does not invest in the evaluation of what government does (what government does (““its programsits programs””) and how it ) and how it performs (performs (““the effectivenessthe effectiveness””) of its programs. Neither ) of its programs. Neither ignorance of performance nor blind eye to performance ignorance of performance nor blind eye to performance is ever a recipe for good governanceis ever a recipe for good governance””

A document from the treasury Board of CanadaA document from the treasury Board of Canada

Recognizing Monitoring and Evaluation as a Recognizing Monitoring and Evaluation as a Core Function of Public sector ManagementCore Function of Public sector Management

•• Monitoring and evaluation is a core function because it Monitoring and evaluation is a core function because it attempts to ascertain and assess the effectiveness of attempts to ascertain and assess the effectiveness of government programs in achieving the expected results, government programs in achieving the expected results, outcomes and impactsoutcomes and impacts

•• The Business of government is to anticipate the needs The Business of government is to anticipate the needs of the population by providing programs of various of the population by providing programs of various kinds to achieve the desired resultskinds to achieve the desired results

Recognizing Monitoring and Evaluation as a Core Recognizing Monitoring and Evaluation as a Core Function of Public Sector Management, ContFunction of Public Sector Management, Cont’’dd

•• Government should be responsive to the needs Government should be responsive to the needs and priorities of its citizensand priorities of its citizens

•• But can government be responsive to the needs But can government be responsive to the needs and priorities of its citizens if the programs it and priorities of its citizens if the programs it initiates for their benefit are not effective?initiates for their benefit are not effective?


•• Government should be accountable and Government should be accountable and responsible in the spending of public monies or responsible in the spending of public monies or resources resources –– Ministers and officials must exercise Ministers and officials must exercise fiscal disciplinefiscal discipline

•• If programs are not effective in achieving If programs are not effective in achieving results, they may matter less results, they may matter less –– An ineffective An ineffective program is a waste of public moneyprogram is a waste of public money


•• How do we know if government or How do we know if government or development programs are being effective or are development programs are being effective or are having the desired results?having the desired results?

•• Application of timeApplication of time--tested monitoring and tested monitoring and evaluation methods, tools and strategies is evaluation methods, tools and strategies is critical to assessing overall effectiveness and critical to assessing overall effectiveness and efficiency of programsefficiency of programs


•• Monitoring and evaluation is not a fad. It is not Monitoring and evaluation is not a fad. It is not an optional method or techniquean optional method or technique

•• It is a core function of governance and It is a core function of governance and managementmanagement

•• Good governance and management require Good governance and management require ongoing monitoring and evaluation of policies ongoing monitoring and evaluation of policies and programsand programs

Embedding Program Evaluation in the Embedding Program Evaluation in the DecisionDecision--Making ProcessMaking Process

•• Ideally, a critical test of decisionIdeally, a critical test of decision--making is how real making is how real evidence is brought to bear on the decisionevidence is brought to bear on the decision--makingmaking

•• Embedding or integrating monitoring and evaluation in Embedding or integrating monitoring and evaluation in decisiondecision--making requires that there be a corporate making requires that there be a corporate policy or a whole government requirement that policy or a whole government requirement that evaluation is conducted and that the findings/evidence evaluation is conducted and that the findings/evidence from it is used in decisionfrom it is used in decision--makingmaking

Some International ExperiencesSome International Experiences

•• Monitoring and evaluation as a core function of public Monitoring and evaluation as a core function of public sector management was introduced in some OECD sector management was introduced in some OECD countries such as Australia, Canada, Britain and USA countries such as Australia, Canada, Britain and USA over three decades agoover three decades ago

•• Recently, developing countries in Africa, Asia, Latin Recently, developing countries in Africa, Asia, Latin America and the Caribbean are integrating M&E into America and the Caribbean are integrating M&E into public sector managementpublic sector management

Some International Experiences, Some International Experiences, ContCont’’dd

•• We will focus on two examples, Canada and We will focus on two examples, Canada and Trinidad and TobagoTrinidad and TobagoCanadaCanada

•• The monitoring and evaluation system in the The monitoring and evaluation system in the Federal Government of Canada is anchored in a Federal Government of Canada is anchored in a governmentgovernment--wide policy first introduced in 1982 wide policy first introduced in 1982 and revised in April 2001and revised in April 2001


The Policy is based on three fundamental principles, The Policy is based on three fundamental principles, namely:namely:

•• that achieving and accurately reporting on results is a primary that achieving and accurately reporting on results is a primary responsibility of public service managers;responsibility of public service managers;

•• that rigorous and objective evaluation is an important tool in that rigorous and objective evaluation is an important tool in helping managers to manage for results; andhelping managers to manage for results; and

•• that departments with the support of the Treasury Board that departments with the support of the Treasury Board Secretariat, are responsible to ensure that the rigour and Secretariat, are responsible to ensure that the rigour and discipline of evaluation are sufficiently deployed within their discipline of evaluation are sufficiently deployed within their jurisdictionjurisdiction


Managing for ResultsManaging for Results

•• The policy stipulates that managing for results is the The policy stipulates that managing for results is the prime responsibility of public service managers who prime responsibility of public service managers who must focus their attention towards results achievementmust focus their attention towards results achievement

•• Managers are held accountable for their performance to Managers are held accountable for their performance to higher management, to ministers, to Parliament and to higher management, to ministers, to Parliament and to Canadians Canadians


Evaluation as a Management ToolEvaluation as a Management Tool

Monitoring and evaluation is seen as an important tool Monitoring and evaluation is seen as an important tool that can help managers to objectively evaluate program that can help managers to objectively evaluate program or policy resultsor policy results

The Canadian M&E policy identifies two main purposes The Canadian M&E policy identifies two main purposes of evaluation:of evaluation:

•• to help managers design or improve policies, programs to help managers design or improve policies, programs and initiatives; andand initiatives; and


•• to provide, where appropriate, periodic assessments of to provide, where appropriate, periodic assessments of policy or program effectiveness, of impacts both policy or program effectiveness, of impacts both intended and unintended, and of alternative ways of intended and unintended, and of alternative ways of achieving expected resultsachieving expected results

ApplicationApplication•• All Canadian Federal Departments and Agencies are All Canadian Federal Departments and Agencies are

required to set up M&E unitsrequired to set up M&E units


Role of the Treasury Board SecretariatRole of the Treasury Board Secretariat

•• It provides leadership, guidance and support to the practice of It provides leadership, guidance and support to the practice of evaluation through the Centre of Evaluation Excellenceevaluation through the Centre of Evaluation Excellence

•• It must use evaluation results where appropriate in decisionIt must use evaluation results where appropriate in decision--making at the centre;making at the centre;

•• It sets standardsIt sets standards-- Annex B of the Policy contains key standards Annex B of the Policy contains key standards and guidanceand guidance

•• It monitors evaluation capacity in the government and where It monitors evaluation capacity in the government and where necessary improve upon it.necessary improve upon it.


Trinidad and TobagoTrinidad and TobagoA sector specific M&E system, focusing on the A sector specific M&E system, focusing on the social sectorsocial sector

A monitoring and evaluation policy for the A monitoring and evaluation policy for the social sector was approved by Cabinet in June social sector was approved by Cabinet in June 20052005


Purpose of the PolicyPurpose of the Policy

•• It seeks to place the practice of monitoring and evaluation withIt seeks to place the practice of monitoring and evaluation within the broader in the broader Government management accountability framework and to clarify thGovernment management accountability framework and to clarify the roles e roles and responsibilities of the Ministry of Social Development and and responsibilities of the Ministry of Social Development and other social other social sector ministries and departmentssector ministries and departments

•• The policy underpins an integrated monitoring and evaluation sysThe policy underpins an integrated monitoring and evaluation system for the tem for the social sector, which is decentralized in approach; with ministrisocial sector, which is decentralized in approach; with ministries and agencies es and agencies undertaking most M&E functions for their respective interventionundertaking most M&E functions for their respective interventionss


Classification of Social InterventionsClassification of Social Interventions

Social interventions are classified by expenditure. Social interventions are classified by expenditure. For example, interventions incurring For example, interventions incurring expenditure of $10,000,000 per year is subjected expenditure of $10,000,000 per year is subjected to a more stringent monitoring and evaluation to a more stringent monitoring and evaluation than interventions costing $500,000than interventions costing $500,000


Key Features of the Integrated M&E System:Key Features of the Integrated M&E System:•• Evaluation Policy and StandardsEvaluation Policy and Standards•• Tools series and Self Assessment ModulesTools series and Self Assessment Modules•• Program Management Information SystemProgram Management Information System•• MultiMulti--Year Evaluation PlanYear Evaluation Plan•• Evaluation Review and AssessmentEvaluation Review and Assessment•• Evaluation Information System (EVIS)Evaluation Information System (EVIS)•• Dissemination of Evaluation Findings/Information Dissemination of Evaluation Findings/Information

through website, Newsletters, Annual Review, etc.through website, Newsletters, Annual Review, etc.


ToolsTools•• Guidelines for Assessing Proposals for New Guidelines for Assessing Proposals for New

ProjectsProjects•• Seven Steps to Planning and Conducting a Seven Steps to Planning and Conducting a

Useful EvaluationUseful Evaluation•• Guidelines for Conducting an Evaluation Guidelines for Conducting an Evaluation

Assessment Assessment •• Self Assessment ModulesSelf Assessment Modules


Self Assessment ModulesSelf Assessment Modules

A selfA self-- assessment module is a toolbox designed assessment module is a toolbox designed to provide program managers with a simple tool to provide program managers with a simple tool for assessing how well services are delivered and for assessing how well services are delivered and impacting on the lives of the beneficiariesimpacting on the lives of the beneficiaries

•• There are eight selfThere are eight self--assessment modules:assessment modules:


•• Project Completion ReportsProject Completion Reports•• Social Sector Initiative Status ReportsSocial Sector Initiative Status Reports•• Physical Aspects of Service Delivery PointsPhysical Aspects of Service Delivery Points•• Client ReceptionClient Reception•• CounselingCounseling•• Client Satisfaction SurveyClient Satisfaction Survey•• Assessment of Core Competencies of Service ProvidersAssessment of Core Competencies of Service Providers•• A Tracer Study InstrumentA Tracer Study Instrument

Key LessonsKey Lessons

•• Need for an M&E Policy either Government Need for an M&E Policy either Government wide or sector specific to anchor the M&E wide or sector specific to anchor the M&E FunctionFunction

•• Need for a center of excellence to provide Need for a center of excellence to provide leadership, guidance and support to the practice leadership, guidance and support to the practice of evaluationof evaluation

•• Need for a tool kit and standards (e.g. selfNeed for a tool kit and standards (e.g. self--assessment modules)assessment modules)

End of this PresentationEnd of this Presentation

Thank you!Thank you!

using monitoring and evaluation methods and ... - world...

Documents