quality of non-lending technical assistance a qag assessment october 25, 2005

Quality of Non-Lending Technical Assistance

A QAG Assessment

October 25, 2005

Quality Assurance Group


2

Table Of ContentsTable Of Contents

I. Executive Summary 3

II. Introduction 13

III. Findings 21

IV. Conclusions 34

V. Recommendations 35


3

I. EXECUTIVE SUMMARYI. EXECUTIVE SUMMARY


4

Summary of Findings

Assessment results show that NLTA has been a highly effective knowledge transfer instrument

But…improper coding of tasks under NLTA is high and calls for remedial actions


5

WHY ASSESS NLTA?

Management concern over rapidly growing share of NLTA in AAA program, almost doubling in dollar terms (from $21m to $42m) and in number (from 175 tasks to 303 tasks) between FY01 and FY04

NLTA quality never systematically reviewed by QAG


6

WHAT IS NLTA? The definition of free-standing NLTA applicable to the FY04

review period was “human and technological inputs to the development of advisory services and instructional activities”. NLTA often took the following form:

– Workshops and conferences: Vehicles to share international “best practice”, exchange knowledge, or build consensus

– Technical advice: Practical assistance to support implementation of policies, programs or projects

– Technical notes: Written practical guides to assist the client in implementation of policies, programs or projects

The distinction between ESW and TA was whether or not the activity aims to produce “original diagnostic/analytic content for the purpose of influencing the client’s policies”


7

THE SAMPLE The FY04 NLTA universe includes 303 tasks for a total cost of $41

million 1/

The NLTA sample includes 752/ randomly selected tasks for a total cost of $19 million, representing 25% of all tasks by number and 46% by cost3/

Tasks costing less than US$ 20,000 were excluded from the sample

Sample yields robust results at the Bank-wide level only

1/ At the time of sampling. This amount has since increased to $42.4 million

2/ The original sample included 85 NLTA tasks delivered to the client during FY04, of which 10 were Reimbursable TA tasks. It was subsequently decided that Reimbursable TA should be subjected to a separate QAG review. The sample size was reduced accordingly

3/ The sample includes 15 tasks (20%) funded under the Public-Private Infrastructure Advisory Facility (PPIAF) for a total cost of $4.3 million (23%)


8

MAIN FINDINGS

48% of tasks sampled (364/ out of 75), accounting for 53% of the sample cost ($10.1 million), were improperly coded as NLTA. Improper coding is primarily caused by a weak overall governance framework (unclear product definition, low SAP user friendliness, insufficient guidance/support to TTLs, unclear accountability)

Only 52% of tasks sampled were assessed and rated (39 tasks5/ out of 75 with a $ amount of 8.9 million or 47% of the total sample cost) as they were appropriately coded as NLTA6/

4/ Including 5 PPIAF funded tasks or 7% of the number of tasks sampled, accounting for 6% of the sample cost and representing 30% of the number of PPIAF tasks sampled and 26% of their cost


6/ The balance of 36 tasks could not be assessed and rated, using the NLTA guidance questionnaire, as they were not NLTA


9

At 99% overall Moderately Satisfactory or better (SAT) for both the weighted number of tasks and their weighted value, the quality of FY04 NLTA is strong. This compares well with the 94% overall Satisfactory or better rating for the FY04 NLTA tasks reviewed under the country AAA assessment

A high number (29%) of tasks were rated Highly Satisfactory (HS) overall

Strategic Relevance and Timeliness, Internal Quality and Dialogue and Dissemination were found particularly impressive by the panels as these dimensions were rated 98% SAT overall with only 4%, 5% and 8% of tasks rated Moderately Satisfactory, respectively

MAIN FINDINGS …Continued


10

Close client involvement and solid consultant and Bank inputs resulted in both high quality content generation and significant knowledge transfer as indicated by the 99% and 96% SAT overall rating for Internal Quality and Likely Impact, respectively

Managerial attention over the implementation of PPIAF funded tasks has often been limited, resulting in missed opportunities for better dialogue and dissemination and greater impact

Although resources were found to have been very effectively used, panels noted a number of instances where task costs had not been fully recorded thus understating the cost of about a quarter of tasks assessed

MAIN FINDINGS …Continued


11

KEY RECOMMENDATIONS

OPCS should:

Simplify the presentation of the ESW and TA Decision Tree (eliminate overlapping definitions, clearly define the terms “client”, “training”, “internal order” and clarify the trigger for task creation)

Prepare and launch an NLTA portal, similar to the ESW portal currently under development, to provide better guidance and support to TTLs

Given SAP’s low user friendliness and the extent of improper coding, allow all NLTA related entries (including the original task coding) into SAP to be made through the NLTA portal

Explore, jointly with SFR and TFO, ways to ensure full accounting of tasks’ costs (including TFs and other external contributions)


12

KEY RECOMMENDATIONS… continued

The Regions and Sector Anchors should monitor the use of the TA product code by their staff to reduce the incidence of improper coding


13

II. INTRODUCTIONII. INTRODUCTION


14

NLTA DEFINITION 1/

The definition of free-standing NLTA applicable to the FY04 review period was “human and technological inputs to the development of advisory services and instructional activities”. NLTA often took the following form:

Workshops and conferences: Vehicles to share international “best practice”, exchange knowledge, or build consensus

Technical advice: Practical assistance to support implementation of policies, programs or projects

Technical notes: Written practical guides to assist the client in implementation of policies, programs or projects.

1/ The key distinction between ESW and NLTA was that ESW aims at generating original analytic content while NLTA aims at disseminating/applying existing knowledge


15

THE NEW NLTA DEFINITION

The new (July 04) NLTA definition2/ includes activities3/ that are:

– Aimed at enabling an external client to implement reforms or strengthen its institutions;

– Free standing; and

– Linked to a Bank unit accountable for the services provided (exclude client-executed grant-funded activities)

2/ The key distinction between ESW and NLTA remains that ESW aims at generating original analytic content while NLTA aims at disseminating/applying existing knowledge

3/ With the following SAP output types: (a) institutional development plan; (b) “how-to” guidance;(c) model/survey; (d) client document review; and (e) knowledge-sharing forum


16

Promote accountability for quality by providing indicators on quality of NLTA work on a Bank-wide basis:

Encourage systemic change through:

- Improved understanding of key determinants of NLTA quality

- Disseminating assessment findings to appropriate Bank units

ASSESSMENT OBJECTIVES


17

Methodology and Guidance Questionnaire were developed by reviewing the NLTA governance framework as well as from lessons learned from a recent assessment of the quality of OESW

Tasks costing between US$ 20,000-$50,000 were reviewed by a “triage panel” and were rated using a simplified version of the Guidance Questionnaire4/. All other tasks were rated by a customized panel

The overall quality of each task was assessed on five quality dimensions (Strategic Relevance and Timeliness, Internal Quality, Dialogue and Dissemination, Likely Impact and Bank Inputs and Processes)

110 panelists conducted the assessment

22 Bank staff participated in the assessments as observers

4/ The simplified questionnaire provides and overall rating and a summary rating for each quality dimension but does not include ratings for individual questions

METHODOLOGY


18

Panelists used QAG’s new six point scale:

- Highly Satisfactory (1): Best practice in several respects and no significant deficiencies

- Satisfactory (2): Satisfactory or better on all aspects

- Moderately Satisfactory (3): Satisfactory on all key aspects but significant missed opportunities

- Moderately Unsatisfactory (4): Significant deficiencies in a few key aspects

- Unsatisfactory (5): Significant deficiencies in several key aspects

- Highly Unsatisfactory (6): A broad pattern of deficiencies

METHODOLOGY…Continued


19

The formal trigger for entry into the universe was the standard SAP milestone “Delivered to the Client”

The FY04 task universe included 303 tasks for a total cost of $41 million 5/

All tasks with a cost of less than $20,000 were excluded from the universe

The sample was stratified by cost ($1m+, $400-$999k, $200-$399k, 50-199k and $20,-$50k) with over sampling of large tasks

5/At the time of sampling. This amount has since increased to $42.4 million

SAMPLE


20

The sample includes 756/ randomly selected tasks with a total cost of $19 million, representing 25% of the universe and 46% of its cost 7/

Statistical Robustness: The sample is only representative at the Bank-wide level (95% confidence level with a 9% sampling margin of error) 8/

Results in this report are presented on a weighted basis to adjust for

over/under representation in the sample because of stratification and sample design

SAMPLE ROBUSTNESS

6/ The original sample included 85 tasks, of which 10 were Reimbursable TA (RTA) tasks. It was subsequently decided that RTA should be subjected to a separate QAG review. The sample size was reduced accordingly

7/ The sample includes 15 tasks (20%) funded under the Public-Private Infrastructure Advisory Facility (PPIAF) for a total cost of $4.3 million (23%)

8/ Results are less robust for Regional and Network cohorts given their small size


21

III. FINDINGSIII. FINDINGS


22

48% of tasks sampled were miscoded (369/ tasks out of 75 sampled) representing 53% of the $ amount ($10.2 million out of $19.1 million)

Miscoding was found across all Regions and Networks

There is no significant correlation between tasks’ cost and miscoding. The average cost of miscoded tasks is $283,000 while the average cost of all tasks is $255,000


CODING


23

CODING …Continued

SAP not sufficiently interactive and user friendly as it does not provide adequate guidance to users to prevent task miscoding

Preparation of the ESW and TA decision tree a good initiative but:

– Several TTLs found it overly complex, confusing, needing simplification

– Some of its definitions are overlapping and could lead to miscoding

– Some basic definitions are missing (who is a “client”, what is a legitimate trigger for task creation, what is training, what is an internal order etc…)

A number of Panelists as well as TTLs wondered why training was not considered part of TA for coding purposes


24

Of the 36 Miscoded tasks:

- 8 were Partnership activities (PT) 10/

- 7 should not have been coded as NLTA since they were conducted for the Bank’s benefit

- 5 should have been coded as ESW given their original analytical content

- 4 should have been coded as project preparation/appraisal and 4 were not yet delivered to the client as work was under way

- 3 were not separate tasks (created simply to separately account for a funding source)

- 2 were a combination of several activities (ESW, preparation, IO etc…)

- 1 was External Training, 1 was project supervision and 1 should have been cancelled as it never started implementation

10/ Clear definition and processing guidance for this product line have not yet been issued

CODING …Continued


25

52% of tasks sampled were assessed and rated (39 tasks11/ out of 75) as they were appropriately coded as NLTA (with a $ amount of 8.9 million or 47% of the total sample $ amount)12/

Overall quality of FY04 NLTA is strong (99% SAT13/ by number of task and by value) and above the 91% satisfactory or better rating for NLTA tasks assessed under the country AAA assessment 14/

The most frequently stated development objective of NLTA is Institutional Development/Capacity Building


12/ The balance of 36 tasks could not be assessed and rated as they were not NLTA

13/ SAT = Moderately Satisfactory or Better

14/ The country AAA assessment reviewed a total of 64 NLTA tasks (of which 1 was delivered in FY01, 9 in FY01, 20 in FY02, 23 in FY03, 10 in FY04 and 1 in FY05). The FY04 NLTA cohort was rated 94% satisfactory or better

RATINGS


26

Distribution of Ratings:

Rating No. of Tasks

% % of $ Amount

Highly Satisfactory (HS) 11 29 29

Satisfactory (S) 21 56 56

Moderately Satisfactory (MS) 6 13 14

Moderately Unsatisfactory 1 1 1

Unsatisfactory 0 0 0

Highly Unsatisfactory 0 0 0

Total 39 100 100

RATINGS …Continued


27

Country Task Title Sector Board

Africa PEFA Workshops Fin. Management

ECA Enhancing Poverty Anal. & Monitoring Poverty Reduction

LCR Indigenous Peoples & Sustainable Dev. Social Development

Moldova Social Protection Policy Dialogue Social Protection

Nepal Living Standards Survey Poverty Reduction

Peru Local Governance & delivery of Services Urban Development

Russian Federation Education Advisory Services Education

Slovenia Pension Model Social Protection

Sri Lanka Primary Dealers Capital Adequacy Framework Financial

Swaziland Railways Legal and Regulatory Private Sector Dev.

World Global Labor Toolkit Private Sector Dev.

OPERATIONS RATED HS


28

11 tasks or 29% of both the weighted number of tasks assessed and their total $ amount ($2.3 million) were rated Highly Satisfactory (HS)

The percentage of HS NLTA tasks (29%) is above the OESW and the FY02 ESW HS levels of 18% and 20%, respectively and comparable to the 29% SAT rating for NLTA tasks assessed under the Quality of Country AAA assessment

The 11 HS tasks were distributed among seven Networks (PREM, HDN, PSD, INF, ESSD, OPCS and FIN) and 4 Regions (ECA, SAR, LCR and AFR)

OPERATIONS RATED HS


29

Outstanding aspects of HS tasks include (by order of importance):

- Strategic relevance, particularly the consistency of the task objectives with the CAS

- Internal quality, resulting from either high quality direct task team contributions or the preparation of thorough consultant TORs, a well managed consultant selection process leading to the hiring of highly qualified consultants and high quality supervision of their work by the Bank

- Strong client involvement and participation and high quality interaction with the client

OPERATIONS RATED HS …Continued


30

RESULTS BY QUALITY DIMENSION

Quality Dimension HS S MS SAT

Overall Assessment 29 56 13 99

Strategic Relevance & Timeliness 38 57 4 100

Internal Quality 29 65 4 99

Dialogue and Dissemination 28 62 7 98

Likely Impact 14 77 6 96

Bank Inputs and Processes 16 54 22 91

<----------------Percent--------------->

Percent of Tasks in Each Rating Category


31

STRENGHTS

Assessed tasks were implemented within a well defined strategic framework

Task objectives were clearly stated

For workshops/conferences the consultative process and high degree of client involvement were usually very instrumental in defining content, structure, venue and participants. High degree of client involvement noted for other tasks as well

Extensive and relevant knowledge from both inside as well as outside the Bank was drawn upon to generate high quality content


32

STRENGHTS…Continued

Both the quality of partnership arrangements with other donors and dissemination arrangements were very appropriate

For a number of tasks, client impact was evident very shortly following delivery (shift in policy positions, setting up of a regulatory body, introduction of competitive bidding process, modeling of pension options etc…)

Solid task team contributions often with help from highly qualified consultants, well supervised by the bank

Many panels noted that task implementation had resulted in significant knowledge transfer


33

WEAKNESSES

Panels noted that for PPIAF funded tasks, beyond the selection of qualified TTLs, managerial attention had often been limited during task implementation, resulting in missed opportunities for better dialogue and dissemination and greater impact

Costs had not been properly recorded for about a quarter of tasks assessed:

Tasks often cross subsidized from other budgets/task codes TF contribution sometimes not accounted for Other contributions (from other donors, NGOs etc…) not accounted for

With about 54% of tasks delivered to the client in June, the Bank’s internal processes, more than client needs, seem to drive the timing of delivery


34

IV. Conclusions

Results show that NLTA is a valuable knowledge transfer tool that has been used very effectively by the Bank to engage clients in a timely manner and to deal with strategically important issues with appropriate expertise

But…improper recording of tasks under NLTA is high and calls for remedial actions


35

V.V. RECOMMENDATIONSRECOMMENDATIONS


36

OPCS should:

- Simplify the presentation of the ESW and TA Decision Tree (avoid overlapping definitions, clearly define the terms “client”, “training”, “internal order” and the trigger for task creation)

- In order to provide better guidance and support to TTLs, finalize and launch an NLTA portal similar to the ESW portal currently under development

- Given SAP’s low user friendliness, allow all NLTA entries (coding, milestones, AIS etc…) into SAP to be made through the NLTA portal

- Explore, jointly with SFR and TFO, ways to ensure full accounting of all task costs (including TFs and other external contributions)

The Regions and Sector Anchors should monitor the use of the TA product code by their staff to reduce the incidence of mis-coding

quality of non-lending technical assistance a qag assessment october 25, 2005

Documents