i

Evaluation of the Multisectoral Team Learning MTL/(TEAMS) Program: Phase 3 Jaime B. Quizon Fumika Ouchi Victoria Gunnarsson Institute Evaluation Group EG04-90 The World Bank Washington, D.C. June, 2004

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

ACKNOWLEDGEMENTS

Jaime B. Quizon, Fumika Ouchi and Victoria Gunnarsson prepared this report for the Learning Board under the overall guidance of Marlaine Lockheed, Manager, Evaluation Group.

Many thanks go to Humberto S. Diaz for their excellent graphics preparation and document production. This report benefited from comments provided by Sukai Prom-Jackson who served as peer reviewer. This paper benefited also from a discussion with the MTL program administrators/coaches and some Regional Learning Board coordinators on June 21, 2004. Finally, the authors wish to extend their thanks to all participants who provided valuable information to this study as well as those who shared their experiences with us.

WBI Evaluation Studies are produced by the Institute Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at:

http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=getPubs&category=Publications&Intro=yes&instructions=no&showDetails=no&ID= Vice President, World Bank Institute Ms. Frannie Léautier Chief Learning Officer Ms. Phyllis Pomerantz Manager, Institute Evaluation Group Ms. Marlaine E. Lockheed Task Team Leader Mr. Jaime B. Quizon

iii

ACRONYMS

AAA Analytical and Advisory Activities AFR Africa Region CAS Country Assistance Strategy CDP Community Development Program CE Capacity Enhancement CT Country Team EAP East Asia and the Pacific Region ECA Europe and Central Asia Region ESW Economic and Sector Work FTL Foundations of Team Leadership FY Fiscal Year GDLN Global Development Learning Network HD Human Development HNP Health, Nutrition and Population HRSLO Human Resources Leadership and Organizational Effectiveness HQ Headquarters IEG World Bank Institute Evaluation Group LCR Latin America and the Caribbean Region MNA Middle East and North Africa MSP Multisectoral Project MTL Multisectoral Team Learning NGO Non-Governmental Organization PID Project Information Document PRSC Poverty Reduction Strategy Credit PRSP Poverty Reduction Strategy Paper QAG Quality Assurance Group QEA Quality at Entry Assessment QER Quality Enhancement Review QSA Quality of Supervision Assessment REG Regional SAP Systems, Applications and Products in Data Processing SAR South Asia Region TLC Team Learning Coach TTL Task Team Leader WB World Bank WBI World Bank Institute

iv

v

TABLE OF CONTENTS

ACKNOWLEDGEMENTS ................................................................................................ ii

ACRONYMS..................................................................................................................... iii

TABLE OF CONTENTS.....................................................................................................v

EXECUTIVE SUMMARY ............................................................................................. vii

1. PROGRAM DESCRIPTION AND EVALUATION OBJECTIVES......................... 1

Program Description..................................................................................................... 1

Evaluation Objectives ................................................................................................... 3

2. EVALUATION METHODOLOGY AND ANALYSIS ............................................ 5

3. EVALUATION RESULTS ........................................................................................ 7

The Independent Expert Panel Review......................................................................... 7 Methodology ........................................................................................................... 7

Evaluation Results................................................................................................... 8

A Special Note on MSP Lending Teams .............................................................. 10

The Supervisors’ Survey............................................................................................. 11 Methodology ......................................................................................................... 11

Evaluation Results................................................................................................. 13

Interviews with In-Country Counterparts ................................................................... 18 Methodology ......................................................................................................... 18

Evaluation Results................................................................................................. 21

Bank Teams........................................................................................................... 21

4. CONCLUSIONS AND RECOMMENDATIONS ................................................... 26

ANNEXES ........................................................................................................................ 29

Annex 1 : Number of MTL Teams FY03-Q4 and FY04 ............................................ 31

Annex 2: Ratings For the Seven PRSC, AAA and CAS Tasks .................................. 34

Annex 3: Supervisors’ Questionnaire ......................................................................... 36

Annex 4: Context of The Visit of Jaime Quizon, Sr. Evaluation Officer, WBI Evaluation Group ......................................................................................... 37

vi

Annnex 5: Correlation of TTLs’ Ratings of Team Characteristics, Inputs and Support vs. Supervisors’ Ratings of Process and Outcome Indicators - MTL Teams Onlya ................................................................. 38

vii

EXECUTIVE SUMMARY

The Multisectoral Team Learning Program (MTL) provides select multisectoral Bank operational teams with, among other things: access to a Team Learning Coach (TLC) and selective access to the Outward Bound Program; Task Team Leader (TTL) training that is tailored to meet specific needs; and a knowledge outreach consisting of access to help desk/online support and to various team learning tools (roundtables, websites, videos, and so on) that disseminate/share new knowledge on teaming issues. The Regions also fund these teams with at least $20K per team. The aim of the program is to transform the way World Bank teams work and learn by integrating team-based learning into operational work. The overall goal is that Bank/borrower teams produce significant improvements in the quality of their operational plans and strategies, processes, products and services.

The analysis presented in this report is based mainly on the results of three separate but related approaches to assessing the impacts of MTL. These include: (a) an independent Expert Panel review of the final written products of MTL vs. non-MTL Bank teams; (b) a survey of the immediate supervisors of the TTLs of both MTL and non-MTL Bank teams; and (c) direct interviews with government /NGO counterparts and donor partners of MTL-assisted Bank teams. Each of these evaluation methods have their own strengths and weaknesses. However, when taken together, they provide a more complete and balanced picture of the real on-the-ground effects of this Bank staff learning program.

This purpose of this study is to assess whether and to what degree the improved teaming and learning practices, processes and tools provided by the MTL program result in: (a) better quality Bank team processes and outputs; (b) enhanced in-country partnerships and teaming and learning with clients; (c) improved engagements with partners and clients in the delivery and dissemination of team outputs; and (d) increased and sustained use of MTL teambuilding and action learning practices. By and large, our main finding is that the MTL program contributes to these desired impacts, albeit in varying degrees.

Our results show that MTL improves the overall quality of final Bank CAS, PRSP and AAA documents. However, these quality gains are not very large because of the already high excellence of these Bank products overall. The largest quality gains are for AAA outputs, although CAS and PRSP documents improve as well. Also, being an MTL-assisted team does not necessarily translate to having better team characteristics, behaviors and performances than similar non-MTL Bank teams. However, a Bank multisectoral team’s use of specific key inputs -- in particular, team/TTL coach, Quality at Entry Assessment (QEA), Quality Supervision Assessment (QSA), self-help tools - and its level of support from relevant sector and other related Bank units are all positively and highly correlated with having some desirable characteristics, behaviors and performances ratings for a multisectoral team. A team’s use of meetings, workshops, and

viii

retreats and/or its attendance in peer gatherings also correlate well with preferred ratings of important team characteristics, behaviors and performances.

These findings suggest that some, and not necessarily all, elements of the MTL program have reassuring connections with improved team characteristics, behaviors and performances. Thus, the current plan to scale back on full-service MTL teams and to provide certain selected elements of the MTL program (e.g., TLCs under its “Quick Coach” initiative) to more multisectoral Bank teams in FY05 are steps already in the right direction. The MTL program might want to consider the dissemination and wider use of non-MTL inputs (QEA, QSA, self-help tools) that appear to be linked with desired team characteristics, behaviors and performances ratings as well.

There are not many Bank MTL teams whose in-country government/NGO counterparts have attended MTL-facilitated gatherings or events. Thus, there are not very many instances where Bank team engagements with their clients can be shown to have improved on account of direct MTL assistance. Nonetheless, in the several instances where MTL-orchestrated Bank-client engagements have happened, their overall effectiveness and impacts have extended often beyond the organized events themselves. The largest of these “spillover” benefits have occurred where large, complex Bank and counterpart multisectoral teams -- such as CAS and PRSP/C teams -- are involved, and in countries that have relatively less developed capacities and/or are moving away from highly centralized governance structures. These are the types of Bank teams that might be targeted for full-service MTL treatment in FY05.

Among the more significant and longer lasting contributions of these MTL-led events are the results-orientation and facilitation processes, teambuilding techniques, and negotiation practices that this learning program introduces not only to the Bank team, but also to the team’s government and NGO counterparts and donor partners. Several government clients have adopted these “new” methods in their own multisectoral, multi-institutional work. In addition, these MTL events have built mutual trust among participants, thereby facilitating a wider dialogue on more than just the issues directly relevant to the MTL Bank team. MTL-facilitated event s have allowed more Bank-client conversations to happen.

MTL-facilitated follow-up activities might be made a regular component of a broader Bank learning strategy focused on “taking the game to a higher level”, that is, not just including MTL coverage of more Bank teams but also of scaling up by inviting more client participation in joint Bank-client learning opportunities. These might entail, inter alia: using GDLN sites or similar media to deliver such learning opportunities more widely; training and/or fielding of region-based TLCs or MTL facilitators; broadening clients’ access to Bank self-help websites (e.g., websites on QEA, PRSP/C, CAS, and so on); opening more joint Bank/client learning opportunities particularly on other immediately relevant concerns (e.g., ethical standards and boundaries, conflict management/resolution, partnership management, listening skills, and so on). Closer collaboration with the World Bank Institute (WBI), who have the Bankwide mandate on client capacity enhancement and who have extensive experience with worldwide dissemination not only of Bank knowledge products but also of best practices through peer learning, can be very helpful here.

1

1. PROGRAM DESCRIPTION AND EVALUATION OBJECTIVES

PROGRAM DESCRIPTION

1.1 The MTL Program, recently part of the larger TEAMS Program, is in its third year.1 Its aim is to help World Bank operational teams fight poverty by transforming the way they work and learn together. The MTL goal is for Bank/borrower teams to change themselves with improved teaming and learning practices, processes and tools. The overall goal is that Bank/borrower teams produce significant improvements in the quality of their operational plans and strategies, processes, products and services. Towards this end, the program provides support both to individual and groups of Bank operational teams working on similar issues. In particular, the FY04 MTL program provides:

(a) access to a Team Learning Coach (TLC) and selective access to the Outward Bound Program for about 40 multisectoral teams nominated by the Regions 2;

(b) Task Team Leader (TTL) training that is tailored to meet the particular needs of the Region, and;

(c) a knowledge outreach consisting of: a help desk and online support to Bank operational teams; various modalities (roundtables, websites, videos, and so on) to disseminate/share new knowledge on teaming issues.

1.2 The FY04 MTL program is slightly different from the MTL programs of prior years. Presently, the Regions, using their own criteria, nominate candidate multisectoral teams for the program. The Regions also fund these teams with at least $20K per team. The MTL program “scopes” these nominees before participating with the Region in making a final selection of the teams for full-service MTL treatment.3

1 TEAMS is an umbrella program. It encompasses the MTL program, including a new MTL component called the MTL Flex, and the Knowledge Outreach program. It also works in cooperation with the Foundation of Team Leadership (FTL) program. 2 At the beginning of FY04, the program aimed at including 50 teams, however the aim was reduced to around 40 teams later on during the year. 3 In previous years, MTL teams were supposed to meet a number of set MTL criteria to qualify for MTL full service. If selected, the MTL teams were provided with team learning coaches, or TLCs, to help them with: identifying goals, related learning needs, and possible knowledge and learning resources relevant to their learning needs; resolving team conflicts; and providing tools for teaming and learning. Last year’s MTL Program granted selected teams up to $20K, if matched by the same amount by the Regions. These combined funds could be used at the discretion of the team to pay for professional and technical and/or learning and knowledge resource needs of the team.

2

1.3 Compared to FY03, the MTL program now operates by demand from the Regions. The MTL budget allocated from the Learning Board declined to $1.8 million4 from $2.2 million in FY03 partly because Regions have to support parts of their own participating teams. Because the direct funding that the Learning Board allocated to the MTL program is lower in FY04 than the previous year, and because Regions now have to use their own Learning Board budget to fund part of their MTL teams, the demand for MTL services has also declined.

1.4 In all, some 23 of the 40 full-service multisectoral teams in the FY04 program are carryovers from the FY03 MTL Program. Annex 1 lists all the MTL teams at the end of FY03 and the distribution of teams over the four quarters in FY04 and across Regions. Annex 1 also reports the number of teams that terminated MTL service during FY03-Q4 to FY04-Q4 and the new teams that entered the program. In all, 22 new teams entered the program in FY04 and six teams left the program. Towards the end of FY04, the MTL program supported 40 teams.

1.5 Because the Regions were given freedom to nominate and fund teams, the regional participation in the MTL program varies depending on priority. For instance, the Latin America and the Caribbean Region (LCR) withdrew almost all of its MTL teams from the program at the end of FY03 while the Africa Region (AFR) continued to support a large number of teams with full MTL service. Those AFR teams, whose MTL support ended in FY03, were closed because their tasks were completed. In the Eastern Europe and Central Asia Region (ECA), a push for a completely new set of MTL teams was initiated in FY04.

1.6 Because the demand for full MTL support has declined, the program has developed several new components in order to make their services more flexible and useful to Bank teams. In FY04, the new component, called MTL Flex, and the Knowledge Outreach program were developed to enable quick but short-term support for teams that are not able to commit to the MTL program for an entire year. Another new addition to the program is its cooperation with the Foundations of Team Leadership (FTL) training program. Experienced MTL coaches contribute as trainers in the FTL program as they, through their experiences with MTL teams, have gathered valuable knowledge on efficient ways for teams to work together. The initiative includes the design of learning modules for the FTL program to reflect the learning from the MTL program and the provision of team coaching and other support.

1.7 The TEAMS and the MTL programs will experience further changes in FY05. Especially, the programs will enhance the flexibility of their offered services. The number of traditional (full-service) MTL teams will drop from 40 to 20, reflecting both the decreased demand and the program’s desire to increase selectivity and focus on strategically important multisectoral teams such as CAS and PRSC teams. In line with increased flexibility, the program will support around 50 MTL Flex teams (30 at Headquarters (HQ) and 20 in the Regions) and will concentrate on knowledge capturing

4 Including the knowledge outreach program which constitutes $270K. Total Learning Board assistance to the larger TEAMS program in FY04 amounted to $2.6 million.

3

in a “Quick Coach” program. The coaching and cooperation with the FTL program will continue in FY05. The proposed budget for the MTL component of the TEAMS program in FY05 is expected to decline to around $1.3 million. 5

1.8 This Phase 3 evaluation only covers the MTL full-service teams. It does not assess the effectiveness and impacts of the MTL Flex or any other components of the TEAMS program, all of which are still relatively new.

EVALUATION OBJECTIVES

1.9 The WBI Evaluation Group (IEG) evaluated each of the first two years (Phases 1 and 2, respectively) of this ongoing MTL learning program for the Learning Board.6 These evaluations were based mainly on:

(a) interviews with team leaders (TTLs), team members and team learning coaches (TLCs);

(b) formal surveys of TTLs of MTL and non-MTL teams, and;

(c) separate discussions with MTL administrators and non-MTL TTLs.

1.10 The main objectives of these earlier MTL appraisals were:

(a) to provide information on the value of key components in supporting work-based learning within a multisectoral team context (in Phase 1), and;

(b) to evaluate MTL program impacts (in Phase 2).

1.11 In Phase 2 however, the impact evaluation results raised certain unresolved questions about the real MTL program impacts, in part because the evaluation methodology used relied mainly on self- ratings of team practices and products by task team leaders (i.e., MTL and non-MTL TTLs), a method whose results, when taken alone without supporting findings from other evaluation approaches, may not be very convincing.

1.12 This Phase 3 study builds on these earlier MTL evaluations. More specifically, its objective is to ascertain whether and to what degree improved teaming and learning practices, processes and tools of the program result in:

(a) higher quality team processes (the way the team works together) and products (CAS, PRSP, AAA, and so on);

(b) enhanced partnerships and effective teaming and learning with clients;

5 This includes the TEAMS Knowledge Outreach program which constitutes $272K. 6 These two studies are:

1. Prom-Jackson, Sukai, et al, “The Multisectoral Team Learning Program: An Evaluation of the Success of 29 Pilot Teams in Achieving the Key Results of the Program.” Phase 1 Study Based on Interviews with Team Leaders, Team Members and Team Learning Coaches, World Bank Institute, Washington, DC, September 2002.

2. Quizon, Jaime, “Phase 2 Evaluation of the Multisectoral Team Learning (MTL) Program,” WBI Evaluation Group Paper No:EG03-70, World Bank Institute, Washington, DC, August 2003.

4

(c) improved effectiveness of processes in the delivery/dissemination of team outputs; and

(d) increased use of team-building and action learning practices.

1.13 This study’s main contribution lies with its use of more independent (or third-party) evaluation approaches to assess the impact of the MTL program on the overall goal of improving the quality and delivery of multisectoral team products and processes.7 We realize that there are many points of view that need to be taken into account when reviewing a complicated learning program like MTL. The aim of our evaluation is to include as many of these viewpoints as possible, i.e., from self-evaluations by TTLs to subjective assessments by in-country counterparts and partners, in the hope that we get a clear, valid and fair picture of program impacts. We realize too that MTL is an organic process and that there are challenges (and limits) to evaluating impacts of a program that is still evolving. However, careful choice of a meaningful evaluation method and of relevant impact indicators and their measurement will help mitigate these challenges.

1.14 We know that in the process of learning by doing, several changes have or are being made to improve the MTL program already. 8 These important initiatives and the overall development of the FY04 program are noted and monitored in this study as part of a “rolling” evaluation of an organic institution. However, the on-the-ground impacts of these recent FY04 MTL program improvements are difficult to capture and measure since more time is necessary for any impacts from new initiatives to manifest themselves.

7 These evaluation approaches were suggested for the Phase 2 MTL evaluation. Unfortunately, these could not be carried out in FY03 for a number of reasons (i.e., lack of time, difficulties with obtaining final products of FY02 MTL teams, and so on) as mentioned in the Phase 2 report. 8 The MTL program counts experienced Bank coaches, learning materials and web pages on team building, a help desk and online support to operational teams, and other useful products and services among its many outputs. However, these products are not the main concerns of this proposed Phase 3 evaluation. The impacts of these intermediate MTL products and services on the Bank’s processes and products are our main interest.

5

2. EVALUATION METHODOLOGY AND ANALYSIS

2.1 As with the earlier Phase 2 MTL evaluation, the methodology used in this evaluation relies on the use of control groups, i.e., similar non-MTL teams against which the performance of MTL-treated groups may be compared, where possible.

2.2 In last year’s Phase 2 evaluation, we administered a survey questionnaire that asked TTL respondents for MTL and non-MTL teams about team-related factors (e.g., team size and composition, frequency of meetings, team-leading experience of TTL, and so on), the resources brought in for use by the task team (e.g., funds, partnerships, non-MTL coaches/mentors, other Bank teaming resources, government agreements, management buy- ins and so on), teaming practices (e.g., team assessments, Outward Bound experiences, use of “team-teaching” websites, and so on), exogenous factors affecting team performance (e.g., in-country conditions, within-Bank constraints, and so on) and TTLs own self-assessments of the outputs and impacts of their teams. These survey data were augmented (and/or elaborated on) with information from several sources, including:

(a) the Phase 1 evaluation of the MTL program, particularly the recorded interviews and other materials gathered for this report;

(b) select interviews and/or focus group discussions with TTLs, team members and TLCs (for MTL teams only);

(c) data from SAP 9 on teams’ use of resources;

(d) data and other documented information from the MTL program’s Activity Room site, and;

(e) informal discussions with MTL program administrators.

2.3 From the Phase 2 evaluation therefore, we have key data and other TTL survey information for: (a) the FY02 and FY03 MTL-treated teams (i.e., 23 completed Phase 2 TTL surveys) and (b) the randomly sampled teams in a control group (i.e., ten completed Phase 2 TTL surveys). Additionally, we have collected a number of final outputs (written products) of FY02-03 MTL teams.

2.4 For this Phase 3 evaluation, we collected more independent measures of the accomplishments of multisectoral teams from various sources. The aim was to get several views from different parties concerning the on-the-ground impacts of the MTL program. We note that each of these approaches have their strengths and weaknesses. However, because different viewpoints are taken into account, the individual findings from each approach, when taken together, provide a more complete and honest accounting of the real effects of the MTL program. In particular, we collected independent ratings of team outputs, ratings by supervisors and assessments by clients: 9 Systems, Applications and products in Data Processing.

6

2.5 For this exercise, we asked three former high- level Bank staff, who sit regularly in the Quality Assurance Group (QAG) panels of the Bank, to form our own Expert Panel. We asked the Panel to develop their own criteria for rating CAS, PRSP and AAA outputs, as based on their QAG experience and focused on the overall quality, relevance and likely effectiveness of these outputs. The Expert Panel then graded and ranked documents that we assigned to them, first individually but later as an expert team. The Panel discussed their individual ratings of each criteria for each document. Throughout this exercise, IEG did not inform the Panel which Bank outputs were from MTL teams and which were not. Quality ratings were totally blind.

2.6 We administered a simple one-page questionnaire to the immediate (or past) supervisors of the MTL and non-MTL teams whose TTLs were respondents to the Phase 2 MTL survey. These surveys were conducted mostly face-to-face, although a few were completed over the telephone. The questionnaire asked supervisors to rate the teams that they supervise, relative to other similar multisectoral teams that work within their own unit. Ratings were not only of the team’s overall performance alone, but also included ratings on several items, such as of team effectiveness; timeliness of team accomplishments (relative to established milestones); and team behaviors that are supposedly encouraged/taught by the MTL program. Our evaluations of the program are then based on comparisons between MTL teams and the control group

2.7 Observations of clients, as gathered from direct interviews, on the conduct and contributions of program-assisted teams can help determine the nature and extent of MTL program impacts.10 Because the majority of Bank clients are high- level government officials, this client-interview approach required face-to-face consultations at the country level. However, on account of costs and other constraints, not all countries with MTL-assisted teams were visited. Rather, careful selection of MTL teams and countries to visit was necessary. With assistance from program administrators, MTL-team TTLs, and their respective TLCs, we chose countries to visit where: (a) the time elapsed since the start of team’s MTL participation in the program had been sufficiently long, so we limited choices to FY02-03 MTL teams/countries; (b) counterparts in government, NGOs and the donor community were available and open to participate in candid, face-to-face interviews; and (c) these counterparts had noteworthy levels of client engagement with the Bank MTL team. In all, we selected and visited seven MTL teams and countries in three Bank regions (ECA, AFR and EAP) for this part of this Phase 3 evaluation.

10 Where valid and feasible, country clients were asked also to assess MTL-treated versus non-MTL treated Bank teams.

7

3. EVALUATION RESULTS

3.1 This section discusses the findings of the review by an independent expert panel review; the supervisors’ survey, particularly their ratings of specific team process indicators; and the IEG in-country interviews with Bank clients and partners concerning their engagement with Bank staff teams.

THE INDEPENDENT EXPERT PANEL REVIEW

3.2 IEG formed a special QAG-like Expert Panel, consisting of three former Bank staff from QAG’s pool of resource persons, to evaluate selected final written “products” of the MTL-assisted Bank teams. These included seven each of Country Assistance Strategy (CAS), Analytical Advisory Activities (AAA) and Poverty Reduction Support Credits (PRSC) documents. The Panel was provided only with the main final document from each task team and the cost information for each task; no other documents or opportunities for the Panel to discuss the task experience with either the team or their client counterparts was provided.

3.3 IEG collected the final CAS, AAA and PRSC documents or “products” from MTL teams directly. For the equivalent non-MTL “products”, we first obtained separate lists of all non-MTL CASs, PRSCs and AAA products that were completed about the same time as these MTL documents. We completed these lists from consulting relevant Bank websites (mainly for CASs and PRSCs) and from data mining the Bank’s Business Warehouse (for AAAs). We then selected randomly from these non-MTL lists to complete the relevant sets of non-MTL Bank products, which would serve as comparators for the MTL documents.

Methodology

3.4 The Expert Panel developed a separate questionnaire for each type of product. For AAA activities, the Panel developed a 16-point questionnaire, based on the questionnaire that the Quality Assurance Group (QAG ) uses for assessing the quality of economic and sector work (ESW). The Expert Panel questionnaire is provided in Annex 2. The Panel simplified the more detailed questionnaire used by QAG because the information provided to it was much less than that typically provided to QAG panels. For example, the Panel did not have the information necessary to adequately judge the dialogue and dissemination dimension or Bank processes contained in QAG evaluations. Moreover the likely impact of the task on the client and the Bank was judged solely on the basis of the substance and presentation in the reports provided to the Expert Panel. As a result of the limited basis and information on which the panel judged and compared the quality of the products, the results should be interpreted with some caution.

3.5 For PRSCs, the Panel based its 17-point questionnaire on that used by QAG for assessing the Quality at Entry of Bank Operations. The Panel also simplified this questionnaire to take account of the more limited information made available to it. The results should similarly be treated with caution.

8

3.6 For CAS papers, the Panel used as a basis the Standard Bank Guidelines for Preparing CASs and developed a 13-point scale to evaluate their quality. Since no background information other than the CASs was provided, the Panel rated only the dimensions related to the quality of the CAS and not the likely impact of the CAS or Bank processes.

3.7 The Panel adopted a 7-point rating scale, where rating of 1 indicated the highest quality and a 7 indicated an unsatisfactory result.11 Two panelists initially rated each product independently. The Panel then met and held discussions to come to an agreed rating among all the three panelists for each of the products. Where differences arose, the Panel thoroughly re-reviewed and discussed the contentions and arrived at final ratings, overall and sub-ratings, for each product. The overall ratings for the products were not derived as a simple average of the individual sub-ratings for the questionnaire dimensions. The Panel agreed on the appropriate overall rating for each product as a weighted average of the sub-ratings, with the weights dependent on the importance that the Panel assigned to each dimension.

3.8 After the Panel finalized its ratings, IEG evaluators informed it as to which of the products they just assessed had been part of the MTL program. The Panel was thus able ex-post to compare its ratings of products “with” and “without” MTL treatment. The ratio of the average “without MTL” score to the average “with MTL” score was computed to compare the overall ratings of “without” versus “with” groups. Because a lower rating score represents a higher quality rating, a computed ratio value of higher than 1.0 indicates that the “with” products got, on average, a higher rating than the “without” products.

Evaluation Results

3.9 With only a few exceptions, the Expert Panel found the overall quality of the products (even of non-MTL teams) to be high, suggesting that, on average, it is in general difficult to improve on ratings of quality. Results are shown in Table 1 below.

Table 1: Summary of Expert Panel ratings of MTL and non-MTL products

Product Reviewed With MTL Without MTL Ratio (Without/With)

AAA (6) 2.00 (4) 2.50 (2) 1.25

CAS (7) 2.50 (4) 2.67 (3) 1.07

PRSC (7) 2.25 (4) 2.33 (3) 1.04

Total (20) 2.25 (12) 2.50 (8) 1.11

Notes: (a) Panel rated products on a 7-point scale: 1=highly satisfactory and 7=unsatisfactory. Reported here are average ratings.

(b) The number of products reviewed is in ( ). AAA has 6 products after one of the final reports is excluded from the analysis.

11 This 7-point scale rating was mapped against the QAG’s 4-point scale rating system, which is 1=highly satisfactory, 2+=top end of fully satisfactory, 2=fully satisfactory, 3+=borderline between satisfactory and marginal, 3=marginally satisfactory, 4+=borderline between marginal and unsatisfactory, and 4=unsatisfactory.

9

3.10 Even discounting for the fact that the Panel was dealing with a relatively small sample, and therefore the Table 2 results should be interpreted with caution, the overall findings should be seen as encouraging. Some of the specific results include the following:

• Among the three types of tasks reviewed, the average quality of the PRCSs was the highest, with CASs being a close second and the AAA tasks third. The above average quality of PRSCs and CASs is probably because both CAS papers and PRSCs receive intense scrutiny and management attention in the Bank. CASs receive attention because of their strategic importance and the highly participatory manner in which they are prepared and PRSCs because of their relative newness and strong link to the Poverty Reduction Strategy Papers (PRSPs).

• Only one of the twenty-one products reviewed was rated 5 (which is equivalent to Marginally Satisfactory on a QAG rating system).12 Eight tasks were rated 3 (QAG equivalent to Fully Satisfactory), one task was rated a 1 (Highly Satisfactory in QAG) and eleven were rated 2 (between Satisfactory and Highly Satisfactory in QAG).

• The ratings on average for the twenty-one tasks are higher for the “with MTL” tasks than “without MTL” tasks. The ratios are 1.04 for PRSC and 1.07 for CAS, which show some improvement in the “with MTL” group over the “without MTL” group. The ratio for the six AAA tasks improves to an average of 1.25 from 0.96 when one of the tasks is excluded. These rankings also indicate which products are more likely to see quality improvements on account of MTL.

• The overall ratio for all three products combined, representing all 21 tasks, is 1.02 but this improves to 1.11 when one of the AAA tasks without the final report is excluded from the analysis. The latter indicates that average product quality of the “with MTL” group rates higher than the “without MTL” group by about 11 percent. It is difficult however to test corresponding levels of statistical significance of these and other estimates because of naturally small sample sizes.

3.11 The Expert Panel’s overall conclusion is that the results of the review should be interpreted with caut ion. While there is some basis to assigning a mildly positive correlation between the presence of MTL and the quality of the products, it is possible that the quality of PRSC and CAS tasks are generally of high quality, with or without MTL, probably because of the intense management scrutiny and attention they receive. For AAA tasks, the robustness of the comparison of AAA products, “with” and “without”, is dubious because of the very limited sample of only two “without MTL” AAA tasks. Finally, given the inherent variation in the quality of Bank teams, there is an assignment problem as to how much difference the presence of the MTL program made

12 Because the final product we had on hand may have been incomplete, we treat this one case as a possible outlier in the analysis that follows.

10

to the quality of a “with MTL” products and how much was due to the quality of the team carrying out the tasks.

A Special Note on MSP Lending Teams

3.12 Because it is very difficult to compare the quality of documents produced by multisectoral project (MSP) lending teams, we instead use a special approach to investigate whether the MTL program introduces more efficiency by shortening the time it takes for Bank MSP lending teams to reach established “lending process” milestones.

3.13 Results, as shown in Box 1 below, suggest that whereas non-MTL lending teams take fewer days, on average, from concept review to the beginning of loan appraisal, MTL teams eventually “catch up” and themselves take fewer days to move from the beginning of project appraisal to Bank board approval. This finding is not too surprising given that MTL teambuilding processes can weigh down on a team’s progress at the initial stages of the team’s overall mission. However, the program may still improve final MSP outputs, outcomes, or products.

Box 1: A brief note on MTL and the efficiency of MSP teams

(Box 1 continues on next page.)

We Use a special approach to study one possible effect of the MTL program on multisectoral project (MSP) lending teams. As the Bank maintains detailed information about the lending volume and process milestones. From SAP, we obtained the data on four actual project milestone dates: concept review; Project Information Document (PID) to Infoshop, beginning of appraisal; and Bank approval, for 13 of the 19 MSP teams that had participated in the MTL program. To serve as a control, we obtained these same data for 13 randomly chosen, regionally stratified, non-MTL teams. Comparing the average times that these teams used to move between the four milestones, we note that non-MTL teams are quicker in accomplishing the steps needed to proceed in early stages of the project cycle, whereas MTL teams take a shorter time towards the final stages of the project. On average, for instance, MTL teams used 227 days between completing the concept review document to getting the PID to the Infoshop. Non-MTL teams, on the other hand, took less than a third of that time, using only 66 days from concept review to getting the PID to the Infoshop. In contrast, non-MTL teams took 347 days, on average, between beginning of appraisal to Bank approval, while MTL teams were much more efficient, using only 116 days for the same process. All these computed average differences, however, are not statistically significant. Nonetheless, one might conjecture that involving the MTL program may not make all MSP processes more timely and efficient, especially in the beginning phases of the projec t. Heavy involvement of clients and emphasis on team work and learning, for example, may slow down the early stages of a project. However, MTL seems to improve the overall efficiency of projects as the extra effort and time spent at the early stages are made up for by greater efficiency in the later stages of the project cycle.

Average number of days between milestones and t-test results, MTL and non-MTL MSP teams

Average number of days between milestones T-tests of difference in mean number of days

Milestones (#)

MTL teams (N=13)

Non-MTL teams (N=13)

All teams (N=26)

MTL advantage* (disadvantage) T-stat P-value

Concept review (1) to PID for Infoshop (2)

226.5 66.3 146.4 (160.2) 1.19 0.246 PID to Infoshop (2) to Begin appraisal (3)

177.1 153.8 165.4 (23.3) 0.30 0.764 Concept review (1) to Begin appraisal (3) 403.6 220.1 311.8 (183.5) 1.32 0.199

11

(Box 1 continued.) Average number of days between milestones and t-test results, MTL and non-MTL MSP teams

THE SUPERVISORS’ SURVEY

3.14 This section summarizes the second of three components of IEG’s ongoing evaluation of the Bank wide MTL (or MTL, henceforth) program for the Learning Board. In contrast to the earlier Expert Panel report which emphasizes team output and outcomes, this second component focuses on team processes and/or progress in arriving at team outputs and outcomes, such as: the client engagement, timeliness of output delivery, team communication, and overall effective teamwork of multisectoral teams.

Methodology

3.15 All teams included in the supervisors’ survey are from the task team leader (TTL) responses from an earlier TTL survey. In that survey, we had asked TTLs to name their/the team’s supervisor and these are the persons we surveyed and interviewed. In total, we contacted 43 team supervisors for interviews. Of these, 28 were supervisors of MTL teams and 15 were supervisors of non-MTL teams. These supervisors are as randomly drawn as the TTL respondents in the earlier TTL survey.

3.16 IEG requested these past and current supervisors of MTL and non-MTL team leaders to complete a short one-page survey. 13 This exercise was not a ve ry easy nor straightforward undertaking, given the high- level positions of these Bank supervisors and because some were located overseas or frequently traveling. In the majority of instances, we administered these surveys face-to-face, explaining the background and purpose of the survey and conducting an informal interview with the supervisor whenever possible. In all, 17 MTL supervisors and seven non-MTL team supervisors responded to the one-page questionnaire. This small sample size limits the depth of the statistical analysis that can be done with the collected data. In all, we employ basic summary statistics and simple correlation analysis of variables to analyze the results of this survey.

3.17 The survey itself asked the responsible supervisor to rate - relative to other similar teams in their units or, if there were none in their unit, relative to other multisectoral

13 A copy of the supervisors’ survey is in Annex 3 of this report.

Average number of days between milestones T-tests of difference in mean number of days

Milestones (#)

MTL teams (N=13)

Non-MTL teams (N=13)

All teams (N=26)

MTL advantage* (disadvantage) T-stat P-value

Begin appraisal (3) to Bank approval (4)

116.4 346.8 241.2 230.4 -1.49 0.150 Concept review (1) to Bank approval (4)

484.2 566.9 530.9 82.7 -0.31 0.759

*Measures the number of days used by non-MTL teams minus the number of days used by MTL teams. A positive (negative) value indicates that MTL teams are more (less) efficient than non-MTL teams.

12

Bank teams of which they were familiar - the characteristics, behaviors and performances of the sampled MTL or non-MTL team. Ratings were on a 7-point scale where 1= very poor and 7 = very good. The specific team characteristics, behaviors and performances that supervisors rated were:

(a) Team spirit/cohesion;

(b) Quality of team outputs;

(c) Engagement of clients;

(d) Timeliness of team deliverables/overall progress;

(e) Team learning practices;

(f) Appropriateness of team size and composition (mix of skills);

(g) Leadership within the team;

(h) Overall cost effectiveness of team’s outputs, and;

(i) Effectiveness of adopted processes, including delivery/dissemination of team outputs.

3.18 Additionally, we computed a single aggregate characteristics, behaviors and performances rating for each team. This aggregate rating is the simple average across all the ratings of the team characteristics, behaviors and performances as listed above.

3.19 Both supervisors and TTLs rated some of the characteristics, behaviors and performances. Table 2 depicts the correlations between the supervisors’ and the TTLs’ rating of the characteristics, behaviors and performances. The “+” signs indicate correlations greater than 0.33. Supervisors and TTLs never had completely opposite ratings of team performance. However, only three of the six comparable characteristics, behaviors and performance ratings were positively correlated, whereas the other three characteristics, behaviors and performances had no significant correlations. This only highlights the importance of taking different viewpoints, wherever feasible, when evaluating any program.

Table 2: Correlation of ratings between TTLs and supervisors on process outcome indicators.

Process Outcome Indicators Correlation of Ratings

Team spirit/cohesion +

Quality of team outputs +

Engagement of clients

Timeliness of team deliverables/ overall progress +

Team learning practices

Appropriateness of team size/ composition (mix of skills)

13

3.20 Using simple correlation estimates, we then related (a) this computed aggregate characteristics, behaviors and performances team rating, and (b) the individual team characteristics, behaviors and performance items, to information for the same team obtained from the earlier completed TTL survey (in the Phase 2 MTL evaluation) or, in particular, to:

(a) Team/TTL characteristics such as whether or not the team is part of the MTL program, frequency of team meetings and workshops etc., number and composition of staff in team and level of TTL experience;

(b) The TTL’s own reported frequency and/or intensity of use of specific team-building and performance improving inputs (team/TTL coach; in-country visits by the team/TTL coach; Outward Bound team building exercises; MTL help desk, outside group facilitators, Quality Enhancement Review (QER), Quality at Entry Assessment (QEA), Quality of Supervision Assessment (QSA), mentors for team members from the Bank’s mentoring program, and self-help sources including Bank-supported resources, such as web-based information and other related Bank documents), and;

(c) TTL-rated (on a five-point scale) levels of support from (a) client counterparts in government; (b) the regional vice-presidency to whom the team was responsible; and (c) the corresponding Bank sectoral units to where team members are mapped, for the aims and related activities of the team.

Evaluation Results

3.21 Table 3 shows how (i) the make-up of the multisectoral team; (ii) the inputs it uses in carrying out its mandate; and (iii) the levels of support it received within and outside the World Bank as rated by the TTLs of these teams (or the row variables), correlate with ratings on team characteristics, behaviors and performances provided by direct supervisors of these TTLs and/or multisectoral teams (or the column variables). The “+” (or “-”) sign signifies that the simple estimated correlation between the corresponding row variable from the TTL survey and column variable from the supervisor survey is positive (negative) and greater than 0.33 (-0.33). These correlation estimates for the aggregate of MTL and non-MTL multisectoral teams. Although correlation estimates do not show causality, taken together with our understanding of the program, they provide helpful indications as to the likely individual and overall team-related effects of the MTL program.

14

Table 3: Correlation of TTLs’ ratings of team characteristics, inputs and support vs. supervisors’ ratings of process and outcome indicators a

Supervisors’ Ratings of Team Behavior and Performance:

TTL Reports on Team characteristics, Inputs and Support Received: Overall

Team spirit/cohesion/sense

of purpose

Quality of team outputs

Engagement of clients

Timeliness of team

outputs/overall progress

Learning practices as a team

Appropriate-ness of team

size and composition

Effective team

leadership

Cost effectiveness

of team’s outputs

Effectiveness of processes

and delivery of team outputs

I. Team/Team leader characteristics a) Team

Being an MTL team Number of team:

Workshops/retreats + + Tours/missions Assessments Meetings + + + + Peer gatherings + + + +

Number of staff in team: Team members - - HQ staff - - - Country Office staff - - Client staff - - - - - Consultants - - - -

b) Team leader Months as TTL of team TTL experience (years) - - - - - - TTL Bank experience (years) - - - - - - -

II. Team inputs/Use of b: Team/TTL coach + + + + + Group facilitator + Quality Enhancement Reviews (QER) Quality at Entry Assessments (QEA) + + + + + + Quality of Supervision Assessments (QSA) + + + + + + Mentoring program + + Outward Bound + MTL help desk Self help job aids (e.g. CAS, QAG, PRSP websites) + + + + + +

III. Level of team support From clients From Bank regional units/vice residencies From sectoral units and/or other Bank units + + + +

a “+” equals correlation greater than 0.33, “ -“ equals correlation less than –0.33. b Correlation between the supervisor outcomes indicators and a dummy variables indicating using the input (1) and not using the input (0).

15

3.22 The second column of Table 3 shows that a multisectoral team’s use of certain inputs (team/TTL coach, QEA, QSA, self-help tools) and its level of team support from sector and other related Bank units are positively and highly correlated with supervisor’s aggregated rating of a team’s characteristics, behaviors and performances. These team inputs and source of team support are more likely than other considered factors to result in higher team characteristics, behaviors and performances ratings.

3.23 In contrast, Table 3 suggests that certain elements are inversely correlated with our aggregated supervisor’s rating of a multisectoral Bank team’s characteristics, behaviors and performances. The years of TTL experience (and of Bank experience) of a multisectoral team’s TTL, and the number of client staff who are core members of the multisectoral team are both highly and negatively correlated with the team supervisor’s aggregated rating of the team’s characteristics, behaviors and performances. These results are not very surprising; they suggest that TTLs with longer experiences within the Bank and as team leaders are less likely to rate high in team-related measures of its characteristics, behaviors and performances. Also, the greater the number of client staff who are part of the core Bank multisectoral team, the more likely are lower team ratings for the following desirable team characteristics, behaviors and performances: team cohesion/spirit/sense of purpose, the quality of team outputs, learning practices as a team, and effectiveness of within- team leadership.

3.24 Table 3 notes that “being an MTL team” is not highly correlated with having the preferred characteristics, behaviors and performance ratings of a multisectoral team. Additionally, while the use of a team/TTL coach is highly correlated with successful multisectoral teams, other MTL-related inputs, particularly team tours/missions, team assessments, Outward Bound experiences and MTL help desk, are not similarly correlated. In short, the use - and therefore also the provision - of these inputs cannot be correlated, as yet, with higher characteristics, behaviors and performance ratings of a multisectoral team.

3.25 The above results hold even when we consider MTL program teams only. 14 Also, these results show that for MTL teams, their use (or attendance) of MTL-facilitated team workshops and peer gatherings correlate well with desired ratings of important team characteristics, behaviors and performances. In short, an MTL team’s use of workshops and its attendance of MTL-organized peer gatherings link well with some individual characteristics, behaviors and performance ratings for the team.

3.26 Again, these results indicate only that certain elements or independent features of the MTL program (and not necessarily all its elements or features) have reassuring connections with desired team characteristics, behaviors and performance outcomes. It would be good if these useful MTL-supplied inputs can be costed, so more informed managerial decisions on their provision can be made.15

14 Annex 5 provides a table similar to Table 3 above but for the sample of MTL-assisted teams only. 15 We note too that because MTL-related expenditures are not readily available by the MTL team, it is also very difficult to know the exact levels of support given by the program to the teams. Are teams with more

16

3.27 Not all Bank multisectoral teams require full-service MTL support for them to perform better. Most Bank operational teams are likely to be deficient in one or a few desired team characteristics, behavior and performance features only, and this can be addressed suitably by the provision of selected MTL teambuilding inputs. For instance, Table 3 notes that the number of team workshops, retreats or meetings are highly and positively correlated with better team spirit, cohesion, and sense of purpose. Likewise, the Outward Bound experience and the number of team workshops appear to promote more effective team leadership. The number of peer gatherings attended by the TTL is significantly associated with having, among other things: better team learning practices, a more appropriate team size and composition, more cost effective team outputs, and greater effectiveness of the processes and delivery of these outputs. The number of team meetings is also associated with having better quality of team outputs.

3.28 Thus, certain key MTL inputs can be immediately useful to a large number of Bank teams who may require targeted assistance only, and less-than-full MTL service, for overcoming a certain team characteristics, behaviors and performance deficiency. The Table 3 findings, in fact, suggest what specific MTL inputs might be offered to Bank multisectoral teams to address their specific diagnosed shortcomings.16 These Table 3 results support the decision to define a more flexible MTL program in FY05 that provides quick and short-term services to Bank teams requiring assistance.

3.29 As already noted, certain non-MTL Bank-provided inputs (in particular having Quality at Entry Assessments (QEAs), Quality of Supervision Assessments (QSAs) and using self-help tools) are positively and significantly correlated with supervisor’s aggregated rating of a team’s characteristics, behaviors and performances. These inputs are not unique features of the MTL program. They are available to all multisectoral Bank teams and teams use them in varying degrees. These Bank-provided inputs might be the main reason why non-MTL teams, in particular, score relatively high on the quality of their final outputs as determined in the independent QAG-like review of the previous section.

3.30 Table 4 shows, among other things, the percentage of our MTL and non-MTL sampled teams who report use of these specific team inputs. As noted here, more non-MTL teams report use of these team characteristics, behaviors and performance-enhancing inputs as well as QER services and the mentoring program. In a way, non-MTL teams compensate for their being outside the MTL program by partaking more of other Bank-assisted inputs - relative to their MTL counterparts - thereby improving the quality of their final outputs. In the absence of the MTL program then, it is still possible to produce quality multisectoral outputs with assistance from other parallel Bank programs. But, as we note from the third evaluation approach or component used in this study (i.e., the IEG interviews with counterparts and partners in government, NGOs and

MTL resources generally more successful and better performing? It will be very helpful for later evaluations if the program used a specific account for each team or new initiative wherein all inputs that are provided by the MTL program (e.g., TLC time, Outward Bound, etc.) are duly recorded. 16 See also Annex 4, which are results for MTL-supported teams only.

17

the donor community), there is certainly more to MTL than improving the quality of multisectoral team outputs.

3.31 Table 4 also shows the size and composition of MTL and non-MTL teams in our survey sample. While the average team size of MTL teams is about the same as non-MTL teams (i.e., an average of 20.1 members for MTL teams vs. 19.9 members for non-MTL teams), there appears to be more Bank staff and fewer client staff who are members of MTL teams than of non-MTL teams. The TTLs of MTL teams also appear to have slightly more years of task manager experience and of Bank experience (i.e., 8.6 years and 4.4 years, respectively) than their non-MTL counterparts (i.e., 6.0 years and 3.9 years, respectively). These team and TTL differences might be explained by the underlying bias of the criteria used to select the FY02-03 MTL teams in our sample. In these initial MTL years, criteria for selection into the program included, inter alia: that the team’s work is an important business priority and that the team had potential for high performance.17

Table 4: Summary statistics team characteristics, inputs and support – percent number of teams or absolute number Team Characteristics and Inputs All teams MTL teams Non-MTL teams

I. Team/Team leader characteristics

a) Team HQ staff 7.16 8.52 4.55 Country Office staff 4.11 4.35 3.50 Client staff 5.46 3.50 9.38 Consultants 3.25 3.73 2.44

b) Team leader Months as TTL in team 16.13 14.68 19.30 TTL experience as TTL (years) 7.70 8.55 6.01 TTL Bank experience (years) 4.24 4.41 3.91

II. Team Inputs/Use of Team/TTL coach 71% 87% 42% Group facilitator 41% 55% 10% Quality Enhancement Reviews (QER) 9% 5% 20% Quality at Entry Assessments (QEA) 17% 13% 25% Quality of Supervision Assessments (QSA) 3% 0% 11% Mentoring program 13% 10% 18% Outward Bound 22% 32% 0% MTL help desk 12% 17% 0% Self help job aids (e.g. CAS, QAG, PRSP websites) 71% 70% 75%

III. Level of team support (1=low, 5=high) From clients 3.94 3.86 4.09 From Bank regional units/vice presidencies 4.18 4.36 3.82 From sectoral units and/or other Bank units 3.62 3.52 3.82

17 See: Quizon, Jaime, “Phase 2 Evaluation of the Multisectoral Team Learning (MTL) Program,” WBI Evaluation Group Paper No:EG03-70, World Bank Institute, Washington, DC, August 2003.

18

INTERVIEWS WITH IN-COUNTRY COUNTERPARTS

3.32 At the behest of the Learning Board in a November 2003 meeting, IEG agreed to visit and interview client counterparts (i.e., relevant high- level government officials, private sector and NGO officers and donor partners) of Bank MTL teams in at least six countries in three regions. The main objective of these interviews is to investigate the nature and extent of Bank MTL teams’ engagement with their main in-country counterparts and partners. In short, does the MTL program result in broader and more responsive Bank team engagement with relevant counterparts and partners?

Methodology

3.33 All Bank operational teams, by their very nature, engage country counterparts at some time and level and to some degree or another. Indeed, there are many types and possible ratings of what are generally regarded in the Bank as “client engagement”. For this study, we used the definition of client engagement as used by Human Resources Leadership and Organization Effectiveness (HRSLO), that is:

• “Client Engagement is the process of interaction with clients and client systems. In the World Bank context, all staff and managers in the Bank have two types of client groups – internal and external. The definition of clients has broadened significantly over the past years – beyond that of sovereign governments to include the poor and disadvantaged, civil society, donors and the private sector.

• Client Engagement relates to all aspects of the Bank’s activities, programs, systems and staffing, and involves the continuous improvement of the World Bank’s interaction with and responsiveness to our clients and client systems. The World Bank needs to understand what the client wants and why, and respond accordingly recognizing that what the client wants may not always be aligned with the Bank’s requirements. The client engagement program aims to create a framework within which a range of activities can focus on improving the Bank’s client responsiveness.

• Emphasis is placed on those skills, values and behaviors that can be directly influenced through targeted activities and actions that can be applied at the individual, work group, and business unit level. All staff and managers are potential targets for the activities to be developed. Initial focus is on operational staff and managers and Country Offices.”

3.34 In light of the above definition of client engagement, the main questions that the IEG client visits and interviews answered are the following: (a) Do the client counterparts of Bank-supported MTL teams go beyond the sovereign governments to include and involve relevant non-government partners? If yes, how and to what degree relative the same MTL-assisted team in the past or to other Bank teams? If no, why? (b) Have these Bank MTL teams’ engagements resulted in their improved interactions with and

19

responsiveness to clients? If so, how and to what extent? (c) How might Bank team engagements with counterparts be improved?18

3.35 The selection of the MTL team counterparts and countries to visit was not random. Because only very few countries could be visited, IEG evaluators requested MTL administrators to provide them with a list of FY02-04 MTL teams which have had some client engagement. Table 5 lists these teams, including some of their features. We note that while this list of 21 teams is not necessarily exhaustive of all past and current MTL teams where there has been direct client engagement, this list includes all, if not nearly all, teams where MTL program inputs directly affected or involved in-country counterparts.19 In all these cases, client counterparts supposedly attended MTL-facilitated events, such as multisectoral workshops, Outward Bound learning encounters, stocktaking seminars, discussion groups and/or similar gatherings. In all, the total number of MTL teams with this level of client engagement is really not very large, particularly considering that there were about 75-80 MTL teams in the FY02-03 period.

Table 5: List of FY02-04 MTL teams that have had direct MTL-led engagement with in-country counterparts/clients.

Region Country of MTL team Years in MTL Outward Bound Cluster Visited Comments

Azerbaijan FY02-03 2 MSP visited Uzbekistan FY02 1 AAA difficult to visit Albania FY02-03 1 PRSC visited Russia FY03-04 1 AAA difficult to visit Moldova FY03-04 MSP too early Ukraine FY04 MSP

ECA

Slovakia FY03 1 MSP little MTL input

Laos FY02-03 CAS visited too early Thailand FY02-03 CAS visited Mongolia FY03 CAS difficult to visit

EAP

Cambodia FY03-04 CAS

Pakistan (AJK) FY02-04 3 MSP difficult to visit Bangladesh HNP FY02-04 MSP SAR Bangladesh CAS FY04 CAS too early

Uganda FY02-03 PRSC visited Nile Basin FY02-04 REG no reply from team coach Senegal FY03-04 1 PRSC visited Cape Verde FY04 PRSC too early Benin FY02-04 PRSC Congo FY02-04 CT too early

AFR

Eritrea FY03-04 MSP visited

Source: MTL Program Administration, February/March 2004.

18 Annex 4 lists an example of the set of guideline questions that accompanied an introductory letter sent to all potential client interviewees prior to IEG’s actual vis its and interviews in their offices. 19 As noted earlier, all Bank operational teams involve direct client engagement at some level. Table 5 lists only those Bank teams which, according to MTL administrators, had what it considers as direct “client engagement”, i.e., MTL-assisted teams that directly involved in-country counterparts at events that were organized, facilitated and/or orchestrated by MTL coaches. MTL administrators last updated this list on April 5, 2004.

20

3.36 Of the 21 MTL teams listed in Table 5, we exempted five teams from being selected for the counterpart interviews because, after consultations with their respective MTL coaches, we found that these teams’ engagements with clients were still at a very early stage and not very likely to have had any meaningful impacts at this time. We excused one team because the provided MTL inputs had only very limited direct engagement with clients. Four other MTL teams were excluded because, if selected, site visits were going to be very difficult to arrange on short notice, on account of either uncertain security, difficulties with arranging meetings with counterparts because of a changed political climate, or extreme weather conditions at the time.

3.37 We eventually selected seven MTL teams for the IEG interviews with these teams’ counterparts. These teams operate in seven different countries in three Bank regions: ECA (2 teams), AFR (3 teams) and EAS (2 teams). IEG visited and interviewed the in-country counterparts and resident Bank MTL team members in these sites.20 Of these visits, one site has not had any MTL-assisted direct client engagement as yet, even though prior to IEG’s scheduled travel to the country, MTL administrators had named it in its original list of program-supported teams which have had direct client engagement.

3.38 Nonetheless, the actual six sites polled are operations centers of very selected MTL Bank teams. All six selected teams have performed extremely well, with two of them having been nominated for the Presidential Award for Excellence and one having actually won this prestigious Bank recognition. While it is difficult to attribute these citations to effects stemming from the MTL program, it is only fair to say that all six teams in our sample benefited greatly from various inputs provided by the program.

3.39 The meetings with MTL team counterparts - government officials, private and NGO officers, donor partners - were arranged with the help of the Bank resident missions in the visited countries. The TTL and members of the MTL-assisted teams first provided the names and addresses of their counterparts and donor partners for interview in each country. Then, a formal letter and some background material explaining the nature of the requested interview were mailed to each person in this list of counterparts and donors. Local resident mission staff finalized meeting arrangements with all those who were available for the interviews. Each interview lasted 45-60 minutes and was conducted face-to-face and often, one-on-one. The sessions made use of local interpreters when necessary.

3.40 IEG interviewed some 60 counterparts and partners (i.e., government officials, NGO officers, and staff of bilateral donor agencies) in all six countries.21 Additionally, we also interviewed several WB resident staff members of the MTL teams, including a few TTLs. The majority of these Bank staff were not interviewed in the Phase 1 and 2 evaluations of the MTL because they reside in the field and are generally difficult to reach. However, for this same reason, they offer excellent perspectives on the nature and

20 Jaime Quizon, a senior evaluation officer in IEG, visited all seven sites and conducted the interviews with MTL Bank team counterparts and with resident WB staff members of the MTL teams. 21 The main questions used in the client interviews are included in Annex 4.

21

extent of the day-to-day engagements between the Bank MTL team and their many local counterparts.

Evaluation Results

3.41 All six MTL teams reviewed have a number of common features and they face similar environments that give them an advantage:22

Bank Teams

3.42 Task team leader are commonly described by MTL team members, donor partners and government/NGO counterparts as being strong team leaders, i.e., frequent adjectives used include: well-selected, accommodating, focused on results, responsive, and inspiring confidence.

3.43 The Bank resident missions are very well regarded in all countries visited. As with the MTL TTLs, the current and former resident country directors or managers are (or were) likewise dynamic, personable and results-oriented. All are (or were) either members of the MTL team themselves or had supported MTL assistance to these teams with few reservations. In several instances, it was the country manager or director who directly campaigned for MTL services for the team.

3.44 As noted earlier, Bank MTL teams are neither unusually larger (i.e., have more team members) nor smaller than their non-MTL counterparts. But, for the MTL teams in our interview sample, the bigger difference perhaps lies in the size of their counterpart teams, with MTL team counterparts being more in number than those of non-MTL teams, in the same country or region, with similar tasks. PRSC teams have the largest number of counterparts, directly involving as many as 200 officials and experts. Because of their size alone, there are clear and strong needs for proper delegation and coordination of activities even among the counterparts alone.

Government officials and donor partners

3.45 Support of respective country governments/ministries for MTL team activities has been and/or remains very strong. Government counterparts perceive their related activities as originating from within themselves (demand-driven) and not from the Bank At the very least, there has always been strong local participation in all MTL-assisted program activities. At best, there has been an enthusiastic sense of government ownership of these events.

3.46 Overall, how do donors and/or counterparts regard their engagement with the Bank in general and with the MTL-assisted Bank team in particular? What has been MTL’s role in shaping these perceptions?

22 Because this report is an assessment of the MTL program and not of the individual Bank teams that participated in the program, we avoid, to the extent possible, any reference to characteristics, behaviors and performances of specific teams.

22

3.47 In all countries, the donors and government/NGO counterparts interviewed note overall satisfaction with the Bank activities in which they were involved. In the majority of cases, the Bank serves as the standard for the country’s relationships with its other donors. Also, the Bank’s engagement with its in-country partners and counterparts has improved significantly since two years ago, especially in three of the seven countries visited. As one interviewee aptly summarizes: “The Bank is fully-respected, less imposing, more transparent, and very inclusive. It is hardly a dogmatic counterpart, where decisions are taken in Washington without local consultations, and more a real partner than a funding agency.”

3.48 How do government officials, civil society participants, and donors describe the specific MTL-supported events they attended? The majority agreed that that these were extremely useful and participative gatherings that raised their own awareness of the major constraints (mainly, political, budgetary and legal) faced by other sectors. The event also allowed them to learn, particularly from other participants, various approaches to dealing with the difficulties in their own work. Others noted that they were forced to think more creatively and holistically, given their better understanding of multisectoral linkages and their accompanying bottlenecks. As one high- level government participant concluded, “Many strategies are holistic in nature. The event allowed us to consider integrated approaches and solutions which we could not consider by ourselves (in the Ministry).”

3.49 In all, the majority of MTL-supported event participants agreed that event activities were arranged well and that their own time and efforts were spent fruitfully despite the large size of the gathering. Preparations that were required prior to these workshops allowed the entire gathering to focus on the key points raised by the attendees. In all, participants in these MTL-assisted events noted “real dialogue”, including on technical matters, taking place and resulting in greater ownership and accountabilities for agreed upon follow-up activities. MTL-assisted events were focused and succeeded at moving from concepts to agreed actions and measurable results. One participant, echoing the replies of several others, thought that what was most important was that the MTL-led event “… provided participants with many useful insights into their own behaviors.”

3.50 Among the non-Bank participants in these MTL-assisted events, government officials appeared to be the most satisfied with the conduct and outcomes of these events.23 As one government official explained. “It (the MTL-led gathering) opened lines of communication across government sectoral agencies and even within our own ministerial hierarchies.” In all countries where the NGO sector participated in the MTL events, government participants claimed to have learned to appreciate more the NGOs’ important role in their own countries. The NGO sector itself, however, was by and large less pleased. They were the least prepared for these gatherings perhaps because, by their own admission, of their more limited knowledge/information and less developed skills with these kinds of high- level engagements. While generally satisfied with the

23 Only two government respondents claimed that the composition of the MTL team was flawed, as these Bank experts knew little of the realities of their countries. Also, at least three government participants interviewed mentioned that the conclusions of these workshops were not very clear. However, the follow-up activities engendered by the MTL gathering allowed them to clarify these ambiguities.

23

composition and conduct of these gatherings, NGO participants had the most to say on how to improve them.24 Several NGO participants noted that the MTL-supported events were too short; much more could be accomplished with more time. However, these same participants realize the difficult coordination problems (i.e., mainly the time and expense) with getting many high- level attendees in the same room over an extended period of time.

3.51 The donors’ views of these MTL-assisted events are similar to the majority views of participants already described. Donors note their greater appreciation as true partners during these events, particularly because of the harmonization and better coordination of sector-based activities among themselves, the more efficient government use of the technical expertise and other resources provided by the donor community, and the larger in-country impacts that these MTL-assisted gatherings allowed them to have. As one donor aptly summarized: “The World Bank plays a unique role in this country. It has brought together the individual programs of each donor under a more coherent structure. The now more regular donor-government meetings that the (MTL) gathering engendered has allowed us to have a seat at the same table and to converse with counterparts in government. This is something we would never have had without our partnership with the World Bank” However, several MTL-organized events developed some problems for certain donors because these events, including before and after the gatherings, drew a large number of experienced government officials (already scarce in many countries) away from their other day-to-day, donor-supported activities.25

3.52 One of the Bank’s main contributions, with assistance from MTL, is with the process in which governments operate.26 Joint learning has taken place and continues to happen, including: more intersectoral dialogue and coordinated activities within government (except in two countries where this was already happening and gaining momentum even before MTL came into the picture27); new Bank/counterpart teaming processes and arrangements for attaining and sharing results which, in at least three instances have been used/copied by counterparts themselves in their own work; new ways of organizing and managing operational teams and large workshops (everyone is heard even with limited time; results oriented; “humanizing” without losing efficiency) despite people having different levels of knowledge (e.g., NGO representatives vs. government

24 Two NGO representatives interviewed suggested that there were no real consultations in these gatherings, as they were not conferred with properly prior to or after the main workshop. They felt that there was insufficient discussion and that decisions were already made prior to these gathering; their token participation was intended only for them to “rubber stamp” the proceedings. In at least two countries, NGO representatives wondered if they will still be consulted in the implementation phase of the agreed activities. 25 In at least one instance, a major bilateral donor complained that despite his/her country’s large annual development aid package, the World Bank, on account of the MTL event, was unfairly getting all the attention and credit from government. The same donor observed that the event did not invite the sectoral experts provided by his/her country. In another instance, several donors noted that the Bank team was too large, although this was in keeping with the growing size of the joint endeavor. 26 As several interviewees put it, it is the entire process of developmental change that the Bank introduces, rather than just the grant/loan from the Bank that is most important and useful. 27 We can also exclude one country where no MTL-sponsored direct engagement within the country has happened as yet.

24

participants) and skills. With these desirable changes having occurred, overall expectations among counterparts have increased as well, and this is now a great challenge in all countries. Can and will the partnership deliver the intended results in a timely and effective manner? Will the Bank/donor/government partnership survive when major difficulties arise in the future?

3.53 It is difficult to attribute a large portion of the major changes that have happened, or are happening, in these country programs to MTL. The government and NGO officials and in-country donors we interviewed maintain that the successes of the MTL-assisted Bank teams lie, by and large, with the relevant Bank country directors/managers, resident missions, TTLs and their team members, who have interacted or are interacting with them very regularly.

3.54 However, all Bank country directors/managers, TTLs and their team members view MTL as an extra and very valuable resource that adds to their limited resources. To them, this is one clear benefit of the learning program and therefore quite welcome. Among the MTL resources received, they value the services of TLCs the most, particularly the coaches’ assistance with the organization and facilitation of meetings/workshops, especially the larger gatherings with government, civil society and donor participants. While the majority of TLCs hardly participate in team activities beyond the large gatherings/workshops that they orchestrate, a few coaches are more engaged than others in the follow-up activities and day-to-day welfare of the MTL teams they mentor.

3.55 Nonetheless, the effects of MTL coaches go beyond the large gatherings or workshops that they direct for the team. The majority of the TTLs and their team members continue to practice the teambuilding practices and facilitation processes from the MTL-organized events in their ongoing and other work. Some TTLs and MTL team members note that several donors, government counterparts and other partners themselves use these new learned techniques in their own other multisectoral, multi-institutional work settings. Also, MTL-facilitated events allow the Bank to have conversations on other issues with government/NGO counterparts and donors, because of the familiarities among the gathered participants that these MTL-organized workshops engender. In this way, MTL-organized events facilitate dialogue on more than just the issues directly relevant to the MTL Bank team.

3.56 As already noted, a certain amount of “learning by doing” has taken place in several MTL-supported in-country events. Government and NGO counterparts in at least three countries note that their participation in MTL-supported activities has taught them a lot about workshop facilitation, particularly the results-orientation and teambuilding processes that they continue to practice. These include: new listening and negotiation techniques; methods for raising awareness of multisector linkages at large gatherings, including the need to work in multisector teams; processes for organizing and facilitating large gatherings towards results; and cooperative procedures for scheduling deadlines and necessary follow-up activities. A lot of mutual trust, including some feel good element of being collectively empowered as part of a working group, is also there, and this is a strong work-motivating factor. As a MTL-event participant noted: “The (MTL) event

25

allowed us to break down barriers across and within ministerial silos; between the ‘old’ and the ‘new’ bureaucrats; between the ‘top’ and the ‘bottom’ of work hierarchies. It allowed us to think differently, more freely and constructively for the cause of the country.”

3.57 Would these benefits have materialized in the absence of MTL? Would the client/donor engagement in these MTL-supported activities, and hence the consequent outcomes, have been the same in the absence of MTL? It certainly is very difficult to describe what would have happened in the absence of MTL inputs; there are not easy comparable in-country experiences that can describe a likely counterfactual. What is more certain are that government/NGO counterparts and in-country donors agree that:

(a) The Bank’s engagement has improved significantly in these countries. They attribute a large part of this marked change to the respective country managers and TTLs whose approaches with dealing with them may be on account of changing Bank wide recruitment policies and in-house training.

(b) The Bank has a very important role to play in these countries, as outside of the government structure itself, the Bank is the only other institution that can encourage, if not cajole, the shift in the way normal development efforts are carried out within the country and refocused towards comprehensive multisectoral approaches to development. Much of the Bank’s work therefore is quite unique and its teams need considerable support to consistently deliver excellence. The Bank also needs to make it easier for lessons learned from operational work to flow to its counterpart teams that need them; to bring to Bank clients and partners not only the most relevant knowledge but also the most useful processes (like MTL) that it has to offer.

(c) There is the need to actively sustain the large inroads at teambuilding already achieved in these countries. In many of the countries visited, the Bank-client teams are now being challenged at the implementation phase of their program/project cycle. Government counterparts, in particular, note their raised expectations, yet certain Bank and donor partner rules (e.g., procurement) are impinging already on the speed in which their own invigorated multisectoral teams are now working. In addition, in several countries, some government, donor and Bank staff have been reshuffled or have left, thereby making it more difficult to sustain the momentum from the original MTL-assisted event. As on donor partner suggested: “Each change in personnel means a relearning exercise that can be expensive for all in-country partners in terms of time and other resources.”

26

4. CONCLUSIONS AND RECOMMENDATIONS

4.1 This Phase 3 study used three separate, albeit complementary, approaches to assess the impacts of the Multisectoral Team Learning (MTL) program, namely: (a) an independent Expert Panel review of the final written products of MTL vs. non-MTL Bank teams; (b) a survey of the immediate supervisors of the task team leaders (TTLs) of both MTL and non-MTL Bank teams; and (c) direct interviews with government /NGO counterparts and donor partners of MTL-assisted Bank teams. The main idea was to get as many relevant and independent viewpoints concerning the on-the-ground impacts of the MTL program. Each of these evaluation approaches have their own strengths and weaknesses. However, because different perspectives are taken into account, the individual findings from each approach, when taken together, should provide a more complete and balanced picture of the real effects of this Bank staff learning program.

4.2 The main purpose of this study was to ascertain whether and to what degree improved teaming and learning practices, processes and tools of the MTL program result in: (a) better quality team processes and outputs; (b) enhanced in-country partnerships and teaming and learning with clients; (c) improved engagements with partners and clients in the delivery and dissemination of team outputs; and (d) increased and sustained use of MTL teambuilding and action learning practices. In general, our main finding is that the MTL program contributes to these desired impacts, albeit in varying degrees.

4.3 Our results show that, in general, MTL improves the overall quality of final Bank CAS, PRSP and AAA documents. However, these quality gains are not very large because of the already high excellence of these Bank products overall. The largest quality gains are for AAA outputs, although CAS and PRSP documents improve as well.

4.4 Being an MTL-assisted Bank team does not necessarily translate to having better team characteristics, behaviors and performances than similar non-MTL Bank teams. However, whether or not certain resources are provided by the MTL program, a Bank multisectoral team’s use of specific key inputs (in particular, team/TTL coach, QEA, QSA, self-help tools) and its level of support from relevant sector and other related Bank units are all positively and highly correlated with having some desirable characteristics, behaviors and performances ratings for a multisectoral team. A team’s use of team meetings/workshops/retreats and/or its attendance of peer gatherings also correlate well with preferred ratings of important team characteristics, behaviors and performances.

4.5 These results suggest that some, and not all, elements of the MTL program have reassuring connections with improved team characteristics, behaviors and performances. Thus, the current plan to scale back on full-service MTL teams and to provide certain selected elements of the MTL program (e.g., TLCs under its “Quick Coach” initiative) to more multisectoral Bank teams in FY05 are steps already in the right direction. The MTL program might want to consider the dissemination and wider use of other inputs

27

(QEA, QSA, self-help tools) that appear to be linked with desired team characteristics, behaviors and performances ratings as well.

4.6 There are not many Bank MTL teams whose in-country government/NGO counterparts have attended MTL-assisted events, such as multisectoral workshops, Outward Bound learning encounters, stocktaking seminars, discussion groups and/or similar gatherings. Thus, there are not very many instances too where Bank team engagements with their clients can be shown to have improved on account of direct MTL assistance. Nonetheless, in the several instances where MTL-orchestrated joint Bank/client engagements have happened, their overall effectiveness and impacts have extended often beyond the organized event s themselves. The largest of these “spillover” benefits from MTL assistance have occurred where large, complex Bank and counterpart multisectoral teams -- such as CAS and PRSP/C teams -- are involved, and in countries that have relatively less developed capacities and/or are moving away from highly centralized governance structures. These are the types of Bank teams that might be targeted for full-service MTL treatment in FY05.

4.7 Among the more significant and longer lasting contributions of these MTL-led events are the results-orientation and facilitation processes, teambuilding techniques, and negotiation practices that this learning program introduces not only to the Bank team, but also to the team’s government and NGO counterparts and donor partners. Several government clients have adopted these “new” methods in their own multisectoral, multi-institutional work. In addition, these MTL events have built mutual trust among participants, thereby facilitating a wider dialogue on more than just the issues directly relevant to the MTL Bank team. MTL-facilitated event s have allowed more Bank-client conversations to happen.

4.8 There are raised expectations among government/NGO counterparts now that better relationships and levels of engagement with the Bank and other donors are in place in these countries. Alongside this, there are ongoing changes in the compositions of Bank and its in-country client teams that are happening. Thus, there is some need to sustain the important gains in multisectoral teambuilding and multi- institutional partnerships that MTL helped engender in these countries.

4.9 MTL-facilitated follow-up activities might be made a regular component of a broader Bank corporate learning strategy focused on “taking the game to a higher level”, that is, not just including MTL coverage of more Bank teams but also of scaling up by inviting more client participation in joint Bank-client learning opportunities. These might entail, inter alia: using the Global Development Learning Network (GDLN) sites or similar media to deliver such learning opportunities more widely; training and/or fielding of region-based TLCs or MTL facilitators;28 broadening clients’ access to Bank self-help websites (e.g., websites on QEA, PRSP/C, CAS, and so on) ; opening more joint Bank/client learning opportunities particularly on other immediately relevant concerns (e.g., ethical standards and boundaries, conflict management/resolution, partnership

28We note from the Phase 2 study that already, some one-third of operational TTLs are located in the resident missions.

28

management, listening skills, consensus building, and so on).29 For Regional Learning Coordinators, closer collaboration with the WBI, who have the Bankwide mandate on client capacity enhancement and who have the experience with worldwide dissemination not only of Bank knowledge products but also of best practices through peer learning, can be very helpful here. Already, WBI (through WBIRCET) has embarked on a country-focused Capacity Enhancement (CE) strategy, starting with 30 priority countries.30 Inclusion of client-engaged MTL activities in this WBI CE strategy will well complement the Institute’s own work with raising client awareness and with disseminating cutting-edge knowledge and skills among clients.

4.10 The above-mentioned recommendation, on having more joint client-staff learning opportunities, needs further study if this is to be carried out successfully in the near term. What lessons can be drawn from past and ongoing joint client-staff learning opportunities, within and outside of MTL? Where are the largest client-staff demands for training (e.g., region vs. HQ, in what sectoral areas, and so on)? What types of Bank- led learning activities are most likely to produce the largest returns in a joint client-staff environment? How might the funding issues of joint learning opportunities be addressed? These are just some questions that deserve further research attention in the year ahead if other joint client-staff learning opportunities are to be pursued effectively.

29 These are some of the concerns that several government/NGO/donor participants of MTL-organized events mentioned as important to their own work. 30 MTL collaboration with WBI might begin with some, if not all, of the 30 WBI priority countries, as these favored countries were themselves chosen in consultation with the Regions.

29

ANNEXES

30

31

ANNEX 1 : NUMBER OF MTL TEAMS FY03-Q4 AND FY04 FY04

Region Project Name Country Theme Team Leader FY03 Q1 Q2 Q3 Q4

AFR CAS/CT South Africa CAS Fayez S Osman X X X Community and Local Gov't Dvlpmt. Nigeria MSP Talib Esmail C Country Team Pilot (ongoing) Madagascar CT Hafez Ghanem X X X X X CT Support/ CAS Chad CT Ali Mahmoud Khadr C CT Support/ CAS/2 PRSCs Uganda/Tanzania CT Judy O'Connor N X X X

Decentralization Sierra Leone PRSP Yongmei Zhou X X Diversification Zambia MSP Constantine Chikosi C DR Congo DR Congo CT Emanuel Mbi X X X X X HD PRSC Cluster Regional AAA Trina Haque N X X X

HIV/AIDS Eritrea MSP Eva Jarawan/Albertus Voetberg X X X X X HIV/AIDS Kenya MSP Albertus Voetberg X X X X X HIV/AIDS Gambia MSP John May N X HIV/AIDS Malawi MSP Christine Kimes N

Nile Basin Initiative Nile Countries REG David R.C. Grey X X X X X Pharmaceutical South Africa MSP Dave DeGroot C PRSC Benin PRSC Antoinette Sayeh X X X X X

PRSC Burkina Faso PRSC Jan Walliser C PRSC Cabe Verde PRSC Julie Van Domelene N X X X PRSC Ethiopia PRSC Ishac Diwan/Gaiv Tata X X X X X PRSC Senegal PRSC Nancy Benjamin X X X X X

PRSC2 Uganda PRSC Satu/Praud Murphy C PRSP/CAS Cote D'Ivoire CAS Mamadou Dia C Rehabilitation Angola MSP Adbelmoula M. Ghzala X X X X X

EAP Cambodia Indonesia Exchange Indonesia/Cambodia CAS Scott Guggenheim N X X C CAS Cambodia CAS Steve Schonberger N X C CAS Laos CAS Ian Porter/Zahar Ahmed X X X X X CAS Mongolia CAS Zafar Ahmed X X X X X

CAS Timor-Leste CAS Elizabeth Huybens N X C CAS Implementation Thailand CAS Ian Porter X X X X X

(Annex 1 continues on the next page.)

32

(Annex 1 continued.) FY04

Region Project Name Country Theme Team Leader FY03 Q1 Q2 Q3 Q4

CAS Local Services Platform Indonesia CAS Joel Hellman N X C CDD Learning Group Philippines MSP Bhuvan Bhatnagar C Country Office Team Philippines CT Robert V. Pulley (Maribelle Zonaga) X X X X X Decentralization Study Philippines CT Amit Mukherjee C

PRSP/Poverty Assessment Team Indonesia AAA Jehan Arulpragasam N X X X ECA ECA Quality Work Regional REG Richard Westin N X X X CAS Impementation Turkey CAS Marie-Helene Bricknell N X X

Fiduciary Safeguards Regional REG Laura Tuck/Alain Colliou N X X X Gender Focal Points Regional PRSC Pierella Paci N X X X Integrating PRSP into CAS Moldova CAS Jariya Hoffman N X X Joint Program and Portfolio Review Turkey AAA Sally Zeijlon C

Koykent Development Project Turkey MSP Sudipto Sarkar C Poverty Monitoring Eval. and Policy Bulgaria AAA Boryana Gotcheva X X X X X Poverty Work Russia AAA Radwan Shaban X X X X X PRSC Albania PRSC Philip Goldman / Mansour Farsad C

Results Based CAS Armenia CAS Julia Bucknell N X X Rural Community Investment Project Azerbaijan MSP Julien Lampietti/Julia Bucknall C School Health PHRD Grant Tajikistan MSP Yang-Ro Yoon C

Social Development Fund Slovakia MSP Dena Ringold C Social Risk Mitigation Ukraine PAL MSP Galina Sotirova N X X X Youth Exclusion/Moldova Moldova MSP Carine Clert C

LCR CAS Mexico CAS Connie Luff N X C Country Team Pilot (ongoing) Brazil CT Vinod Thomas C Decentralization Peru Peru CT Conny Luff/Oscar Ovalle N X X Education for All 1 Bolivia/Hond./Nic/Guyana REG Marito H. Garcia C

Education for All 2 Bolivia/Hond./Nic/Guyana REG Marito H. Garcia C Education for All 3 Bolivia/Hond./Nic/Guyana REG Marito H. Garcia C GUAPA (Guat. Pov Ass)CAS Guatemala CAS Eduardo Somensatto / Kathy Lindert C HD Brazil CT Kathy Lindert C

(Annex 1 continues on the next page.)

33

(Annex 1 continued.) FY04

Region Project Name Country Theme Team Leader FY03 Q1 Q2 Q3 Q4

LCR Lima Poverty Strategy Peru AAA David Sislen X X X X X Racial Inequality in Brazil Brazil AAA Josefina Stubbs C Rural Infrastructure Guatemala AAA Jennifer Sara N X X Rural Infrastructure Chile AAA Jennifer Sara C

Youth at Risk Multi-Nat'l Carib. Proj. AAA Wendy Cunningham C MNA CAS/CT Algeria CAS Ted Ahlers X X X X X CAS Tunesia CAS Ted Ahlers N X X X

Country Team Jordan CT Joe Saba X X X X X Country Team/CAS Yemen, Republic CT Robert Hindle/John Macgregor X X X X X Governance Team Regional REG Chuck Humphreys C PER Iran AAA Charles Humphreys C

Port Cities Devlpt Pgm (PCDP) Yemen, Republic MSP Omar Razzaz/Steve Karam C Programmatic ESW Regional REG Nadereh Chamlou C Rural Alliance Project Regional REG Douglas Lister/Matthias Grueninger C

SAR AJK-Community-Distr.Infra Services Regional/Pakistan MSP Julie G. Viloria-Williams X X X C CAS Bangladesh CAS Fred Temple/Chingboon Lee/J.Sinclair X X X X X Country Team/CAS Sri Lanka CT/CAS Peter Harold C

HNP Team - Health, Nutrition & Pop. Bangladesh MSP Birte Holm-Sorensen X X X X X Social Investment Bangladesh MSP Wahida Huq N X X South Asia Decentralization Team Regional AAA L. Constantino/C. Bernard/T. Sharafudeen X X X X X

AAA Analytical and Advisory Activities Total AAA 12 X = Ongoing Teams per Quarter: 23 23 38 40 40 CAS Country Assistance Strategy Total CAS 17 N = New Teams: 0 14 6 1 1 CT Country Team Effectiveness Total CT 11 C = Closed Teams per Quarter: 30 0 0 5 1 MSP Multi-Sector Lending Project Total MSP 19 Quarterly Grand Totals: 53 37 44 46 42

PRSC Poverty Reduction Support Credit Total PRSC 8 REG Regional Projects Total REG 9

34

ANNEX 2: RATINGS FOR THE SEVEN PRSC, AAA AND CAS TASKS

PRSC Tasks

1. Consistency of the program objectives with the Country's key development priorities and with CAS.

2. Level of borrower ownership, commitment and participation to go through with the program.

3. Clarity of development objectives.

4. Quality of economic analysis and rationale underpinning the program.

5. Technical soundness of the program.

6. Appropriateness and realism of program conditionality. Are conditions of effectivenes s appropriate?

7. Quality, depth and appropriateness of the poverty and social analysis.

8. Quality of gender analysis (optional).

9. Stakeholder participation/involvement.

10. Assessment of environmental impact and risks.

11. Adequacy and appropriateness for financial management and procurement arrangements.

12. Assessment of institutional and human capacity and TA arrangements to implement the program.

13. Readiness of program for implementation.

14. Appropriateness of the program to combat corruption.

15. Quality of Monitoring and Evaluation arrangements.

16. Clarity and candor of risk assessment and quality of risk mitigating measures.

17. Appropriate partnership arrangements with other donors (multilateral & bilateral).

AAA Tasks

1. Relevance of task to initiating or moving forward the policy dialogue with country.

2. Relevance of task as underpinning to strategy development [next CAS or PRSP].

3. Relevance of task as underpinning to lending operations.

4. Relevance of task as instrument for capacity -building and knowledge sharing.

5. Consistency of task objectives with the Bank's sector/thematic strategy and with the MDGs.

6. Appropriateness of coverage/scope in view of task objectives.

7. Clarity of identification of critical issues.

8. Quality of the analysis [analytical / methodological rigor].

9. Quality of the empirical evidence [from both within and outside the Bank].

10. Appropriate consideration of / sensitivity to poverty reduction dimension.

11. Appropriate consideration of / sensitivity to gender dimension.

12. Are conclusions and policy recommendations specific and consistent with the analysis?.

13. Are policy recommendations adequately selective and prioritized?

14. Are policy recommendations realistic in the light of social, political and administrative realities?

15. Given the substance and presentation of the report, it is likely to have an impact upon the client?

16. Given the substance and presentation of the report, it is likely to have an impact upon the Bank?

(Annex A continues on next page.)

35

(Annex A continued.)

CAS Tasks

1. Quality of description of political, economic and social context.

2. Quality of diagnosis of poverty issues [profile, trends and analysis of policy issues].

3. Adequacy of description of recent economic developments and government policy reforms

4. Quality of diagnosis of governance issues [incl. governance and public financial accountability].

5. Quality of diagnosis of other cross-cutting issues [gender, environment, private and financial sectors].

6. Adequacy of discussion of the country's own development strategy and program.

7. Quality of self-evaluation of performance under the previous CAS; indications of lessons learned.

8. Extent of participatory process in CAS preparation [client government and civil society; other donors].

9. Adequacy of discussion of non-lending services.

10. Justification for proposed lending levels [including triggers for low, base and high cases].

11. Thoroughness of analysis of risks.

12. Quality of Annex B9 [the CAS Program Matrix].

13. Completeness of "Key Economic Indicators" annex and "Key Exposure Indicators" annex.

36

ANNEX 3: SUPERVISORS’ QUESTIONNAIRE

Supervisor Name:______________________________________________

TTL Name: ___________________________________________________

Task Team Name: _____________________________________________

How long have you known of the activities of this task team? Since the team’s inception ____________ or __________ months

Taking into account all the constraints that may affect a team’s performance, how would you rate this task team relative to other multi-sectoral task teams in your work unit in the following areas: (Select one answer on the scale where 1=Much Worse, 2=Worse, 3=Somewhat Worse, 4=Same, 5=Somewhat Better, 6=Better and 7=Much Better)

Medium

� �� ��z { a) team spirit/cohesion/sense of purpose ¡ ¡ ¡ ¡ ¡ ¡ ¡ b) quality of team output ¡ ¡ ¡ ¡ ¡ ¡ ¡ c) engagement of clients, such as enhanced partnerships and

effective teaming and team learning with clients ¡ ¡ ¡ ¡ ¡ ¡ ¡

d) timeliness of team outputs/overall progress ¡ ¡ ¡ ¡ ¡ ¡ ¡ e) learning practices as a team ¡ ¡ ¡ ¡ ¡ ¡ ¡ f) appropriateness of team size and composition (mix of skills) ¡ ¡ ¡ ¡ ¡ ¡ ¡ g) effective team leadership ¡ ¡ ¡ ¡ ¡ ¡ ¡ h) cost effectiveness of team’s outputs ¡ ¡ ¡ ¡ ¡ ¡ ¡ i) effectiveness of processes and delivery of team outputs ¡ ¡ ¡ ¡ ¡ ¡ ¡

Relative to other multi-sectoral task teams in your work unit, to what degree does (or did, if the team is already disbanded) the team have: (Select one answer on the scale where 1=Much Worse, 2=Worse, 3=Somewhat Worse, 4=Same, 5=Somewhat Better, 6=Better and 7=Much Better)

Medium

� � �� �z { a) support (or buy-in) from relevant client/ministries ¡ ¡ ¡ ¡ ¡ ¡ ¡

b) support (or buy-in) from relevant Bank country/regional units ¡ ¡ ¡ ¡ ¡ ¡ ¡ c) support (or buy-in) from relevant sectoral and/or other Bank units ¡ ¡ ¡ ¡ ¡ ¡ ¡

To what degree did exogenous factors (e.g., in-country conditions, within Bank constraints, and others) affect the overall performance of the team (Circle only one answer.)

Significant negative effect

No effect Significant positive effect

-3 -2 -1 0 1 2 3

Much

Worse

Much

Better

Much

Worse

Much

Better

37

ANNEX 4: CONTEXT OF THE VISIT OF JAIME QUIZON, SR. EVALUATION OFFICER, WBI EVALUATION GROUP

The World Bank has a program called the Multisectoral Team Learning (MTL) Program that aims at improving the working relationships and overall engagement of World Bank (WB) teams with their Uganda team counterparts like yourself. My assignment is to evaluate this program to see whether it is delivering what it is supposed to and to outline ways of improving its features. Your time and effort is greatly appreciated. Over the past two years or so, the WB team responsible for the monitoring the preparation of the Uganda PRSC has been a beneficiary of this in-house MTL program. I would like to ask you your candid opinion regarding: (a) your relationship with the aforementioned team and (b) your assessment of this team’s interactions/engagement with you and your colleagues involved with the PRSC. Questions:

• What has been your involvement/relationship with the WB staff/client team responsible for monitoring the preparation of the PRSC? Do you consider yourself a member (or partner?) of this Bank MTL-PRSC team or do you see yourself as outside this team? To what degree were you engaged in this team? In what capacity? Can you cite specific examples of engagement with members of this team?

• What elements of your engagement with (or participation in) the WB staff/client team did you find most useful? least useful?

• Other than this specific Bank team, have you had any dealings with other WB teams in the past? How would you rate or assess the nature and quality of your engagement with the WB MTL-PRSC team relative to these other WB teams? Please elaborate. (Note that if comparisons with other WB teams are not relevant, then look for comparisons with other non-Bank multisectoral teams.) How would you rate or assess the overall performance of the Bank MTL-PRSC team relative to other WB teams (or other non-Bank multisectoral teams of which you are aware)?

Can you suggest ways that WB multisectoral teams can improve further their engagement with counterparts like yourself? their overall effectiveness, impact and performance?

38

ANNNEX 5: CORRELATION OF TTLS’ RATINGS OF TEAM CHARACTERISTICS, INPUTS AND SUPPORT VS. SUPERVISORS’ RATINGS OF PROCESS AND OUTCOME INDICATORS - MTL TEAMS ONLYA

Supervisors’ Ratings of Team Behavior and Performance:

TTL Reports on Team Characteristics, Inputs and Support Received: Overall

Team spirit/cohesion/

sense of purpose

Quality of team outputs

Engagement of clients

Timeliness of team outputs/

overall progress

Learning practices as a team

Appropriateness of team size

and composition

Effective team

leadership

Cost effectiveness

of team outputs

Effectiveness of processes and delivery

of team outputs

I. Team/Team Leader characteristics

a) Team

Being an MTL team

Number of team

Workshops/retreats + + + + + Tours/missions

Assessments

Meetings +

Peer gatherings + + + + + + Number of staff in team

Team members - - - - - - - -

HQ staff - - - - -

Country Office staff + - - - - Client staff - - - - - - -

Consultants - - - + - - -

b) Team leader Months as TTL of team - - -

TTL experience (years) - - - - - -

TTL Bank experience (years) - - - - -

II. Team inputs/Use of*

Team/TTL coach + + + + + + + +

Group facilitator +

Quality Enhancement Reviews (QER) Quality at Entry Assessments (QEA)

Quality of Supervision Assessments (QSA)

Mentoring program + + + Outward Bound +

MTL help desk +

Self help job aids (e.g CAS, QAG, PRSP websites)

+ + + + + +

III. Level of team support

From clients - -

From Bank regional units/vice presidencies From sectoral units and/or other Bank units + + +

a “+” equals correlation greater than 0.33, “-“ equals correlation less than –0.33. b Correlation between the supervisor outcomes indicators and a dummy variables indicating using the input (1) and not using the input