center for education innovations p.o. box 15593-00509 … · combines approaches in a new way—cei...
TRANSCRIPT
TRAINING REPORT
EVALUATION FROM START-UP TO SCALE
9TH FEBRUARY 2015
CENTER FOR EDUCATION INNOVATIONS
P.O. BOX 15593-00509
NAIROBI
Prepared by:
ii
ACKNOWLEDGEMENT
This capacity building training report was developed through the effort of a team from
Center for Education Innovations (CEI) and facilitated by Shape Afrika.
The consultant, Shape Afrika sincerely acknowledges the support and contributions of
departments and individuals who made the training and the development of this report
possible.
Special acknowledgments to:
1. Dr. Gordon Carver; Project Director, Caroline Jordan; Project Manager, Namulanta
Kombo; Project Officer and Events Manager and Esther Sifuma; Project Officer for their
leadership and logistical support in the implementation of the capacity building
workshop.
2. Prof. David Macharia and Dr. Kinyanjui Nganga, the lead trainers, for effective
facilitation of the training program.
3. The training participants for their active participation during the training and without
whom the training exercise would not have taken place.
4. The various organizations and institutes that allowed their staff members to participate
in the training program.
1
TABLE OF CONTENTS CONTENTS PAGE
ACKNOWLEDGEMENT ............................................................................................................ 2
TABLE OF CONTENTS ............................................................................................................ 1
ABBREVIATIONS ..................................................................................................................... 3
EXECUTIVE SUMMARY ........................................................................................................... 4
CHAPTER ONE: INTRODUCTION ............................................................................................ 5
1.1 Introduction of Chapter One ...................................................................................................... 5
1.2 Background of the Training ....................................................................................................... 5
1.3 Objectives of the Training .......................................................................................................... 6
1.4 Significance of the Training ....................................................................................................... 6
1.5 Limitations of the Training ......................................................................................................... 6
1.6 Organization of the Training Report ......................................................................................... 7
CHAPTER TWO: ORGANIZATIONAL BACKGROUND ........................................................... 8
2.1 Introduction of Chapter Two ...................................................................................................... 8
2.2 Organizational Historical Background ..................................................................................... 8
2.3 Organizational Vision ................................................................................................................. 9
2.4 Organizational Mission ............................................................................................................... 9
2.5 Organizational Approach ......................................................................................................... 10
2.6 Organizational Network ............................................................................................................ 12
CHAPTER THREE: TRAINING METHODOLOGY ...................................................................14
3.1 Introduction of Chapter Three ................................................................................................. 14
3.2 Training Approach .................................................................................................................... 14
3.3 Training Participants ................................................................................................................. 15
3.4 Training Duration ...................................................................................................................... 15
2
3.5 Training Modules ...................................................................................................................... 15
3.6 Training Evaluation ................................................................................................................... 17
CHAPTER FOUR: TRAINING FINDINGS AND ANALYSIS .....................................................18
4.1 Introduction of Chapter Four ................................................................................................... 18
4.2 Findings from the Training ....................................................................................................... 18
4.3 Analysis of the Training ............................................................................................................ 20
CHAPTER FIVE: CONCLUSIONS AND RECOMMENDATIONS .............................................23
5.1 Introduction of Chapter Five .................................................................................................... 23
5.2 Conclusions on the Training .................................................................................................... 23
5.3 Recommendations .................................................................................................................... 25
APPENDICES...........................................................................................................................28
(i) Appendix I: List of Participants ............................................................................................... 28
(ii) Appendix II: Training Program ................................................................................................ 30
(iii) Appendix III: Feedback Form .................................................................................................. 32
3
ABBREVIATIONS
CEI Center for Education Innovations
CHMI Center for Health Market Innovations
DFID Department for International Development
ECD Early Childhood Development
LFA Log Frame Approach
M&E Monitoring and Evaluation
NGO Non-Governmental Organization
OVI Objectively Verifiable Indicator
R4D Results for Development
SGA Skills Gaps Analysis
ToR Terms of Reference
4
EXECUTIVE SUMMARY
Center for Education Innovations (CEI) embarked on training program implementers
and associates across the ECD (Early Childhood Development) and Skills education
sectors on project/program evaluation. The training was expected to provide
participants with practical guidance for understanding evaluation and its application in
respective education programs.
Forty four participants were targeted in a two-days training conducted on 3rd and 4th
February 2015 in Nairobi, Kenya. The training approach was participatory through
analysis of case studies in focus group discussions with the aid of PowerPoint
presentations and flip charts to assess knowledge acquisition. Under the training theme
of “Evaluation from Start-up to Scale”, participants were trained on the following six
modules: introduction to evaluation; evaluation stakeholders; evaluation indicators;
evaluation frameworks; evaluation report; and evaluating for program results. Learning
materials and participation certificates were given to participants after the training. In
addition, participants were given an opportunity to evaluate the effectiveness of the
training by filling a feedback form.
Findings and analysis of the training indicated that participants appreciated the concept
of evaluation and were able to link evaluation with performance. As such, participants
appreciated evaluation as an important requirement for project/program performance. In
addition, participants enhanced their skills on designing evaluation indicators and
evaluation frameworks in a manner that can guide the performance of projects/program.
Moreover, participants understood the concept of conducting an evaluation including
writing and presenting evaluation reports to the satisfaction of stakeholders.
Based on the findings and analysis of the training, it was recommended that more time
be allocated for similar trainings in future. It was also recommended that participants be
trained in project management and in M&E and projects/programs be assisted in
conducting evaluations. In addition, it was recommended that a more thorough skills
gaps analysis be done prior to implementing such trainings to profile respondents’
capacity.
5
CHAPTER ONE
INTRODUCTION
1.1 Introduction of Chapter One
This chapter examines the background of the training; objectives of the training;
significance of the training; limitations of the training; and organization of the training
report.
1.2 Background of the Training
Center for Education Innovations (CEI) seeks to identify, analyse and connect
innovative education programs globally. As part of its global network, CEI has a fully
established office in Kenya – GEMS Education Solutions, that is serving the East
African region. Based on this mandate, CEI organizes capacity building events for
innovative education programs in the region that have been profiled in the CEI website
as part of the CEI network.
These programs range from small to large for-profit start-ups and small NGOs. For
programs to be profiled on the CEI network, it is required that they demonstrate
existence of some form of Monitoring and Evaluation (M&E) in their organization as well
as education innovation.
The training participants were program implementers and associates across the ECD
(Early Childhood Development) and Skills education sectors. The reason for selecting
these two thematic areas is that they are a focus of the work at CEI and the workshop
built upon Thematic Forums held last year for Skills and ECD implementers. The
workshop was expected to provide participants with practical guidance for
understanding evaluation and its application in respective education programs.
6
1.3 Objectives of the Training
The specific objectives of the training were:-
i. Help participants appreciate the purpose of project/program evaluation and link
evaluation to overall project/program performance to the satisfaction of all
stakeholders;
ii. Help participants conduct project/program evaluation using standard tools,
frameworks and objectively verifiable indicators;
iii. Help participants to write a standard evaluation report and effectively present
findings to key stakeholders for continuity of the project/program.
1.4 Significance of the Training
Training on “Evaluation from Start-up to Scale” was expected to help participants to
conduct evaluation and write a standard evaluation report and present the same in a
convincing manner to stakeholders. In addition, the training was expected to help
participants to design evaluation indicators that can access the progress of the
projects/programs. Further, the training was expected to equip the participants with
skills to design evaluation frameworks that would not only summarize projects/programs
progress but also help project/program implementers to be specific in identifying
project/program direction in respect to desired project/program results while designing
and planning for project/program implementation. The training was also expected to
help participants to appreciate and undertake projects/programs stakeholders analysis.
1.5 Limitations of the Training
The duration of the training, two-days, was relatively short in respect to anticipated
objectives of the capacity building exercise especially since the training methodology
7
included the use of interactive sessions by participants. The trainers, however, made
the most of the duration allocated by triangulating training methodology with the help of
training aids like LCD projector, flip charts and focus group discussions. In addition, the
learners’ educational level and M&E skills varied significantly which in turn limited the
level of engagement in the training. The trainers, however, implemented the exercise in
a manner that accommodated the majority of the participants comfortably and the
participants were able to learn from one another during the focus group discussions.
1.6 Organization of the training report
This report is organized in five chapters. Chapter one of this report is an introduction
and it focuses on the background of the training; objectives of the training; significance
of the training; limitations of the training; and organization of the training report. Chapter
two of this report is on organizational background and it focuses on organizational
historical background; organizational vision; organizational mission; organizational
approach; and organizational network. Chapter three of this chapter is on training
methodology and it focuses on the training approach; training participants; training
duration; training modules; and training evaluation. Chapter four of this report focuses
on findings from the training; and analysis of the training based on the training
objectives. Chapter five of this report focuses on training conclusions and
recommendations. The list of participants, participants’ feedback form and the training
program are appended in the last section of this report.
8
CHAPTER TWO
ORGANIZATIONAL BACKGROUND
2.1 Introduction of Chapter Two
This chapter examines the organizational historical background; organizational vision;
organizational mission; organizational approach; and organizational network.
2.2 Organizational Historical Background
Center for Education Innovations (CEI) is a go-to resource on education innovations in
low-income communities; providing data, analysis and connections as a global public
good. Launched in 2013, CEI is an initiative of Results for Development (R4D) and
modelled on the sister program Center for Health Market Innovations (CHMI). CEI aims
to identify, analyse, and connect innovative education programs in developing countries
so as to increase access to quality education and improve learning outcomes,
especially for the poorest and most vulnerable.
CEI pursues this goal using two mutually-reinforcing mechanisms: a virtual platform and
a network of partners that drive in-country activities. GEMS Education Solutions
implement CEI in East Africa, currently covering Kenya and Uganda. By mapping the
education landscape, CEI East Africa has actively engaged with implementers, funders,
policymakers and researchers in the region and developed analysis tools and connect
opportunities meeting stakeholder needs. Thematic Forums have been held and what
works syntheses produced in select thematic areas of ECD and Skills. This Workshop
builds upon the connections established and knowledge shared to deliver capacity
building to ECD and Skills implementers.
9
2.3 Organizational Vision
CEI promotes programs, policies, and practices that increase access to quality,
affordable, and equitable education for the world’s poor.
The vision of CEI is for education systems around the world to capitalize on innovation
so as to increase access to quality education and improve learning outcomes,
especially for the poorest and most vulnerable.
2.3.1 Why focus on education innovations?
The development of CEI was based on the understanding that there is an incredible
amount of innovation and activity in education, but it is not systematically documented
and there is little evidence about the most effective ways to improve access to quality
education for the poor. CEI was developed in order to address this gap.
2.3.2 What makes a program innovative?
CEI realize that effective education systems are complex. They include delivery, but
also policies and financing mechanisms that foster equity and quality, as well as support
products and services for schools, teachers, and students. When a program for the poor
addresses any one of these system components using approaches that are new—or
combines approaches in a new way—CEI consider it innovative. CEI see innovation as
relative: what is commonplace in one community may not be in another; an application
of an existing approach in a new place or sector is innovative.
2.4 Organizational Mission
CEI increases access to quality education for the poor by identifying, analyzing, and
connecting non-state education innovations.
10
2.5 Organizational Approach
CEI works to create impact—expanded access to quality, affordable education for the
poor—by providing information, analysis, and connections for program managers,
funders, researchers, and policymakers who work to enable the scale-up, replication
and improvement of promising innovations in education.
2.5.1 Identify
CEI identifies and profiles innovative education programs, building a global network of
innovators. CEI team members around the world conduct interviews with every program
they profile so they can fully understand the model, the challenges it faces, and its plans
for the future.
The CEI profiles provide information on the program’s approach, target beneficiaries,
financing, scale, monitoring and evaluation strategy, results achieved and external
evidence
2.5.2 Analyze
The CEI website includes the Research & Evidence Library, a searchable database of
over 600 analytical products about education in developing countries. The Library
includes country landscapes, evaluations, case studies, policy documents, meta-
analyses, global initiative reports, and more.
With over 500 entries in the Programs Database, CEI is now beginning to focus on
using the database as a starting point for understanding and sharing lessons about
what works in education. Throughout 2014, CEI produced topic and country briefs that
will surface common themes and developments from an analysis of CEI data, on top of
existing literature and available evidence.
11
CEI works with global partners such as UNICEF to better understand what types of
innovative programs and interventions have the greatest potential for scalability and
impact. CEI also works with partners to identify programs interested in formative
evaluation and works with them to identify challenges, strategize about solutions,
develop monitoring and evaluation capacity, and ultimately share lessons about their
experience.
2.5.3 Connect
The CEI website enables users to directly contact managers of innovative programs,
enabling direct engagement with a diverse community of innovators. CEI hosts
customized virtual collaboration spaces for various groups involved in education
innovation, including the Girls’ Education Challenge (GEC), the Partnership to
Strengthen Innovation and Practice in Secondary Education (PSIPSE) and STIR
Education.
CEI’s most critical connections are facilitated by our network of regional partners
working select geographies to foster connections and learning among innovative
programs and link them to funding, operational and research opportunities. Read more
about CEI’s regional partners here.
Recognizing the value partners can offer high-potential education programs, CEI
collaborates with a wide range of organizations seeking to scale up and improve
education innovations by connecting them with essential financial, operational,
technical, and advisory support.
CEI builds upon the example of the Center for Health Market Innovations (CHMI), a
companion R4D initiative that is now the world’s largest freely accessible information
resource of innovations in the non-state health sector. CEI is funded by the United
Kingdom’s Department for International Development (DFID) and UNICEF.
12
2.6 Organizational Network
CEI is coordinated by the Results for Development Institute (R4D) and relies on a broad
network of regional partners and global collaborators.
2.6.1 Regional Partners
To foster the uptake of promising education innovations, country-based activities that
complement the online platform are driven by both the CEI team at R4D and a network
of CEI regional partners:
i. INDIA | Catalyst Management Services
ii. EAST AFRICA| GEMS Education Solutions
iii. SOUTH AFRICA | University of Cape Town, Graduate School of Business,
Bertha Centre for Social Innovation and Entrepreneurship
iv. NIGERIA | TEP Centre
These organizations represent CEI in their country and region, carrying out a range of
activities such as convening innovators with high‐potential business models, brokering
relationships between innovators and policymakers to initiate cooperation between the
public and private sectors, and building relationships between innovators and donors,
investors, and others that can help them grow and scale‐up their activities.
2.6.2 Global Collaborators
For profiled programs, CEI provides access to forums, competitions, fundraising
platforms, and networks run by our Global Collaborators shown below. CEI provides
Global Collaborators with a pipeline of excellent candidates for these opportunities.
Collaborators often search CEI’s database to identify candidates. CEI may share
relevant programs that meet the nomination criteria thematically, geographically, or are
reporting specific results. It is important for programs to keep their profiles up to date, so
13
that they can be considered for potential opportunities. Global collaborators include:
UNICEF; GLOBAL GIVING; OECD; STIR EDUCATION; GIRLS’ EDUCATION
CHALLENGE; UKAID; TONIC; ACUMEN; IDEAS; WISE; AND SKOLL FOUNDATION.
14
CHAPTER THREE
TRAINING METHODOLOGY
3.1 Introduction of Chapter Three
This chapter examines the training approach; training participants; training duration;
training modules; and training evaluation.
3.2 Training Approach
The scope of the training as well as the Terms of Reference (ToR) for the training was
deliberated on by a project team comprising officials from CEI and senior trainers from
Shape Afrika, an organizational research and corporate training private firm. The
training methodology, objectives, modules and the program of the training were all duly
agreed upon by the two parties prior to preparation of the training materials.
Training on “Evaluation from Start-Up to Scale” was facilitated by two senior trainers
from Shape Afrika: Prof. David Macharia and Dr. Kinyanjui Nganga, with practical
expertise and experience in capacity building in M&E and project management. The
training was done in accordance with the norms of adult learning through interactive
sessions and focus group discussions with the aid of flip charts and LCD projector.
Participants, therefore, would be given space not only to clarify issues during the
training but also to learn from one another through focus group discussions to ensure
the acquisition of actual practical skills on the subject matter under examination.
15
3.3 Training Participants
There were forty four (44) participants in the training. These participants were program
implementers and associates across the ECD (Early Childhood) and Skills education
sectors who were expected to use the skills acquired from the training in project
management and specifically in project evaluation. The list of the participants and the
respective organizations where the participants work is appended in this report as
Appendix i.
3.4 Training Duration
The training was conducted in two working days on Tuesday 3rd and Wednesday 4th
February 2015 from 8.00am to 4.30pm on both days. The training program is appended
as Appendix ii in this report.
3.5 Training Modules
The modules that were delivered in the training as well as the learning objectives and
learning outcomes for each of the training days are as shown in Table 3.1.
Table 3.1: Training Modules
(i) Modules for day I
Module Learning Objectives Learning Outcomes
Module I
Introduction to
Evaluation
Evaluation as a key phase of the project life cycle
Purpose of evaluation
Self-efficacy in evaluation
Components of evaluation
Appreciation of evaluation as a prerequisite of performance
Taking personal responsibility for projects / programs outcomes
Appreciation of the concept of evaluation and various
16
Types of program evaluations
views about evaluation
Module II
Evaluation
stakeholders
Evaluation ethics
Identifying evaluation stakeholders
Stakeholders needs analysis (cost, time, program efficiency, program impact etc)
Integrity in programs implementation
Appreciation that programs comprise various stakeholders with dynamic interests that must be satisfied – hence the focus of the evaluator should not be her/himself but program stakeholders
Module III
Evaluation
Indicators
Purpose of evaluation indicators
Designing evaluation indicators
Critiquing evaluation indicators
Participants should appreciate the importance of evaluation indicators
Learners should be able to design Key Performance Indicators (KPIs)
Learners should allow other evaluators to critique whether the indicators they have developed meet desired project/program objectives
(ii) Modules for day II
Module Learning Objectives Learning Outcomes
Module IV
Evaluation
Frameworks
Purpose of evaluation frameworks
Types of frameworks used in evaluation
Contextualizing evaluation frameworks
With or without a full evaluation report, participants should be able to summarize their evaluation findings in recognized evaluation frameworks
Appreciation and acceptance of various evaluation frameworks that are used in projects/programs
Learners should be able to contextualize existing evaluation frameworks in their program setting
17
Module V
Evaluation report
Conducting a program evaluation
Designing program evaluation instruments
Writing an evaluation report
Skills to conduct an evaluation cognizant of multiple realities
Ability to design instruments that are valid and reliable for evaluation
Ability to write a standard evaluation report
Module VI
Evaluating for
program results
Handling evaluation findings
Communicating evaluation findings
Utilization of evaluation report
Integrity in accepting evaluation results as scientific findings
Communication / Presentation skills in dispensing evaluation report in a convincing manner to all stakeholders
Ability to apply recommendations from evaluation reports in implementation of similar future projects
3.6 Training Evaluation
At the end of the training, participants were asked to evaluate the program including the
trainers’ ability to effectively train, mode of delivery, content of training materials,
relevance of the training and to make any other comment especially on future trainings
and areas of improvement. The feedback from the participants was used in compiling
this training report and formed part of the recommendations herein. In addition,
participants were awarded with participation certificates on successful completion of the
course. Participants were also given the training materials.
18
CHAPTER FOUR
TRAINING FINDINGS AND ANALYSIS
4.1 Introduction of Chapter Four
This chapter presents findings from the training; and analysis of the training based on
the training objectives.
4.2 Findings from the Training
The findings from the training were identified and categorized in accordance with the
training’s specific objectives.
4.2.1 Findings from the Training on Training Objective 1
Training objective 1 was to help participants appreciate the purpose of project/program
evaluation and link evaluation to overall project/program performance to the satisfaction
of all stakeholders. Based on both the participants’ participation in the workshop and
feedback received from the evaluation forms, indications were that participants
appreciated the purpose of project/program evaluation including linking project/program
evaluation to the overall project/program performance. In addition, based on class
assessments in the workshops on stakeholders’ analysis, it can be argued that
participants appreciated the concept of stakeholder analysis in respect to
projects/programs evaluation. As such, indications from the training are that
participants appreciated the concept of self-efficacy in evaluation, taking responsibility
for project/program performance and appreciation of evaluation as a necessary phase
in the project life cycle to quantify and enhance performance rather than witch hunt from
senior management and project/program sponsors.
19
4.2.2 Findings from the Training on Training Objective 2
Training objective 2 was to help participants conduct project/program evaluation using
standard tools, frameworks and objectively verifiable indicators. Based on assignments
allocated in the focus group discussions in designing project/program indicators, it can
be argued that majority of the participants could individually and collectively design
Objectively Verifiable Indicators (OVIs) that can be helpful in measuring actual
project/program performance. In addition, on-job training assessments and feedback
given by participants on the training indicate that participants appreciated the types of
evaluation frameworks available for project managers including the ability to decide and
design relevant evaluation frameworks for given project/program context.
4.2.3 Findings from the Training on Training Objective 3
Training objective 3 was to help participants to write a standard evaluation report and
effectively present findings to key stakeholders for continuity of the project/program.
Informed by this training objective, participants were trained on how to conduct an
evaluation including writing and presenting an evaluation report. Following the training
on conducting an evaluation, participants appreciated the need for conducting
project/program evaluations, basis for selecting internal or external evaluators,
conceptualizing a project/program evaluation, tools used in conducting project/program
evaluation, evaluation designs, analysis of project/program findings, ethical issues in
evaluation and compiling and presenting an evaluation report to the satisfaction of key
project/program stakeholders. In addition, in the event participants chose to outsource
evaluation from external evaluators, the participants were equipped with requisite skills
that would enable them to draw clear evaluation Terms of Reference (TORs) for the
said evaluators based on appreciation of the evaluation conceptual framework. On the
same note, the knowledge acquired from the training would enable participants not only
be able to follow what an external evaluator is doing on behalf of the project/program
but also be able to assess whether an external evaluator understands the tasks
allocated in respect to project/program objectives.
20
4.3 Analysis of the Training
This section is on the analysis of the training. The analysis of the training was done
qualitatively based on class participation and assessments and feedback from
participants. Analysis of the training was carried out in accordance with the training
objectives.
4.3.1 Analysis of the Training on Training Objective 1
Training objective 1 was to help participants appreciate the purpose of project/program
evaluation and link evaluation to overall project/program performance to the satisfaction
of all stakeholders. While participants appreciated evaluation as a key phase of the
project life cycle prior to the training, findings from the training indicate that many
participants had not previously appreciated evaluation as a direct prerequisite to
project/program performance. Evaluation was previously perceived by many
participants just as a requirement by sponsors and senior management. In addition,
prior to the training, most participants viewed the project life cycle linearly rather than
inter-connected phases with M&E taking place throughout the life of a project.
Additionally, while most participants appreciated the broad concept of evaluation prior to
the training, most of the participants assumed that evaluation could only be done at the
end of a given project. The training, therefore, helped the participants appreciate
various types of evaluations including their diverse applications. Further, while most
participants understood that evaluation was core to project funders, the fact that
individual career growth and development was, to a great extent, linked to
project/program outcomes was not clear to many participants prior to the training. In
addition, many participants were not confident in respect to project/program evaluations.
While self-efficacy in evaluation focused on demystifying evaluation, the entire course,
to a large extent, served the same purpose of building confidence in participants in
respect to project/program evaluation.
21
4.3.2 Analysis of the Training on Training Objective 2
Training objective 2 was to help participants conduct project/program evaluation using
standard tools, frameworks and objectively verifiable indicators. Assessments from the
training indicate that participants appreciated the importance of evaluation indicators
prior to the training but most participants could not design objectively verifiable
indicators that could guide the project/program direction in a manner that could aid in
objective project/program evaluation prior to the training. In addition, indications from
the training revealed that many participants did not prior to the training expose their
evaluation indicators to peer review which undermined in-depth understanding of the
concept of evaluation indicators.
Although participants appreciated the value of evaluation frameworks prior to the
training, weak designs of evaluation indicators compromised the quality of evaluation
frameworks previously designed by the participants. While there were time constraints
in working through evaluation frameworks in the workshops, participants were exposed
to various evaluation models in use as required by various development partners.
However, while participants were able to work through focus group discussions on case
studies chosen by individual groups, the time allocated for the training was not
adequate for participants to be tested on ability to contextualize evaluation frameworks
in respect to the projects/programs presented in the workshops.
22
4.3.3 Analysis of the Training on Training Objective 3
Training objective 3 was to help participants to write a standard evaluation report and
effectively present findings to key stakeholders for continuity of the project/program.
While participants knew the importance of conducting project/program evaluation, not
as many could effectively undertake a project/program evaluation prior to the training.
While it was not possible to quantify actual acquisition of skills to conduct a
project/program evaluation during the workshop, participants were equipped with skills
to conduct an evaluation cognizant of multiple realities. Although learners were trained
on designing evaluation data collection instruments, time allocated for the training could
not allow a critique of important aspect of evaluation data collection instruments like the
practical usefulness of validity and reliability of evaluation data collection instruments.
Similarly, while evaluation designs and evaluation statistical tools of analysis were
included in the module on conducting an evaluation, the training was limited in helping
participants appreciate the concepts through contextualizing the skills in a project case
study. While some of the participants could comfortably write an evaluation report,
others could neither conduct nor write an evaluation report nor disseminate evaluation
findings. The training, however, was able to unearth learners’ capacity in respect to
conducting an evaluation which formed the basis for the report’s recommendations.
While dissemination of evaluation findings had been identified as a key deliverable from
the training, the same was the most adversely affected in respect to time allocation due
to time constraints. However, in as much as this was not done as a stand-alone module
as originally envisioned, the concept was intertwined in the entire course. Assessments
from the workshops indicated that most participants were relatively comfortable in
dispensing evaluation report in a convincing manner to all stakeholders to buy into the
evaluation results. However, that conducting evaluation must be scientific in approach
for results to be reliable and valid for utilization by project stakeholders needed more
time for internalization by participants. This way, evaluation findings are accepted as
having the required integrity threshold to quantify either an on-going or a concluded
project and to form the basis for implementation of future similar projects.
23
CHAPTER FIVE
CONCLUSIONS AND RECOMMENDATIONS
5.1 Introduction of Chapter Five
This chapter presents conclusions and recommendations made based on the training
objectives.
5.2 Conclusions on the Training
This section is on conclusions made from the training based on the training objectives.
5.2.1 Conclusions on Training Objective 1
Training objective 1 was to help participants appreciate the purpose of project/program
evaluation and link evaluation to overall project/program performance to the satisfaction
of all stakeholders. Based on the findings and analysis of training objective 1, it was
concluded that participants appreciated evaluation as a prerequisite of performance
rather than a fault finding mission. It was also concluded that participants, based on the
training, appreciated taking personal responsibility for projects/programs outcomes
informed by the conventional usage of the concept of evaluation and various scholarly
views about project/program evaluation. In addition, it was concluded that participants
appreciated integrity in programs implementation and the fact that programs comprise
various stakeholders with dynamic interests that must be satisfied. It was, therefore,
concluded that participants appreciated that the focus of the evaluator should not be
personal interests but satisfaction of all project/program stakeholders.
24
5.2.2 Conclusions on Training Objective 2
Training objective 2 was to help participants conduct project/program evaluation using
standard tools, frameworks and objectively verifiable indicators. Based on the findings
and analysis of training objective 2, it was concluded that participants appreciated the
importance of evaluation indicators and were able to design Key Performance Indicators
(KPIs) and to allow other evaluators to critique whether the indicators they develop meet
desired project/program objectives. Based on the training, participants were also able to
summarize and contextualize evaluation findings in acceptable and recognizable
various evaluation frameworks.
5.2.3 Conclusions on Training Objective 3
Training objective 3 was to help participants to write a standard evaluation report and
effectively present findings to key stakeholders for continuity of the project/program.
Based on the findings and analysis of training objective 3, it was concluded that
participants acquired necessary skills to conduct a standard evaluation cognizant of
multiple realities including ability to design simple data collection evaluation instruments
that are valid and reliable for evaluation. In addition, it was concluded that participants
could write and present a standard evaluation report based on skills acquired from the
training. Further, it was concluded that participants appreciated the concept of integrity
in accepting and utilization of evaluation results as scientific findings useful for advising
the implementation of similar future projects
25
5.3 Recommendations
This section is on training recommendations and it was organized in accordance with
the training objectives.
5.3.1 Recommendations from Training Objective 1
Training objective 1 was to help participants appreciate the purpose of project/program
evaluation and link evaluation to overall project/program performance to the satisfaction
of all stakeholders. Based on the findings and analysis of training objective 1, the
following was recommended:-
i. Training on performance management to be undertaken not only to the
project/program managers that participated, but also to other project/program
officers from the institutions/organizations that participated to help project team
members link their individual performance to the broader project/program
performance through performance measurement.
ii. Training on project management to precede trainings on evaluations so that
participants can link evaluation to project/program performance. Further, participants
ought to appreciate that evaluation is not a stand-alone phase in the project life cycle
and hence interlinkages with other project phases ought to be appreciated prior to
zeroing on project/program evaluation.
iii. Participating organizations need capacity building in the area of understanding the
various types of evaluations that can be executed rather than just undertaking
evaluations at the end of a project.
iv. Based on findings and analysis done from the training and in view of the fact that
evaluation is both a science and an art, it was recommended for project/program
managers to be trained on soft skills and in particular team work in project/program
evaluations, confidence in projects/programs evaluations, management of
project/program stakeholders and self-efficacy in project/program evaluations.
26
5.3.2 Recommendations from Training Objective 2
Training objective 2 was to help participants conduct project/program evaluation using
standard tools, frameworks and objectively verifiable indicators. Based on the findings
and analysis of training objective 2, the following was recommended:-
i. While indicators were taught through presentations and focus group discussions,
time allocated for the same was inadequate for satisfactory appreciation of this
important concept in evaluation. It was, therefore, recommended that more time to
be allocate for future similar practical hands-on training on designing and developing
evaluation indicators and linking OVI with project/program evaluation and
performance.
ii. Although learners appreciated evaluation frameworks and were able to design the
same through focus group discussions, it would have been more efficient to allow
participants to individually design an evaluation framework based on a given project
case study. In addition, time available for the workshop only allowed participants to
work through one evaluation framework: Log Frame Approach (LFA) Model. While it
is generally argued that LFA has wider usage than other evaluation models, it would
have been better to allow participants to work through various frameworks
collectively and individually both for knowledge acquisition and to appreciate
requirements from different development partners on different evaluation
frameworks. It was, therefore, recommended that more time, say, a whole working
day, to be allocated in future similar trainings on evaluation frameworks.
27
5.3.3 Recommendations from Training Objective 3
Training objective 3 was to help participants to write a standard evaluation report and
effectively present findings to key stakeholders for continuity of the project/program.
Based on the findings and analysis of training objective 3, the following was
recommended:-
i. While participants were trained on conducting an evaluation including writing an
evaluation report, no time was allocated to undertake either individual or group
assignment on this particular training objective partially because of the complexity of
the task. While more time may be allocated in future similar trainings, an evaluation
cannot be conducted from a workshop context. As such, it was recommended that
evaluation experts be attached to the projects/programs that are in the CEI network
to help project/program managers on the ground understand the entire process of
conducting project/program evaluation from conceptualization to writing and
presentation of evaluation report.
ii. Since conducting a project/program evaluation is conventionally a rigorous scientific
process, it was recommended that a pre-training Skills Gap Analysis (SGA) be
undertaken to profile participants’ entry behavior. This would assist trainers to
engage participants at their skills level in such a manner that all participants can
effectively be trained on how to conduct an evaluation depending on their
educational level, learning capacity and specific skills in M&E. In addition, this would
assist in development of the training curriculum and materials in respect to content
depth, breadth and mode of delivery.
iii. While planning for future trainings, it is recommended that conducting an evaluation
to be done as an entire course on its own merit. In this case, the modules in this
course would range from designing an evaluation, developing evaluation objectives,
evaluation designs, evaluation approaches, evaluation respondents, evaluation tools
of data collection, evaluation statistical tools of analysis, analyzing evaluation
findings, writing an evaluation report and presenting an evaluation report.
28
APPENDICES
(i) Appendix I: List of Participants
No. Organisation Forename Surname
1 Center for Education Innovations Caroline Jordan
2 Center for Education Innovations Esther Sifuma
3 Center for Education Innovations Namulanta Kombo
4 Daraja Civic Godfrey Otieno
5 DOT Trust Antony Omutobe
6 Educate Africa Nuala Alibhai
7 Emerging Leaders Foundation (ELF)Africa Caleb Odhiambo Ouma
8 Equity African Leaders Program (EALP) Michael McCreary
9 Future First Pauline Wanja
10 Global Education Fund Daniel Masawi
11 GRADIF Kenya Gladys Miriti
12 Hope for Teenage Mothers Lucy Ndungu
13 ICRI - International Child Resource Institute Stephen Waweru
14 Innovate Kenya Richy Bikko
15 Kesho Kenya Dr. Kate Nokes
16 Kibera Girls Soccer Academy (KGSA) Richard Teka
17 Kidogo Janet Mwitiki
18 Kids Comp Camp Caleb Ndaka
19 Kindergarten Experts Peter Odour
20 Kito International Wiclif Otieno
21 Koinoina Community Okada Buluma
22 Livelyhoods Millicent Chepkemoi
23 Mentor Coach Empower (MCE) Uganda James Katumba
24 Mobjap Children Center Bernard Asanya
25 Mortely Youth Group Stephen Kimani
26 Nairobi Parenting Clinic Vivianne Matuku
27 Nairobits Fatuma Ramadhan
28 NAPS Kenya Juliet Mwangi
29 PACEMaker International Doris Kiogora
30 PACT Global Business Wangari Kabiru
31 Riziki Kenya Mwirigi Gatobu
32 Safe Spaces Immaculate Agusta
33 Samburu Girls Foundation Wanjiru Wahome
34 Samsung Engineering Academy Beverlyne Mudeshi
29
No. Organisation Forename Surname
35 Sense of Worth Trust Peggie Kalie
36 Spire Education Pauline Atieno
37 Star of Hope Daycare Centre Miriam Nyongesa
38 STEM Africa Dr. Susan Musembi
39 The Kuza Institute Gilbert Mitullah
40 The Supply John Ng'ang'a
41 Tushinde Children's Trust Paul Otieno
42 Ufahamu Youth Okumu Churchil
43 VICDA - Volunteer International Community Development Africa Irene Ngatia
44 Youth Fund Sharu Huka
30
(ii) Appendix II: Training Program
Day One:
Time Module Learning Objectives Learning Outcomes
8.00 – 8.30 a.m. REGISTRATION & OPENING REMARKS
8.30 – 10.30
a.m.
Module I
Introduction
to
Evaluation
Evaluation as a key phase of the project life cycle
Purpose of evaluation
Self-efficacy in evaluation
Components of evaluation
Types of program evaluations
Appreciation of evaluation as a prerequisite of performance
Taking personal responsibility for projects / programs outcomes
Appreciation of the concept of evaluation and various views about evaluation
10.30 – 11.00
a.m.
TEA BREAK
11.00 – 1.00
p.m.
Module II
Evaluation
stakeholder
s
Evaluation ethics
Identifying evaluation stakeholders
Stakeholders needs analysis (cost, time, program efficiency, program impact etc)
Integrity in programs implementation
Appreciation that programs comprise various stakeholders with dynamic interests that must be satisfied – hence the focus of the evaluator should not be her/himself but program stakeholders
1.00 – 2.00 p.m. LUNCH
2.00 – 4.00 p.m. Module III
Evaluation
Indicators
Purpose of evaluation indicators
Designing evaluation indicators
Critiquing evaluation indicators
Participants should appreciate the importance of evaluation indicators
Learners should be able to design Key Performance Indicators (KPIs)
Learners should allow other evaluators to critique whether the indicators they have developed meet desired project/program objectives
4.00 – 5.00 p.m. PLENARY SESSION & REFRESHMENTS
31
Day Two:
Time Module Learning Objectives Learning Outcomes
8.30 – 10.30
a.m.
Module IV
Evaluation
Frameworks
Purpose of evaluation frameworks
Types of frameworks used in evaluation
Contextualizing evaluation frameworks
With or without a full evaluation report, participants should be able to summarize their evaluation findings in recognized evaluation frameworks
Appreciation and acceptance of various evaluation frameworks that are used in projects/programs
Learners should be able to contextualize existing evaluation frameworks in their program setting
10.30 – 11.00
a.m.
TEA BREAK
11.00 – 1.00
p.m.
Module V
Conducting
and reporting
evaluation
Conducting a program evaluation
Designing program evaluation instruments
Writing an evaluation report
Skills to conduct an evaluation cognizant of multiple realities
Ability to design instruments that are valid and reliable for evaluation
Ability to write a standard evaluation report
1.00 – 2.00 p.m. LUNCH
2.00 – 4.00 p.m. Module VI
Evaluating
for program
results
Handling evaluation findings
Communicating evaluation findings
Utilization of evaluation report
Integrity in accepting evaluation results as scientific findings
Communication / Presentation skills in dispensing evaluation report in a convincing manner to all stakeholders
Ability to apply recommendations from evaluation reports in implementation of similar future projects
4.00 – 5.00 p.m. CLOSING REMARKS, AWARDING CERTIFICATES & REFRESHMENTS
32
(iii) Appendix III: Feedback Form
Evaluating your Education Program from Start-Up to Scale CEI Evaluation Training Workshop – 3rd and 4th February 2015
Participants Feedback Form Introduction
Following your successful completion of the “Evaluation from Start Up to Scale”
course, we would like to request you to fill this Participants Feedback Form to help us
improve the implementation of future similar undertakings.
Section A: Course Relevance 1) Based on the course that you have been trained, kindly rate the following factors /
statements using a scale of SA – Strongly Agree; A – Agree; N – Neutral; D – Disagree; and SD – Strongly Disagree
Parameters S
tro
ng
ly
Ag
ree
Ag
ree
Neu
tral
Dis
ag
ree
Str
on
gly
Dis
ag
ree
(SA) (A) (N) (D) (SD)
a) I am happy that I attended the training
b) The training is relevant in my work
c) The training has enhanced my evaluation skills
d) The course content is relevant in my work
e) The course will help me improve in program evaluation
f) The time allocated for the training was satisfcatory
g) I understood what was being taught
h) The course has met my expectations
33
Section B: Competence of Trainers
2) Based on the course that you have been trained, kindly rate the following factors /
statements using a scale of SA – Strongly Agree; A – Agree; N – Neutral; D – Disagree; and SD – Strongly Disagree
Parameters
Str
on
gly
Ag
ree
Ag
ree
Neu
tral
Dis
ag
ree
Str
on
gly
Dis
ag
ree
(SA) (A) (N) (D) (SD)
a) I enjoyed the training
b) Training delivery was satisfactory
c) There was clarity in the trainers’ language
d) The training was participatory
e) The trainers were competent in the subject matter
f) I would be happy to be trained by the same trainers again
Section C: Improvement Strategies 3) Which was the most useful component of the workshop and why?
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
4) What would you like to see included in future workshops?
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
34
5) Suggest future workshops that you would like to attend facilitated by CEI
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
6) Are there any stakeholders you would like CEI to help you make linkages with?
(Either attendees or others)
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
7) Is there any support you would want in your organization in light of the training that
you have undergone? (e.g. conducting an evaluation, reviewing your evaluation
framework, training other staff members on evaluation etc)
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
8) Any other comment
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
Thank you.