panacea evaluation
TRANSCRIPT
-
7/31/2019 Panacea Evaluation
1/60
PANACeA Formative
Network EvaluationReport
May 2011
Afroz Sajwani,BScN
Sr. Research Assistant PANACeA - Department of Community Health Sciences,The Aga Khan University, Stadium Road. P.O. Box 3500. Karachi, Pakistan
74800, Ph: +92-21-34801315 Fax: 92-21-4934294
Email: [email protected]
Shariq Khoja,MBBS. MSc. PhD (eHealth)
Director eHealth Program and Asst Professor,
The Aga Khan University, Stadium Road. P.O. Box 3500. Karachi Pakistan
74800, Ph: +92-21-4930051 (Ext: 4920) Fax: 92-21-4934294Email: [email protected]
Hammad Durrani,MBBS. MSc (Health Policy and Management)
mailto:[email protected]:[email protected]:[email protected]:[email protected] -
7/31/2019 Panacea Evaluation
2/60
, ( y g )
Table of Contents
Acknowledgements .............................................................................................................................. 4
Abbreviations ........................................................................................................................................ 5
1. Executive Summary ......................................................................................................................... 6
2. Introduction and Background .......................................................................................................... 8
2.1 Network Background: ................................................................................................................ 8
2.2 DECI Background: ..................................................................................................................... 9
2.3 Rationale for Network Evaluation of PANACeA: ................................................................... 9
3. Methodology and Approach .......................................................................................................... 10
3.1 Programs readiness assessment for evaluation:..................................................................... 11
3.2 Evaluator readiness and capability assessment: ..................................................................... 11
3.3 Identification of Primary Intended Users: ............................................................................... 11
3.4 Situational analysis: .................................................................................................................. 12
3.5 Identification of primary intended uses: ................................................................................. 123.6 Focusing the evaluation: .......................................................................................................... 15
3.7 Evaluation design: .................................................................................................................... 15
3.8 Simulation of uses: ................................................................................................................... 16
3.9 Data collection: ......................................................................................................................... 17
3.10 Data Analysis: ......................................................................................................................... 17
3.11Facilitation of use: ................................................................................................................... 20
3.12 Meta-evaluation: ..................................................................................................................... 214. Findings ........................................................................................................................................... 22
Category 1: Collaboration and Teamwork Subcategories: Network approach,
C i ti & P ti i ti 22
-
7/31/2019 Panacea Evaluation
3/60
5.3 Category 3: Knowledge Management: .................................................................................... 55
5.3.1 Subcategory: Dissemination ............................................................................................. 555.3.2 Subcategory: Policy Impact .............................................................................................. 57
Appendix 5: Table for Utilization Action Steps ............................................................................... 58
-
7/31/2019 Panacea Evaluation
4/60
Acknowledgements
We would like to extend our sincere thanks to PANACeA team members for their participation
and active involvement throughout the evaluation process.
Our gratitude to DECI Team and Mr. Chelladurrai Solomon for developing our Evaluation
Capacity in Information Communication Technology for development (ICT4D) and for
mentoring us to carry out formative evaluation of PANACeA Network via Utilization Focused
Evaluation (UFE) approach.
We are also grateful to International Development Research Centre, Canada for providing us
with ongoing support and funding for the formative evaluation of PANACeA Network.
-
7/31/2019 Panacea Evaluation
5/60
Abbreviations
PANACeA: PAN Asian Collaborative for Evidence-based eHealth Adoption and Application
IDRC: International Development Research Centre
DECI: Developing Evaluation Capacity in ICT4D
ICT4D: Information Communication Technology for Development
PCTA: PANACeA Common Thematic Areas
MDF: Molave Development Foundation
AKU: Aga Khan University
UFE: Utilization Focused Evaluation
PIUs: Primary Intended Users
KEQs: Key evaluation Questions
AMT: Advisory and Monitoring Team
PL: Project Lead
PP: Project Partner
ST: Supporting Team members
ISRO: Indian Space Research Organization
IIM: Institute of Management (Bangalore)
CST: Canadian Society of Telehealth
GK3: 3rd Global Knowledge Conference
eHAP: eHealth Association of Pakistan
ISfTeH: International Society for Telemedicine and eHealthATA: American Telemedicine Association
DFID: Department for International Development
-
7/31/2019 Panacea Evaluation
6/60
1. Executive Summary
PANACeA is a network of health researchers and institutions belonging to 12 developing
countries of Asia, funded by International Development Research Centre (IDRC) Canada as a
four year project from August 2007 to July 2011.
The objective of this network is to generate evidence on how eHealth could improve the health
programs and services in the member countries through collaborative multinational eHealth
projects. This report illustrates the findings of formative evaluation of the network.
The purpose of PANACeAs formative evaluation is to determine how the network supported
networking, capacity building, and research projects, and generate recommendations for the
future.
Utilization Focused Evaluation (UFE) approach was used for the formative evaluation. All 12
steps of UFE were followed under the guidance of IDRC-DECI (Developing Evaluation capacity
for ICT4D) team. PANACeA leadership identified the needs and showed readiness for the
evaluation, and helped selecting Primary Intended Users (PIUs). In total, 25 PIUs were involved
to identify Primary Intended Uses for the evaluation. The evaluation was conducted using in-
depth interview guides to collect information from PIUs on PANACeAs role in facilitating
collaboration and teamwork, knowledge management and building capacity. The findings of the
evaluation were organized and analyzed using NVivo (Qualitative data analysis) software.
The major findings regarding the performance of PANACeA under the three main categories are
as follows:
-
7/31/2019 Panacea Evaluation
7/60
mentors, providing interaction avenues, and ensuring equal opportunities and representation for
both the sexes.
Capacity Building:
Network built partners capacity in research design, improved eHealth infrastructure and
enhanced their readiness for conducting independent eHealth projects by providing mentorship,
courses, workshops and trainings, opportunities for collective learning and sharing, and by
providing human, material and monetary resources. As a result of these initiatives, members
came up with communication plans, monitoring and evaluation indicators, and successful
implementation of their projects. Though members capacity in eHealth was enhanced, some
members demanded more capacity building in certain research aspects for which they suggested
more time and frequency of sessions and trainings, follow-ups, ground-level training, and
administrative and financial support.
Knowledge Management:
Network not only provided encouragement for dissemination but also built members capacity in
dissemination, and provided expenses for participation in dissemination conferences. This
resulted in internal and external dissemination of PANACeA projects and researches by its
members. Although the results of PANACeA projects are not disseminated yet, PANACeA
members were able to make impact through initiatives, such as development of eHealth
Association of Pakistan and influencing government and institutions in partner countries. In
order to bring evident policy change, the members suggested that they should consider focuseddissemination, use better marketing strategies, collaborate with government and policy makers,
other NGOs and related institutions, identify funding sources, and use country specific strategies
to bring policy impact in the field of eHealth
-
7/31/2019 Panacea Evaluation
8/60
2. Introduction and Background
2.1 Network Background:
The PAN Asian Collaboration for Evidence-based eHealth Adoption and Application
(PANACeA) is a network of health researchers and institutions to conduct collaborative research
on eHealth applications in Asian context. The network is funded by International Development
Research Centre (IDRC) Canada as a four year project from August 2007 to July 2011. Experts
from Pakistan, India, Afghanistan, Bangladesh, Nepal, Malaysia, Indonesia, Philippines, Sri
Lanka, Mongolia, and Canada volunteered to be part of the network and collect evidence on how
eHealth could improve the health programs and services in their countries. Eight research
projects were initiated involving multi-national teams. List of these projects and along with its
countries is given as appendix 1. The Key objectives of the network were:
To support a set of multi-country research activities to address the four core research
questions
1. Which eHealth applications and practices have had most beneficial outcomes?2. What are the best ways for ensuring that beneficial outcomes can reach the
population?
3. What is the potential of using new pervasive technologies?4. What types of technologies/applications are best suited to help prepare for, or
mitigate the effects of disasters, pandemics and emerging and re-emerging
diseases?To create a theoretical model for evaluating good practice in eHealth programs in Asia
To build research capacity amongst Asian researchers to evaluate and adopt appropriate
H lth t h l i d ti d i fl li d d i i k
-
7/31/2019 Panacea Evaluation
9/60
2.2 DECI Background:
Utilization-focused and outcome oriented evaluation is very important for InformationCommunication Technology for Development (ICT4D) research projects; therefore, in August
2009, IDRC initiated a two year project Developing Evaluation capacity for ICT4D (DECI) to
enhance evaluation capacity among IDRC-ICT4D project partners (http://www.idrc.ca/en/ev-
149323-201-1-DO_TOPIC.html). Under the leadership of Ricardo Ramirez and Dal Broadhead,
DECI team appointed a regional mentor, Mr. Chelladurrai Solomon, to support and facilitate
PANACeAs Utilization-focused formative network evaluation process.
2.3 Rationale for Network Evaluation of PANACeA:
PANACeA has its objectives defined at both Network and project levels, for which separate
activities are undertaken at each level. PANACeA realizes the importance of network at both
levels, but for the purpose of formative evaluation, only the activities at network level have been
evaluated. Through this formative evaluation, PANACeA aims to determine how the network
supported its research projects, how and what it achieved as a network and what else should be
done in future, by whom, and at what level to ensure the fulfillment of the networks objectives.
-
7/31/2019 Panacea Evaluation
10/60
3. Methodology and Approach
Though several evaluation approaches are available, Utilization Focused Evaluation (UFE)
approach as given by Michael Quinn Patton, gives a highly systematic approach which
emphasizes on identification of the Primary Intended Uses and Primary Intended Users (PIUs),
who hold a stake in the evaluation and who would actually use those findings. It then directs the
identification of the aspects in which those primary evaluation stakeholders would intend to use
the evaluation findings. With this stepwise and systematic approach, the evaluator facilitates and
involves the PIUs in identification of uses of the evaluation, focusing the evaluation, selecting
study designs, interpreting results, applying the evaluation findings and executing the
recommendations. This active involvement in decision making and in evaluation process
enhances a sense of ownership for the evaluation in the PIUs and fosters utilization of evaluation
finding and process by the PIU.
Patton presented the course of UFE approach using 12 concrete steps; from the programs
assessment till the utilization of findings, in which involvement of the Primary Intended Users is
crucial. For PANACeA evaluation, the UFE checklist was referred and followed. The milestones
for the 12 UFE steps as given in the checklist, with their timelines are as follows:
Timeline Milestones
January 2010 Programs readiness assessment for evaluation
February 20101
stto 4
thFeb, 2010
Identification of primary intended usersAnnual PANACeA Workshop in Bangkok, where
PIUs were identified
-
7/31/2019 Panacea Evaluation
11/60
Utilization Focused Steps used for PANACeA Formative Network Evaluation are as follows:
3.1 Programs readiness assessment for evaluation:
The AMT of PANACeA Network determined the need and readiness for PANACeAs formative
evaluation. The leadership and the network members were committed to spend their time and
resources for the evaluation since they wanted to identify the achievements of the network and
were prepared to identify gaps in the overall functioning of the network for their rectification. To
enhance readiness and commitment of all the network members, a workshop was organized
where the evaluation mentor assessed and enhanced readiness, and gave briefings regarding the
evaluation process so as to ensure understanding and enhance members commitment for the
evaluation.
3.2 Evaluator readiness and capability assessment:
After ensuring network members preparedness for the evaluation, network identified an external
evaluator who could facilitate them in carrying out the formative evaluation of PANACeA
Network, so that the network could be evaluated from a neutral lens. The evaluator was assessed
for capabilities and readiness for the evaluation. It was ensured that the evaluator had prior
experience of research and evaluation. The evaluator was based at AKU, and was guided by the
evaluation mentor throughout the process of evaluation. Evaluator was briefed regarding the
structure of the network and objectives of the evaluation. It was ensured that the evaluator is
ready and willing to collaborate with and engage the PIUs throughout the process of evaluation.
3.3 Identification of Primary Intended Users:
Identification of PIUs is a crucial step because they are the ones who would apply the evaluation
process and its findings in the way they intend Identification of such people who hold a stake in
-
7/31/2019 Panacea Evaluation
12/60
levels and achievement of networks objective could also be ensured. During this meeting, the
leadership and mentors not only identified PIUs but also ensured that they understood their rolein every step of this evaluation.
3.4 Situational analysis:
In order for evaluation to be useful, it is very important that the evaluation team along with the
PIUs assess the situational factors that would affect evaluation process and usage of evaluation
findings. This situational analysis not only helps in identifying existing and potential barriers but
also helps in looking for favorable factors, which, in turn, enhances a partners adaptive capacity,
and fosters the utilization of the evaluation. Following were the situational factors for PANACeA
formative network evaluation that could affect a users involvement in evaluation and evaluation
usage:
Programs prior evaluation experience: PANACeA was part of Formative Evaluation of
PANs Networking Approach conducted by IDRC for its four PAN Networks, done by
Mary Jane Real and Ricardo Wilson-Grau in July 2008. PANACeA itself as a program
was undergoing explicit evaluation for the first time, so there was no documented
learning on evaluation. The network members in different capacities did experience
evaluation, so they were clear about the concept of evaluation, but the UFE Approach
was new for all of them, which resulted in curiosities among the PIUs.
Availability of PIUs: PANACeA tried to ensure availability, willingness and commitment
of all its 25 members (AMT and project partners) as the PIUs. These PIUs were theenthusiastic, knowledgeable, cooperative and relevant to the subject.
Availability of resources: PANACeA identified and ensured availability of enough
t i l d t t t d t thi l ti
-
7/31/2019 Panacea Evaluation
13/60
purposes of PANACeA formative evaluation were to identify strengths and areas of
improvement of the network, list down the lessons learnt as a network, take major decisionsrelated to network and its activities so that improvements could be brought in the overall
functioning of the network. In order to fulfill this objective, PANACeA intended to use
evaluation findings to improve multiple aspects of the network, for which they referred to the
evaluation aspects of Formative Evaluation of PANs Networking Approach used by Mary
Jane Real and Ricardo Wilson-Grau (2008) [2]. These evaluation aspects include:
Communication, Participation, Knowledge Management and Learning, Leadership, Legitimacy,
Financial Management, Network Management, Resource Mobilization, Adaptive Capacity, and
Mentorship. The PIUs grouped some related aspects and elaborated one aspect to come up with
following three broad categories and thirteen subcategories.
1) Communication Collaboration and Teamwork2) Participation3) Knowledge Management and Learning Capacity building
Networked Approach
Dissemination
Policy impact
4) Leadership5) Legitimacy6) Financial Management7)
Network Management Enabling Environment8) Resource Mobilization
9) Adaptive Capacity10) Mentorship
-
7/31/2019 Panacea Evaluation
14/60
To ensure participation from all partners and promoting gender equality in
communicationTo enhance use of appropriate technology and tools for collaboration and clear
documentation system
To enhance availability and accessibility of software (open source) for research and
collaboration
To ensure transparency and accountability
To explore other ways or methods of enhancing collaboration and participation among
the teams and within the network.
Category 2: Knowledge Management and Learning
Subcategories: Capacity Building, Network Approach, Dissemination and Policy Impact.
The process of knowledge management and learning in a successful network includes reflection
on the experiences of network, refining goals, policies and priorities based on these experiences,and sharing of the relevant and meaningful knowledge of the network within the network
members, with other like-minded individual researchers and research networks, as well as with
other audience groups within their immediate and extended social network [2]. Keeping in mind
this broad understanding of the process of knowledge management and enhancing learning, this
aspect was elaborated and subcategories of Capacity building, Network Approach,
Dissemination and Policy impact were added into it. The uses identified by the PIUs under this
category include:
To enhance the eHealth knowledge of PANACeA partners
-
7/31/2019 Panacea Evaluation
15/60
Category 3: Enabling environment
Subcategories: Leadership, Mentorship, Legitimacy, Financial Management, NetworkManagement, Resource Mobilization and Adaptive Capacity.
An enabling environment is a set of interrelated conditions - such as legal, organizational, fiscal,
informational, political, and cultural that impact on the capacity of development actors to
engage in development processes in a sustained and effective manner [4]. As aspects of
leadership, mentorship, legitimacy, financial management, network management, resource
mobilization and adaptive capacity, can contribute towards enabling environment of the
Network, these aspects were merged under the category of Enabling Environment. The uses
identified by the PIUs under this category include:
To appreciate and improve the leadership and managerial strategies of the Leads and
AMT members
To strengthen relationship between network lead, project leads, AMT and network
members
To efficiently manage financial and administrative aspects of the network
To effectively plan future activities to advance and sustain the network, such as resource
mobilization and adaptive capacity.
3.6 Focusing the evaluation:
The focus of evaluation emerged from the intended uses by the PIUs. PIUs decided to focus andevaluate those aspects which they think would have best utility and which would bring the most
positive outcome from the evaluation. Keeping in mind the timelines and time period for
PANAC A f ti t k l ti th PIU i iti d th i d d th i i t d d
-
7/31/2019 Panacea Evaluation
16/60
Network did not tend to measure the initiatives that it took; rather, it focused towards
identification of the lessons learnt and challenges faced by the network members in the aboveareas. For this same reason, they collected data from all network members and hence their PIUs
were actually also their data source.
In order to stimulate respondents thinking and extract their opinions, an interview guide was
designed, containing open-ended Key Evaluation Questions (KEQs) pertaining to Collaboration
and Teamwork, and Knowledge Management and Learning. These KEQs Include:
Collaboration and Teamwork:
1) How has PANACeA facilitated communication and interaction amongst the networkmembers?
2) How the network members have collectively worked towards achieving commonPANACeA objectives?
Knowledge Management and Learning:
1) To what extent has PANACeAs Network approach benefited towards your researchskills and achievements?
2) To what extent has PANACeA enabled capacity building among its partners?3) How has PANACeA helped in disseminating the research findings inside and outside the
network?
4) To what extent will the PANACeA Network contribute to policy change in the partnercountries?
These KEQs were then circulated to all PIUs and to the mentors so that the questions could be
verified
-
7/31/2019 Panacea Evaluation
17/60
of this exercise and the modifications that were brought in the tool were discussed with all other
remaining primary intended users via emails. This exercise helped finalize the key evaluationquestions and enhance the sense of ownership among primary intended users. The original KEQs
and the modifications brought in the KEQs along with the rationale are illustrated as Appendix 2.
3.9 Data collection:
After the finalization of the study design and data collection tool, data collection was initiated.
Initially, 25 PIUs were identified but until the time of data collection, two of them left the
network. Out of the remaining 23 PIUs, the evaluator was able to collect data from 22 users(95% response rate). These interviews were conducted online by the evaluator using Skype. The
evaluator took consent from all the respondents and ensured confidentiality and anonymity of the
data. The time period of each interview varied between 30 to 60 minutes. The interviews were
recorded using ifree Skype Recorder1, and the recordings were then used for transcription of the
interviews. The transcription was then shared with the respective respondent for their review.
Out of 22 PIUs, 14 sent their reviewed transcripts (63.6% response rate). Remaining transcripts
were accepted in their original form.
3.10 Data Analysis:
Data analysis is not done in a single step; rather, it involves a chain of steps, which includes data
organization, integration and recommendation. Through proper data management and
organization, the data becomes understandable and meaningful, and hence its utilization could be
facilitated.
In order to organize the data gathered from the interviews, qualitative data analysis software,
QSR NVivo was used This software does not conduct analysis for us but it actually facilitates
-
7/31/2019 Panacea Evaluation
18/60
were basically representing the collaborative approach used for research, Capacity Building
was made as a separate category, while category Knowledge Management and Learning wasrenamed to Knowledge Management. The KEQs under each subcategory were taken as the
sub-subcategories, as illustrated in the diagram below:
Fi 1 P i i ( i b i d b b i )
-
7/31/2019 Panacea Evaluation
19/60
to look at the similarities and differences in the opinion of the mentors and the project members
(project leads and project partners) working in different capacities and at different levels of thenetwork. The three major groups are as follows:
1. Core AKU Team - This includes five members, of which three are AMT and two workedas support team members for PANACeA
2. Non -AKU AMT - This includes remaining non-AKU based four AMT members3. Project Members (PM) - This includes remaining non-AKU based thirteen PIUs who
were project leads and project partners overlapping in eight different projects.
The responses from group 1 were tagged with a small green box, from group 2 with small purple
box, and from group 3 with small orange box. The example of one of such graphical
representation of the findings with complete analysis is displayed below:
-
7/31/2019 Panacea Evaluation
20/60
During the step of data analysis, the revised categorization and the preliminary findings models
were shared with 13 PIUs during a face-to-face meeting of PANACeA partners (October 25,2010, in Colombo) so that they have an idea of the data that they would be using. Along with
these findings, the list of the primary intended uses that they identified initially were given to
them, and they were guided to revisit and refine the intended uses. The table illustrating original
and revised primary intended uses pertaining to revised categories along with the justifications/
discussion points as given by the PIUs is given as appendix 3. The exercise not only helped in
refining the uses but also reinforced and reminded PIUs regarding the utility of the evaluation
findings as they felt a sense of ownership while revisiting these uses.
3.11Facilitation of use:
In order to fulfill the purpose of UFE i.e. to facilitate the utilization of the PANACeA Formative
Network Evaluation findings, some utilization action steps were proposed to the PIUs. Along
with the action steps, timelines and members or level of the network that would be responsible to
carry out the action points, was also suggested. A table illustrating these suggestions was
circulated to all PIUs and based on their comments some modifications was done.
During PANACeAs fourth Annual Workshop in Kuala Lumpur, Malaysia on April 16, 2011,
final comments, a consensus was regarding the action actions. Some of the utilization actions for
were shifted to the second phase of PANACeA (PANACeA 2.0).
Major utilization actions points for the current phase were:
Ensure support from institutions and maintain communication with the heads of
institutions
-
7/31/2019 Panacea Evaluation
21/60
The timelines and responsible person/team/level responsible to execute these utilization actions
are illustrated as appendix 5.
3.12 Meta-evaluation:
There is a plan to determine the extent to which intended and additional uses were achieved by
the PIUs of PANACeA formative evaluation; this step would be conducted by the DECI team.
-
7/31/2019 Panacea Evaluation
22/60
4. Findings
The graphical illustrations of the findings with the in-depth analysis under each KEQ are presented as models in the appendix 4. The
description of the major findings under each KEQ i.e. Sub-subcategories with the uses are given in the table below. Some of the
important statements of the respondents that give examples, rationale and explanation to the findings are also presented.
Category 1: Collaboration and Teamwork Subcategories: Network approach, Communication & Participation:
Primary
Intended Uses
KEQs (Sub-
subcategories)
Findings from the Evaluation Statements
1. To promote
open, effective
and timely
communication,
participation
and
collaboration
between and
among
PANACeA
partners and
with the
external
stakeholders.
2. To enhance
use of
appropriate
technology and
tools for
PANACeA
strengths & factors
for effective
communication
All Groups: PANACeA strengths and factors for
effective communication include diversity in the
network and collaboration among members. This
effective collaboration is because of a common
goal, common language, open communication
and participation, collective learning and sharing,
good interpersonal relation, interest and
enthusiasm of members and supervision of
mentors.
By choosing network approach we are
able to get the best out of people, put
them together and basically they fill
each others gap and the output that
they come up with is much richer than if
just one person created or attempted to
create the same output. [Document 'PL
01', Section 1.1.1.2, Paragraph 13.]
Weaknesses and
Challenges of
PANACeA
Networks weaknesses and challenged include:
All Groups: Networks focus on individuals
rather than institutions because of which
individuals could give less time as they had their
day jobs and they faced administrative issues.
Communication gap, decreased participation from
some members and issues in online
communication such as technological glitches,
time zone differences and use of too many ICT
tools.
AKU team: Hierarchical structure of network.
We cannot move forward with
implementation as we have not yet
received the operational expenses, we
have already cleared our finances but
the other partner countries members
have not done it. So because of one or
two members all other members of the
research in partner countries are
affected. [Document 'PL 05', Section
1.1.1.6, Paragraph 25]
-
7/31/2019 Panacea Evaluation
23/60
PANACeA Formative Network Evaluation Report 23
documentation
and
collaboration,
among andbetween project
partners.
Non AKU AMT: Informal approach and absence
of a legal framework.
Project Members: Limited training, haphazard
communication, non-adherence to deadline, lessexpenses for face to face meeting and less
involvement of government sector.
Changes for the
future in network
All groups: Involve institutions rather than
individuals, make good blend in teams, and make
following communication efforts: more
encouragement to interact, more reminders to
respond, more face to face meetings, use better
quality internet connection, make flexibility to the
routine to adjust time zone difference, use ICT
tools like Twitter, Facebook, Youtube, Webinars,
Webportals but choose and stick to one or two
technological solutions that you find appropriate.
AKU team: Decrease hierarchy in network to
enhance free flow of information.
Non-AKU AMT: Make the network more
formalized.Project members: Adhere to timelines, involve
multi stakeholders, individualize financial
reporting for each partner country, provide
administrative support to individual members,
arrange mandatory meetings for partners,
if we could have brought in some
project through ministry of public health
than this could have created an enabling
environment for ministry of public
health to think how this eHealth is
changing the healthcare industry, they
could have learnt benefited and
contributed to this and I think that was
what the area I would say can be
improved. [Document 'PP 01', Section
1.1.1.3, Paragraph 15]
General tools and All levels: Different communication tools and The annual face to face is most
-
7/31/2019 Panacea Evaluation
24/60
PANACeA Formative Network Evaluation Report 24
modes of
communication
used within
PANACeA
modes such as telephone, face-to-face modes
(seminars, meetings, workshops and
conferences), SMS and onlline modes (Skype,
Elluminate Live!, audioconference, Facebook,website, online forums, instant messenger, emails
and mailing list) were used. Out of these,
telephone, face-to-face modes, Skype, emails and
mailing list are the most effective ones for
communication.
valuable in clarifying issues and moving
forward research activity
implementations. [Document 'PL 05',
Section 1.1.2.9, Paragraph 59]
Networks support
for communication
within the projects
All groups: Network provided structure of
reporting and meetings between AMT and project
members.
AKU team: Network provided both formal andinformal communication modalities.
Non-AKU AMT: Network supported
communication via Communication PCTA.
Project members: Network provided expenses to
carryout online and face to face communication.
Effective communication was fostered
by offering relevant activities which kept
us in touch with the network such as
workshops, ongoing activities whichwere very encouraging, the incentive for
us to gather was a good motivation for
us to participate and communicate .
[Document 'PL 06', Section 1.1.2.7,
Paragraph 52]
Networks support
for communication
between different
projects
All Groups: Network provided online and face to
face modes, but this communication is less
effective; it needs more support.
Non-AKU AMT: Network offered project
members to join PCTA and mentor it, but that
didnt work out.
Project Members: Network provided expertise
and technical support, enhanced sharing via
newsletter and created a sense of accountability
Networking itself is a big facilitator.
You are in a team and when you give
consent to be part of the team in a
network, you give up many of your
rights to hold all the results with you
and make sure that you are the sole
custodian of knowledge and results, so
you give up with that and you start
contributing, you start sharing right
-
7/31/2019 Panacea Evaluation
25/60
PANACeA Formative Network Evaluation Report 25
for communication. from the beginning. [Document 'AMT
01', Section 1.1.2.7, Paragraph 52]
Network members
functioningtogether
All Groups: PANACeA members worked
together as a team because they were motivated,dedicated, had good interpersoanl relationship
and were supervised by mentors adn leader. But
since the teams were unbalanced, there was less
involvement from some members.
Project Members: Members from diverse
countries and background had different expertise
but they collectively learnt and shared as they had
a common goal which helped them work well as a
team. But time zone differences, technologicalglitches and busy schedule limited effective
teamwork in different projects.
If the teams are not made properly, and
the people do not have the capacity toperform well, I dont think they are able
to actually do it no matter what they do.
There are cases where everything has
been done; all kind of support made
available but still people could not
deliver. [Document 'AMT 01', Section
1.1.3.1, Paragraph 64]
Networks help in
fostering
partnership
All Groups: Network provided interaction
avenues, built capacity and ensured clear
understanding of goals to foster partnership. The
structure of the network along with the efforts
from IDRC, AMT, PCTA, and project members
also fostered effective partnership.
3. To build in
gender equality
in
communication
and in eHealth
and eHealth
Networks help
for participation &
capacity building
of different sexes
All Groups: Network provided equal opportunity
for both sexes to communicate and to build their
capacities by creating gender PCTA, ensured
female representations at all levels, provided an
open sharing environment, provided
encouragement and gave understanding of gender
-
7/31/2019 Panacea Evaluation
26/60
PANACeA Formative Network Evaluation Report 26
research. issues.
AKU Team and Project Members: Gender
balance was not achieved at network.
AKU Team: Network could not reach the fieldlevel to ensure participation of both sexes.
Project Members: Project members tried to give
opportunities to both sexes at the field levels.
Category 2: Capacity Building in eHealth and eHealth Research:
Primary
Intended Uses
KEQs (Sub-
subcategories)
Findings from the Evaluation Statements
1. To extend
eHealthknowledge and
research
capability of
network
members and
other external
stakeholders.
2. To build
network
capability to
evaluate itself.
3. To obtain
Capacity building
activities foreHealth and
eHealth research
All Groups: Network built members capacity in
their entire research design, supported them toenhance eHealth infrastructure at their project
sites and enhanced their readiness for conducting
independent eHealth projects by providing
mentors, courses, workshops and trainings,
avenues for networking, encouragement and
expenses to present in conferences, collective
learning and sharing, and by providing human,
material and monetary resources.
Non-AKUAMT: Network invited external
speakers, shared information to attend webinars,
provided PCTAs and also planned to initiate but
could not execute plan of GeWOK (web of
knowledge).
All levels: More capacity building in needs for
Within our CBA group we actually
have the opportunity to take oneparticular hospital from the pre-
computer state through implementing
and giving them computers to help them
do a computerized registration and
laboratory test analysis software
package. So we have done very much for
that particular site, we have very much
given huge amount of infrastructure but
in terms of physical infrastructure, the
computers, the networking capability
that we put the networking in place for
them. [Document 'AMT 04', Section
1.2.1.8, Paragraph 100]
-
7/31/2019 Panacea Evaluation
27/60
PANACeA Formative Network Evaluation Report 27
support for
future and
existing eHealth
projects.
some research aspects such as: needs assessment,
data collection, implementation, analysis,
evaluation and report writing.
Outputs andoutcomes of
capacity building
initiatives
All levels: The output and outcomes of thesecapacity building initiatives are yet to come, but
because of above initiatives, we have successful
implementation of PANACeA projects,
development of communication plans, initiation
of dissemination activities for PANACeA
researches, and enhancement in the ICT usage,
eHealth knowledge and hunting for new eHealth
projects.
Non-AKU AMT: Members came up withmonitoring and evaluation indicators.
There are couple of examples come inmy mind regarding the funding we
obtained, one is the eHealth association
of Pakistan. I think the networking we
had there is originally the PANACeA
team members and Pakistan started this
network and actually you see 5 board
members who are part of PANACeA. We
took this initiative, arranged funding
and now we have made it eHealthrepresentative body of Pakistan so that
was big success. [Document 'AMT 01',
Section 1.2.1.6, Paragraphs 93-94]
Suggestions for
capacity building
in eHealth
research:
All Groups: Time and frequency of sessions and
trainings for the capacity building should be
increased. Institutions should be involved instead
of individuals, more ground level training should
be provided, and capacity building needs should
be addressed earlier.
Project Members: There should be more follow-
ups, and more administrative and financial
support should be provided to the projects.
Network should document learning that has
occurred, increase network members interaction,
it was almost as the way we
designed projects first and then asked
them to go back and do their needs
assessment which is not the right way to
do it we perhaps could have done
better in that sense because if people
tried to put a square peg into a round
hole i.e. that you design research and
then find a problem to fit it, so thats not
the right way, so I think thats one thing
that could have been done to give a
-
7/31/2019 Panacea Evaluation
28/60
PANACeA Formative Network Evaluation Report 28
and provide more monetary and human resources
for some areas because some areas are far less
developed than others.
perspective about the capacity
building. [Document 'AMT 04',
Section 1.2.1.5, Paragraph 92]
4. To enhanceavailability and
accessibility of
software (open
source) for
research and
collaboration.
Findings similar to the response of theKEQGeneral tool sand modes of communication
used within PANACeA as described in category
1 findings.
5. To extend
existing project
interventions ona large scale.
6. To increase
uptake of
expertise from
network
members
resource pool.
Use added later data not collected explicitly on
it. This used could be postponed till summative
evaluation.
Category 3: Knowledge Management Subcategories: Dissemination and Policy Impact:
Primary
Intended Uses
KEQs (Sub-
subcategories)
Findings from the Evaluation Statements
1. To build
capacity to
disseminate
Networks support
for dissemination
and dissemination
All Groups: Network provided encouragement for
dissemination, built members dissemination
capacity, and provided information and expenses
The first newsletter was designed
intentionally so that it can be
disseminated basically within the
-
7/31/2019 Panacea Evaluation
29/60
PANACeA Formative Network Evaluation Report 29
research
findings,
communicate to
policy makersand inform
policy
development
e.g. publishing
in journals,
conference
presentations,
etc.
2. To
disseminate/
share
knowledge to
network and
(others) relevant
stakeholders
3. To
systematically
document the
results/learning/
lessons from the
dissemination
done by members for participation in dissemination conferences;
this resulted in internal and external
dissemination of PANACeA projects, researches,
and literature reviews by the AMT members, PLsand PPs via publications, conferences, seminars,
workshops, IDRC events, social networking,
emails, website, newsletter, website, digital
stories, posters, media, wikis, web portals,
brochures, field visits, meetings and discussion
with internal and external institutions and
agencies.
network partners and their institutions
for awareness at that level; for the
second newsletter it is planned to be
disseminated more widely, viaconferences and forums where we would
be disseminating newsletter.
[Document 'AMT 05', Section 1.3.1.1,
Paragraph 124]
Members access
to externalstakeholders via
network and use of
PANACeA
findings by
externals
All Groups: Through PANACeA, members were
able to get hold of journals and libraries, externalgroups and institutions, different professional
experts, and government and policy people.
All Groups: The use of PANACeA findings by
externals is yet to be done.
AKU Team & Non-AKU AMT: As a result of
sharing of PANACeA activities, institutes in
Africa, S. America and polar region have started
thinking about eHealth research network.
Non-AKU AMT & Project Members: As a result
of PANACeA researches there has been
enhancement in eHealth in different institutions
of Afghanistan, India and Philippines.
I have been able to share this with four
other nursing and medical schools in thePhilippines, the application we use is
iPath so because of this project we have
learnt how to use iPath, how to
configure iPath and we have taught this
in 3 medical schools and one nursing
school on how to use the application so
I think they will use the application in
their respective curriculums not yet may
be at policy level but for training their
own medical and nursing students in
Tele-medicine. [Document 'PL 01',
Section 1.3.1.5, Paragraph 139]
Policy impact
brought via
All Groups: Policy impact yet to be brought, but
PANACeA members have developed eHealth
I know that TBDOTS have been
speaking themselves to policy and
-
7/31/2019 Panacea Evaluation
30/60
PANACeA Formative Network Evaluation Report 30
activities (i.e.
policy
influence, reuse
of knowledge,feedbacks, etc)
network Association of Pakistan and have got major
influence in different institutions of the partner
countries via external dissemination. PANACeA
members have also started taking steps for policychange which includes collaboration with
government and policy makers, provision of
infrastructure, and enhancement in eHealth
knowledge and awareness in its partner countries.
decision makers in their own countries
and raising awareness and speaking
about their preliminary results
Pakistan does not formally haveTBDOTS committees together, I think
this is going to be significant impact
within Pakistan. [Document 'AMT 04',
Section 1.3.2.1, Paragraph 147]
Suggestions for
activities for
dissemination and
policy change:
All Groups: Network should involve and
collaborate with government, policy makers,
NGOs, eHealth associations and other relevant
external institutes.
AKU Team: Members should contribute tonational eHealth strategy and should share
successes and failures of the network.
Project Members: Members should understand
policy structure and take country specific
measures. There should be proper dissemination
guidelines, measures to identify and attract
funding sources.
Non-AKU AMT & Project Members: Members
should enhance the dissemination through
publishing research papers, reports and policy
papers, via visual modes, media, and advocacy
and via use of modes like handbook, breeze,
podcast etc.
For policy impact I think very
important thing would be to build
relationship with policy makers,
especially government officials. You can
never publish anything or put somethingon website or do one seminar or a forum
and expect a change in policy at
government level; you have to work with
the governments, so it is very important
that we work with government in
different countries, build relationship
with them, have them understand what
PANACeA is and why researches
conducted by PANACeA should be made
part of policy or should contribute to
not just at institutional level but also at
national level in the countries where we
are working. [Document 'AMT 01',
Section 1.3.2.3, Paragraph 150]
-
7/31/2019 Panacea Evaluation
31/60
5. Reflections on the process from the evaluator
Utilization Focused approach accentuates identification of uses by the PIUs, and the usage ofevaluation findings/processes by them to fulfill the uses. For PANACeAs Formative Network
Evaluation, 12 UFE steps were referred because the evaluation team and the PANACeA
Network found these steps helpful to carry out a systematic and rigorous evaluation. They found
these steps promising in terms of enhancing evaluation utility, as these steps encourage the
involvement of PIUs, which in turn increases their sense of ownership for this evaluation.
There was a diverse pluralistic group of 23 PIUs for this evaluation, who were involved at
different levels at each step of the UFE. These PIUs were from different countries, different
cultures and had different professional expertise. Their expert opinion and involvement
strengthened the process, increased their sense of ownership and brought richness to the
evaluation uses and findings, but on the other hand, there were some delays and communication
challenges because the PIUs had time zone differences, they had different level of involvement
because of their busy schedule, and their understanding about the UFE was not very clear from
the beginning of the process. In order to overcome these challenges good rapport was built with
the PIUs, encouragement and timely reminders were sent to them, and adjustments in the
timelines were made according to their needs and circumstances.
The formative evaluation of the PANACeA Network was initiated in January 2010 and took
about one year to complete the whole process. As the PANACeA project concludes in July 2011,
there remained very less time for PIUs to fully utilize the evaluation findings. It was a challengefor the evaluation team to facilitate full utilization and for PIUs as well to carry out complete
usage; it was suggested to park some utilization actions for the phase two of PANACeA.
-
7/31/2019 Panacea Evaluation
32/60
6. Conclusion
The PANACeA Network was evaluated in the aspects of Collaboration and Teamwork, Capacity
building and Knowledge management. The major strengths of the Network include: diversity,
effective communication, good interpersonal relations, increased eHealth understanding, and
initiation of dissemination activities. The major areas of improvement that were identified were:
participation of the members on equal footing and less hierarchical structure , blending of the
team with similar interests, involvement of institutions rather than individuals, capacity building
in eHealth research and evaluation, ground level training, and working towards bringing a policyimpact.
The formative evaluation used utilization focused approach to conduct the evaluation. This
approach directed the users of the evaluation to focus on the utilization of the findings right from
the start of the process. The evaluation team also involved PIUs in all steps of UFE and
facilitated them to plan and take specific actions, based on these findings, to bring improvement
in the network. The network members at different levels have already started implementing these
action steps. For future, the network plans to evaluate the results of these utilization action steps
by assessing the changes and improvement brought in the Network, this assessment would be
conducted via summative evaluation of PANACeA Network. The utility of this evaluation is also
planned to be evaluated i.e. Meta-evaluation (evaluation of this evaluation), which would be
conducted by DECI team.
Due to time limitation, network did not evaluate the third identified area of enabling environment
which includes sub areas such as leadership, legitimacy, financial management, network
management resource mobilization adaptive capacity and mentorship These areas are expected
-
7/31/2019 Panacea Evaluation
33/60
References
[1]Patton, MQ. Utilization-Focused Evaluation 2008; 4th
ed, USA: SAGE Publications.
[2]Real MJ, Wilson-Grau R.Formative Evaluation of PANs Networking Approach, IDRC
Report, 2008.
[3]Merriam-Websters online dictionary, Retrieved November 15, 2009 from
http://www.merriam-webster.com/dictionary/collaborate
[4]Thindwa J. Enabling environment for Civil Society in CDD Projects , Washington, DC: World
Bank, Social Development Family, CDD Learning Module, 2001. Retrieved November 20, 2009from
http://www.worldbank.org/participation/enablingenvironment/EnablingenvironmentCECDD.pdf
-
7/31/2019 Panacea Evaluation
34/60
Appendix 1: List of Projects
Projects Leading Country Other Partners
Multi institutional study on the cost benefit of hospital
automation (patient registration and laboratory systems).
Pakistan Philippines
India
Afghanistan
Portable system for Telemedicine and Health
Informatics in rural and remote areas.
Malaysia Sri Lanka
Nepal
Afghanistan
Improving maternal health care services by using ICTs
for remote consultation and education.
Mongolia Philippines
A framework to identify gaps in the use of eHealth in
primary health care settings.
India Pakistan
Philippines
Exploratory intervention research on eHealth for the
visually challenged.
Bangladesh Philippines
IndiaOnline TB diagnostic committees for clinically
suspected sputum negative patients in the TB-DOTS
program.
Philippines Pakistan
A systematic review of current ICT applications in
Disaster: A potential for integrating TM.
Indonesia Philippines
India
Community-based eHealth promotion for safe
motherhood: Linking community maternal health needswith health services system.
Philippines Pakistan
Indonesia
-
7/31/2019 Panacea Evaluation
35/60
Appendix 2: Key Evaluation Questions
Original Key Evaluation Questions Revised Key Evaluation Questions Rationale
Category 1: COLLABORATION AND TEAM WORK(Communication and Participation)
Communication1) How has PANACeA facilitated
communication and interaction
amongst the network members?
Communication1) How has PANACeA facilitated
communication and interaction amongst
the network members?
No change.
a) How has the network supported
communication between the project
partners, project leads and the AMT?
Which communication method was
effective?
a) How has the network supported
communication between the project
partners, project leads and the AMT?
Which communication method was
effective? Can you think of any technology
or tool?
The question is further specified by the
addition of technology component as PIUs
wanted to utilize evaluation findings to
enhance use of appropriate technology and
software (open source) for research and
collaboration as illustrated in Original Use
Category B use number 6 and in Category C
use 3.
b) How has the network supported
communication between different
partners? Which communication
method was more effective?
b) How has the network supported
communication between partners from
different projects? Which communication
method was more effective?
Question is further specified.
c) How has the network facilitated the
participation of different sexes,
experiences and perspectives to
engage at all levels of the network?
Do you feel that certain partners were
not included for communication by
the network?
c) How has the network facilitated the
participation of different sexes, experiences
and perspectives to engage at all levels of
the network? Do you feel that certain
partners were not included for
communication by the network?
No change.
d) How has the network supported
communication with people external to the
Question added because PIUs wanted to
utilize evaluation findings to promote
-
7/31/2019 Panacea Evaluation
36/60
PANACeA Formative Network Evaluation Report 36
network? communication with external stakeholders as
well as illustrated in Original Use Category C
use number 1.
d) What factors enhanced networks
ability of fostering effective
communication?
e) What factors enhanced networks
abilityof fostering effective communication?
No change.
e) What factors hindered networks
ability of effective communication?
How did network respond to them?
f) What factors hindered networks ability
of effective communication? How did
network respond to them?
No change.
g) What other methods of communication
would you suggest to enhance
collaboration among PANACeA members?
Question added because PIUs wanted to
utilize evaluation findings to explore other
ways or methods for enhancing collaboration
and participation among and within thenetwork as illustrated in Original Use
Category C use number 6.
Participation
2) How the network members have
collectively worked towards
achieving common PANACeA
objectives?
Participation
2) How the network members have
collectively worked towards achieving
common PANACeA objectives?
No change.
a) How well are the network
members functioning together?
a) How well are the network members
functioning together?
No change.
b) To what extent the network helped
towards achieving the objectives of
individual projects?
b) To what extent the network helped
towards achieving the objectives of
individual projects?
No change.
-
7/31/2019 Panacea Evaluation
37/60
PANACeA Formative Network Evaluation Report 37
Category 2: KNOWLEDGE MANAGEMENT AND LEARNING (Capacity Building, Network Approach, Dissemination and
Policy Impact)
Network Approach
1) To what extent has PANACeAs
Network approach benefited towards
your research skills and
achievements?
Network Approach
1) To what extent has PANACeAs
Network approach contributed to the
development of your research skills and
management (or conducting) of your
research?
Question is further specified.
a) What is your understanding of
network approach?
a) What is your understanding of network
approach?
No change.
b) What is the contribution of the
network in designing, planning,
implementing, analysis andfacilitating your research?
Deleted as it was a repetition to question 2a.
c) What are the strengths and
weaknesses of the network approach
adopted by PANACeA?
b) What are the strengths and weaknesses
of the network approach adopted by
PANACeA?
No change.
d) Do you think Network Approach
adopted by PANACeA is a better way
of conducting research?
c) Do you think Network Approach
adopted by PANACeA is a better way of
conducting research?
No change.
e) Would you recommend Network
Approach for future research
initiatives? Why? If not, what would
you change next time around?
d) Would you recommend Network
Approach for future research initiatives?
Why? If not, what would you change next
time around?
No change.
e) Do you feel the network approach
increases the transparency and
accountability in conducting research?
Question added because PIUs wanted to
utilize evaluation findings to ensure
transparency and accountability in the
-
7/31/2019 Panacea Evaluation
38/60
PANACeA Formative Network Evaluation Report 38
Please explain. network as illustrated in Original Use
Category B use number 5.
Capacity Building
2) To what extent has PANACeAenabled capacity building among its
partners?
Capacity Building
2) To what extent has PANACeA enabledcapacity building in eHealth and eHealth
research among its partners?
Question is further specified.
a) To what extent PANACeA
supports its researchers in research
design - hypothesis, feasibility,
contextual relevance, research
questions, methodology,
implementation, analysis, reporting
etc?
a) To what extent PANACeA supports its
researchers in research design - hypothesis,
feasibility, contextual relevance, research
questions, methodology, implementation,
analysis, reporting etc?
No change.
b) What is the nature of these
capacity building activities in
knowledge & skills?
b) To what extent PANACeA supports its
researchers in developing eHealth research
methodologies?
Question change because PIUs wanted to
focus on eHealth research methodologies and
eHealth research methodologies are different
from other research methodologies.
c) To what extent the knowledge and
skills of the capacity relevant to the
researches being undertaken?
c) To what extent the knowledge and skills
of the capacity building activities, relevant
to eHealth being enhanced through
PANACeA?
Question is further specified.
d) What are the outputs and outcome
thus far from the capacity building
support?
d) What are the outputs and outcome thus
far from the capacity building in eHealth
and eHealth researches?
Question is further specified.
e) Based on your experience with
PANACeA, how ready do you feel
that you are in carrying out an
e)Based on your experience with
PANACeA, how ready do you feel that
you are in carrying out an independent
No change.
-
7/31/2019 Panacea Evaluation
39/60
PANACeA Formative Network Evaluation Report 39
independent eHealth project i.e.
hypothesis, designing, data collection,
analysis and reporting?
eHealth project i.e. hypothesis, designing,
data collection, analysis and reporting?
f) Is there anything that PANACeAcould have done do increase partner
capacity building? Suggest specific
activities.
f) Is there anything that PANACeA couldhave done do increase partner capacity
building? Suggest specific activities.
No change.
g) How PANACeA helped you in writing
other eHealth proposals and in obtaining
funding?
Question added because PIUs wanted to
utilize evaluation findings to obtain funding
for similar projects as illustrated in Original
Use Category B use number 4.
h) How does PANACeA allow equal
opportunity for members from both sexesto build capacity in eHealth and eHealth
research?
Question added because PIUs wanted to
utilize evaluation findings to ensure genderequality in all aspects of learning and
knowledge management as illustrated in
Original Use Category C use number 4.
i) How has PANACeA helped in
enhancing eHealth infrastructure at the
project sites?
Question added because PIUs wanted to
utilize evaluation findings to improve ICT
infrastructure for eHealth in project sites as
illustrated in Original Use Category B use
number 5.
Dissemination
3) How has PANACeA helped in
disseminating the research findings
inside and outside the network?
Dissemination
3) How has PANACeA helped in
disseminating the research findings inside
and outside the network?
No change.
a) What dissemination activities have
you done till now of PANACeA
a) What dissemination activities have you
done till now of PANACeA research
No change.
-
7/31/2019 Panacea Evaluation
40/60
PANACeA Formative Network Evaluation Report 40
research findings? findings?
b) How PANACeA supports
dissemination of research findings
through different methods or forums?Do you propose any other forum or
medium?
b) How PANACeA supports dissemination
of research findings through different
methods or forums? Do you propose anyother forum or medium?
No change.
c) What groups of stakeholders could
you (or will you) access through
PANACeA that you may not have
accessed individually?
c) What groups of stakeholders could you
(or will you) access through PANACeA
that you may not have accessed
individually?
No change.
d) How have your findings been used by
others to strengthen eHealth research
program or policy?
Question added because PIUs wanted to
utilize evaluation findings to increase reuse of
knowledge by other advocates as illustrated inOriginal Use Category A use number 2.
Policy Impact
4) To what extent will the PANACeA
Network contribute to policy change
in the partner countries?
Policy Impact
4) To what extent will the PANACeA
Network contribute to policy change in the
partner countries?
No change.
a) Has PANACeA Network brought a
policy impact in the partner
countries? Please specify with
examples?
a) Has PANACeA Network brought a
policy impact in the partner countries?
Please specify with examples?
No change.
b) What were the dissemination
activities adopted by the Network that
might prove potential to bring policy
influence in eHealth? How?
b) What were the dissemination activities
adopted by the Network that might prove
potential to bring policy influence in
eHealth? How?
No change.
c) What activities should be adapted c) What activities should be adapted by No change.
-
7/31/2019 Panacea Evaluation
41/60
PANACeA Formative Network Evaluation Report 41
by PANACeA to further bring the
policy impact in its respective
countries?
PANACeA to further bring the policy
impact in its respective countries?
-
7/31/2019 Panacea Evaluation
42/60
PANACeA Formative Network Evaluation Report 42
Appendix 3: Revised Primary Intended Uses with justifications
Category 1: Collaboration and Teamwork
Original Revised Discussion Points1. To promote open and effective
communication, participation and
collaboration between and among
PANACeA partners and with the
external stakeholders.
1. To promote open and effective and
timely communication, participation and
collaboration between and among
PANACeA partners and with the external
stakeholders.
Not a major change, PIUs just want to ensure
timely communication.
2. To ensure participation from all
partners and promoting gender equality
in communication.
It is not dropped; they have included it in
revised third use.
3. To enhance use of appropriatetechnology and tools for collaboration,
clear documentation system.
2. To enhance use of appropriatetechnology and tools for documentation
and collaboration, among and between
project partners.
Not a major change. Exclusion of words, suchas clear and system do not change the
meaning/use significantly. They have brought
specification by adding among and between
project partners that was missing earlier.
4. To build in gender equality in
eHealth and eHealth research.
3. To build in gender equality in
communication and in eHealth and
eHealth research.
It is now more specific.
5. To ensure transparency and
accountability.
PIUs collectively came to the point that this
time this use was not their focus, so it was
removed.
6. To explore other ways or methods of
enhancing collaboration and
participation among and teams and
within the network.
It is not dropped; this is already covered in the
first revised one.
-
7/31/2019 Panacea Evaluation
43/60
PANACeA Formative Network Evaluation Report 43
Category 2: Capacity building
Original Revised Discussion points
1. To enhance the eHealth knowledge
of PANACeA partners.
1. To extend eHealth knowledge and
research capability of network membersand other external stakeholders.
External stakeholders refers to those
individuals and teams, who are notPANACeA partners, but are involved in
designing and implementing the projects.
PANACeA is actively involved in improving
both eHealth Knowledge and Research
Capability of researchers involved in the
projects.
2. To enhance the eHealth research
capacity of PANACeA partners and
their teams.
2. To build network capability to
evaluate itself.
Research capacity is now covered in the first
point. This point talks about capacity in
evaluation, which itself is a crucial area inresearch.
3. To promote research skills of
PANACeA members to enable conduct
of robust studies.
Already included in the first two points.
4. To obtain funding for similar
projects.
3. To obtain support for future and
existing eHealth projects.
No major change.
5. To improve ICT infrastructure for
eHealth of health care providers in
project sites.
PIUs found it out of their scope for this time.
6. To enhance availability and
accessibility of software (open source)
for research and collaboration.
4. To enhance availability and
accessibility of software (open source)
for research and collaboration.
No change.
5. To extend existing project
interventions on a larger scale.
Towards the end of their project, PANACeA
researchers found it important to measure
-
7/31/2019 Panacea Evaluation
44/60
PANACeA Formative Network Evaluation Report 44
scalability and sustainability of their research.
In case sufficient data is not available on this
use, it will be postponed till the final
evaluation
6. To increase uptake of expertise from
network members resource pool.
The main purpose for adding this use is to
measure the use of capacity built among
PANACeA partners. In case sufficient data is
not available on this use, it will be postponed
till the final evaluation
7. To build capacity to communicate to
policy makers.
It is an extension on A1, focusing specifically
on capacity building for policy change.
Category 3: Knowledge ManagementOriginal Revised Discussion points regarding revised Uses as
discussed by the PIUs
1. To build capacity to disseminate
research findings and impact inform
policy development e.g. publishing in
journals, conference presentations, etc.
1. To build capacity to disseminate
research findings and inform policy
development e.g. publishing in journals,
conference presentations, etc.
Not a major change; the idea was that since
PANACeA projects are pilot projects,
especially with regard to developing
countries, it is difficult to claim policy impact.
Based on their research, the partners can
inform policy developments through policy
recommendations and publications.
2. To increase re-use of knowledge by
other advocates.
2. To disseminate/share knowledge to
network and (other) relevant
stakeholders.
Others would reuse the knowledge only if it is
disseminated well. PIUs considered
dissemination important, because good
dissemination would lead to re-use by others.
3. To influence policy change. 3. To systematically document the As pointed out in the first point, PANACeA
-
7/31/2019 Panacea Evaluation
45/60
PANACeA Formative Network Evaluation Report 45
results/learning/lessons from the
dissemination activities (i.e. policy
influence, re-use of knowledge,
feedbacks, etc)
cannot bring policy change on its own. But
through proper dissemination (point 1) it can
influence policy makers and trigger a change
process through informed use (or re-use) of
the knowledge. It is important to carefully
document such changes.
-
7/31/2019 Panacea Evaluation
46/60
Appendix 4: Models with Analysis for each KEQ i.e. Sub-subcategories
The graphical illustrations of the findings with the analysis under each KEQ are presented below.
The change in shape from rectangle to bullets illustrates change in the level of categorization.There is no significance to the change in color of the rectangles in different models.
5. 1 Category 1: Collaboration and Teamwork
5.1.1 Subcategory: Network Approach
5.1.1.1Sub-subcategory:
5.1.1.2. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
47/60
5.1.1.3. Sub-subcategory:
5.1.1.5. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
48/60
5.1.1.5. Sub-subcategory:
5.1.1.6. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
49/60
5.1.2 Subcategory: Communication
5.1.2.1Sub-subcategory:
5.1.2.2. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
50/60
5.1.2.3. Sub-subcategory:
5.1.2.5. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
51/60
5.1.2.5. Sub-subcategory:
5.1.2.6. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
52/60
5.1.3 Subcategory: Participation
5.1.3.1Sub-subcategory:
5.1.3.2. Sub-subcategory
-
7/31/2019 Panacea Evaluation
53/60
5.2 Category 2: Capacity Building
5.2.1 Subcategory: capacity building in eHealth and eHealth Research
5.2.1.1Sub-subcategory:
-
7/31/2019 Panacea Evaluation
54/60
5.2.1.2. Sub-subcategory:
5.2.1.3. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
55/60
5.3 Category 3: Knowledge Management:
5.3.1 Subcategory: Dissemination
5.3.1.1Sub-subcategory:
5.3.1.2. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
56/60
5.3.1.3. Sub-subcategory:
5.3.1.5Sub-subcategory:
-
7/31/2019 Panacea Evaluation
57/60
5.3.2 Subcategory: Policy Impact
5.3.2.1. Sub-subcategory:
5.3.2.1. Sub-subcategory:
-
7/31/2019 Panacea Evaluation
58/60
Appendix 5: Table for Utilization Action Steps
The utilization action along with the suggested Network level that is responsible for carrying out particular utilization with an
approximate time is presented in the table below:
Category 1: Collaboration and Teamwork Subcategories: Network approach, Communication & Participation
Utilization Action Responsible levelfor execution
Approximate timeof execution
Specific tasks
Ensure and encourage diversity
among partners.
Interim steering
committee for
PANACeA 2.0
PANACeA 2.0 This could be ensured via membership guideline
for PANACeA 2.0
Ensure support from institutions
and maintain communication with
the heads of institutions
Project leads,
project partners and
IDRC
(March-July 2011) PP and PL will talk to institutional heads and
would share project results and reports with them.
After PL and PP have contacted their institutional
heads and disseminated project results, IDRC will
be asked to write letter of thanks/acknowledgingInstitutions work in eHealth research
Improve communication between
different levels of the network and
promote and implement
communication strategy.
All levels (March-July 2011) Communication PCTA (AJR & SN) will update
the mailing list. They will also re-circulate
communication protocol.
Everybody will take responsibility to share the
lessons learnt and project achievements and
findings among the network members.
Use simple and inclusive ICT tools All Levels (March-July 2011) Continue using email, Skype and Elluminate.
Ensure participation of members in
project activities, network activities
and PCTAs, and ensure that
partners meet the deadlines
AMT and Project
leads
(March-July 2011) AMT will encourage and send more reminders to
the project leads and project leads will also
encourage and send more reminders to the partners
for participation and meeting deadlines.
Balance project teams for better
coordination & teamwork in
Interim steering
committee for
PANACeA 2.0
-
7/31/2019 Panacea Evaluation
59/60
PANACeA Formative Network Evaluation Report 59
terms projects requirement. PANACeA 2.0
Ensure gender balance at all levels. Interim steering
committee for
PANACeA 2.0
PANACeA 2.0
Category 2: Capacity Building in eHealth and eHealth Research
Utilization Action Responsible level
for execution
Approximate time
of execution
Specific tasks
Enhance formal learning
opportunities for network partners
Interim steering
committee for
PANACeA 2.0
PANACeA 2.0 Formal learning opportunities would be added in
the proposal for PANACeA 2.0
Plan for GeWOK (web of
knowledge) to be implemented
Interim steering
committee forPANACeA 2.0
PANACeA 2.0 GeWOK or a similar platform (knowledge
repository) for PANACeA 2.0
Increase frequency of capacity
building sessions
AMT and PCTA
leads
(March-July 2011) Invitations of CMEs from different member
institutions will be extended to PANACeA
members.
PANACeA members will participate in eHealth
Educational Sessions organized by AKU.
Provide administrative training to
project partners
Interim steering
committee for
PANACeA 2.0
PANACeA 2.0 Trainings for administration and technical aspects
of eHealth project would be added in the proposal
for PANACeA 2.0
Category 3: Knowledge Management Subcategories: Dissemination and Policy Impact
Utilization Action Responsible level
for execution
Approximate time
of execution
Specific tasks
Ensure dissemination and use of All levels April onwards Every Project will make their dissemination plan
-
7/31/2019 Panacea Evaluation
60/60
PANACeA Formative Network Evaluation Report 60
PANACeA findings to broader
community
and accordingly will write papers, reports and
policy briefs and would take steps to share it
outside PANACeA to relevant institutions,
government officials, decision makers and policy
makers. This sharing of results and lessons learnt
will be done in a cohesive fashion.
Ensure policy impact through
proper dissemination of PANACeA
findings and policy dialogues
All Levels April onwards RES along with AMT for each project to keep on
guiding project members on making and
dissemination of policy briefs.
Build partnerships with
governments, eHealth Associations
and institutions for policy change
All Levels for
PANACeA 1.0 and
Interim steering
committee forPANACeA 2.0
April onwards
And for
PANACeA 2.0
Share PANACeA findings, results and learning
with relevant institutions, associations,
government officials, decision makers and policy
makers.Involve policy and decision makers on board for
PANACeA 2.0
Members take country-specific
actions for policy change
All levels April onwards Project members will organize and share their
project findings, results and learning with relevant
institutions, associations, government officials,
decision makers and policy makers in their own
countries
AMT will take responsibility to check the content
of dissemination before project membersshare/disseminate.