ecd in the 21 st century: what’s the best way forward?
DESCRIPTION
ECD in the 21 st Century: What’s The Best Way Forward? . Karen Russon and Michele Tarsilla Evaluation Capacity Development Group (ECDG) AEA Presentation - October 18, 2013. ECDG organized an Int’l Workshop on ECD 17-21 October 2011 . Today is a sort of anniversary .. - PowerPoint PPT PresentationTRANSCRIPT
ECD in the 21st Century: What’s The Best Way Forward?
Karen Russon and Michele TarsillaEvaluation Capacity Development Group (ECDG)
AEA Presentation - October 18, 2013
ECDG organized an Int’l Workshop on ECD 17-21 October 2011
Today is a sort of anniversary ..Two years ago today, evaluation practitioners, commissioners and members of Standards organizations gathered to discuss the current situation and the future of ECD.
As a Result of the Workshop, Four Key ECD Principles were Identified
Ownership
Integration
Usefulness
Relevance
1. “What does ECD look like in practice?”
2. “How could ECD processes be enhanced and evaluated in the future?”
Today’s Questions
Today’s ECD Stories Are Global
A Global ECD was Conducted to Understand ECD better
• Collection and Analysis of stories from multiple perspectives: actors, levels and contexts• Focus on:
-ECD partnerships -Infrastructure changes-Organizational processes improvements
Key ECD Trends
• Tri-partite ECD collaboration• Indigenous-to-indigenous ECD cooperation
• Creative use of action learning/research to enhance ECD buy-in
• ECD as a tool for social emancipation
A Few Selected ECD Stories
The Government of Papua New Guinea
Actors/Location: Gov’t Agencies, Papua New Guinea
Level: National, Provincial, Local
Sector: Government
Time period: 2003-present
Documentation: Paper presented at AES 2013
The Government of Papua New Guinea
Lessons Learned• Developing a “fit-for-use M&E system” was conducive to
enhancing coordination and harmonization among stakeholders in the public sector• The ECD participatory process eased the tensions among a
variety of ECD stakeholders at different levels• The availability of better data translated into stronger
district planning and decision-making.
The Papua New Guinea Assn. of Professional Evaluators (PNGAoPE)
Actors/Location: Australian Evaluation Society, PNGAoPE, Coffey Int’l, Expat evaluators, AusAID, Gov’t of Papua New Guinea
Level: Organizational
Sector: Evaluation, multi-sectors
Time period: 2006 – present
Documentation: Guest blog posted on www.ecdg.net
The Papua New Guinea Assn. Of Professional Evaluators (PNGAoPE)
Lessons Learned• The sustained support provided by international development
partners is often key to ECD success but it is not sufficient.• The official government recognition of a national evaluation
association was critical• The critical friend and support role played by expats could be
effective during the development of a national evaluation association
• The selection of topics to be discussed during monthly meetings was a sign of ownership and the first necessary step towards sustainability
Innovation NetworkActors/Location: non-profit providing consulting services in research and evaluation to other non-profits and funders, three funders and their respective grantees; Washington, D.C.
Level: Organizational
Sector: women’s economic empowerment, social justice, corporate philanthropy
Time Period: 2006-2011
Documentation: Personal communication with Innovation Network staff.
Innovation Network
Lessons Learned:• Provision of “default” services is better, especially if within the scope of an existing evaluation assignment
• ECD often consists of a Gradual Approach (shifting from funder’s interests to organizational needs and strategies)
• It is important to respond to strategic grantees’ demand (that is, demand for critical evaluation services) and not to confused or disjointed requests;
• The service provider should not be the one “running” the ECD program: leadership/management and internal staff or the funders themselves should play such role
DECI-2 Actors/Location: Funder, organizations, regional consultants
Level: Organizational and Regional
Sector: Information and Communication Technology (ICT) paired with UFE curriculum development
Time Period: (DECI-1: 2009-2011; DECi-2:2012-2015)
Documentation: E-mail Exchange with ECDG
DECI-2
Lessons Learned• Pairing ECD with capacity building in other technical sectors
reinforces learning and use;• Participatory curriculum development is lengthy but ensures
training uptake;• Two-level mentoring was effective: mentoring to partners and
mentoring to regional mentors (“at their own pace” without a classic workshop approach on pre-established dates );
• Mostly virtual + 2 face-to-face mentoring sessions;• Timeliness of support enhanced and upfront investment to ensure
organizational readiness;• Presented as a research project: joint learning- facilitator of
learning process NOT external expert;
Habitat For Humanity - LAC
• Actors/Location: HQ staff, regional managers, national technical specialists (planning and implementation officers) in Central and South America
• Level: HQ, Regional and National
• Sector: Housing Solutions, Community Development
• Time Period: 2009-2012
• Documentation: Direct Exchange with LAC Regional Office Coordinator
Lessons learned• The development of a qualitative instrument capturing outcome-
related information could become the entry point for a learning and collaborative ECD effort across levels;
• Getting managers and Implementers (not just researchers or evaluation specialists) involved in the development of a data collection tool made it possible to emphasize the need for timely and adequate resources as a prerequisite for better evidence;
• Balance between technical rigor (19 staff and 10 experts) and face validity is critical for capacity development within and outside an organization (the HFH tool was also validated through focus groups with target populations in four countries)
• Sustainability and scaling-up of the approach: this process will be replicated for the development of indicators on risk mitigation associated with HPC disaster response programs globally
Habitat For Humanity - LAC
Actors/Location: UN (IFAD), regional capacity building platform, indigenous population groups, evaluation professionals in the Latin American region
Level: national, regional, local
Sector: Agriculture, Rural Community Development
Time period: Participatory Research on the state of 15 projects M&E Systems (2004-2007) + Training on Innovative PME and Image-based M&E (2008-2010)
Documentation: Reflection Paper submitted by the Program Coordinator
PREVAL (Latin America)
PREVAL (Latin America)Lessons Learned• ECD could be a remarkable tool for social emancipation and an
effective enabler of transformational learning in marginalized communities• Joint Regional training aimed at both rural community leaders
and evaluation specialists enhanced the ECD process• The use of several communication means (talking maps,
thematic maps, photographs, info-graphs) and visits to indigenous communities enhanced inter-learning among the trainers’ workshop participants• The organization of peasant contests on who (within the
community) had used the evaluation findings more effectively than others was a good entry point for transfer of evaluation knowledge and skills in the following months
Ukk'aa – Alaskan ECD
• Actors/Location: ANZEA; indigenous research group from Massey University, NZ; Director & students at Interior-Aleutians Campus, Univ of Alaska Fairbanks; Evaluation Research Associates, Native Elders, Fulbright-scholar-in-residence; Ukk'aa - the Alaska Native group
• Level: Individual - statewide; organizational
• Sector: various
• Time period: 2008 – present
• Documentation: AJE article, email correspondence
Ukk'aa – Alaskan ECD
Lessons Learned• Interactions and collaboration among indigenous groups
promotes an authentic horizontal ECD process and collaborative learning
• Buy-in by native Elders critical to project success• Critical friends are often key enablers of ECD processes: New
Zealand Evaluation Association members (Maori) mentored Alaskan Native population on indigenous evaluation theory & practice
• A research grant might bring ECD actors to the table for the first time and enable the development of an ECD plan
Civic Theater (Kalamazoo, MI)
Actors/Location: Community theater, Kalamazoo, MI
Level: Organizational
Sector: Arts
Time period: ongoing
Documentation: Key informant interviews
Civic Theater (Kalamazoo, MI)Lessons Learned• The systemic integration of evaluation throughout an
organization where all of management acts as evaluators is a remarkable ECD strategy;
• Identifying and maximizing opportunities for conducting and using evaluation is a truly ECD-savvy strategy. Activities included:--Seeking informal feedback as patrons exit theater lobby;- -Using evaluation data for marketing purposes;- -Establishing a mechanism where staff and supervisors could evaluate each other;
--Evaluating the quality of vendors and software;
Quick Observations on the Process
The way a story was told was sometimes as illuminating as the content of the story. Two contrasting storytellers:• New Zealand (Maori) and Alaskan (Native) began telling their
story by introducing themselves, providing backgrounds and personal perspectives from participating in the ECD activity. • A different story of the creation of a new evaluation
association began by describing political/economic background as one of deficit, needing to be developed and improved.
What’s the Best Way Forward?
• Develop long-term collaborative relationships with funders as partners in the development process.
• Don’t Develop Capacity, work with actors who are WILLING and MOTIVATED to develop it further
• Work on at all three levels (individual, organizational and contextual) within your system
• Step back and let ownership emerge: Home grown, not externally driven - imposed by funders, gov't or others "forcing" evaluation on an institution/gov't.
• Need internal champions who are leaders/decision-makers and external champions who can act as critical friends/coaches/mentors.
What’s the Best Way Forward?• Give ECD a chance to take hold - long-term. Nonetheless, don't be
averse to change: tactics/strategies, partnerships, actors, policies, budgeting, training methods, and infrastructural arrangements
An evaluation system design should be appropriate to the needs of the institution (considering size, complexity, political environment, culture, etc.). Consider organizational design, job descriptionsevaluation approaches that are unique to meet those needs.
• Seeing the value of evaluation enhances its use/usefulness:providing evidence for policy-makers can dramatically affectresource Allocation
• Acknowledge the power of peer-to-peer support: indigenous-to-indigenous
“It takes time for an organisation to find its way into evaluation; and for evaluation to find its way into organisational functioning. This involves overcoming resistance, seeking best approaches, trial and error, staff training and exposure and eventual 'absorption' of the practices as part of ordinary functioning.”
Community Development Resource Assn. (CDRA), South Africa
QUOTE
Thank You Very Much!
Michele Tarsilla, Ph.D.E-mail: [email protected]
Karen [email protected]
Please follow us on:www.ecdg.net