evaluation: a means to gain insights into and improve the roer4d project
TRANSCRIPT
Sarah GoodierROER4D Evaluation Advisor
EDN4502W: Research & Evaluation of Emerging Technologies9 September 2015
Evaluation: a means to gain insights into and improve the ROER4D project
9/11/20151
WHAT IS EVALUATION?
… the systematic determination of the quality or value of something (Scriven, 1991)
Evaluation is…
Scriven, M. (1991). Evaluation Thesaurus. Sage.
… the systematic determination of the quality or value of something (Scriven, 1991)
• It is done according to a specific plan/method, in a thoughtful way
Evaluation is…
Scriven, M. (1991). Evaluation Thesaurus. Sage.
… the systematic determination of the quality or value of something (Scriven, 1991)
• It is done according to a specific plan/method, in a thoughtful way
• It measures merit/worth
Evaluation is…
Scriven, M. (1991). Evaluation Thesaurus. Sage.
… the systematic determination of the quality or value of something (Scriven, 1991)
• It is done according to a specific plan/method, in a thoughtful way
• It measures merit/worth• It is focused on a specific
programme/intervention/technology
Evaluation is…
Scriven, M. (1991). Evaluation Thesaurus. Sage.
• Assessment of merit or worth Are specific standards met?
• ImprovementWhat can be improved? And how?
• Oversight and compliance Is the technology cost-effective?
The purposes of evaluation
8
Research vs Evaluation Add knowledge to a specific field
Modified from: http://aea365.org/blog/john-lavelle-on-describing-evaluation/
Purpose of the study Make judgments/decisions in a specific context
Who decides what you study?
Researcher derives the questions (researcher-focused)
Funder/programme specifies the questions
Testing a hypothesis/hypotheses Answering evaluation questions
Social science methods & analyses
Results
The questions
Sharing findings
Research findings (and some explanation)
Findings inform recommendations based on evaluation questions
Publish – share with research community in the field
Evaluation report to stakeholders (may also publish)
Exercise:Come up with a example of 1) Research and 2)
Evaluation that show the difference between the two
(See the previous slide)
EVALUATING ROER4D
in the Global South
In what ways, and under what circumstances can the adoption of OER address the increasing demand for accessible, relevant, high-quality and affordable education and what is its impact in the Global South?
Research on OER for Development
1. Build an empirical knowledge base on the use and impact of OER in education2. Develop the capacity of OER researchers3. Build a network of OER scholars4. Communicate research to inform education policy and practice5. Curate output as open content
ROER4D Objectives
1. Build an empirical knowledge base on the use and impact of OER in education2. Develop the capacity of OER researchers3. Build a network of OER scholars4. Communicate research to inform education policy and practice5. Curate output as open content
ROER4D Key Evaluation Areas
EVA
LUAT
ION
ROER4D Evaluation and Communication supported by DECI-2
DECI-2An IDRC funded research project to build and mentor Communication and Evaluation for IDRC
flagship projects
http://evaluationandcommunicationinpractice.net/
UTILIZATION FOCUSED EVALUATION (UFE)
What is utilization focused evaluation (UFE)
UFE is centered around intended use by intended users
What do the users (key stakeholders) want to know?
How will answers to the KEQs help to improve the project?
Remember that evaluation is
about assessing worth of
something
UFE in 12 steps:
1. Assessing program readiness2. Assessing evaluator readiness3. Identifying primary intended users4. Situational analysis5. Identification of primary intended uses6. Focusing the evaluation7. Evaluation design8. Simulation of use9. Data collection10. Data analysis11. Facilitation of use12. Meta evaluation
Steps in the Utilization Focused
Evaluation (UFE)
process
UFE in 12 steps:
1. Assessing program readiness2. Assessing evaluator readiness3. Identifying primary intended users4. Situational analysis5. Identification of primary intended uses6. Focusing the evaluation7. Evaluation design8. Simulation of use9. Data collection10. Data analysis11. Facilitation of use12. Meta evaluation
Progress towards
fine-tuning evaluation
Steps are iterative, not linear
In progress
UFE in 12 steps:
1. Assessing program readiness2. Assessing evaluator readiness3. Identifying primary intended users4. Situational analysis5. Identification of primary intended uses6. Focusing the evaluation7. Evaluation design8. Simulation of use9. Data collection10. Data analysis11. Facilitation of use12. Meta evaluation
Progress towards
fine-tuning evaluation
In progress
Different ROER4D
objectives are at different
stages in the UFE process
DEVELOPING AN EVALUATION STRATEGY
ROER4D Process of developing evaluation strategy1. Understand what is needed in terms of the scope of evaluating the
ROER4D project – the evaluation work is iterative by nature.2. In collaboration with the ROER4D network hub team, formulate an
evaluation plan, including what to evaluate and how.> The experience of the evaluation process and the effect this has is a key component of the evaluation.
3. Get feedback from DECI-2 around the evaluation work and incorporate this into the process.
ROER4D Process of developing evaluation strategy
ROER4D Process of developing evaluation strategy
ROER4D Process of developing evaluation strategy4. Connect with members of the ROER4D, where needed (surveys, interviews, etc.)5. Assess the findings.6. Share products of the evaluation work (e.g. slides around process and
results, reports, etc.) timeously to allow the findings and recommendations to effect change.
7. Be aware of all components of the evaluation work and collaborate/ share information where possible and where needed.
1. Build an empirical knowledge base on the use and impact of OER in education2. Develop the capacity of OER researchers3. Build a network of OER scholars4. Communicate research to inform education policy and practice5. Curate output as open content
ROER4D Key Evaluation Areas – involving technology use
Webinars
Social media
UFE in 12 steps:
1. Assessing program readiness2. Assessing evaluator readiness3. Identifying primary intended users4. Situational analysis5. Identification of primary intended uses6. Focusing the evaluation7. Evaluation design8. Simulation of use9. Data collection10. Data analysis11. Facilitation of use12. Meta evaluation
In progress
Focus on:• Data collection
and analysis• Facilitation of
use by the ROER4D team
Progress in the evaluation
Next steps More data collection and analysis (ongoing for webinars, Twitter as well as other
social media platforms)
Feedback from the facilitation of use into refinements in the identification of primary intended uses and users to help focus the evaluation further
Constant dialogue with the ROER4D team and PIUs
SUMMARY
… the systematic determination of the quality or value of something (Scriven, 1991)
• It is done according to a specific plan/method, in a thoughtful way
• It measures merit/worth• It is focused on a specific
programme/intervention/technology
Evaluation is…Scriven, M. (1991). Evaluation Thesaurus. Sage.
• Prepare an evaluation plan
• Choose what to measure and how (indicators)
• Collect the data (evidence)
Research and Evaluation are differentbut share methods
Some lessons from evaluating ROER4D Find a framework for your evaluation (e.g. UFE)
For both internal evaluation and external evaluations, never underestimate the importance of iterative engagement – more engagement = better understanding of the project and what matters to the evaluation users
Keep feeding back from the facilitation of use into refinements in the evaluation to help focus the evaluation further
Further reading:About ROER4D:• Hodgkinson-Williams, C. (2013). Research on Open Educational
Resources for Development in Post-secondary Education in the Global South (ROER4D) - Scoping Document. Available online: http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015].
• Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015].
About evaluation:• Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage
Publications Inc.• Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A
Primer for Evaluators. Penang: Southbound.• Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications
Inc.
Links and license
This work is licensed under a Creative Commons Attribution 4.0 International License.
Website: http://roer4d.org
Contact author: Sarah Goodier [email protected] @SarahGoodier
Excluding images, screenshots and logos and/or unless otherwise indicated on content