elizabeth godfrey 1. periodic assessment of results appropriateness, effectiveness, efficiency,...
TRANSCRIPT
Evaluating the success and sustainability of curriculum and
pedagogical innovation
Elizabeth Godfrey
1
Periodic assessment of results• Appropriateness, effectiveness, efficiency, impact, sustainability
Identifies intended and unintended effects Identifies what worked and what didn’t Provides level of judgement about the overall worth of
intervention Key to project improvement Influence decision-making or policy formulation through
the provision of empirically-driven feedback sustainability
Evaluation – gathering of information to understand and make judgements about the project and its outcomes
What is the purpose and scope of the evaluation?
Why is the evaluation being done? How will the information be used? What evaluation form and approach will be most
suitable for this study?
Formative – monitoring progress to improve approaches
Summative – overall perspective – focus on value or worth of project and designed for accountability or continuation purposes
4
Evaluation and assessment have many meanings…one definition:
• Assessment - is gathering evidence
• Evaluation - is interpreting data and making value judgements
Examples of assessment and evaluation• Individual’s performance (grading)
• Program’s effectiveness (accreditation)
• Project’s progress and success (monitoring and validating)
Evaluation and Assessment
Effective evaluation starts with carefully defined project
goals and expected outcomes Goals provide overarching statements of project intention
• What is your overall ambition? • What do you hope to achieve?
Expected outcomes identify specific observable results for each goal
• How will achieving your “intention” reflect changes in student behavior?
• How will it change their learning and their attitudes?
Evaluation and Project Goals/Outcomes
Goals → Expected outcomes → Evaluation questions
Questions form the basis of the evaluation process
Evaluation process collects and interprets data to answer evaluation questions
Goals, Expected Outcomes, and Evaluation Questions
To what extent has the project been implemented as planned? How well has the project been co-ordinated across different institutions? To what extent have the intended outcomes been achieved? Were there any unintended outcomes? How well have the needs of staff been met? To what extent have the intended student learning outcomes been achieved? What measures if any have been put in place to promote sustainability of the
project’s focus and outcomes? Are students better able to describe the effects of changing some feature in a simple
problem as a result of the intervention? What lessons have been learned from this project and how might these be of
assistance to other institutions?
Evaluation Questions like…
How will the information be collected, analysed and presented?
What type of data is most appropriate?
What are the most appropriate methods of data collection?
How will data be analysed and presented?
What ethical issues are involved in the evaluation?
Surveys • Forced choice or open-ended responses
Observations• Actually monitor and evaluate behavior
Interviews• Structured (fixed questions) or in-depth (free flowing)
Concept Inventories• Multiple-choice questions to measure conceptual understanding
Rubrics for analyzing student products• Guides for scoring student reports, test, etc.
Focus groups• Similar to interviews but with group interaction
Examples of Tools for Evaluating Learning Outcomes
An Intelligent Tutoring System for Engineering Mechanics Tanja Mitrovic, Charles Fleischmann, Brent Martin, Pramudi Suraweera
The planned evaluation will focus on the following questions: Is the developed ITS effective? Does it support learning better than the traditional approach? Does the system increase students motivation? �We will compare performances of two groups of students. One group will
learn the material in the traditional way, via lectures and tutorials. The other group will attend lectures, but the tutorials would be replaced by interaction with the system. Time will be controlled. We will require students to sit pre- and post-tests, to measure their knowledge. We will also collect data about their actions while solving problems, and analyze the data.
A NZ exemplar
Judgements will be required for each key evaluation question
Need to develop indicators or targets - standards and levels regarded as acceptable
Could use benchmarks – location or source of best practice for comparison
What are the criteria for making judgements about the project and its outcomes?
Who?
key question underpinning the budget
Should it be an individual or a team? Insiders or outsiders? Skills – qualitative and/or quantitative Prepare an evaluation brief or terms of reference When to involve the evaluator?
What resources and skills are required to conduct the evaluation?
NSF’s User Friendly Handbook for Project Evaluation http://www.nsf.gov/pubs/2002/nsf02057/start.htm
Australian Learning and Teaching Council Grants Scheme – Evaluating Projectshttp://www.altc.edu.au/extras/altc-gsep/evaluation_plan.pdf
Student Assessment of Their Learning Gains (SALG)http://www.salgsite.org/
Other Sources
Think about evaluation, purpose and scope from the beginning of the project
Ensure appropriate data is collected eg pre/post intervention
Evaluation provides supporting evidence to influence decision-making or policy formulation through the provision of empirically-driven feedback sustainability
Summary
Questions…
