Download - Interrogating evaluation 2015 induction
Aim of this part of the session
• Interrogate ‘evaluation’ from own
perspective
• Come up with an evaluation
strategy
Common Purpose of evaluation
• Required by universities near end of
each course *
• Can be used by
administration/departments as an
important element in making
decisions about promotion
• Could be a source of great pride or
trepidation
(Cashin, 1999; Clayson, 2009)
Purposes of evaluation
auditdevelopment
“quality assurance”
(Biggs, 2003; Edström, 2008; Patton, 1997)
appraising teachers
developing/ improving courses and teaching effectiveness
“quality enhancement”
What should the focus of the
evaluation be?
What should the focus of the
evaluation be?
• To check that something is working
What is that something ?(e.g. course,
degree programme, activity)
What do we mean by ‘working’ ? (what goal
should be achieved?)
• To figure out how ‘it’ can be improved
What should the focus of the
evaluation be?
Teaching
( teacher)
Learning (outcomes)
Teaching (process)
Learning
(process)
“Rather than ratings, teachers
should be asked to include
their course analyses in their
teaching portfolio in order to
show their ability to both
analyze the student learning
experience and the quality of
the student learning outcomes ,
and to improve these with
adequate course development
measures”
(Edström, 2008, p.104).
“Evaluation is often viewed as a test of effectiveness – of materials, teaching methods or whatnot - but this is the least important aspect of it. The most important is to provide intelligence on how to improve these things”
What should the purpose of
evaluation be ?
( Bruner, 1966, as cited in Ramsden, 2003, p. 233).
Typical questions
• Have student attitudes changed?
• Has the lecturer approaches changed?
• Have student learnt something in class?
• What is happening in class?
• Have classroom practices changed?
• Are students engaged?
• Am I meeting students’ needs?
Evaluation strategies – student
attitudes & perceptions
Have student attitudes changed?
• Likert-scale questions & surveys
• Could be used to measure the effect of a
course, a degree or a long term change
over a number of years
Evaluation strategies – Lecturer
approaches
Have lecturer approaches to teaching
changed?
• Teaching Practices Inventories
• Can help departments reflect on teaching,
by allowing comparisons between different
courses/departments
Evaluation strategies – Student
learning
Have students learned anything?
• Measure student performance
• Important to standardise testing (using
locally developed or already developed
tools)
• Important to conduct pre and post-testing,
before and after a learning intervention
Is there evidence that the Graduate Attributes
have been adopted by the students?
Evaluation strategies – classroom
practices
What is going on in the class?
• Video recordings
• Peer observations (fellow lecturers) using
observation protocols
• This data can be used in your Teaching
Portfolio
Have classroom practices changed?
• Use of innovative teaching & learning
practices
• Providing useful feedback etc.
Evaluation strategies – student
engagement
Are students engaged?
• Observation of about 10 students to see if
they are engaged by recording student
activity
• What teaching & learning activities result in
high levels of student engagement?
Evaluation strategies – student
learning needs
Are students learning needs being met?
• Focus groups or Interviews
• Anonymous surveys/midterm surveys
(online, or on paper) – letter to the
facilitator
• Assignment by assignment survey (time it
took, approach, resources used etc. )
• Keep, Start, Stop
Kirkpatrick’s (1994) four level model
Level What is measured Examples
1 Reaction - changes in perception , satisfaction levels
How students feel about the learning experience
Feedback forms
2 Learning – changes in knowledge, skills, attitudes
Increase in student knowledge and skills
Informal/informal assessment before and after learning interventions
3 Changes in behavior/practices
How far learning is applied in practice resulting in personal changes
Observations and interviews of students over time
4 Results - noticeable changes in results or conditions.
How far the module/course impacts on program or institutional factors ( student performance, retention, throughput).
Use institutional data to identify whether the program/module shifts the nature of student participation/ performance/engagement.Student surveys.
Some Drawbacks
• Students are not always good at evaluating
teaching effectiveness. Popularity is usually
mistaken for good teaching
• Response rate is always less than 100%
• Susceptible to bias - voluntary
participation/polarization
• Questions are usually general –missing the
finer details of practice
The UWC Evaluation Guideline document
Instruments
GENERIC
EVALUATION DISCIPLINE
SPECIFIC
Question Bank
A standard set of questions that
students are invited to answer
anonymously at the end of each
course. ( often, the results are given
weight in promotion and tenure
decisions)
Run centrally and completed by all
students to provide the institution with
data on how students are performing
with regards to institutional strategies
and plans
Your evaluation strategy
Which evaluation methods could you consider
implementing in your teaching?
References
• Biggs, J. (2003). Teaching for quality learning at university: What the
student does, Buckingham, UK: SRHE and Open University Press.
• Cashin, W. E. (1999). Student ratings of teaching: uses and misuses. Ch 2,
In P. Seldin and Associates (Eds). Changing practices in Evaluating
Teaching: A practical guide to improved faculty performance and
promotion/tenure decisions. Pp 25-44. Bolton, MA. Anker Publishing.
• Clayson, D. E. (2009). Student evaluations of teaching: Are they related to
what students learn? A meta-analysis and review of the literature. Journal
of Marketing Education. 31 (1), 16-30.
• Edström, K (2008), Doing course evaluation as if it matters most. Higher
Education Research and Development, 27 :2: 95-106
• Kirkpatrick, D.L. (1994). Evaluating Training Programs. The Four Levels.
San Francisco: Berrett- Koehler.
• Ramsden, P (2003). Learning to teach in higher education ( 2nd edition).
London: Routledge Falmer.
• Schimpf, N (2015) Evidence-based Active Learning:
Evaluating Classroom Practices. Presentation made at the University of the
Western Cape
Thank you