evaluation pgcert in learning and teaching module 402: course design and evaluation

Evaluation

PGCert in Learning and Teaching

Module 402:Course Design and

Evaluation

Learning outcomeDesign effective evaluation strategies through a critical comparison of alternative strategiesOrWhat forms of evaluation occur in your

module/ programme nowWhat approaches are used in other

programmesWhat are the strengths/weakness of

‘your’ system◦ In comparison with others and with a model

What might you adapt to improve evaluation practice in your module/ programme

Definitions Evaluation is:

A process whereby a teacher, group of students, or their institution,

find out, on the basis of their own perceptions, the overall quality of a programme, a particular course, or a teacher

generally part of a formative process aimed at improving standards. heacademy.ac.uk

Who

Does what

Why

Some key pointsEvaluation is more than student

feedback◦ But includes student feedback as a key source of

evidence◦ Here student feedback means feedback from

students, about the programme, not feedback to students, about their work

USA uses assessment to mean evaluation (UK)

In general:◦ Module evaluation tends to focus on student

feedback◦ Programme evaluation should have a wider more

systematic approach◦ You need to consider both levels in the assignment

Why find out moreTo find out about:Student learningStudent difficultiesYour teaching

◦ Strengths and weaknesses

To engage students in their learning experience

To provide evidence for:Your professionalismQuality control

◦ Organisational requirement Morss & Murray 2005

Why find out moreWhat systems are there in your

area for finding out about teaching, learning, student perceptions for your module

Articulating your own (departmental) practice and listening to what others do is one step toward the assignment's ‘critical comparison’

Evaluation: The systemEvaluation is more than student

feedbackThere are many sources of

informationIt can be thought of as a process

◦CircularUsually driven by the institution

◦Often applied differently in different departments

◦Regularly applied to modules & programmes as well as to individual lecturers

The evaluation cycle

5 Agree actionNotify students, staff

6 Implement changesReport

3 Gather: Data, feedback

2 Devise: Strategy - method(s), timing

4 Analyse & interpret

1 Decide: Reasons, purpose, level

HEAcademy.ac.uk

1 Purpose of EvaluationMain purpose:

• enhance students’ experience of learning and teaching• contribute to monitoring and review of quality and standardsAlso contributes to:• effectiveness of course design and delivery• dialogue with students• students reflecting on their experiences• enhancing the teaching and learning process• identifying good practice• measuring student satisfaction

which can lead to marketing (recent development)

• staff developmentheacademy.ac.uk

Question:Which of these predominate in your area

‘Level’ of the evaluation- where is the focus?

Individual session(s) – delivery, learningModule – delivery,

administration/resourcesProgramme – delivery, integration,

overallUniversity – services

Critical questionsIn your area: Which of these is evaluated as a ruleHow is it done and who causes it to be doneWho gets any value

2 Devise the strategySources of information, and timing

Sources of information:Students:

◦ Feedback◦ Staff student liaison committee◦ Performance in assessment◦ National Student Survey – increasingly

importantSelf, peers, mentors, external

examiners, reviewsExternal sources:

◦ Employers and Professional bodies◦ Other universities’ courses◦ League tables

Question:Which of these has been useful or impacted on your work

When to evaluate - timingTo benefit current students: After an individual session Within a module - for immediate feedback on

comprehension, interest, perception of relevance, workload

To benefit future students: At the end of a module - to gauge whether

learning outcomes have been met, and potential changes for the next time

At the end of the year? Every 5 years? (UK systems)

Which of these are you aware of/ benefit from?

3 Gather: How and what process?

Method:Questionnaires (hard copy and electronic)Focus group – various approachesStudent assessment resultsPeer reviewCATs - Classroom Assessment TechniquesAny other way?Process:Who will collect this data

◦ When - in a session, after it, ◦ What format – paper forms, summary report,

electronic??

Questionnaires: rationale

Decisions to be made:◦Quantitative

◦ for comparison between modules/teachers and over time

◦Qualitative ◦ to allow more in-depth and tailored responses

◦Open or closed questions

◦Hard copy or VLE/ web based (Survey Monkey)

◦Repeating questions from previous years (for comparison) or changing the questions to give new insights

◦Aligning questions to the NSS

Questionnaires: design

Bear in mind:

◦ Is the purpose clearly stated

◦ Are the questions unambiguous. Are the questions answerable

◦ Are they phrased in a way that is easily understood

◦ Is there a mix of open and closed questions

◦ How many questions are being asked

◦ Is the Likert scale being used appropriately?

(adapted from Brennan and Williams 2004)

NB questions themselves are often not researched or evaluated

See the resource in vital: ‘Questions to consider when critiquing the evaluation questionnaires’

4 Analyse - interpretHow are paper responses

collated – put into electronic format?

How analysed?How are qualitative comments

analysed and reported?Who does all this?

Question:What happens in your area?

5 Agree action

What happens to the results of the evaluation?

How do the students or providers of feedback know what will be changed for next time?

Where is the commitment to change?Who is responsible for ensuring that

change happens?Question:In your area: How does this happen?Does it ‘work’?

Change the deliveryUpdate the curriculumProvide opportunities for staff

developmentClarify expectations

look at the student handbook!Update resourcesTell the Students!!

6 Implement the changes

Question:Who does all this in your area?

How can we get students to respond and take it seriously?

Make time in sessions

Require attendance/ completion

Make it easy (e.g. Vital)

Don’t overload - too many questions, too often

Vary evaluation methods

Written comments rather than tick boxes?

Give examples of constructive feedback

Establish ground rules

Give feedback e.g. on notice boards, Vital

• Demonstrate the effect

Question:Which of these is used in your areaWhich could be?

Coming soon……..University purchase of Evasys

◦Will change evaluation practice

Increasing responsiveness to NSS◦Becoming more thoughtful, less

reactive?

NSS Questions: areas & e.gs.

The teaching on my courseStaff have made the subject interesting

Assessment and FeedbackFeedback on my work has helped me clarify things I did not understand

Academic supportI have received sufficient advice and support with my studies

Organisation & ManagementThe course is well organised and is running smoothly

Learning ResourcesThe library resources and services are good enough for my needs

Personal developmentI feel confident in tackling unfamiliar problems

Overall satisfactionOverall, I am satisfied with the quality of the course

Possible additions to NSSAcademic challenge/reflective and

integrative learning, e.g. ◦My course has challenged me to achieve my best

The learning community/ collaborative learning 

◦I have not been encouraged to talk about academic ideas with other students

Student voice ◦Staff appear to value the course feedback given by students

I have been encouraged to use technology to enhance my learning

HEFCE 2014

Alternatives

UK variation on National Survey of Student Engagement; NSSE (US & Aus)

◦ Gibbs (2012): “NSSE provides a more valid predictor of student gains than does NSS”

◦ See HEA website ‘Engagement for enhancement’ & ‘Institutional case studies (NSSE pilot)’

What other processes contribute to evaluation?

Recap: the university has other processes beyond student feedback to evaluate the quality of programmes◦ Annual Subject Review report◦ Internal Periodic Review◦ Professional Body Review◦ External Examiners◦ Institutional Audit – QAA (November 2015)◦ Required responses to NSS

see Student Experience Committees

Have you experience of theseHave you seen value from them

Learning outcomeDesign effective evaluation strategies through a critical comparison of alternative strategiesOrWhat forms of evaluation occur in your

module/ programme nowWhat approaches are used in other

programmesWhat are the strengths/weakness of

‘your’ system◦ In comparison with others and with a model

What might you adapt to improve evaluation practice in your module/ programme

QuestionsIn your

department/programme/team how effectively are evaluation processes leading to change

Consider one major change that you could make to evaluation in your area

What aspects of ‘your’ evaluation systems would you recommend to others

Use the questions on the slides as prompts for your assignment

Resources Angelo, T., & Cross, P. (1993) Classroom Assessment Techniques: A Handbook for

College Teachers, 2nd edition Brennan, J & Williams, R.(2004) Collecting and using student feedback

www.heacademy.ac.uk/ First words on Evaluation- Oxford Brookes University

www.brookes.ac.uk/services/ocsd/firstwords/ Haugen, L, (1999) Classroom Assessment Techniques (CATs), Iowa State

University http://www.celt.iastate.edu/teaching/cat.html HEFCE (2014) NSS Review https://

www.hefce.ac.uk/pubs/rereports/Year/2014/nssreview/Title,92164,en.html Macdonald, R. (2006) The use of evaluation to improve practice in learning and

teaching. Innovations in Education and Teaching International Vol. 43, No. 1, February 2006, pp. 3–13

Morss, K., & Murray, R. (2005) Teaching at University: A Guide for Postgraduates and Researchers

Shevlin M., Banyard P., Davies M., & Griffiths M. (2000) The Validity of Student Evaluation of Teaching in Higher Education: love me, love my lectures? Assessment & Evaluation in Higher Education, Volume 25, Number 4, 1 December 2000, pp. 397-405(9)

Why find out more

Demonstrating evidence based

practice

Improving student experience

Improving your experience!

Raising profile (yours)

Supporting change

One-Minute Paper

Course Knowledge and SkillsDuring last few minutes of class

period, ask students to use a half-sheet of paper and write "Most important thing I learned today and what I understood least."

Review before next class meeting and use to clarify, correct, or elaborate.

Haugen 1999 (from Angelo & Cross 1993)

Muddiest Point

Course Knowledge and SkillsSimilar to One-Minute Paper but

only ask students to describe what they didn't understand and what they think might help.

Same as One-Minute Paper. If many had the same problem, try another approach.

Haugen 1999 (from Angelo & Cross 1993)

How can you gather evidenceAnd what counts as evidence

Your own evidence:Teaching diary: your record,

responses, reflections, rationales, observation of students, informal notes (e.g. on ppt slides)

Video or audio tapingChecklistLearning log

◦Goals and achievements (may be shared)

VariousTeaching methodsQuestionnaires

◦ Standardised◦ Designed by you for specific purposes◦ Input from students

Focus groups◦ Run by neutral person

Staff- student liaison committeesPeer review

◦ Peer observation; mentor; supervisor