JSWEC 2008 WorkshopJSWEC 2008 Workshop

David Gaylard & Sam Baeza David Gaylard & Sam Baeza

Acknowledgements: Acknowledgements: Phil Verrill (Head of Centre for Learning & Teaching)Phil Verrill (Head of Centre for Learning & Teaching)

Traditional evaluation methodsTraditional evaluation methods Tend to concentrate on the mechanics of learning & teaching

‘Modulisation’ of current curriculum's can fragment the student experience, promote a ‘surface’ approach to learning (as opposed to a ‘deep’ approach) and may hamper integrated learning.

Students complain of ‘evaluation burn-out’, as a consequence, the quality and amount of evaluation data declines.

have been described by some as a ritual and not always regarded as a

serious matter by either students or lecturers

students do not believe that universities take their feedback seriously, shown by poor response rates of evaluation forms, which in turn affects their reliability and validity

Provides little data that can actually be used for reflective development for teaching staff.

A growing recognition within HE that the student is the ‘consumer’ so their voice is increasingly important.

Student comments on existing evaluation processJust tick the boxes, it doesn't let you explain. I often feel

we evaluate content & entertainment not the

learning.

Dull, no one cares! Doesn’t let you express yourself and they’re

never totally anonymous.

Current method is really irritating – we

just write anything in order to finish the

forms quickly. I sometimes feel that nothing will improve from our feedback.

Having to do them for every module is a joke – it takes up

so much time and feels as though I am repeating

myself.

Aims of Whole Programme Evaluation: Aims of Whole Programme Evaluation: Creates a more transparent and holistic evaluation culture.

Takes as its starting point the centrality of the student voice.

Aims to enhance integrated and life-long student learning through reflective staff development.

Encourages student to give thoughtful and honest feedback that they value as much as the staff.

Provides a ‘rich picture’ of the student experience that enable teachers to reflect more deeply upon their own practice & consequent effect on student learning.

The whole programme processThe whole programme process

Involves a whole year group whole year group and is conducted face to face with students over a period of

2- 3hrs. 3 elements to the approach:3 elements to the approach:

1. 1. Student QuestionnaireStudent Questionnaire re:

● aspects of the modules that helped them learn;

● evaluation of the current evaluation methods;

● expectations & level of fulfilment of the course;

● students rank concerns in order of importance re: the first two areas

BREAKBREAK: during which questionnaires are scanned for major themes under each heading.

2. 2. Questionnaire discussionQuestionnaire discussion on issues raised in responses, enables clarification , probing of concerns to establish better validity and reliability of data, agreement from the whole group in terms of what was being reported.

3. 3. Students asked to rate staffStudents asked to rate staff (on a 5point scale) in terms of what staff did to help them learn and what they did to prevent them from learning. Students asked to justify what they were saying in terms of student learning - avoiding an ‘academic beauty contest’.

How is data usedAfter data analysis a report is produced, detailing

major issues for general learning and teaching development.

No data on individual lecturers revealed.Individual feedback to lecturers on their student

ratings given.

(i.) as raw data for lecturers to reflect on;

(ii.) a follow up session to discuss thoughts on data & any developmental plans made or support needed.

Student comments following a whole programme evaluationStudent comments following a whole programme evaluation

This process is better as we can be more detailed.

This is much better because there is no

pressure to rush

I feel you are listening more to our feelings and queries. Evaluations could be

improved by using forms such as this taking directed evaluation rather than just random comments

I find it much more valuable and easier to voice my opinion in

group meetings.

Some thoughts?

Are students competent enough to make such judgements?

Are ratings influenced by teachers’ popularity rather than their effectiveness?

Its applicationsIts applications There are a number of ways to access the student voice:

Mid – module evaluations or end of module evaluationsMid – module evaluations or end of module evaluations

Cross – module or cohort evaluationsCross – module or cohort evaluations

Personal interaction (‘Personal interaction (‘coffee & cigarettes’coffee & cigarettes’ approach) approach)

Practice placement evaluations involving students & Practice placement evaluations involving students & practice assessorspractice assessors

Whole programme reviewsWhole programme reviews

Programme Evaluation: Key Learning PointsProgramme Evaluation: Key Learning Points This process requires good trained & empathetic facilitators This process requires good trained & empathetic facilitators

staff and students must have confidence in facilitator staff and students must have confidence in facilitator

Related to development not individual performance Related to development not individual performance

Ownership of the process by students & staffOwnership of the process by students & staff

Prompt feedback to students and staffPrompt feedback to students and staff

Provides an effective mechanism for capturing data to make real Provides an effective mechanism for capturing data to make real programme development programme development

The process can be seen as cost - beneficial, transparent, The process can be seen as cost - beneficial, transparent, confidential & detached from any managerial agenda relating to confidential & detached from any managerial agenda relating to performanceperformance

Efficient from a students’ perspectiveEfficient from a students’ perspective

The ChallengesThe Challenges Time consuming: requires a 2nd person to capture data during the

session.

Depends upon the skill of the facilitator / evaluator

Requires the faculty, teacher, student to ‘buy into’ and engage with the process

Train student reps as evaluators to make process more time efficient

Teachers may withdraw - if uncomfortable.

Possible misinterpretation of themes / problems.

Danger of concentrating on ‘ranking’ instead of developing teachers attitudes towards feedback.

ReferencesReferences Campbell C. et al (2007) ‘Hearing the student voice: promoting & encouraging the

effective use of the student voice to enhance professional development, teaching & assessment within higher education’ Final Report, Napier University, Edinburgh, Pub.

Crisp B.R. Lister P.G. (2002) ‘Assessment methods in social work education : a review of the literature’ in Social Work Education, Vol.21, No.2, 2002 p.259 -269

Darby, J. A. (2007). "Evaluating course evaluations: the need to establish what is being measured." Assessment and evaluation in higher education 32(4): 441-455.

Morley, L. (2003). Quality and Power in Higher Education. Maidenhead, Open University Press.

Richardson, T. E. (2005). "Instruments for obtaining student feedback: a review of the literature." Assessment and evaluation in higher education 30(4): 387- 415.

Ticker T., Rangecroft M. & Long P. (2005) ‘Bridging the gap: an alternative tool for course evaluation’ in Open Learning,Vol.20,No.2 June p.185-192

Verrill P. (2007) ‘Voices Odd? Using student voice for reflective staff development’, Hearing the student voice, project case study.

Useful website: www.napier.ac.uk/studentvoices