oral assessment discussion paper

A discussion paper on oral assessment prepared for the Assessment Subcommittee of the UQ Teaching and Learning Committee

IntroductionOral assessments have long been associated with disciplines such as languages and health sciences but are now receiving wider attention because of the increased recognition of the importance of developing students who can claim effective oral communication skills among their attributes on graduation. As this discussion paper indicates, oral assessment has particular advantages and disadvantages and, as with other forms of assessment, certain conditions must be met to ensure that it is conducted in ways that allow judgements that are consistent, defensible and transparent. Joughin and Collom (2003) argue that oral assessment “has been the subject of little research and only occasional reflection” so it is timely for the Assessment Subcommittee to direct its attention to this area in order to provide guidance to the UQ assessment community.

What is oral assessment?

Oral assessments are tasks designed to provide students with opportunities to develop and demonstrate their command of (1) an oral medium, and/or (2) of content as demonstrated through the oral medium (adapted from Heywood, 2000 p. 316).

Why use oral assessment?

Oral communication skills are now a key part of what any university program aims to develop in its students and “Effective communication” is therefore one of the University of Queensland's graduate attributes. As well as being essential in many personal and professional contexts, the capacity to express, justify and defend ideas and opinions, to engage in discussion with peers and to learn from the ideas of others are some of the most significant aspects of the university experience. However, unless oral communication skills are explicitly addressed through course and program assessment there is a strong possibility that their development will be assumed rather than systematically addressed. As assessment is generally believed to define the curriculum for students, when oral communication skills are omitted from the formal assessment program it is unlikely that students will invest time or effort in their acquisition.

Benefits of oral assessment

Joughin and Collom (2003) identify the key benefits of oral assessment as: allowing students to research, prepare and present findings in a

context similar to that of their future work lending itself readily to group presentations, thereby making it well

suited to those group-base assessment tasks that reflect ‘real life’ teamwork.

providing an assessment experience that is more personal, more challenging, and more engaging than other forms of assessment

allowing academics to probe understanding through follow-up questions which encourages deep approaches to learning.

balancing and developing student strengths. Some students will perform better in oral than in written formats. For other students, oral assessment provides an opportunity to develop important oral skills.

countering plagiarism. The questioning component of oral assessment discourages plagiarism and allows a quick check on who has contributed appropriately to group work; while a short oral component attached to written work can confirm student authorship.

In addition Joughin’s (1999) research found that the particular qualities of oral

Dr Clair Hughes UQ/TEDI April, 2008

assessment have the potential to exert a positive influence on student learning in that students: perceive a greater need to understand the material they are studying

when preparing for oral rather than written forms of assessment when they expect their understanding to be probed through questioning either by peers or other assessors

are less inclined to present ideas they do not understand report oral assessment as a more personal experience which makes

them feel more ownership of the material presented and more accountable to the assessor.

Orals also have great feedback potential and marking can be fast.

Oral assessment and criterion referenced approaches

Effective oral assessment, like all other modes of assessment must adhere to the UQ criterion-referenced approach: In particular: assessment tasks must be congruent with course learning objectives no single assessment item can contribute more than 70% of the final

grade two different forms of assessment must be used (however this would

not preclude oral assessment forming 100% of the assessment mode providing two different forms of oral assessment are used)

students must be provided in advance with the criteria and standards to be used as the basis of assessment judgements

feedback must be provided on all pieces of assessment, feedback must be timely and offer students constructive advice on how to work towards intended course learning outcomes.

Planning for oral assessment

The planning of oral assessments involves program and course coordinators in decisions around the following elements of assessment tasks and associated teaching.

Purpose:What is the purpose of the oral assessment – cognitive abilities as described in Bloom’s revised taxonomy (Krathwohl, 2002), tutorial preparation, interpersonal competence: personal qualities? Which of the course learning objectives will students be developing and demonstrating through participation in the task? Does the development of the oral assessment identify any gaps or lack of clarity end in the list of course learning objectives?

Form of oral communication:What forms of oral communication are traditionally associated with this discipline - debate, history-taking, provision of advice or instruction? Which form will best suit the purposes of assessment in this course? (See the following section for further possibilities)

Subject matterWhat will be the subject matter or topic of the oral assessment?

Role or position of student Will the student be asked to assume a particular role (patient, customer, interviewer, chair) or perspective on the subject matter (in favour of or against a certain proposition or stance on an issue)?


Interaction with audience and/or assessorWho will be the audience of the oral assessment – peers, assessor, ‘authentic’ audience from industry or profession, other, none? Who will make the assessment judgement – peer/s, teacher, tutor or other such as industry representative or client? What level of interaction will be encouraged/permitted between students and audience – presentation with no interaction, presentation followed by questions or critique, continuous dialogue between assessor and student?

MediumWill the oral presentation the face-to-face or will some audio/visual equipment be used?

ModeWill the assessment task be completely oral or will other modes of communication be incorporated– a poster, presentation slides or other visual aids or a requirement for an accompanying paper?

Criteria and standardsCareful thought to all of the elements of assessment task design will make it much easier to identify the criteria and standards that will serve as a basis for assessment judgements. Criteria and standards for oral assessment tasks can be similar to or even the same as those developed for many written tasks – especially in relation to cognitive abilities such as understanding, critical thinking and so on.

When developing criteria and standards for oral tasks, coordinators need to focus on the reason for choosing an oral task in the first place –empathy, team work, communication o f ideas in oral mode, flexible or spontaneous thinking – and to then ensure these are prioritised as the basis of assessment judgements.

The use of criteria and standards when tutorial participation is a component of summit of assessment can be problematic. Criteria such as preparation and/or quality of contribution and interaction are difficult to apply while facilitating tutorial activities and that this reason tutorial attendance is often used as a proxy measure of participation. It is therefore worthwhile to investigate alternative methods such as peer assessment goes efficiencies in the assessment of valued tutorial behaviours to avoid rewarding students turning up but making no other contributions to either their own or others’ learning.

Teaching and learningAs for any other form of assessment, when oral assessment is used coordinators need to determine: what will students need to do to undertake these tasks effectively? what will teachers need to do to support students in developing the

required learning?

Students will therefore need opportunities to reach an understanding of what is required of them in performing oral assessment tasks and also opportunities to practise and receive feedback in formative contexts. While it is fair to assume that students have some experience of oral assessment by the time they reach university, it cannot be assumed that they will have equal facility with all types


of oral tasks that are assigned. Some unfamiliar oral tasks, especially those characteristic of professional disciplines will need systematic and explicit development through planned teaching episodes.

FeedbackThough oral assessment expedites the provision of immediate feedback to some extent, it also poses some problems. Making and recording assessment judgements while observing oral assessment is a challenging task, especially if the assessor is also an active participant in an interactive assessment process. Efficiencies such as easily annotated rubrics and checklists can support the provision of feedback. Students can engage effectively in peer feedback provided their skills in this area are developed through appropriate teaching activities.

Forms of oral assessment

Oral assessment can take many forms. When deciding on an appropriate form coordinators may draw on the ‘signature’ or authentic forms of communication of a particular discipline. However there is no need to be restricted to these, particularly as they are not always feasible in the academic context. Well-chosen, innovative forms of assessment can often be highly motivational while still providing students with authentic opportunities to demonstrate intended course learning outcomes. It is also worth considering combinations of tasks (e.g. a prepared presentation followed by questions from an audience - with both elements assessed) or combinations of modes (e.g. an oral presentation of a poster with criteria relating to both the presentation and the visual elements of the poster). Some alternatives include: prepared presentations - sometimes followed by questioning or

critique interview – e.g. role-play by two or more students, between tutor

and student role-play, with or without a structured (assessable) de-briefing

component forms of clinical practice – e.g. history taking, presentation and

defence of diagnosis stimulated recall based on patient charts, lesson plans random selection of a problem-based scenario from a set of

scenarios provided to students for preparation more complex forms such as the triple-jump developed as part of the

McMaster problem-based curriculum (Heywood, p. 318) consisting of three components:

o a vignette or clinical problem to test hypothesis generation, data gathering and interpretation, current knowledge, early problem formulation, and the ability to define learning issues

o the intermediary step comprised of searching the literature, using either a personal or institutional library, or other learning resources

o a final component, permitting the learner to present the final conclusions about the case and findings from the literature, thus permitting an assessment of information retrieval or self-directed learning skills, and self-assessment ability

Issues related to oral assessment

Oral assessment has its own set of distinctive issues.


Student factors:Individual student characteristics, particularly high or low levels of confidence, can advantage or disadvantage particular students. Unconfident students may make self-deprecating comments which disadvantage them in the eyes of examiners. Highly-anxious students report distraction by a greater incidence of task-irrelevant thoughts. Students therefore need guidelines on preparing for particular oral assessment tasks and opportunities for practice and feedback prior to summative assessment.

Students within a class can vary in their interpretation of the requirements of oral tasks – from regurgitation to critical evaluation on the same task - so, as for any assessment task, particular care needs to be taken in task wording to ensure all students are aware of how best to prepare for and participate in these assessments.

Assessor factorsAssessors can differ greatly in the way they conduct oral examinations and in the judgements they make.

Assessors establish different affect environments which may impact on student confidence and consequent performance. Some fail to maintain the same cognitive structure (type, level and sequence of questions) for each student and others are unduly influenced by unfavourable aspects of student performance. Standardisation of interview procedures, questions, prompts and levels of intervention are needed to ensure reliability and validity and equitable conditions for all students.

Assessors can differ greatly in the standards that they apply when making summative assessment judgements and so intra-rater (the judgements of a single assessor) or inter-rater (the judgements of two or more assessors) reliability can sometimes be low. This is a particular difficulty with oral assessments as there is generally no record of the student's performance available for moderation activities or to serve as a reference in the case of student appeals. It has therefore been suggested that where possible audio and video recordings are used, especially when the oral assessment constitutes a significant proportion of the final grade. Even where it is not possible to tape all student performances, it is beneficial to make a set of tapes illustrating contrasting levels of performance and to use these in professional development activities conducted with experienced and new staff in order to promote a shared understanding of the standards to be applied. When creating records of students’ oral assessment performance it is necessary to consider privacy issues related to matters including gaining permission, storage and access.

Assessment conditions:Where students are presented with practical data during oral assessment it is difficult to ensure consistency of factors such as patients’ health complexity or levels of cooperation in clinical contexts. Simulated patients, the use of actors and structured questions have been investigated for their capacity to address such issues.

Other conditions such as the number of stations students rotate through when completing Observed Structured Clinical Examinations (OSCE) may also


require attention to ensure that students are given the most favourable opportunities to demonstrate their learning.

There is also debate about other variables which affect assessment conditions. In some circumstances students are presented with unseen information such as a practical problem or patient data and asked to ‘think on their feet’. Alternatively they can be given quite a bit of information beforehand and time to prepare a response. There is no right or wrong answer to decisions such as these - when determining appropriate task design it is the course learning objectives that must guide coordinators in setting conditions that will best enable them to observe required learning.

Lack of attention to practical details can provide quite diverse opportunities for students to demonstrate their learning. For example, if strict time limits are not prescribed and adhered to, students rostered at the end of the assessment time period can find themselves with a severely curtailed time allocation. It is therefore advisable to consider time limits carefully, to communicate these clearly, to establish timekeeping processes (e.g. 5 and 1-minute signals) and processes to enforce them. Student should also clearly understand which equipment and visual aids are required, permitted and/or disallowed, responsibility for photocopying handouts and protocols for leading discussions or offering feedback.

Practical arrangements also need to be considered in relation to those students who will need to be offered supplementary or special examinations. Will comparable conditions be available to students and, if not what alternative arrangements will need to be made? (for a discussion on supplementary examinations in clinical contexts see Pell, Boursciot, & Roberts, 2008 )

Assessment resourcingOral assessment can be more resource-intensive than assessment conducted in other modes. It is time-consuming to conduct oral assessment, particularly in large classes, and this can sometimes be at the expense of teaching time. It has therefore been argued that courses which incorporate particular forms of assessment including oral assessment should receive favourable budgetary consideration.

Equipment used in oral assessment (e.g. audio-visual recorders, projectors) must be available and reliable and it is advisable to decide in advance how equipment failure will be addressed. There is also a need to investigate the capacity of existing and emerging technologies to support oral assessment. Voice recognition and data interpretation software such as Leximancer, which allows automation of marking through analysis of voice recordings for content/themes, may prove an effective way of resourcing oral assessment.

Staff availability and expertise and other resourcing implications of particular oral assessment tasks must be considered as a normal part of program and course planning in order to determine their feasibility and fairness for both staff and students.

Professional developmentAs for other forms of assessment, academic staff need support in designing and


implementing effective oral assessments. Where support staff are not available academics also needed opportunities to develop the expertise to operate equipment efficiently – e.g. audio and visual recorders and the forms of technology discussed above.

Moderation can be an effective form of professional development and is one way of addressing variations in the type and level of oral assessments set, the feedback received by students, standards of the type of feedback to be given and consistency of judgements when multiple assessments are involved in assessing the learning of a single cohort. In particular moderation prior to the making of assessment judgements can be effective in circumventing many of the variations discussed.

Questions for discussion

1. How can program design ensure systematic student development of communication skills in oral modes?

2. Should oral assessment be limited to courses where the development of oral communication skills is specifically listed in course learning objectives?

3. To what extent, if any, should allowances be made for students who through lack of confidence or other factors prefer written to oral forms of assessment?

4. What forms of oral assessments require specific teaching in order that students are well prepared to undertake them?

5. What conditions or stipulations need to be in place to ensure equitable conditions for all students? Where oral assessment tasks contribute significantly to a final grade (say >30% weighting) what additional precautions need to be taken to ensure equity for all students?

6. What are the resourcing implications of some forms of oral assessment? Does the inclusion of oral assessment constitute grounds for preferential resourcing of courses?

7. How can academics (coordinators, teachers and tutors) best be supported in designing and conducting effective oral assessment that is consistent with the UQ criterion-referenced approach?

8. What special consideration needs to be given to the provision of supplementary or special examinations when oral assessment is used?

9. How can the effectiveness of oral assessment be monitored? What are suitable questions or prompts to guide evaluation of oral assessment?

10. What aspects of oral assessment would be appropriate for institutional investigation through teaching and learning projects or research?

References


oral assessment discussion paper

Documents