Bond University

Legal Education Review

Volume 28 Issue 1

2018

(Re)Introducing a Closed Book Exam in Law

Cathy S Sherry University of New South Wales Leon Terrill University of New South Wales Julian Laurens University of New South Wales

_____________________________________________________________________________________

Follow this and additional works at: https://ler.scholasticahq.com/

This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 Licence.

(RE)INTRODUCING A CLOSED BOOK

EXAM IN LAW

CATHY S SHERRY, LEON TERRILL AND JULIAN LAURENS

I INTRODUCTION

Historically, closed book examinations were the dominant form of

examination in higher education. In recent decades many academics

and institutions have moved towards open book assessments. The

primary motivations for the change appears to be a concern that closed

book assessments require ‘rote learning’, which is associated with

undesirable surface, rather than desirable deep learning;1 that closed

book assessments do not replicate real world work, where employees

will always be able to look up material; 2 and that closed book

assessments cause unnecessary student anxiety.3 Consistent with this

trend, the University of New South Wales law school (‘UNSW Law’)

has avoided closed book exams. Established in the 1970s, with a strong

progressive ethos, the law school has employed a variety of assessment

methods including research essays, class participation, moots, group

work, posters, take-home and open book exams. UNSW Law open book

exams are typically formal, time-limited exams, invigilated by the

University, into which students are permitted to bring any written

material.

In 2016, for the first time, UNSW Law used a closed book final

exam in the compulsory course Land Law. Against a review of the

relevant literature, this article describes the reasons for the introduction

of a closed book exam and sets out the outcomes of a research project

that evaluated its impact. The evaluation found inter alia that students

reported preparing differently for a closed book exam, many described

higher levels of stress, and a range of views were expressed as to

whether closed books exams should be used as part of a broader mix.

Lecturers found exam answers to be substantially similar, with slightly

Associate Professor, University of New South Wales Senior Lecturer, University of New South Wales Teaching Fellow, University of New South Wales

1 John Biggs, ‘What the Student Does: Teaching for Enhanced Learning’ (1999) 18

Higher Education Research and Development 57, 68–9; Christos Theophilides and

Mary Koutselini, ‘Study Behavior in the Closed-Book and the Open-Book Examination: A Comparative Analysis’ (2000) 6 Educational Research and

Evaluation 379. 2 Jeremy B Williams, ‘The Place of the Closed Book, Invigilated Final Examination in

a Knowledge Economy’ (2006) 43 Educational Media International 107, 110. 3 Theophilides and Koutselini, above n 1.

2 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

less ‘dumping’ of information, and a limited statistical analysis reveals

no significant change to how well students fared on the exam compared

to other assessment for the subject. While we do not suggest that closed

book exams should become the dominant form of assessment, both the

literature and the evaluation suggest that when used well closed book

exams can be a useful addition to a balanced assessment strategy in a

modern law school, as they encourage students to adopt different

learning strategies.

The article is in five parts. Part I considers the literature on closed

and open book exams. Part II provides background to the research and

the reasons for introducing a closed book exam. Part III sets out the

methodology of the research — which comprised a staff survey, an

anonymous student survey and a statistical comparison of marks

between the year a closed book exam was conducted and the previous

year when an open book exam was used — and sets out the results. Part

IV discusses the significance of the results, and the article concludes in

Part V.

II LITERATURE REVIEW

A Literature on Open and Closed Book Exams

One of the earliest analyses of open book exams was conducted in

1958 by Kalish, who defined open book exams as:

an examination [in which] the student is allowed to make use of any

materials at his disposal, including textbooks, lecture notes, and

dictionaries, but does not obtain answers either directly or indirectly from

other students.4

Research on closed and open book exams has not been particularly

extensive, and its findings are surprisingly varied. The transferability of

some research findings is limited by the discipline-specific nature of the

research; by the small size of cohorts in empirical research or an

absence of empirical research; and by the general lack of research on

long-term learning outcomes. The law-specific research on closed and

open book exams is very limited.5

However, as noted above, there has been a general trend towards

open book exams because they are thought to reflect employment more

4 Richard A Kalish, ‘An Experimental Evaluation of the Open Book Examination’

(1958) 49 Journal of Educational Psychology 200, 200. 5 Paul Maharg, ‘The Culture of Mnemosyne: Open-Book Assessment and the Theory

and Practice of Legal Education’ (1999) 6 International Journal of the Legal Profession 219; Lawrence Donnelly, ‘A Modest Proposal: The Case for the Open-

Book Law Exams’ (2005) 2 European Journal of Legal Education 105; Amanda

Cahill-Ripley, ‘Innovative Methods and Assessment in Law: The Value of Open-Book Exams as a Catalyst for Improving Teaching and Learning in the Law School’

(2015) 49 The Law Teacher 206.

2018______________________________________________CLOSED BOOK EXAM 3

accurately, 6 reduce rote learning, 7 and minimise student anxiety. 8

Studies have found that in preparing for open book exams, students

strive to make connections with previous learning on the same topic, as

well as with other topics and subjects, and that in writing their exams

they engage in creative use of acquired knowledge. 9 In contrast,

students undertaking a closed book exam have been found to adopt a

more ‘atomistic perspective’ based around ‘memorisation’ in which

knowledge manifested as ‘unconnected parts’ with a ‘lack of

understanding’.10

However, other research also reveals negative aspects of open book

exams or concludes that the impact of a shift to open book exams can

be neutral rather than positive. For example, Agarwal et al found that

while students’ test scores were higher on initial open book rather than

closed book tests, when tested subsequently, their scores showed that

students’ retention was the same.11 Brightwell et al expressed surprise

at ‘very similar overall results’ obtained in a closed book and

subsequent open book test.12 Testing the hypothesis that open book

exams would lead to fewer student errors, Kalish found that this was

not the case.13 In relation to the common assertion that open book

exams stimulate deep learning more than closed book exams, Heijne-

Penninga et al found the opposite; closed book exams were more

strongly related to deep learning.14

Investigating students’ use of texts and notes during exams,

Boniface found significant reliance on notes during open book exams,

6 Williams, above n 2; Jeremy B Williams and Amy Wong, ‘The Efficacy of Final

Examinations: A Comparative Study of Closed-Book, Invigilated Exams and Open-

Book, Open-Web Exams’ (2009) 40 British Journal of Educational Psychology 227;

Sabbas Shine, Cheddi Kiravu and Jeremy Astley, ‘In Defence of Open-Book Engineering Degree Examinations’ (2004) 32 International Journal of Mechanical

Engineering Education 197, 198; Maharg, above n 5; Donnelly, above n 5; Morris

Feller, ‘Open-Book Testing and Education for the Future’ (1994) 20 Studies in Educational Evaluation 235.

7 Maharg, above n 5; Donnelly, above n 5; Niels Krarup, Noe Naeraa and Christian

Olsen, ‘Open-Book Tests in a University Course’ (1974) 3 Higher Education 157; David Boniface, ‘Candidates’ Use of Notes and Textbooks During an Open-Book

Examination’ (1985) 27 Educational Research 201; Christos Theophilides and

Omiros Dionysiou, ‘The Major Functions of the Open-Book Examination at the University Level: A Factor Analytic Study’ (1996) 22 Studies in Educational

Evaluation 157. 8 Theophilides and Dionysiou, above n 7; Boniface, above n 7; John Francis, ‘A Case

for Open-Book Examinations’ (1982) 34 Educational Review 13; Afshin Gharib,

William Phillips and Noelle Mathew, ‘Cheat Sheet or Open-Book? A Comparison of the Effects of Exam Types on Performance, Retention, and Anxiety’ (2012) 2

Psychology Research 469. 9 Theophilides and Koutselini, above n 1, 391. 10 Evangelia Karagiannopoulou, ‘Effects of Classroom Learning Experiences and

Examination Type on Students’ Learning’ (2010) 17 Psychology 325, 338–9. 11 Pooja K Agarwal et al, ‘Examining the Testing Effect with Open- and Closed-Book

Tests’ (2008) 22 Applied Cognitive Psychology 861, 873. 12 Richard Brightwell, Janine-Helen Daniel and Angus Stewart, ‘Evaluation: Is An

Open Book Examination Easier?’ (2004) 3 Bioscience Education 1, 7. 13 Kalish, above n 4, 202. 14 Marjolein Heijne-Penninga et al, ‘Influence of Open- and Closed-Book Tests on

Medical Students’ Learning Approaches’ (2008) 42 Medical Education 967, 971.

4 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

with students spending on average one third of their time consulting

texts and notes.15 There was clear positive correlation between heavy

reliance on notes and texts and lower test scores,16 and a clear negative

correlation between students’ previous results and their reliance on texts

(but not notes). The weakest 10 students in Boniface’s study spent two

and a half times as long consulting texts as the strongest 10 students.17

In the same vein, in a study of undergraduate education students

Ioannidou found that students who sat an open book exam spent more

than 50 per cent of their test time consulting books, with the result that

they used books as ‘substitutes for their own thinking’.18 Ioannidou also

found that there was no significant difference between the overall final

exam scores of students who sat a closed or open book examination,

however students who sat the closed book exam scored ‘significantly

higher’ on exam sections on terminology and problem solving. 19

Overall, Ioannidou concluded that there are advantages and

disadvantages to both forms of assessment.

Gharib, Phillips and Mathew found that while anxiety levels were

lower for open book and cheat sheet exams, students overestimated how

well they would do on these exams, and post-exam quizzes showed no

difference in retention between different kinds of exams.20 Moore et al

conducted an experiment in which biology students completed a closed

book final semester exam, but some students did their mid-semester test

in open book conditions while others did the mid-semester test in closed

book conditions. The material tested for both mid and final exams was

similar. The researchers found that students who had done the open

book mid-semester test did ‘significantly worse’ on the final exam than

students who had done the closed book mid-semester test. They

concluded that:

These results indicate that on a closed-book final exam, students…recalled

significantly less about topics that were covered on open-book exams than

those covered by closed-book exams. These results suggest that open-book

exams may impede long-term learning of material covered in an

introductory biology course.21

The research also suggested that open book formats may produce

worse academic practices, with students doing the open-book exam

being less likely to attend classes and help sessions or hand in extra-

credit assignments. However, researchers noted that some students

doing the open book exam still came to class and prepared well for the

open book exam, using very little time in the exam to look up answers.

15 Boniface, above n 7, 207. 16 Ibid 204. 17 Ibid. 18 Mary Koutselini Ioannidou, ‘Testing and Life-Long Learning: Open-Book and

Closed-Book Examination in a University Course’ (1997) 23 Studies in Educational

Evaluation 131, 137. 19 Ibid 136. 20 Gharib, Phillips and Mathew, above n 8, 476. 21 Randy Moore and Phillip A Jensen, ‘Do Open-Book Exams Impede Long-Term

Learning in Introductory Biology Courses?’ (2007) 36(7) Journal of College Science

Teaching 46, 49.

2018______________________________________________CLOSED BOOK EXAM 5

In contrast, other students spent most of their time looking up answers

to even ‘the most basic, straightforward question’.22

A common theme in research is that students do less preparation for

open book exams.23 In Boniface’s study, 83 per cent of students said

they would have done more revision if the exam had been closed

book.24 While Boniface argued that this figure may have been inflated,

he accepted that ‘many of the candidates would have done more

preparation for a closed book examination’.25

Finally, Durning et al conducted a systematic review of the literature

on 37 studies comparing open book exams and closed book exams

which confirmed that research demonstrates that students either do

similar preparation for both forms of exams or prepare less for open

book exams; no research demonstrated greater preparation for open

book exams.26 Contrary to other research,27 they found no research in

the 37 studies correlating open book exams with deep learning. In

relation to anxiety, Durning et al concluded that all studies that

considered emotion lacked theoretical grounding, but that there seemed

to be some evidence that students ‘overestimate’ the effect that an open

book exam has on reducing anxiety.28 In relation to exam performance,

Dunning et al’s literature review favoured closed book exams, but the

authors noted that many students had little or no experience with open

book exams, which may have affected their performance.29 Both closed

and open book exams produced a ‘testing effect’, that is, the well

demonstrated phenomenon of test-taking improving subsequent

performance.30 The review concluded that there is insufficient evidence

to justify using open or closed book exams exclusively, and that a mixed

approach may be best, particularly where the aim is to assess

competencies rather than the maintenance of knowledge.31

B The Current Research Project

There are a number of ways in which the research conducted for this

article differs from the existing literature. First, the current study is

about the introduction of a closed book exam into a course where the

students’ prior experience of all law (but not other)32 exams has been

22 Ibid. 23 Ioannidou, above n 18. 24 Boniface, above n 7, 205. 25 Ibid 206. 26 Steven J Durning et al, ‘Comparing Open-Book and Closed-Book Examinations: A

Systematic Review’ (2016) 91 Academic Medicine 582, 585; Ioannidou above n 18,

136; M Heijne-Penninga et al, ‘Directing Students to Profound Open-Book Test Preparation: The Relationship Between Deep Learning and Open-Book Test Time’

(2011) 33 Medical Teacher e16, e16. 27 See above n 1. 28 Durning et al, above n 26, 586. 29 Ibid 585. 30 Ibid 587. 31 Ibid 588. 32 As described below, undergraduate students at UNSW Law study a combined degree

and many students experience closed book exams in their non-law degree.

6 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

open book, while existing studies relate to faculties and disciplines in

which closed book exams have been the normal or even exclusive

method of examination. Most students in the studies cited in the

Durning et al review had little or no prior experience with open book

exams.33 Studies have found that experience with open book exams

changes the way students prepare and are motivated.34 This current

UNSW Law study is thus unique amongst current literature —open

book examination is the orthodoxy.

Second, while the overall empirical literature comparing open and

closed book exams is limited, it is almost non-existent in relation to law.

This lack of discipline-specific research may be significant for a

number of reasons. One obvious reason is that multiple choice questions

are common in many disciplines, but rare in law. For example, of the

37 studies considered by Durning et al, the overwhelming majority

related to multiple choice exams.35

Only three law-specific articles were identified. Maharg’s 1999

article is critical of the dominant position that memory holds in legal

education,36 and argues that such a focus on memory is not as relevant

as it once was considering the more readily available nature of

knowledge today. Being able to memorise is not the only skill required

for legal analysis nor does it indicate whether you will be able to

appropriately apply that information. Maharg situates his support for

open book exams in the context of constructivism, and importantly that

situated or ‘authentic’ learning is vital for professional and vocational

learning — open book exams are more authentic because they better

replicate what would occur in practice. Donnelly’s 2005 article also

argues that an open book exam ‘most closely replicates the work of

practitioners’. 37 Donnelly acknowledges that memory is required in

legal education and the profession but like Maharg thinks there is too

great an emphasis placed on it in legal education.38 He suggests that an

open book exam requires an exam response that ‘should reflect a greater

level of sophistication’ than a closed book exam.39 However, crucially,

neither the Maharg nor Donnelly articles involved empirical research in

relation to law students.

Cahill-Ripley’s 2015 article includes the results of a small-scale

empirical study on law students, but the study related to their perception

of assessment generally. The course that the students had completed

was assessed by 50 per cent coursework and 50 per cent closed book

exam. Students were asked, ‘What kinds of assessment do you prefer

and why?’ Ten students, of 47 who responded to the survey, expressed

33 India L Broyles, Peggy R Cyr and Neil Korsen, ‘Open Book Tests: Assessment of

Academic Learning in Clerkships’ (2005) 27 Medical Teacher 456; M Heijne-

Penninga et al, ‘Open-Book Tests to Complement Assessment-Programmes:

Analysis of Open and Closed-Book Tests’ (2008) 13 Advances in Health Sciences

Education 263. 34 Durning et al, above n 26, 588. 35 Ibid 586. 36 Maharg, above n 5, 222. 37 Donnelly, above n 5, 107. 38 Ibid. 39 Ibid 109.

2018______________________________________________CLOSED BOOK EXAM 7

a preference for open book or seen exams.40 Cahill-Ripley concludes

that she is not advocating for either all closed or all open book exams,

but rather multi-method assessment that allows students to develop and

exhibit a range of competencies needed for work inside and outside

law.41

There is clearly a need for more research into the use of closed and

open book exams in legal education.

Whether research relates to law students or not, the differences

between student cohorts in different institutions raises fundamental

questions about the transferability of research on assessments. For

example, a 2007 study conducted by Williams, which strongly

supported the use of open book exams, drew on a cohort of 91 students,

85 per cent of whom were male and 59.1 per cent of whom were

between the ages of 30–39.42 They were studying at an entirely online

university with 4000 students from 60 countries. How this cohort of

students, most of whom were well into their careers, approached

assessment may differ radically from how a group of mixed-gender

school leavers approach assessment. Student approach to assessment is

critical. For example, research essays offer students an outstanding

opportunity to develop research and writing skills, to think deeply and

critically, and to read extensively. However, if a student pays someone

to write the essay for them or collaborates with classmates in order to

minimise their own work, a research essay may be a very poor form of

assessment. Student motivation and academic methods matter.43 While

there will be variation within any student body, factors like age,

previous experience of study, entry marks, status of the discipline,

degree and institution, will influence the way cohorts of students

approach assessment. This applies to our research as much as others’;

our conclusions may have limited applicability to student cohorts unlike

our own.

III BACKGROUND TO THE RESEARCH

This section sets out some of the history behind, and reasons for, the

introduction of a closed book exam in the subject Land Law. As the

literature review demonstrates, the decision to move from an open book

to a closed book exam is unusual. It reflects the particular history of

UNSW Law as well as shifts in technology and perceptions about the

changing nature of student practices.

When UNSW Law was founded in 1971, it set out to take a distinct

approach to teaching, which later came to be characterised as ‘student

centred’. Tilbury describes how:

40 Cahill-Ripley, above n 5, 216. 41 Ibid 217. 42 Williams and Wong, above n 6, 231. In contrast, a study done by Theophilides and

Koutselini, above n 1, 381, relied on a cohort of 201 students, 88.1 per cent of whom were female and all within the age of 20–24. Heijne-Penninga et al, ‘Influence’,

above n 14, 971, found significant gender differences in relation to exam preparation. 43 Moore and Jensen, above n 21, 46.

8 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

Classes were small and a culture of student preparation for, and discussion

in, those classes developed. The advantages of this style of teaching, at least

theoretically, include the acquisition by students of a deep knowledge of

law, which facilitates the development of a number of skills along the way

and leads to teacher satisfaction.44

A core element of this approach is the use of relatively small classes

in which student participation is highly encouraged and often assessed.

For core subjects such as Land Law, this means that each week there

are two, two-hour seminars. There are no lectures or tutorials in addition

to this. Seminars are intended to encourage a style of teaching that

draws ‘upon the form of Socratic dialogue-based teaching common in

the USA’,45 modified to provide for informed discussion rather than

confrontation or interrogation.46 Over time, student numbers at UNSW

Law have grown significantly and today there are well over 2000

students. 47 A consequence has been an increase in class sizes,

particularly for subjects in later years such as Land Law, which has

classes of up to 44 or 56 students.

UNSW Law’s student cohort and their educational background were

significant to this research. Like most Australian law schools, UNSW

Law teaches both an undergraduate and graduate (JD) law degree, with

70 per cent of students in the former category. Unlike the United States,

Australia (like the United Kingdom) admits students to law degrees

straight from high school. However, unlike the United Kingdom, high

school graduates in Australia will frequently combine their law degree

with another discipline, such as Arts, Commerce or Science. Such

students typically obtain two degrees in five years, rather than the six

years it would take to obtain the degrees separately. By allowing

students to combine law with another degree on graduation from high

school, combined law is popularly considered ‘a good general degree’

and the demand for entry is very high. Admission to university in

Australia predominantly relies on the Australian Tertiary Admission

Rank (ATAR).48 The average ATAR in Australia is 70 and the median

ATAR for admission to UNSW Law’s combined degree in 2017 was

98.49 Along with a number of other law and medical schools, UNSW

Law has one of the highest entry marks of any degree in Australia.

44 Michael Tilbury, ‘Marion Dixon, Thirty Up: The Story of the UNSW Law School

1971–2001’ (2002) 25 University of New South Wales Law Journal 255. 45 Alex Steel, Julian Laurens and Anna Huggins, ‘Class Participation as a Learning and

Assessment Strategy in Law: Facilitating Students’ Engagement, Skills Development and Deep Learning’ (2013) 36 University of New South Wales Law Journal 30, 34.

46 Ibid; see above n 16. 47 As at October 2016: see UNSW Law, Facts in Brief (6 October 2016)

<http://www.law.unsw.edu.au/about-us/facts-brief>. 48 Unlike the United States and the United Kingdom, it is unusual for Australian

universities to use interviews, personal statements or letters of recommendation for

admission to university. While universities take into account socio-economic

disadvantage, and occasionally give bonus points for extra-curricular achievements, the dominant method of selection is the Australian Tertiary Admission Rank which

is based on moderated school assessments and state-administered final exams: see

Universities Admission Centre, What is the ATAR? <http://www.uac.edu.au/atar/>. 49 UNSW, Admissions Transparency (31 August 2017)

<https://www.futurestudents.unsw.edu.au/admission-unsw>. UNSW Law recently

2018______________________________________________CLOSED BOOK EXAM 9

As a result, the cohort of undergraduates at UNSW Law could fairly

be described as high achieving, and the students’ approach to learning

and assessment is still strongly influenced by their experience of high

school where they excelled. It is inherent in the process of aggregating

high-achieving students and marking them relative to each other that

some students will find themselves consistently obtaining marks that

are lower than what they are accustomed to. As will be discussed later,

higher rates of anxiety and depression among law students are well

documented.50

Across the curriculum, UNSW Law uses a variety of forms of

assessment and Land Law has followed a fairly typical approach: class

participation is worth 20 per cent, a mid-semester assignment worth 30

per cent and the final exam worth 50 per cent. Prior to 2016, all exams

had been either open book or take-home exams, usually with a problem-

question and an essay. When it emerged that a closed book exam was

being introduced for Land Law in 2016, there was debate at the school

level whether it should proceed. However, an overwhelming majority

of staff supported a resolution allowing the course organisers to set their

own assessment.

Also significant was the reaction from students. A group of current

students organised an online survey to obtain student views, something

that had not been done previously for other changes to assessment.

Through student representatives, the student body also put a motion to

the law school asking that the use of closed book exams be prohibited.

Some students said that the absence of closed book exams was

instrumental in their electing to study at UNSW Law rather than the

alternatives. It was also argued that it was unfair on students to impose

this new form of assessment on them in their penultimate or final year

of study, when they had not had the opportunity to become accustomed

to it,51 and were under pressure to obtain good marks in order to obtain

clerkships and jobs. The topic generated widespread discussion,

including in online forums, with arguments being expressed both for

and against the change. This widespread debate about the exam, and the

introduced a separate Law Admission Test, but the students in our study entered law

school well before its introduction. 50 See, eg, Natalie Skead and Shane L Rogers, ‘Stress, Anxiety and Depression in Law

Students: How Student Behaviours Affect Student Wellbeing’ (2014) 40 Monash

University Law Review 564; Sharon Medlow, Norm Kelk and Ian Hickie, ‘Depression and the Law: Experiences of Australian Barristers and Solicitors’ (2011)

33 Sydney Law Review 771; Massimiliano Tani and Prue Vines, ‘Law Students’ Attitudes to Education: Pointers to Depression in the Legal Academy and the

Profession?’ (2009) 19 Legal Education Review 3; Justice Shane Marshall,

‘Depression: An Issue in the Study of Law’ (Paper presented at the National Wellness for Law Forum, Australian National University College of Law, 6 February 2015)

<http://www.fedcourt.gov.au/__data/assets/pdf_file/0004/26608/Marshall-J-

201502.pdf>; Jessie Agatstein et al, ‘Falling Through the Cracks: A Report on Mental Health at Yale Law School’ (Report, Yale Law School Mental Health Alliance,

December 2014)

<https://law.yale.edu/system/files/falling_through_the_cracks_120614.pdf>. 51 This argument applied only to law subjects. All undergraduate students are enrolled

in combined degrees, and many were exposed to closed book exams in their other

degrees.

10 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

campaign by some students to have the decision reversed, may have

influenced student attitudes when completing the survey we

administered for this research.

Why, then, did the course organisers decide to pursue a closed book

exam? The reasons were several. One was concern about the practice

employed by some students of simply copying down or perhaps

paraphrasing their notes in open book exams, thereby regurgitating pre-

prepared summaries of the law, and perhaps even pre-prepared answers,

rather than applying the law to the fact scenario or exploring the issues

raised by the essay question. Where those notes have been prepared by

other people, and not the students themselves, this also raises issues of

plagiarism, and does so in a forum where it is more difficult to detect.

Most open book exam answers are handwritten and are not put through

anti-plagiarism software, in contrast to assignments or take-home

exams, (although several universities, including UNSW, are now

trialling or have introduced, typed exams).

It appears that these concerns about the assessment process extend

beyond UNSW Law. Also in 2016, the New South Wales Law

Extension Committee, which runs a Diploma in Law course for

admission as a lawyer under the oversight of the University of Sydney,

instituted a closed-book exam policy for all exams, as well as banning

the publication of past exam papers.52 The Board advised students:

The introduction of the closed-book exam policy reflects the need for

increased attention to maintaining the integrity of the Board's exams

process … particularly as a result of recent media reporting of widespread

cheating in tertiary assessment tasks, and the University of Sydney's

subsequent report into the prevention and detection of academic

misconduct.53

While there is a long history of law students sharing notes,

technological advances have made the practice easier. Websites such as

Thinkswap, Coursehero, StudentVIP Notes and Nexus Notes make it

possible to buy notes from, or swap them with, a wider group of people.

Often those notes are institution and subject specific. A recent survey of

Nexus Notes, Thinkswap and Course Hero identified over a thousand

sets of notes associated with UNSW Law courses across these

platforms.54

Some staff also identified concerns about changing learning

practices among students, a perception that strategic approaches to

performing well in high school — such as the use of tutors and pre-

prepared answers 55 — carry through to the way students approach

52 Eryk Bagshaw, ‘Sydney University, University of NSW and UTS Crack Down on

Cheating Students’, Sydney Morning Herald (online), 25 April 2016

<http://www.smh.com.au/national/education/universities-crack-down-on-cheating-

students-20160421-gobq1m.html>. 53 Ibid. 54 Results on file with authors. 55 New South Wales final school exams, the HSC (Higher School Certificate) is a

standards-based assessment. Highly specific syllabi, detailed rubrics and examiners notes are available to all students online; see, eg, NSW Education Standards

Authority, English Stage 6

2018______________________________________________CLOSED BOOK EXAM 11

assessment at university, in a way that privileges the recitation of

information rather than deep learning. Not all staff agreed with this

observation, nor did all staff agree that closed book exams might assist,

but it was a significant motivating concern.

A further rationale was that closed book exams are used in many

other institutions, including for practitioners sitting the Bar exam in

New South Wales, and so it would be of benefit to students to be

exposed to that form of assessment during the law degree.

Finally, some staff questioned the need to teach or examine Land

Law in the detail it had traditionally been taught. As a compulsory core

course, it was preferable to focus on students mastering the foundational

principles, rather than running the risk of swamping students with detail

they could not possibly understand. 56 A closed book exam might

encourage students to focus on the framework of land law, rather than

the excessive detail included in notes they bring into open book exams.

A Implementation of the Closed Book Exam

Some changes were made to the exam paper and course information

in light of the decisions to introduce a closed book exam. While an essay

question had often been included in Land Law exams in addition to a

problem question, it was decided that an essay question was less suited

to the closed book format. If a central concept was examined by essay,

there was a risk that many students would draw on their high school

experience and memorise a pre-written essay, possibly written by

others;57 if a more tangential concept was examined by essay, some

students may not be able to answer the question at all, despite having a

sound understanding of the fundamental principles of the course.

<http://www.boardofstudies.nsw.edu.au/syllabus_hsc/english-std-adv.html>. One of the (presumably unintended) consequences of this is that exam questions, particularly

in the compulsory English courses are very predictable, with the result that many

students memorise pre-written essays which they rewrite in exams, ‘topped and tailed’ to address any specific aspects of the question: Anna Patty, ‘Memorised Essays

a Poor Test of HSC Skill, Says Expert’, Sydney Morning Herald (online), 2

November 2009 <http://www.smh.com.au/national/memorised-essays-a-poor-test-of-hsc-skills-says-expert-20091101-hrl2.html>. The key problem with this method is

that students are not always memorising essays they have written themselves. Dr Kim

Jaggar, principal of high ranking, academically selective government school, Sydney Boys High School, said that problem of students submitting essays written by tutors

or other students for HSC assessments or exams, was widespread and that students did not see it as immoral. Dr Jaggar said that, ‘Kids are doing what comes naturally

to maximise their chances’: Suzanne Smith, ‘Students Taking ‘Extreme’ Measures to

Succeed in HSC’, ABC News (online), 12 November 2011 <http://www.abc.net.au/news/2011-11-11/students-cheating-to-excel-in-

hsc/3661820>; Eryk Bagshaw, ‘Cheating “Endemic” in NSW High Schools’, Sydney

Morning Herald (online), 7 June 2015 <http://www.smh.com.au/national/education/cheating-endemic-in-nsw-high-

schools-20150507-ggw8h9.html>. HSC marks, scaled for subject difficulty, are the

principal determinant for university entry: see above n 48. 56 See Cathy Sherry, ‘Teaching Land Law: An Essay’ (2016) 25 Australian Property

Law Journal 129. 57 See ibid.

12 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

Consequently, two problem questions were used and designed to be

slightly easier than the problem questions used for an open book exam.

Students were also advised through the course outline that the lecturers

‘do not expect you to answer questions with the same level of detail that

you could answer an open book exam so do not expect that of

yourselves’. Attached to the exam paper was a list of the names of all

major cases (75 in total) that were covered in the course. That list had

also been made available to students through semester so that they

would be familiar with it prior to the exam. Students were advised that

they were not required to memorise case names and that, for example,

if they wrote ‘the NZ husband and wife farm case’ then this would be

sufficient: the examiner would know they meant Frazer v Walker.58 The

course also covers a small number of legislative provisions in the Real

Property Act 1900 (NSW) and the Conveyancing Act 1919 (NSW).

Students were advised that they were expected to know the provisions

but that examiners were ‘more interested in whether you have

understood their meaning than whether you get a little Roman numeral

right or wrong’.

In other respects, the exam was the same as previous open book

exams. It was a two-hour invigilated exam with 10 minutes reading

time. Students were assessed on the usual criteria, being a combination

of their ability to identify the issues and correctly apply the law, the

coherence of their conclusions, their prioritisation of issues and their

writing style.59 As per previous semesters, the exam formed 50 per cent

of the overall mark and tested all of the material taught in the course.

Land Law is taught primarily through a casebook, supplemented by

materials on the University’s online Moodle site. While students were

advised through the course outline that the closed book exams would

focus on ‘the fundamental principles’, no changes were made to the

course content or materials.

IV RESEARCH METHODOLOGY AND RESULTS

We adopted a three-pronged approach to assess the impact of the

closed book exam: a staff questionnaire, an anonymous student survey,

and a limited statistical analysis of student results. Ethics approval for

the student survey was applied for and given by the UNSW Human

Research Advisory Panel B (approval number HC16533).

A Staff Questionnaire

1 Methodology

The views of the six lecturers who taught Land Law were sought

through a written questionnaire comprising eight open-text questions.

Lecturers were asked what changes they made to their teaching style,

58 [1967] 1 AC 569. 59 See UNSW Law, Common Assessment Types (10 September 2018)

<http://www.law.unsw.edu.au/common-assessment-types>.

2018______________________________________________CLOSED BOOK EXAM 13

what changes they observed in the students and whether they had any

concerns about the closed book exam.

Of the six lecturers, two were permanent members of staff and four

were sessional staff, two of whom were practising barristers. Staff

teaching experience ranged from a year to over 20 years. One sessional

member of staff also taught at another university and had considerable

experience examining law using both closed and open book exams. As

there were only six teachers, the survey was not anonymous as it would

have been artificial and ultimately ineffective to make it so.

2 Results

The first finding of the staff survey was how little difference

lecturers saw in the exam answers written in closed and open book

conditions. While slightly shorter than answers written in open book

conditions, the closed book answers were still substantial and detailed.

However, there seemed to be less ‘information dump’ in the closed book

exam. As one lecturer said:

[S]tudents got to the point. Again, an incredibly valuable skill for any law

graduate to possess. There was definitely less of a ‘kitchen sink’ inclusion

of completely irrelevant material. This is something I have observed in the

many times I have now taught this subject at various institutions over the

last 15 years. It makes exams easier to grade because it is obvious very

quickly who does or does not understand the issues—rather than who can

write faster and copy-write!

Unfortunately, closed book exams do not entirely eradicate the

practice of information dumping. Staff report that some students still

followed the practice of writing down what they knew rather than

answering the question.

Staff felt that the closed book exam allowed them to differentiate

the students better, identifying both the students who understood the

material very well, and those who understood it very little. One lecturer

commented that they gave more high distinctions than usual because

they felt it was clear that the students had genuinely attained that level.

Another staff member noted that,

Two [of my eight] students who failed the exam had done very well on the

assignment. One received a D, and one an HD. They were also clearly

bright, capable students. However, they simply had not done enough

reading or work for the exam. They were perfectly capable of understanding

land law, but had not even mastered the basics. They were caught out by the

closed book format and I am sure would have passed, and possibly done

well, if it had been open book. However, they were a very small minority;

2/90 students.

In the authors’ experience, the failure rate in UNSW Law exams is

typically less than 5 per cent. Some lecturers reported a slight increase

in the failure rate for the closed book exam, an observation borne out

by the statistics. The failure rate for the closed book exam was around

5.2 per cent, whereas for an open book exam the previous year it had

been around 3.8 per cent. While there were a higher number of fails, as

14 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

described below our statistical analysis revealed substantially similar

results for students as a whole. In other respects, lecturers’ marks

naturally conformed with the results that are achieved every year in

Land Law when it is examined by open book exam.60

Staff were aware that some students seemed to be suffering from

high levels of anxiety about the exam format, and one teacher suggested

that

it would be worth running some form of resilience workshop/s for students.

Should they choose to practise as solicitors, they will often be subject to

great pressure, setbacks and other obstacles (far more serious than a closed

book exam). I think it is important for the law school to equip them not only

with brilliant legal training, but also with life skills to manage and overcome

common challenges faced in practice. I think this may not only enhance

their lives, but contribute to building a better, more resilient and supportive

profession as a whole.

On the issue of whether the closed book exam tested or helped

students to acquire transferrable skills, one teacher said:

As a practitioner, whilst I understand that in practice we always go back and

find sources to base an opinion on ie duty to do that! But it is also not correct

to suggest that we do not also recognise potential legal issues from having

a proper understanding and recollection of basic principles (which is all that

really gets taught or even can be taught in a law school). The assumption

by other practitioners is that law graduates will at least possess that basic

understanding in order to be fit to practice.

No lecturer had radically altered their seminar-style teaching, but

most reported doing more problem questions with students in class,

including problem questions answered in closed book conditions.

B Statistical Comparison of Current and Previous Year’s Results

The second form of review was a statistical analysis of student

results. For several reasons, this provided only limited information.

This was not a situation where two groups of students sat the same

exam, one in closed book conditions and the other in open book

conditions, thereby enabling a direct comparison.61 While it would have

been possible to compare results from the closed book exams with the

results obtained for open book exams in previous years, the exam

questions are different each year and there is often a change in teaching

staff. The authors do not necessarily teach Land Law every year, and

thus do not determine the exam content year after year. Further, to

reduce the risk of inconsistency between lecturers, at UNSW Law each

lecturer is required to ensure that their spread of grades falls within

certain bands. 62 So while it possible to have some variation in the

60 UNSW Law has a policy requiring results to fall within bands. While there was some

concern that the closed book Land Law marks would have to be scaled to match those

bands, this proved unnecessary. 61 See, eg, Agarwal et al, above n 11. 62 Or to provide an explanation as to why they do not: where, for example, a class is

particularly small or unusually strong or weak.

2018______________________________________________CLOSED BOOK EXAM 15

overall results between two years, this will both be moderated by the

use of bands and may be due to several factors other than the form of

assessment.

In this regard, our research, like other research in this area, was

limited by the overriding imperative to set assessment for pedagogically

justifiable reasons, not research efficacy. For example, while it might

be possible to run longer-term experiments with cohorts of students

sitting open book exams and closed book exams in each or alternate

semesters so that results can be compared, it would arguably be

unethical to use students in this way. To their credit, contemporary

university students would be unlikely to tolerate such an experiment

with their marks.

As a result, we conducted the most meaningful statistical

comparison we could devise, given the available data. We compared the

difference between the results for the exam and the results for the two

other forms of assessment in the subject (the mid-semester assignment

and class participation). This tested the extent to which the same

students fared better or worse on the exam compared to the other

assessment in the course. We ran this check for both 2016 (the year in

which a closed book exam was introduced) and 2015 (where an open

book exam had been used). The results were very similar. The mean

difference between students’ results for the exam and for other

assessment was -1.09 in 2015 and -1.50 in 2016. However, a problem

with this measure is that positive differences and negative differences

cancel each other out, so we also measured the root mean square (RMS)

of the difference for each year, which returned a figure of 6.52 for 2015

and 6.7 for 2017. In other words, the mean difference between the mark

each student obtained for the exam as compared to other assessment

was between 6 and 7 per cent, and this did not change significantly with

the introduction of a closed book exam. As the other forms of

assessment remained constant, an increase in variability would have

indicated that closed book exams had had a disruptive impact on student

performance. However, there was no real increase. While only two

years were compared, the results indicate that students performed

similarly relative to the other assessment tasks.

C Student Survey

1 Methodology

The final method we used to investigate the effects of a closed book

exam was an anonymous survey of students. The survey was

administered online, but during class time for a compulsory subject that

followed Land Law in the curriculum; students who did not have

laptops in class were given hard copy surveys. This led to a high

response rate for undergraduate students, with 162 responses out of 290

students. Unfortunately, it also led to a much lower response rate for JD

16 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

students, as their subjects follow a different timetable. Only 12 out of

60 JD students completed the survey. The JD cohort includes a far

higher proportion of international and non-English speaking

background students. The lower representation of JD students had an

impact here, as only 10 out of the 174 respondents identified as

international students with a first language other than English. This

number would have been higher if a greater proportion of JD students

had completed the survey.

The substantive part of the survey contained 35 multiple-choice

questions and one open-text question. The multiple-choice questions

were divided into four parts: the first (nine questions) asked students

about their approach to study across all law subjects; the second (nine

questions) about their approach to preparing for and undertaking prior

exams; the third (six questions) about their approach to preparing for

the closed book exam in Land Law; and the fourth (eleven questions)

about their experience of the closed book exams and their reflections

afterwards. The questions allowed for scaled responses with respect to

strength of agreement/disagreement or frequency, depending on the

question.63 The final, open-text question asked students: ‘Do you have

any general comments you would like to make regarding the closed

book exam in Land Law?’.

The survey was administered through Survey Monkey, which

allowed us to check the data for correlations between answers while

retaining student anonymity. It was possible to check, for example,

which proportion of respondents who answered ‘agree’ or ‘strongly

agree’ to one question also answered ‘agree’ or ‘strongly agree’ to

another question.

2 Results

The student survey confirmed a number of preliminary assumptions

we had about our student body. First, our students are hard-working and

engaged, with high levels of commitment to their studies. Seventy-eight

per cent of students think about whether they will need to understand

the material they are studying when they graduate; 70 per cent are

aiming to get the highest marks they can; 90 per cent always try to

actually understand the material they are covering and 85 per cent find

studying law interesting.64 Only 14 per cent of students do not want to

practise law on graduation, with 29 per cent being unsure and 57 per

cent being certain that they do.65

63 Twenty-nine questions were scaled as Strongly agree, Agree, Neither agree or

disagree, Disagree and Strongly disagree. Five were scaled Never, Once or twice,

More than once or twice and Always. The question ‘I want to practise law after

graduation’ was simply scaled Yes, No and Unsure. 64 Percentages are the aggregate of students who answered ‘Strongly agree’ and ‘Agree’

or ‘Strongly disagree’ and ‘Disagree’ to questions. 65 We made no assumption that only students who want to practise law will be engaged

in their studies. However, students’ perception that combined law is a ‘good general

degree’, particularly commerce/law for those intending to work in the finance industry, has led to staff concern that some students have no interest in law beyond

the combined law graduate label.

2018______________________________________________CLOSED BOOK EXAM 17

In relation to open book exam preparation, the otherwise exclusive

method of formal examination in our School, only 22 per cent of

students said they spend less time reading and revising for a course

because there is an open book exam and 88 per cent believe that to do

well in an open book exam they need to have a good understanding of

the material. When writing open book exam answers, 96 per cent try to

ensure their answers are well-reasoned and persuasive and 50 per cent

never include material they do not entirely understand.

Second, despite the general conscientiousness of our student body,

the survey confirmed concerns we had about the way that they prepare

for and complete open book exams. While only 25 per cent of students

have purchased unofficial notes or study guides, almost 70 per cent have

worked in groups to create a shared set of notes or answers for exams

or other assessment, and 68 per cent of students have used other

students’ notes in exams. Seventy-nine per cent of students refer to

notes frequently during open book exam,66 and 59 per cent of students

have copied sections directly from notes, articles or books into their

exam paper. Fifty-eight per cent of students include every detail they

can in case it will earn them more marks and 50 per cent of students

have included material they do not entirely understand.

Thirdly, the survey results reveal that there are high levels of stress

in our student body, possibly higher than we anticipated.67 Ninety per

cent of students said that they felt pressure on them to do well in their

law studies, although we do not know if this pressure comes from

parents, themselves, concern about job markets or simply the inherently

competitive nature of law schools in which students are continually

graded on tasks.68 Not surprisingly, 73 per cent of students found the

prospect of a closed book exam more stressful than an open book exam,

but worryingly, 73 per cent of students said that they had experienced

‘significant anxiety’ during their law studies and 66 per cent said they

were ‘very anxious’ about the closed book exam.

When it came to preparing for the closed book exam, 26 per cent of

students did their reading and revised more consistently throughout the

semester knowing that the exam was closed book, although 51 per cent

66 This is consistent with Ioannidou, above n 18. 67 Stress levels of Australian university students seem to be very high. In 2015 and

2016, 42 per cent and 41 per cent of students respectively considered leaving tertiary

education because of ‘health and stress’: Quality Indicators for Learning and Teaching, ‘2016 Student Experience Survey: National Report’ (Report, Quality

Indicators for Learning and Teaching, March 2017) 76. Law students have atypically

high levels of psychological stress relative to the general community: Norm Kelk, Sharon Medlow and Ian Hickie, ‘Distress and Depression among Australian Law

Students: Incidence, Attitudes and the Role of Universities’ (2010) 32 Sydney Law

Review 113, 122. 68 Other studies have revealed a range of causes for law students’ stress including

workload, competition, perceived value of marks, clerkships, financial concern and

limited time with family and friends. See, eg, Adele Bergin and Kenneth Pakenham, ‘Law Student Stress: Relationships Between Academic Demands, Social Isolation,

Career Pressure, Study/Life Imbalance and Adjustment Outcomes in Law Students’

(2015) 22 Psychiatry, Psychology and Law 388, 389–90.

18 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

said they did not.69 Thirty-eight per cent agreed that they had made

more of an effort to write notes in their own words to increase their

understanding, while another 38 per cent said that they did not. Twenty-

six per cent of students asked more questions in class to increase their

understanding, and 51 per cent said they did not.70

We asked students if they had changed their ‘study strategy’

knowing there was a closed book exam and 66 per cent said that they

had. Fifty-one per cent also said that the closed book exam had

encouraged them to learn in a ‘different way’. While there were some

follow up questions (see below), unfortunately the questionnaire did not

allow them to elaborate on the changes they had made, and so we could

not identify the full range of ways that they changed strategies and

whether those changes are desirable. 71 However, it is arguable that

almost any change or adaption is preferable to formulaic, repetitive

strategies which can develop when students are presented with

unvarying assessment.

While not allowing students to give freely worded answers on

changes to strategy and learning, the survey did ask if the closed book

exam had required them to memorise ‘substantial amounts of material’.

Eighty-three per cent of students said that it had. Memorisation and rote

learning are some of the primary criticisms levelled at closed book

exams,72 and so this finding is concerning.

However, perhaps because they had been required to retain

information, just under 50 per cent of students said that they went into

the closed book exam knowing more than they would usually know for

an open book exam, and 36 per cent of students said that they

understood the material better having studied for and taken a closed

book exam.

When it came to answering the closed book exam, 51 per cent of

students said that the closed book exam encouraged them to write

answers in their own words and 35 per cent of students said that not

having notes encouraged them to think more. Fifty per cent said that the

closed book exam allowed them to demonstrate their knowledge of the

course, while 33 per cent said that it did not. Thirty-one per cent of

students said that they included material they did not entirely

understand, in contrast to 50 per cent of students who say they have

done this in open book exams. Thirty-eight per cent of students did

better in the closed book exam than they had anticipated. In relation to

the statement ‘the closed book exam was not as stressful as I originally

thought’, 43 per cent of students agreed, 24 per cent neither agreed nor

disagreed, and 33 per cent disagreed.

69 This is consistent with Durning et al, above n 26, in which a review of research on

closed book exam and open book exams in health education was undertaken. The

authors found at 585 that while some research showed that students studied more for

closed book exams, research results were inconsistent. The crucial difference between our students and the students in the studies considered by Durning et al, is

that our students have significant experience of open book exams. 70 23.5 per cent of students said ‘Neither agree nor disagree’ to this statement. 71 However, some specific changes were elicited from other questions, discussed below. 72 See above n 7.

2018______________________________________________CLOSED BOOK EXAM 19

Finally, 42 per cent of students concluded that it was worthwhile

having a mix of closed book exams and open book exams at law school,

18 per cent neither agreed nor disagreed with the idea, while 40 per cent

of students thought that it was not worthwhile.

3 Statistically Significant Pairs

The survey data allowed us to investigate whether there were any

‘statistically significant’ pairs of questions. 73 These were pairs of

questions where the students who answered one question in a particular

way were not evenly distributed across the answers for another

question, but rather answered the second question in distinct ways.

Of particular interest to us were the answers of the 22 per cent of

students who said that they spend less time reading and revising during

semester knowing the exam is open book. As our students have no other

experience of doing closed book law exams, it is possible that this figure

is an understatement; it is hard for students to know if they would spend

more time revising for closed book law exams because they have never

done them. The 22 per cent figure is particularly low in contrast to other

studies, such as Boniface’s in which 83 per cent of students doing both

closed and open book exams said they prepared less for open book

exams.74

However, looking at this 22 per cent, who we will call ‘lesser

reading students’, 90 per cent of them had used another student’s notes

in an exam, in contrast to 70 per cent of all students. It is likely that

these students feel they can afford to not read and revise consistently

for open book exams because they know that even if they have not

created their own set of notes, they can always use another student’s

notes in the exam.

While it would be tempting to assume that the lesser reading

students are simply lazy or disengaged, their answers to other questions

suggest that this may not always be the case. Some may be, but some

seem to be academically able and insufficiently challenged by open

book exams. So, while 88 per cent of students generally think that to do

well in an open book exam they have to have a good understanding of

the course material, only 68 per cent of the lesser reading students think

that a good understanding of the material is necessary to do well. As

Land Law is taken two thirds of the way through most students’ degree,

it is possible that some lesser reading students have first-hand

experience of having done well in open book exams, knowing they do

not have a good understanding of the material. In contrast, almost 100

per cent of other students — students who said they did not read less

because an exam was open book — thought that to do well they had to

have a good understanding of the material.

73 The analysis was done by an Honours student from the School of Mathematics and

Statistics, UNSW. The methodology used a contingency table, Chi Square Test and

Cramer’s V. The explanation of the methodology is on file with the authors. 74 Boniface, above n 7.

20 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

The closed book exam had the most marked positive effect on the

lesser reading students. Fifty-five per cent of lesser reading students

said that they did their reading more consistently knowing it was a

closed book exam, compared to 26 per cent of students overall; 68 per

cent of them made more of an effort to write notes in their own words,

as opposed to 38 per cent of students generally, and 50 per cent of them

made more effort to ask questions in class to increase their

understanding, as opposed to 26 per cent of students generally.

When it came to writing the closed book exam, 71 per cent of the

lesser reading students said the closed book format encouraged them to

write in their own words and 63 per cent said that not having notes

encouraged them to think more; this was in contrast to 51 per cent and

35 per cent of students generally. Not surprisingly, 66 per cent of the

lesser reading students said they understood the material better having

studied for and taken a closed book exam, in contrast to 36 per cent of

students generally. Interestingly, of the 59 per cent of other students —

students who said they did not read less knowing an exam was open

book — 23 per cent still said they understood the material better having

studied for and done a closed book exam.

One of the more striking results was in relation to the question of

whether students thought it was ‘worthwhile having a mix of closed and

open book exams at law school’. It might be expected that lesser-

reading students — that is, students who read and revise less when there

is an open book exam — would be more likely to oppose the

introduction of closed book exams, which require them to work harder.

The opposite was true. Sixty-three per cent of lesser-reading students

were in favour of a mix of open book exams and closed book exams,

compared to 43 per cent of students generally. That is, this 63 per cent

of lesser-reading students are not opposed to working harder, but it

would appear that they do not feel compelled or incentivised to do so

for open book exams. To be clear, there were very few students who felt

that open book exams were not a good form of assessment,75 but that

does not mean that some students do not recognise their limitations. As

one student said:

Although I am shooting myself in the foot by suggesting more closed book

exams might be appropriate, I feel that actually having to put some effort

into studying the course instead of relying on notes would probably be

beneficial for me in the long term.

This finding is important as research indicates that challenging

retrieval processes promote long-term retention of material.76

The second set of statistically significant pairs that were instructive

related to anxiety. Along with the students who read less for open book

exams, the other group of students who were most in favour of a mix of

open book exams and closed book exams was the 24 per cent of students

who said that they were not ‘very anxious’ about the closed book exam.

75 Only 7.5 per cent of students overall disagreed with the statement that ‘open book

exams are a good form of assessment’. 78.5 per cent agreed and 14 per cent neither agreed nor disagreed.

76 Agarwal et al, above n 11, 872.

2018______________________________________________CLOSED BOOK EXAM 21

A striking 80 per cent of these students were in favour of a mix of exams

at law school. In contrast, of the 66 per cent of students who were ‘very

anxious’ about the closed book exam, only 28 per cent thought that a

mix of exams was worthwhile, while 43 per cent of students generally

thought a mix was worthwhile.

V DISCUSSION

Some of the most significant findings of the research relate to the

way students use material taken into open book exams, in particular

notes that have been written by other students. Kalish’s definition of

open book exams, cited at the beginning of the article and used by many

researchers, states that in open book exams ‘the student is allowed to

make use of any materials at his disposal, including textbooks, lecture

notes, and dictionaries, but does not obtain answers either directly or

indirectly from other students’.77 Tussing, who was cited by Kalish,

went so far to assert that in open book exams, ‘Cheating with cribs and

other devices is eliminated’.78 While contemporary academics might

not be quite as confident as their 1950s counterparts, some still assume

that formal exams, including invigilated open book exams, eradicate or

minimise the possibility of plagiarism.79

However, if students are using other students’ notes in exams, as 68

per cent of our students have done,80 and they are copying directly from

notes, articles and books into their exam booklets, as 59 per cent of our

students have done, 81 it is impossible to assert that they are not

obtaining answers ‘directly or indirectly from other students’. On the

contrary, that is precisely what they are likely to be doing. Further, while

a student might reference an article or book in an exam, (and as a result

of a rise in exam plagiarism, our Faculty’s standard cover page for open

77 Kalish above n 4, 200. 78 Lyle Tussing, ‘A Consideration of the Open Book Examination’ (1951) 11

Educational and Psychology Measurement 597, cited in Kalish, above n 4, 200. 79 For example, it has been reported that the University of Technology, Sydney, ‘had

moved more in the direction of open-book exams in order to minimise cheating by

asking students to come up with creative rather than rote-learned answers’, with a staff member stating: ‘We are trying to prepare people to enter the real world of work.

The assessments are much harder to design but people can’t pass just by copying. It

is much harder to cheat in that way.’: Eryk Bagshaw, ‘Sydney University, University of NSW and UTS Crack Down on Cheating Students’, Sydney Morning Herald

(online), 23 April 2016 <http://www.smh.com.au/national/education/universities-crack-down-on-cheating-students-20160421-gobq1m.html>. This trend is not

uniform: the same article notes that the Legal Profession Admissions Board was

instituting a new closed-book exam policy in response to ‘the need for increased attention to maintaining the integrity of the Board's exams process … particularly as

a result of recent media reporting of widespread cheating in tertiary assessment

tasks.’ 80 43.5 per cent said that they had used other students’ notes in an exam once or twice,

19.5 per cent that they had done it more than once or twice and 5 per cent that they

always did so. 81 36 per cent said they had copied sections from their notes/articles/books directly into

the exam paper once or twice, 16.5 per cent said they had done it more than once or

twice and 6.5 per cent said they always did so.

22 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

book exams now instructs students to reference),82 no student is going

to reference another student’s notes. And unlike unreferenced articles

or books, with which markers may be familiar, no marker is familiar

with students’ notes, and so they are not able to detect directly copied,

plagiarised material. Further, it is likely that it may not occur to many

students that copying from another student’s notes in an exam is

plagiarism. The consequence is that it is not safe to simply assume that

exam scripts are entirely students’ own words. While closed book

exams do not eradicate the use of other students’ notes, it is arguable

that there is more academic merit in a student attempting to understand

and remember the legal detail in another student’s notes, than a student

simply copying directly from another student’s notes into their exam

script. The latter could involve no real understanding of the content

beyond being able to identify the issue being examined.83

Even if students are not copying directly from notes, articles or

books into their exam scripts, an overwhelming majority of students (79

per cent) said that they refer to their notes frequently during exams. As

noted in the literature review, research that observed students’ use of

material during exams has revealed that weaker students spend

substantially more time consulting material than stronger students.84

While not all our students are academically strong and hard-working, a

large majority are, as demonstrated by their exceptional secondary

school attainment, their responses to the survey about their work

practices, and our perception of them as teachers. This raises the

question, ‘why are very academically able students frequently referring

to notes when writing problem question exams?’ One possible answer

is that open book exams discourage students from having confidence in

their own knowledge and opinions, and that consistent with other

research, students are using materials as ‘substitutes for their own

thinking’. 85 This is concerning, particularly as an ability to think

independently and confidently will arguably be the single greatest skill

82 As a result of increased incidents of plagiarism in exams, the Faculty’s standard cover

page for open book exams states:

Note that the university’s rules on plagiarism apply to examination answers. This

means that: I. You must not copy or closely paraphrase any source or set of notes that is not your

own work without appropriate attribution.

II. Quotations must be placed in quotation marks and the source acknowledged, typically the case or author’s name is sufficient (eg Smith; D McBarnett).

III. In addition you must acknowledge the author or judgment when using their arguments or ideas, eg ‘McBarnett argues that there are two tiers of justice.’

83 Readers might wonder why students copy out prewritten material into problem

question exams or how they might be awarded marks for doing so. Students are

engaging in what academics describe as ‘information dump’. For example, if a

problem question relates to easements, students will typically copy out the four

criteria for validity of easements in Re Ellenborough Park [1956] Ch 131. Even if the material is partially or even wholly irrelevant, having been copied from notes it

is unlikely to be wrong. While a student will not earn a high mark for answering

exams this way, they are also unlikely to fail. 84 Boniface, above n 7; Moore and Jensen, above n 21. 85 Ioannidou, above n 18, 137.

2018______________________________________________CLOSED BOOK EXAM 23

our students can have in a profession where mundane and predictable

work (like cutting and pasting) will increasingly be automated.86

Although the closed book exam did not alter all students’ study and

exam preparation, for a significant number of students it did produce

what most academics would agree are desirable work practices; that is,

students did more reading throughout semester, they asked more

questions in class, and they made more of an effort to write notes in

their own words. For the students who admitted to doing less reading

for open book exams, these results were most pronounced.

Similarly, the closed book format produced more desirable exam

practices for a significant minority of students, including thinking more

during the exam and not including material that they did not entirely

understand. Just over half of the students said that the closed book exam

encouraged them to write in their own words. Again, the better exam

practices were most pronounced for the lesser-reading students.

The most negative finding of the research was that 83 per cent of

students said that the closed book format required them to memorise

‘substantial amounts of material’. We would make two comments on

this. First, it is possible that this was a result of a failure on our part to

adjust the course content. From the beginning of semester, the course

outline clearly instructed students to prepare for the exam by

understanding ‘the fundamental principles, as well as the framework of

Land Law and how different areas fit together’. Students were explicitly

told that it did not matter if they could not remember exact case names

or sections of legislation, as long as they understood their function.

However, we continued to use a casebook method of teaching which

presents students with substantial detail which they may have assumed

they needed to retain for the exam. This suggests that changes to a

closed book exam format might be better made in conjunction with

adjustments to course material. This is not to suggest that assessment

form should drive course content, but that assessment should always

reflect course content. If we primarily want students to demonstrate

understanding of fundamental principles in the final exam, we should

partially limit our teaching to those fundamental principles.

Second, while memorising ‘substantial amounts of material’ is not

desirable, some retention of material is. We would challenge the idea

that closed book exams are inappropriate because lawyers will always

be able to look up material in practice.87 One of the arguments in favour

of retention of closed book exams in medicine is that doctors are

frequently required to respond to patients in circumstances in which

they cannot look up answers.88 We found no research arguing in favour

of exclusively open book exams in medicine. While lawyers may not

need to recall information in the same way as medical professionals,

86 Deloitte, ‘Developing Legal Talent: Stepping into the Future Law Firm’ (Insight

Report, Deloitte, February 2016) 4

<https://www2.deloitte.com/content/dam/Deloitte/uk/Documents/audit/deloitte-uk-developing-legal-talent-2016.pdf>.

87 See above n 6. 88 Durning et al, above n 26, 583; Heijne-Penninga et al, ‘Directing’, above n 26, e17.

24 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

there will still be circumstances in which lawyers have to rely on

retained foundational knowledge and it is misleading to teach students

otherwise. For example, it is not acceptable to read continually from

notes in court or in client meetings, nor to look up answers to every

question posed by a client or judge. Basic professional legal

competence includes the ability to answer or at least offer tentative

answers to fundamental questions from retained knowledge. 89

Assignments and take-home exams test students’ ability to provide legal

answers by looking up reference material. It is arguable that open book

exams are simply retesting the same skill but under highly artificial time

pressure; in contrast, closed book exams test a different, but equally

important skill, that is, the ability to retain and apply foundational

knowledge. Further, retention of information is not necessarily the

product of rote-learning. As academics, we frequently teach from

retained information, none of which we are likely to have memorised;

on the contrary, we remember legal principles, cases and legislation

because we understand them. That is a skill that all lawyers need and

which we must ensure our students develop.

Like the results on memorisation, the survey results on student

anxiety were concerning. We must emphasise that Durning et al’s

comment on the lack of theoretical grounding in other research on

anxiety applies equally to our study.90 We have no expertise in anxiety

research and asked the questions on anxiety simply because the topic

loomed large in student discussions.

While law students are known to have high levels of anxiety,

excessive anxiety about a closed book exam was difficult to reconcile

with the fact that all of our students have done multiple closed book

exams and excelled. All final high school exams in New South Wales

are closed book and almost without exception, all our students did very

well in those exams; it was the only way they could obtain admission

to the Faculty.91 Further, all our undergraduate cohort are completing

another degree in other faculties where closed book exams are common.

While mental health is quite rightly an area of great concern in

education, the legal profession and in relation to young people

generally,92 it is possible that when objecting to the closed book exam

89 See comment above from one of the lecturers in the staff survey who is also an

experienced, practising barrister. The lecturer said that in legal practice it is ‘not

correct to suggest that we do not also recognise potential legal issues from having a

proper understanding and recollection of basic principles’. 90 See above n 26. 91 See above nn 48 and 49. 92 David Said, Kypros Kypri and Jenny Bowman, ‘Risk Factors for Mental Disorder

Among University Students in Australia: Findings from a Web-Based Cross-

Sectional Survey’ (2013) 48 Social Psychiatry and Psychiatric Epidemiology 935,

936; A John et al, ‘Recent Trends in the Incidence of Anxiety and Prescription of

Anxiolytics and Hypnotics in Children and Young People: An E-Cohort Study’

(2015) 183 Journal of Affective Disorders 134, found that for children and young people incidence of anxiety symptoms recorded in primary care had tripled since

2003, but that anxiety diagnoses remained relatively stable. However new

prescriptions for anxiolytics (anxiety inhibiting medication) had significantly increased for 15–18-year olds. Other studies have found marked increases in the

prescription of stimulants and antidepressants for children and adolescents: Mark

2018______________________________________________CLOSED BOOK EXAM 25

during semester, the students consciously or unconsciously felt they

would get most traction from focussing on a closed book exam’s

potential to generate stress and anxiety. A number of comments in the

survey identified stress and anxiety as the most significant or even sole

reason for objecting to a closed book exam. They also linked stress to

having to do a closed book law exam for the first time in the latter part

of their law degree. For example, one student said:

As someone with anxiety issues, the timing of the exam was difficult. It was

introduced in fourth year, at a time where marks are particularly important

to getting into honours or obtaining a clerkship. It should have been

introduced earlier in law school, based on these factors.

It does appear that the prospect of a closed book exam generated

significant anxiety for some students. This raises the further question of

how law schools should respond to student anxiety. Earlier studies

suggest that law students experience higher levels of stress due to a

range of factors within the law school environment, rather than it being

due to the makeup of the cohort.93 Based on those studies, and their own

study of 647 law students in Queensland, Bergin and Pakenham

conclude that ‘a significant proportion of Australian law students are

sufficiently stressed to warrant intervention’. 94 Consequently, they

argue that ‘[w]here possible, it is imperative that law schools act to

reduce … sources of stress [and where] not possible, it is crucial that

both law students and law schools implement strategies to manage

stress and promote well-being’.95

There is, however, considerable scope for debate about the way law

schools should best respond to stress and support students to develop

resilience as they prepare to enter a challenging profession. In some

circumstances, it may do students a disservice to simply remove sources

of stress. With respect to assessment, the priority is to find methods that

are pedagogically justified and best suited to the aims of the course.

While the impact on student anxiety needs to be considered, avoiding

the use of a pedagogically sound form of assessment because students

report finding it stressful is a step that cannot be taken lightly. It is also

significant that students are frequently exposed to closed book exams

in other degrees. There are steps that might be taken to make the use of

closed books exams less stressful, such as carefully explaining the

rationale for their use, giving students the opportunity to practise

throughout the semester, providing feedback, monitoring the impact on

student workloads and ensuring that students are aware that previous

exams results for open and closed book exams have been similar.96

Olfson et al, ‘National Trends in the Use of Psychotropic Medications by Children’

(2002) 41 Journal of the American Academy of Child and Adolescent Psychiatry 514. 93 See Bergin and Pakenham, above n 68, 388–90, and the studies cited therein. 94 Ibid 401. 95 Ibid. 96 See ibid 389–90.

26 LEGAL EDUCATION REVIEW_________________________________VOLUME 28

VI CONCLUSION

The results of this research are important for a number of reasons.

First, while there is some research from other disciplines analysing

closed and open book exams, much of it involves discipline-specific or

small-scale studies. In relation to law, the empirical research is almost

non-existent. As a result, this research, based on a staff and large student

survey, makes an important contribution to legal education research.

Second, the research concerned an attempt to address challenges

faced by academics in many disciplines; that is, the effect of technology

on students’ work practices, in particular the widespread sharing and

use of student notes, inside and outside a student cohort. While the

research does not claim to have identified a single, correct response to

these challenges, it describes one possible response and some positive

outcomes. It demonstrates that academics should be both prepared and

allowed to trial new assessment, including those that may not be

perceived as ‘progressive’.

Consistent with all other research on closed and open book exams,

our research demonstrated that there are advantages and disadvantages

to both forms of assessment. The survey of staff demonstrated

academics’ support for a closed book exam in the compulsory land law

course. The statistical comparison of students’ marks between years

revealed no significant difference between results achieved in closed

and open book exams relative to the other forms of assessment in the

course.

The student survey had some important findings that may assist

academics with similar concerns to our own. The survey demonstrated

that closed book exams create anxiety in students, along with a real

and/or perceived obligation to memorise material. Altering course

content to reduce excessive detail may alleviate the stress that students

experience when being required to retain information for closed book

exams. The survey also revealed that closed book exams encouraged

students to learn differently. A significant minority of students reported

reading more, asking more questions and preparing differently for the

exam. Perhaps most importantly, the research demonstrated that most

students doing open book exams, including conscientious students,

refer to material frequently during exams, and that the majority have

copied directly from that material into their exam scripts at some point.

This raises concerning questions about whether open book exams

discourage students from having confidence in their own intellectual

ability. Moreover, as the material students rely on in exams is likely to

contain notes from other students with which markers will not be

familiar, the risk of plagiarism in open book exams was clearly

demonstrated by the research.

All forms of assessment have flaws and in choosing assessment we

are balancing the advantages and disadvantages of each form. As a

result of the mixed findings about closed and open book exams, we

conclude that when used well closed book exams can be a useful

addition to a balanced assessment strategy.