The impact of case studies formatively and summatively assessed on students’ examination performance.

Geeta Hitch (Senior Lecturer, Dept of Pharmacy)

Janet Webber (Senior Lecturer, Dept of Physiotherapy/Principal Lecturer LTI)

Background

• PMM module /2nd year MPharm degree • Previously taught as lectures only• Concern- ‘decontextualised’ as well as there

was a divide between experience of learning and that as a practitioner

Aim

• To investigate the the impact of case studies formatively and summatively assessed on students’ examination performance

Seven Principles of Good Practice in ‘Higher’ Education (Chickering & Gamson, 1987).

• Encourages contact between students and lecturers • Develops reciprocity and cooperation among students • Encourages active learning • Gives prompt feedback • Emphasises time on task • Communicates high expectations • Respects diverse talents and ways of learning

Method

• Use of ‘hybrid’ PBL in PMM module– Bridge the divide between theory and practice; real-life

situation problems. • Incorporation of constructive alignment (Biggs, 2000)

in the use of problem solving exercise– exam question format is case study style – Inclusion summative assessment of presentation &

poster– PMM module 50% exam (25% on case study style

question); 50% CW (of which 5% was allocated to this assignment)

Method: encompassed Seven Principles of Good Practice in ‘Higher’ Education (Chickering & Gamson, 1987).

• Students placed in groups of 6; communicated with peers online via Studynet discussion site

• ‘Hybrid’ PBL- case study with structured questions

• Tutor acted as facilitator

• Students given a timeline• Constant feedback was provided online• Had an oral presentation /poster (10 mins)

– Followed by Q&A– Assessed by panel of 3 lecturers– CW mark of 5% in summatively

assessed cohort

– Encourages contact between students and lecturers

– Develops reciprocity and cooperation among students

– Encourages active learning – Respects diverse talents and ways of

learning • Emphasises time on task • Gives prompt feedback

– Encourages contact between students and

lecturers – Develops reciprocity and cooperation

among students – Encourages active learning – Communicates high expectations

• Respects diverse talents and ways of learning

Exam question:25% of the marks- had a choice of doing one of the 2 case study style based questions

Method• Cohorts in which study was carried out:

Academic year Lectures only (no hybrid PBLs so used as Control Group)

Lectures plus Hybrid PBLs (formatively assessed)

Lectures plus Hybrid PBLs (summatively assessed)

2007-08 (n=95) ×

2008-09(n=123)

×

2009-10(n=132)

×

Feedback Questionnaire

× ×

Data analysis

• Overall examination performance between all 3 cohorts was compared- Statistical Package for Social Science (SPSS for MS windows version 16). – The Students t test was used to compare the probability level set

at 5% i.e. P < 0.05. If the calculated p- value was below the threshold chosen for statistical significance of 0.05, then the null hypothesis was rejected in favour of the alternative hypothesis.

– Therefore any P values less than 0.05 were regarded as significant. • Null hypothesis: assessed case studies should make NO

impact on students’ final exam performance• Alternative hypothesis: assessed case studies should make

an impact on students’ final exam performance

Data analysis

• Questionnaires– A qualitative analysis of the student feedback

questionnaire was also carried out from the academic cohorts of 2008-09 (‘formative cohort’) and 2009-10 (‘summative cohort’)

– Group dynamics and the impact of inclusion of summative and formative assessment was examined as to whether it was a driving force in participation of group work.

Results: Percentage of student cohorts in 2007-08 (Control cohort), 2008-09 (formative cohort) and 2009-10 (summative cohort) showing marks range scored (%) in the final PMM Examination question.

0 to

und

er 1

0

10 to

und

er 2

0

20 to

und

er 3

0

30 to

und

er 4

0

40to

und

er 5

0

50 to

und

er 6

0

60 to

und

er 7

0

70 to

und

er 8

0

80 to

und

er 9

0

90 to

100

0

5

10

15

20

25

30% of students 2007-08 cohort (Lectures only; No case studies)

% of students 2008 -09 cohort lec-tures and (Formative assessment of case studies)

% of students 2009-10 cohort lectures and (Summative assess-ment of case studies)

% s

tude

nts

Range of marks (%)

Mean marks:2007-08: 38.48 %2008-09: 38.41 %2009-10: 46.04 %

Results

40 to

und

er 5

0

50 to

und

er 6

0

60 to

und

er 7

0

70 to

und

er 8

0

80 to

und

er 9

0

90 to

100

tota

l % o

f stu

dent

s sc

orin

g fr

o...

0

10

20

30

40

50

60

70

80

% of students 2007-08 cohort (Lectures only; No case studies)

% of students 2008 -09 cohort lectures and (Formative assessment of case stud-ies)

% of students 2009-10 cohort lectures and (Summative assessment of case studies)

% s

tude

nts

Percentage of student cohorts in 2007-08, 2008-09 and 2009-10 scoring more than 40% in final PMM Examination question

More than 65% of students have scored greater than 40% in their final PMM exam question in 2009-10 cohort (summative vs lectures only) p≤0.0001;

Results

• No significant difference in exam performance for students when formatively assessed case studies were used (2008-09 cohort) and when lectures only were used (2007-08 cohort) to deliver the curriculum (student T test; p≤0.97).

• Significantly better overall performance in the relevant exam question when the case studies were summatively assessed in comparison to formative assessment (student T test; p≤0.0005);

2007-08 (control cohort)

0 to under

10

11 to under 20

20 to under 30

30 to under 40

40 to under 50

50 to under 60

60 to under 70

70 to under 80

80 to under 90

90 to 1000

5

10

15

20

25

30

35

40

45

2007-08 MPP3 exam

2007-08 PMM exam

Mean mark: PMM: 38.48 %MPP3: 55.38 %

% s

tude

nts

Marks range %

Comparison of final exam performance in the same cohort between 2 different pharmacy modules (PMM- pharmaceutical Microbiology & Manufacture and MPP3- Medicines & Professional Practice Level 3)

2008-09 (formative cohort)

0 to under

10

11 to under

20

20 to under

30

30 to under

40

40 to under

50

50 to under

60

60 to under

70

70 to under

80

80 to under

90

90 to 100

0

5

10

15

20

25

30

35

40 2008-09 MPP2 exam

2008-09 PMM exam

Mean mark:PMM: 38.41 %MPP2: 51.5 %

% s

tude

nts

Marks range %

Comparison of final exam performance in the same cohort between 2 different pharmacy modules (PMM- pharmaceutical Microbiology & Manufacture and MPP2- Medicines & Professional Practice Level 2)

2009-10 (summative cohort)

0 to under

10

11 to under

20

20 to under

30

30 to under

40

40 to under

50

50 to under

60

60 to under

70

70 to under

80

80 to under

90

90 to 100

0

5

10

15

20

25

30

35

40

452009-10 MPP1 exam

2009-10 PMM exam

Mean Mark:PMM: 46.05 %MPP1: 51.2 %

% s

tud

ents

Marks range %

Comparison of final exam performance in the same cohort between 2 different pharmacy modules (PMM- pharmaceutical Microbiology & Manufacture and MPP1- Medicines & Professional Practice Level 1)

Cohorts general performance in other examinations

• All 3 cohorts appear to be performing at the same level in 2 sets of exams results compared.

• This bears a significance in terms of inclusion of case studies because:– The overall performance appears to be similar in

2007-08 and 2008-09 cohorts. – The overall exam performance in case study style

based question is performed significantly better in the 2009-10 summative assessed cohort.

Questionnaires• Response rate:

– 65% from 2008-09 (n=123) (Formative cohort)– 73% from 2009-10 (n=132) (Summative cohort)

• Enjoyed taking part in group work:– 75% of formatively assessed cohort - yes– 90% of summative assessed cohort- yes• 58% of students participated in equal input into group work when case

studies were formatively assessed in comparison to 83% when summatively assessed.

• When students were asked if they participated more when summative assessment took place,

– 76% replied yes and 66% of these stated that they were driven by a genuine desire to study by participating in the case study.

– The other 34% were just performing a task as it was summative and exam marks mattered more to them.

Questionnaires

– 56% of the formatively assessed cohort felt that they learnt better in group work than working alone

– 64% of the summative assessed cohort felt that they learnt more in group work than by working alone.

Questionnaires

• Formative assessments were not viewed as important to the students in achieving their learning outcomes for a variety of reasons. In the case of this cohort (2008-09), several reasons were cited by students in the feedback questionnaire:– “Too many summative assignments to hand in and therefore

formative assignments are my last priority”– “Many students tend to rely on others to do the group work and

then when they are told to get on with it, the work they hand in is not reliable”

– “Some students were clearly not willing to prepare or participate in the presentations as there was no marks awarded for this work and so the rest of the group ended up doing their part of the work”

Discussion

• Gibbs, (1999) - “Assessment is perceived by students as the curriculum and as such the power of assessment needs to be used strategically to help students learn”

• Biggs (2002) identifies this- states that “students will only learn what they think will be assessed on as opposed to what is required of them to learn in the curriculum”.

Discussion

• General observations: inclusion of case studies does improve overall performance- students engage more effectively when learning is contextualised and is of relevance to practice

• Students also perform better when marks are allocated to assignments adding to overall module mark

• If marks are not allocated towards assignments and towards module marks in general, students can be reluctant to participate in CW elements of the curriculum.

Finally we conclude:

“That by inclusion of the 7 principles of good practice in undergraduate education, we have shown in this small study that how we practice the teaching is as much as important as to what the students learn.”

Seven Principles of Good Practice in ‘Higher’ Education (Chickering & Gamson, 1987).

• Encourages contact between students and lecturers • Develops reciprocity and cooperation among students • Encourages active learning • Gives prompt feedback • Emphasises time on task • Communicates high expectations • Respects diverse talents and ways of learning

References

1. Chickering, A. W. & Gamson, Z. F. (1987). ‘Seven principles for good practice in undergraduate education.’ American Association for Higher Education Bulletin, 39(7), 3–7.

2. Gibbs, G (1999) Using assessment strategically to change the way students learn, in: S. Brown & A. Glasner (Eds) Assessment Matters in Higher Education (Buckingham, SRHE & Open University Press).

3. Biggs, J. (2002) Aligning the curriculum to promote good learning, paper presented at the Constructive Alignment in Action: Imaginative Curriculum Symposium, LTSN Generic Centre, November 2002. Available online at: www.ltsn.ac.uk

Thank you

Any questions?