VIEWING SUMMATIVE ASSESSMENT

THROUGH DIFFERENT LENSESPeter Reed

eAssessment Scotland Keynote Presentation

SELF REFLECTION

STUDENTS PEERS

CONTINUING THE THEME…

PROGRAMME SYSTEM(S)

LITERATURE

ANALYTICS

DATA CAPTURE

LIFTUPP TURNITIN

ANALYTICS @ LIVERPOOL

Curriculum Review – mapping to GDC domains

What should a Dentist be able to do upon qualification?

LIFTUPP

1 2 3 4 5 6

WORKING BELOW EXPECTED LEVEL

WORKING AT EXPECTED LEVEL

WORKING ABOVE EXPECTED LEVEL

DEVELOPMENTAL INDICATORS

STUDENT VIEW

Particular domain

STUDENT VIEW

Skills within domain & development indicator

STUDENT VIEW

Areas of the mouth & procedures

STUDENT VIEW

Materials used

STUDENT VIEW

STAFF VIEW

GENERATED DATA

TURNITIN / GRADEMARK

Pilot in School of Medicine SSMs

• Tutor name included in submission title;• SSMs marked against 7 objectives (intro, methodology, etc);• Objectives marked as Unsatisfactory (1), Fair (2), Good (3), Excellent (4);• Total possible marks was 7 * 4 = 28;• 6 Markers & 36 Student assignments;

TURNITIN / GRADEMARK

Unsatisfactory Fair Good Excellent0

20

40

60

80

100

120

Count of marks awarded

Coun

t

TURNITIN / GRADEMARK

Ob 1 Ob 2 Ob 3 Ob 4 Ob 5 Ob 6 Ob 70

5

10

15

20

25

Frequency of Grades Awarded for Each Objective

UnsatisfactoryFairGoodExcellent

TURNITIN / GRADEMARK

Objective 1 Objective 2 Objective 3 Objective 4 Objective 5 Objective 6 Objective 70

20

40

60

80

100

120

140

Number of QM Comments Made by Objective

TURNITIN / GRADEMARK

Marker 1 Marker 2 Marker 3 Marker 4 Marker 5 Marker 60

5

10

15

20

25

30

Mean Points awarded by marker

TURNITIN / GRADEMARK

1 2 3 4 5 6 70

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Mean Points Awarded per Objective, by Marker

Marker 1Marker 2Marker 3Marker 4Marker 5Marker 6

QUESTIONS

Is anybody doing anything with this information?

Should we?

Share with Staff? With Students?

Ethical considerations?

Hawthorne effect?

Predictive analytics?

Email Interviews - Do you think the data available from the system is useful and why?

EMAIL INTERVIEWS

“Yes, no Hawthorne for me, I didn't know marks were being assessed in that way. Helps reassure students and externals that we are monitoring for consistency.”

“I think it very useful as it will make it easy to identify conveners who mark outside normal parameters and identify who is giving feedback to the students and who isn't.”

EMAIL INTERVIEWS

“I’m not quite sure what I think about this. Obviously we’re all

interested in what our own performance is (although I’m not sure we

should be told!). Major inconsistencies between markers or between

objectives would be worrying, but I don’t think we have a solid basis

for expecting any particular result – to put it crudely, would ‘all

excellent’ be a good thing (great students and course-management)

or bad (slack marking)?”

ASSESSMENT ANALYTICS

For the Greater Good or Big Brother?