how honest are the students when assessing each other’s work? diogo montalvão

1/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

How honest are the students when assessing each other’s work?

Diogo Montalvãod.montalvao@herts.ac.uk

Senior Lecturer, Dipl MSc PhD CEngSchool of Engineering & Technology

2012 Annual L&T Conference, 3rd May

2/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

1. Why student’s peer-assessment?2. Description of a case study at IST3. Some of the students’ works4. Peer-assessment results5. Conclusions

Outline

3/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Learn from their peers

Encourages critical analysis of student work

Increased feedback to

students

Helps them perceiving their

own learning

Helps them interpreting assessment

criteria

Why student’s peer-assessment?Student benefits

source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

4/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Summative grading purposes

Reduced marking loads

To get students more involved and to make

them take more responsibility

Assess individual effort in team work

Engagement

Why student’s peer-assessment?Teacher benefits

source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

5/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Why student’s peer-assessment?

Disadvantages may be encountered when students:

• Lack the ability to evaluate each other

• Do not take it seriously

• Fear discrimination

source: http://www.heacademy.ac.uk/hlst/resources/a-zdirectory/peer_assessment

6/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

1. Why student’s peer-assessment?2. Description of a case study at IST3. Some of the students’ works4. Peer-assessment results5. Conclusions

Outline

7/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Description of a Case Study at IST**Instituto Superior Técnico, Technical University of Lisbon

Modules:• Technical Drawing and Geometrical

Modelling I & II

• 1st year students• 2 groups/module, ~50 students/module

• BEng / MEng in Mechanical, Aerospace and Marine Engineering

• 2010/2011 session

8/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Description of a Case Study at ISTTechnical Drawing and Geometrical Modelling I & II - Assessment

In-class tests30%

Group presenta-

tions10%

Group project40%

CAD* assign-ment10%

“Other” tests10%

*CAD – Computer Aided Design

In-class tests30%

Group presenta-

tions10%

Group project40%

CAD* assign-ment10%

“Other” tests10%

“Other” tests 10%

9/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

(“Other” tests – 10%)Every week in-class 15~30 mins assignments marked from 0 to 1

0 1

Wrong Right

Goals:• To engage the students in practice and study• Without significant increase of the marking load

How do we define the boundary in a binary marking scheme for drawings?

This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

10/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

(“Other” tests – 10%)Every week in-class 15~30 mins assignments marked from 0 to 1

0 1

Wrong Very good

Goals:• To engage the students in practice and study• Without significant increase of the marking load

How to differentiate a student that does not deliver the assignment from the others?

.5

Acceptable

This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

11/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

(“Other” tests – 10%)Every week in-class 15~30 mins assignments marked from 0 to 1

0 1

Zero

Goals:• To engage the students in practice and study• Without significant increase of the marking load

.5.25 .75

Poor Accept. Good Very good

2 levels of fail 3 levels of pass

Problems:• Poor feedback• “Ethical” question: marking is based on comparison

This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

12/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

(“Other” tests – 10%)Examples of “other” tests

This slide is outside the main scope of the presentation (peer-assessment), but is shown for helping the author explain a concept.

1 0.75 0.5

13/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Description of a Case Study at ISTTechnical Drawing and Geometrical Modelling I & II - Assessment

In-class tests30%

Grouppresenta-

tions10%

Group project40%

CAD* assign-ment10%

“Other” tests10%

*CAD – Computer Aided DesignLecturer

5%Peer-assessment

5%

In-class tests30%

Public presenta-

tions10%

Group project40%

CAD* assign-ment10%

“Other” tests10%

Group presentations

10%

14/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Description of a Case Study at IST

• Done in-class, during last week

• 10~15 minutes/group

About the public presentation

• Students had to attend all the presentations

• Marks ranged from 1 to 10 (0 for no shows)

• Peer-assessment marks kept confidential by the lecturer

• Non-differentiating peer-assessment marks were rejected

1-2 9-105-63-4 7-8Very Poor Poor Accept. Good Very good

15/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

1. Why student’s peer-assessment?2. Description of a case study at IST3. Some of the students’ works4. Peer-assessment results5. Conclusions

Outline

16/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Mower

Rodrigo Ferreira

Boring mill

Wilson CarreiraHélder Camacho

Filipe Moreira

Some of the students’ works

Reactive suspension

Leonor NetoSoraia Ribeiro

17/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Some of the students’ works

A Racing bike!!! (Solidworks)

(Author deliberately left anonymous)

18/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

1. Why student’s peer-assessment?2. Description of a case study at IST3. Some of the students’ works4. Peer-assessment results5. Conclusions

Outline

19/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

First observation:• Both the teacher’s and students’ marks follow a Gaussian

distribution.

20/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

21/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

The students did not fail their colleagues

22/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

The two students with very poor presentations are given a pass by their

peers...

23/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

... and the four students just below pass were given a comfortable pass by their peers

24/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

72% of the marks given by the

students are in the 7-8 (Good) range

25/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

However, the students seem to

be afraid to overrate too much their colleagues

26/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

• The averaged values are not very different• The teacher’s standard deviation is practically twice as much

as the peers’

27/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Peer-assessment results

The marks are concentrated around the same mode, although the teacher’s assessment “leaks” towards the whole marking range.

28/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

1. Why student’s peer-assessment?2. Description of a case study at IST3. Some of the students’ works4. Peer-assessment results5. Conclusions

Outline

29/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Conclusions

Overall conclusions from this study:

• Students have a tendency to overrate their “mates”

• They are able to make a correct relative judgement between their peers’ performances (and thus themselves)

• Most often, they will avoid to:• Fail their “mates”• Give their peers topper marks

30/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

Conclusions

How honest are the students when assessing each other’s work?

Students move by their own code of conduct and live in a competition environment.

Qualitatively honest Quantitatively biased

31/31How honest are the students when assessing each other’s work?2012 Annual L&T Conference, Diogo Montalvão

THANK YOU

Diogo Montalvãod.montalvao@herts.ac.uk

Senior Lecturer, Dipl MSc PhD CEngSchool of Engineering & Technology

Acknowledgments are due to Prof. António Ribeiro, Assistant Professor at IST, for sharing the data from two of their groups of students.