Using teacher-to-student feedback comments in rating

scale developmentRicky Jeffrey

EdD candidateSchool of Education, University of Nottingham Ningbo

Chinazx15255@nottingham.edu.cn

Context

• One-year EAP programme with a range of modules• 1500 students per year• Around 60 teaching staff

Rating

Task Fulfilment and Organisation

5

Task addressed comprehensively. Thorough understanding and application of ideas from texts using appropriate referencing conventions. Clear and sophisticated stance. Effective paragraph organisation/text structure to support stance.

4

Task addressed fully. Clear understanding and application of ideas from texts using appropriate referencing conventions. Clear stance with a degree of sophistication. Appropriate paragraph organisation/text structure to support stance.

3

Task addressed. Adequate understanding and application of ideas from texts using appropriate referencing conventions. Clear stance. Adequate paragraph organisation/text structure to support stance.

2

Task addressed but could be developed further. Acceptable but inconsistent understanding and application of ideas from texts. Acceptable but inconsistent use of the appropriate referencing conventions.

Evidence of simple stance. Acceptable but inconsistent paragraph organisation/text structure to support stance.

1

Task attempted but is only partly addressed, or has been misinterpreted. Partial understanding and application of ideas from texts, which lacks clarity or depth. Limited understanding of referencing conventions. Indication of basic stance but lacks clarity. Paragraph organisation/text structure is mostly inappropriate or illogical.

Rater

Scale

Test-taker response

Score

Problem: Vague, irrelevant descriptors5

Effective paragraph organisation/text structure to support stance.

4

Task addressed fully. Clear understanding and application of ideas from texts using appropriate referencing conventions. Clear stance with a degree of sophistication. Appropriate paragraph organisation/text structure to support stance.

3

Task addressed. Adequate understanding and application of ideas from texts using appropriate referencing conventions. Clear stance. Adequate paragraph organisation/text structure to support stance.

2

Task addressed but could be developed further.

• Different raters interpret scale descriptors in different ways – reduced reliability

• Raters spend more time and effort trying to interpret – strain on resources

• Raters get frustrated – reduced staff motivation

“Empirical” scale development

Task Fulfilment and Organisation

5

Task fulfilled comprehensively. Stance is clear. The argument is logical, well-organised, and easy to follow, and has some elements of sophistication. Paragraph structure is effective. Claims are supported by a range of sources, with effective source synthesis. Citations and references are accurate, apart from rare minor lapses (e.g. "Date accessed" is not always provided for websites), and they match in all or almost all cases.

4

Task fulfilled. Stance is clear, perhaps with minor lapses. The argument is mostly logical and well-organised. Introduction and conclusion are mostly effective. Paragraph structure is mostly effective. Claims are generally supported by sources. There is some effective source synthesis, but this could be developed further. Citations and references are mostly accurate, and they mostly match.

3

Task addressed. Stance is generally clear, but may have some problems. Introduction and/or conclusion have some problems (e.g. introduction is too long). A thesis statement is present, although it may not be entirely effective. Paragraph structure has some problems (e.g. some topic sentences unclear).

Claims are not always supported by sources. More source synthesis is needed. There may be over-reliance on particular sources. Citations and references show a number of types of error, and there are a number of mismatches.

2

Task addressed, but with problems (e.g. confusion about one part of the task). Stance is not clear. Introduction and/or conclusion are missing important features or have signficant problems (e.g. conclusion introduces several new sources). Thesis statement is ineffective or absent. Paragraph structure frequently ineffective.

Claims need more support from sources, and a wider range of sources is needed. Source synthesis is minimal or absent. The essay does not show enough understanding of the reading. Citations and references are frequently inaccurate, with some basic errors (e.g. citations do not include name and year). There are many mismatches.

1

Task only partly addressed. Some parts of the text are irrelevant to the task. Stance is not clear. The text is illogical and difficult to follow. Introduction and conclusion display several basic problems (e.g. introduction does not indicate stance). Paragraphs are badly organised and mostly ineffective.

Claims often unsupported by sources. There are not enough citations, so that long sections are presented without any citation. No source synthesis. Not enough reading has been done, and key ideas have been misunderstood or ignored. Several inappropriate sources may be used. Citations and references have basic errors throughout, and may at times be incomprehensible. They show many mismatches, or even mostly mismatch.

• Strain on resources• Artificial conditions –

potential reactivity

Spoken verbal

protocols

Improved scale

Lots of responses

Group of raters

An overlooked part of the process

Task Fulfilment and Organisation

5

Task addressed comprehensively. Thorough understanding and application of ideas from texts using appropriate referencing conventions. Clear and sophisticated stance. Effective paragraph organisation/text structure to support stance.

4

Task addressed fully. Clear understanding and application of ideas from texts using appropriate referencing conventions. Clear stance with a degree of sophistication. Appropriate paragraph organisation/text structure to support stance.

3

Task addressed. Adequate understanding and application of ideas from texts using appropriate referencing conventions. Clear stance. Adequate paragraph organisation/text structure to support stance.

2

Task addressed but could be developed further. Acceptable but inconsistent understanding and application of ideas from texts. Acceptable but inconsistent use of the appropriate referencing conventions.

Evidence of simple stance. Acceptable but inconsistent paragraph organisation/text structure to support stance.

1

Task attempted but is only partly addressed, or has been misinterpreted. Partial understanding and application of ideas from texts, which lacks clarity or depth. Limited understanding of referencing conventions. Indication of basic stance but lacks clarity. Paragraph organisation/text structure is mostly inappropriate or illogical.

FeedbackThe stance of the essay is made fairly clear in the introduction. Although this task does require an evaluation, you should still avoid emotional language (e.g. "commendably", "outstanding") in academic writing. Make sure to provide references when referring to others' ideas (e.g. the point about Drucker does not have any citation as support). Examples are generally used well to support the argument. You do need to be more careful about your referencing: there are several mismatches between in-text citations and the reference list (e.g. Berlo). The argument is linked together well with cohesive devices, which makes the argument easy for the reader to follow. Your use of lexis and grammar makes your meaning mostly clear, and you do well to attempt more complex grammar structures.

Feedback

Feedback contains rater cognitions

FeedbackYour stance is made clear in the introduction. Make sure to provide references when referring to others' ideas (e.g. the point about Drucker does not use any citation). Examples are generally used well to support the argument. Be careful with referencing: there are several mismatches between in-text citations and the reference list. Your use of lexis and grammar

• Feedback contains rater cognitions about a test-taker performance, and is associated with a score

Feedback-informed empirical scale development

Task Fulfilment and Organisation

5

Task fulfilled comprehensively. Stance is clear. The argument is logical, well-organised, and easy to follow, and has some elements of sophistication. Paragraph structure is effective. Claims are supported by a range of sources, with effective source synthesis. Citations and references are accurate, apart from rare minor lapses (e.g. "Date accessed" is not always provided for websites), and they match in all or almost all cases.

4

Task fulfilled. Stance is clear, perhaps with minor lapses. The argument is mostly logical and well-organised. Introduction and conclusion are mostly effective. Paragraph structure is mostly effective. Claims are generally supported by sources. There is some effective source synthesis, but this could be developed further. Citations and references are mostly accurate, and they mostly match.

3

Task addressed. Stance is generally clear, but may have some problems. Introduction and/or conclusion have some problems (e.g. introduction is too long). A thesis statement is present, although it may not be entirely effective. Paragraph structure has some problems (e.g. some topic sentences unclear).

Claims are not always supported by sources. More source synthesis is needed. There may be over-reliance on particular sources. Citations and references show a number of types of error, and there are a number of mismatches.

2

Task addressed, but with problems (e.g. confusion about one part of the task). Stance is not clear. Introduction and/or conclusion are missing important features or have signficant problems (e.g. conclusion introduces several new sources). Thesis statement is ineffective or absent. Paragraph structure frequently ineffective.

Claims need more support from sources, and a wider range of sources is needed. Source synthesis is minimal or absent. The essay does not show enough understanding of the reading. Citations and references are frequently inaccurate, with some basic errors (e.g. citations do not include name and year). There are many mismatches.

1

Task only partly addressed. Some parts of the text are irrelevant to the task. Stance is not clear. The text is illogical and difficult to follow. Introduction and conclusion display several basic problems (e.g. introduction does not indicate stance). Paragraphs are badly organised and mostly ineffective.

Claims often unsupported by sources. There are not enough citations, so that long sections are presented without any citation. No source synthesis. Not enough reading has been done, and key ideas have been misunderstood or ignored. Several inappropriate sources may be used. Citations and references have basic errors throughout, and may at times be incomprehensible. They show many mismatches, or even mostly mismatch.

• Already recorded – less strain on resources

• Authentic conditions – no reactivity

FeedbackThe stance of the essay is made fairly clear in the introduction. Although this task does require an evaluation, you should still avoid emotional language (e.g. "commendably", "outstanding") in academic writing. Make sure to provide references when referring to others' ideas (e.g. the point about Drucker does not have any citation as support). Examples are generally used well to support the argument. You do need to be more careful about your referencing: there are several mismatches between in-text citations and the reference list (e.g. Berlo). The argument is linked together well with cohesive devices, which makes the argument easy for the reader to follow. Your use of lexis and grammar makes your meaning mostly clear, and you do well to attempt more complex grammar structures.

More responses

More raters

FeedbackThe stance of the essay is made fairly clear in the introduction. Although this task does require an evaluation, you should still avoid emotional language (e.g. "commendably", "outstanding") in academic writing. Make sure to provide references when referring to others' ideas (e.g. the point about Drucker does not have any citation as support). Examples are generally used well to support the argument. You do need to be more careful about your referencing: there are several mismatches between in-text citations and the reference list (e.g. Berlo). The argument is linked together well with cohesive devices, which makes the argument easy for the reader to follow. Your use of lexis and grammar makes your meaning mostly clear, and you do well to attempt more complex grammar structures.

FeedbackThe stance of the essay is made fairly clear in the introduction. Although this task does require an evaluation, you should still avoid emotional language (e.g. "commendably", "outstanding") in academic writing. Make sure to provide references when referring to others' ideas (e.g. the point about Drucker does not have any citation as support). Examples are generally used well to support the argument. You do need to be more careful about your referencing: there are several mismatches between in-text citations and the reference list (e.g. Berlo). The argument is linked together well with cohesive devices, which makes the argument easy for the reader to follow. Your use of lexis and grammar makes your meaning mostly clear, and you do well to attempt more complex grammar structures.

Written feedback

Improved scale

Detailed method: CodingAlthough (1) you do have a thesis statement, this does not address the second part of the question. In fact, (2) the second question has been rather poorly dealt with as you only appear to refer to the Mao era in China and not to contemporary China as required. In the conclusion, you consider two possible ways the theory can be used in China, indicating that not enough thought had been given to this previously. Note also that (3) a conclusion should summarise the key points of the essay. In the first part of the essay, there are (4) some ideas, but not all have been supported. Also (5) the only source used for the second body paragraph (McGregor) is (6) misrepresented and (7) incorrectly cited. Although your language is understandable throughout, work on producing a more sophisticated precise academic style. Include more hedging in your work, rather than phrases such as ‘obviously’ and ‘as we all know’, and a wider range of cohesive devices rather than relying on firstly/secondly.

(1) Thesis statement: Thesis statement does not address the second part of the question. (2) Task fulfilment/relevance: Task not fully addressed (i.e. second part of the task not clearly addressed, and only Mao-era China discussed). (3) Conclusion: Conclusion has a problem (i.e. does not summarise key points of the essay). (4) Source support for claims: Some claims are not supported by sources. (5) Source support for claims: Some parts of the essay do not use enough source support (i.e. only 1 source used in body paragraph 2). (6) Understanding of the reading: Misunderstanding of sources on at least one occasion. (7) Form of citations and references: Form of citations inaccurate on at least one occasion.

Detailed methodSample• 150 feedback

comments

Coding• Each comment coded

into a set of evaluative statements

• Coding scheme with 20 categories

Identify consensus statements• Any statement expressed

by >1 raters at a band• If there are contradicting

statements, ratio must be greater than 2:1

'Exploded' scale• Evaluative statements

used to create descriptors on an exploded scale with 19 columns

Assessor-oriented scale• Descriptors combined

to create a scale convenient for operational rating

Piloting

5

Effective paragraph organisation/text structure to support stance.

4

Task addressed fully. Clear understanding and application of ideas from texts using appropriate referencing conventions. Clear stance with a degree of sophistication. Appropriate paragraph organisation/text structure to support stance.

3

Task addressed. Adequate understanding and application of ideas from texts using appropriate referencing conventions. Clear stance. Adequate paragraph organisation/text structure to support stance.

2

Task addressed but could be developed further.

An improved, empirically-grounded scale

Curriculum coherence as knowledge management

• Feedback-informed scale development can’t replace other approaches to scale development – but it can usefully supplement other approaches

• More broadly, it’s another way in which verbal information produced within an organisation can be mined to capture knowledge – which can then be communicated to benefit later processes

References

• Jeffrey, R. 2015. Using feedback comments to develop a rating scale for a written coursework assessment, Journal of English for Academic Purposes, 18 (Special issue: Assessment in EAP), http://dx.doi.org/10.1016/j.jeap.2015.03.002

• Icons made by Freepik at www.flaticon.com

Thank you for listening.

And a very big thanks to the British Council

for their support.