ASSESSING WRITING

Prepared by Olga Simonova, Maria Verbitskaya, Elena Solovova, Inna ChmykhBased on material by Anthony Green

Outline of the lecture

1. Why teach writing? 2. Stages of teaching writing. 3. Writing: process vs. product.4. Why test writing? Types of tests & their

purpose for different test users.5. Writing test specifications.6. Rating scales in assessing writing skills.

WHY TEACH WRITING?

Why teach writing?

1. Children (school-based writing):– writing as a means of learning;– writing as a process;– writing as a product.2. Adults:– socio-cultural experience (travelling, migration,

etc.);– academic experience (furthering one’s education);– professional experience (career development).

STAGES OF TEACHING WRITING

6

Productive language processing

Stages of learning and teaching writing:

• graphics (level of letters/ their combinations);

• spelling (word level);• note-taking

(words/chunks/sentences/plan, etc.);• writing as a communicative skill.

Micro skills:

• produce graphemes and orthographic patterns• produce writing at an efficient speed to suit the purpose• produce an acceptable core of words and use appropriate

word-order patterns• use an appropriate grammar system (tense, agreement,

pluralisation, etc.)• express a particular meaning in different grammar forms.• use cohesive devices in written discourse.

(Brown 2003, p.221)

• use the rhetorical form and conventions of written discourse;

• appropriately accomplish the communicative functions of written texts according to form and purpose;

• convey links and connections between events;• communicate the main idea and supporting ideas;• communicate given information and the new

information, generalization and exemplification;

Macro skills:

• distinguish between literal and implied meanings when writing;

• correctly convey culturally specific references in the context of the written text;

• develop and use a battery of writing strategies such as:– assessing the audience’s interpretation;– using pre-writing devices;– writing the first draft:– using paraphrases and synonyms;– seeking peer and instructor’s feedback;– using feedback for revising and editing.

Macro skills:

Types of tasks

Copying Dictations Gap filling (word/ word

combination/ sentence level)

Scrambled words/ sentences

Sequencing/ ordering Find and correct the

mistakes

Short answer and sentence completion

Multiple choice questions Multiple matching Paraphrasing Grammatical

transformations Vocabulary exercises Cloze Picture-cued tasks

WRITING AS PROCESS AND PRODUCT

Writing: Process vs Process

Process:•drill / controlled / guided ex-s; •brainstorming, planning, all draft materials.

Product:•different types of written texts within different types of language course; •letters / e-mails / questionnaires, transactional letters; memos / minutes / business plan;•essays, reports, articles, presentations, etc.

Genres and text types

1. Academic writing: general subject reports, essays, compositions, term/ course papers, academically focused journals, short-answer test responses; technical reports (lab reports); theses, dissertation, etc.

2. Job-related writing: messages/phone messages; letters/e-mails/ transactional letters; memos; minutes; reports (job evaluation, project reports); schedules, labels, signs; announcements, advertisements; manuals, etc.

3. Personal writing: letters, emails, greeting, cards, invitations; messages, notes, reminders, shopping lists; financial documents (checks, tax forms, loan applications); forms, questionnaires, medical reports, immigration documents; diaries, personal journals; fiction, etc. (Brown 2003, p.221)

WHY ASSESS WRITING?

Construct Purpose

Task types Scoring criteria

Assessing writing: key questions

Why assess writing?

No single answer:• different groups of language learners have different needs, such as:

– international travellers: language for travel, leisure;– migrants: survival skills, access to employment;- students: notes, essays, dissertations;- professionals: emails, reports, letters, memos.

• different users have different purposes when they seek information from tests;

• but most users of language do need to write.

1. SOCIAL NEEDS:• international travellers: language for travel, leisure;• migrants: survival skills, access to employment;• professionals: emails, reports, letters, memos.

2. ACADEMIC NEEDS:• students: mobility, exchange programs, notes, essays,

course work, dissertations, national exams.3. EDUCATIONAL NEEDS:

• teachers: monitoring, control, assessment;• learners: progress check, skills development.

18

Why assess writing?

Bachman and Palmer (2010) Language Assessment in Practice

Performance testing in writing:

samples of written language that simulate real-life behaviour (are authentic in situation and interaction);

complex events involving unpredictable variables;

observed & evaluated by agreed judging process, where facets of measurement interact to produce score.

McNamara (1996)

WRITING TEST SPECIFICATIONS

Construct Purpose

Task types Scoring criteria

Assessing writing: key questions

Writing test specifications:

• topic• text type• length• areas to be assessed• timing• weighting• pass level.

(Coombe, 2011)

Parts of writing test tasks:

• rubric

• prompt / input

• expected response

• post-task evaluation.

Rubric: instructions and criteria

The instructions and the set of criteria for evaluation of the writing task.

It is supposed to:• specify a rhetorical pattern, length, time;• indicate the resources available for students

(dictionaries, spell checker, etc.) and the delivery method (paper and pencil, etc.);

• indicate whether a draft or an outline is required;• include the overall weighting of the writing task in

the exam.(Coombe, 2010)

Writing prompt

‘the stimulus the students must respond to’ Hyland (2003 p.221)3 types of prompt patterns:•base prompt (see the handouts);•framed prompt (see the handouts);•text-based prompt (see the handouts).

Kroll and Reid (1994 p. 233)

26

Expected response and relationship between input and

response:• length• purpose• text type• functions• linguistic features• interactiveness

• reactivity: degree of reciprocity;• scope: amount of input needed to respond;• directness: direct / indirect correlation / real world

knowledge involved apart from the input.(Bachman & Palmer, 1996)

RATING SCALES

Construct Purpose

Task types Scoring criteria

Assessing writing: Key questions

Task analysis

IELTS Academic Writing Task 2: UpbringingWRITING TASK 2You should spend about 40 minutes on this task,Write about the following topic:Children who are brought up in families that do not have large amounts of money are better prepared to deal with the problems of adult life than children brought up by wealthy parents.To what extent do you agree or disagree with the opinion?Give reasons for your answer and include any relevant examples from your own knowledge and experience.Write at least 250 words.

Rating scales

Holistic (see handouts)

+ fast (practicality) – not easy to interpret results/ remember all to arrive at score Analytical (see handouts)

parameters (task fulfillment, content, organization, cohesion, register, accuracy, etc.) and criteria (3-2-1-0 and their description)

+ clear, explicit, detailed diagnostic info, reliable, good for guiding learners

- time consuming

Holistic rating scales

• Positive features:– practicality – fast;– rating holistically may be more naturalistic.

• Disadvantages:– no useful diagnostic information - single score;– not always easy to interpret - raters not required

to use same criteria to arrive at score.

Analytic rating scales

Positive features:•can provide diagnostic information if scores reported separately;•potentially clear, explicit and detailed;•usually more reliable (multiple scores);•useful in training raters to focus on our construct;•potentially useful in guiding learners.Disadvantages:•time-consuming;•may overburden raters.

How to reduce subjectivity ?

• specification of the content of the assessment;• adopting standard procedures;• basing judgments on specific defined criteria;• requiring multiple judgments;• undertaking appropriate training;• moderation of judgments;• checking validity and reliability by analyzing

assessment data.