centre for research in english language learning and assessment

66
CRELLA Centre for Research in English Language Learning and Assessment Однажды…

Upload: vinaya

Post on 25-Feb-2016

38 views

Category:

Documents


0 download

DESCRIPTION

Centre for Research in English Language Learning and Assessment. Однажды …. Centre for Research in English Language Learning and Assessment. Однажды …. ….. в 1974-o м году …. Centre for Research in English Language Learning and Assessment. Assessing reading – how and why? - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Centre for Research in English Language Learning and Assessment

CRELLA

Centre for Research in English Language Learning and Assessment

Однажды…

Page 2: Centre for Research in English Language Learning and Assessment

CRELLA

Centre for Research in English Language Learning and Assessment

Однажды…

….. в 1974-oм году …

Page 3: Centre for Research in English Language Learning and Assessment
Page 4: Centre for Research in English Language Learning and Assessment

CRELLA

Centre for Research in English Language Learning and Assessment

Assessing reading – how and why?An introduction to assessing reading

for language teachers

Dr Lynda Taylor

Page 5: Centre for Research in English Language Learning and Assessment

CRELLA

Designing tasks for testing L2 reading

Page 6: Centre for Research in English Language Learning and Assessment

We need to think about….1. What types of mental (or cognitive)

processing are typically involved in reading?

2. What types of text are typically involved in reading activity?

3. How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests?

Page 7: Centre for Research in English Language Learning and Assessment

Starting from the right place….1. the nature of the cognitive processes in

reading

2. the nature of the reading texts

3. the nature of the reading question formats

Page 8: Centre for Research in English Language Learning and Assessment

Starting from the WRONG place?1. the nature of the reading question

formats

2. the nature of the reading texts

3. the nature of the cognitive processes

Page 9: Centre for Research in English Language Learning and Assessment

Good reading task design1. What types of mental (or cognitive)

processing are typically involved in reading?

2. What types of text are typically involved in reading activity?

3. How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests?

Page 10: Centre for Research in English Language Learning and Assessment

CRELLA

What actually happens when we read?

A brief look at cognitive processes in reading

Page 11: Centre for Research in English Language Learning and Assessment

Remediation wherenecessary

Monitor:goal checking

Goal setterSelecting appropriate

type of reading:

Careful reading

LOCAL:Understanding sentence

GLOBAL:Comprehend main idea(s)Comprehend overall textComprehend overall texts

Expeditious reading

LOCAL:Scan/search for specifics

GLOBAL:Skim for gist

Search for main ideas andand important detail

Visual input

General knowledgeof the world

Topic knowledge

Meaning representation of text(s) so far

Syntactic knowledge

LexiconLemma:Meaning

Word class

LexiconForm:

OrthographyPhonologyMorphology

Building a mental modelIntegrating new information

Enriching the proposition

Inferencing

Establishingpropositional meaning

at clause and sentence levels

Syntactic Parsing

Lexical access

Word recognition

Text structure knowledge:

Genre

Rhetorical tasksCreating a text level representation:

Construct an organised representation of a single text

Creating an intertextual representation:

Construct an organised representation across texts

Page 12: Centre for Research in English Language Learning and Assessment

Visual input

Clearly, the starting point for any reading activity is the set of marks on a handwritten or printed page, or on a computer screen – i.e. a combination of letters, symbols, pictures, etc.

Page 13: Centre for Research in English Language Learning and Assessment

Word recognition

Word recognition involves matching the form of a word in a written text with a mental representation of the orthographic forms of the language. Field (2004:234) refers to this as “the perceptual process of identifying the letters and words in a text”.

Page 14: Centre for Research in English Language Learning and Assessment

Lexical (vocabulary) accessAccessing the lexical entry containing stored information about a word’s form and its meaning from the lexicon. The form includes orthographic and phonological mental representations of an item and possibly information on its morphology. The lemma (the meaning-related part of the lexical entry) includes information on word class and the syntactic structures in which the item can appear and on the range of possible senses for the word.

Page 15: Centre for Research in English Language Learning and Assessment

Syntactic parsing

Once the meaning of words is accessed, the reader has to group words into phrases, and into larger units at the clause and sentence level to understand the text message.

Page 16: Centre for Research in English Language Learning and Assessment

Establishing propositional meaning at the clause or sentence level

An abstract representation of a single unit of meaning: a mental record of the core meaning of the sentence without any of the interpretative and associative factors which the reader might bring to bear upon it.

Page 17: Centre for Research in English Language Learning and Assessment

Inferencing (1) Inferencing is necessary so the reader can go beyond explicitly stated ideas as the links between ideas in a passage are often left implicit.

Inferencing in this sense is a creative process whereby the brain adds information which is not stated in a text in order to impose coherence.

Page 18: Centre for Research in English Language Learning and Assessment

Inferencing (2) A text cannot include all the information that is necessary in order to make sense of it. Texts usually leave out knowledge that readers can be trusted to add for themselves.

If there was no such thing as inferencing, writing a text which includes every piece of information would be extremely cumbersome and time consuming.

Page 19: Centre for Research in English Language Learning and Assessment

Constructing a mental model Once the reader has processed the incoming sentence and elaborated it where necessary and possible through inferencing, the new information needs to be integrated into a mental representation of the text so far.

This process entails an ability to identify main ideas, to relate them to previous ideas, distinguish between major and minor propositions and to impose a hierarchical structure on the information in the text.

Page 20: Centre for Research in English Language Learning and Assessment

Creating a discourse-level structure

At a final stage of processing, a discourse-level structure is created for the text as a whole. The skilled reader is able to recognise the hierarchical structure of the whole text and determines which items of information are central to the meaning of the text. The skilled reader determines how the different parts of the text fit together and which parts of the text are important to the writer or to reader purpose.

Page 21: Centre for Research in English Language Learning and Assessment

Establishing a mental representation across texts

In the real world, the reader sometimes has to combine and collate macro-propositional information from more than one text. The need to combine rhetorical and contextual information across texts would seem to place the greatest demands on processing.

Page 22: Centre for Research in English Language Learning and Assessment

Cognitive processing at A2 to C2

KET A2 PET B1 FCE B2 CAE C1 CPE C2 Word recognition * * * * * Lexical access * * * * * Parsing * * * * * Establishing propositional meaning * * * * * Inferencing (*) * * * * Building a mental model (*) * * * * Creating a text level structure * * Creating an organised representation of several texts

*

Page 23: Centre for Research in English Language Learning and Assessment

Goal setter

The goal setter in the left hand column is critical in that the decisions taken on the purpose for the reading activity will determine the relative importance of some of the processes in the central core of the model.

Page 24: Centre for Research in English Language Learning and Assessment

Types of reading

Reading is either careful or expeditious

and comprehension takes place at the local and global level.

Page 25: Centre for Research in English Language Learning and Assessment

Local comprehension Local comprehension refers to the understanding of

propositions at the level of microstructure i.e., the sentence and the clause. Basic comprehension questions are used to assess lexical, syntactic, and semantic abilities and the ability to understand important information presented in sentence-level propositions. The information used in the question and the information required for the answer are usually in the same sentence.

Page 26: Centre for Research in English Language Learning and Assessment

Global comprehension

Global comprehension refers to the understanding of propositions beyond the level of microstructure, that is, any macro-propositions including main ideas, the links between those macro-propositions and the way in which the micro-propositions elaborate upon them.

Page 27: Centre for Research in English Language Learning and Assessment

Careful reading Careful reading is intended to extract complete

meanings from presented material. This can take place at a local or a global level, i.e. within or beyond the sentence right up to the level of the complete text or texts. The approach to reading is based on slow, careful, linear, incremental reading for comprehension.

Page 28: Centre for Research in English Language Learning and Assessment

Expeditious reading Expeditious reading involves quick, selective

and efficient reading to access desired information in a text. Expeditious reading would appear likely to include skimming, search reading, and scanning.

Page 29: Centre for Research in English Language Learning and Assessment

Skimming

Skimming is generally defined as reading to obtain the gist, general impression and/or superordinate main idea of the whole text.

Page 30: Centre for Research in English Language Learning and Assessment

Search readingSearch reading involves locating information on predetermined topics. The reader only wants information necessary to answer set questions or to provide data for example in completing written assignments. It differs from skimming in that the search for information is guided by predetermined topics so the reader does not necessarily have to establish a macro-propositional structure for the whole of the text.

Page 31: Centre for Research in English Language Learning and Assessment

Scanning

Scanning involves reading selectively, to achieve very specific reading goals, e.g. looking for specific words/phrases, figures/percentages, names, dates of particular events or specific items in an index at the local word level.

Page 32: Centre for Research in English Language Learning and Assessment

Types of reading tested at levels A2 to C2

KET

A2 PET B1

FCE B2

CAE C1

CPE C2

Careful Reading Local Understanding propositional meaning at clause and sentence level

* * *

Careful Reading Global

Comprehend across sentences * * * * * Comprehend overall text (*) * * Comprehend overall texts * Expeditious Reading Local Scanning or search reading * * * Expeditious Reading Global Skim for gist Search reading * (*) (*)

Page 33: Centre for Research in English Language Learning and Assessment

Let’s look at a reading test task

• Look at Task 1 which comes from a reading test.

• Which of the core cognitive reading processes we have discussed does this task seem to elicit?

• Which processes does this task apparently NOT elicit?

Page 34: Centre for Research in English Language Learning and Assessment

Let’s look at a reading test task

Word recognition

Lexical access

Syntactic parsing

Establishing propositional meaning at clause/sentence level

Inferencing

Building a mental model

Creating a text-level representation

Creating an intertextual representation

Page 35: Centre for Research in English Language Learning and Assessment

Let’s look at a reading test task

Word recognition Yes

Lexical access Yes

Syntactic parsing Yes

Establishing propositional meaning at clause/sentence level Yes

Inferencing ?

Building a mental model

Creating a text-level representation

Creating an intertextual representation

Page 36: Centre for Research in English Language Learning and Assessment

When we design a reading test task…

We need to be explicit about:

• the types of reading that the task requires

• the various cognitive processes we believe the reading task is eliciting

• how well matched the cognitive processes are to the level of our students

E.g. Is there a shift from tasks that focus on decoding to tasks that focus on meaning building from main ideas, to a text level representation to intertextual representation

Page 37: Centre for Research in English Language Learning and Assessment

Cognitive validity in the testing of Reading

The extent to which the tasks we employ in a reading test elicit the cognitive processing involved in target reading contexts beyond the test itself.

Page 38: Centre for Research in English Language Learning and Assessment

CRELLA

What is the nature of the texts we read?

A brief look at what makes a text difficult to read

Page 39: Centre for Research in English Language Learning and Assessment

Contextual features of texts and tasks in reading tests

This relates to the appropriateness of both :

the linguistic and content demands of the text to be processed (i.e. read and comprehended)

and

the features of the task setting that impact on task completion (e.g. responding to comprehension questions or writing a summary)

Page 40: Centre for Research in English Language Learning and Assessment

Controlling contextual features of reading texts and tasks – 3 questions

• What are the key contextual features (within-text) that we need to think about when selecting texts for reading test tasks?

• What degree of complexity should we aim for in these features across each of the proficiency levels we want to test?

• What are the methods that can help us to understand the difficulty level of a reading text?

Page 41: Centre for Research in English Language Learning and Assessment

Contextual features in readingSome illustrative features relating to: • lexical complexity (decoding)

• structural complexity (syntactic parsing)

• cohesion (the construction of meaning)

Page 42: Centre for Research in English Language Learning and Assessment

Some key lexical parameters

L1 Syllables per wordL2 Type token ratioL3 Word frequencyL4 Lexical densityL5 Proportion of academic words

Page 43: Centre for Research in English Language Learning and Assessment

L1 : Average syllables per word• the mean number of syllables per content word

multisyllabic words take longer to read and process than monosyllabic words [Rayner & Pollatsek, 1989]

“In general, the more syllables per word and the more words per sentence, the higher the associated grade level of the text”[White, S. (2011) Understanding Adult Functional Literacy: Connecting Text Features, Task Demands, and Respondent Skills. Taylor & Francis]

Page 44: Centre for Research in English Language Learning and Assessment

L2 : Type token ratio• the number of unique words divided by the number of tokens of

the words

Each unique word in a text is a word type. Each instance of a particular word is a token.

When the type: token ratio is 1, each word occurs only once in the text; comprehension should be comparatively difficult because many unique words need to be encoded and integrated with the discourse context. A low type: token ratio indicates that words are repeated many times in the text, which should generally increase the ease and speed of text processing.

[Templin, M (1957) Certain Language Skills in Children: Their development and interrelationships.

Institute of Child Welfare Monograph Series No. 26. Minneapolis, MN: University of Minnesota Press]

Page 45: Centre for Research in English Language Learning and Assessment

L3 : Word frequency

• the relative frequency of occurrence of words

Frequency effects have been shown to facilitate decoding:– frequent words are processed more quickly and understood better

than infrequent ones (Haberlandt & Graesser, 1985; Just & Carpenter, 1980).

– rapid or automatic decoding = strong predictor of L2 reading performance (Koda, 2005)

– texts which assist such decoding (e.g., by containing a greater proportion of high frequency words) are easier to process….

The more frequent a word, the more likely it is to be processed with a fair degree of automaticity, thus increasing reading speed (even among lower level learners) and freeing working memory for higher level meaning building. (Crossley, Greenfield and McNamara, 2008)

Page 46: Centre for Research in English Language Learning and Assessment

L4 : Lexical density• depends on distinguishing between different word types, i.e.

lexical (content) and function words– lexical: verbs, nouns, adjectives, adverbs– function: auxiliaries, determiners, pronouns, prepositions,

conjunctions

Accessing the meaning of lexical items requires accessing the mental lexicon, function words can be dealt with by pattern matching. Reading focuses mainly on lexical items and readers tend to skip function words.

Page 47: Centre for Research in English Language Learning and Assessment

L5 : Proportion of academic words• the incidence of academic words in a text

• proved to be a good predictor of level in a study of FCE, CAE and CPE reading texts (Weir et al, 2012)

Mean SD FCE (B2) 1.61% 1.26%

CAE (C1) 1.63% 1.41%CPE (C2) 5.82% 2.84%

Page 48: Centre for Research in English Language Learning and Assessment

Some key syntactic parameters

S1 Sentence length

S2 Readability formulae

S3 Higher level constituents

Page 49: Centre for Research in English Language Learning and Assessment

Syntactic complexity

• linear processing of text in careful reading, with the reader decoding word by word

• assembly of decoded items into larger scale syntactic structure

(Just & Carpenter, 1987; Rayner & Pollatsek, 1994)

• cognitive demands imposed vary considerably according to how complex the structure is

(Perfetti, Landi & Oakhill, 2005)

(Crossley ,Greenfield and McNamara, 2008)

Page 50: Centre for Research in English Language Learning and Assessment

S1 : Sentence length in Cambridge ESOL reading papers

Main Suite Level Average number of words per sentence

Range

KET (A2) 13.2 8 - 17PET (B1) 14.9 10 - 20FCE (B2) 18.4 11 - 25CAE (C1) 18.6 13 - 27CPE (C2) 19.6 13 - 30

Page 51: Centre for Research in English Language Learning and Assessment

S2 : Readability formulae• are long-established and widespread in use• rely heavily on word length and sentence length• ignore many language and discourse components

that are theoretically expected to influence reading and comprehension difficulty

Page 52: Centre for Research in English Language Learning and Assessment

…nevertheless…• texts with longer words and lengthier sentences are more

difficult to read– longer words tend to be less frequent in the language and

infrequent words take more time to access and interpret during reading

– longer sentences place more demands on working memory

– real-time processing means holding information in your head until you can parse sentences syntactically

– the longer the sentence, the more difficult this may be

Page 53: Centre for Research in English Language Learning and Assessment

S2 : Difficulty/readability estimates in Cambridge ESOL reading papers

Main Suite level

Flesch reading ease score

Flesch-Kincaid grade level

Flesch-Kincaid range

KET (A2) 78.3 5.5 2 – 7.4

PET (B1) 64.7 7.9 5 – 10.1

FCE (B2) 66.5 8.4 5 – 12.3

CAE (C1) 58.4 9.6 5.7 - 16

CPE (C2) 57.7 9.9 5.6 – 16.1

Page 54: Centre for Research in English Language Learning and Assessment

S3 : Higher level constituents• the number of main verbs in a sentence is broadly indicative

of the number of clauses - thus of complex syntactic composition

• the more complex the syntactic composition, the greater the load on cognitive processing

• the more clauses you have to process in a sentence, the more propositions you have to hold in working memory and link together

Page 55: Centre for Research in English Language Learning and Assessment

Cohesion (and coherence)

Cohesion is an objective property of the explicit language and text. There are explicit features, words, phrases, or sentences that guide the

reader in interpreting the substantive ideas in the text, in connecting ideas with other ideas, and in connecting ideas to higher level global units (e.g., topics and themes). These cohesive devices cue the reader on how to form a coherent representation.

The coherence relations are constructed in the mind of the reader and depend on the skills and knowledge that the reader brings to the situation…coherence is a psychological construct, whereas cohesion is a textual construct.

[Graesser et al 2004: 193]

Page 56: Centre for Research in English Language Learning and Assessment

Cohesion • two forms of textual cohesion can be estimated : • referential cohesion (the extent to which words

in the text co-refer)• conceptual cohesion (the degree of similarity

between concepts in different parts of a text)

Page 57: Centre for Research in English Language Learning and Assessment

Let’s look at a reading text

• Read the text and, as you do so, think about aspects of the text’s complexity related to :– the difficulty of the lexis/vocabulary– the difficulty of the syntax– the cohesion of the text

• Don’t worry about answering the questions over the page. We shall look at those later.

Page 58: Centre for Research in English Language Learning and Assessment

Analysing the text’s complexity

Page 59: Centre for Research in English Language Learning and Assessment

Overall Number of words

Lexis Structure

KET

Approximately 740-800words

Restricted to common items which normally occur in the everyday vocabulary of native speakers.

Mainly simple sentences

PET

Approximately 1460-1590 words

General vocabulary sufficient for most topics in everyday life.

Mostly simple sentences but some use of relative and other subordinate clauses.

FCE

Approximately 2000 words

Good range of vocabulary. Topics are addressed in detail and with precision.

A range of sentence patterns– from the simple to the complex.

CAE

Approximately 3000 words

Broad range of vocabulary including idiomatic expressions and colloquialisms as well as language relating to opinion, persuasion and ideas.

This level is typified by: many complex sentences Frequent use of modals Some use of ellipsis Complex approaches to referencing – range of pronouns and adverbials, as well as use of synonymy.

CPE

Approximately 3000 words

Very wide range of vocabulary including idiomatic expressions and colloquialisms as well as language relating to opinion, persuasion and abstract ideas.

Most sentences are long and complex. No restriction on the types of structure employed by the text. Many examples of structures typically used for effect in writing – sentences with several subordinate clauses, for example.

Page 60: Centre for Research in English Language Learning and Assessment

CRELLA

What is the nature of the reading task?

A brief look at question formats in reading tests

Page 61: Centre for Research in English Language Learning and Assessment

Response method

• Selected response• Multiple choice (MCQ)• True/false• Right/wrong doesn’t say• Multiple matching• Gapped text

• Constructed response• Short answer questions• Information transfer• Random deletion cloze• Selective deletion gap-

filling• Reading into writing

(e.g. summary)

Page 62: Centre for Research in English Language Learning and Assessment

Position of the test questions

• Before the reading text?

or

• After the reading text?

Page 63: Centre for Research in English Language Learning and Assessment

Let’s look at some test questions

• Question 1

• Question 2

• Question 3

• Question 4

Page 64: Centre for Research in English Language Learning and Assessment

Context validity in Reading

Context validity relates to the appropriateness of both the linguistic and content demands of the text to be processed, and the features of the task setting that impact on task completion.

Page 65: Centre for Research in English Language Learning and Assessment

When designing reading test tasks we need to take account of….

1. What types of mental (or cognitive) processing are typically involved in reading?

2. What types of text are typically involved in reading activity?

3. How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests?

Page 66: Centre for Research in English Language Learning and Assessment

Спасибо за внимание!