Using assessment to improve the teaching of

reading

18 September 2013David Carroll and Keith Prenton

• The Basic Education Program (BEP) is a five-year initiative designed to benefit all public primary and lower secondary schools in Kosovo, (serving grades 1 – 9). It began on 1 October 2010 and is scheduled to end on 30 September 2015.

• BEP’s motto is “Developing students’ 21st Century skills with schools and communities”. To this end, BEP seeks to improve the capacity of Kosovo’s schools to provide relevant skills for its students.

• BEP’s overarching goal is to improve the Government of Kosovo’s institutional capacity in the education sector and improve the quality of primary education.

• BEP is funded by USAID and the Government of Kosovo. FHI 360 is implementing BEP on behalf of USAID in partnership with the Ministry of Education, Science and Technology and Kosovo Education Center (KEC) which manages Component 3 of the Program.

•

USAID/GoK Basic Education Program

CLICK TO ADD TITLE

1. Inviting all Kosovo’s primary school teachers to complete a three day, “Assessment for Learning” (AfL) course, which aims to increase the quality of teaching and learning process and enhance the progress of individual students. To date, over 7500 teachers have attended the course.

2. Assisting the MEST Assessment Division to improve the quality of test items, using PISA models.

3. Developing an Early Grades Reading Assessment (EGRA) in Albanian language; conducting a baseline study of 800 students using this EGRA; and, based on the findings ,developing a training course and supplementary reading materials for Grade 1 – 5 teachers.

Using Assessment to Improve Student Competencies

Deconstructing the PISA Reading Literacy Framework

Getting from where we are to where we want to be

Predicting Kosovo’s 2015 Performance

Country 2009 Reading Mean Score

Albania 385Montenegro 408Romania 424Bulgaria 429Serbia 442Croatia 476

The PISA Reading Literacy Framework

• Texts: medium, environment, format, type• Aspects: access and retrieve; integrate and interpret;

reflect and evaluate• Situations: personal, public, educational, occupational

PISA Proficiency Levels

Six levels are defined:• Average percent of students able to perform at the level

or above is given• Characteristics of tasks at each level• Examples of released questions at the level, with their

scaled difficulty.

PISA 2009 – Cumulative % of test takers at or above each level

  Level 1a Level 2 Level 3Albania 69.9 43.3 17.7Montenegro 78.2 50.4 22.4Romania 83.2 59.6 28.0Bulgaria 79.1 59.0 35.6Serbia 89.3 67.2 34.0Croatia 94.1 77.6 50.2OECD avg. 94.3 81.2 57.2

Level 3 – the Median OECD Test Taker

Access and retrieve Integrate and interpret Reflect and evaluateLocate several pieces of information, each of which may need to meet multiple criteria. Combine pieces of information within a text. Deal with competing information.

Integrate several parts of a text in order to identify the main idea, understand a relationship or construe the meaning of a word or phrase. Compare, contrast or categorise taking many criteria into account. Deal with competing information.

Make connections or comparisons, give explanations, or evaluate a feature of a text. Demonstrate a detailed understanding of the text in relation to familiar, everyday knowledge, or draw on less common knowledge.

Comparing Level 2 and Level 3

Level 3 Level 2 Locate several pieces of information, each of

which may need to meet multiple criteria. Locate one or more pieces of information,

each of which may need to meet multiple criteria.

Combine pieces of information within a text.   Identify the main idea, understand a

relationship or construe the meaning of a word or phrase based on integration of parts of a text.

Identify the main idea, understand a relationship, or construe the meaning of a word or phrase within a limited part of the text when the information is not prominent and low-level inferences are required.

Compare, contrast or categorise taking many criteria into account.

Form or apply simple categories

Make connections or comparisons, give explanations, or evaluate a feature of a text.

Make connections or comparisons between the text and outside knowledge

Demonstrate detailed understanding of the text in relation to familiar, everyday knowledge

Explain a feature of the text by drawing on personal experience or attitudes.

Draw on less common knowledge.   Deal with competing information. Deal with some competing information

What to do?

Two broad options:

• Focus on training grade 8 and 9 students in the required competences.

• Design teaching of reading to develop the required competences from early grades on.  

These are not mutually exclusive, of course.

Deconstructing the PISA Competences

• PISA describes exit competences. • The national education system to determines how the

competences will be achieved. • The BEP aimed to develop a sequence of competences

across grades, based on the PISA proficiency level descriptors.

• We used the sequence of complexity across the levels to guide us in envisaging what level 3 might look like at grade 5 and at grades 2/3.

The “PISA-based” Model of Reading Comprehension at Early Grades

• Locate one or more independent pieces of information, where the information is explicitly stated, each piece of information meets a single criterion and there is little or no competing information.

• Locating the information may or may not require making a synonymous match; and the target information may or may not be prominent in the text.

• Recognise the main theme or author’s purpose in a text where the topic of the text is familiar and the required information is prominent in the text.

• Make very simple inferences• Make a simple connection between information in the text and

common, everyday knowledge.• Relate a simple feature of the text to their personal experience or

attitudes.

Current Practice

Ishte njëherë një djalosh i quajtur Pikalosh, që jetonte me babain, nënën dhe qenin e tij Bupin. Ata jetonin në një kasolle afër pyllit. Kur po darkonin pranë oxhakut, jashtë dëgjuan të lehurat e Bupit. Pikaloshi dhe babai i tij dolën dhe panë Bupin duke u përleshur me një ujk. Pikaloshi zuri të qante. Por, kur pa ujkun që po ia mbathte, kuptoi që Bupi ishte heroi i tij i vërtetë.

1) Me kë jetonte Pikaloshi? (Me babain, nënën dhe qenin e tij, Bupin).

1) Çfarë dëgjoi kur po darkonte Pikaloshi?

(Ai dëgjoi të lehurat e Bupit).

1) Çfarë panë Pikaloshi dhe babai i tij?

(Ata panë Bupin duke u përleshur me një ujk).

1) Ç’kuptoi Pikaloshi? (Ai kuptoi që Bupi ishte heroi i tij i vërtetë).

A “PISA-based” model

Questions might have the following features:• Retrieval of explicitly-stated information might involve

some linguistic transformation or retrieval of less obviously salient information

• There would be questions requiring simple inferences• There might be a question about the author’s purpose or

the main theme of the text• There would be one or more questions requiring the

reader to make a connection with their own knowledge of the world or attitudes.

What to do?

• Differences in PISA performance between different nations reflect broad characteristics of student learning at age 15. These characteristics matter.

• The PISA frameworks are the state of the art in specifying exit competences. But they define goals, not how to achieve them.

• If PISA performance is to change significantly, the place to start is not grade 8. It is grade 1. You need a thorough review of objectives and instruction, and it needs to be guided by the exit competences, as defined by PISA