5/05/2015

1

New Zealand approaches to assessment that contribute to improved teaching and learning

Assoc. Prof. Gavin T L BrownThe University of Auckland

Email: gt.brown@auckland.ac.nzPresentation at East China Normal University, 5 May 2015

New Zealand Schooling StructureNominal Age Year Level Curriculum Level School Type Qualifications

17 13 8

Secondary

NCEA L316 12 7 NCEA L215 11 6 NCEA L114 10 513 912 8 4

primary

11 710 6 39 58 4 27 36 2 15 14 K

pre-school321

5/05/2015

2

NZ Assessment Tool Box

Voluntary, School-basedCurriculum aligned

High-stakes Student Certification

SEA/AKA(School Entry Assessment)• Introduced 1997 (& evaluated by ACER 2001)• Standardised assessment procedure for

schools in English & Maori• Determines the nature and extent of certain

knowledge and skills shown by new entrants• Three components

– Concepts about print (CAP) literacy– Checkout numeracy– Tell Me oral language

• Teacher rated against rubrics

5/05/2015

3

Six Year Net

• Diagnostic survey to check progress in reading and writing

• Usually administered around sixth birthday. • Includes Concept About Print (developed by

Dame Marie Clay for this survey)• Identifies children suitable for Reading

Recovery• Reports in stanines

Exemplarshttp://toolselector.tki.org.nz/Assessment-areas

• Assessed and graded samples of authentic student work

• English and Maori medium, levels 1-5• Annotations on samples• Scoring rubrics provided• Clear links to NZ Curriculum objectives• Support for teachers to inform judgments about

student achievement against objectives

5/05/2015

4

NZCER tests (http://www.nzcer.org.nz)

PAT(Progressive Achievement Tests)

• Reading Comprehension and Vocabulary; Listening Comprehension; & Mathematics

• Ages 7-14• Group administration, 30-55 minutes• All multiple choice items• Parallel forms, A and B for alternate years• Rank Order information only

– Norms as percentiles and stanines– But derived with item response theory scoring

5/05/2015

5

National Education Monitoring Project (http://nemp.otago.ac.nz/)

•1995-2010

•Identifies and reports on trends in educational performance

•Assesses knowledge, skills, motivation and attitudes BUT not tied to curriculum

•Four year cycle of learning areas and skills—4 content per year

•Two levels, four years apart (Y4 and Y8)

•Meaningful tasks to capture rich picture•Involves teachers as assessors•Four approachesOne to oneGroupPencil and paperStations

•About 2/3 of tasks released after each cycle•Light sample ~3000 students in 260 schools each cycle

National Monitoring Study of Student Achievement (NMSSA)http://nmssa.otago.ac.nz/

• Similar to NEMP– national sample Year 4 and Year 8; – broad range of skills, knowledge, key competencies,

as well as literacy and numeracy; – variety of assessment tasks; – focus on change over time; – provision of trustworthy information for

improvement. • IRT model of measurement applied to provide a

common scale for Year 4 and 8 data, • effect sizes used to interpret differences between

groups • Approximately 2000 students from 100 schools studied

at each of the two year levels

5/05/2015

6

Assessment Resource Bank(ARB)• http://arb.nzcer.org.nz/• Assessment resources for teachers• English (levels 2-5), Mathematics (2-5),

Science (2-6) only• Broad range of assessment tasks from

– simple objectives, to– extended responses in problem solving

• Includes tasks from other sources (e.g. TIMSS)

• Mixture of closed and open-ended items

Sample ARB item in Reading

This comprehension task assesses student ability to evaluate the ideas and information in a transactional text about an environmental issue. Students are asked to read a text, then respond to four questions.Keywords (comprehension; evaluating; transactional; School Journal; persuasion;)Publication date (02/10/2006)Level 4, Making Meaning

5/05/2015

7

Why so much testing but so little threat?

It’s information for improvement before evaluation of students or schools

Formative assessment: Government policy• Assessment for the purpose of improving student

learning is best understood as an ongoing process that arises out of the interaction between teaching and learning. It involves the focused and timely gathering, analysis, interpretation, and use of information that can provide evidence of student progress. Much of this evidence is ‘of the moment’. Analysis and interpretation often take place in the mind of the teacher, who then uses the insights gained to shape their actions as they continue to work with their students. • Ministry of Education. (2007). The New Zealand Curriculum for

English-medium teaching and learning in years 1-13. Wellington, NZ: Learning Media. (p. 39)

5/05/2015

8

Essential facets of AfL• Formative

–Before the end–Leads to changes in teaching &

learning practices–Results in progress–Evidence must be capable of

accurately and appropriately guiding changed practices

–Low threat to integrity of scores—safe to have low scores

Chinese approach to assessment

• Public examinations (zhong kao, gao kao)– Expected– Social rewards for success– Not avoiding the challenge

• But exams have limited information– Rank order– Total score

• So is this good enough? Probably not…• But should exams & tests be abandoned?

5/05/2015

9

Does Testing Improve?• John Hattie’s (2009) meta-analysis

• Frequent testing=.34; • Test-taking coaching=.22Hattie, J. (2009). Visible learning: A synthesis of meta-analyses in education. London: Routledge.

• These are small effects• But Chinese teachers believe in this anyway

– Testing results in improvement as a person as well as learning

–考考考，老师的法宝；分分分， 学生的命根– [exam, exam, exam, teacher’s magic weapon, grade

grade grade, students’ lifeblood]

Effective strategies for learning

• Effective teaching practices that MIGHT look like assessment? (Hattie, 2009)

• Teachers doing formative evaluation of data & evidence based models (not teacher judgements) with data graphing d=.90

• feedback=.73; • spaced practice=.71; • meta-cognitive strategies=.69; • self-questioning=.64 • mastery learning=.58; • goals=.56; • Peer tutoring=.55;

5/05/2015

10

Assessment for Learning• Emphasise the importance of the teacher as the

expert in content and pedagogy and pedagogical content knowledge whose job is to enable progress among novices

• Emphasise pedagogical practices that are known to effect learning

• Systematic, formal use of high-quality, highly-informative tests (e.g., asTTle), but not too frequently– To inform teaching– To calibrate judgements– As part of a ‘toolbox’ of validated methods/tools– Make sure tests & exams tell teachers “who

needs to be taught what next”

Source: Hattie & Brown (2008, 2010)

Formative Assessment through Testing• Carless (2011): Formative Use of Summative

Testing• Tests are inevitable but inadequate• What we need is diagnostic analysis of results

and alignment of tests to curriculum and student abilities

• New Zealand’s solution from 2000 is Assessment Tools for Teaching & Learning (asTTle & e-asTTle)

5/05/2015

11

• Developed since 2000; now in 7th version• A software system that allows

– creation by schools & teachers of standardised tests in Reading, Mathematics, & Writing drawn from a bank of calibrated test items AND

– Instant graphical analysis of student performance relative to the various informational needs of educators

– Reports designed to help teachers meet their educational goals and to ensure accuracy of comprehension

New Zealand Assessment Tools for Teaching & Learning (asTTle) system

Theoretical Considerations of Test Scores– Observed—What you actually get on a test – True—What you should get if test were

perfect, bearing in mind test is a sample of domain

– Ability—What you really are able to do or know of a domain independent of what’s in any one test. And this is what we want to know.

Ability

True Score Range

Hard to increase

Easy to increase

5/05/2015

12

How answers relate to Teaching

Answer StatusItem difficulty relative to ability Correct Wrong

Hard Unexpected! Celebrate.

Expected. Teach these things

My ability estimate on the test

Easy

Expected. Mastered. Stop

teaching

Unexpected but not serious! Don't skill & drill. These are easy

to fix.

Content Areas (Objectives)

MCQ Short Answer

Completion Total Items

1. Basic Operations 10 102. Order of Operations 5 5 103. Decimal Operations 5 5 5 154. Fraction operations 5 5 5 15

How items relate to curriculumAll questions relate to important content in a curriculum we are teaching. No test can cover everything but every item relates to something. We need at least 5 items to estimate proficiency in a content area. Different styles of items help validity of analysis. A blueprint is needed.

5/05/2015

13

But items are not equally difficult

• Assumptions: Each item is equally difficult and has equal weight towards total score. Total score based on sum of items correct is a good estimate of true ability

• Inference: these students are equally able.• BUT are they equally difficult items?

Item 1 2 3 4 5 6 7 8 9 10

A 60

B 60

C 60

% correct

Weight the score according to item difficulty

Item 1 2 3 4 5 6 7 8 9 10Difficulty -3 -2 -1 -1 0 0 1 1 2 3

A 60 530

B 60 545

C 60 593

% correct asTTle v4

Conclusions: C > A and B; B ≈ A C answered all the hardest items correctly—no penalty for skipping or getting easy items wrong

BUT to do this is difficult—need computer and calibration of items

HENCE, custom software for TEACHERS

5/05/2015

14

Key Characteristics of asTTle

• Curriculum• Choice• Calibration• CommunicationNOT• Central Control or

Reporting• Compulsory

Read more:Archer, E., & Brown, G. T. L. (2013). Beyond rhetoric: Leveraging learning from New Zealand’s Assessment Tools for Teaching and Learning for South Africa. Education as Change: Journal of Curriculum Research, 17(1), 131-147. doi:10.1080/16823206.2013.773932

Brown, G. T. L. (2013). asTTle—A national testing system for formative assessment: How the national testing policy ended up helping schools and teachers. In M. Lai & S. Kushner (Eds.). A National Developmental and Negotiated Approach to School and Curriculum Evaluation (pp. 39-56). London: Emerald Group Publishing. doi: 10.1108/S1474-7863(2013)0000014003

e-asTTle demonstration

• Test: Reading Yr8 Term3 07A– End of primary– Levels 3-4 to check completion of expected

curriculum• View Reports

– Console—how did we do compared to others (the nation or schools like mine)?

– Individual Learning Pathway—what are S&W of a student? (Shanice Khoo)

– Group Learning pathway—what are S&W of a group?– What Next?—resources to teach new stuff.

5/05/2015

15

e-asTTle teacher login

Reports: Telling the teacher what she needs to know• How is student x doing? Strengths &

Weaknesses• How are my students doing compared to

others? (national & similar schools)• What are the teaching needs of my

class/group?• Which students should I group together?

5/05/2015

16

5/05/2015

17

5/05/2015

18

5/05/2015

19

e-asTTle demonstration

• Create test– Adaptive test– 20 minutes– High School—Level 5 & 6 YES– Most Algebra– Some Number Sense & Operations– Attitude=Motivation Mathematics

– Inspect, accept or change

5/05/2015

20

5/05/2015

21

5/05/2015

22

5/05/2015

23

e-asTTle

• A tool to help teachers and school leaders• Used by the school for the school’s priorities• Professional capacity required to link testing to

curriculum and teaching• Helps improve teaching AND learning

• But it’s not the only approach to evaluating students

5/05/2015

24

• In-house evaluations– Refined design of interfaces to ensure accurate

understanding • Especially the box-and-whisker plots

– Accurate interpretations predicted by professional development AND belief that assessment is for improved teaching and learning

Effects of asTTle system on educational practice

Hattie, J. A., Brown, G. T. L., Ward, L., Irving, S. E., & Keegan, P. J. (2006). Formative evaluation of an educational assessment technology innovation: Developers’ insights into Assessment Tools for Teaching and Learning (asTTle). Journal of Multi-Disciplinary Evaluation, 5(3), 1-54.

• Useful in school-wide data analysis for improvement– some teachers are making use of tests for

improvement & accountability– Provides nationally referenced information for school

leaders PLUS gives the teacher information about MY students

– Brown, G. T. L., & Harris, L. R. (2009). Unintended consequences of using tests to improve learning: How improvement-oriented resources engender heightened conceptions of assessment as school accountability. Journal of MultiDisciplinary Evaluation, 6(12), 68-91. URL: https://tinyurl.com/o82pkv4

independent, post-release evaluations

5/05/2015

25

• Archer 2007 visit from South Africa Positive impacts on

teachers who know better who needed to be taught what students who knew they were learning from feedback and

from teacher grouping and instruction

– Archer, E., & Brown, G. T. L. (2013). Beyond rhetoric: Leveraging learning from New Zealand’s Assessment Tools for Teaching and Learning for South Africa. Education as Change: Journal of Curriculum Research, 17(1), 131-147. doi: 10.1080/16823206.2013.773932

independent, post-release evaluations

• High School self-report—personal communication from Principal and Head of Mathematics– Below average Grade 9 on entry become above

average grade 11 through focused teaching to identified cohort needs

independent, post-release evaluations

5/05/2015

26

• Professional development literacy development studies– Students gains in writing (d=1.28) and reading

(d=.48). – use of the asTTle system led to improved

pedagogical content knowledge– teachers improved in their ability in this project to

use data from test reports to make data-referenced interpretations and recommendations.

• Parr, J. M., & Timperley, H. (2008). Teachers, schools and using evidence: Considerations of preparedness. Assessment in Education: Policy, Principles and Practice, 15(1), 57-71.

independent, post-release evaluations

But it’s not just tests

5/05/2015

27

NZ National Standards

• The National Standards set clear expectations for students in reading, writing and mathematics in the first eight years at school.– 2010 came into effect

• Schools report to parents at least twice a year in writing about their child's progress and achievement in relation to the National Standards.

– 2012 school annual reports how progress to targets

National Standards NZ: Mathematics example

5/05/2015

28

National Standards as Judged by Teachers• Overall Teacher Judgements (OTJ)

– use test tools within schools to set targets and report progress relative to national standards

Judging National Standards: Kiwi style

On the balance of evidence does the student perform:• Above: looks like what is expected at the next year

level• At: looks like this year’s expected level• Below: looks like last year’s expected level• Well Below: looks like two or more years’ ago

levelStudents who are well below the expected standard may require modified classroom programmes and extra support to continue their development and maintain positive attitudes.<http://assessment.tki.org.nz/Overall-teacher-judgment/Definitions-of-achievement>

5/05/2015

29

Tools to aid JudgementsPAT: Maths and the National Standards -

probabilities of 'At' judgmentE-asTTle mathematics and the National Standards -

probabilities of 'At' judgment

To conclude

• Tests can be formative but – assessment developers have to be a lot more

effective in giving teachers information that teachers need not information that psychometricians need

– Policy has to give an incentive to teachers and students to discover what they haven’t taught well or which hasn’t been learned well

• Truth has to be more important than inflated scores

• THIS SEEMS TO BE WHAT TEACHERS WORLD WIDE WANT—LET’S GIVE IT TO THEM

5/05/2015

30

The NZ way

• High trust in professionalism of teachers– But validate through multiple methods

• Low consequences for low performance– The focus is on improvement

• Low dependence on testing as the sole source of valid information– Teachers know what students can do

• Better testing– Aligned to curriculum and aimed at supporting

teaching