naplan data analysis school leadership module - ais...

NAPLAN Data Analysis

School Leadership Module

August 2015

NAPLAN 2015 Analysis – School Leadership Module 1

NAPLAN Analysis: School Leadership Module

PURPOSE

This module is designed for Principals and school Leadership Teams to lead the school-level analysis of NAPLAN data. It is intended to help school leaders to:

understand the state/territory, Archdiocesan and school results

confidently lead the analysis of data on the cohort and skills development at the whole-school level

develop strategies to promote staff engagement with the data throughout the school, not just the testing-year teachers

identify issues for further exploration

contribute to the development of the school’s Annual Improvement and Strategic plans, and

inform the school’s planning for ongoing Internal School Review

The Teacher Module concentrates on Value-added Student Growth, Item Analysis, Teaching Strategies, and the identification of individual students needing support. This is centred on supporting the teacher in the classroom to align testing results with pedagogy in a diagnostic sense.

CONTEXT

ACCESSING THE SMART 2015 WEB-BASED APPLICATION

The SMART application is accessed via the web and requires authentication via a username and password. Users then agree to the terms and conditions of use according to the relevant Education Act. In schools, Principals and one delegate have the authority and responsibility, as Executive users, to provision staff to access and use the application.

It is expected that all teaching staff are given access to the SMART web application to promote a school-wide understanding of the data, its implications for teaching and learning, and development of priorities for using the results for literacy and numeracy planning.

In the school context, the main areas of analysis involve NAPLAN data at the state/territory, Archdiocesan, school, classroom and student levels. Specifically, the strategies relate to the following areas:

diagnosing school results and making comparisons with Archdiocesan and state/territory information

analysing skill/item strengths and weaknesses

identifying trends related to school-level literacy/numeracy skills and particular cohorts in the school

developing school-level priorities for literacy & numeracy

formulating intervention plans for cohorts and individual students

determining appropriate ways for teachers to integrate SMART 2011’s Teaching Strategies into classroom curriculum delivery and assessment.

When considering the results, the three fundamental questions to ask are:

How are we going?

How do we know?

What are we going to do about it?

These questions form the basis for leading the analysis and use of the data at the school level.

National

State/Territory

Archdiocese

Cohort Skills

Cohort

Skills

Student

Whole School


ANALYSING THE DATA

For school leaders, the focus is on the school results, identification of issues from the data, and strategies for effective leadership in analysis and use of the information across the whole school. This involves three types of analyses:

point-in-time analysis (to identify concerns for this testing period),

trends over time (to identify longer term issues), and

value-added growth of cohorts and individual students (from one testing period to the next).

PROCESS In analysing NAPLAN results at the school level, the following process outlines the phases undertaken to firstly identify the specific issues from the data, then to develop a literacy/numeracy plan and implement the appropriate and targeted strategies. The final phase involves crucial reflection on the measurable changes in student achievement and, importantly, the impact of these on teaching practices within the school.

This process is important in developing, implementing and evaluating the school’s literacy/numeracy plan. A key element is to lead the integration of such information across curriculum areas and classroom-based pedagogy.

It is strongly recommended that the following process be undertaken:

Initial analyses undertaken before the end of Term 3 by the school Leadership Team using the School Leadership Module to ascertain the main points of interest, and possible focus areas to be included in the Annual Improvement Plan.

Members of the Leadership Team then lead the subsequent analysis early in Term 4 with groups of teachers using the Teachers Module as a guide. This promotes staff engagement with the process and helps to promote the essential linkages between NAPLAN results and classroom curriculum delivery and pedagogy related to the skills of literacy and numeracy.

The staff, as a whole, agree on the main issues and challenges presented by the results. This helps to promote the likelihood of a whole-school understanding of the results and coordinated response to any issues ascertained.

Term 3 (Aug/ Sept)

Analyse testing results

Identify issues for follow-up

Term 4

Formulate whole-school response involving critical mass of staff

Develop specific strategies to implement Literacy/Numeracy Plan

Relate action plan to Internal School Review process

Include in the Annual School Improvement Plan

Determine adjustments to pedagogy to meet learning needs of students

Next year

Implement Literacy/Literacy Plan

Explore links between NAPLAN and classroom pedagogy

Integrate SMART2 teaching strategies with curriculum delivery

Monitor and evaluate improvements in student achievement: evidence of change

Critique impact on teaching practices

Plan

Act

Reflect

Identify


It is important that the school’s Leadership Team:

- is familiar with, and competent in, analysing and understanding the NAPLAN data; and

- takes an active role in leading the analysis and use of the data with the staff to effect measurable improvements in student learning.

In the following analysis:

the text in RED indicates the questions to ask of the data

the text in BLUE is the actual worksheet to record your answers

The final section draws together the individual analyses to arrive at a picture of the cohort at a point in time and over time.


SMART 2015 Application

Landing Screen

When you open the application this screen shows the main features of the SMART 2015 application. The key elements are:

1. Scope of Analysis (Region, School or Student) and available Reports

2. Test, year, cohort group and test Domain

3. Subgroups (by gender, school/system-created groups, etc)

4. Data analysis tools and displays

5. Summary of the key features from the NAPLAN data

6. Key data in relation to the National Minimum Standards.

‘% below NMS’ is the bottom band ‘% at or below NMS’ is the bottom two bands ‘% at proficiency’ is the top two bands

7. Groups of schools (at least 5 schools containing a minimum of 30 students) and students within schools can be created for closer comparison

8. elearning Modules: self-paced tutorials for navigating SMART2 and analyzing the data.

1

2

3

4 5 6

7

8

This is a screen shot for System analysis.

The school’s landing page and subsequent screens are similar,

and are GREEN.


The questions to ask are: WHOLE COHORT:

1. In which Domains and cohorts are the school’s Mean scores ABOVE or BELOW those for the state/territory?

2. Where are the greatest differences?

GENDER DIFFERENCES: 1. For each Year and Cohort, are there any gender differences in the Mean Scores between

the school and state/territory? This could have implications for pedagogy in the school.

2. For each Domain, where are the greatest gender differences between the school and state/territory?

COHORT ANALYSIS

Select ‘Reports’ from the menu, then ‘Means and Standard Deviations’

The following example explains how to calculate the significance of these differences.

Means and Standard Deviations


To understand a school’s performance in NAPLAN it is important to calculate the ‘practical significance’ of the difference between its Mean scores on each Domain and those of the state/territory. In reality, it is a simple process, but is shrouded in perceptions of difficulty.

The process involves finding how far away the school’s Mean is from the state/territory Mean. That is, we compare the difference in Mean scores with the Standard Deviation (SD) or spread of scores across the state/territory. There are two stages in this:

A. Calculate: 1. Mean difference between the school and state/territory 2. 0.2 critical value of the state/territory Standard Deviation

In this example, for Writing:

Whole Cohort:

the school is 10 Mean scores below the State mean (429 – 419); this is inside the 0.2 Standard Deviation threshold (= ±13). Consequently, the school’s achievement is ‘around’ the State result.

Gender Differences:

the Boys Mean is 22 scores below the state Mean (417-395) -- outside the Standard Deviation threshold of ±14. However, Girls are around the state Mean (+2 difference) – inside the Standard Deviation threshold of ±13.

This means there is a gender difference between boys’ and girls’ results in Writing, with boys performing relatively worse than girls. This may have implications for classroom pedagogy when the particular skills are analysed in the ‘Item Analysis’ section of SMART,

B. Compare Mean difference with Standard Deviation critical values according to the key:

more than +0.2 SD above state/territory Mean Above state/territory Mean: keep working on strengths

between -0.2 SD and +0.2 SD from state/territory Mean Around state/territory Mean: cannot infer anything

more than -0.2 SD below state/territory Mean Below state/territory Mean: worth further investigation

Difference = -10

0.2 = ±13

Difference = -22

0.2 = ±14

Difference = +2

0.2 = ±13


In the Table below, record the differences in Mean scores for each of the groups, and note those that fall OUTSIDE the Standard Deviation threshold. This will indicate areas of success and requirements for possible intervention and further investigation.

NAPLAN 2015: School Means ABOVE and BELOW State/Territory Means

Cohort Boys Girls

Reading

Y3

Y5

Y7

Y9

Writing

Y3

Y5

Y7

Y9

Spelling

Y3

Y5

Y7

Y9

Grammar & Punctuation

Y3

Y5

Y7

Y9

Numeracy

Y3

Y5

Y7

Y9


The questions to ask are:

1. Is there any under-representation in the top band in any Domain or cohort? Also, are there any areas of persistent under-representation across the three years? This could have implications for the engagement of the top students in each cohort.

2. Is the school over-represented in the middle bands at the expense of the top band?

3. How does the proportion of students below the National Minimum Standard (NMS) - the bottom band - compare with the state/territory?

Select ‘Percentage in Bands’ from the menu

This will provide a time series picture of the proportion of students in each Band.

For each Year, Domain and Cohort, indicate if the school is over-represented (+) or under-represented (-) in each of the following.

PERCENTAGE OF STUDENTS IN BANDS: Comparison with State/Territory

2013 2014 2015

Below NMS

Middle Bands

Top Band

Below NMS

Middle Bands

Top Band

Below NMS

Middle Bands

Top Band

Reading

Y3

Y5

Y7

Y9

Writing

Y3

Y5

Y7

Y9

Spelling

Y3

Y5

Y7

Y9

Percentage in Bands

1.

2.

3.


PERCENTAGE OF STUDENTS IN BANDS: Comparison with State/Territory

2013 2014 2015

Below NMS

Middle Bands

Top Band

Below NMS

Middle Bands

Top Band

Below NMS

Middle Bands

Top Band


Y3

Y5

Y7

Y9

Numeracy

Y3

Y5

Y7

Y9

As well as the quantum of growth for the school, the distribution of growth within the cohort can provide important information. Growth in student scores for Reading, Writing and Numeracy can be viewed by comparing test scores this year with those of the same students two years ago (e.g. Year 5 students this year compared with their Year 3 results).

The example shows that, even though the ‘Average Scaled Score Growth’ of the school (52.14) is higher than that of the state (43.06), the growth of more than half of the students (55.2%) was less than ‘expected’. This is due to the distribution of growth across the cohort.

Distribution of Student Growth

What is shown by the two subgroups

of students?

What are the implications for the classroom?


The questions to ask are:

1. Is there a discernable pattern of cohort subgroups shown by the Student Growth graphs?

2. Is there a pattern between different years, e.g. 2012-2014, 2013-2015?

3. If the filter is set according to gender, is there a discernable pattern of subgroup differences for boys and girls?

The growth shows two distinct subgroups of students: the growth of those at the bottom end (orange arrows) is far greater than that of the top students (blue arrows), most of whom had little growth. Here, students have either remained stationary since the previous NAPLAN test, or have actually suffered a decline during this period. This result is worrying since it may indicate a degree of disengagement of these students with learning or classroom pedagogy that may target one subgroup of students at the expense of another. It may highlight a lack of challenge or extension work for these students in the particular skills associated with the test Domain. The result may also consistent with under-representation of students in the top achievement band. This pattern suggests that further exploration may be needed at the school level to see if it is replicated in the other literacy and numeracy results from NAPLAN. If so, it may indicate the need for a reassessment of classroom pedagogy to determine how these skills are covered across the Key Learning Areas, and to see if one subgroup of students is being disadvantaged compared with another.

For each Year Range, Domain and Cohort, indicate if there are differences between:

Top end / bottom end for the Cohort

Top end / bottom end for boys and girls

STUDENT GROWTH – Distribution between Subgroups

2012 - 2014 2013 - 2015

Cohort Boys / Girls Cohort Boys / Girls

Reading

Y3 Y5

Y5 Y7

Y7 Y9

Writing

Y3 Y5

Y5 Y7

Y7 Y9

Spelling

Y3 Y5

Y5 Y7

Y7 Y9



Y3 Y5

Y5 Y7

Y7 Y9

Numeracy

Y3 Y5

Y5 Y7

Y7 Y9


PUTTING THE INFORMATION TOGETHER

The analysis of NAPLAN results for the school Leadership Team raises the following questions for further investigation:

Mean Scores

1. a. For this year, how do the school’s Mean scores for each cohort differ from those of the state/territory?

b. What has been the pattern over the last three years?

2. a. What GENDER DIFFERENCES exist this year between the school and state/territory?


Percentage in Bands

3. a. In which Domains and cohorts is the school under-represented and over-represented this year?



School Cohort Growth

4. How has the school’s growth for each cohort compared with the state/territory between 2012-2014 and 2013-2015? Are any issues indicated by this?

Distribution of Student Growth

5. Is there any discernable pattern of the distribution of student growth between 2012-2014 and 2013-2015 in relation to two sets of subgroups?

Top and Bottom students

Boys and Girls


Synthesis

1. What school-level focus areas from the literacy/numeracy Domains can be identified for further action and professional learning of staff?

2. How can these focus areas be integrated into the school’s Internal School Review process and Annual Improvement Plans?

3. What is the most effective way to lead the response to these results across the whole school?

Achievement Growth


4. How is staff engagement with the NAPLAN data to be facilitated across the school? What strategies and resources are appropriate for such intervention (e.g. time, group/stage/faculty meetings, role of school Leadership Team, etc)?


Linking Data with Pedagogy (SMART eLearning Module)

This model is a feature of the SMART application as a strategy for linking the data from NAPLAN testing with whole-school planning and classroom-based pedagogy, assessment and reporting. It is suggested as a structure for, and approach to, leading learning throughout the school based on information from NAPLAN testing. Each element of the model is explained in the eLearning Module of the SMART2 application.


Using Data to Inform Learning Information on student achievement helps to inform decisions about learning and teaching. Used wisely, data is diagnostic; it suggests questions to be asked and informs decisions that ultimately affect student learning and teacher pedagogy. Data forms the basis for information about assessment for learning, assessment of learning and assessment for teaching. The following principles should be applied to use data wisely for the cohort, the class and the student.

1. Engage with the data

If learning is everyone’s business, so is data analysis. Staff engagement with data on student achievement is crucial for developing a whole-school understanding of, and response to, student performance. To promote this engagement with the data and with the process of analysis, it is essential to involve a critical mass of teachers in analysing data on student achievement, and then share this with all teachers. The conversations are as important as the analysis!

2. Ask questions of the data

Questions are starting points for future action; judgements are stopping points. Use the data to ask ‘How are we going?’ ‘How do we know?’ ‘What are we going to do about it?’ The third question forms the plan for action: if we collect and analyse data on achievement, we are obliged to use this information to improve student outcomes. Any judgements should be as a result of questions arising from the data analysis.

3. Investigate the important issues

Any analysis of data on student achievement should be a planned process, not the result of ‘busy work’. Be strategic in what to analyse and how to analyse it.

With NAPLAN the key areas are:

Status: achievement at a point in time (means, percentage in bands, item analysis).

Gain: the difference for the student between this test and the previous score on the same skill. This measures value added of matched students, and can raise questions about classroom pedagogy. Also, determine if student gain is different for different subgroups (top/bottom; boys/girls, etc).

Improvement: how the school is going on any indicator compared with previous years. How does this year’s Year 5 compare with last year’s Year 5 (different cohort)? Is the pattern of results similar? If so, it could indicate issues with sequencing of learning or pedagogy across particular skill areas.

With semester reporting the key issues involve analysing any marked differences in grade distributions between:

Semester 1 and Semester 2

different Learning Areas within the same cohort

different cohorts in the same Learning Area

Maths and Science compared with Religious Education, English and Social Science/HSIE

4. Drill down

Start looking at an overview of the cohort. Then look at the class, then the student. How are the student results distributed? What are the patterns? What are the questions arising from the analysis? Is student performance consistent across multiple data sources?

5. Identify students

Put a face on the data. Having a clear idea of individual students and their results is the aim. Are there patterns across different students or student groups that produce good questions for analysis? Are there any surprises? How do these results compare with other sources of student achievement data for the same students?

6. Use questions AND evidence to inform decisions

This is the point of the exercise. Analysing and reporting student achievement is meaningless unless there is a diagnostic purpose for doing so. This involves asking ‘What are our students’ learning needs?’ and, therefore, ‘What are our learning needs?’. Until we get relevant answers to these questions, we do not have a way forward. When we do, we frame a course of action that plans for improvement in student learning, by improving our learning.

naplan data analysis school leadership module - ais...

Documents