national assessment program ict literacy 2008 year assessment program ict literacy 2008 ... national

Download National Assessment Program ICT Literacy 2008 Year Assessment Program ICT Literacy 2008 ... National

Post on 09-Jun-2018

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • National Assessment Program

    ICT Literacy 2008

    Year 6 and Year 10

    Technical Report

    Christopher Freeman

    John Ainley

    Julian Fraillon

    Eveline Gebhardt

    Kate OMalley

    Wolfram Schulz

    May 2010

  • ii

    Prepared in 2010 by The Australian Council for Educational Research Ltd 19 Prospect Hill Road, Camberwell, Victoria, 3124, Australia.

    Copyright 2010 Australian Council for Educational Research

  • i

    Contents

    Chapter 1 Introduction ................................................................................................................. 1

    Project overview .................................................................................................................. 1

    National Assessment Program ICT Literacy ........................................................................ 1

    Participants in the assessment ............................................................................................. 2

    The assessment format ........................................................................................................ 2

    Reporting of the assessment results ..................................................................................... 3

    Structure of the Technical Report ......................................................................................... 4

    Chapter 2 Assessment Domain and Instrument Development ...................................................... 5

    The assessment domain ....................................................................................................... 5

    Summary of the assessment domain .................................................................................... 5

    Trend modules ..................................................................................................................... 6

    New item development........................................................................................................ 6

    The Field Trial ...................................................................................................................... 7

    Assessment system .............................................................................................................. 8

    The score guide .................................................................................................................... 9

    Student survey ................................................................................................................... 10

    Summary ........................................................................................................................... 10

    Chapter 3 Sampling and Weighting ............................................................................................ 12

    Sampling ............................................................................................................................ 12

    Weighting .......................................................................................................................... 14

    Summary ........................................................................................................................... 16

    Chapter 4 Field Administration and Data Management .............................................................. 17

    Field administration ........................................................................................................... 17

    Contact with schools .......................................................................................................... 17

    Assessment delivery methods ............................................................................................ 18

    Test Administrators ............................................................................................................ 20

    Test Administration ............................................................................................................ 20

    Marking procedures and marker training ........................................................................... 20

    Data management ............................................................................................................. 21

    School reports .................................................................................................................... 23

    Summary ........................................................................................................................... 23

  • ii

    Chapter 5 Scaling Procedures ..................................................................................................... 25

    The scaling model .............................................................................................................. 25

    Item calibration.................................................................................................................. 27

    Plausible values .................................................................................................................. 29

    Assessment of item fit ........................................................................................................ 29

    Differential item functioning by sex.................................................................................... 31

    Item deletions .................................................................................................................... 34

    Vertical and horizontal equating ........................................................................................ 35

    Uncertainty in the link ........................................................................................................ 36

    Scaling of questionnaire items ........................................................................................... 38

    Summary ........................................................................................................................... 39

    Chapter 6 Proficiency Levels And the Proficient Standards ......................................................... 40

    Proficiency levels ............................................................................................................... 40

    Setting the standards ......................................................................................................... 42

    Summary ........................................................................................................................... 43

    Chapter 7 Reporting of Results .................................................................................................... 44

    Estimation of sampling and measurement variance ........................................................... 44

    Reporting of mean differences across States and Territories .............................................. 45

    Reporting of mean differences across subgroups other than States and Territories ............ 46

    Reporting of differences across cycles 2005 to 2008 ........................................................ 46

    Other statistical analyses ................................................................................................... 47

    Summary ........................................................................................................................... 49

    References ............................................................................................................................. 50

    Appendix A Student Background Survey....................................................................................... 52

    Appendix B Sample Characteristics ............................................................................................. 58

    Appendix C Survey Administration Documents............................................................................. 61

    Document C1. Letter to Principal........................................................................................ 61

    Document C2. Letter to School Contact .............................................................................. 63

    Document C3. Year 10 School Contact Guide ..................................................................... 65

    Appendix D Example School Reports And Explanatory Material.................................................... 74

    Appendix E Student Background Variables Used For Conditioning ................................................ 77

    Appendix F ICT Literacy Proficiency Level Descriptors and Examples ............................................ 80

  • iii

    Tables

    Table 1.1 Achieved ICTL08 sample by State and Territory ........................................................ 2

    Table 2.1 Main aspects of NAP-ICT field trial ........................................................................... 8

    Table 3.1: Numbers of students and schools in the achieved sample...................................... 13

    Table 4.1 Procedures for field administration........................................................................ 17

    Table 4.2 Percentages of schools utilising each available delivery method for ICT literacy assessment ............................................................................................................ 19

    Table 4.3 Standard timing of the assessment session. ........................................................... 20

    Table 4.4 Summary of analy