National Assessment Program - Literacy and Numeracy 2012 RFT

Responses to Prospective Tenderers’ Questions – second set of queries Issued 10 February 2011

Question No.

RFT Reference

Question Response

1.

Part A, Project Timeline Writing Page 25

The Project timeline for Writing schedules ‘Online common activity marking by Marking Quality Team’ for 5/12/11 – 01/12 with the contractor to supply online marking access for MQT members. 1. How many members are there in the MQT? 2. How many scripts will they mark each? 3. Can we have any more details about the type

of online marking access required? 4. Will it be acceptable for MQT members to

record their marks for these scripts in a secure spreadsheet?

1) This information is in the footnote; i.e. writing task

managers for states and territories (8) and the Catholic and Independent schools’ representatives (2). The chief assessor is also a member and the contractor would attend meetings of the group.

2) The Marking Quality Team (MQT) members will mark approximately 200 scripts each (including the 80 scripts selected for consensus marking). In a few cases the MQT member delegates part of this work to another marker in their jurisdiction.

3) All student responses to the NAPLAN tests that are not machine scored are scored by markers using online marking technology. Each state and territory contracts the online marking technology solution from a suitable supplier. The marking of writing that is required by ACARA for this project includes the marking of 200 scripts, by approximately 15 markers (including the members of the MQT and members’ delegates). ACARA’s requirement is that marking utilise an online marking technology, as this is

considered to be current best practice.

As this process is the one from which marker-training scripts, control scripts, and warm-up scripts are selected for marking the writing task, ACARA's preference is for the Contractor to provide a similar system to that which the states and territories currently use for the marking of the actual tests. In terms of the requirements for such a system, if an online system is used, it must be a secure system, capable of showing a scanned writing script and marking rubric on-screen simultaneously. The marking rubric contains fields for each score point within each of the criteria (48 score points in total). The marking process must also support marker-monitoring by the chief assessor, by script and score, in a similar way to how the national marker monitoring occurs. Each marker’s scores must be easily exported for analysis by script and marker. Markers must get the same set of scripts in the same order.

4) Refer answer to question 3 above, it is a matter for the tenderer to determine whether any proposed system will meet ACARA’s requirements. Please note that this requirement is only one matter that will be taken into account when evaluating the tender. If you are unable to meet ACARA’s requirement in this regard, you are free to advise other options to meet the requirement.

2.

Contractors are required to attend up to four face-to-face meetings with ACARA and the NTWG as well as the close out meeting. Will these meetings be held at the ACARA offices in Sydney or elsewhere?

Meetings to be held in Sydney, at the ACARA offices.

3.

4.

Part A, page 24, test Construction period: “Analysis of data by contractor/s, …” Part A, page 30, Deliverables Test Construction 1 “Nominate items for inclusion in final test papers in accordance with the test specifications provided, based on expert analysis of the trial data and advice provided by ACARA” Part B, page 33, Payment Schedule, Milestone 3

1) Is the contractor responsible for the analysis of data from the Trial or is ACARA providing the analysis of the data and the contractor’s responsibility is to evaluate the results of data analysis to make decisions about the construction of the final test forms?

2) What is the expected scope of the “trial analysis report” referred to in the Payment Schedule?

1) The successful Tenderer is not responsible for analysing the data. ACARA will provide the analysis of data, and then the contractor’s responsibility will be to evaluate the results of the analysis to inform the decisions for the construction of the final test papers.

2) The successful Tenderer is not responsible for preparing the trial data analysis report. The trial data analysis report will be provided by ACARA and will contain a summary report of invigilator comments, in addition to the trial data, and any other information provided by the successful Tenderer relating to the quality of the trial process.

(Receipt of trial data analysis report) Appendix E: #16. “Following the in-house analysis of item performance at trialling, the contractor will receive data relevant to their items, and confirmation of items to be used for draft final test form construction.”

5.

Part A p.22 states that “A separate tender will be offered for the administration of trialling and marking. The analysis of the trial data will be undertaken independently.”

In the payment schedule in Part B p.33, line item 3 is Receipt of trial data analysis report. Could you please confirm if the receipt of a trial data analysis report is a deliverable within the current tender, or is that item there only in reference to a milestone within the wider project context?

This report is not a deliverable for the successful tenderer. However its purpose is relevant, in that the report will indicate the quality of the test materials, and whether the contractor has met the test specifications; e.g. spread of items.

6.

Would an Adobe Acrobat PDF file be an acceptable file format for the submission of the

Yes

tender response?