hawaii statewide assessment program: maintaining test security and reporting testing incidents fall...
TRANSCRIPT
Hawaii Statewide Hawaii Statewide Assessment Assessment Program:Program:Maintaining Test Maintaining Test Security andSecurity andReporting Testing Reporting Testing IncidentsIncidents
Fall 2008Hawaii Department of EducationSystems Accountability Office (SAO)
1
This presentation focuses on testing incidents and the procedures for reporting testing incidents and inquiries into those reports.
2
Board of Education Policy Board of Education Policy No. 2520No. 2520STATEWIDE ASSESSMENT STATEWIDE ASSESSMENT PROGRAMPROGRAM POLICYPOLICY
“Systematic school, curricular, and program improvement efforts directed toward student attainment of the Hawaii Content and Performance Standards (HCPS) requires the collection of relevant and accurate student assessment information.”
3
Maintaining Integrity of theMaintaining Integrity of theHawaii State Assessment Hawaii State Assessment
(HSA)(HSA)Criterion Referenced Test (CRT) and Norm
Referenced Test (NRT) data
Curricular, instructional, personnel, and financial decisions
Improved student achievement
Strict adherence to test administration and security procedures is required to ensure that the student assessment results are accurate.
4
HSA ProceduresHSA Procedures
All school personnel involved in the administration of the HSA are accountable for knowledge of and adherence to all test security procedures as specified for their roles.
5
HSA guidelines and procedures can be found in the Test Administrator’s Handbook (TAH), the Test Coordinator’s Handbook (TCH) and the Student Participation and Accommodations Handbook (SPAH).
Additional mandatory resources:◦HSA District Meeting sessions or school
level trainings◦DVD: Hawaii Statewide Assessment
Program
Watching the DVD and attending the mandatory training are not substitutes for reading the HSA handbooks.
6
Hawaii State Alternate Assessment Hawaii State Alternate Assessment (HSAA) &(HSAA) & Hawaiian Aligned Portfolio Hawaiian Aligned Portfolio
Assessment (HAPA)Assessment (HAPA)
HSAA procedures can be found in the:◦HSAA Rater/Coordinator Manual◦HSAA Test Coordinator’s Handbook
HAPA procedures can be found in the:◦School Administrator - Test
Administrator - Test Coordinator Handbook
7
Testing IncidentsTesting Incidents
Any event or procedure that could potentially impact the integrity of the tests, the data, and the test results before, during, and after the test administration is a testing incident.
All testing incidents must be reported immediately.
8
Types of Testing IncidentsTypes of Testing Incidents
◦Testing irregularity
◦Standardized administration irregularity
◦Test security violation9
Testing IrregularityTesting Irregularity
Is an incident that affects the student’s performance on the test, but may, or may not, compromise the test result.
Must be reported May or may not trigger an inquiryMay require the administration of the
replacement test formMay require invalidation of the student’s
score
10
Testing Irregularity Testing Irregularity ExamplesExamples
Include, but are not limited to:Electrical outage during testingFire alarm during testingMany students taken ill due to vogHurricane evacuation during testingMedical emergencies involving key
personnelDefective materialsStudent bubbled the wrong portion of
the response booklet11
Standardized Administration Standardized Administration IrregularityIrregularity
Is a change in the established test administration procedure and protocols that may, or may not, compromise the test result.
Must be reported May or may not trigger an inquiryMay affect the student’s performance on the
testMay result in the administration of the
replacement test formMay require invalidation of the student’s
score12
Standardized Administration Standardized Administration Irregularity ExamplesIrregularity Examples
Include, but are not limited to: Disallowed classroom displays containing content
information visible during test sessions (see handbooks for details)
Test related content instruction, review or practice tests are provided between testing sessions (if it implies prior knowledge of test items, this could be a test security violation)
Test administrator fails to read the directions or conduct the examples as directed
Providing inappropriate accommodations Selecting specific students to “not test” Restricting or extending time to complete the testing
sessions13
Test Security ViolationTest Security ViolationA test security violation is a serious incident
that compromises test security, data security, or both, and the integrity of the student score results.
Must be reportedWill trigger an inquiry into the incidentMay affect the student’s performance on the
testMay result in the administration of the
replacement test formMay require invalidation of the student’s scoreSuperintendent will be notified
14
Test Security Violation Test Security Violation ExamplesExamples
Include, but are not limited to:• Use of mobile phones or cameras during testing
sessions to record test items• Test items or student responses are copied,
discussed, reviewed or analyzed before, during or after the assessment is administered
• Test administrator or proctor leaves the classroom unattended or with an unqualified adult during a session (see handbooks for qualifications)
• Test booklets are assigned to test administrator or proctors for their reference and review
• Test responses are altered• Prompting (verbal or non-verbal) or giving answers
to students during testing to review specific items• Students sharing answers, creating cheat sheets or
developing group answers
15
Procedure, Procedure, Sequence and Sequence and Expectations forExpectations forReporting a Reporting a Testing IncidentTesting Incident
16
Who Reports? Who Reports? All Hawaii Department of Education
employees are required to report testing incidents to the school Administrator.
Test Coordinators and school Administrators must report any knowledge of testing incidents to:
Student Assessment Section (SAS) Administrator
Complex Area Superintendent
17
Students, Parents, and Students, Parents, and Community Members Community Members
Non-DOE personnel may report testing incidents to school personnel.
School personnel receiving such
information are responsible for reporting the information to the school Administrator.
18
Test Coordinators and Test Coordinators and School AdministratorsSchool Administrators
If you have witnessed, been informed of, or suspect the possibility of a testing incident that could potentially impact the integrity of the tests, the data, and the test results, immediately contact:
Student Assessment Section (SAS) Administrator at 808-394-1300
In no answer, call… System Planning and Improvement (SPIS)
Administrator at 808-586-3285
19
If No Answer or After If No Answer or After HoursHours
Send a LN email to Kent Hinton
(cc: Christina Tydeman and Cara Tanimura) Provide: ◦ Your name, ◦ Your school, ◦ Phone numbers with hours for contact, and◦ The nature of your call.
You will be contacted by one of the individuals within your specified hours of contact.
The SAO Director will inform the Superintendent of all reported testing incidents.
20
Besides School Besides School Personnel Reporting to Personnel Reporting to
SAS…SAS…
State level data processing can reveal potential testing incidents or school level data quality concerns that may trigger an inquiry.
21
“ “Red Flag” ExamplesRed Flag” Examples
Include, but are not limited to:• Reviews of student demographic data
– Subgroup identifiers (i.e., ethnicity, SPED, ELL, disadvantaged)
– Grade level/birth date• Analyses of testing data– Student performance trends (e.g., school level,
grade level, subgroups)– Irregular scoring patterns– Inappropriate accommodations provided– Out of grade level testing
• Missing testing materials• Altered documents• Test scores for students who have been
exempted from taking the test by parent letter
22
Testing Irregularity FormTesting Irregularity Form
All schools reporting any testing incidents must complete and submit a “Testing Irregularity Form” for each incident to the Student Assessment Section.
23
24
Information Requested When Information Requested When Reporting to SASReporting to SAS
• Role/position of individual reporting – Name is optional
• School name• Date(s) of incident(s)• Description of incident(s)–Type (e.g., cheating, missing materials, copying
items)
–Scope (e.g., individual, class, grade level, school-wide)
–By whom (e.g., student, school personnel)
25
Step 1: Did a Testing Incident Step 1: Did a Testing Incident Occur?Occur?
What happened?Who was involved?How widespread?Who initiated the incident?
26
Step 1: OutcomesStep 1: Outcomes
If no testing incident is found, no further action is needed.
If a test/administration irregularity is found, SAS shall direct school personnel with appropriate actions.
If a test security violation is suspected, the school shall be directed to take immediate action (e.g., stop all testing) and a inquiry team shall be sent to the school.
27
Step 2: If an Inquiry is Step 2: If an Inquiry is NecessaryNecessary
School Administrator and the respective Complex Area Superintendent or Charter School Executive Director will be notified.
Upon notification, School Administrator must immediately:◦Cease testing◦Provide full cooperation and assistance
to the inquiry team Private room for interviews by the inquiry team Access to all individuals involved in the incident Copies of documents upon request of SAO
28
Documents to Have Documents to Have HandyHandy
FERPA forms – with parent/guardian signature for students involved in the incident
Test Administration Training Session Attendance Verification Form
Hawaii State Assessment Security and Administration Procedures Acknowledgment Form
29
Step 2: OutcomesStep 2: Outcomes
Inquiry team is established.◦Systems Accountability Office (SAO) personnel
◦Dispatched to the school upon notification of alleged test security violation
30
Step 3: The Site VisitStep 3: The Site Visit
Upon arrival at the school, the team will:
Meet with Test Coordinator/School Administrator◦Summarize and clarify the testing
incident◦Share the scope of the inquiry◦Clarify the roles of school personnel
within the inquiry◦Discuss possible outcomes◦Review copies of all FERPA forms
Conduct interviews◦Staff may have union representation
31
Scope of the InquiryScope of the Inquiry
Who will be interviewedHow the findings will be determinedWhen and if testing will be resumedPossible consequences regarding
test security and the validity of test scores
When the report will be completed
32
At the End of the Site VisitAt the End of the Site Visit
The inquiry team will brief the Administrator and Complex Area Superintendent or Charter Schools Executive Director on the findings.
33
Components of the Inquiry Components of the Inquiry ReportReport
• School context• Summary of the inquiry methodology• Descriptions of the incident, findings,
and conclusions• Recommended actions and timeline–Test security and administration
procedures–Validity of scores No recommendations for disciplinary
actions shall be included.
34
Suppressed Test ScoresSuppressed Test Scores
In the event the inquiry team determines a student’s scores to be invalid, the scores and the related test materials will be suppressed.◦In other words, it will be as if the
student(s) did not take the test session(s).
Schools may not view any testing materials or scores that have been suppressed as a result of a testing incident inquiry.
35
Effect of Suppressed Effect of Suppressed ScoresScores
Scores from individual testing sessions may be deemed invalid resulting in partial scores in the subject area for the affected students.
If all of a student’s scores for a subject area are deemed invalid, the student will have no score and will be considered a non-participant.
36
Step 3: OutcomesStep 3: OutcomesSAO makes the final determinations of score validity.
Inquiry report is presented by SAO Director:
Complex Area Superintendent or Charter Schools Executive Director
State Superintendent.
Superintendent approves the final recommended actions for improving test security and administration procedures.
37
Possible DeterminationsPossible Determinations
1. No testing incident occurred 2. Test/administration irregularity occurred =
valid scores◦ Did not compromise test security◦ Did not compromise test results (i.e., scores)
3. Test/administration irregularity occurred = invalid scores
◦ Did not compromise test security◦ Compromised test results (i.e., scores)
4. Test security violation occurred = invalid scores◦ Compromised test security◦ Compromised test results (i.e., scores)
38
At the Superintendent’s At the Superintendent’s Discretion…Discretion…
The Superintendent may share the testing incident inquiry report with the Board of Education or the Charter Schools Review Panel.
The school may be required to cover the costs of any replacement test forms used and other related costs (e.g., re-printing of score reports and testing materials, hiring raters).
39
Step 4: Public DisclosureStep 4: Public Disclosure
Approved recommendations are implemented.
The school Administrator must notify the affected parents in writing of the test security violation and its impact on their children’s scores.
The school, in consultation with the DOE’s Communications Office, must respond to Freedom of Information requests regarding the incident.
40
Overall OutcomesOverall OutcomesImproving test security contributes to
Improved validity and accuracy of assessment scores.
Transparency of the test security inquiry process for schools.
Public accountability through full compliance with No Child Left Behind.
Increased credibility and overall confidence in the student performance results. 41
For more information For more information regarding Testing regarding Testing Incidents and Inquiries, Incidents and Inquiries, please contact:please contact:
Systems Accountability Office at 808-586-3283
42