request for proposals (rfp) 20-004: grades 3-8 tests · web viewrequest for proposal (rfp) rfp....

RFP #20-004

REQUEST FOR PROPOSAL (RFP)RFP 20-004

NEW YORK STATE EDUCATION DEPARTMENT

Title: Continuation of the Development of the Grades 3-8 English Language Arts and Mathematics Tests, computer-based and paper-based, and Provision of a Computer-based

Testing Platform for the Grades 5 & 8 Science Tests

The New York State Education Department (NYSED) Office of State Assessment is seeking proposals from highly qualified respondents with expertise in the development and administration of Statewide tests in both paper- and computer-based testing platforms. Services required by this RFP include ongoing development and administration of tests, both computer- and paper-based, in English Language Arts (ELA) and Mathematics in Grades 3, 4, 5, 6, 7, and 8 (3-8) for multiple school years, as defined in this RFP. The requested services include conducting field testing using both stand-alone field tests and items embedded within the operational tests; scoring the field tests; developing all operational and field test items and forms, ancillary materials, and scoring materials; and performing all analysis and research. Proposals must include the printing, shipping, collection, and return of the paper-based ELA and Mathematics Tests, as well as the provision of a computer-based testing platform which includes the administration and test delivery systems and a scoring platform. Test development for the Grades 3-8 ELA and Mathematics Tests will start in July 2020 with the creation of field tests for administration in Spring 2021 and operational tests beginning in the 2020-21 school year. Bidders must bid on both content areas, ELA and Mathematics, and all requested grade levels.

Additionally, proposals must include the provision of the same computer-based testing platform for the administration of the Grades 5 and 8 Science Written Tests. Note that the development, printing, and shipping of the Grades 5 and 8 Science Written Tests will continue to be done exclusively by NYSED and is, therefore, not part of this RFP. Use of the computer-based testing platform for the Grades 5 and 8 Science Tests will begin with administration of computer-based field tests beginning in Spring of 2021 and the first administration of computer-based operational tests occurring in Spring of 2022.

The services described in this RFP are divided into two components, with Component 1 being divided into 1a, 1b, and 1c for services required for development and administration of the Grades 3-8 ELA and Math Tests. Component 2 covers the request for services for the provision of a computer-based testing platform for Grades 5 and 8 Science. Bidders are required to submit a technical proposal that clearly delineates the contractor’s plan for providing the services required in Component 1a, 1b, and 1c separately from its plan for providing the additional services required in Component 2. Bidders are also required to submit costs for Component 1a, 1b, and 1c and Component 2, as shown in the schedules of deliverables in the Bid Form Cost Proposal (separate attachment). Following the scoring of the technical and cost proposals submitted for all four components, NYSED will make a determination whether, based on available funding, to award a contract that includes Component 2 to the bidder who received the highest total score. If it is determined that the cost of the highest ranked proposal with the inclusion of Component 2 is above the amount of funding available, then a second ranking will be completed using the technical and cost proposals submitted for Component 1a, 1b, and 1c only. Please see Section 3 on Method of Award for additional information.

Component 1 services cover the development and administration of computer- and paper-based Grades 3–8 ELA and Mathematics Tests and is divided into three sections. Component 1a services

i

RFP #20-004

are for the development of an operational test item bank and construction of stand-alone field tests and operational test forms. Component 1b services are for computer-based delivery of the Grades 3-8 ELA and Math tests. Component 1c services are for the printing, shipping, collection, and return of printed test forms for those schools that continue to administer the Grades 3-8 ELA and Math Tests on paper rather than computer. As part of services required by Component 1, the contractor will provide accommodated versions for students with disabilities and English Language Learners (ELLs/MLLs) as part of both the computer-based and paper-based tests.

Component 2 services are for the provision of the computer-based testing platform for the Grades 5 and 8 Science Tests. Because these tests will continue to be developed exclusively by NYSED, this component requires only the provision of a computer-based testing platform for the Grades 5 and 8 Science Written Tests. Embedding of field test items on the operational tests will necessitate the administration of multiple forms via computer-based testing (CBT) enabling New York State to eliminate the need for stand-alone field tests. Services required for the provision of a computer-based testing platform for Grades 5 and 8 Science Written Tests includes a scoring platform.

New York State’s Transition from Paper-based Tests (PBT) to Computer-based Tests (CBT)

New York State’s (NYS) transition to CBT began with the 2016 stand-alone field tests in ELA and Math. In the most recent year, approximately 35% of NYS schools participated in the NYS Grades 3-8 ELA and Math field tests or operational tests on computer. One intent of this project is to continue the shift from PBT to CBT administration in Statewide large-scale operational testing and may include Grades 5 and 8 science in computer-based testing should it be part of the contract as awarded.

The contractor must provide the technical specifications for a multi-faceted electronic assessment technical solution that includes, at minimum, an online test delivery system, online test administration system, and online scoring platform. Proposals must also include a detailed plan for the transition to NYS school use of the contractor’s solution for computer-based testing. Technical specifications must include the following:

Minimum specifications for all devices on which the test is delivered;

Minimum specifications for any administrative computer associated with the devices on which the test is delivered (if applicable);

Minimum specifications for the networking associated with any devices involved in the delivery of tests (if applicable);

Minimum infrastructure specifications for Internet, printing, and any other applicable requirements involved in the delivery of the tests; and

Minimum infrastructure specifications for contractor-based distributed scoring platform.

Grades 3-8 ELA and Math — It is currently at the discretion of each school and revisable annually, as to whether they will administer the Grades 3-8 ELA and Math tests by paper or computer, so the contractor must be prepared to provide CBT to schools within every local school district. Approximately 35% of schools administer either the operational or stand-alone field tests for Grades 3-8 ELA and Math using the CBT delivery system. However, it is anticipated that the percentage of schools and test-takers within those schools taking CBT will increase each school year of this contract. Some schools and individual test-takers may continue to use PBT, due to lack of school capacity or individual student needs. The contractor must be flexible and must be prepared for the

ii

RFP #20-004

tests to be administered via both PBT and CBT in unknown proportions by schools throughout the state.Grades 5 and 8 Science — With this contract, NYSED will initiate the transition from PBT to CBT for the Grades 5 and 8 Science Tests. It will be at the discretion of each school and revisable annually, as to whether they will administer the Grades 5 and 8 Science Tests by paper or computer or both. For Spring 2021, schools will have the option of administering the stand-alone field tests (SAFTs) by CBT or PBT, but all administrations of the operational tests will be by PBT. For all subsequent school years of this contract, schools will have the option of administering the operational tests as well as the SAFTs by either CBT or PBT. Included in the services required for a computer-based testing platform for the Grades 5 and 8 Science Tests is the provision of a computer-delivered scoring platform.

The contractor may choose to subcontract for the provision of a computer-based testing platform for the Grades 3-8 ELA and Math Tests and Grades 5 and 8 Science Tests. For tests administered by computer, the contractor and its subcontractor, if there is one, must ensure that the test administration and test delivery systems and scoring platform are compatible with the existing devices and infrastructure most often available in New York State’s schools. See Attachment A for details on the minimum technical specifications for devices to be used for CBT.

See number 1. in the Mandatory Requirements section, for a description of the eligible applicants.

Subcontracting will be limited to thirty percent (30%) of the total contract budget. A higher subcontracting limit will be allowed only when a bidder is proposing to subcontract for the provision of the computer-based testing platform. In this case, the subcontracting limit will be increased to fifty percent (50%) of the total contract budget. In all other cases, the subcontracting limit will be 30%. NYSED reserves the right to approve all subcontractors. Subcontracting is defined as non-employee, direct, personal services and related incidental expenses, including travel. For purposes of this RFP, the subcontracting limit does not include expenses related to printing, shipping, or the purchase of equipment, supplies, and materials.

The contract resulting from this RFP will be for a single award with a term of five years and five months. The contract is anticipated to begin on July 1, 2020 and end on November 30, 2025. Bidders should refer to the Mandatory Requirements section of this RFP and to the Schedule of Deliverables when preparing their proposals. Bidders must adhere to all Mandatory Requirements and perform all required deliverables.

Bidders are required to comply with NYSED’s Minority and Women-Owned Business Enterprises (M/WBE) participation goals for this RFP through one of three methods. Compliance methods are discussed in detail in the Minority/Women-Owned Business Enterprise (M/WBE) Participation Goals section.

Service Area: Statewide

Mandatory Requirements: See Mandatory Requirements section of the RFP.

Components contained in RFP Proposal #20-004 are as follows:

1.) Description of Services to Be Performed2.) Submission3.) Evaluation Criteria and Method of Award

iii

RFP #20-004

4.) Assurances5.) Attachments6.) Submission Documents (separate document)

Each bidder will be scheduled for a mandatory demonstration of the online delivery system it proposes. The demonstration will be scheduled in consultation with the contractor. It is anticipated that all demonstrations will be scheduled for one hour on a date mutually agreed upon by NYSED and the bidder. Demonstrations will be conducted either in person at NYSED’s Albany, NY office or by webinar. This demonstration must take place between 2/10/2020 and 2/14/2020. To schedule the demonstration, bidders may contact Victoria Mahar via email at [email protected] prior to the deadline for submitting proposals. Bidders who have not scheduled a demonstration time in advance will be provided a time once bids are received. See RFP section “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Mathematics and the “Computer-Based Testing Demonstration” for additional information.

Questions regarding the request must be submitted by E-mail to [email protected] no later than the close of business December 18, 2019. Questions regarding this request should be identified as Program, Fiscal or M/WBE. A Questions and Answers Summary will be posted to Competitive Procurement Contracts website no later than January 2, 2020.

The following are the designated contacts for this procurement:

Program Matters Fiscal Matters M/WBE MattersVictoria Mahar Jessica Hartjen Brian HackettEmail Address:[email protected]

The following documents must be submitted in separately sealed envelopes, as detailed in the Submission section of the RFP, and be received at NYSED no later than January 23, 2020 by 3:00 PM:

1. Submission Documents labeled Submission Documents - RFP #20-004 Do Not Open2. Technical Proposal labeled Technical Proposal - RFP #20-004 Do Not Open3. Cost Proposal labeled Cost Proposal – RFP #20-004 Do Not Open4. M/WBE Documents labeled M/WBE Documents—RFP #20-004 Do Not Open5. CD-ROMs or Flash Drive containing the technical/cost proposal, M/WBE and Submission

Documents labeled CD-ROM/Flash Drive-RFP#20-004 Do Not Open.

The mailing address for all the above documentation is:

NYS Education DepartmentBureau of Fiscal ManagementAttn: Jessica Hartjen, RFP# 20-004 Contract Administration Unit89 Washington Avenue, Room 501W EBAlbany, NY 12234

iv

mailto:[email protected]

RFP #20-004

Table of Contents1.) DESCRIPTION OF SERVICES TO BE PERFORMED........................................................................1

Work Statement and Specifications.................................................................................................................1Mandatory Requirements.................................................................................................................................1Minority and Women-Owned Business Enterprise (M/WBE) Participation Goals Pursuant to Article 15-A of the New York State Executive Law............................................................................................................2Service-Disabled Veteran-Owned Business (SDVOB) Participation Goals Pursuant to Article 17-B of New York State Executive Law...............................................................................................................................3Background Information..................................................................................................................................5Deliverables and/or Project Description..........................................................................................................7Tentative Test Administration Schedule.........................................................................................................8Component 1a: Test Development Requirements for Grades 3-8 English Language Arts and Mathematics.......................................................................................................................................................................13English Language Arts and Mathematics Test Designs................................................................................14Sequencing.....................................................................................................................................................23Internal Test Blueprints for ELA and Mathematics......................................................................................31Test Development Process for ELA and Mathematics..................................................................................31Sequence of Test Development Events.........................................................................................................32Educator Committees.....................................................................................................................................33External Staffing Requirement for Development of Grades 3-8 ELA and Math..........................................34Field Testing for ELA and Mathematics.......................................................................................................34Field Test Sample Design..............................................................................................................................38Analysis of Field Tests..................................................................................................................................39Field Test Technical Report...........................................................................................................................39Liquidated Damages Related to Passage Selection and Item Development for ELA and Math...................39Item Banking for the Grades 3-8 ELA and Math Tests.................................................................................40Operational Test Administration for Grades 3-8 ELA and Math..................................................................41Operational Forms Preparation......................................................................................................................42Local Scoring of Operational Tests for Grades 3-8 ELA and Math..............................................................43Scoring Materials for Operational Test Constructed-Response Items..........................................................44Liquidated Damages Related to Operational Tests – Translated, Braille, and Large-Type Editions and all Scoring Materials for Constructed-Response Items......................................................................................45Grades 3-8 ELA and Math Manuals (Nonsecure Materials).........................................................................47Updating the Educator Guides.......................................................................................................................47Updating the Style Guide...............................................................................................................................47Forms Scaling and Equating..........................................................................................................................48

i

RFP #20-004

Operational Test Technical Report................................................................................................................48Technical Advisory Committee.....................................................................................................................49Standard Setting.............................................................................................................................................49Performance Level Descriptors (PLD)..........................................................................................................50Alignment Studies for ELA and Math...........................................................................................................50Feasibility Study for Implementing Computer-adaptive Testing..................................................................51Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math................................51Computer-Based Operational and Field Test Forms.....................................................................................52Testing System Specifications.......................................................................................................................52User Interface.................................................................................................................................................55Administrative System...................................................................................................................................57System Monitoring and Reporting.................................................................................................................58System Availability, Maintenance, and Updates...........................................................................................59Scoring Platform and Data Management.......................................................................................................60Readiness Verification Process......................................................................................................................61Online Question Sampler and Secure Browser Practice Tests......................................................................62Training and Technical Support....................................................................................................................63Computer-Based Testing Hosted Solution, Minimum Technical Specifications..........................................64Simulation Testing.........................................................................................................................................65Comparability of Testing Modes (PBT versus CBT) for Grades 3-8 ELA and Math...................................65Liquidated Damages Related to CBT for Grades 3-8 ELA and Mathematics..............................................65Component 1c: Printing, Shipping, Collection and Return of Grades 3-8 ELA and Math Operational Tests Materials and Liquidation of Damages Related to Printing, Shipping and Collection of Paper-based Grades 3-8 ELA and Math Operational Materials.....................................................................................................67Printing of Regular English, Large Type and Translated Editions of Operational Tests..............................67Shipping.........................................................................................................................................................69Collection, Return, and Secure Destruction..................................................................................................70Secure Handling of Returned Assessment Materials.....................................................................................70Missing Materials Report/Support Locating Missing Materials...................................................................71Clarification of Contractor Responsibility.....................................................................................................71Liquidated Damages Related to Printing and Shipping of Paper-based Operational Test Materials............72NYSED Assumption of Printing, Shipping, Collection and Return..............................................................74Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests....................................................................................................................................74Timeline for CBT Services to be Provided by the Contractor for the Grades 5 and 8 Science Tests...........75Computer-Based Test Forms.........................................................................................................................78Testing System Specifications.......................................................................................................................78

ii

RFP #20-004

Online Student Testing Accommodations.....................................................................................................78Online Question Sampler and Secure Browser Practice Tests......................................................................79Training and Technical Support....................................................................................................................80Computer-Based Testing Hosted Solution, Minimum Technical Specifications..........................................81Provision to NYSED of the Raw Data Records of Students’ CBT-Entered Responses for Stand Alone Field Tests...............................................................................................................................................................81Provision to NYSED of the Raw Data Records of Students’ CBT-Entered Responses for Operational Tests.......................................................................................................................................................................81Hosting a Secure Site at Which School Administrators Can Access Their School’s Student CBT-Entered Constructed Responses for the Science Operational Written Tests...............................................................82Comparability of Testing Modes (PBT vs CBT) for Grades 5 and 8 Science..............................................82Liquidated Damages Related to CBT Services for Grades 5 and 8 Science Tests........................................82Requirements for All Components................................................................................................................84Staffing Requirements and Program Management........................................................................................84Program Management Requirements............................................................................................................84Program Manager Duties...............................................................................................................................84Staffing Requirements for the Grades 3-8 ELA and Mathematics Test Development.................................85Publications and Editorial Requirements.......................................................................................................86Technical Writer Requirements.....................................................................................................................86Technical Writer Duties.................................................................................................................................86Business Analyst of Data Requirements........................................................................................................86Notification Procedures.................................................................................................................................87NYSED Approval..........................................................................................................................................87Ownership......................................................................................................................................................88Data Security, Data Privacy and Appropriate Use........................................................................................88Requirements of Education Law § 2-d..........................................................................................................89Promotion of Products and Services Outside the Contract............................................................................89Invoices and Payments..................................................................................................................................90Accessibility of Web-Based Information and Applications..........................................................................90Subcontracting Limit.....................................................................................................................................90Consultant Staff Changes..............................................................................................................................91Contract Period..............................................................................................................................................91Electronic Processing of Payments................................................................................................................91

2.) SUBMISSION.......................................................................................................................................95Project Submission........................................................................................................................................95Technical Proposal.........................................................................................................................................96Computer-Based Testing Demonstration....................................................................................................103

iii

RFP #20-004

Cost Proposal...............................................................................................................................................104M/WBE Documents.....................................................................................................................................104

3.) EVALUATION CRITERIA AND METHOD OF AWARD..............................................................106Criteria for Evaluating Bids.........................................................................................................................106Technical Criteria........................................................................................................................................106Financial Criteria.........................................................................................................................................107Method of Award.........................................................................................................................................107NYSED’s Reservation of Rights.................................................................................................................108Post Selection Procedures............................................................................................................................109Debriefing Procedures.................................................................................................................................109Contract Award Protest Procedures.............................................................................................................110Vendor Responsibility.................................................................................................................................110Procurement Lobbying Law........................................................................................................................111Consultant Disclosure Legislation...............................................................................................................111Public Officer’s Law Section 73..................................................................................................................112NYSED Substitute Form W-9.....................................................................................................................113Workers’ Compensation Coverage and Debarment....................................................................................113Proof of Coverage Requirements.................................................................................................................113Sales and Compensating Use Tax Certification (Tax Law, § 5-a)..............................................................114

4.) ASSURANCES...................................................................................................................................116State of New York Agreement....................................................................................................................117Appendix A: Standard Clauses for NYS Contracts............................................................................................119Appendix A-1..............................................................................................................................................124Appendix R: Data Security and Privacy Plan Provisions............................................................................129Appendix S: Parents’ Bill of Rights for Data Privacy and Security............................................................132Appendix S1: Attachment to Parents’ Bill Of Rights For Contracts Involving Disclosure of Certain Personally Identifiable Information.............................................................................................................140

5.) ATTACHMENTS......................................................................................................................................142Attachment A: Minimum Technical Requirements for Computer Devices to be Used by Schools for CBT.....................................................................................................................................................................142Attachment B: Grades 3-8 ELA and Math Test Development Process.......................................................144Attachment C: Passage Development..........................................................................................................145Attachment D: Item Development...............................................................................................................149Attachment E: Educator Committees for ELA and Math............................................................................184Attachment F: External Staffing Requirements for Test Development Process.........................................188Attachment G: Grades 3-8 ELA Tests Guidelines for Passages..................................................................189Attachment H: Assessment Specialist Content Verification Checklist.......................................................191

iv

RFP #20-004

Attachment I: Specifications for Contractor/Subcontractor Transcribing Braille.......................................195Attachment J: Guidelines for Large-Type Print Editions............................................................................196Attachment K: Technical Manual Outline for New York State Assessments.............................................197Attachment L: Print Quantities....................................................................................................................199Attachment M: Security Guidelines for the New York State Assessment Program...................................200

v

RFP #20-004

1.) DESCRIPTION OF SERVICES TO BE PERFORMED

Work Statement and Specifications

This section of the bid package details the services and products to be acquired. Please note that the contract process also includes general New York State administrative terms and conditions, as well as terms and conditions required by New York State law. These terms and conditions address issues related to both the submission of bids and any subsequent contract; they are included separately in this bid package for your information. Please review all terms and conditions.

Mandatory Requirements

The eligible bidder must agree to the Mandatory Requirements found below and must submit the Mandatory Requirements Certification Form located in 6.) Submission Documents (separate document) signed by an authorized person. If the bidder’s proposal fails to meet any of these mandatory requirements or submit the signed Mandatory Requirement Certification Form, it will be disqualified.

1. Bidders must be either for-profit or not-for-profit entities, including large testing companies, consultants, and/or educational organizations with a demonstrated capacity to complete all required services for large-scale assessments. Bidders must, at a minimum, have three years of experience working with the development and administration of print-based and computer-based large-scale assessments and educational measurement, including test development, scoring, scaling and equating, data analysis, standard setting, technical reporting, and computer-based testing.

2. Bidders must bid on all aspects of both Component 1 and 2 for the Grades 3-8 ELA and Mathematics Tests and the Grades 5 and 8 Science Tests as described in this RFP for all years listed.

3. Bidders must provide a full staffing plan with FTEs specified for this project that, at a minimum, must include and maintain:

a. three fulltime program managers who are: 1) fulltime employees of the entity, and 2) each have a Bachelor’s Degree or above.

b. three fulltime Mathematics content specialists, one each for Grades 3–4, 5–6, and 7–8, and three fulltime ELA content specialists, one each for Grades 3–4, 5–6, and 7–8, for a total of six content specialists, all of whom meet the minimum qualifications specified in the “Staffing Requirements for the Grades 3-8 ELA and Mathematics Test Development” section of this RFP.

c. at least one fulltime development supervisor for the duration of the contract who is: 1) a fulltime employee of the entity, and 2) meets the minimum education and experience requirements in either ELA or Mathematics specified for content specialists in the “Staffing Requirements for the Grades 3-8 ELA and Mathematics Test Development” section of this RFP, and is not serving as one of the content specialists.

d. at least two technical writers who must have a Bachelor’s Degree or above, as well as

1

RFP #20-004

5 years of experience in the field of technical writing/editing.

e. three editors.

f. one fulltime business analyst of data who meets the minimum qualifications specified in the “Staffing Requirements for the Grades 3-8 ELA and Mathematics Test Development” section of this RFP.

4. Each bidder must provide NYSED a mandatory demonstration of its computer-based testing and scoring system. All demonstrations will be scheduled for one hour on a date mutually agreed upon by NYSED and the bidder. Demonstrations may be conducted either in person at NYSED’s Albany, NY, office or by webinar. This demonstration must take place between 2/10/2020 and 2/14/2020.

Minority and Women-Owned Business Enterprise (M/WBE) Participation Goals Pursuant to Article 15-A of the New York State Executive Law

For purposes of this procurement, NYS Education Department hereby establishes an overall goal of 30% of the total contract amount for M/WBE participation, 17% for Minority-Owned Business Enterprises (“MBE”) participation and 13% for Women-Owned Business Enterprises (“WBE”) participation based on the current availability of qualified MBEs and WBEs. All bidders must document good faith efforts to provide meaningful participation by MWBEs as subcontractors or suppliers in the performance of this Contract. Minority and Women-Owned Business Enterprise (M/WBE) participation includes any and all services, materials or supplies purchased from New York State certified minority and women-owned firms. Utilization of certified Minority and Women-Owned firms will be applied toward the goals. Bidders can achieve compliance with NYSED’s Minority and Women-Owned Business Enterprise goals as described below.

ACHIEVE FULL COMPLIANCE WITH PARTICIPATION GOALS (PREFERRED)Bidders should submit subcontracting/supplier forms that meet or exceed NYSED’s participation goals for this procurement. All subcontracting/supplier forms must be submitted with the bid proposal. In addition, bidders must complete and submit M/WBE 100: Utilization Plan, M/WBE 102: Notice of Intent to Participate and EEO 100: Staffing Plan. Instructions and copies of these forms are located in the Submission Documents. All firms utilized must be certified with the NYS Division of Minority and Women Business Development before beginning any work on this contract. For additional information and a listing of currently certified M/WBEs, see the NYS Directory of Certified Minority and Women-Owned Business Enterprises.

The contact person on M/WBE matters is available throughout the application and procurement process to assist bidders in meeting the M/WBE goals. NYSED reserves the right to approve the addition or deletion of subcontractors or suppliers to enable bidders to comply with the M/WBE goals, provided such addition or deletion does not impact the technical proposal and/or increase the total cost of the bid proposal.

DOCUMENTATION OF GOOD FAITH EFFORTSBidders must undertake a good faith effort to solicit NYS Certified M/WBE firms as subcontractors and/or suppliers in fulfillment of this procurement. Means of solicitation may include but are not limited to: advertisements in minority centered publications; solicitation of vendors found in the NYS Directory of Certified Minority and Women-Owned Business Enterprises; and the solicitation

2

https://ny.newnycontracts.com/FrontEnd/VendorSearchPublic.asp?TN=ny&XID=4687



RFP #20-004

of minority and women-oriented trade and labor organizations. Bidders will be required to certify and attest to their good faith efforts by completing NYSED’s Certification of Good Faith Efforts (Form M/WBE 105). See the M/WBE Submission Documents for detailed examples of and required forms to document good faith efforts.NYSED reserves the right to reject any bid for failure to document “good faith efforts” to comply with the stated M/WBE goals.IN THE EVENT BIDDERS CANNOT COMPLY WITH NYSED DESIGNATED PARTICIPATION GOALS, SAID BIDDERS MUST DOCUMENT THEIR “GOOD FAITH EFFORTS” TO COMPLY AND SUBMIT ONE OF THE FOLLOWING REQUESTS:

REQUEST A PARTIAL WAIVER OF PARTICIPATION GOALSIn order to request a partial waiver of the participation goals for this procurement, Bidders must provide documentation of their good faith efforts to obtain the use of certified M/WBE enterprises along with their bid proposal forms. The subcontracting forms must include the participation percentage(s) for which they seek approval. Bidders will be required to certify and attest to their good faith efforts. Bidders should submit a request for a partial waiver (Form M/WBE 101) and document their Good Faith Efforts (Form M/WBE 105) at the same time as the bid is submitted. Bidders must also complete and submit M/WBE 100: Utilization Plan, M/WBE 102: Notice of Intent to Participate and EEO 100: Staffing Plan. The M/WBE Coordinator is available throughout the procurement process to assist in all areas of M/WBE compliance.

REQUEST A COMPLETE WAIVER OF PARTICIPATION GOALSIn order to request a complete waiver of the participation goals for this procurement, Bidders must provide documentation of their Good Faith Efforts to obtain the use of certified M/WBE enterprises along with their bid proposal forms. Bidders will be required to certify and attest to their good faith efforts. Bidders should submit a request for a complete waiver on Form M/WBE 101 and document their Good Faith Efforts (Form M/WBE 105) at the same time as they submit their bid. The M/WBE Coordinator is available throughout the procurement process to assist in all areas of M/WBE compliance.

All payments to Minority and Women-Owned Business Enterprise subcontractor(s) must be reported to NYSED M/WBE Program Unit using M/WBE 103 Quarterly M/WBE Compliance Report. This report must be submitted on a quarterly basis and can be found at NYSED’s M/WBE Forms and Compliance Forms webpage.

Service-Disabled Veteran-Owned Business (SDVOB) Participation Goals Pursuant to Article 17-B of New York State Executive Law

Article 17-B of Executive Law was enacted to ensure that certified SDVOBs are provided opportunities for meaningful participation in the performance of state contracts. To this end, NYSED strongly encourages bidders to make maximum possible use of SDVOBs as subcontractors and/or suppliers under this contract, consistent with the requirements of State Finance Law and State procurement guidelines, as well as NYSED policies and procedures. Bidders should consider fulfilling the requirements of this contract through the participation of SDVOBs at a rate of 6%. For additional information about this program, including a list of SDVOBs, please visit the Office of General Services, Division of Service-Disabled Veterans’ Business Development website .

3

http://www.ogs.ny.gov/Core/SDVOBA.asp

http://www.ogs.ny.gov/Core/SDVOBA.asp

http://www.oms.nysed.gov/fiscal/MWBE/forms.html

http://www.oms.nysed.gov/fiscal/MWBE/forms.html

RFP #20-004

Glossary of Terms and Acronyms Used in this RFP

Term DescriptionBOCES Board of Cooperative Educational ServicesCAU NYSED’s Contract Administration UnitCBT Computer-Based TestCR Constructed ResponseData Dictionary A set of information describing the contents, format, and structure of a

database and the relationship between its elements, used to control access to and manipulation of the database

Department New York State Education DepartmentDIF Differential Item FunctioningELA English Language ArtsELL/MLL English Language Learner/Multilingual LearnerESSA Every Student Succeeds ActFT Field TestIRT Item Response TheoryISP Internet Service ProviderLarge Scale Assessment

The measuring of student achievement on a specified set of academic constructs at numerous schools or testing locations, such as across a large city, region, state, or nation, using test instruments administered and scored in strict accordance with test publisher-provided standardized procedures and within the same specified interval of time.

MC Multiple-Choice M/WBE Minority and Women-Owned Business EnterpriseNYCDOE New York City Department of EducationNYS New York StateNYSED New York State Education DepartmentOIB Operational Item BankOSA NYSED’s Office of State AssessmentPBT Paper-Based TestPL Performance Level PLD Performance Level DescriptorRFP Request for ProposalRIC Regional Information CenterSAFT Stand-alone Field TestSAM School Administrator’s ManualSIRS Student Information Repository SystemSLH Scoring Leader HandbookTAC NYSED’s Technical Advisory Committee on assessmentsTD Teacher’s DirectionsUD Universal DesignUIN Unique Identification Number–used to identify items in the item bank

4

RFP #20-004

Background Information

New York State School Systems

The NYS Grades 3-8 Assessments are administered to approximately 1.2 million students in approximately 4,500 public, religious, independent, and charter schools throughout the State.

Many public schools throughout New York State partner with Boards of Cooperative Educational Services (BOCES). BOCES provide educational programs and services to school districts as a way of consolidating these services to make them more cost-effective. BOCES are also important liaisons between the State Education Department and schools. There are 37 BOCES throughout the state that assist NYSED in the implementation of many aspects of the Grades 3-8 assessments. District Superintendents, the chief executive officers for the BOCES, are an important part of the partnership between the State and the BOCES and are central points of contact for communication. It should be noted that BOCES membership is not currently available to the “Big Five” city school districts in New York: New York City, Yonkers, Syracuse, Rochester, and Buffalo.

New York State also relies heavily on its partnership with Regional Information Centers (RICs) and large city scanning centers. RICs are located throughout the State and offer technology-related services to school districts. RICs provide answer sheets and scanning services to schools for paper-based tests and have been important partners with the State in the transition to computer-based testing. Two of the “Big Five” city school districts, NYC and Yonkers, operate their own scanning centers. The other three cities obtain services from one of the RICs.

New York State Grades 3-8 Assessments

The Grades 3-8 New York State Tests fulfill the federal requirements of the “Every Student Succeeds Act” (ESSA). ESSA mandates that 95% of students in each public school, including charter schools, and each school district participate yearly in assessments in English Language Arts and Mathematics in Grades 3-8, and in science at least once during Grades 3-5 and 6-9. This 95% participation requirement applies to all subgroups, which include racial/ethnic groups, ELLs/MLLs, low-income students, and students with disabilities. While not required, many religious and independent schools in New York State also participate in this testing program.

Currently, the Grades 3-8 assessments are based on the New York State Learning Standards (NYSLS). However, on September 11, 2017, the Board of Regents, which is responsible for setting educational policy and standards, adopted the newly revised Next Generation English Language Arts and Mathematics Standards. The Grades 3-8 ELA and Math assessments administered in Spring 2020 will be based on the current NYSLS. Beginning in 2021, these tests will be based on the Next Generation Standards. These standards are the foundation for a system of assessments designed to:

evaluate the achievement of the knowledge and skills delineated by the learning standards and their constituent key ideas and performance indicators;

provide information that helps teachers adapt instruction to students' strengths and needs; provide useful information to students, parents, educators, and the general public about

what students know and can do; and serve as an important source of data for student performance at the school, district, and

State levels.

5

RFP #20-004

Educator Guides, which outline test design and blueprint information for the current ELA and Math tests and previously released items with item maps, are available at the EngageNY ELA and Mathematics Tests web site. The Next Generation Standards in English Language Arts and Mathematics, as well as related information, are also available on the EngageNY Next Generation Learning Standards web site.

CBT Preparedness in NYS Schools

No single technical architecture for WAN, LAN, server hardware, or desktop hardware is in use by New York State’s school districts. Currently, four primary types of testing devices can be used by students in New York State schools, Windows, iPads, Chromebooks, and Macintosh. The contractor must be prepared to support testing on all four types of devices. Local school districts contract with multiple Internet Service Providers (ISPs) statewide for varying levels and types of Internet connectivity. Any proposed online test delivery solution must also accommodate this variability and make use of existing technology within schools at the time of test administration. It is anticipated that schools will continue to indicate on the NYSED Exam Request Portal whether they choose to use CBT or PBT for operational tests in a specified school year. The contractor must develop a method for capturing what testing device(s) each CBT school anticipates using, as well as a method for determining the technological preparedness of a school new to computer-based testing.

It will be at the discretion of the local school and school district whether each school will administer the tests by paper or computer, so the contractor must be prepared to provide CBTs to schools within every local school district. It is anticipated that the percentage of schools and test-takers within those schools taking the CBT will increase each school year of this contract. Some schools and individual test-takers may continue to use PBTs, due to lack of school capacity or individual student needs. The contractor must be flexible and prepared for the tests to be administered by schools via both PBT and CBT in unknown proportions.

During the term of the contract resulting from this RFP, NYSED will want to explore with educators and other stakeholders the feasibility of implementing computer-adaptive testing. Toward this end, NYSED will hold a limited number of educator engagements at which the contractor would be expected to attend and help inform the discussions. In addition, the contractor will be required to conduct a feasibility study for implementing computer-adaptive testing to assist the Department with its planning. (See the section Feasibility Study for Implementing Computer-adaptive Testing). Please note that if such discussions result in NYSED wanting to move forward with the development of computer-adaptive testing, NYSED would enter into negotiation with the contractor for an amendment to this contract. Such amendment would require the approval of the Office of the State Comptroller.

In 2014, New York State voters approved the Smart School Bond Act, which authorized the issuance of $2 billion of general obligation bonds to finance improved educational technology and infrastructure to improve learning and opportunities for students throughout the State. Information and guidance can be found on NYSED’s Office of Education Management Services website. Please note that the New York State Education Department does not regulate, mandate, or make decisions on technology, device, or infrastructure purchases; all purchasing decisions are under local school district control. Attachment A provides NYSED’s minimum device specifications for New York State schools for use with CBT at the time of the development of this RFP.

6

http://www.p12.nysed.gov/mgtserv/smart_schools/

https://www.engageny.org/next-generation-learning-standards

https://www.engageny.org/next-generation-learning-standards

https://www.engageny.org/3-8


RFP #20-004

For tests administered by computer, the contractor must ensure that the test administration and test delivery systems and scoring platform are compatible with the existing computers and infrastructure available in New York State’s schools.Tentative Test Administration Schedule for Grades 3-8 ELA and Mathematics and for the Grades 5 and 8 Science TestsCurrently, the test administration schedule provides for administration of the operational Grades 3-8 ELA Tests in March/April, operational Grades 3-8 Math Tests in April/May and the Elementary- and Intermediate-level Science Tests in May/June. Stand-alone Field Tests in Grades 3-8 ELA and Math and the Elementary- and Intermediate-level Science Tests are administered in May/June.

Deliverables and/or Project Description

Component 1a encompasses providing the services necessary for the continued development of secure tests, measuring the New York State Learning Standards (NYSLS), to assess the achievement of elementary- and middle-level students. The tests will be developed by the contractor in:

ELA (Grades 3, 4, 5, 6, 7, and 8) Mathematics (Grades 3, 4, 5, 6, 7, and 8)

Component 1b encompasses the provision of a computer-based testing platform (test delivery system, administration system, and scoring platform) for the Grades 3-8 ELA and Math Tests. The computer-based platform must accommodate the provision of accommodated tests, including translated editions for Math – Chinese (Simplified), Haitian Creole, Korean, Russian, and Spanish.

Component 1c encompasses providing printing, shipping, collection, return and secure destruction of the paper-based Grades 3-8 ELA and Math Tests to schools that continue to administer paper-based tests. Additionally, paper-based tests are required as part of the provision of accommodations for certain students with disabilities and for certain students identified as ELLs/MLLs. Component C also includes the provision of large-type paper-based tests, Braille, and translated editions in Arabic, Bengali, Chinese (Simplified), Chinese (Traditional), Haitian Creole, Korean, Russian, and Spanish for the Grades 3-8 Mathematics Tests for those students requiring these accommodations.

Component 2 services encompass the provision of the same computer-based testing platform to administer the NYS Grades 5 and 8 Science Tests as is used for the Grades 3-8 ELA and Math Tests, including accommodated test forms and translated editions in Chinese (Simplified), Haitian Creole, Korean, Russian, and Spanish. As for ELA and Math, the computer-based testing platform for Grades 5 and 8 Science must include a test delivery system, administration system for both the Stand-alone Field Tests (SAFT) and operational tests and a scoring platform for the operational tests.

The following table provides the tentative test administration schedule for both operational and stand-alone field tests for each of the five years of the contract resulting from this RFP. This table includes information on the new contractor’s responsibility with regard to construction of tests using items developed by the previous contractor, as well as inclusion of new items on the Grades 3-8 ELA and Math Tests developed as part of the services required under Component 1a and provision of a computer-based testing platform for the NYSED-developed Grades 5 and 8 Science Written Tests.

7

RFP #20-004

Tentative Test Administration Schedule

Contract Year Month of Test Administration

Test Administered How Developed Contractor’s Responsibility for

Test DeliveryYear 1July 1, 2020 – June 30, 2021

March/April 2021

Grades 3-8 ELAOperational Tests

The contractor constructs operational tests drawing from bank of items developed and field tested by previous contractor. Test will also include embedded FT items developed by the previous contractor.

CBT and PBT

April/May 2021 Grades 3-8 MathOperational Tests

The contractor constructs operational tests drawing from bank of items developed and field tested by previous contractor. Test will also include embedded FT items developed by previous contractor.

CBT and PBT

May/June 2021 Grades 3-8 ELA and Math SAFTs

The contractor constructs ELA and Math SAFTs drawing from bank of items developed by previous contractor.

CBT and PBT

May/June 2021 Component 2: Grades 5 & 8 Science SAFT

NYSED develops SAFT forms; contractor provides CBT platform.

CBT

8

RFP #20-004

Contract Year Month of Test Administration Test Administered How Developed

Contractor’s Responsibility for Test Delivery

Year 2July 1, 2021 – June 30, 2022

March/April 2022


The contractor constructs operational tests drawing from bank of items originally developed by previous contractor and field tested in and prior to Spring 2021;Exception: contractor will embed field test items that it developed.

CBT and PBT


The contractor constructs operational tests drawing from bank of items originally developed by previous contractor and field tested in Spring 2021;Exception: contractor will embed field test items that it developed.

CBT and PBT


The contractor develops items and constructs forms for the Grades 3-8 ELA and Math SAFTs.

CBT and PBT



CBT

June 2022 Component 2: Grades 5 & 8 ScienceOperational tests

NYSED develops operational test forms; contractor provides CBT platform.

CBT

Contract Year Month of Test Administration

Test Administered How Developed Contractor’s Responsibility for

9

RFP #20-004

Test DeliveryYear 3July 1, 2022 – June 30, 2023

March/April 2023


The contractor constructs operational tests using items field tested in and prior to Spring 2022 as well as some items drawn from the bank originally developed by previous contractor.Contractor also develops embedded FT items.

CBT and PBT


The contractor constructs operational tests using items field tested in and prior to Spring 2022 as well as some items drawn from the bank originally developed by previous contractor.Contractor also develops embedded FT items.

CBT and PBT

May/June 2023 Grades 3-8 ELA and MathSAFTs


CBT and PBT



CBT

June 2023 Component 2: Grades 5 & 8 Science Operational Tests


CBT

10

RFP #20-004



Year 4July 1, 2023 – June 30, 2024

March/April 2024

ELA -- Develop Operational Tests

The contractor constructs operational test forms using items field tested in and prior to Spring 2023, as well as some items originally developed by the previous contractor.Contractor also develops embedded FT items.

CBT and PBT

April/May 2024 Math – Operational Tests The contractor constructs operational test forms using items field tested in and prior to Spring 2023, as well as some items originally developed by the previous contractor.Contractor also develops embedded FT items.

CBT and PBT



CBT and PBT

May/June 2024 Component 2: Grades 5 & 8 Science – SAFT


CBT

June 2024 Component 2: Grades 5 & 8 Science – Operational tests


CBT

11



Year 5July 1, 2024 – November 30, 2025

March/April 2025

ELA – Operational Tests The contractor constructs operational test forms using items field tested in and prior to Spring 2024;Contractor also develops embedded FT items.

CBT and PBT

April/May2025

Math – Operational Tests

The contractor constructs operational test forms using items field tested in and prior to Spring 2024; the contractor also develops embedded FT items.

CBT and PBT

May/June2025

Grades 3-8 ELA and Math SAFTs


CBT and PBT

May/June2025

Component 2: Grades 5 & 8 Science – SAFT


CBT

June2025

Component 2: Grades 5 & 8 Science – Operational Tests


CBT

RFP #20-004

Component 1a: Test Development Requirements for Grades 3-8 English Language Arts and Mathematics

The test development process and related outcomes are governed by certain requirements. While further details about these requirements are provided throughout this section and in Attachments B, C, D, and E, NYSED requires that:

1. All operational tests adhere to the design specifications (test designs, blueprints, embedded field test designs, linking designs, etc.)

2. All test content adheres to content specifications, including:

a. Adherence to the Passage Review Criteria, and

b. Adherence to the specific Item Criteria.

3. All items are written by educators at a series of item writer workshops held and facilitated by the contractor at locations around the State. At the item writing workshops, the contractor facilitates educators to draft items that meet expectations set forth in item criteria documents and Universal Design (UD) guidelines.

4. The Grades 3-8 Test Development Process is followed or enhanced to ensure:a. Adherence to principles for Universal Design (UD) for Learning Guidelines

b. Conformance with the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and the National Council of Educational Measurement in Education, 2014)

5. As part of the test development process, the contractor will conduct cognitive labs with NYS students to provide validation of test item types. Results of these labs must be documented to provide evidence that:

a. Test items elicit the intended response processes appropriate for each grade level as represented in the NYS P-12 Next Generation Learning Standards;

b. Test items require complex demonstrations or applications of knowledge and skills; and

c. Ancillary constructs needed for success on the assessments do not provide inappropriate barriers to measuring the achievement of all students.

Cognitive labs will be conducted for each of the Grades 3-8 ELA and Math Tests for the purpose of content validation and peer review only once during the contract resulting from this RFP, at such time as specified by NYSED.

6. All documentation stipulated by the test development process is complete and accurate.

7. All documentation is maintained by contractor and available to NYSED in a digitally searchable format.

8. NYSED will approve, reject, or recommend for revision all materials where indicated in the development process.

9. To the fullest extent feasible, the contractor must make changes requested by NYSED prior to:12

RFP #20-004

a. Internal Item Review,

b. Educator Review,

c. Field Test Forms Finalization,

d. Operational Forms Publication.

10.All expectations for item banking are met.

11.Contractor meets or exceeds minimum expectations for staffing as specified in the “Staffing Requirements” section of this RFP.

12.All items on the field tests and operational tests become the sole property of New York State. In addition, the contractor shall reproduce, use, display, and include copies of NYSED’s trademarks, trade name, logos, copyrights and other intellectual property (collectively, the “Marks”) on all copies of materials produced for NYSED. The contractor acknowledges that the Marks are owned solely and exclusively by NYSED, and nothing contained in the resulting contract shall give the contractor any ownership right or interest in such Marks.

13.All materials must be provided to NYSED error free and proofed by the contractor’s publications team.

English Language Arts and Mathematics Test Designs

At this time, NYSED is working with NYS educators to revise the test designs to measure the English Language Arts and Mathematics Next Generation Learning Standards. These new test designs will first be implemented with the Spring 2021 Operational Tests.

Future ELA and Math test designs are expected to be of similar test length but may include a slightly different mix of passages, multiple-choice, and constructed-response questions. For ELA, one such possible change that is currently under consideration is to have a limited number of passages that have both multiple-choice and short response questions associated with them. For math, NYSED is exploring including 1-point constructed-response items in the design. However, as stated above, such changes would not result in an appreciable difference in test length, and it is anticipated that the test design will continue to include two sessions and similar numbers of items.

For informational purposes, the current test designs are detailed in the following pages.

Grades 3-8 English Language Arts

Current English Language Arts Test Design(Through Spring 2020)

Test Design: Each operational Grade 3–8 ELA Test is currently made up of two test sessions. The two test sessions are usually administered on two successive school days.

English Language Arts Test DesignThe following charts provide a description of the current test design for grade bands.

13

RFP #20-004

Grades 3-4 English Language Arts Test Design

Session 1 Session 2 Total

Reading Writing

Passages 4 3 7

Multiple-Choice Questions 24* 0 24

Short-Response Questions 0 6 6

Extended-Response Questions 0 1 1

Total Number of Literary Passages 3-4

Total Number of Informational Passages 3-4*Session 1 includes 6 embedded MC questions for Grade 3 and includes 7 embedded MC items for Grade 4. The embedded MC questions do not count toward student scores.



Reading WritingPassages 5 3 8





Total Number of Informational Passages 3-5

*Session 1 includes 7 embedded MC questions. The embedded MC questions do not count toward student scores.



Reading Writing

Passages 5 3 8





Total Number of Informational Passages 3-5

*Session 1 includes 7 embedded MC questions. The embedded MC questions do not count

14

RFP #20-004

toward student scores.Passages on ELA TestsAll questions on NYS ELA tests are passage-based. Passages on NYS ELA tests must meet the specific requirements set forth in Passage Selection Guidelines for Assessing ELA and Passage Review Criteria (Attachment G). Additionally, passages on ELA Tests must:

1. Be authentic and come from reputable sources. Authentic published literature used on each ELA test must come from a variety of publications, books and magazines. Unless in the public domain, these pieces may only be abridged from longer pieces if the permission of the author is obtained prior to the items related to that passage being field-tested.

2. Be of general interest

3. Sufficiently represent the diversity that exists within NY

4. Only include illustrations, photographs, or charts if used to measure specific skills

In developing passages for approval by NYSED, the contractor must adhere to NYSED’s Passage Review Criteria for each passage. General guidelines for the selection of passages is provided in Attachment G. The comprehensive requirements will be provided upon award of the contract. The contractor will provide documentation for each passage selected that shows:

1. Results of three quantitative analyses to justify grade-appropriate complexity

2. Results of qualitative text complexity review (using appropriate text complexity rubric) to justify the level of complexity

3. Explanation of how the passage meets each criterion

4. Justification for inclusion on the NYS ELA test as a passage that is worth reading and supports measurement of particular standards

NYSED staff will review and approve passages prior to their being reviewed by teachers and will have the final authority over passages selected for use on the tests.

REQUESTS FOR PERMISSION

Requests for Permission obtained by the contractor from copyright holders to use their copyrighted works and materials on any NYSED ELA operational tests administered under this contract must include explicit permission from the author for NYSED to use the copyrighted work in NYS operational tests, whether or not developed by the contractor. The written agreement between the contractor and the copyright holders, for works used in field tests and operational tests, must expressly state that NYSED is being granted permission to post the test containing the copyrighted work on the NYSED web site for no less than three years following the administration of the operational test.

The contractor must obtain permission of the author or copyright holder for use of copyrighted passages prior to associated items being submitted to NYSED for review. If the contractor cannot obtain the copyright holder’s approval to use the copyrighted work on NYSED’s operational tests, the passage and all associated items cannot be used for any purpose, including, but not limited to field tests and operational tests.

15

RFP #20-004

External Test Blueprints

The charts below show the current percentage of points that relate to Reading, Language, and Writing Standards. When reading these charts, it is essential to remember that most questions assess many standards simultaneously. Additionally, Reading Standards are divided by focus (Key Ideas, Craft and Structure, and Integration of Knowledge) to help guide instruction.

Grades 3, 4, and 5 ELA TestsArea of Focus Approximate Percentage of Points

Reading Standards 100% of points require close readingLanguage and Writing Standards Up to 50% of points require writing and command of

languageApproximate Percent of Reading Points

Key Ideas and Details Craft and Structure Integration of Knowledge and Ideas

Up to 65% Up to 35% Up to 30%

Grades 6, 7, and 8 ELA Tests

Area of Focus Approximate Percentage of Points

Reading Standards 100% of points require close readingLanguage and Writing Standards Up to 40% of points require writing and command of

languageApproximate Percent of Reading Points

Key Ideas and Details Craft and Structure Integration of Knowledge and Ideas

Up to 60% Up to 40% Up to 40%

It should be noted that Standards RL3.1, RI3.1, RL4.1, RI4.1, etc., provide a foundation for all questions on the tests, as all will require text-based responses. Likewise, Standards RL3.10, RI3.10, RL4.10, etc., form the heart of all text-based instruction. While not assessed directly in questions, RL3.10, RI3.10, RL4.10, etc., are present on the test in the form of rigorous, worthwhile texts.

Items on ELA Tests

QUESTION FORMATS

The ELA Tests currently contain multiple-choice (1 point), short-response (2 point), and extended-response (4 point) questions. For multiple-choice questions, students select the correct response from four answer choices, labeled A, B, C, and D. For short- and extended-response questions, students write an answer to an open-ended question. The short and extended responses are scored according to the rubrics shown in the Educator Guides.

Multiple-Choice Questions

Multiple-choice questions are designed to assess Reading and Language Standards. They will ask students to analyze different aspects of a given text, including central idea, style elements, character and plot development, and vocabulary. Almost all questions, including vocabulary questions, will only

16

https://www.engageny.org/resource/test-guides-english-language-arts-and-mathematics

RFP #20-004

be answered correctly if the student comprehends and makes use of the whole passage. For multiple-choice questions, students will select the correct response from four answer choices.

Multiple-choice questions will assess Reading and Language Standards in a range of ways. Some will ask students to analyze aspects of text or vocabulary. Many questions will require students to combine skills. For example, questions may ask students to identify a segment of text that best supports the central idea. To answer correctly, a student must first comprehend the central idea and then show understanding of how that idea is supported. Questions will require more than rote recall or identification. Students will also be required to negotiate plausible, text-based distractors. 1 Each distractor will require students to comprehend the whole passage.

Writing from Sources

For each passage, students will apply the skills and knowledge gained in the classroom by answering reading comprehension questions that demonstrate their understanding of the passages. Each passage will give the student all of the information that is needed to successfully answer the attached questions without using their background knowledge of the subject. Higher order thinking skills are required on some questions that accompany reading passages, including summarizing, analyzing, inferring, interpreting, predicting, and evaluating information.

Tests at every grade level will contain both literary and informational reading passages. Literary passages may include short stories, folk tales, poetry, or other forms of literary writing; informational passages may include articles, excerpts from biographies or autobiographies, essays, or other forms of informational writing. A minimum of five genres must be used on each test for interest and literary diversity.

Short-Response Questions

Short-response questions are designed to assess Reading and Language Standards. These are single questions in which students use textual evidence to support their own answers to an inferential question. These questions ask the student to make an inference (a claim, position, or conclusion) based on their analysis of the passage, state it in their own words, and then provide two pieces of text-based evidence to support their answer.

The purpose of the short-response questions is to assess a student’s ability to comprehend and analyze text. In responding to these questions, students will be expected to write in complete sentences. Responses should typically require no more than three or four complete sentences. It is important to note that students who answer the question only using details from the text will NOT receive full credit. A full-credit response is characterized by both a valid inference and sufficient textual support.

Extended-Response Questions

Extended-response questions are designed to assess Writing from Sources. They will focus primarily on Writing Standards. In Grade 3, extended-response questions will require comprehension and analysis of an individual text. In Grades 4-8, extended-response questions will require students to read and analyze paired texts. Paired texts are related by theme, genre, tone, time period, or other 1 A distractor is an incorrect response that may appear to be a plausible correct response to a student who has not mastered the skill or concept being tested.17

RFP #20-004

characteristics. Many extended-response questions will ask students to express a position and support it with text-based details. Extended-response questions allow students to demonstrate their ability to write a coherent essay using textual evidence to support their ideas.

Student responses will be evaluated based on Writing Standards and a student’s command of evidence to defend their point.

English Language Arts Rubrics

Both the English Language Arts 2-Point and 4-Point Rubrics reflect the demands called for by the Learning Standards. The rubrics currently used to score the short- and extended-response questions are available in the 2019 English Language Arts Educator Guide.

Item Specifications

All items on NYS ELA tests are passage-based. Items on NYS ELA tests must meet the specific requirements provided by NYSED upon award of the contract. An abbreviated set of requirements is provided in Attachment H.

Additionally:

1. All operational test items must be unique items taken from the existing and growing bank of items developed for this testing program. No item may have appeared on a previously administered test in another state unless NYSED has given specific approval for its inclusion.

2. Items must be written by NYS teachers who are certified in the content area. The new contractor will organize and facilitate item writing workshops around the State.

3. Items must be designed to meet targets set in Test Blueprints.

4. Items must be tagged to:

a. Reading or Writing Standardsb. Primary, secondary, and tertiary standards within eachc. Targeted performance level (PL)d. Item type/Representation.

5. Items must measure tagged standards at the identified performance level in ways consistent with the item specifications.

6. The contractor must develop at least ten multiple-choice items per passage, so that operational tests may be constructed using items that have been found appropriate through NYSED and educator committee review and have been successfully field tested.

7. Items and item types must reflect the best educational research available. Items must match the cognitive demands defined for each standard at the intended performance level.

8. Items must be clear to the reader.

9. Constructed-response items must clearly define the expectations for the students, such as the necessity of providing details from a story or other piece of literature as support.

10. Items on the assessment must be text-dependent questions that require close attention to the text and no information or evidence from outside the text. A text-dependent question establishes what follows and what does not follow from the text itself.

18

https://www.engageny.org/resource/test-guides-english-language-arts-and-mathematics

RFP #20-004

Grades 3-8 MathematicsCurrent Mathematics Test Design

(Through Spring 2020)

Mathematics Test Design

Test Design: Each operational Grades 3-8 Mathematics Test currently consists of two test sessions. The two test sessions are usually administered on two successive school days.

The following charts provide a description of the current test design for each grade.

Grade 3 Test Design

Session

Number of Multiple-Choice

Questions(1 credit)

Number of Short ResponseQuestions(2 credits)

Number of Extended-ResponseQuestions(3 credits)

Total Number of Questions

1 25* 0 0 252 8 6 1 15

Total 33 6 1 40*Session 1 includes 7 embedded MC questions.

Grade 4 Test Design

Session


Questions(1 credit)

Number of Short Response Questions(2 credits)

Number of Extended-Response Questions(3 credits)


1 30* 0 0 302 8 6 1 15


Grade 5 Test Design

Session


Questions(1 credit)




1 30* 0 0 302 8 6 1 15


19

RFP #20-004

Grade 6 Test Design

Session


Questions(1 credit)




1 31* 0 0 312 7 7 1 15


Grade 7 Test Design

Session


Questions(1 credit)




1 33* 0 0 332 7 7 1 15


Grade 8 Test Design

Session


Questions(1 credit)




1 33* 0 0 332 7 7 1 15


Items on Mathematics Tests

QUESTION FORMATS FOR MATHEMATICS TESTS

The 2019 Grades 3–8 Mathematics Tests contain multiple-choice (1-point), short-response (2-point), and extended-response (3-point) questions. For multiple-choice questions, students select the correct response from four answer choices. For short- and extended-response questions, students write an answer to an open-ended question and may be required to show their work. In some cases, they may be required to explain, in words, how they arrived at their answers. Some test questions target more than one standard or assesses an entire cluster. As such, many individual test questions assess September-to-April standards in conjunction with May-to-June standards from past grades.

20

RFP #20-004

Multiple-Choice Questions

Multiple-choice questions are designed to assess learning standards for Mathematics. Mathematics multiple-choice questions will mainly be used to assess standard algorithms and conceptual standards. Multiple-choice questions incorporate both Standards and Standards for Mathematical Practices, some in real-world applications. Many multiple-choice questions require students to complete multiple steps. Likewise, some of these questions are linked to more than one standard, drawing on the simultaneous application of multiple skills and concepts. Within answer choices, distractors will all be based on plausible missteps. For all multiple-choice questions, students select the correct response from four answer choices, labeled A, B, C, and D.

Short-Response Questions

Short-response questions ask students to complete a task and show their work. Like multiple-choice questions, short-response questions will often require multiple steps, the application of multiple mathematics skills, and real-world applications. Many of the short-response questions will cover conceptual and application standards.

Extended-Response Questions

Extended-response questions ask students to show their work in completing two or more tasks or a more extensive problem. Extended-response questions allow students to show their understanding of mathematical procedures, conceptual understanding, and application. Extended-response questions may also assess student reasoning and the ability to critique the arguments of others.

Item Specifications

All items on NYS Mathematics Tests must meet the specific requirements for potential Grades 3-8 Math questions which will be provided by NYSED upon award of the contract. An abbreviated set of requirements is provided in Attachment H.

Additionally:

1. All operational test items must be unique items taken from the existing and growing bank of items developed for this testing program. No item may have appeared on a previously administered test in another state unless NYSED has given specific approval for its inclusion.

2. Items must be designed to meet targets set in Test Blueprints.

3. Items must be tagged to:

Primary, secondary, and tertiary standards or cluster

Targeted PL

Procedural, Conceptual, or Applied Focus

Item type/Representation

All meta-data specified for item bank.

4. Items must measure tagged standards at the identified performance level in ways consistent with the item specifications.

21

RFP #20-004

5. All items are written by educators at a series of item writer workshops held and facilitated by the contractor at locations around the state. At the item writing workshops, the contractor facilitates educators to draft items that meet expectations set forth in item criteria documents and Universal Design (UD) guidelines.

6. Items and item types must reflect the best educational research available. Items must match the cognitive demands defined for each standard at the intended performance level.

7. Items must be clear to the reader.

8. Math items must show evidence of review by grade-level math teachers.

9. Constructed-response items must clearly define the expectations for the students.

Sequencing

Currently, standards designated as September-to-April will be assessed on that grade level’s test; likewise, standards designated for the May-to-June instructional period will not be assessed on that grade level’s test but could be assessed on the next grade level’s test.

More information about the current September to April/May to June learning standards can be found in the Grades 3–8 Mathematics Testing Program Guidance: September-to-April/May-to-June Learning Standards.

Content Emphases and Sequencing

The current NYS learning standards for math were designed with the understanding that not all clusters should be emphasized equally in instruction or assessment. Some clusters require greater emphasis than others based on the time that they take to master and/or their importance to future mathematics or the demands of college and career readiness. The charts below detail the recommended instructional focus and the percentage of test questions that currently assess what had been designated as Major, Supporting, and Additional Clusters. These charts also illustrate the different clusters and standards currently recommended for instructional emphasis. Standards that are recommended for greater emphasis are indicated with a check mark while those that are recommended for instruction after the administration of the spring operational tests are indicated by the word “Post.”

22

http://www.p12.nysed.gov/assessment/ei/2014/math-sept-april-may-june.pdf

http://www.p12.nysed.gov/assessment/ei/2014/math-sept-april-may-june.pdf

RFP #20-004

= Standards recommended for greater emphasisPost = Standards recommended for instruction in May-June

Grade 3 Math Test

Cluster Emphasis Domain Cluster Standard

Major Clusters

Operations and Algebraic Thinking

Represent and solve problems involving multiplication

and division.

3.OA.13.OA.23.OA.3

3.OA.4Understand the properties of

multiplication and the relationship between multiplication and

division.

3.OA.5

3.OA.6

Multiply and divide within 100. 3.OA.7Solve problems involving the four

operations and identify and explain patterns in arithmetic.

3.OA.8

3.OA.9

Number and Operations –

Fractions

Develop understanding of fractions as numbers.

3.NF.13.NF.23.NF.3

Measurement and Data

Solve problems involving measurement and estimation of intervals of time, liquid volumes,

and masses of objects.

3.MD.1

3.MD.2

Geometric measurement: understand concepts of area and relate area to multiplication and

to addition.

3.MD.53.MD.63.MD.7

Supporting Clusters

Measurement and Data Represent and interpret data.

3.MD.33.MD.4 Post

Geometry Reason with shapes and their attributes.

3.G.1 Post3.G.2

Additional Clusters

Number and Operations in

Base Ten

Use place value understanding and properties of operations to perform multi-digit arithmetic.

3.NBT.13.NBT.23.NBT.3


Geometric measurement: recognize perimeter as an

attribute of plane figures and distinguish between linear and

area measures.

3.MD.8 Post

23

RFP #20-004

Grade 4 Math Test

Cluster Emphasis Domain Cluster Standard

Major Clusters


Use the four operations with whole numbers to

solve problems.

4.OA.14.OA.2 4.OA.3


Base Ten

Generalize place value understanding for multi-digit

whole numbers.

4.NBT.14.NBT.24.NBT.3

Use place value understanding and properties of operations to perform multi-digit arithmetic.

4.NBT.44.NBT.5

4.NBT.6

Number and Operations –

Fractions

Extend understanding of fraction equivalence and ordering.

4.NF.14.NF.2

Build fractions from unit fractions by applying and extending previous understandings of

operations on whole numbers.

4.NF.3

4.NF.4

Understand decimal notation for fractions and compare

decimal fractions.

4.NF.5 Post4.NF.6 Post4.NF.7 Post

Supporting Clusters


Gain familiarity with factors and multiples. 4.OA.4


Solve problems involving measurements and conversion of measurements from a larger

unit to a smaller unit.

4.MD.1 Post

4.MD.2 Post

4.MD.3

Represent and interpret data. 4.MD.4

Additional Clusters


Generate and analyze patterns. 4.OA.5


Geometric measurement: understand concepts of angles

and measure angles.

4.MD.54.MD.64.MD.7

Geometry

Draw and identify lines and angles and classify shapes

by properties of their lines and angles.

4.G.14.G.2

4.G.3

24

RFP #20-004

Grade 5 Math TestCluster

EmphasisDomain Cluster Standard

Major Clusters


Base Ten

Understand the place value system.

5.NBT.15.NBT.25.NBT.35.NBT.4

Perform operations with multi-digit whole numbers and with

decimals to hundredths.

5.NBT.5 5.NBT.6 5.NBT.7

Number and Operations–

Fractions

Use equivalent fractions as a strategy to add and subtract

fractions.

5.NF.1

5.NF.2

Apply and extend previous understandings of

multiplication and division to multiply and divide fractions.

5.NF.35.NF.45.NF.55.NF.6 5.NF.7


Geometric measurement: understand concepts of

volume and relate volume to multiplication and to addition.

5.MD.3

5.MD.4

5.MD.5

Supporting Clusters


Convert like measurement units within a given

measurement system.5.MD.1

Represent and interpret data. 5.MD.2

Additional Clusters

Operations and Algebraic

Thinking

Write and interpret numerical expressions.

5.OA.1

5.OA.2

Analyze patterns and relationships. 5.OA.3 Post

Geometry

Graph points on the coordinate plane to solve.

5.G.1 Post5.G.2 Post

Classify two-dimensional figures into categories based

on their properties.

5.G.3

5.G.4

25

RFP #20-004



Major Clusters

Ratios and Proportional

Relationships

Understand ratio concepts and use ratio reasoning to

solve problems.

6.RP.16.RP.26.RP.3

The Number System

Apply and extend previous understandings of

multiplication and division to divide fractions

by fractions.

6.NS.1

Apply and extend previous understandings of numbers to

the system of rational numbers.

6.NS.56.NS.66.NS.76.NS.8

Expressions and Equations

Apply and extend previous understandings of arithmetic to

algebraic expressions.

6.EE.16.EE.26.EE.36.EE.4

Reason about and solveone-variable equations

and inequalities.

6.EE.56.EE.66.EE.7 6.EE.8

Represent and analyze quantitative relationships between dependent and independent variables.

6.EE.9

Supporting Clusters Geometry

Solve real-world and mathematical problems

involving area, surface area, and volume.

6.G.1

6.G.2

6.G.3

6.G.4

Additional Clusters

The Number System

Compute fluently with multi-digit numbers and find

common factors and multiples.

6.NS.2

6.NS.36.NS.4

Statistics and Probability

Develop understanding of statistical variability.

6.SP.1 Post

6.SP.2 Post

6.SP.3 Post

Summarize and describe distributions.

6.SP.4 Post

6.SP.5 Post

26

RFP #20-004



Major Clusters


Relationships

Analyze proportional relationships and use them to solve real-world

and mathematical problems.

7.RP.1

7.RP.2

7.RP.3

The Number System

Apply and extend previous understandings of operations with fractions to add, subtract, multiply,

and divide rational numbers.

7.NS.1

7.NS.2

7.NS.3


Use properties of operations to generate equivalent expressions.

7.EE.1

7.EE.2

Solve real-life and mathematical problems using numerical and

algebraic expressions and equations.

7.EE.3

7.EE.4a7.EE.4b

Supporting Clusters


Use random sampling to draw inferences about a population.

7.SP.1

7.SP.2

Investigate chance processes and develop, use, and evaluate

probability models.

7.SP.5

7.SP.67.SP.7

7.SP.8

Additional Clusters

Geometry

Draw, construct, and describe geometrical figures and describe the relationships between them.

7.G.1

7.G.2 Post

7.G.3 Post

Solve real-life and mathematical problems involving angle measure,

area, surface area, and volume.

7.G.4

7.G.5 Post7.G.6 Post


Draw informal comparative inferences about two populations.

7.SP.3

7.SP.4

27

RFP #20-004


Emphasis Domain Cluster Standard

Major Clusters


Work with radicals and integer exponents.

8.EE.18.EE.2 Post8.EE.38.EE.4

Understand the connections between proportional

relationships, lines, and linear equations.

8.EE.5

8.EE.6Analyze and solve linear equations and pairs of

simultaneous linear equations.

8.EE.7

8.EE.8

Functions

Define, evaluate, and compare functions.

8.F.18.F.28.F.3

Use functions to model relationships

between quantities.

8.F.4 8.F.5

Geometry

Understand and apply the Pythagorean Theorem.

8.G.6 Post8.G.7 Post8.G.8 Post

Understand congruence and similarity

using physical models, transparencies, or geometry software.

8.G.18.G.28.G.38.G.48.G.5

SupportingClusters

Number SystemKnow that there are numbers that are not rational and approximate

them by rational numbers.

8.NS.1 Post

8.NS.2 Post

Statistics andProbability

Investigate patterns of association in bivariate data.

8.SP.18.SP.28.SP.38.SP.4

Additional Clusters Geometry

Solve real-world and mathematical problems

involving volume of cylinders, cones, and spheres.

8.G.9

28

RFP #20-004

The tables below illustrate the current domain-level and cluster-level test blueprints.

Domain-Level Test Blueprint—Percent of Test Points on Grade 3 Math Test

Number and Operations in Base

Tens

Number and Operations-

Fractions


Measurement and Data Geometry

5–15% 15–25% 40–50% 15–25% 5–15%

Cluster-Emphasis Test Blueprint—Percent of Test Points on Grade 3 Math Test

Major Clusters Supporting Clusters Additional Clusters

70–80% 10–20% 5–10%



Tens


Fractions



20–30% 20–30% 15–25% 15–25% 5–15%



70–80% 10–20% 5–10%



Tens


Fractions



20–30% 30–40% 5–15% 20–30% 5–15%



70–80% 10–20% 5–10%


The Number System



RelationshipsGeometry Statistics and

Probability

15–25% 35–45% 20–30% 10–20% 0%



70–80% 10–20% 5–10%


29

RFP #20-004


Relationships

The Number System

Expressions and Equations Geometry Statistics and

Probability

20–30% 15–25% 30–40% 5–15% 10–20%


Major Clusters Supporting Clusters

Additional Clusters

70–80% 10–20% 5–10%


The Number System

Expressions and Equations Functions Geometry Statistics and

Probability

0% 40–45% 25–30% 20–25% 10–15%



70–80% 10–20% 5–10%

Internal Test Blueprints for ELA and Mathematics

In addition to the external ELA and Math test blueprints shown on the previous pages of this RFP, NYSED has internal blueprints that include further details concerning how test items are to be written and test forms are to be constructed. Most notably, the internal test blueprints identify the target number of points for which items should measure each standard and in some cases the type of item (e.g. multiple choice) that is to be used in measuring a standard. These internal blueprints will be shared with the successful contractor. In addition, NYSED is presently developing both external and internal blueprints for the assessments measuring the recently adopted Next Generation NYS ELA and Mathematics Standards. These blueprints will ensure that Evidence Based Design is integral to the development of all future test forms. For example, NYSED intends to specify the performance level(s) at which each newly developed item should be targeted in measuring a specific standard. The contractor is expected to work collaboratively with NYSED in completing the internal test blueprints that will guide the development of future test items and forms. This will ensure comparability in test forms from year to year and internal consistency in the test construction across New York State’s K-12 assessment program.

Test Development Process for ELA and Mathematics

The contractor will develop the items and test forms in accordance with Attachments B, C, and D, which detail the Grades 3-8 Test Development Process. These attachments explain each event in the test development process by defining:

Outcomes

30

RFP #20-004

Responsibilities

High Level Process

Decision Making Authority

Quality Control/Assurance

Necessary Documentation

Necessary Metrics

Presented below is each of the sequential test development events that the contractor is required to adhere to:

Sequence of Test Development EventsMajor Events Process Step Time Frame

Fiel

d Te

st B

ank

Dev

elop

men

t

Passage Development(ELA Only)

1. Passage Selection Summer2. Content Review* Early Fall3. Educator Passage Review Meeting Late Fall

Item Development

4. Regional Item Writing Workshops* Winter5. Item Editing Winter6. NYSED Item Pre-Review Late Winter/Early Spring7. Internal Item Review* Early Summer8. First Copy Edit/Content QC Early Summer9. External Quality Control* Early Summer10. Educator Item Review Meeting Summer11. Reconciliation Summer

Field Test Form Construction

12. Formal Copy Edit Early Fall13. Stand-alone Field Test Form Construction Fall – Late Fall14. Field Test Form NYSED-Review

Rangefinding15. Rangefinding Meeting Mid-July16. Rangefinding Reconciliation Mid-July17. SAFT CR Scoring Late August

Ope

ratio

nal F

orm

s D

evel

opm

ent

Forms Construction/Form

Finalization

18. First Draw (Contractor) Late Summer19. Form Pre-Review (NYSED) Early Fall20. Content Review/Risk Assessment* October21. Forms Construction Meeting November22. Final Eyes Meeting/Reconciliation November23. Form Copy Edit 1 November24. Form Copy Edit 2 December25. Final Executive Review December

Scoring Ancillaries

26. Proposed Operational Scoring Materials Sets Late Fall27. NYSED Review of Scoring Materials Sets Late Fall28. Scoring Copy Edit and Senior Management

Approval Early Winter

Item Release 29. Preparation of Selected Items for Public Release JuneAll Activities Requiring

Educatorsa. Educator Committee Selection Ongoing

* These process steps require the contractor to procure external content expert review. See the section of this RFP, “External Staffing Requirement” for further details.

A full description of the test development process is presented in Attachments B , C, and D .

31

RFP #20-004

Educator Committees

NYSED requires the contractor to involve New York State educators in many aspects of the testing program. These events are detailed in Attachment E: Educator Committees and in Attachments B, C, and D: Grades 3-8 Test Development Process.

All activities involving New York State educators, with the exception of the Regional Item Writing Workshops, must be held within a radius of 30 miles of the NYSED building at 89 Washington Avenue in Albany.

NYSED has a database of selected educators that it will provide to the contractor to use to contact educators and determine their willingness and availability for participation in educator committees. New names will be added by NYSED. Committees must be composed of educators from all parts of the State, be diverse in terms of race, gender, and the needs/resource capacities of their schools, and be composed of a mixture of both experienced and new reviewers. A waiting list of educators for each event must be established by the contractor in case of last-minute problems with attendance.

For all educator committees, except as time constraints in the first year of the contract may preclude, the contractor must send out written invitations to educators to participate at least 40 days in advance to allow sufficient time for educators to plan accordingly, especially during the school year.

For all educator committee meetings, the contractor is required to:

1. Convene and conduct meetings;2. Arrange and pay for the meeting space;

3. Pay an honorarium of $200 per 7.5-hour work day;4. With the exception of the regional item writing workshops, pay for educator participant travel

expenses at State-approved rates. Educator participants in the regional item writing workshops are responsible for their own travel expenses;

5. With the exception of the regional item writing workshops, pay for educator participant lodging expenses and dinner allowance(s) at State rates for those educator participants residing 50 or more miles from the meeting location. All educator participants in the regional item writing workshops are responsible for their own lodging/dinner if needed;

6. Contact the educators recommended by NYSED to determine willingness and availability to participate on the committee and to inform them of the logistics around participation. All contractor correspondence with committee members must be provided to NYSED for review and approval prior to dissemination by the contractor;

7. Provide meals during the meeting hours for all participants;

8. Notify NYSED in advance of all committee meetings and provide all committees with the information needed to analyze and review assessment objectives and items;

9. Record and summarize the recommendations of these committees per the specifications in Attachments B, C and D, or other agreed-upon requirements, including the result of those recommendations, and report this information in writing to NYSED;

10.Provide NYSED-approved evaluation forms for educators to submit near the end of each meeting providing feedback on the quality of the contractor’s session, and host and offer an online survey created by NYSED to allow for additional educator feedback;

32

RFP #20-004

11.Maintain records of all changes made to test content and be prepared to provide them to NYSED upon request.

12.Provide NYSED with educator and staff attendance sheets and copies of the evaluation forms for each meeting.

13. In arranging meeting space, allow space for a limited number of NYSED staff who will also be attending these educator committee meetings. (Please note that the contractor does not provide any honoria or travel reimbursement for the NYSED staff attending these meetings.)

External Staffing Requirement for Development of Grades 3-8 ELA and Math

Additionally, the contractor is responsible for procuring external content experts who are not employees of the contractor and who do not write items for the contractor. For each event in the test development process requiring external capacity, the contractor will facilitate identification and approval with enough time to ensure that the event can occur as planned. Those events requiring external content capacity are described in the test development process in Attachments B, C and D and the list of development steps/events for which external staffing is required is provided in Attachment F.

NYSED-approved content consultants must:

Be capable of executing the necessary work, per the test development process Have documented leadership in their content field or the ability to hire national leaders Have documented experience working on major testing programs in or outside of New York

State Use NY Item Specifications as they were intended Be approved by NYSED

Field Testing for ELA and Mathematics

The contractor will conduct field testing for ELA and mathematics in the 2020-21, 2021-22, 2022-23, 2023-24, and 2024-25 school years to assess the difficulty, fairness, and appropriateness of the items proposed for operational tests. (The contractor will also select passages and develop items for the 2025-26 field testing in ELA and mathematics but will not construct or administer the 2026 field test forms.) Field testing will be conducted through a combination of embedding field test questions in operational tests and conducting stand-alone field tests (SAFTs), administered separately from the operational tests. Nearly all SAFTs will contain only constructed-response (CR) items.

NYSED reserves the right to require the contractor to administer the SAFTs during a different month or season from that indicated in the timelines in this RFP, provided that NYSED gives the contractor a minimum of nine months’ notice.

For all field testing, the contractor must ensure that:

1. Field test items have been developed in accordance with all item and passage specifications and using the state item development process.

2. Those items that were rejected through the review process including the Final QC for Item Development have been removed and replaced as necessary.

33

RFP #20-004

3. A large enough field test item pool is developed so that the operational test forms can be created using “successfully” field-tested items, and those items found unusable for an immediate operational test can be revised and reworked in order to be used in future field test cycles, thus building an item bank for NYSED’s use.

4. The contractor develops and field tests at least 4 times the number of multiple-choice items and three times the number of constructed-response items needed per operational Mathematics test and builds a Mathematics item bank that contains 100% more items than needed each year during operational content selection to allow for the best possible selection of items.

5. For the Operational ELA test, the contractor develops and field tests at least three times as many passage items and passage sets for each grade.

6. The contractor develops, implements, and documents a strategy to avoid item cloning.

7. The contractor documents and executes a strategy to maintain the security of all field test items and test forms.

8. The contractor scores all field-tested items, verifies the results, forwards the score results to NYSED in a mutually agreed-upon format, and produces a Field-Testing Technical Report for each year of field testing.

9. The contractor retains all field test data throughout the contract and for an additional five years thereafter.

For the stand-alone field tests (SAFT), the contractor must:

1. Administer stand-alone field tests within two months of the administration of the operational examination.

2. Develop, print, and ship administrative SAFT materials, including teacher directions, School Administrator’s Manual, and directions for administering and returning the tests to the contractor.

3. Provide one field test form per grade in Braille. (See Attachment I: Specifications for Contractor/Subcontractor Transcribing Braille.)

4. For PBT SAFTs:

a) Design the field test answer sheets or use scannable field test booklets for students to record their responses.

b) Develop secure SAFT booklets.

c) Print, ship, and manage collection and return of secure field test materials, including answer sheets.

d) Provide one field test form per grade in large-type. (See Attachment J: Specifications for Large-Type.)

e) For Mathematics SAFTs at each grade level, translate one stand-alone field test form into Spanish. The contractor must be prepared to offer Spanish editions of the SAFT for Mathematics as both PBT and CBT forms. Schools are not specifically selected for populations using the Spanish version.

34

RFP #20-004

5. For CBT Field Tests:

a) Provide the CBT platform consistent with the platform provided for operational tests for the administration and delivery of the ELA and Math SAFTs.

b) Provide for delivery of a Spanish edition of one form of the Mathematics SAFT via CBT for each grade.

6. Organize field tests in mini-forms such that students can generally complete them in one 40-minute class period.

7. Provide each participating school with an easy-to-follow set of instructions that explains how to administer a field test in the grade and subject assigned to that school.

8. Provide notice to schools on each page of the field test forms and administration directions that photocopying materials and any reference to test content is strictly prohibited. The notice must read as follows:

Secure materials. All rights reserved. No part of this booklet may be reproduced or transmitted by any means. Use of these materials is expressly limited to the New York State Education Department.

9. Collect a minimum of 1,500 usable field test student responses per SAFT item. The contractor must oversample in order to achieve the target of 1,500.

10. In collaboration with NYSED, select the schools to participate in the SAFTs based on NYSED’s sample design. Field tests are administered by New York State public schools (including charter schools) and religious and independent schools.

11.Notify the selected schools in writing of the field tests. The solicitation by the contractor must be explicit as to the grade level and subject for which the school is selected. All contractor correspondence to school administrators must be provided to NYSED for review and approval prior to dissemination by the contractor.

12.Provide all schools participating in the field tests with the appropriate quantities of field test materials for that grade(s) and subject even if it appears to the contractor that more than the target number of student responses will be achieved.

13.Pay for the shipping costs for schools to return the printed field tests to the contractor.

14.Serialize all secure printed materials distributed to schools as part of the field tests. The contractor’s records and the shipping manifest must indicate the range of serial numbers shipped to schools. The contractor must also account for each individualized secure item in checking returned materials. Follow-up correspondence to schools failing to return all materials must include the serial number of the missing materials.

15.Ensure that all field tests and related materials are returned in a timely and secure manner, and notify schools repeatedly, if necessary, that do not return materials by the due date. The contractor must provide a list to NYSED of school districts, charter, and religious and independent schools that have failed to return materials within 2 weeks after the due date and again after the contractor has made a minimum of three concerted efforts to effect their return. The contractor must provide documentation to NYSED of its efforts to contact schools and have materials returned.

35

RFP #20-004

16.Perform all activities needed for the scoring of constructed-response items. NYSED must approve the contractor’s plan for rangefinding. See Attachments B, C, and D for a description of the test development process and rangefinding and the scoring process, including the scoring rubrics and training materials (practice sets and annotated anchor sets).

17.Employ only scorers with a minimum of a bachelor’s degree. For scoring ELA Tests, such scorers must have at least nine college credits in English and/or Education. For scoring Mathematics Tests, such scorers must have at least nine college credits in Mathematics and/or Education. The contractor must provide all training for scorers.

Each year, the contractor must provide evidence to NYSED that its scorers meet these qualifications.

18.Propose a plan to equate those items administered in SAFTs to the operational test scale. Field tests must be matched by student to their operational test results to permit common scaling of items to the operational scale.

19.Securely retain all student responses to constructed-response items, on paper or as scanned images, through the end of the contract, and for one year thereafter.

For all field test items embedded in operational tests, both paper-based and computer-based, the contractor must:

1. Develop and execute a strategy to embed field test items in similar positions within the test sessions each year, such that location effects are minimal while still ensuring that it is not obvious which items are part of the field test and which are operational items.

2. Provide NYSED with operational test forms, including the embedded field test items according to the timelines and specifications set forth in this RFP. See section “Operational Forms Preparation” for additional information.

3. Receive raw data files containing student responses from the Student Information Repository System (SIRS), in the same way such files are received for operational tests. (See items #2, #4, and #5 of the “Local Scoring of Operational Tests” section of the RFP for information.) The contractor must score all student responses and conduct all analyses.

For the field test questions embedded in the paper-based operational tests:

The contractor must be flexible and prepared for the tests to be administered by schools via both PBT and CBT in unknown proportions. Currently, there are multiple operational test forms containing embedded field test items for both ELA and Math that are spiraled within schools administering paper-based operational tests. As the number of schools administering the Grades 3-8 ELA and Math Tests on computer increases, the need for multiple paper-based operational forms may decrease. Depending upon the proportion of schools that choose to administer operational tests on computers, NYSED hopes to reduce the number of paper-based operational test forms needed.

For the paper-based operational tests, the contractor must translate mathematics field test items that are being embedded into one form of the operational tests into the required alternate languages. For this contract, paper-based operational tests must be translated into eight languages: Arabic, Bengali, Chinese (Simplified), Chinese (Traditional), Haitian Creole, Korean, Russian, and Spanish. All translated items must be back translated. The contractor and its subcontractor, if there is one, must refer to bilingual mathematics glossaries and past translations of Grades 3–8 Mathematics Tests, which are posted on NYSED’s 3-8 Tests website, for guidance as to NYSED’s preferred word

36


https://steinhardt.nyu.edu/metrocenter/resources/glossaries

RFP #20-004

usages, formatting, and style for all translations. The contractor must be prepared to administer translated editions as computer-based, as well as paper-based tests.

For field test items embedded in the computer-based operational tests:

The contractor should plan to embed future MC items in as many operational test forms per grade/subject as necessary to continue to prevent the need to include MC items in Stand-alone Field Tests. As with print-based operational test forms, the contractor should plan for all computer-delivered operational test forms to be spiraled within each classroom. As the number of schools participating in computer-based operational tests continues to increase, the contractor should plan for reduced need for multiple paper-based operational forms.

For the computer-based operational tests, the contractor must translate mathematics field test items that are being embedded into one form of the operational tests into the required alternate languages. For this contract, operational computer-based tests must be translated into five languages: Chinese (Simplified), Haitian Creole, Korean, Russian, and Spanish. As with paper-based tests, all translated items must be back translated. The contractor and its subcontractor, if there is one, must refer to bilingual mathematics glossaries and past translations of Grades 3–8 Mathematics Tests, which are posted on NYSED’s 3-8 Tests website, for guidance as to NYSED’s preferred word usages, formatting, and style for all translations. The contractor must be prepared to administer the Chinese (Simplified), Haitian Creole, Korean, Russian, and Spanish translated editions as both paper-based and computer-based tests. The Arabic, Bengali, and Chinese (Traditional) translated editions will be made available to schools exclusively as paper-based tests.

Field Test Sample Design

A stratified random sampling design will be used for the SAFTs. All public and charter schools with Grades 3–8 enrollment will be required to participate in the annual field tests. Since religious and independent school participation in the Grades 3-8 tests is voluntary, these schools are expected to participate in SAFTs only in the same grade(s)/subject(s) for which they choose to administer the operational tests. Each school will be asked to administer mini field test forms that can be administered in one class period (approximately 40 minutes).

Every school with Grades 3–8 enrollment will be assigned to a field test sample in one grade level in either ELA or Math, not both. All samples represent stratified random samples that are representative of New York State schools. Additionally, schools selected by NYSED to administer field tests for NYS elementary- or intermediate-level science will not be asked to administer field tests in ELA or Math.

NYSED will provide the contractor with a complete list of New York State schools and related demographic variables for sample selection. The contractor is responsible for selecting the stratified random samples. The contractor must ensure that all samples are representative of the New York State schools with regard to number of schools and students N counts by grade, distribution across the Need/Resource Capacity Categories, ethnic composition, school type, and mean scale scores in Grades 3–8 ELA and Mathematics Tests. (For more information see the 3-8 Data Release from September 2018). The sample file must be submitted to NYSED for review and approval.

37

http://www.p12.nysed.gov/irs/pressRelease/20180926/home.html

http://www.p12.nysed.gov/irs/pressRelease/20180926/home.html


https://steinhardt.nyu.edu/metrocenter/resources/glossaries

RFP #20-004

Analysis of Field Tests

The contractor will provide statistical analyses for the field test administration. All field test items must be equated to the operational scale. The analyses must include both classical item analysis and IRT statistics, but not be limited to, the following:

1. Analyses of field test participation rates;

2. Evaluation of the field test sample representativeness;

3. Analyses of classical item statistics;

4. Analyses of Item Response Theory (IRT) item parameter estimate including the item difficulty and discrimination;

5. Evaluation of the item fit and model fit statistics;

6. Evaluation of the classical and IRT Differential Item Functioning (DIF) results;

7. Overall statistical indices of field test forms reliability;

8. Reliability and validity information; and

9. Inter-rater reliability.

The Mantel-Haenszel process for selected-response questions, a standardized mean difference (SMD) process for constructed-response questions, or an IRT-based statistical process for both item types, must be used to evaluate DIF on field test items for subgroups (i.e., ethnicity, test mode, and Needs/Resource Capacity Category) and for population focal groups identified by NYSED. The minimum sample size for a focal group is 200. DIF analysis will also be conducted on the field test versions of mathematics test items field tested in English compared to Spanish, plus any other alternate language edition taken by 200 or more students. All high DIF items will be evaluated by the contractor and NYSED for suitability for inclusion in operational tests. A copy of each research file containing all field test item responses as well as the demographic information must be provided to NYSED after each field test.

Field Test Technical Report

The contractor will develop and submit to NYSED a technical report for each school year of field testing that will include, but not be limited to, the following information:

1. Field test design;

2. Field test sample and participation rate;

3. Field test and operational data match;

4. Classical analysis (item analysis, inter-rater reliability, and DIF analysis);

5. IRT Scaling (item calibration, item equating, item fit evaluation and IRT DIF); and

6. Item flag summary and research guide for item selection.

Liquidated Damages Related to Passage Selection and Item Development for ELA and Math

Because quantifying losses arising from the contractor’s failure to perform under the contract would be inherently difficult insofar as the contractor’s failure to deliver or perform its obligations may impact the State’s reputation, the contractor and State shall agree to certain liquidated damages provisions

38

RFP #20-004

throughout the contract. The contractor and the State shall agree that the agreed-upon sums are not a penalty, but rather represent reasonable measure(s) of damages, based upon the parties’ experience and given the nature of the losses that may result therefrom. The liquidated damages clauses herein are intended to cover reputational and other losses suffered by the State in the event the contractor fails to provide a deliverable or perform a duty as outlined therein.

NYSED will provide the contractor with initial feedback and allow the contractor to provide acceptable replacements of passages and/or items pursuant to the timeframes set forth herein, as needed, for the field testing due to occur in spring 2021 and thereafter. If after this has occurred, the contractor fails to provide the requisite minimum numbers of passages for ELA and/or items for ELA or Mathematics that in NYSED’s judgment meet the criteria provided in Attachments G and H, then the liquidated damages provisions contained herein shall apply. A failure to provide a passage or item can include, but shall not be limited to: delivering fewer items or passages than required; delivering passages or items which contain inadequate substance, as determined by NYSED. The requisite minimum number of passages for ELA and items for ELA and Mathematics are specified in this RFP under the heading “Field Testing for ELA and Mathematics.”

NYSED shall reduce by a specified amount the sum of the agreed upon consideration for each of the following activity line items as specified in the agreed-upon budget for the impacted field test(s) (i.e., ELA and/or Math) for the associated year:

For ELA AssessmentsObtain Passages for [relevant year] ELA Field TestsDraft Items for [relevant year] ELA Field TestsSubmit Proposed [relevant year] Field Test Passages and Items to NYSEDConstruct Stand-alone ELA Field Test Forms for [relevant year] Including Anchor Items

For Mathematics AssessmentsDraft Items for [relevant year] Math Field TestsSubmit Proposed [relevant year] Math Field Test Items to NYSEDConstruct Stand-alone Math Field Test Forms for [relevant year] Including Anchor Items

If after NYSED has provided the Contractor with initial feedback and allowed the Contractor to provide acceptable replacements, the Contractor fails to provide at least

Amount to be Reduced By

90% of the requisite minimum number of passages/items 20%80% of the requisite minimum number of passages/items 40%70% of the requisite minimum number of passages/items 60%50% of the requisite minimum number of passages/items 100%

The criteria specified in Attachments G and H are subject to revision by mutual agreement of NYSED and the contractor during the course of the contract resulting from this RFP.

Item Banking for the Grades 3-8 ELA and Math Tests

Following the administration of field tests, the contractor must compile an operational item bank (OIB) that is available to NYSED one month after field-testing. The bank will be added to with each subsequent development cycle and house all items. At minimum, the OIB must catalogue each item

39

RFP #20-004

by unique identification number (UIN), be searchable by specific tags and capable of generating cross-tabbed reports on any and all of the minimum tags. These tags include:

1. UIN

2. Grade

3. Primary, Secondary, Tertiary Focus of measurement

4. Item Type

5. Targeted Performance Level

6. Depth of Knowledge (DOK)

7. Key

8. IRT statistics

9. Year Developed

10.Year Used

Additionally, the contractor must make available to NYSED

1. Each viable, field-tested operational item (both past and present)

2. Iteration histories of each item

3. All documentation for each item

4. All metrics/evaluations for each item

At a minimum, the item banking system provided and maintained by the contractor must:

1. Be secure

2. Be searchable by tag

3. Provide printing options for NYSED

4. Be capable of pulling together mini- and full-forms (with aggregated and disaggregated meta-data)

5. Support Windows/Mac OSX platforms

6. Allow access to outside agents (external content expert consultants and agencies) approved by NYSED

7. Allow migration of items to future systems

Operational Test Administration for Grades 3-8 ELA and Math

The contractor must adhere to the Form Construction process described in Attachments B, C, and D and annually submit a timeline for each test that provides specifics on major events, phases, and deliverables. The timeline must include appropriate time for NYSED to review and approve carefully edited copies of all test items, all test administration materials having to do with both paper-based and computer-based testing and scoring materials.

40

RFP #20-004

Operational Forms Preparation

1. The contractor must provide to NYSED, for its approval, proposed updates to the style guide for the NYS Grades 3–8 ELA and Mathematics Tests delineating the content specific terminology, phrases, graphics, symbols, and grammatical style. The Style Guide must include specifications for the layout of the test for both the paper- and computer-based forms.

2. The contractor must annually update the style guide with approval from NYSED.

3. Materials to be produced include all test sessions and test booklets, including Braille, large-type, and translated editions (for mathematics only); CBT Teacher’s Directions, PBT Teacher’s Directions, and School Administrator’s Manual; scoring guides for local scoring; and any other examination-related materials necessary for valid administration. Instructions in manuals for teachers and administrators must be clear, complete, and specific. Whenever possible, directions across grade levels and within content areas should be consistent.

4. Prior to printing the PBT forms, the contractor must provide camera-ready proofs of all operational forms, as well as access to rendering of the matching operational test questions in the CBT platform. The contractor must be prepared to provide the following prepress deliverables to NYSED:

a) A high resolution (i.e., 1200 dpi or better), press-ready PDF file (i.e., all exam fonts and graphics embedded) for the final NYSED-approved version of each exam booklet and its associated scoring materials; these PDF files that are approved for printing are in grayscale with no additional colors;

b) Clean printouts of the above-mentioned PDF files, single-sided and numbered (except the cover);

c) Native layout files (either in QuarkXpress 6.5 or higher, or Adobe InDesign Creative Suite 3 or higher) of the above-mentioned PDF files; and

d) All related, native art files (either in Freehand MX or Illustrator Creative Suite 3 or higher).

5. Prior to the production of CBT forms, the contractor must provide the following deliverables to NYSED: presentation of all forms and all grades for ELA and Math, in the same secure test environment that students will test in. All features and functionalities used by students must be active in these forms for NYSED to test. This includes all tools and accommodations features provided for both students with disabilities and ELLs/MLLs.

6. Grade and subject designations, as well as New York State Testing Program must appear on all test materials. Grade appropriate graphics, content, style, typeface, and font size must be in accordance with all requirements outlined in the NYS Style Guide. Any proposed changes must be agreed upon by NYSED.

7. The contractor will provide NYSED with the materials noted in this section prior to the date of test administration according to a mutually agreed upon schedule. All products and procedures proposed will be reviewed and approved in draft and final form by NYSED, with a minimum of 20 days for review. Materials for NYSED approval will be provided by the contractor in a finished form.

41

RFP #20-004

8. The contractor must propose/construct the operational test forms based on the forms specification blueprints, field test results and research guidelines for item selection. The contractor must also propose additional field-tested items for possible replacements.

9. Forms construction must adhere to the process described in Attachments B, C, and D.

10.The contractor must provide proposed forms and all field test statistical data for review per the process described in Attachments B, C, and D.

11.At Process Step 18, Form Pre-Review, the contractor must also provide NYSED staff with a mapping of all proposed items, per test blueprint, and all form statistics.

12.At least one month prior to forms construction, the Item Bank must be operational, with the possible replacement items.

13.Annually, the contractor must facilitate a forms construction meeting for review of draft forms by NYS educators as specified test development process in Attachments B, C, and D.

14.The final operational forms will be jointly reviewed by the contractor and NYSED and approved by NYSED per the process set out in Attachments B, C, and D.

15.Annually, the contractor must facilitate a risk review and convene and execute a final eyes committee review per Attachments B, C, and D.

16.At the conclusion of the risk review and final eyes meeting, the contractor must conduct debriefings with NYSED to resolve any concerns raised by the committees.

17.Mathematics tests will be translated into eight target languages other than English. Arabic, Bengali, Chinese (Simplified), Chinese (Traditional), Haitian Creole, Korean, Russian, and Spanish. All eight languages will be offered for the paper-based tests. Five of the languages will be offered for the computer-based tests – Chinese (Simplified), Haitian Creole, Korean, Russian, and Spanish. Translated editions must be back translated. The ELA tests are not translated. The contractor and any subcontractor must refer to bilingual mathematics glossaries and past translations, which are posted on NYSED’s website, for guidance as to NYSED’s preferred word usages, formatting, and style.

18.The contractor will transcribe into Braille one form each of the Grades 3-8 ELA and Math Tests. The contractor will ship Braille forms directly to schools, based on the quantity requested placed by schools in NYSED’s online examination request system. The Braille editions must be accurately labeled. In addition to fulfilling requests placed by schools, the contractor is also required to provide an additional 25 booklets to NYSED. It is expected that the contractor will be required to provide a total of 125 booklets for each grade and content area in Braille. (See Attachment I: Specifications for Contractor/Subcontractor Transcribing Braille.)

19.The contractor will prepare a large-type edition of one form each of the Grades 3-8 ELA and Math Tests. The contractor will ship quantities of the large-type editions to schools based on orders placed by schools in NYSED’s online examination request system. (See Attachment J : Specifications for Large-Type.)

20.The contractor will prepare, print and ship all regular, Braille, large-type, and translated edition Grades 3-8 ELA and Math tests based on orders placed by schools in the online examination request system. For additional information on printing requirements, see the section of this RFP for Printing, Shipping, Collection and Return of Operational Test Materials.

42

http://www.nysedregents.org/

RFP #20-004

Local Scoring of Operational Tests for Grades 3-8 ELA and Math

1. The contractor will provide a toll-free number which is staffed from 7:30 A.M.–4:30 P.M. EST each business day to handle all test-related questions for the duration of the scoring period, which begins after each testing period ends. In addition, the contractor will ensure that a toll-free FAX number, with a minimum of three FAX machines, shall be operational 24 hours a day for the duration of the scoring period.

2. Student operational test answer sheets for students testing with PBT are developed by the RICs and large-city school district scanning centers. The contractor must cooperate with the RICs and large-city school district scanning centers to facilitate the development of the most appropriate student operational test answer sheets. The contractor must also work cooperatively with the SIRS to facilitate the transfer of statewide student operational test data. NYSED will provide the contractor with contact information for the RICs, large-city school district scanning centers, and the SIRS.

3. All paper-delivered operational test multiple-choice questions, including embedded field test and anchor items, will be scanned by the Regional Information Centers (RICs) and two of the large city (New York City and Yonkers) school district scanning centers. RICs are data processing centers operated by twelve Boards of Cooperative Educational Services (BOCES) that provide answer sheets and scanning services.

4. All operational constructed-response questions are scored by New York State teachers in regional or district-level groups and scanned at the RICs, including the large-city school districts. New York State is allowing local districts to choose regional scoring at sites identified by the BOCES and the large city school districts or choose district-level scoring. In addition, a growing number of public-school districts, charter schools, and religious and independent schools have in recent years been contracting on their own with BOCES or other for-profit and not-for-profit entities for their students’ CR responses to be scored.

5. Contractors must propose a plan to NYSED to collect data and provide quality control for all data records. In this plan, all answer documents for students testing with PBT will be scanned at the RICs where both the multiple-choice responses and the scores for constructed-response questions are added to an electronic file. These electronic records are sent to a central location, New York State’s Student Information Repository System (SIRS), for aggregation into one data file. This file is then provided to the contractor for scoring and scaling analysis. The contractor’s plan for the collection of data must include combining the data for test sessions administered with CBT with the data for test sessions administered with PBT. For each operational examination, the contractor will provide to NYSED a research file containing statewide item level responses for each MC question and scores for each CR question for each operational examination with NYSED.

Scoring Materials for Operational Test Constructed-Response Items

1. The contractor will provide clear, complete, and grade-specific scoring guides that are derived from the contractor-prepared field test scoring materials. The contractor will also provide rubrics, student paper exemplars at each score point, practice sets for scorer training and consistency assurance sets. The scoring materials for ELA constructed-response items should include 8 exemplars, 10 practice papers, and 10 consistency assurance papers for each 2-point item and 14 exemplars, 10 practice papers, and 10

43

RFP #20-004

consistency assurance papers for each 4-point item. The scoring materials for Math constructed-response items should include 8 exemplars, 10 practice papers, and 5 consistency assurance papers for each 2-point item and 11 exemplars, 10 practice papers, and 7 consistency assurance papers for each 3-point item.

2. The contractor will continue to use generic scoring rubrics for constructed-response items provided in the Educator Guides. Generic rubrics communicate how point values will be assigned for each level of response in terms of the general competency displayed.

3. Camera-ready scoring materials must be delivered to NYSED by the first week in October for ELA and the first week in November for mathematics.

4. The contractor-prepared PDF file scoring materials will be furnished to schools by the contractor on CDs, flash drives, or another vehicle proposed by the contractor. The contractor will be required to reproduce 7,500 CDs or flash drives to send to schools. The contractor must provide all scoring materials in a format that can be posted online by NYSED should NYSED choose to do so. Alternatively, the contractor may propose, subject to NYSED approval, to provide schools and districts access to the printable scoring PDF files on a contractor-hosted secure SFTP site or something similar.

Liquidated Damages Related to Operational Tests – Translated, Braille, and Large-Type Editions and all Scoring Materials for Constructed-Response Items

Because quantifying losses arising from the contractor’s failure to perform under the contract would be inherently difficult insofar as the contractor’s failure to deliver or perform its obligations may impact the State’s reputation, the contractor and State shall agree to certain liquidated damages provisions throughout the contract. The contractor and the State shall agree that the agreed-upon sums are not a penalty, but rather represent reasonable measure(s) of damages, based upon the parties’ experience and given the nature of the losses that may result therefrom. The liquidated damages clauses herein are intended to cover reputational and other losses suffered by the State in the event the contractor fails to provide a deliverable or perform a duty as outlined therein.

Translated Editions of Mathematics Tests

1. The contractor shall provide a forward and backward translation and a third-party independent translator to verify and document the decisions made to resolve any inconsistencies between the two versions. The parties acknowledge that NYSED does not have the capacity to verify the accuracy of the translation proofs provided by the contractor.

2. Errors found in three to six foreign language translation(s) of test questions delivered by the contractor, which require NYSED to make an adjustment to students’ scores (e.g., a change in what constitutes an acceptable student response for an examination question, or which require the elimination of an examination question from consideration in scoring) in the aggregate in any one contract year, shall cause NYSED to reduce by twenty percent (20%) the agreed upon consideration as specified in the line item entitled “Provide [relevant year] Math Operational Tests Translated Editions (including Embedded Items)” in the agreed-upon budget for the associated contract year. Errors found in seven to eleven foreign language translation(s) of test questions delivered by the contractor, which cause NYSED to make an adjustment to students’ scores in the aggregate in any one contract year, shall cause NYSED to reduce by forty percent (40%) the agreed upon consideration as specified in the line item entitled “Provide [relevant year] Math Operational Tests Translated Editions (including Embedded

44

RFP #20-004

Items)” in the agreed-upon budget for the associated contract year. Errors found in greater than eleven foreign language translation(s) of test questions delivered by the contractor, which cause NYSED to make an adjustment to students’ scores in the aggregate in any one contract year, will be deemed by NYSED to constitute a total failure of the contractor to provide this deliverable and shall cause NYSED to reduce by one hundred percent (100%) the agreed upon consideration as specified in the line item entitled “Provide [relevant year] Math Operational Tests Translated Editions (including Embedded Items)” in the agreed-upon budget for the associated contract year.

Braille Transcriptions

Errors found in two Braille transcriptions of test booklets delivered by the contractor, which cause NYSED to make an adjustment of students' scores (e.g., a change in what constitutes an acceptable student response for an examination question, or which require the elimination of an examination question from consideration in scoring) in the aggregate in any one contract year, shall cause NYSED to reduce, by twenty percent (20%) the agreed upon consideration as specified in the line item entitled “Provide Production Copies [relevant year] Operational Tests Braille Editions (Including Embedded Items)” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year. Errors found in three or four Braille transcriptions of test booklets delivered by the contractor to NYSED, which cause NYSED to make an adjustment of students' scores in the aggregate in any one contract year, shall cause NYSED to reduce, by fifty percent (50%) the agreed upon consideration as specified in the line item entitled “Provide Production Copies [relevant year] Operational Tests Braille Editions (Including Embedded Items)” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year. Errors found in five or more Braille transcriptions of test booklets delivered by the Contractor to NYSED, which cause NYSED to make an adjustment of students' scores in the aggregate in any one contract year, shall be deemed by NYSED to constitute a total failure of the contractor to provide this deliverable and shall cause NYSED to reduce by one hundred percent (100%) the agreed upon consideration as specified in the line item entitled “Provide Production Copies [relevant year] Operational Tests Braille Editions (Including Embedded Items)” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year.

Large Type

Errors found in two Large Type Editions of test booklets delivered by the contractor, which cause NYSED to make an adjustment of students' scores (e.g., a change in what constitutes an acceptable student response for an examination question, or which require the elimination of an examination question from consideration in scoring) in the aggregate in any one contract year, shall cause NYSED to reduce, by twenty percent (20%) the agreed upon consideration as specified in the line item entitled “Provide Production Copies [relevant year] Operational Tests Large Type Editions (Including Embedded Items)” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year. Errors found in three or four Large Type Editions of test booklets delivered by the contractor to NYSED, which cause NYSED to make an adjustment of students' scores in the aggregate in any one contract year, shall cause NYSED to reduce, by fifty percent (50%) the agreed upon consideration as specified in the line item entitled “Provide Production Copies [relevant year] Operational Tests Large Type Editions (Including Embedded Items)” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year. Errors found in five or more Large Type Editions of test booklets delivered by the contractor to NYSED, which cause NYSED to make an adjustment of

45

RFP #20-004

students' scores in the aggregate in any one contract year, shall be deemed by NYSED to constitute a total failure of the contractor to provide this deliverable and shall cause NYSED to reduce by one hundred percent (100%) the agreed upon consideration as specified in the line item entitled “Provide Production Copies [relevant year] Operational Tests Large Type Editions (Including Embedded Items)” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year.

Scoring Materials for Constructed-Response Items

Three to six errors found in the Scoring Leader Materials or Scorer Materials for constructed-response items that cause NYSED to post adjustments to scorer training materials resulting in a change in what constitutes an acceptable student response to achieve a certain score point for a constructed-response item in the aggregate in any one contract year, shall cause NYSED to reduce, by twenty percent (20%) the agreed upon consideration as specified in the line item entitled “Submit Scoring Files Including Consistency Assurance Sets for Operational Tests” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year. Greater than six errors found in the Scoring Leader Materials or Scorer Materials which cause NYSED to post adjustments to scorer training materials resulting in a change in what constitutes an acceptable student response to achieve a certain score point for a constructed-response item in any one contract year, shall cause NYSED to reduce, by forty percent (40%) the agreed upon consideration as specified in the line item entitled “Scoring Materials for Operational Test Constructed-Response Items” in the agreed-upon budget for the impacted assessment(s) (i.e., ELA or Math) for the associated contract year.

Grades 3-8 ELA and Math Manuals (Nonsecure Materials)

Updating the Educator Guides

The contractor must annually update the Educator Guide for each content area, ELA and Math. Updates will be agreed upon by NYSED and the contractor no later than August 1st each year. Updates must be ready for posting by October 1. Final versions must be ready for use by educators at Forms Construction and Final Eyes Review Events. The current Guides may be found on NYSED's website. The Guides must address: how the tests were developed; how they are administered; how scores will be used and in what format they will be reported; how many test sessions there will be by grade and the length of testing time for each session; what types of test questions (items) will be included (with examples); what standards will be assessed; and test specifications.

Additionally, the contractor must prepare each of these documents for posting on the NYSED website, ensuring that all documents posted on the NYSED website meet NYSED’s web accessibility guidelines as outlined below in the portion of this RFP entitled “Web Accessibility.”

Updating the Style Guide

The contractor must make any needed changes to the Style Guide throughout the year, as agreed upon by NYSED. A finalized version of the Style Guide must be produced by the contractor for use, as needed, at Forms Construction and Final Eyes Review Meetings.

Updating the School Administrator’s Manual (Operational and SAFT), Operational ELA and Math Teacher’s Directions (PBT and CBT), SAFT ELA and Math Teacher’s Directions (PBT and CBT), and Scoring Leader Handbook (Operational)46

RFP #20-004

The contractor must also annually update the School Administrator’s Manual (SAM), all Teacher’s Directions (TDs), both PBT and CBT, and Scoring Leader Handbook (SLH) to reflect the current year’s testing policies and procedures. The current SAM, TDs, and SLH may be found at Test Manuals for School Administrator’s and Teachers. Print ready copies must be produced for printing and distribution to schools as outlined in the section on Printing, Shipping, Collection, and Return of Operational Test Materials. Additionally, the contractor must prepare each of these documents for posting on the NYSED website, ensuring that all documents intended to be posted on the NYSED web site meet NYSED’s Web Accessibility Guidelines as outlined in the “Accessibility of Web-Based Information and Applications” section of this RFP.

Forms Scaling and Equating

Contractors must use IRT 3-Parameter Logistic Model for multiple-choice items and 2-Parameter partial credit model or graded response model for constructed-response items to calibrate, equate, and scale the field test and operational test forms. The equating process must allow for year-to-year comparisons for each test through the development of a common scale across years.

Post-equating methodology must be used to equate the operational test forms using the non-equivalent groups anchor test design. The contractor must propose an anchor design and items. The contractor must revise or augment the anchor items as necessary to ensure alignment to any changes to the learning standards or test specifications. The full set of anchor items for each test must be representative of the operational test with regard to the test contents and statistical specifications. Currently, a subset of 12-14 multiple-choice anchor items (two passage sets) are embedded into each operational ELA test form and a subset of 14 multiple-choice anchor items are embedded into each operational Math test form.

Each operational test form can be placed onto the operational scale through the secure anchor items using the Stocking and Lord Method. The stability of the anchor items must be examined before being used as anchors in the equating process using appropriate method and statistics.

Additional extended-response items needed for constructing future operational tests will require a separate field test administration. The bidder must select the field test samples and use the common student equating method to equate and place the SAFT items onto the operational base scale.

Operational test items can be selected from the item bank established through embedded field test items and SAFTs based on content coverage and statistical specifications. Operational test data must be used for item calibrations and equating and for creating the operational conversion tables. Number correct scoring approach must be used to produce the raw score to scale score tables using the test characteristic curve (TCC). As part of the equating process, the contractor must conduct PBT/CBT Test Mode Comparability Studies for each of the twelve operational tests and provide appropriate raw score to scale score conversion tables for each mode. The contractor must submit to NYSED the data files used for equating and calibration for review and verification.

If the contractor is using proprietary software to scale and equate the test forms, the contractor must provide NYSED’s research staff with access to the software and training in its use at no additional charge to NYSED for the duration of the contract and three years after the end of the contract.

47

RFP #20-004

Operational Test Technical Report

Contractor must submit a comprehensive technical report after each operational test administration. The technical report must include, but not be limited to, all the technical information such as test design, test development, scoring, test reliability and validity, calibration and equating, classical and IRT statistics, and analysis of the operational test results.

All research files must be submitted to NYSED containing complete student item-level data as well as scale scores and the demographic information necessary to replicate all analysis.

The contractor must retain all operational test data through the contract period and for five-years after the end of the contract.

The contractor will provide a comprehensive technical report for each test (see Attachment K for an outline of the current report) to include:

1. Disaggregating test results at various levels;

2. All psychometric and statistical data;

3. Reliability data, including internal consistency and decision reliability, and validity data;

4. Documentation of item linkages to the standards;

5. Inter-correlations by content-specific sections and item types;

6. Generalizability analyses must be provided; and

7. Item-level data must also be provided.

The 2017 Grades 3-8 Technical Manual should be used as a template. This document can be found on NYSED’s website under Technical Reports.

Technical Advisory Committee

The Contractor is required to work with NYSED’s Technical Advisory Committee (TAC) to ensure independent guidance is given to the assessment program. This may require travel to TAC meetings to receive guidance and/or present information as needed (a maximum of three one-day meetings per year in New York City).

Standard Setting

Following one year’s administration of the Grades 3-8 ELA and Math Tests measuring the Next Generation Learning Standards, the contractor will conduct standard setting. This standard setting will take place Spring 2021. For the standard setting, the contractor will handle all logistics, work with NYS teachers to revise performance level descriptors, and conduct the standard setting and all necessary analyses. Technical reports must be produced documenting the standard setting process. The contractor is required to advise NYSED on any psychometric/methodological matters relating to standard setting.

The process for standard setting will consist of the following elements:

48

http://www.p12.nysed.gov/assessment/reports/

RFP #20-004

1. Hold meetings with NYS teachers to develop and/or validate content-area performance level descriptors (PLDs) (see Attachment E for further details regarding educator participation).

2. Design and execute appropriate external benchmark data and analysis

3. Hold standard setting meetings to set performance standards (see Attachment E for further details regarding educator participation).

4. Develop a technical report detailing the overall recommendations of panelists, the judgments of each subject area expert in each phase of judgment (as appropriate to the standard setting model) and sections on the development of PLDs and the analysis and presentation of external benchmark data.

Performance Level Descriptors (PLD)

For each test, the contractor will handle all logistics and work with NYS educators to review and/or to validate content-area performance level descriptors describing the range of student performance within each performance level on the given assessment (“range PLDs”). These meetings will be approximately one day each and take place in March 2021, the year in which the standard setting will be conducted.

The contractor will be responsible for arranging and paying for conference room space within 30 miles of Albany for the PLD meetings. In addition, the contractor must provide all meeting facilitation and training for participants and develop and produce all materials necessary to conduct the meetings.

The PLD meetings will consist of three grade-band groups for each content area, Grades 3-4, 5-6 and 7-8, for a total of six groups. Each group will include a minimum of eight (8) NYS-certified educators in the examination content area. NYSED will provide the contractor with names of educators for the contractor to contact in order to determine availability and willingness to participate. The contractor will be responsible for arranging and paying for participants’ travel reimbursement, meals, and hotel accommodation for participants residing more than 50 miles from the meeting location. Educator participants will also be paid honoraria of $200 per day.

In addition to the range PLDs developed at the PLD meetings, threshold PLDs may be developed if time allows. Threshold PLDs describe the knowledge, skills, and practices demonstrated by students at the borderline of each performance level. Threshold PLDs will be developed and/or finalized by standard setting panelists in the beginning of the standard setting meeting.

Alignment Studies for ELA and Math

In order to meet the ESSA requirement that each state submit evidence regarding its assessments in English Language Arts and Mathematics to the United States Department of Education for peer review, the contractor will arrange and provide for a one-time alignment study by an independent party with demonstrated expertise in high-quality, large-scale assessment, at the contractor’s sole cost. These studies must review the test blueprints and the operational test forms for the revised ELA and Math Tests measuring the new NYS Next Generation Learning Standards in ELA and Math and revised language arts and mathematics progressions. The studies must determine:

1. The extent to which the tests are aligned to the full range of New York State’s relevant academic standards, and

49

RFP #20-004

2. The extent to which test items intended to require complex demonstration or applications of knowledge and skills relate to other measures that require similar levels of cognitive complexity in the content area.

A report from the independent-expert review of the English Language Arts and Mathematics Tests is expected to be completed by September 30, 2021, following the administration and scoring of the 2021 operational tests.

The contractor shall notify NYSED of any third-party reviewing field test items and/or operational items. Any such third party is subject to the review and approval of NYSED and its staff must sign non-disclosure agreements.

Feasibility Study for Implementing Computer-adaptive Testing

To enable the Department to gather crucial input from stakeholders and experts into what features of adaptive testing should be considered, if any, the contractor will conduct a cost and feasibility study of the changes to the assessment program that would be required and any issues that need to be addressed in implementing such a transition. The study must include an examination of 1) applicable federal and state laws and regulations; 2) school and district policies; and 3) school and district technological infrastructure and capacity. Significant consultation with national experts and NYS stakeholders such as teachers, students, parents, administrators, and members of professional organizations is also required. The Department will assist the contractor in identifying and contacting the appropriate NYS stakeholders. Some of the consultations may occur in person and the contractor is encouraged to take advantage of gatherings of professional organizations and scheduled educator test development events to efficiently and cost-effectively collect input from these stakeholders. The contractor is not expected to travel to individual school districts to assess infrastructure and capacity.

The final product will be a written report which the contractor is expected to submit to the Department by September 30, 2021, unless the deadline for this report has been revised by mutual agreement of the contractor and NYSED. This report must provide recommendations from the contractor on what features, if any, of computer adaptive testing best meet the goals and purposes of the NYS Testing Program and how to best transition to computer-adaptive testing, if appropriate. It should also include estimates of the costs of implementing these recommendations.

Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math

Component 1b services require the contractor to provide a computer-based testing platform that includes a test delivery system, test administration system, and scoring platform for the Grades 3-8 ELA and Mathematics Tests. The contractor must have the staffing levels needed to maintain a high level of quality assurance and customer service throughout the duration of the contract including peak levels leading up to and during test administration dates.

The contract that results from this RFP requires ongoing flexibility on the part of the contractor to provide schools the option to administer operational tests and SAFTs as either paper-based or computer-based tests. It is NYSED’s goal to require all schools and all grade levels to test on computer. However, any contractor should be cognizant of special considerations and challenges that have not been fully vetted to complete a successful transition of all NYS schools and should be prepared to offer paper-based and computer-based tests for ELA and Math in unknown proportions.

50

RFP #20-004

The contractor will be required to provide NYSED with a detailed plan to successfully complete the transition to CBT.

In preparation for administering operational forms on computer, the contractor must ensure that the system will function without incident using existing computers within the State’s schools (see Attachment A for minimum specifications), provide a CBT administration and delivery platform, and work with NYSED to ensure the testing platforms are compatible with the existing computers and infrastructure available in New York State’s schools and districts. The testing platform must have the capability to serve the translated editions of the Math tests to ELLs/MLLs requiring such accommodation. For CBT Mathematics translated test administration, all aspects of the testing platform must be adapted to fully meet the needs of ELLs/MLLs. This means that all words appearing on the computer screen, including icons and test taking directions, must be translated. The CBT platform must enable students to respond to questions in their native languages. This means the platform must be compatible with alternate language keyboards used in conjunction with all acceptable computer devices.

The platform must also offer certain accommodations to students with disabilities, as outlined later in this RFP in the section “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math Computer-based Testing

A goal of Computer-Based Field Testing is to reduce the amount of school/student participation that is necessary in stand-alone field testing by developing as many of the future multiple-choice (MC) and short-response items as possible through embedding them in computer-delivered operational test forms to eliminate the need for stand-alone field testing of those items. With this goal in mind, the contractor should propose to embed future MC items in as many computer-delivered operational test forms per grade/subject as necessary to prevent the need to include them in the SAFTs. As with the paper-based operational test forms, the contractor should plan for all computer-delivered operational test forms to be spiraled within each classroom. It is NYSED’s goal that by some point in the contract term it may only be necessary for the contractor to produce only one paper-based form of each test.

Computer-Based Operational and Field Test Forms

The contractor must provide a timeline to develop test forms for computer-based field testing and computer-based operational testing that mirrors the development of paper-based test forms. The contractor must ensure that NYSED has, at minimum, 60 days to review and quality check the computer-based test forms. Representatives of NYSED will review all test forms developed under the contract. During the Forms review process, the contractor will provide NYSED with the test forms and needed platforms to complete a review for the online and paper forms for accuracy and functionality. This will include all forms, including accommodated and alternate language forms. The contractor will provide a checklist for reviewers to utilize to validate each form. The checklist provided by the contractor must include all known online functionality. NYSED and the contractor will communicate all edits requested through a secure site. Additionally, NYSED will confirm that all edits requested were applied correctly and are resolved in the review process. If the edits are resolved, NYSED will give approval. If the edits are not resolved, NYSED will provide feedback to the contractor. The contractor is responsible for correcting any errors in work products arising from activities that are responsibility of the contractor at the contractor’s expense. Such corrections may involve activities that include, but are not limited to, the following:

1. Analyses to identify the cause and extent of errors.

51

RFP #20-004

2. Edit, revise, and reproduce items or materials.

The CBT operational tests and Stand-alone Field Tests forms for ELA and mathematics must be available for participating schools to securely access the encrypted test content between 7 a.m. and 6 p.m. Eastern Time each day the schedule allows administration for those tests.

Testing System Specifications

The contractor is responsible for providing an online test delivery platform and an administration platform that is detailed in a technical systems manual and at a minimum provides:

1. Test access control

NYSED requires a system that provides student authentication with the ability to maintain student and test security, including:

a. Student authentication (e.g., login ID, password) must be required to access an online test. The required authentication information for gaining access to a computer-based test should be generated only within a secure administrative system. A student test record should exist in the administrative system before the corresponding authentication information may be generated.

b. Additional means of access control are required (e.g., no concurrent logins, access controls based on time and date, randomly generated authentication information).

2. Administrative access control

NYSED requires administrative authentication and access, that will enable school-based non-technical staff to manage the assessment successfully.

a. Administrative authentication (e.g., login ID, password) must be required to gain access to administer computer-based tests, view/maintain student data, and access student performance reports.

b. Administrative user accounts must be manageable to include the capability to be assigned varying levels of access and functionality and be activated or inactivated as needed. Multiple roles and responsibilities must be given administrative permissions by functionality to regional, district, and building users.

3. Security of test content and student data

Test security and the security of student data is a primary concern of the NYSED. The contractor must use a specific method of encryption that is supported by research. The following are additional considerations:

a. All test content must be encrypted at the host server and remain encrypted during all network transmissions.

b. Only valid authentication information may enable test content to be decrypted to a viewable format. Decryption to memory only (versus being written to disk) is highly desirable by NYSED for protection of its test content. No decrypted or unencrypted test content should exist on equipment outside the hosting facility or NYSED’s physical location.

52

RFP #20-004

c. All confidential data must be stored on computer and storage facilities maintained within the United States using appropriate firewalls and security measures.

d. Test content accessed via valid authentication information must be displayed only while the student is taking the test. Upon completing the test, any residual, decrypted test content must automatically be removed from any systems outside of the host systems.

e. When caching technology is utilized, the cached content is secured, managed, and purged in accordance with the description of the caching technology provided in the technical systems manual.

f. All transmissions of student data must occur over secure network connections that utilize authentication and encryption technologies.

g. The contractor will review all system security features quarterly with NYSED and meet all student data security requirements as noted in Education Law §2d and other component rules and regulations in New York State. Additional information on the regulations can be found in Appendix R of this RFP.

4. Desktop, laptop, and tablet security during testing

a. While students are testing, decrypted test content must be protected through control of the desktop, laptop, and/or tablet. Contractor must provide a solution by which access to other applications or web sites will be disabled while a test is being accessed. Strict controls must be maintained over operating system functionality, printing, copy and pasting, screen captures, keyboard shortcuts, right-mouse clicks or other functionality that could compromise test content.

b. Security on the desktop, laptop, and/or tablet should exist equally for all approved operating systems as specified under the hardware requirements for managed and non-managed environments.

5. Wireless Networking

Some schools and districts utilize wireless networking. The proposed solution should be compatible with the current wireless networking standards (IEEE.802.11x) but should not assume that all wireless networks have been configured as secure networks. Details of the security of the online testing solution in a wireless environment must be provided in the technical systems manual.

6. Network Availability a. The platform must have an ability to continue administration during periods of network

congestion or interruption that may include caching of content and responses on the local machine and have a recovery plan which includes communication to NYSED and the field.

b. The contractor will provide a system that allows for online testing simultaneously to no less than 600,000 students in NYS with optimal page-turn, system performance and user experience during peak testing hours during the test administration windows.

53

RFP #20-004

c. The contractor will investigate any reports of systems slow down and seek immediate solutions specified by NYSED to improve reports of poor online test performance.

d. The contractor will conduct extensive load testing and provide reports to NYSED prior to the start of the NYS test administration windows.

i. The contractor must maintain appropriate technical expertise and also provide third party external verification and validation of all load testing of the CBT system to include the capacity of database servers and the ability of the system to accommodate the anticipated high volume of users.

e. The contractor will provide twice daily “health” reports to NYSED on the test delivery system, administrative, and scoring system once in the morning, and once in the evening. The contractor must provide immediate reports to NYSED if at any time adverse conditions occur. The contractor will provide guidance to the field as requested by NYSED.

7. Data Interoperability

a. The contractor will accept any data, such as student enrollment information, provided to it by NYSED in a non-proprietary interoperable format. It is the contractor’s responsibility to use the format provided by NYSED to integrate the NYSED data into the contractor’s system.

b. The contractor has an established system of data transmission which will limit the number of integrations in order to extract data from multiple sources. In addition, the contractor will securely connect applications to ensure data privacy.

c. The contractor has the capability to manage a migration from flat-file delivery methods to electronic data transmission, including XML and development and adoption of data standards.

d. The contractor will ensure that all components function properly, and, the interchange of data between all components of the assessment system and across diverse networks meets key interoperability requirements.

User Interface

This user interface that will be used by students should provide:

1. Standardized display of text and graphics for all students regardless of monitor type and resolution setting. (If a specific resolution setting is required, this should be indicated.)

2. Ability for students to navigate throughout complete test form (forward, backward, and skipping items).

3. Ability for students to mark a multiple-choice answer using an input device or touchscreen.

4. Ability for students to change a multiple-choice answer using an input device or touchscreen.

5. Ability for students to complete constructed-response items by typing their response and selecting basic text formatting options. Italics, underlining, undo/redo functionality are required text formatting capabilities. Additionally, the “tab” functionality must indent a paragraph within the constructed response box, rather than move the cursor/focus to the next control.

54

RFP #20-004

6. Constructed-response boxes comparable in size to the answer space provided in the printed test booklets. Constructed response boxes must have the ability to expand to view so that the student can view his/her entire response. Text within the extended response box should automatically wrap and move dynamically if/when the box is expanded or contracted. Horizontal scrolling is not allowed.

7. Ability for students to select or insert any necessary mathematical equations, symbols, numbers, and special characters that are not found on a standard keyboard (e.g. degree symbol, non-Latin letters, subscripts, superscripts) while still being able to type alpha-numeric characters (i.e. text) for verbal explanations and justifications.

8. Ability and capacity for students to draw their mathematical responses. This tool must be available by Spring 2021 administration.

9. Ability for students to pause the test, log out, and, with proper authority, be re-located to another computer to login and resume taking the test, and answers should be available to be viewed and changed by the student upon resuming the secure assessment with appropriate permission.

10.Ability for students to view germane text or graphics simultaneously with the test item stem and distractors when an item or set of items is text or graphic intensive.

11.Ability for students to view the text, item stimulus, question, and answers without the need for scrolling horizontally and vertically.

12.Availability of NYSED-approved ancillary testing features and tools for use by students (e.g., calculator, ruler, protractor, reference sheets). Information on the features and tools currently used for each test is provided in the School Administrator’s Manual.

13. Indication (on the workstation display) of the name of the student login used to access the test.

14. Indication via a pop-up warning that appears on the screen to the student for confirmation to end their test. The pop-up warning screen should allow the student to return to continue taking the test and/or submit their test.

At a minimum, the contractor will provide an online testing environment that provides options to support additional test-taking strategies such as:

1. Visually eliminating one or more distractors;

2. Highlighting key words, text, images, and/or graphics to recall and/or emphasize text;

3. Flagging items as incomplete or in need of review using a visible icon or other option prior to completing the test;

4. Indicators of which items have been answered/unanswered which also allows the users to move to different sections of the assessment;

5. Using an embedded Notepad application in which students are able to write and edit text;

6. Blocking certain parts of the test so a student can focus on one portion at a time, line reader;

7. Zoom text capabilities, ability for the student to enlarge or reduce text and graphics on the screen, up to 300% and no less than 50% (while preserving clarity, contrast, and color);

8. Split screen controls;

55

http://www.p12.nysed.gov/assessment/sam/ei/eisam19.pdf

RFP #20-004

9. Online ruler, calculator, and protractor capabilities;

10.Ability for student to copy and paste any text within a test item into a Notepad or constructed-response entry box for all assessments;

11.Ability to use writing process tools for written responses, such as underline, bold, and italics;

12.Ability to pause the test and retrieve all saved work, including but not limited to any typed text, highlighting, and eliminated distractors.

At a minimum, the contractor will provide the following online student testing accommodations in its administrative and test delivery system:

1. Text to Speech – Allows all text, mathematical symbols, special characters, as well as any text embedded in graphics to be read aloud to the student, highlighting the text as it is spoken. The text, mathematical symbols, special characters, as well as, any text embedded in graphics must be highlighted word by word.

2. Reverse Contrast – Presents all text in white on a black background;

3. Answer Masking – Hides answers that the students select, allowing the student to focus on just the answers they want;

4. Background Color – Changes the color of the background and text based on the selection made;

5. Initial Zoom (with responsive design) - Provides any text, graphics, and/or images on the assessment enlarged in increments up to 300% initially when the student logs into the test delivery system. As indicated by responsive design the display must dynamically adjust to each zoom setting rather than require students to scroll horizontally;

6. Alternate Language – this accommodation feature provides ELLs/MLLs with the option to test online in a student’s native language for New York State mathematics stand-alone field and operational tests using an external or online alternate language keyboard. The translated content must encompass all text content embedded in its CBT platform, such as online tools, to be translated into and presented to students in the same target languages.

Administrative System

The administrative system used to manage the delivery of online tests must be capable of supporting the various aspects of the assessment program.

1. Data Management. The administrative system must be:

a. Able to allow administrative users to view and edit student information and corresponding demographic information entered as part of the pre-id process as defined by NYSED;

b. Able to allow administrative users to manually enter new student records prior to or at the time of testing;

c. Capable of maintaining unique student test records with both optional and required data fields;

d. Capable of maintaining both student-specific data fields and test-specific data fields;56

RFP #20-004

e. Capable of maintaining information on student enrollment requiring students to be tested in the grade/content area but who do not complete the associated test or whose test is invalidated (e.g., absence from all or part of the test session(s), administrative error, etc.);

f. Capable of handling multiple individual test records for a single student;

g. Capable of providing a database and user interface for managing all NYS educational organizations and users;

h. Capable of managing accommodations and not tested reason codes for classes and students;

i. Capable of providing access to scored student constructed responses for school-level authorized users for no less than 2 years;

j. Capable of allowing authorized users to download reports by region, district, school, testing group, and student levels;

k. Capable of allowing authorized users to export reports at the region, district, school, testing group, and student levels;

2. Test Management. The administrative system must have:

a. Ability for administrative users with appropriate access to schedule students for online tests and generate necessary student login information prior to testing.

b. Ability to manage test assignments for individual students such that a test may not be administered to a student more than one time in a test session.

c. Ability for authorized users to view individual student test information including initial logins, subsequent logins, pausing of the test session, and submission entries.

d. Ability for authorized users to view aggregate test information by subject including number of test sessions, number of tests being administered (real-time), number of tests submitted.

e. Ability for authorized users to set Not Tested reason codes on a student and group level.

f. Ability for authorized users to set accommodations on a student and group level.

g. Ability for authorized users to print student login tickets individually, by groups of students, and by batch.

h. Ability for authorized users to organize testing groups.

3. Security Management. The administrative system must have:

a. Ability to control access based on specific password-protected user accounts (e.g., school-level, district-level, state-level accounts)

b. Capable of secure transmission of all student information

c. Availability through role-based access 24/7 and year-round to all authorized NYS educational organization entities. All scheduled and/or unscheduled system updates to the test administration system must be agreed upon by NYSED.

57

RFP #20-004

System Monitoring and Reporting

Various types and levels of system monitoring shall be implemented by the contractor to ensure availability of the online testing system and to provide NYSED with user statistics.

1. All system monitoring shall be conducted by the contractor with results available to NYSED daily and upon request during the pre-administration, administration and scoring windows.

2. The contractor’s system will provide a means for automatically monitoring system availability to authorized users. The proposed monitoring system should automatically notify the contractor on an on-going basis of system warnings and/or system failures to appropriate contractor staff for corrective action.

3. The contractor’s system will provide real-time and archived users statistics to include, but not be limited to, concurrent administrative system users, concurrent testers, number of test sessions scheduled, number of items completed by test takers, items remaining for students with test sessions in progress, and number of test sessions submitted.

4. Statistics should be by test administration, test session, and school district as determined by NYSED. Authorized users should be able to filer/sort data by test session, test grade, and test subject area. Authorized users will be provided the capability of exporting data in a format agreed upon by NYSED. The contractor will provide users statistics post administration for time spent on individual questions.

5. The contractor will provide regular reports to NYSED of online system usage that include, but are not limited to, usage over time, by browser type, and operating system type.

6. The contractor will conduct extensive load testing and provide reports to NYSED prior to the start of the NYS test administration window.

a. The contractor must maintain appropriate technical expertise and also provide third party external verification and validation of all load testing of the CBT system to include the capacity of database servers and the ability of the system to accommodate the anticipated high volume of users.

7. The contractor shall conduct comprehensive capacity planning prior to each Testing Window to ensure that all system resources are appropriately sized and configured for the expected size and volume of that Testing Window.

8. The contractor will provide a narrative that illustrates the ability of the contractor's CBT delivery system to function in low bandwidth scenarios and explains the compatibility of the contractor’s test delivery system that student responses are not lost if there is a disruption in the students’ device connection to the local or contractor’s servers.

9. The online test delivery system must provide confirmation on each student testing device that the test delivery system is installed properly.

10.The contractor must provide a method for reporting of transactions at school, district and state level including but not limited to: number of attempted logins, number of successful logins, length of delay from attempt to login to successful login, number of interrupted test session

58

RFP #20-004

administrations, number of students paused transactions, and number of successfully submitted tests.

System Availability, Maintenance, and Updates

The contractor’s online test delivery system must be available to schools, with the exception of scheduled downtime as approved by NYSED, to deliver and score online tests and conduct related administrative functions.

1. System Availability and Maintenance:

a. The online test delivery system must be capable of delivering online tests and allowing access to conduct related administrative functions on all days during scheduled testing periods and three weeks before and after. Scheduled downtime should occur outside of the above timeframes as agreed upon by NYSED.

b. In the event that system downtime is determined necessary to resolve a critical issue within the above timeframes, NYSED must be notified and approve the downtime prior to the system becoming unavailable.

c. A communication plan must be designed to be used for planned and unplanned system outages, including communication with NYSED, districts, and schools.

d. The contractor must provide NYSED with a recovery plan for its approval.

2. System Updates:

a. Throughout the life of the awarded contract, it is assumed the proposed online solution may change due to necessity and system enhancements. NYSED must be made aware of any planned changes to the online testing system that will occur no less than six (6) months prior to the deployment of any system changes.

b. The contractor conducts elicitation sessions with NYSED’s designated staff to finalize the enhancements and associated requirements into a formal document.

c. Enhancements are agreed-upon and finalized between contractor and NYSED by October 1 each year.

d. Final documentation is presented to NYSED by October 15 of each year.

e. Accurate, detailed documentation of all changes must be provided to NYSED for review at least three (3) months prior to deployment into production. The updated functionality also should be made available in a non-production test environment for 8 weeks prior to deployment for NYSED to review. Requests for exceptions to this timeline shall be presented to NYSED for case-by-case consideration.

f. NYSED reserves the right not to have system upgrades deployed immediately prior to or during a statewide test administration period. System changes should be deployed only when approved by NYSED and with the intent to maintain or improve the level of system functionality. System updates should take place only after 4:00 pm with completion by no later than 7:00 am the next day. The contractor may perform these updates at a different time only if agreed upon with NYSED.

g. The contractor must provide NYSED with a detailed description of how system updates, system patches, and/or system enhancements are planned, managed, and fully tested prior to deployment into production.

59

RFP #20-004

h. If the contract provides locally installed software as part of the proposed online testing solution, NYSED must approve the updating process. NYSED strongly desires a solution that includes minimal update efforts to be put forth by school or school district personnel especially within an academic year. Ideally, any system update processes will be automated when possible to reduce the need to configure or update individual workstations within schools prior to administering tests.

Scoring Platform and Data Management

The contractor must provide a scoring platform addressing the following criteria:

1. Administration. The scoring platform must be:

a. Synchronized with the contractor’s CBT administration and test delivery systems;

b. Able to accept school and district demographic information that is captured by NYSED’s data collection systems;

c. Able to display scoring consortia data- meets all NYS Scoring Consortia needs which include BEDS code associated with the Scoring Consortia, scoring consortia, email addresses, names and BEDS codes of schools where student-constructed responses are scored by the consortia;

d. Able to allow for NYS Role and Responsibilities: The scoring platform must allow for user roles that are comparable to NYSED’s paper-based testing scoring organization which are Scorer, Table Facilitator, and Scoring Leader and Scoring Site Coordinator. Additional information on these roles can be found in the Scoring Leader Handbook ;

e. Able to provide a scoring dashboard;

f. Able to allow for custom scoring window timeframes;

g. Able to allow for scoring reports by groups, items scored, scoring times, and scorers.

2. Scoring. The scoring system must:

a. Allow for scoring according to all NYS scoring metrics and scoring models which ensure that responses are not fed to the raters from the same school in which the students are enrolled;

b. Allow for read behinds;

c. Allow for alerting in which a scorer has the capability to escalate any sensitive content to a Scoring Leader.

3. Data Management. The contractor must:

a. Develop Data Dictionary for student score and score data;

b. Develop business rules document in conjunction with NYSED business rules for scoring;

c. Provide quality control mechanisms that ensure that the scores inputted by raters and the student responses are securely stored and transferred to the contractor’s exam data management system with 100% accuracy and reliability;

60

RFP #20-004

d. Ensure the scoring system meets all NYS Education Law privacy regulations and guidelines. Additional information on the regulations can be found in Appendix R of this RFP;

e. Securely store and provide secure platform for schools to access student constructed responses for no less than two years from the date of release.

Readiness Verification Process

Prior to conducting computer-based operational tests, the contractor must provide an online tool, utility, or process for schools and school districts to use to verify the capacity of their technical infrastructure for conducting computer-based tests that use the contractor’s test administration platform. Districts and schools should be able to verify that workstations meet minimum specifications, that bandwidth meets minimum requirements, and that their network can support concurrent users during normal school hours.

The verification tool must reflect the minimum connectivity and hardware needed for successful online testing using the contractor’s online delivery solution. The tool should be available for school districts to use for benchmarking throughout the academic year.

By October 1 of each school year, the contractor will begin contacting those schools identified by NYSED to have the schools verify their technology readiness. The contractor’s administration database:

a. Is refreshed each year for the new school-year educational organization and user data.

b. Offers an online form for school users to verify they meet NYS CBT technology readiness requirements.

c. Collects student testing device information.

d. Collects user contact information for operational testing communication from contractor and NYSED

e. Provides reporting capabilities.

f. Provides messaging capabilities to school users about CBT Technology Readiness.

g. Offers tool for schools to test bandwidth and test student testing device minimum requirements.

h. Provides NYSED and schools with a quick reference guide to assist schools with important information and clarification to complete CBT Technology Readiness Checklist.

The contractor will provide access to Online Technology Readiness Checklist, including:

a. Confirmation of requirements met;

b. Collection and display of Scoring Consortia;

c. Collection and display of School Device Use;

d. Collection and display of Operational Contact Information, including technical coordinators and other users who are not collected in SEDREF;

e. Ability to export data.

61

RFP #20-004

Online Question Sampler and Secure Browser Practice Tests

For the 2020-2021 school year, the contractor will provide a minimum of 3 online student practice tests for each grade level and content area with a minimum of ten questions to allow students and test administration staff to become familiar with the secure browser online testing environment, and how to verify proper configuration and functioning of the network and devices, prior to a live test administration. Working with NYSED, the contractor will eliminate items from the existing practice tests that are aligned solely to the current standards and replace with items aligned to the New York State Next Generation Learning Standards drawing from recently released operational items beginning in year one of this contract term. In subsequent years, the contractor will provide one additional practice test per grade level and content area annually using recently released operational items.

The contractor will be required to provide web-based online question sampler tests that will allow parents, students and educators to access. The web-based online sampler tests must be duplicated and accessible on the contractor’s secure browser allowing students to experience computer-based testing in the same environment they will use for operational CBT. All students must have the opportunity to practice with and become familiar with the online tools and navigation available before sitting for the operational computer-based tests.

The format of the web-based online question sampler and secure browser practice tests must mirror that of operational tests. The multiple-choice questions will each feature four answer choices. The four answer choices must be rendered by the contractor in such a manner that all four of the answer choices will be visible at the same time to the student on the computer screen without the student having to scroll horizontally or vertically. For some multiple-choice questions, it may be necessary for the contractor to render the four answer choices in a two by two column arrangement. Additionally, empty constructed-response boxes and mathematical symbol boxes need to be provided by the contractor in the practice test environment to allow schools to familiarize students with the CBT platform in answering questions provided by the teacher. Test questions must reflect the New York State Next Generation Learning Standards on the web-based online question sampler and in the contractor’s secure browser practice tests.

Each practice test need not be as long as an actual operational test but should include a sample of previously released items representing item types and should be appropriate for familiarizing students with each system feature they will need to use during operational testing. These items will be taken from NYSED-developed sample items or previously administered exams. NYSED and the contractor will work collaboratively to determine the appropriate items to be included on the practice tests.

For the Mathematics practice tests, in addition to the regular English edition, the contractor will be required to provide multiple alternate language practice tests in Chinese (simplified), Haitian Creole, Korean, Russian, and Spanish. The translated content for these online practices must encompass all text content embedded in its CBT platform, such as online tools, to be translated into and presented to students in the same target languages.

The contractor will be required to provide the same online accommodations that are available to NYS students during operational testing in the web-based question sampler and the online student practice tests for each grade level and content area to allow students and test administration staff to become familiar with the secure browser online testing environment.62

RFP #20-004

It is NYSED's expectation that the online practice tests will be available throughout the school year beginning with three months prior to the spring 2021 operational and field test administration. NYSED also reserves the right to have the contractor input updated and expanded content for the CBT ELA and math practice tests no more frequently than once annually over the course of the contact term

Training and Technical Support

The contractor must provide local schools and districts with clear and complete training materials and complete technical support to enable them to successfully administer tests using the online delivery system.

The following types of system documentation shall be written, maintained, and provided to NYSED and local schools and school districts. The following represents a minimum requirement; additional documentation types and formats may be proposed.

1. System User’s Guide that fully details the functionality of the online testing system for an end user in a school or district

2. System Infrastructure Guide that details the minimum and recommended technical specifications and configurations needed to successfully access the online testing system and deliver online tests

3. System Training Workbook that provides step-by-step details for how to complete the most commonly needed tasks in the online testing system

4. System Update Notifications that detail the specifics of newly deployed system changes. Such information also should be included in the previous documents; however, this documentation will summarize system changes for advanced users who would not likely reread a complete user’s guide.

5. Set-up and Installation Guide that details a high-level overview, guidelines for deployment and implementation, and troubleshooting tips.

6. Develop and distribute transition communication to educational organizations.

7. Scoring manual that details guidelines, information, and procedures to facilitate scoring of operational tests.

Prior to test administration, the contractor must provide written training materials for schools and districts. In addition, the contractor must plan and host at a minimum six regional Fall training sessions and six regional Winter training sessions annually for schools and districts in person. Each training session shall accommodate both in-person and virtual (video and online) participation. The contractor will also be required to record the training session(s) and to make asynchronous training available online featuring material from the recorded sessions. In addition, the contractor must provide a minimum of four scoring training sessions to participating schools and districts in person or via webinar, or a combination of both. Furthermore, the contractor will develop and host a minimum of 4 video tutorials about transition and enhancements introduced yearly.

In addition to training materials and system documentation, the contractor must provide a telephone support helpline for schools to contact during practice testing, field testing, operational testing, and online scoring. Telephone support must be available each business day from 7 am – 7 pm EST during critical periods and from 7:30 am – 4:30 pm EST during the duration of the contract to answer

63

RFP #20-004

calls related to the use of the online delivery and scoring systems with no longer than a two-minute wait time. School personnel must be able to communicate by email and fax to the contractor.

The contractor shall provide NYSED with a report showing the number of calls received each day, the concerns or issues prompting the calls, their escalation, if necessary, their resolution, the number of remaining open tickets per time period, and the average wait time for a response, the duration of calls, and the total time from start of the call to resolution of the issue.

Should the State choose to include Component 2 as part of the contract resulting from this RFP, these training sessions and video tutorials should incorporate the Grades 5 and 8 Science Tests into the training materials and video tutorials that the contractor is providing for the ELA and Mathematics Tests and need not be discrete from those.

Computer-Based Testing Hosted Solution, Minimum Technical Specifications

The hosted solution for CBT offered by the contractor must meet all NYS education laws and regulations pertaining to data privacy which can be found in Appendices R, S, and S1 of this RFP. The solution’s online testing component must be fully compatible with all student testing devices most often purchased by NYS schools, including but not limited to, iPad, Chromebooks, Windows laptops and personal computers, and Mac laptops and personal computers. The contractor must provide NYSED with specific browser(s) that are compatible with the contractor’s online test delivery and administration system that operates on and functions as designed while meeting minimum technical requirements (See Attachment A). The contractor’s proposed hosted solution for CBT must support the release of new testing devices, operating systems, and browsers or communicate otherwise.

Simulation Testing

During the winter months, the contractor will conduct with NYS schools two simulations per year of the contractor’s testing and administration online system in an effort to ensure a successful online operational testing administration. Simulation testing should enable authorized NYS districts, schools, students, and teachers to perform tasks in a simulated testing environment which mirrors its usage during operational testing by accessing the test delivery and administration system. The contractor should propose methodology for conducting the simulation and must provide a report of the system usage detailing the number of practice tests submitted and the capability to flow this data through the contractor’s servers. These simulations will serve as a part of its contractor’s ability to perform load testing on the system. Each simulation test will provide opportunities for schools to evaluate and confirm technology readiness including adequate bandwidth, wireless connectivity, and functionality of devices. Furthermore, the contractor will provide NYS students and teachers the opportunity to gain exposure and familiarity with the test delivery and administration systems.

Comparability of Testing Modes (PBT versus CBT) for Grades 3-8 ELA and Math

As more school’s transition to computer-based testing, some students will continue to use a paper and pencil test. Therefore, it is essential that both delivery modes are comparable, and students are neither advantaged nor disadvantaged by the mode of test taken.

The contractor will be responsible for providing any required data for conducting the research and analysis needed to verify such comparability. One or more research studies must be conducted by the contractor that documents the comparability of the test under PBT and CBT administration

64

RFP #20-004

conditions. The contractor must produce a report of the comparability study for NYSED and must apply any adjustments to the final scale scores as agreed upon by NYSED.

Liquidated Damages Related to CBT for Grades 3-8 ELA and Mathematics


General timelines for deliverables and services under this Contract are set forth in this RFP.

Key Deliverables: It is hereby agreed that, without limitation to other rights and remedies of the State under this contract, the State shall be entitled to liquidated damages for the contractor’s failure to deliver the CBT platform for the Grades 3-8 ELA and Math Tests:

1. Uninterrupted access to the secure CBT platform for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time for the entire test administration period.

CBT Platform: The term “CBT platform” means the test administration and test delivery systems by which schools administer the tests to students using computer devices. The term includes any servers on which the system operates. The term test administration period includes initial test administration dates as well as makeup dates. The State will be entitled to liquidated damages of any school day on which the CBT platform is not accessible and fully functional for schools or the State for 20 or more minutes during the test administration window. For each day during the test administration period that the CBT platform is inaccessible and not fully functional for 20 or more minutes, the State is entitled to liquidated damages in the amount of $50,000 for the first day and then $75,000 for each additional day during which the test delivery system is inaccessible and not fully functional for schools or the State for 20 or more minutes.

2. Uninterrupted access to online testing accommodations for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time for the entire test administration period.

The full selection of online testing accommodations, both for Students with Disabilities and ELLs/MLLs as described in the section Component 1b: Computer-Based Testing and Scoring, must be available, without interruption and during the entire test administration period, to all NYS eligible students requiring them. The State will be entitled to liquidated damages for any school day on which the CBT system is functioning continuously but one or more of the testing accommodations in the CBT platform are inaccessible or not fully functional for schools, as follows: for each day during the test administration period, that an accommodation tool is inaccessible or not fully functional for 20 or more minutes, the State is entitled to liquidated damages in the amount of $50,000 for the first day and then $75,000 for each additional day during which an accommodation tool is inaccessible or not fully functional for schools for 20 or more minutes.

65

RFP #20-004

3. Uninterrupted access to the test administration portion of the contractor’s CBT platform for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time for the six weeks prior to each test administration period.

The term “test administration system” includes the portion of the CBT platform whereby schools and districts can set up the testing environment for their students and schools. For each day that the test administration portion of the CBT platform is inaccessible or not fully functional for a period of three hours or more during the six week period prior to the test administration window established by NYSED for the Grades 3-8 ELA and Math Tests, the State is entitled to liquidated damages in the amount of $10,000 for the first day and then $25,000 for each additional day during which the system is inaccessible or not fully functional for three hours or more.

4. Uninterrupted access to the secure scoring platform for CBT for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time during the one week prior to and for every school day during the scoring window.

The term “scoring platform” refers to the portion of the CBT platform provided by the contractor for schools, districts, and private contractors contracted by a school to use to score the constructed-responses submitted by students taking the tests with CBT. For each day during the week prior to the scoring window for the Grades 3-8 ELA and Mathematics Tests that the scoring platform is inaccessible or not fully functional for three hours or more, the State is entitled to liquidated damages in the amount of $10,000 for the first day and then $25,000 for each additional day during which the system is inaccessible or not fully functional for three hours or more.

For each day during the scoring window for the Grades 3-8 ELA and Math Tests that the scoring platform is inaccessible or not fully functional for 30 or more minutes, the State is entitled to liquidated damages in the amount of $25,000 for the first day and then $50,000 for each additional day during which the system is inaccessible or not fully functional for 30 or more minutes.

Component 1c: Printing, Shipping, Collection and Return of Grades 3-8 ELA and Math Operational Tests Materials and Liquidation of Damages Related to Printing, Shipping and Collection of Paper-based Grades 3-8 ELA and Math Operational Materials

While New York State continues to make the transition to computer-based testing, the contractor must be prepared to provide paper-based test materials to schools within every local school district. It is anticipated that the percentage of schools and students taking the CBT will increase each school year of this contract and additionally, due to the embedding of field test questions on computer-based operational forms, the need for multiple paper-based forms will be reduced to the extent that only one form of the paper-based tests will be needed over the course of the contract. While the numbers for paper-based testing are expected to be reduced dramatically over the course of the contract, the contractor should anticipate that some schools and individual test-takers may continue to use PBT, due to lack of school capacity to participate in CBT or due to individual student needs. The contractor must be flexible and prepared for the tests to be administered by schools via both PBT and CBT in unknown proportions.

66

RFP #20-004

Component 1c services require printing, shipping, collection, and return of printed test forms for the Grades 3-8 Operational Tests for those schools who continue to choose to administer the Grades 3-8 ELA and Mathematics Tests on paper rather than computer. The staffing plan proposed by the contractor must identify key staff who will be responsible for managing each of the areas of printing, shipping, collection, and return of printed test forms for the Grades 3-8 Operational Tests. The contractor must provide the staffing needed to maintain a high level of quality assurance and customer service throughout the duration of the contract including peak levels leading up to and during test administration dates.

Printing of Regular English, Large Type and Translated Editions of Operational Tests

Based on specifications received from NYSED, the contractor will manage printing, shipping, and collection of the test materials for each grade and subject, in a manner consistent with NYSED’s specifications and expectations. For the Grades 3-8 ELA and Mathematics Tests, final determinations of quantities will be proposed by the contractor to NYSED in January of each contract year, based on an order file provided electronically by NYSED, containing school shipping and contact information and order information (including quantities of students testing by CBT and PBT, by content area, and by grade). The actual print quantities that were produced for this program for the spring 2019 operational tests can be found in Attachment L .

In the contractor’s cost proposal submission for this RFP, the contractor provides costs for the printing, distribution, and collection-related deliverables associated with the operational tests based on the assumption of printing 100% of the volume indicated in Attachment L. The cells for these deliverables are highlighted in yellow in the Cost Proposal: Schedule of Deliverables. However, when submitting invoices for these deliverables, the contractor will bill NYSED for final printing, distribution, and collection costs based on the actual approved print quantities for that contract year, by applying the following formula: for every 1% reduction in the total print volume page count, as compared to the baseline in Attachment L, the cost will be reduced by .9%, up to a maximum reduction in printing, distribution, and collection costs by 54%. For example, if the actual mutually agreed upon Total Print Page Count for a given contract year is 10% less than that shown in Attachment L, the contractor will invoice NYSED at 9% below the costs contained in the Cost Proposal: Schedule of Deliverables for the printing, distribution, and collection- related deliverables for that contract year.

RevisedPrint Reduction Quantities

100% 80% 60% 40%

Total Pages 205,594,980 164,475,984 123,356,988 82,237,992

Once the primary period for schools to place their online examination requests has closed, NYSED will provide to the contractor an order file containing each school’s request for both CBT and PBT. Once the contractor receives that file it will propose to NYSED, for NYSED’s approval, the quantities of each type of exam booklet to be printed. These quantities will provide an allowance for the high probability that in the months following the contractor’s receipt of this file, schools will submit Additional Materials Orders (AMOs) beyond those reflected in the file. The print quantities must be inclusive of all printing and shipping necessary to accommodate schools’ requests up through the administration of the exams, including both initial orders and AMOs.

Each grade and content area currently consists of two test sessions which are printed in two test booklets. A limited number of multiple-choice field-test items are embedded in each ELA and 67

RFP #20-004

Mathematics Operational Test Session 1. The contractor is expected to develop and print as many discrete operational Session 1 forms as are necessary to build the bank of future multiple-choice test questions. These operational Session 1 test forms will feature the same test questions in every position in each of the test forms with the exception of the embedded field test questions. The embedded field test questions vary by operational test form and do not contribute to students’ test scores. Over the course of the contract term, NYSED hopes to be able to increase reliance on multiple-choice field testing through CBT and decrease reliance on multiple-choice field testing through PBT. If in any contract year, CBT becomes the predominant mode (65% or more of the test takers) of testing for a grade and content area, the contractor will begin printing only one form for that grade and content area beginning with the next contract year.

All accommodated materials (Braille, Large Type, Alternative Languages for Mathematics only, and Test Read) will use the same form, dubbed Form A.

The contractor will arrange printing services for the Grades 3–8 test booklets for ELA and Mathematics, as well as the School Administrator’s Manual, PBT and CBT teacher directions, and scoring CDs for all participating schools. NYSED is hopeful that the proportion of test administrations by CBT will increase over the term of the contract, resulting in the need for less printing. If in any given school year covered by the contract term, the agreed upon total printing volume is less than 40% of the total volume shown in Attachment L, NYSED may elect to assume all responsibility for the printing, distribution, and collection of the required operational test printed test materials beginning with the next school year.

Those secure test materials the contractor produces for NYSED which must be returned to the contractor must be printed with a barcode security ID number, in both machine- and human-readable formats, that enables pinpoint tracking of materials at every stage and location. It also must allow the contractor to produce accurate reporting of unreturned materials, by school, to facilitate reconciliation and accounting.

After completion of the content review and approval process, the contractor will transfer the approved, packaged files to their printing facility or contractor via a secure FTP site.

Test booklets will be printed in black ink, on 42.5# white offset paper, using press-paste binding; all booklets will have a secure barcode number for tracking.

Regular edition test booklets will be shrink-wrapped in packages of 10 and 25 booklets.

In all contract years during which multiple operational printed forms are used, booklets will be bundled with spiraling of test forms in each package.

Spiraling of forms will occur at the classroom level.

All test book packages will be labeled with grade, content area, test book type, and the secure barcode range of the test booklets.

Printed Mathematics Tests will be translated into eight target languages other than English: Arabic, Bengali, Chinese (Traditional), Chinese (Simplified) Haitian Creole, Korean, Russian, and Spanish. Schools will need to place special orders for translated test materials. As required by NYSED, the contractor will also provide a printed English-version test booklet to accompany the translated mathematics test booklets for both PBT and CBT, on a one-to-one basis.

Braille and Large Type editions of the test are packaged by exact number ordered.

68

RFP #20-004

Test Read copies, for use in administering this accommodation to students testing via PBT and CBT, are packaged in packs of two, one for the student who has the accommodation of Test Read and one for the proctor reading the test to the student. Students who require an oral translation of a lower incidence language, one for which NYSED doesn’t provide a translated edition, also receive Test Read Copies in packets of two, one for the student and one for the proctor providing the oral translation.

For both PBT and CBT, the contractor will provide a Mathematics Reference Sheet for each grade level. In accordance with the NYSED requirements for this Reference Sheet, the contractor will:

For PBT, supply the Reference Sheet as a perforated sheet, located at the front of each mathematics test booklet, for students testing via paper.

For CBT, print as a single page and ship Reference Sheets for students testing online.

For CBT, print and ship copies of the one-page printed Mathematics Reference Sheets for online test-takers, in sufficient quantities, to the approximately 75 regional centers.

Provide copies in the kits for the Braille and large-type editions.

The contractor will also print and distribute the School Administrator’s Manual (1 per school), Teacher Directions for PBT and CBT (ELA and mathematics versions for Grades 3–5 and 6–8 for PBT and CBT; 8 editions of teacher’s directions total, 1 copy per 20 students), and Scoring Materials CDs for ELA and mathematics (1 per content area per school).

Shipping

NYSED will provide a district/school file with shipping and contact information, order information (including quantities of students testing by mode, by content area, and by grade), and any specific distribution/delivery details, so that the contractor may create shipping algorithms and orders. The following list provides details on the main distribution processes:

Materials will be packaged by school and shipped directly to schools or their approved storage site.

The contractor will provide traceable and trackable distribution of materials via UPS.

The contractor will supply backup inventory at regional centers, shipped separately, to facilitate rapid fulfillment if shortages occur.

All orders/packages will have packing lists and box content lists as appropriate.

The contractor will provide a Secure Materials Form, to summarize all secure barcode numbers for the test booklets being delivered. The contractor will verify the return of all test materials that NYSED designates as secure using this barcode approach, described in the Printing section.

Deliveries will be made in multiple shipments per subject, as detailed below

o Shipment 1: ELA Test Materials and Return Kits – Operational ELA Test Sessions 1 and 2 booklets, School Administrator’s Manual, Teachers Directions, and return kits containing all needed supplies for the return of ELA Test Session 1 booklets.

o Shipment 2: ELA Scoring Materials CD for scoring of constructed response items.

69

RFP #20-004

o Shipment 3: Mathematics Test Materials and Return Kits – Operational Math Test Sessions 1 and 2 booklets, including translated editions, School Administrator’s Manual, Teachers Directions, and return kits containing all needed supplies for the return of Mathematics Test Session 1 booklets.

o Shipment 4: Mathematics Scoring Materials CD for scoring of mathematics constructed response items.

Shipment of secure test materials and Return Kits for ELA and Mathematics must occur in a timeframe agreed upon by NYSED that takes into consideration religious holidays and school vacations. Shipment delivery dates for both ELA and Mathematics must occur over a three business-day period no sooner than one week prior to each of the first test administration dates and no later than four business days prior to each of the first test administration dates. Any exception must be the result of need based on school or religious holidays and must be agreed upon by NYSED.

The contractor must accommodate all schools’ printing and shipping requests up through the administration of the exams, including both requests placed in NYSED’s online examination request system and those orders that are placed through the contractor after NYSED’s online examination request system has closed. The contractor must provide a means for processing orders placed after the primary period for schools to place their orders in the NYSED online examination request system has closed.

Collection, Return, and Secure Destruction

Secure Handling of Returned Assessment Materials

The contractor will collaborate with NYSED to develop processes and materials return kits that are both tailored to the assessment and geared toward safeguarding the secure nature of the contents being sent back to the contractor, with the goal of achieving 100-percent accounting for all secure materials required to be returned. Highlights of this process include:

Supply tools for school staff to pack for return/arrange pick-up. The contractor will provide the appropriate kits necessary for the collection and return of ELA and Mathematics Test Session booklets as outlined by NYSED, including all labels, forms, and instructions. The return kits will contain pre-paid UPS return labels for schools and bills of lading for centralized locations. The contractor may plan that the boxes that were used to deliver the materials to schools may be re-used by schools to return secure materials as required. The contractor will send out additional return kits as needed by schools. Once properly packaged and sealed, the boxes will be ready to ship. For UPS returns, schools will contact the contractor’s preferred customer care representative at UPS to schedule pick-up.

Receive shipment and account for all returned boxes. The contractor will confirm that all the cartons that were shipped have arrived using barcodes, incoming scan data, and tracking information from the shippers to account for every piece. If there are issues, the contractor will follow up with the carrier and the school contact to initiate resolution.

Sort received materials. Once logged into receiving, the contractor will sort the cartons based on their color-coded labels and scan again to verify they have been sorted correctly. Materials are grouped as used/unused for continued processing, through storage. All contents are examined, to ensure that no inappropriate documents were sent back.

70

RFP #20-004

Verify quantities of materials. The contractor will confirm that receiving logs are correct and that all materials are in a condition suitable for storage. The contractor will apply NYSED-approved processes to mitigate incidental document damage and notify test coordinators of any unresolved issues, including missing items. At the end of this stage, materials are ready to be sent to storage, for one year. After that, they will be disposed of in accordance with NYSED-defined procedures.

Missing Materials Report/Support Locating Missing Materials

If, following check in, secure materials are determined to be missing, the contractor will provide a missing materials report and timeframe for resolving the issue. The contractor will generate an email to be sent to the district superintendents, superintendents of schools, and principals that notifies them of the missing materials. The contractor’s customer and technical support team, coordinating with the program management team, will work with schools to determine the location of the missing serial numbers. NYSED will schedule the alerts and frequency of delivery appropriately.

The contractor will work out a schedule with NYSED as to how final notifications are sent to schools and what the appropriate message will be for those that still have missing secure materials after the window for return to the contractor has closed. The contractor will provide finalized secure materials missing reports, in accordance with agreed-upon dates in the master schedule for each assessment.

Clarification of Contractor Responsibility

The contractor accepts full responsibility for ascertaining the correct quantities of exam materials needed to be printed to furnish to NYS schools the quantities of exams they requested. The contractor is also responsible, without additional expense to NYSED, for any reprinting and/or re-shipping that may be necessary due to errors in quantities printed and/or shipped.

Liquidated Damages Related to Printing and Shipping of Paper-based Operational Test Materials


Printing of Test Materials

NYSED expects that the contractor will exercise due diligence, such that 100 percent of the passages/items in the printed test booklets that are distributed to NYS schools by the contractor will be completely legible for the students. The contractor is expected to provide thorough quality control over the printing process to ensure that all test booklets shipped to schools are completely legible, as determined by NYSED at its sole discretion, on every test page. The contractor is also expected to make every effort to remedy any reported problems with the legibility of any test booklets by

71

RFP #20-004

immediately reprinting any test booklets for which it may be necessary and immediately shipping replacement test booklets to the affected schools.

If the final set of test booklets for any grade/subject test that are shipped to schools contain errors in the printing of between one or two test questions and/or, for ELA, related passage(s), owing to which causes NYSED to make adjustments in scoring the affected students’ tests, NYSED will reduce by twenty five percent (25%) the agreed upon remuneration to the contractor for the printing of that grade/subject test, as described below.

If the final set of test booklets for any grade/subject test that are shipped to schools contain printing errors in the printing of three or four test questions and/or, for ELA, related passage(s), owing to which causes NYSED to make adjustments in scoring the affected students’ tests, NYSED will reduce by fifty percent (50%) the agreed upon remuneration to the contractor for the printing of that grade/subject test, as described below.

If the final set of test booklets for any grade/subject test that are shipped to schools contain printing errors in the printing of between five to eight test questions and/or, for ELA, related passage(s), owing to which causes NYSED to make adjustments in scoring the affected students’ tests, NYSED will reduce by seventy five percent (75%) the agreed upon remuneration to the contractor for the printing of that grade/subject test, as described below.

If the final set of test booklets for any grade/subject test that are shipped to schools contain printing errors in the printing of nine or more test questions and/or, for ELA, related passage(s), owing to which causes NYSED to make adjustments in scoring the affected students’ tests, NYSED will reduce by one hundred percent (100%) the agreed upon remuneration to the contractor for the printing of that grade/subject test, as described below.

The remuneration to the contractor for the printing of each grade/subject test will be calculated based on the total deliverable of printing of the Grades 3-8 ELA and Mathematics Tests materials. To the extent that a cost reduction is provided to NYSED, based on the formula specified in this RFP for a reduction in the overall print volume as outlined in the contract, the amount of the liquidated damages will be based on the actual expected remuneration to the contractor for printing.

72

RFP #20-004

Shipping of Test Materials

NYSED expects that the contractor will exercise due diligence to ensure that all schools that submitted exam requests receive the quantities of exam materials they have requested within the delivery windows specified in the contract such that they will be able to administer the tests to their students during the specified administration/makeup period. The contractor is expected to provide thorough quality control over the packing and distribution process to ensure that all required test materials are distributed to schools during the contract-specified delivery windows. The contractor is also expected to closely monitor all shipments of exam materials using its carrier-provided shipment tracking online tools and to quickly resolve exceptions or mis-deliveries.

Resolution of problems detected may be achieved by the contractor through communication with and correction of mis-deliveries with its carrier or immediate reshipment of exam materials to the affected schools so as to prevent shortages of exam materials in schools.

If, owing to packing/distribution related shortage(s) of required exam materials in NYS schools, NYSED deems it necessary to extend the administration/makeup period for the Grades 3-8 ELA or Grades 3-8 Mathematics Tests by one or more days, NYSED will reduce the agreed upon remuneration to the contractor, for the packing/distribution of the applicable Grades 3-8 ELA or Mathematics Tests by five percent for each school day that NYSED found it necessary to extend the administration/makeup period.

NYSED Assumption of Printing, Shipping, Collection and Return

During the course of the contract term, NYSED may, by providing the contractor with six months advanced notice and at its sole discretion, elect to assume all responsibility for the printing, distribution, and collection of the required operational test printed test materials.

Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests

Background

Component 2 services, should the State choose to include this component as part of the contract resulting from this RFP, requires the contractor to provide the same computer-based testing platform that includes a test delivery system, test administration system, and scoring platform for the NYSED-Developed Grades 5 and 8 Science Tests as provided for the Grades 3-8 ELA and Mathematics Tests. As such, the system must meet all requirements stated for the contractor hosting of the NYSED-developed Grades 3-8 ELA and Math Tests.

It is expected that each school year approximately 190,000 students in approximately 2,875 schools will be participating in the Grade 5 Science Test and approximately 125,000 students in approximately 1,750 schools will be participating in the Grade 8 Science Test. The NYS Science Tests in Grades 5 and 8 will most likely include both a hands-on performance test component and a written test component. The written test components of both tests will include both multiple-choice and constructed-response questions. It is anticipated that schools will be given the option of administering the written test component of the Grades 5 and 8 Science Tests with either CBT or PBT.

73

RFP #20-004

For Component 2, the contractor will be interacting with and providing services only to those schools that elect to administer the science field tests and operational tests via CBT and only for the written test component of these two assessments. For those schools that elect to administer the Grades 5 and/or 8 Science written tests via computer, NYSED will be responsible for printing and distributing to participating schools any ancillary materials, such as CBT Science teacher’s directions, that may be necessary to provide to schools in printed form to support the schools’ CBT administration of these tests. However, the contractor will be expected to assist NYSED with developing the content of the ancillary materials that explains to school administrators how their faculty and students will use the contractor’s CBT administration and scoring systems and platform.

A goal of computer-based field testing for the Science written tests is to reduce the amount of school/student participation that is necessary in stand-alone field testing by field testing as many of the future multiple-choice (MC) science test items through embedding them in computer-delivered operational test forms. With this goal in mind, the contractor will be expected to administer, starting with the 2022 operational tests, as many as 12 written test operational test forms for each of the two grade levels. These forms will include a maximum of eight discrete embedded field test multiple-choice questions within each operational test form. The embedded field test questions will be in the same position in each of the operational test forms. As with the Grades 3-8 ELA and Math Tests, the contractor should plan for all computer-delivered operational test forms for Grades 5 and 8 Science written tests to be spiraled within each exam room.

The contractor must provide a staffing plan that identifies key staff who will be responsible for managing the contractor’s CBT-hosting of the NYSED-developed Grades 5 and 8 Science Written Test field test and operational test forms. The contractor must have the staffing levels needed to maintain a high level of quality assurance and customer service throughout the duration of the contract including peak levels leading up to and during the Science test administration and scoring dates.

Timeline for CBT Services to be Provided by the Contractor for the Grades 5 and 8 Science Tests

Activity Months/Year

Year 1: July 1, 2020 – June 30, 2021

Develop/Customize Online Test Delivery Platform to be Used by Students for the Online CBT Science Practice Tests and Spring 2021 Field Tests

Jul 20-Dec 20

Customize Online Test Administration System to be Used by School Administrators for the Spring 2021 Science Field Tests Jul 20-Dec 20

Host Ongoing Online CBT Science Practice Tests on Contractor's Website with Sample Test Forms Developed By NYSED Jan 21-June 21

Maintain CBT Helpline for Full Duration of the Spring 2021 Science Field Testing Period May 21

Provide to NYSED Data Files for Student Responses to MC Field test Questions and Digital Files for NYSED to Rate Student Field Test Responses to CR Items

Jun 21

Year 2: July 1, 2021 – June 30, 2022

Develop/Customize Online Test Delivery Platform to be Used by Students for Spring 2022 Science Operational and Field Testing Jul 21-Dec 21

74

RFP #20-004

Customize Online Test Administration System to be Used by School Administrators for 2022 Science Field and Operational Testing Jul 21-Dec 21

Develop/Customize CBT Scoring Platform for Local Scoring Consortia to Use for Scoring Student Constructed Responses for 2022 Grades 5 and 8 Science Operational Tests

Jul 21-Dec 21

Host Ongoing Online CBT Practice Tests in Six Languages Including English on Contractor's Website using Sample Science Forms Developed and Translated by NYSED

Sept 21-June 22

Maintain CBT Helpline for Full Duration of 2022 Science Operational and Field-testing Periods May 22-June-22

Maintain Customer Support Helpline for Full Duration of 2022 Scoring Period for the Operational Tests May 22-June 22

Manage the Process for Scoring by Local Scoring Consortia from the Delivery of Responses to Scoring Consortia through to the Collection of Scores for all CBT-entered Responses for the 2022 Science Operational Tests

May 22-June 22


June 22

Provide to NYSED Data Records of the Student Responses to the MC Items and Local Scoring Consortia-Determined Scores for the Student Constructed Responses for the 2022 Grades 5 and 8 Operational Tests

June 22

Year 3: July 1, 2022 – June 30, 2023

Host a Secure Site at Which School Administrators and Other NYSED-Approved Parties Can Access and Print Their School’s Student CBT-Entered Responses and the Associated Scores for the Spring 2022 Grades 5 and 8 Science Operational Written Tests

July 22-June 23

Develop/Customize Online Test Delivery Platform to be Used by Students for 2023 Science Operational and Field testing July 22-Dec 22

Customize Online Test Administration System to be Used by School Administrators for Science Operational and Field testing July 22-Dec 22

Develop/Customize CBT Scoring Platform for Local Scoring Consortia to Use for Scoring Student Constructed Responses for 2023 Grades 5 and 8 Science Operational Tests

July 22-Dec 22

Host Ongoing Online CBT Science Practice Tests in Six Languages Including English on Contractor's Website Using Sample Forms Developed and Translated by NYSED

July 22-June 23

Maintain Customer Support Helpline for Full Duration of 2023 Scoring Period for the Grades 5 and 8 Science Operational Tests May 23-June 23

Maintain Scoring Helpline for Full Duration of 2023 Scoring Period for the Operational Science Tests May 23-June 23

Manage the Process for Scoring by Local Scoring Consortia from the Delivery of Responses to Scoring Consortia through to the Collection of Scores for all CBT-entered Responses for the 2023 Operational Science Tests

May 23-June 23


June 23

75

RFP #20-004

Provide to NYSED Data Records of the Student Responses to the MC Items and Local Scoring Consortia-Determined Scores for the Student Constructed Responses for the 2023 Operational Science Tests

June 23

Year 4: July 1, 2023 – June 30, 2024


July 23-June 24

Develop/Customize Online Test Delivery Platform to be Used by Students for 2024 Science Operational and Field testing July 23-Dec 23

Customize Online Test Administration System to be Used by School Administrators for Science Operational and Field testing July 23-Dec 23

Develop/Customize CBT Scoring Platform for Local Scoring Consortia to Use for Scoring Student Constructed Responses for 2024 Grades 5 and 8 Operational Science Tests

July 23-Dec 23

Host Ongoing Online CBT Science Practice Tests in Six Languages Including English on Contractor's Website using Sample Forms Developed and Translated by NYSED

July 23-June 24

Maintain Customer Support Helpline for Full Duration of 2024 Scoring Period for the Operational Tests May 24-June 24

Maintain Scoring Helpline for Full Duration of 2024 Scoring Period for the Operational Science Tests May 24-June 24

Manage the Process for Scoring by Local Scoring Consortia from the Delivery of Responses to Scoring Consortia through to the Collection of Scores for all CBT-entered Responses for the 2024 Operational Science Tests

May 24-June 24


June 24

Provide to NYSED Data Records of the Student Responses to the MC Items and Local Scoring Consortia-Determined Scores for the Student Constructed Responses for the 2024 Operational Science Tests

June 24

Year 5: July 1, 2024 – Nov 30, 2025


July 24-June 25

Develop/Customize Online Test Delivery Platform to be Used by Students for 2025 Science Operational and Field testing Jul 24-Dec 24

Customize Online Test Administration System to be Used by School Administrators for Spring 2025 Operational and Field testing Jul 24-Dec 24

Develop/Customize CBT Scoring Platform for Local Scoring Consortia to Use for Scoring Constructed Responses for 2025 Grades 5 and 8 Operational Science Tests

Jul 24-Dec 24

Host Ongoing Online CBT Science Practice Tests in Six Languages Including English on Contractor's Website using Sample Forms Developed and Translated by NYSED

July 24-June 25

76

RFP #20-004

Maintain CBT Helpline for Full Duration of 2025 Science Operational and Field-testing Periods May 25-June 25

Maintain Customer Support Helpline for Full Duration of 2025 Scoring Period for the Science Operational Tests May 25-June 25

Manage the Process for Scoring by Local Scoring Consortia from the Delivery of Responses to Scoring Consortia through to the Collection of Scores for all CBT-entered Responses for the 2025 Science Operational Tests

May 25-June 25


June 25

Provide to NYSED Data Records of the Student Responses to the MC Items and Local Scoring Consortia-Determined Scores for the Student Constructed Responses for the 2025 Science Operational Tests

June 25


July 25-Dec 25

Computer-Based Test Forms

The contractor must ensure that NYSED has sufficient time to review and quality check the computer-based test forms, comparable to the requirements specified in Component 1b of this RFP. The CBT operational tests for Grades 5 and 8 Science must be available for participating schools to securely access the encrypted test content between 7 am and 6 pm eastern time each day the schedule allows administration. Stand-alone field test forms must be available for participating schools to securely access the encrypted test content during the window schools administer the tests.

Testing System SpecificationsThe contractor is responsible for providing an online test delivery platform and an administration platform for the provision of the Science written tests and local scoring consortia scoring of the constructed-responses for the science written tests that meet all of the requirements specified in Component 1b of this RFP for the CBT administration and scoring of the Grades 3-8 ELA and Mathematics Tests. These requirements include but are not limited to test access control; administrative access control; security of test content and student data; desktop, laptop, and tablet security during testing; wireless networking; network availability; data interoperability; user interface including online student testing accommodations; data management and test management; security management; scoring platform; administrative system; system monitoring and reporting; system availability, maintenance, and updates; readiness verification process; computer-based testing hosted solution, minimum technical specifications; training and technical support.

Online Student Testing Accommodations

At a minimum, the contractor is expected to provide the same online student testing accommodations for the science tests as is specified in this RFP for the Grades 3-8 Mathematics Tests:

1. Text to Speech – Allows all text, mathematical symbols, special characters, as well as, any text embedded in graphics to be read aloud to the student which highlights the text as it is spoken.

77

RFP #20-004

The text, mathematical symbols, special characters, as well as, any text embedded in graphics must be highlighted word by word.

2. Reverse Contrast – Presents all text in white on a black background;

3. Answer Masking – Hides answers that the students select, allowing the student to focus on just the answers they want;

4. Background Color – Changes the color of the background and text based on the selection made;

5. Initial Zoom (with responsive design) - Provides any text, graphics, and/or images on the assessment enlarged in increments up to 300% initially when the student logs into the test delivery system. As indicated by responsive design the display must dynamically adjust to each zoom setting rather than require students to scroll horizontally;

6. Alternate Language – this accommodation feature provides current ELLs/MLLs with the option to test online in a student’s native language for New York State science stand-alone field and operational tests using an external or online alternate language keyboard. The translated content must encompass all text content embedded in its CBT platform, such as online tools, to be translated into and presented to students in the same target languages.

Online Question Sampler and Secure Browser Practice Tests

The contractor will be required to host multiple online student practice tests in the six required languages including English for each of the two grade levels of science tests to allow students and test administration staff to become familiar with the secure browser online testing environment, and how to verify proper configuration and functioning of the network and devices, prior to a live test administration. The contractor will use the same NYSED-provided Grades 5 and 8 Science practice test forms to populate the online question samplers and the secure browser practice tests. The complete content for the online student practice tests in science and the translations of this content will be provided to the contractor by NYSED. The contactor will submit to NYSED for its approval online instruction for students in how to use and navigate the practice test using the contractor’s CBT platform. The purpose of these CBT Science practice tests is to afford students the opportunity to experience computer-based testing in the same environment they will use for operational CBT. All students must have the opportunity to practice with and become familiar with the online tools and navigation before sitting for the operational computer-based science tests.

The contractor will be required to provide an alternate language secure browser for student CBT science practice tests in Chinese (Simplified), Haitian Creole, Korean, Russian, and Spanish to allow students and test administration staff to become familiar with the secure browser online testing environment. The translated content for these online practice tests will be furnished to the contractor by NYSED except that the contractor must arrange for the text content embedded in its CBT platform, such as online tools, to be translated into and presented to students in the same target languages.

The practice test forms developed by NYSED will not be as long as an actual operational written test but will include a sample of released items representing all of the item types that will be included on the operational written tests. The purpose of these practice tests is to familiarize students with each system feature they will need to use during operational science testing with CBT.

It is NYSED's expectation that the online practice tests will be available throughout the school year beginning with three months prior to the spring 2021 field test administration. NYSED also reserves 78

RFP #20-004

the right to have the contractor input updated and expanded content of the CBT science practice tests no more frequently than once annually over the course of the contact term.

Features of the CBT Platform for the Science Tests

The Grades 5 and 8 Science Tests will include multiple-choice and constructed-response items. The multiple-choice questions will each feature four answer choices. The four answer choices must be rendered by the contractor in such a manner that all four of the answer choices will be visible at the same time to the student on the computer screen without the student having to scroll horizontally or vertically. For some multiple-choice questions it may be necessary for the contractor to render the four answer choices in a two by two column arrangement.

To help enable students to solve some of the constructed-response questions, it may be necessary for the students testing on computer to be given access to an equation editor and to an online drawing tool similar to those the contractor will be providing as part of the computer test delivery platform for the Grades 3-8 Mathematics Tests. For both the Grades 5 and 8 Science Written Tests, the CBT platform must provide an online four-function calculator that the student may access in attending to all of the written test questions. This online calculator should not include any special features or functionality that is not ordinarily available to students using a hand-held four function calculator.

Training and Technical Support

The contractor must furnish for NYSED to provide to local schools and districts clear and complete training materials and complete technical support to enable them to successfully administer the science written tests using the online delivery system. The training materials developed by the contractor for this component may be the same materials as the contractor is providing for the ELA and Mathematics Tests as long as these materials have been adapted to address aspects of the online delivery system that are unique to the written science tests. NYSED will be responsible for the printing and distribution to schools of any materials that NYSED deems it necessary for schools to receive in printed form.

The following types of system documentation shall be written, maintained, and provided to NYSED and local schools and school districts. The following represents a minimum requirement; additional documentation types and formats may be proposed.

1. System User’s Guide that fully details the functionality of the online testing system for an end user in a school or district

2. System Infrastructure Guide that details the minimum and recommended technical specifications and configurations needed to successfully access the online testing system and deliver online tests

3. System Training Workbook that provides step-by-step details for how to complete the most commonly needed tasks in the online testing system

4. System Update Notifications that detail the specifics of newly deployed system changes. Such information also should be included in the previous documents; however, this documentation will summarize system changes for advanced users who would not likely reread a complete user’s guide.

5. Set-up and Installation Guide that details a high-level overview, guidelines for deployment and implementation, and troubleshooting tips.

6. Scoring manual that details guidelines, information, and procedures to facilitate scoring of the constructed responses for the operational tests.

79

RFP #20-004

Prior to science test administration, the contractor must provide written training materials for schools and districts. In addition, the contractor must plan and host a minimum of sixteen regional training sessions annually for schools and districts. These training sessions must include the CBT administration and test delivery systems as well as training in the use of the contractor’s scoring system and platform for schools and districts administering the Grades 5 and 8 Science Tests with CBT. In addition, the contractor will develop and host a minimum of 4 video tutorials about the transition to CBT and enhancements introduced yearly. All of these training sessions and video tutorial may be incorporated into the training materials and video tutorials that the contractor is providing for the ELA and Mathematics Tests and need not be discrete from those.

In addition to training materials and system documentation, the contractor must provide a telephone support helpline for schools to contact during practice testing, field testing, operational testing, and online scoring. Telephone support must be available each business day from 7am – 7pm EST during these critical periods and from 7:30 am – 4:30 pm EST during the duration of the contract to answer calls related to the use of the online delivery and scoring systems with no longer than a two-minute wait time. In addition, school personnel must be able to send in concerns by email and fax. The contractor shall provide NYSED with a report showing the number of calls associated with the CBT science practice tests, operational tests and field tests received each day, the concerns or issues prompting the calls and their resolution, the average wait time for a response, the number of remaining open tickets per time period, the duration of calls, and the total time from start of the call to resolution of the issue prompting the call.

During the scoring of the operational Grades 5 and 8 Science written tests, the contractor will be responsible for providing technical support for each local school-based science scoring site that is using the contractor-provided computer-based CR scoring platform. For the science tests the support provided by the contractor is limited to the use of the scoring system and should not provide any guidance to schools in the rating of responses, with the exception of how to record the scores. The contractor shall provide NYSED with a report showing the number of calls received each day, the concerns or issues prompting the calls and their resolution, and the average wait time for a response.

Computer-Based Testing Hosted Solution, Minimum Technical Specifications

The hosted solution for computer-based testing (CBT) offered by the contractor must meet all NYS education laws and regulations. The solution’s online testing component must meet minimum specifications to perform on all student testing devices allowable for purchase by NYS schools, including iPad, Chromebooks, Windows laptops and personal computers, and Mac laptops and personal computers. The solution’s online testing component must meet minimum specifications to perform on complimentary testing device operating systems for the last three released versions or older.

Provision to NYSED of the Raw Data Records of Students’ CBT-Entered Responses for Stand Alone Field Tests

Within three business days immediately following the close of the window for schools to administer Grades 5 and 8 Science field written tests, the contractor must provide to NYSED in a secure manner the raw data records of the student responses to the MC items and the digital files for NYSED to rate the Student Constructed Responses for the 2025 Science Operational Tests.

80

RFP #20-004

Provision to NYSED of the Raw Data Records of Students’ CBT-Entered Responses for Operational Tests

Within three business days immediately following the close of the scoring period for schools to administer Grades 5 and 8 Science operational written tests, the contractor must provide to NYSED in a secure manner the raw data records of the student responses to the MC items and the Local Scoring Consortia-Determined Scores for the Student Constructed Responses for the 2025 Science Operational Tests.

Hosting a Secure Site at Which School Administrators Can Access Their School’s Student CBT-Entered Constructed Responses for the Science Operational Written Tests

As a component of either its CBT administration system or its CBT scoring platform, the contractor must securely store and provide a secure platform for schools to access student constructed responses and the associated student scores for school administrators and other NYSED-approved parties for no less than one full year from the date of release on July 1 immediately following test administration. On this site, NYSED-approved viewers must also be enabled to view the selected students’ scores for the constructed responses and to print the responses.

Comparability of Testing Modes (PBT vs CBT) for Grades 5 and 8 Science

As schools’ transition to computer-based testing, some students will continue to use a paper and pencil for the Grades 5 and 8 Science written tests. Therefore, it is essential that both delivery modes are comparable, and students are neither advantaged nor disadvantaged by the mode of test taken.

For the Grades 5 and 8 Science Tests, NYSED, rather than the contractor, will be responsible for conducting any test mode comparability studies NYSED deems to be necessary. The contractor will be responsible for providing to NYSED, in a timely manner consistent with the requirements and timeline for delivery of CBT field and operational test data files, as specified above, any data for students who complete the written tests with CBT that NYSED reasonably requires to conduct the research and analysis needed to verify comparability of the two test modes, CBT and PBT. NYSED will apply any adjustments to the final scale scores of students who test with CBT that NYSED deems appropriate to ensure comparability between the two modes of testing.

Liquidated Damages Related to CBT Services for Grades 5 and 8 Science Tests


General timelines for deliverables and services under this contract are set forth in this RFP and are herein incorporated.

81

RFP #20-004

Key Deliverables: It is hereby agreed that, without limitation to other rights and remedies of the State under this contract, the State shall be entitled to liquidated damages for the contractor’s failure to deliver the CBT platform for the Grades 5 and 8 Science Tests, should NYSED choose to include Component 2 in the contract resulting from this RFP:

1. Uninterrupted access to the secure CBT platform for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time for the entire test administration period.

CBT Platform: The term “CBT platform” means the test administration and test delivery systems by which schools administer the science tests to students using computer devices. The term includes any servers on which the system operates. The term test administration period includes initial test administration dates as well as makeup dates. The State will be entitled to liquidated damages on any school day on which the CBT platform is inaccessible or not fully functional to schools or the State for 20 or more minutes during the science test administration window. For each day during the science test administration period that the CBT platform is inaccessible or not fully functional for 20 or more minutes, the State is entitled to liquidated damages in the amount of $50,000 for the first day and then $75,000 for each additional day during which the test delivery system is inaccessible or not fully functional for 20 or more minutes.

2. Uninterrupted access to online testing accommodations for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time for the entire test administration period.

The full selection of online testing accommodations, both for Students with Disabilities and ELLs/MLLs as described in the section Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests, must be available, without interruption during the entire science written test administration period, to all NYS eligible students requiring them. The State will be entitled to liquidated damages for any school day on which the CBT platform is accessible and largely functional but one or more accommodation tool(s) in the CBT platform are inaccessible or not fully functional to schools, as follows. For each day during the science written test administration period that an accommodation tool(s) is inaccessible or not fully functional for 20 or more minutes, the State is entitled to liquidated damages in the amount of $50,000 for the first day and then $75,000 for each additional day during which an accommodation tool(s) is inaccessible or not fully functional for 20 or more minutes.

3. Uninterrupted access to the test administration portion of the contractor’s CBT platform for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time for the six weeks prior to the science written test administration period.

The term “test administration system” includes the portion of the CBT platform whereby schools and districts can set up the testing environment for their students and schools. For each day that the test administration portion of the CBT platform is inaccessible or not fully functional for three hours or more during the six week period prior to the test administration window established by NYSED for the Grades 5 and 8 Science Written Tests, should Component 2 be included in the contract resulting from this RFP, the State is entitled to liquidated damages in the amount of $10,000 for the first day and then $25,000 for each additional day during which the CBT test administration system is inaccessible or not fully functional for three hours or more.

82

RFP #20-004

4. Uninterrupted access to the secure scoring platform for CBT for every school day from 7:00 a.m. to 6:00 p.m. Eastern Time during the one week prior to and for every school day during the science written test scoring window.

The term “scoring platform” refers to the portion of the CBT platform provided by the contractor for schools, districts, and private contractors contracted by a school to use to score the constructed-responses submitted by students taking the tests with CBT. For each day during the one week prior to the scoring window for the Grades 5 and 8 Science Written Tests that the scoring platform is inaccessible or not fully functional for three hours or more, should Component 2 be included in the RFP resulting from this contract, the State is entitled to liquidated damages in the amount of $10,000 for the first day and then $25,000 for each additional day during which the system is inaccessible or not fully functional for three hours or more.

For each day during the scoring window for the Grades 5 and 8 Science Written Tests that the scoring platform is inaccessible or not fully functional for 30 or more minutes, should Component 2 be included in the RFP resulting from this contract, the State is entitled to liquidated damages in the amount of $25,000 for the first day and then $50,000 for each additional day during which the system is inaccessible or not fully functional.

Requirements for All Components

Staffing Requirements and Program Management

The contractor must provide personnel with experience and expertise congruent with staff assignments for all aspects of the tasks outlined in the RFP. The contractor must ensure that staffing is commensurate with the project’s scope of work throughout the duration of the contract and immediately notify NYSED of any changes in staffing including any subcontractors. NYSED reserves the right to review and approve any proposed changes in key staffing and/or subcontractors.

Program Management Requirements

The contractor’s program management team must, at a minimum, include and maintain three fulltime program managers who:

1. are each fulltime employees of the entity, 2. have a Bachelor’s Degree or above, 3. will be devoted exclusively to the NYSED contract, 4. will be the central points of contact with NYSED for the contract,5. each have at least three years of experience managing large scale assessment projects from

conception through completion following industry recognized project management methodology,

6. each have previous experience with managing a large-scale P-12 assessment project and knowledge of testing procedures, and

7. each have strong organizational, managerial, and communication skills

The individuals identified as program managers must have sufficient authority across departments within the organization to ensure that the work of the contract has the necessary priority to be completed with the highest quality and on time.

83

RFP #20-004

A master’s degree or above, and project management certification through the Project Management Institute (PMI) as a Project Management Professional (PMP), or other recognized program management certification, is preferred.

Program Manager Duties

The program managers will:

1. Travel to Albany at least four times a year to meet with NYSED program staff. A start-up meeting must be held in Albany at the beginning of each contract year.

2. Develop and submit an annual detailed project plan to NYSED for review and approval. The purpose of the plan is to provide NYSED with an overall analysis of the methods the contractor will utilize to perform all aspects of the contract in the required timeframe. The contractor will not perform work on the project until NYSED has approved the contractor’s project plan.

3. Maintain accurate, up-to-date information of the current status of all contractor and subcontractor(s) work on the project, and timely communicate such information to NYSED.

4. Coordinate and participate in quarterly review meetings and weekly conference calls to discuss the project status and any issues related thereto. Within one week following each quarterly review meeting, the contractor must provide meeting minutes to NYSED. The contractor must also provide a written statement on all important decisions made during weekly conference calls.

5. Submit bi-weekly status reports electronically updating NYSED on all phases of the development and implementation of the project and progress made in accordance with the detailed project plans and contractual requirements. Such bi-weekly reports must include the following:

a. current detailed listing of all work and activities completed and in progress with corresponding dates;

b. a detailed list of upcoming work;c. a list of all information needed from NYSED in order for work to proceed;d. external dependencies, if any, which may affect the schedule, and the variance by

which they would affect the schedule; e. all dates must correspond to all timelines and schedules for deliverables; andf. a detailed listing of work and timelines will be provided in calendar view format

upon request by NYSED.

6. Submit all deliverables and other work products to NYSED in the manner designated by NYSED for review and approval prior to any release or distribution by the contractor. All materials must be appropriately proofed by the contractor for errors. The program manager must submit materials to NYSED according to a mutually agreed upon timeline to allow for appropriate review.

7. Oversee the development and implementation of changes as necessary to ensure that the projects remain within specified scope and are within time, cost, and quality objectives.

The contractor must provide a means for the program managers to be reached between 6 am and 10 pm EST Monday through Sunday during periods of operational testing.

Staffing Requirements for the Grades 3-8 ELA and Mathematics Test Development

For both ELA and Mathematics, the contractor must maintain a full staff consisting of a minimum of three fulltime Mathematics content specialists, one each for Grades 3–4, 5–6, and 7–8, and three fulltime ELA

84

RFP #20-004

content specialists, one each for Grades 3–4, 5–6, and 7–8 for a total of six content specialists. The contractor must also provide and maintain one fulltime development supervisor for the duration of the contract, who will oversee the development of both the ELA and Mathematics Tests.

The six content specialists and the development supervisor assigned to this project must each be fulltime employees of the entity and, must not have primary responsibility for any large-scale testing program in another state or with a consortium serving multiple states. The development supervisor must be an individual other than the six content specialists assigned by the contractor to this project. The development supervisor must have a master’s degree or higher, at least 24 college credits in either ELA or mathematics, and at least three years of experience teaching in one of these two content areas.

Each content specialist must have a master’s degree or higher, at least 24 college credits in the content area, and at least three years of experience teaching in the content area.

The contractor is precluded from assigning any staff (including per diems) who perform work for the contractor on developing customized tests for any school districts, charter, religious, independent schools in New York State from having any responsibilities for developing any of the components for the New York State Grades 3-8 ELA and Mathematics Tests.

Publications and Editorial Requirements

For the full duration of the contract term, the contractor must continuously provide at least two technical writers for this project. The technical writers will be the central point of contact with regard to the publication and production of all training materials, manuals, and communications having to do with the administration of both paper- and computer-based tests. The two technical writers must lead a publications team that includes, at a minimum, three editors.

Technical Writer Requirements

The two technical writers must each have:1. education at the Bachelor’s Degree level or above2. 5 years of experience in the field of technical writing/editing3. previous experience on an education project, preferably for Grades 3-8, and4. demonstrated technical writing ability

Technical Writer DutiesThe technical writers will

1. manage the work of the contractor’s publications team, which includes, at a minimum, three editors to ensure Quality Assurance/Quality Control for all NYSED publications

2. become familiar with NYSED’s testing procedures and policies as outlined in NYSED administrative publications, including the School Administrator’s Manual, Teacher’s Directions, and Educator Guides, as well as in the regular communications that go to schools and districts such as any memorandums, emails, web postings, etc.

3. become familiar with NYSED’s Style Guide which outlines NYSED’s style requirements for secure and nonsecure materials.

4. oversee the yearly updates to all administrative manuals such as the School Administrator’s Manual, Teacher’s Directions, and Educator Guides and work with the program manager, NYSED, and publications team to ensure revisions follow the necessary timelines to meet NYSED deadlines for publication.

85

RFP #20-004

5. oversee the publication of all CBT-related administrative materials, including but not limited to all webinars, guides, email and written communications, web postings, etc.

Business Analyst of Data Requirements

The contractor must provide at least one fulltime business analyst of data to oversee the data collection and reporting needs for the Grades 3-8 ELA and Mathematics Testing Program and for the Grades 5 and 8 Science Testing Program, should that be included as part of the contract resulting from this RFP.

The business analyst of data for this contract must:1. not have primary responsibility for any large-scale testing program in another state or with a

consortium serving multiple states 2. have at least three years of experience as a data analyst for a large-scale assessment

program (previous experience in a Grades 3–8 education project is preferred), and3. have a bachelor’s degree (a master’s degree or above is preferred)

Business Analyst of Data Duties

1. The business analyst of data will travel to the NYSED building in Albany at least four times a year to meet with NYSED program staff. A start-up meeting must be held in Albany at the beginning of each contract year.

2. The business analyst of data will have primary responsibility for coordinating the data needs for the Grades 3-8 ELA and Mathematics Testing Program and for the Grades 5 and 8 Science Testing Program, should that be included as part of the contract resulting from this RFP, ensuring adherence to the NYSED Office of Information and Reporting Services Business Rules.

3. Participate in weekly conference calls with NYSED throughout the year.4. Oversee the planning, coordination, and implementation for NYSED’s data needs with

NYSED, the contractor, the State Data Warehouse, the contractor’s technical staff, and other NYSED-designated partners.

Notification Procedures In the event that a problem or potential problem arises with regard to security, quality, timeliness, or any other issue with respect to deliverables and services at any time during the contract term, regardless of when the problem arises, the program manager must immediately notify the Director of State Assessment, or his/her designee, via telephone and in writing of the issue and the vendor’s proposed solution and shall also include the issue and NYSED approved solution on any subsequent report(s).

In the event that a delay in performance occurs as a result of a force majeure, the affected party will contact the other party in writing as soon as the delay is known and provide a written contingency plan. The non-affected party acknowledges that the affected party will not be held liable for failure to perform any provision of the contract if such failure is caused by a force majeure. Should such events occur, the contractor will use financially reasonable efforts to overcome the difficulties and will resume work as soon as reasonably possible. Notwithstanding the foregoing, if the force majeure continues beyond thirty (30) days, the parties shall decide on an appropriate course of action that will permit fulfillment of the parties' objectives hereunder.

86

RFP #20-004

NYSED Approval NYSED shall have approval authority over all aspects of the Grades 3–8 ELA and Mathematics Testing Programs including, but not limited to, the following:

1. All material, products, and services produced by the vendor, including, but not limited to, all test items and passages, test forms (test booklets and answer documents), brochures, guides, student information labels and answer sheet identifiers, manuals, teacher’s directions, and any and all other printed materials produced for this project. In addition, all test forms produced in Braille or large type must be reviewed and approved by NYSED before final copies are provided to NYSED.

2. All development and operational plans for field testing items.

3. All plans and timelines for item development, item replenishment, scoring, and reporting.

4. All schedules, including but not limited to training schedules, rollout schedules, implementation schedules, scoring/reporting schedules and item review schedules.

5. All reports, including but not limited to, technical reports (ongoing and post-operational), and field test reports.

6. Any changes to project staff, including program managers, content, or research staff on and after award of the contract and commencement of work for the duration of the contract.

7. All communications and correspondence by the contractor with the media, the field, or entities other than NYSED, including to the districts, schools, and committee members.

Any changes to services or deliverables under this procurement must be in accordance with a written description clearly justifying the need for the change and identifying all rational for the change (Change Order). The New York State Office of the State Comptroller is the only entity that has the authority to approve the modification of an agreement between NYSED and a vendor. No work that is not already contracted for should be started by the vendor prior to its official approval by NYSED and the Office of the State Comptroller.

Ownership

NYSED is the sole owner of materials produced and data collected under the contract, including but not limited to:

all test item banks and items developed under the contract in all stages of development all New York State tests and test-related materials prepared for printing or posting on NYSED’s

website all materials prepared for training sessions or other presentations all research conducted by the contractor pursuant to the requirements of the contract all data related to New York State tests and test-takers

These materials and data can only be used by, or with permission of, NYSED. NYSED may, at its discretion, make public, and may publish on its web site, all examinations and related materials and data noted in the preceding paragraph.

The contractor must be prepared to deliver these materials and data to NYSED at any point during the term of the contract, even if materials are only partially completed. All materials and data are to be turned over to NYSED, in the format specified by NYSED, prior to the final payment to be issued under the contract.

87

RFP #20-004

Data Security , Data Privacy and Appropriate Use

The contractor must comply with all data security, data privacy and appropriate use laws, regulations, policies and procedures required by the State of New York and the NYSED, in accordance with the contract requirements of the Department. Such requirements include the Security Guidelines for New York State Assessment Program (Attachment M), Data Security and Privacy Plan (Appendix R), and Parents’ Bill of Rights (Appendix S). These security guidelines were developed by NYSED to set forth the management of secure materials and should be reflected in the contractor’s procedures.

Security of individual student data and test items, including the thorough retrieval and accounting for all secure field test materials, is essential. The contractor must adhere to all of NYSED’s security protocols regarding transmission and shipping of secure materials. The contractor must adhere to all of NYSED’s security protocols regarding transmission and shipping of secure materials. This includes secure shipment of all physical materials using a carrier with ground tracking capability.

The contractor must use encrypted files and design, host, and maintain a secure file transfer protocol (FTP) site as a means of file transfer. Access to the NYS test information on this site must be limited to the contractor and NYSED unless further sharing with other parties is authorized in writing by NYSED. Any other electronic transfer via e-mail, Internet, or facsimile (FAX) of individual student information or any secure test materials is not permitted unless authorized by NYSED to do so on a case-by-case basis. When shipping secure materials, the contractor must use a delivery service with online ground tracking capabilities. All shipments between the contractor and NYSED or the contractor’s subcontractors must be in locked boxes, which will be supplied by NYSED.

All confidential data must be stored on computer and storage facilities maintained within the United States using the strictest industry standards and state-of-the-art best practices, including appropriate firewalls and security measures.

All contractor staff and subcontractors having access to secure or confidential information, including but not limited to test questions, scoring materials, student data and personally identifiable information, shall sign a non-disclosure agreement provided by NYSED prior to receiving access to such materials. The contractor will maintain all signed agreements for the duration of the contract term and five years thereafter and will provide them to NYSED upon request.

Requirements of Education Law § 2-d

The contractor agrees to comply with FERPA and New York State Education Law § 2-d. The New York State Data Security and Privacy Plan (Appendix R), the Parents Bill of Rights (Appendix S) and the Attachment to the Parents’ Bill of Rights for Contracts Involving Certain Personally Identifiable Information (Appendix S-1) are included in 4. Assurances of this RFP, the terms of which are incorporated herein by reference, and which shall also be part of the Contract.

Promotion of Products and Services Outside the Contract

If the vendor also develops or publishes textbooks or practice tests for adoption within the State of New York, the vendor agrees that, when promoting to New York State customers such textbooks, ancillary materials, and/or practice tests, the vendor will not make any reference to the vendor having been awarded the contract or the vendor's performance of the services for NYSED contemplated by the contract.

88

RFP #20-004

If the vendor is under ownership with another entity that develops or publishes textbooks or practice tests for adoption within the State of New York: a) ("Related Publisher"), the vendor reiterates the restriction against disclosure as stated; b) the Related Publisher is restricted from providing any information to the vendor pertaining in any way to content uses, or proposed to be used, in the Related Publisher's development of textbooks or practice tests for adoption within the State of New York; and c) the Related Publisher is restricted, when promoting to New York State customers, the Related Publisher's practice tests and/or textbooks developed for adoption within the State of New York, from making any reference to the common ownership of the Related Publisher and the vendor or any reference to the vendor's performance.

89

RFP #20-004

Invoices and Payments

The contractor will furnish an invoice to NYSED on a monthly basis for the deliverables provided, and must include dates of services and an itemized list of activities and costs consistent with the approved schedule of deliverables contained in the executed contract. Payment(s) to subcontractor(s) should be indicated on the invoice and should list the subcontractors’ names, payment amount, and nature of services provided. Invoices with incomplete information will be returned to the contractor. Invoices should reflect only the deliverables that have been completed and submitted to NYSED. Payments will be made upon 100% completion of each deliverable and approval by NYSED.

Accessibility of Web-Based Information and Applications

The NYSED web accessibility policy applies to all internal or external web content and functionality whether developed by, maintained by, or offered either by NYSED or through a third-party vendor or open source. This policy ensures that all people with disabilities have an equal opportunity to participate in our benefits, programs, and services through web content. Any documents, web-based information and applications development, or programming delivered pursuant to the contract or procurement, will comply with New York State Education Department IT Policy NYSED-WEBACC-001, Web Accessibility Policy as such policy may be amended, modified or superseded, which requires that state agency web-based information, including documents, and applications are accessible to persons with disabilities. Documents, web-based information and applications must conform to NYSED-WEBACC-001 as determined by quality assurance testing. Such quality assurance testing will be conducted by NYSED employee or contractor and the results of such testing must be satisfactory to NYSED before web-based information and applications will be considered a qualified deliverable under the contract or procurement.

Subcontracting Limit

Subcontracting will be limited to thirty percent (30%) of the total contract budget. A higher subcontracting limit will be allowed only when a bidder is proposing to subcontract for the provision of the computer-based testing platform. In this case, the subcontracting limit will be increased to fifty percent (50%) of the total contract budget. In all other cases, the subcontracting limit will be thirty percent (30%).NYSED reserves the right to approve all subcontractors. Subcontracting is defined as non-employee, direct, personal services and related incidental expenses, including travel. For purposes of this RFP, the subcontracting limit does not include expenses related to printing, shipping, or the purchase of equipment, supplies, and materials.

For contractors using subcontractors, a Vendor Responsibility Questionnaire and a NYSED vendor responsibility review are required for a subcontractor where:

the subcontractor is known at the time of the contract award; the subcontractor is not an entity that is exempt from reporting by OSC; and the subcontract will equal or exceed $100,000 over the life of the contract;

For additional information about Vendor Responsibility, see the “Vendor Responsibility” section contained in 3.) Evaluation Criteria and Method of Award of this RFP.

If the contractor proposes to change subcontractors during the contract period, NYSED must be notified prior to the change. NYSED reserves the right to reject any replacement subcontractors 90

http://www.nysed.gov/webaccess

RFP #20-004

proposed by the contractor and reserves the right to approve all changes in subcontractors. The Subcontracting Form located in the Submission Documents must be updated annually and submitted to NYSED. Using this form, the contractor must also report to NYSED, on an annual basis, actual expenditures incurred for all subcontractors and indicate which subcontracting costs are associated with M/WBE.

Consultant Staff Changes

The Contractor will maintain continuity of the Consultant Team staff throughout the course of the contract. All changes in staff will be subject to NYSED approval. The replacement Consultant(s) with comparable skills will be provided at the same or lower hourly rate.

Contract Period

NYSED will award one contract pursuant to this RFP. The contract resulting from this RFP will be for a term anticipated to begin July 1, 2020 and to end November 30, 2025.

Electronic Processing of Payments

In accordance with a directive dated January 22, 2010 by the Director of State Operations - Office of Taxpayer Accountability, all state agency contracts, grants, and purchase orders executed after February 28, 2010 shall contain a provision requiring that contractors and grantees accept electronic payments. Additional information and authorization forms are available at the State Comptroller's website.

M/WBE AND EQUAL EMPLOYMENT OPPORTUNITIES REQUIREMENTS CONTRACTOR REQUIREMENT AND OBLIGATION UNDER NEW YORK STATE EXECUTIVE LAW, ARTICLE 15-A (PARTICIPATION BY MINORITY GROUP MEMBERS AND WOMEN WITH RESPECT TO STATE CONTRACTS)

In an effort to eradicate barriers that have historically impeded access by minority group members and women in State contracting activities, Article 15-A, of the New York State Executive Law §310-318, (Participation By Minority Group Members and Women With Respect To State Contracts) was enacted to promote equality of economic opportunities for minority group members and women.

The New York State Education Department (“NYSED”) has enacted its policies Equal Opportunity, Non-Discrimination and Affirmative Action and on Minority and Women-Owned Business Enterprise Procurements, consistent with the requirements as set forth under the provisions of Article 15-A (the “Article”) incorporated by reference, requiring Contracting Agencies to implement procedures to ensure that the “Contractor” (as defined under Article 15-A, §310.3 shall mean an individual, a business enterprise, including a sole proprietorship, a partnership, a corporation, a not-for-profit corporation, or any other party to a state contract, or a bidder in conjunction with the award of a state contract or a proposed party to a state contract, complies with requirements to ensure Equal Employment Opportunities for Minority Group Members and Women, in addition to providing Opportunities for Minority and Women-Owned Business Enterprises on all covered state contracts.

In keeping with the intent of the Law, it is the expectation of the Commissioner and the responsibility of all contractors participating in and/or selected for procurement opportunities with NYSED, to fulfill their obligations to comply with the requirements of the Article and its implementing regulations.

91

http://www.osc.state.ny.us/epay/index.htm

http://www.osc.state.ny.us/epay/index.htm

RFP #20-004

In accordance with these requirements, the contractor hereby agrees to make every good faith effort to promote and assist the participation of certified Minority and Women-Owned Business Enterprises (“M/WBE”) as subcontractors and suppliers on this project for the provision of services and materials in an amount at least equal to the M/WBE goal (Included in the procurement document) as a percentage of the total dollar value of this project. In addition, the contractor shall ensure the following:

1. All state contracts and all documents soliciting bids or proposals for state contracts contain or make reference to the following provisions:

a. The contractor will not discriminate against employees or applicants for employment because of race, creed, color, national origin, sex, age, disability, marital status, gender, religion, veteran status, sexual orientation, genetic disposition or carrier status and will undertake or continue existing programs of affirmative action to ensure that minority group members and women are afforded equal employment opportunities without discrimination.

For purposes of the Article, affirmative action shall mean recruitment, employment, job assignment, promotion, upgrading, demotion, transfer, layoff or termination and rate of pay or other forms of compensation.

b. The contractor shall request each employment agency, labor union, or authorized representative of workers with which it has a collective bargaining or other agreement or understanding, to furnish a written statement that such employment agency, labor union or representative will not discriminate on the basis of race, creed, color, national origin, sex, age, disability, marital status, gender, religion, veteran status, sexual orientation, genetic disposition or carrier status and that such union or representative will affirmatively cooperate in the implementation of the contractor’s obligation herein.

c. The contractor shall state in all solicitations or advertisements for employees, that, in the performance of the State contract, all qualified applicants will be afforded equal employment opportunities without discrimination because of race, creed, color, national origin, sex, age, disability, marital status, gender, religion, veteran status, sexual orientation, genetic disposition or carrier status.

2. The contractor will include the provisions of subdivision one of this section in every subcontract as defined under §310.14, except as provided under §312.6 of the Article, in such a manner that the provisions will be binding upon each subcontractor as to work in connection with the State contract.

3. Contractors or subcontractors shall comply with the requirements of any federal law concerning equal employment opportunity, which effectuates the purpose of this section.

4. Contractors and subcontractors shall undertake programs of affirmative action and equal employment opportunity as required by this section2. In accordance with the provision of the Article, the bidder will submit, with their proposal, Staffing Plan (EEO 100).

5. Certified businesses (as defined under Article 15-A, §310.1 means a business verified as a minority or women-owned business enterprise pursuant to §314 of the Article) shall be given the opportunity for meaningful participation in the performance of this contract, to actively and affirmatively promote

2 Notice – Contractors are provided with notice herein, NYSED may require a contractor to submit proof of an equal opportunity program after the proposal opening and prior to the award of any contract. In accordance with regulations set forth under Article 15-A §312.5, contractors and/or subcontractors will be required to submit compliance reports relating to the contractor’s and/or subcontractor’s program in effect as of the date the contract is executed.

92

RFP #20-004

and assist their participation in the performance of this contract, so as to facilitate the award of a fair share of this contract to such businesses3.

6. Contractor shall make a good faith effort to solicit active participation by enterprises identified in the Empire State Development (“ESD”) directory of certified businesses. The contractor must document its good faith efforts as set forth in 5 NYCRR 142.8. This document, Contractors Good Faith Efforts, can be found in the M/WBE Submission Documents.

7. Contractor shall agree, as a condition of entering into said contract, to be bound by the provisions of Article 15-A, §316.

8. Contractor shall include the provisions set forth in paragraphs (6) and (7) above, in every subcontract in a manner that the provisions will be binding upon each subcontractor as to work in connection with this contract.

9. Contractor shall comply with the requirements of any federal law concerning opportunities for M/WBEs which effectuates the purpose of this section.

10.Contractor shall submit all necessary M/WBE documents and/or forms as described above as part of their proposal in response to NYSED procurement.

11. The percentage goals established for this RFP are based on the overall availability of M/WBEs certified in the particular areas of expertise identified under this RFP. These goals should not be construed as rigid and inflexible quotas which must be met, but as targets reasonably attainable by means of applying every good faith effort to make all aspects of the entire Minority and Women-Owned Business Program work.

12. Contractor shall ensure that enterprises have been identified (M/WBE 102) within the Utilization Plan, and the contractor shall attempt, in good faith, to utilize such enterprise(s) at least to the extent indicated in the plan, as to what measures and procedures contractor intends to take to comply with the provisions of the Article.

13. Contractor shall upon written notification from NYSED M/WBE Program Unit as to any deficiencies and required remedies thereof, the contractor, within the period of time specified, will submit compliance reports documenting remedial actions taken and other information relating to the operation and implementation of the Utilization Plan.

14. Where it appears that a contractor cannot, after a good faith effort, comply with the M/WBE participation requirements, contractor may file a written application with NYSED M/WBE Program Unit requesting a partial or total waiver (M/WBE 101) of such requirements setting forth the reasons for such contractor’s inability to meet any or all of the participation requirements, together with an explanation of the efforts undertaken by the contractor to obtain the required M/WBE participation.

For purposes of determining a contractor’s good faith efforts to comply with the requirements of this section or be entitled to a waiver, NYSED shall consider at the least the following:

3 Should the contractor identify a firm that is not currently certified as an M/WBE, it should request that the firm submit a certification application Empire State Development by the deadline for submission of proposals for eligibility determination. It is the responsibility of the contractor to ensure that a sufficient number of certified M/WBE firms have been identified in response to this procurement, in order to facilitate full M/WBE participation.

93


http://www.esd.ny.gov/MWBE/Certification.html

http://www.esd.ny.gov/MWBE/Certification.html

RFP #20-004

I. Whether the contractor has advertised in general circulation media, trade association publications and minority-focused and women-focused media and, in such event;

a. Whether or not the certified M/WBEs which have been solicited by the contractor exhibited interest in submitting proposals for a particular project by attending a pre-bid conference; and

b. Whether certified businesses solicited by the contractor responded in a timely fashion to the contractor’s solicitations for timely competitive bid quotations prior to the contracting agency’s deadline for submission of proposals.

II. Whether there has been written notification to appropriate certified M/WBEs that appear in the Empire State Development website.

All required Affirmative Action, EEO, and M/WBE forms to be submitted along with bids and/or proposals for NYSED procurements are attached hereto. Bidders must submit subcontracting forms which:

1) fully comply with the participation goals specified in the RFP; OR

2) partially comply with the participation goals specified in the RFP, and include a request for partial waiver, and document its good faith efforts to fully comply with the percentage goals specified in the RFP; OR

3) do not include certified M/WBE subcontractors or suppliers, and include a request for a complete waiver, and document its good faith efforts to fully comply with the participation goals specified in the RFP.

All M/WBE firms are required to be certified by Empire State Development (ESD) or must be in the process of obtaining certification from ESD. Online Certification can be found at the New York State Contract System website.

Failure to comply with the requirements of Article 15-A as set forth under this procurement and in conjunction with the corresponding contract, will result in the withholding of associated funds and other enforcement proceedings set forth under Article 15-A.

94

https://ny.newnycontracts.com/FrontEnd/StartCertification.asp?TN=ny&XID=2029


RFP #20-0042.) SUBMISSION

Documents to be submitted with this proposal

This section details the submission document or documents that are expected to be transmitted by the respondent to the State Education Department in response to this RFP. New York State Education Department shall own all materials, processes, and products (documentation and other written materials and data) developed under this contract. Materials prepared under this contract shall be in a form that will be ready for copyright in the name of the New York State Education Department. Any subcontractor is also bound by these terms. The submission will become the basis on which NYSED will judge the respondent’s ability to perform the required services as laid out in the RFP. This will be followed by various terms and conditions that reflect the specific needs of this project.

In case of discrepancies between hardcopy and electronic Proposals, the original signed hardcopy shall take precedence.

Project Submission

The proposal submitted in response to this RFP must include the following documents:

1. Submission Documents—Two (2) copies (one bearing an original signature)2. Technical Proposal—Eight (8) copies (one bearing an original signature)3. Cost Proposal—Two (2) printed copies (one bearing an original signature, be sure to include

printed copies of all 6 worksheets contained in the Cost Proposal Workbook)4. M/WBE Documents—One (1) copy (bearing an original signature)5. Microsoft Office (CD or Flash Drive format)—One (1) electronic version with the submission,

technical, cost, and M/WBE proposals. Please place the CD-ROM or Flash Drive in a separate envelope. Please note: the cost proposal should be submitted in Excel format, using the template provided by NYSED.

The proposal must be received by January 23, 2020 by 3:00 PM at NYSED in Albany, New York.

Bidders must submit one technical proposal to include all services required for both Components 1 and 2, and a cost proposal containing a budget for Years 1-5 Component 1 and Years 1-5 Component 2. Component 1 is for services detailed in this RFP associated with the Grades 3–8 ELA and Mathematics Tests development, printing and dissemination for schools choosing to administer the tests with PBT, and computer-based test administration and online scoring for schools choosing to administer the tests with CBT. Component 2 is for the services and support detailed in this RFP necessary to enable all schools statewide to have the choice of administering the NYSED-developed Grades 5 and 8 Science Test stand-alone field tests and the operational tests on school-owned computer devices and to enable schools choosing CBT to score the constructed-response questions on a vendor-provided computer scoring platform

Proposals should be prepared simply and economically, avoiding the use of elaborate promotional materials beyond those sufficient to provide complete presentation. If supplemental materials are a necessary part of the proposal, the bidder should reference these materials in the technical proposal, identifying the document(s) and citing the appropriate section and page(s) to be reviewed.

95

RFP #20-004The proposal must communicate an understanding of the deliverables of the RFP, describe how the tasks are to be performed and identify potential problems in the conduct of the deliverables and methods to identify and solve such problems.

Bidders should specify all details and dates required to evaluate the technical proposal and should limit aspects of the project plan which are to be determined only after the award of a contract. No optional deliverables to be provided only at an additional cost should be included and will not be considered in the evaluation of the technical proposal. Contractual terms, conditions and assumptions are inappropriate for inclusion in the proposal.

Any proprietary material considered confidential by the bidder will specifically be so identified, and the basis for such confidentiality will be specifically set forth in the proposal by submitting the form “Request for Exemption from Disclosure Pursuant to the Freedom of Information Law,” located in 5) Submission Documents.

Technical Proposal

Eight copies (one bearing an original signature) of the Technical Proposal should be organized in a binder with tabs clearly labeling each section as detailed below and include the following:

1. Mandatory Requirements Certification Form Signature Required

2. Work Plan: In its proposal, each bidder should include a comprehensive five-year detailed work plan. The work plan should describe how the activities required in the RFP will be conducted. A detailed timeline for each year of the contract, consistent with the Timeline for Required Services, provided in this RFP should be included in the work plan.

3. Bidders should complete items #3 and #6 of Appendix S-1 and return it with their technical proposal for NYSED review.

4. Project Description as outlined below:

Technical Proposal for Components 1a and 1c Components 1a and 1c Total (90 Points)1. Staffing and Program Management (15 points)

The proposal should provide a staffing plan with FTEs specified demonstrating that personnel experience and expertise are congruent with staff assignments for all aspects of the tasks outlined in Components 1a and 1c of the RFP. The proposal should include plans and procedures ensuring that staffing commensurate with the project’s scope of work throughout the duration of the contract resulting from this RFP.

The proposal should provide enough detail to enable reviewers to evaluate the appropriateness of the proposed staff, describing how different staff members and divisions will interact with one another, subcontractors, NYSED staff, teacher committees, schools, and other contractors working with NYSED. The proposal should describe how the bidder will ensure that the work of the contract has the necessary priority within the organization to be completed with the highest quality and on time. Please refer to the RFP sections “Staffing Requirements and Program Management,” “Staffing Requirements for the Grades 3-8 ELA and Mathematics Test Development,” and “Component 1c: Printing, Shipping, Collection and Return of Grades 3-8 ELA and Math Operational Tests Materials

96

RFP #20-004and Liquidation of Damages Related to Printing, Shipping and Collection of Paper-based Grades 3-8 ELA and Math Operational Materials” for a complete description of what will be required in the contract resulting from this RFP.

The program managers and all other key staff, including the three ELA content specialists, three mathematics content specialists, development supervisor, business analyst of data, technical writer and editorial staff, and research staff, must be identified by name and experience in writing. Current resumes for these staff members and any other key staff should be included as part of the proposal. The proposal should provide detailed information including:

an organizational diagram that indicates staffing arrangements for this project, the time commitments of the proposed staff described in terms of projected hours per week

devoted exclusively to the NYSED project, detailed description of responsibilities and qualifications,

resumes and two professional references for the program managers, technical writers and the business analyst of data positions.

For the program managers, a bachelor’s degree is required but a master’s degree or above is preferred. Project management certification through the Project Management Institute (PMI) as a Project Management Professional (PMP), or other recognized program management certification is preferred. If the proposed program managers hold such certification, evidence of the certification should be included in the proposal.

Writing samples should be included along with the resumes and references for the technical writer positions assigned to this project.

The bidders’ staffing plan should identify key staff that will be responsible for managing each of the areas of printing, shipping, collection, and return of printed test forms for the Grades 3-8 Operational Tests. The staffing plan should demonstrate that the bidder has or will have the staffing levels needed to maintain a high level of quality assurance and customer service throughout the duration of the contract including peak levels leading up to and during test administration dates.

2. Test Development Requirements for ELA and Mathematics (10 Points)Please refer to the RFP section “Component 1a: Test Development Requirements for Grades 3-8 English Language Arts and Mathematics” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe the bidder’s:

experience in developing passages and both multiple-choice and constructed-response items for large-scale assessment programs

plans and capacity to develop the passages, multiple-choice, and constructed-response items required by the RFP

plans for assembling tests according to the test design and other specifications of the RFP using items from NYSED’s existing bank, newly developed items, or a combination of both

plans to develop more items than will be needed for the operational tests, as specified in the RFP, in order to build operational tests from successfully field-tested items, as well as enriching the item bank for NYSED’s future use

capacity to develop a variety of passages and items in order to ensure that multiple genres are reflected in each ELA operational test and that the items on both the ELA and mathematics operational tests cover the widest range of learning standards possible

97

RFP #20-004 specifics on the required regional item writing workshops, review committees and the

operational forms construction meetings to be held with NYSED staff capacity to revise the existing style guide for the Grades 3–8 ELA and Mathematics Tests

3. Preparation of Printed Test Booklets and Printed Ancillary Materials (10 points)Please refer to the RFP section “Component 1c: Printing, Shipping, Collection and Return of Grades 3-8 ELA and Math Operational Tests Materials and Liquidation of Damages Related to Printing, Shipping and Collection of Paper-based Grades 3-8 ELA and Math Operational Materials” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should demonstrate the bidder’s:

experience in preparing printed test materials for large-scale assessment programs plans and capacity to develop all the materials required by the RFP, including printed and

online field test materials (test booklets, answer sheets, teacher directions, and instructions for administration) and camera-ready operational test materials (test booklets, school administrator’s manuals, teacher directions, and scoring guides and materials

plans for the development of translated, Braille, and large-type versions of the tests technical capacity and staffing levels needed to provide quality control and to produce all

materials in the timeframe required

4. Field Test Development (5 points)Please refer to the RFP section “Field Testing for ELA and Mathematics” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe the bidder’s:

experience in field testing large-scale assessments and its plans and capacity to conduct field testing as required by the RFP, including both stand-alone and embedded field tests

plans for selecting field test samples for stand-alone field tests, ensuring they are representative of NYS schools, and oversampling to ensure that the minimum number of actual student responses is obtained for each field test form

plans for the development, printing, shipping, and retrieval of all stand-alone field test materials, for schools choosing to administer SAFTs with PBT, and communication with schools in order to notify them of the field tests and work to obtain any unreturned field test materials

plans for scoring constructed responses, including conducting rangefinding with NYS teachers, developing scorer training materials, and recruiting and training appropriate scorers

plans for embedding multiple-choice field test items into operational test forms methodology to construct each field test and related materials with a timeline for each test specifics for field testing, including specifics on major events, phases, and deliverables timeline for the development of operational test sessions in ELA and Math with embedded field

test items including providing camera-ready copies and computer-ready sessions of operational tests to NYSED

technical specifications for the item banking system it intends to use to provide the test development services specified in this RFP

5. Research (10 points)Please refer to the RFP section “Component 1a: Test Development Requirements for Grades 3-8 English Language Arts and Mathematics,” “Analysis of Field Tests,” and “Forms Scaling and Equating” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe the bidder’s:

experience in conducting research for large-scale assessments

98

RFP #20-004 plans and capacity to perform the research required by the RFP, including the feasibility study

described in the RFP section “Feasibility Study for Implementing Computer-adaptive Testing.” methodology and rationale for such methods for appropriate field test analyses, forms scaling

and equating, embedding anchor forms, conducting a linking study, conducting standard setting with NYS teachers, providing all data files to NYSED, and preparing all required technical reports

6. Printing and Dissemination (10 points)Please refer to the RFP section “Component 1c: Printing, Shipping, Collection and Return of Grades 3-8 ELA and Math Operational Tests Materials and Liquidation of Damages Related to Printing, Shipping and Collection of Paper-based Grades 3-8 ELA and Math Operational Materials” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe the bidder’s:

experience in managing the printing, shipping, collection, and return of large volumes of secure printed test forms for large scale assessments

plans and capacity to perform the printing and dissemination tasks required by the RFP plans to arrange for the spiraling of multiple forms of the test booklet for each test session

within the same shrink-wrapped packages and ensuring that approximately 4,500 schools receive within the agreed upon date windows the exact quantities they request of each requested exam including special editions such as Braille and alternate language editions

plans for arranging for the return of all non-released test booklets and accounting for all materials that schools are required to return to the bidder, and

provision for processing orders placed after the primary period for schools to place their orders in the NYSED online examination request system has closed

7. Test Administration and Field Support for PBT and Scoring (10 points)Please refer to the RFP section “Component 1a: Test Development Requirements for Grades 3-8 English Language Arts and Mathematics” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe the bidder’s:

experience in test administration provision for field support for large-scale testing programs plans and capacity to perform all services required by the RFP provisions for developing all administration manuals, teacher’s directions, and scoring

materials needed for local scoring of the tests plans for annually updating the “Educator Guides” for the English Language Arts and

Mathematics Tests plans for providing a scoring helpline to answer test-related questions during the local scoring

period plans to work with local scanning centers (RICs and large city school district scanning centers)

and SIRS to coordinate the development of answer sheets and the transfer of student test data, and

plans for receiving, scoring, and analyzing student test data and for providing appropriate quality controls for all data.

99

RFP #20-0048. Security (10 points)

Bidders should describe all aspects of their security procedures handling large scale testing products in their proposals, which must be at a minimum, consistent with the Security Guidelines for New York State Assessment Program (Attachment M), Data Security and Privacy Plan (Appendix R), and Parents’ Bill of Rights (Appendix S). These security guidelines were developed by NYSED to set forth the management of secure materials and should be reflected in the bidder’s procedures.

Please refer to the RFP section “Data Security, Data Privacy and Appropriate Use” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe in detail, the bidder’s:

facilities, infrastructure, and security protocols, including specifics about each stage of the test development and administration process

plans for maintaining the security of test items prior to operational test administration, including during committee meetings and field testing

plans for maintaining the security of all test and individual student data, and plans for adhering to all requirements of the RFP, including Attachment M

9. Data Privacy Protection (5 points)Please refer to the RFP section “Data Security, Data Privacy and Appropriate Use” and “Requirements of Education Law § 2-d” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe in detail how the bidder will safeguard all individual student data and personally identifiable information and comply with all pertinent requirements of the RFP, including Appendices R, S, and S-1.

Bidders should complete items #3 and #6 of Appendix S-1 and return it with their technical proposal for NYSED review.

10. Qualifications and References (5 points)The proposal should demonstrate how the bidder:

meets the qualification requirements for the development of large-scale assessment programs, including test development, field testing, scoring, research, material preparation, and administration

has demonstrated experience in the development of secure, high volume, standardized assessment programs including both multiple-choice and constructed-response items

has the hardware, software, and staffing to complete all required tasks

The proposal should include a minimum of three current professional references to substantiate these qualifications. Current shall mean relevant within the past three years. NYSED offices and staff may not be used as references. Each reference should include the name, title, organization, address, telephone number, and e-mail address. The dates of service and a brief summary of the services provided should be included for each reference. The bidder should ensure that contact information is current and accurate.

100

RFP #20-004Technical Proposal for Components 1b and 2 Components 1b and 2 Total (85 Points)

1. Organizational Capacity (18 points)The proposal should provide a staffing solution and organizational arrangement demonstrating that personnel experience and expertise are congruent with staff assignments for all aspects of the tasks outlined in Component 1b and Component 2 of the RFP. The proposal should include plans and procedures ensuring that staffing will be commensurate with the project’s scope of work throughout the duration of the contract resulting from this RFP.

The proposal should provide enough detail to enable reviewers to evaluate the appropriateness of the proposed staffing, describing how different staff members and divisions will interact with one another, subcontractors, NYSED staff, teacher committees, schools, and other contractors working with NYSED. The proposal should describe how the bidder will ensure that the work of the contract has the necessary priority within the organization to be completed with the highest quality and on time and how staffing levels will maintain a high level of quality assurance and customer service throughout the duration of the contract including peak levels leading up to and during the test administration and scoring windows.

Please refer to the RFP sections “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math” and “Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should include the bidder’s:

staffing solution and organizational arrangement including an organization diagram, detailed descriptions of responsibilities, resumes, and time commitments for all proposed staff associated with performing the work detailed in Components 1b and 2,

proposed key staff, including the program managers, information technology, and customer support/training staff, by name, their resumes and a detailed description of responsibilities, qualifications, and the number of hours per week spent exclusively on Component 1b and Component 2 of the NYSED project.

2. Training and Technical Support for CBT Administration and Online Scoring (17 points)Please refer to the RFP sections “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math” and the “Training and Technical Support” section in Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests for a complete description of what will be required in the contract resulting from this RFP. All the requirements of the RFP should be addressed. At a minimum, the proposal should describe the bidder’s:

experience in hosting and supporting online test administration and providing field support for large-scale CBT programs

plans for providing local schools and districts with clear and complete training materials and complete technical support, including tools to check technology readiness, to enable them to successfully administer computer delivered field tests and to administer and score operational tests using the online test delivery and scoring systems

plan for providing written training materials for schools and districts well in advance of the administration period

provision for a minimum of sixteen regional training sessions annually for schools and districts: 6 in the fall, 6 in the winter and 4 in late winter as described in the “Training and Technical Support” section of the RFP

101

RFP #20-004 provision for a telephone support helpline for schools to contact during ELA, mathematics, and

science field testing and operational testing

3. Preparation of Online Operational and Field Test Forms (10 points)Please refer to the RFP section “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math” for a complete description of what will be required in the contract resulting from this RFP. At a minimum, the proposal should describe the bidder’s:

experience in developing test forms for computer-based field testing and computer based operational testing for large-scale assessment programs The test forms should include reading passages, and both multiple-choice and constructed-response items according to the test design and other specifications of the RFP.

plans for spiraling in classrooms a sufficient number of computer-delivered test forms to support the future development of the tests

plans for successfully supporting the changing numbers of schools administering computer-based assessments in each year of the contract

technical capacity and staffing level needed to provide quality control and to produce all materials in the timeframe required

4. Online Operational and Field Test CBT Administration and Online Scoring Systems

(20 points)

Please refer to the RFP section “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math” and “Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests” for a complete description of what will be required in the contract resulting from this RFP. All the requirements of the RFP should be addressed. At a minimum, the proposal should describe the bidder’s:

test delivery system for operational and field tests in English and required alternate languages plans and capacity to conduct the required online tests, including embedded MC field test

items, online SAFTs spiraled within exam rooms, student practice tests, system tests, and system monitoring

user interface for both the test administration and test delivery platforms detailed description of how students will interact with the test delivery platform when taking the

online tests and field tests quality control procedures used to verify that student responses are accurately saved and

associated with the correct data provision for computer delivery system for local scoring of constructed-response questions

submitted by students on computer devices provision for any adjustments that schools may need to make to accommodate ELLs/MLLs

using alternate language mathematics and science tests scoring platform addressing the criteria as outlined in the RFP section “Scoring Platform and

Data Management” specific browser(s) that are compatible with the bidder’s online test delivery and administration

system that operates on and functions as designed while meeting minimum technical requirements (See Attachment A) and supports the release of new testing devices, operating systems, and browsers

plan for submitting to NYSED, in a suitable format, the responses submitted by students to online field test science questions

102

RFP #20-004 5. System Management/Monitoring and Security (20 points)

Please refer to the RFP section “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Math” and “Component 2: Hosting Computer-Based Testing and Scoring for NYSED-Developed Grades 5 and 8 Science Written Tests” for a complete description of what will be required in the contract resulting from this RFP. All the requirements of the RFP, including Attachment C and Appendices R, S, and S-1, should be addressed. At a minimum, the proposal should describe the bidder’s:

facilities, infrastructure, and security protocols, including specifics about each stage of the online test development, administration, and vendor-provided platform for local online scoring process

plans for maintaining the security of test items prior to test administration, during computer-based operational test administration, and during contractor platform hosted online scoring of student responses to CR questions

quality control mechanisms that ensure that the scores inputted by raters and the student responses are securely stored and transferred to the bidder’s exam data management system with 100% accuracy and reliability

types and levels of monitoring plan for load testing and for system maintenance, updates and availability during both

operational and field-testing periods plans for student authentication and access, and administrative authentication and access security plan, with an explanation of the encryption to be used and research that supports its

use which meets all requirements specified within this RFP as well as all relevant federal and New York State laws

plans for maintaining the security of all test and individual student data description of recovery plan plan for managing purging, when caching technology is utilized

Computer-Based Testing Demonstration Scores for the Technical Proposal for Components 1b and 2 are subject to change following the mandatory demonstration of the computer-based testing and scoring system. Demonstrations will be used for Validation/Adjustments to the final scores for these sections.

Each bidder submitting a proposal will be scheduled for a mandatory demonstration of its online delivery system to NYSED’s RFP proposal reviewers. It is anticipated that all demonstrations will be scheduled for one hour on a date mutually agreed upon by NYSED and the bidder. Demonstrations will be conducted either in person at NYSED’s Albany, NY office or by webinar. This demonstration must take place between 2/10/2020 and 2/14/2020. To schedule the demonstration, bidders may contact Victoria Mahar via email at [email protected] prior to the deadline for submitting proposals. Bidders who have not scheduled a demonstration time in advance will be provided a time once bids are received.

Such bidder demonstration shall help NYSED’s reviewers understand what systems are being offered and any additional features that the bidder plans to develop to support CBT for ELA, Math, and Science. The bidder should demonstrate its CBT testing platform, including the online test delivery, administration, and scoring systems. The demonstration should include, but not be limited to, demonstration of how students will respond to constructed-response questions in both ELA and Mathematics and online testing tools. Additionally, the bidder should present mathematics test materials in at least one language other than English. See RFP section “Component 1b: Computer-Based Testing and Scoring for Grades 3-8 ELA and Mathematics and the “Computer-Based Testing Demonstration” for additional information.103


RFP #20-004

Cost Proposal Cost Proposal (30 points)

The original plus one (1) copy of the completed Cost Proposal must be mailed in a separate envelope labeled RFP #20-004-Cost Proposal-Do Not Open and must include the following six sheets contained in the Excel workbook:

1. Bid Form Cost Proposal—Schedule of Deliverables: Years 1-5 Component 1 Signature Required

2. Subcontracting Form for Component 13. M/WBE Purchase Form for Component 14. Bid Form Cost Proposal—Schedule of Deliverables Years 1-5 Component 2

Signature Required5. Subcontracting Form for Components 1 and 26. M/WBE Purchase Form for Components 1 and 2

On the Bid Form Cost Proposal for Components 1 and 2, bidders are to provide a price for each deliverable. Please note that the cost submitted by the bidder for Component 1c: Printing, Shipping, Collection and Return of Grades 3-8 ELA and Math Operational Tests Materials should be based on costs for the printing, distribution, and collection-related deliverables associated with the operational tests based on the assumption of printing 100% of the volume indicated in Attachment L . The cells for these deliverables are highlighted in yellow in the Cost Proposal: Schedule of Deliverables.

Budgets must be submitted using whole dollar numbers.

The Financial Criteria portion of the RFP will be scored based upon either the five (5)-year grand total for Component 1 plus the five (5)-year grand total for Component 2 or upon the five (5)-year grand total for Component 1 only. (See “Method of Award” in Section 3 of this RFP for further details.

M/WBE Documents

The original of the completed M/WBE Documents must be mailed in a separate envelope labeled RFP #20-004-M/WBE Documents-Do Not Open. Please return the documents listed for the compliance method bidder has achieved:

Full Participation-No Request for Waiver1. M/WBE Cover Letter, Signatures Required2. M/WBE 100 Utilization Plan3. M/WBE 102 Notice of Intent to Participate 4. EEO 100 Staffing Plan

Partial Participation-Partial Request for Waiver1. M/WBE Cover Letter, Signatures Required2. M/WBE 100 Utilization Plan3. M/WBE 102 Notice of Intent to Participate 4. EEO 100 Staffing Plan 5. M/WBE 101 Request for Waiver104

RFP #20-0046. M/WBE 105 Contractor’s Good Faith Efforts

No Participation-Request for Complete Waiver1. M/WBE Cover Letter, Signatures Required2. M/WBE 101 Request for Waiver3. M/WBE 105 Contractor’s Good Faith Efforts

105

RFP #20-0043.) EVALUATION CRITERIA AND METHOD OF AWARD

This section begins with the criteria the agency will use to evaluate bids and closes with the “method of award” or how the contractor will be selected. This will be followed by various terms and conditions that reflect the specific needs of this project as well as New York State contract guidelines and requirements.

Criteria for Evaluating Bids

All eligible proposals received by the deadline will be reviewed using the following criteria and ratings. Applicants must ensure that all components of this application request have been addressed, the required number of copies has been provided, all forms and assurances have been completed, and the original signatures in blue are included as required.

An evaluation committee will complete a review of all proposals submitted. The committee will review each proposal based upon the submitted proposal and the requirements of the RFP only. Bidders should not assume that committee review members will be familiar with the current program or have any previous experience with the bidder. Appropriate description should be included to inform review committee members about the bidder’s qualifications and capacity to perform all required deliverables.

The committee will review each proposal to determine compliance with the requirements described in the RFP. The committee retains the right to determine whether any deviation from the requirements of this RFP is substantial in nature and may reject in whole or in part any and all proposals, waive minor irregularities and conduct discussions with all responsible bidders.

Technical CriteriaTechnical Criteria for Components 1 and 2 (70% of total score)

175 Points

Technical Criteria for Components 1a and 1c (90 Points)Each proposal received by the deadline will be reviewed and rated on the quality and extent to which the bidder meets the following criteria:

Technical Score Evaluation Points1. Staffing and Program Management 152. Test Development Requirements for ELA and Mathematics 103. Preparation of Printed Test Booklets and Printed Ancillary Materials 104. Field Test Development 55. Research 106. Printing and Dissemination 107. Test Administration and Field Support for PBT and Scoring 108. Security 109. Data Privacy Protection 510. Qualifications and References 5

Technical Evaluation Total Points 90

106

RFP #20-004Technical Criteria for Components 1b and 2 (85 Points)

Each proposal received by the deadline will be reviewed and rated on the quality and extent to which the bidder meets the following criteria:

Technical Score Evaluation Points1 Organizational Capacity 182 Training and Technical Support for CBT Administration and Online

Scoring17

3 Preparation of Online Operational and Field Test Forms 104 Online Operational and Field Test CBT Administration and Online

Scoring Systems20

5 System Management/Monitoring and Security 20Technical Evaluation Total Points 85

Financial CriteriaFinancial Criteria (30% of total score) 30 Points

The Financial Criteria portion of the RFP will be scored based upon either the five-year grand total for Component 1 plus the five-year grand total for Component 2 or upon the five-year grand total for Component 1 only. (See the Method of Award section for more details.) The cost proposal will be awarded points pursuant to a formula, which awards the highest score of thirty (30) points to the proposal with the lowest cost. The remaining cost proposals will be awarded points based on the relative difference of each proposal against the lowest budget submitted. The resulting percentage is then applied to the maximum point value of thirty points. This calculation will be computed by the Contract Administration Unit upon completion of the technical scoring by the technical review panel.

NYSED reserves the right to request best and final offers. In the event NYSED exercises this right, all responsive bidders will be asked to provide a best and final offer. The Contract Administration Unit will recalculate the financial score.

Method of Award

The aggregate score of all the criteria listed will be calculated for each proposal received.

The contract issued pursuant to this proposal will be awarded to the vendor whose aggregate technical and cost score is the highest among all the proposals rated. If NYSED exercises the right to request best and final offers, the contract must be issued to the vendor with the highest aggregate technical and financial score that results from the best and final offer.

In the event that more than one proposal obtains the highest aggregate score, the contract will be awarded to the vendor in that group of highest aggregate scores whose budget component reflects the lowest overall cost.

Initial Scoring and Award

All proposals which are received by the bid due date and meet the mandatory requirements of the RFP will be reviewed by the technical evaluation committee as described above. A combined total cost for Components 1 and 2 will then be calculated for all proposals received. The combined cost for

107

RFP #20-004scoring purposes will be the five-year grand total for Component 1 plus the five-year grand total for Component 2.

The total score for each proposal will represent the 70/30 percent split between technical and cost scores. The sum of the average technical score for Components 1a and 1c plus the average technical score for Components 1b and 2 will be divided by 2.5. Consequently, the final technical score for any proposal will have a maximum value of 70 points. For example, if a proposal receives an average technical score of 80 for Components 1a and 1c, an average technical score of 70 for Components 1b and 2, this proposal will receive a final technical score of 60, [(80 + 70)/2.5 = 60]. This score will be added to the cost score to obtain the final score. The successful bidder will be that bidder whose aggregate of the final technical score and cost score is the highest among all the proposals rated. In the event that more than one proposal obtains the highest aggregate score, the successful bidder will be the bidder in that group of highest aggregate scores whose proposal reflects the lowest cost.

Prior to making an award based on the original cost scoring, NYSED reserves the right to request that all bidders submit a best and final offer for their proposal. If best and final offers are requested, these prices will be used to calculate the final cost score for each contractor.

After determining the successful bidder, NYSED will evaluate the cost proposed by the successful bidder for Components 1 and 2 in comparison to the maximum feasible cost determined by NYSED given the availability of funding and budget constraints. If the cost proposed by the successful bidder is equal to or below the maximum feasible cost established by NYSED, NYSED will proceed to award a contract containing both Components 1 and 2 to the successful bidder.

Second Scoring and Award, if Necessary

If, following the initial scoring, the combined cost proposed by the successful bidder for Components 1 and 2 is above the maximum feasible cost established by NYSED, no award will be made containing Component 2. Instead, NYSED will recalculate the cost score for each bidder using only the five-year grand total for Component 1.

If best and final offers were requested by NYSED during the initial round of scoring or are requested during this second round of scoring, the best and final offers quoted by the bidders for Component 1 will be used to calculate the comparative financial score for each bidder in the second round of scoring.

After determining the new successful bidder in the second round of scoring, NYSED will proceed to award a contract containing only Component 1 to the successful bidder.

NYSED’s Reservation of Rights

NYSED reserves the right to: (1) reject any or all proposals received in response to the RFP; (2) withdraw the RFP at any time, at the agency’s sole discretion; (3) make an award under the RFP in whole or in part; (4) disqualify any bidder whose conduct and/or proposal fails to conform to the requirements of the RFP; (5) seek clarifications of proposals; (6) use proposal information obtained through site visits, management interviews and the state’s investigation of a bidder’s qualifications, experience, ability or financial standing, and any material or information submitted by the bidder in response to the agency’s request for clarifying information in the course of evaluation and/or selection under the RFP; (7) prior to the bid opening, amend the RFP specifications to correct errors or oversights, or to supply additional information, as it becomes available; (8) prior to the bid opening, 108

RFP #20-004direct bidders to submit proposal modifications addressing subsequent RFP amendments; (9) change any of the scheduled dates; (10) waive any requirements that are not material; (11) negotiate with the successful bidder within the scope of the RFP in the best interests of the state; (12) conduct contract negotiations with the next responsible bidder, should the agency be unsuccessful in negotiating with the selected bidder; (13) utilize any and all ideas submitted in the proposals received; (14) unless otherwise specified in the solicitation, every offer is firm and not revocable for a period of 90 days from the bid opening; (15) require clarification at any time during the procurement process and/or require correction of arithmetic or other apparent errors for the purpose of assuring a full and complete understanding of an offerer’s proposal and/or to determine an offerer’s compliance with the requirements of the solicitation; (16) request best and final offers.

Post Selection Procedures

Upon selection, the successful bidder will receive a proposed contract from NYSED. The selected bidder may be given an opportunity to reduce its cost proposal in accordance with the agency's right to negotiate a final best price. The contents of this RFP, any subsequent correspondence during the proposal evaluation period, and such other stipulations as agreed upon may be made a part of the final contract prepared by NYSED. Successful bidders may be subject to audit and should ensure that adequate controls are in place to document the allowable activities and expenditure of State funds.

Debriefing Procedures

In accordance with section 163 of the NY State Finance Law, NYSED, upon request, must provide a debriefing to any unsuccessful bidder regarding the reasons their proposal was not selected for an award.

1. All unsuccessful bidders may request a debriefing within fifteen (15) calendar days of receiving notice from NYSED of non-award. Bidders may request a debriefing by submitting a written request to the Fiscal Contact person at:

NYS Education DepartmentContract Administration Unit89 Washington AvenueRoom 501W EBAlbany, NY 12234

2. Upon receipt of a timely written request from the unsuccessful bidder, NYSED will schedule the debriefing to occur within a reasonable time following receipt of the request. Debriefings will be conducted in person, unless NYSED and the bidder mutually agree to utilize other means, including but not limited to telephone, video-conferencing or other types of electronic communication.

3. The debriefing will include: a) the reasons that the proposal submitted by the unsuccessful bidder was not selected for an award; b) the qualitative and quantitative analysis employed by NYSED in assessing the relative merits of the proposals; c) the application of the selection criteria to the unsuccessful bidder’s proposal; and d) when the debriefing is held after the final award, the reasons for the selection of the winning proposal. The debriefing will also provide, to the greatest extent practicable, general advice and guidance to the unsuccessful bidder concerning potential ways that their future proposals could be more responsive.

109

RFP #20-004Contract Award Protest Procedures

Bidders who receive a notice of non-award or disqualification may protest the NYSED award decision subject to the following:

1. The protest must be in writing and must contain specific factual and/or legal allegations setting forth the basis on which the protesting party challenges the contract award by NYSED.

2. The protest must be filed within ten (10) business days of receipt of a debriefing or disqualification letter. The protest letter must be filed with:

NYS Education DepartmentContract Administration Unit89 Washington AvenueRoom 501W EBAlbany, NY 12234

3. The NYSED Contract Administration Unit (CAU) will convene a review team that will include at least one staff member from each of NYSED’s Office of Counsel, CAU, and the Program Office. The review team will review and consider the merits of the protest and will decide whether the protest is approved or denied. Counsel’s Office will provide the bidder with written notification of the review team’s decision within seven (7) business days of the receipt of the protest. The original protest and decision will be filed with OSC when the contract procurement record is submitted for approval and CAU will advise OSC that a protest was filed.

4. The NYSED Contract Administration Unit (CAU) may summarily deny a protest that fails to contain specific factual or legal allegations, or where the protest only raises issues of law that have already been decided by the courts.

Vendor Responsibility

State law requires that the award of state contracts be made to responsible vendors. Before an award is made to a not-for-profit entity, a for-profit entity, a private college or university or a public entity not exempted by the Office of the State Comptroller, NYSED must make an affirmative responsibility determination. The factors to be considered include: legal authority to do business in New York State; integrity; capacity- both organizational and financial; and previous performance. Before an award of $100,000 or greater can be made to a covered entity, the entity will be required to complete and submit a Vendor Responsibility Questionnaire. School districts, Charter Schools, BOCES, public colleges and universities, public libraries, and the Research Foundation for SUNY and CUNY are some of the exempt entities. A complete list of exempt entities can be viewed at the Office of the State Comptroller’s website.

NYSED recommends that vendors file the required Vendor Responsibility Questionnaire online via the New York State VendRep System. To enroll in and use the New York State VendRep System, see the VendRep System Instructions or go directly to the VendRep System on the Office of the State Comptroller's website.

Vendors must provide their New York State Vendor Identification Number when enrolling. To request assignment of a Vendor ID or for VendRep System assistance, contact the Office of the State

110

https://portal.osc.state.ny.us/

https://portal.osc.state.ny.us/

http://www.osc.state.ny.us/vendrep/vendor_index.htm

http://www.osc.state.ny.us/vendrep/resources_docreq_agency.htm

http://www.osc.state.ny.us/vendrep/resources_docreq_agency.htm

http://osc.state.ny.us/vendrep/

RFP #20-004Comptroller’s Help Desk at 866-370-4672 or 518-408-4672 or by email at [email protected].

Vendors opting to complete and submit a paper questionnaire can obtain the appropriate questionnaire from the VendRep website or may contact NYSED or the Office of the State Comptroller’s Help Desk for a copy of the paper form.

Subcontractors:For vendors using subcontractors, a Vendor Responsibility Questionnaire and a NYSED vendor responsibility review are required for a subcontractor where:

the subcontractor is known at the time of the contract award; the subcontractor is not an entity that is exempt from reporting by OSC; and the subcontract will equal or exceed $100,000 over the life of the contract.

Note: Bidders must acknowledge their method of filing their questionnaire by checking the appropriate box on the Response Sheet for Bids (5. Submission Documents).

Procurement Lobbying Law

Pursuant to State Finance Law §§139-j and 139-k, this solicitation includes and imposes certain restrictions on communications between the New York State Education Department (“NYSED”) and an Offerer/bidder during the procurement process. An Offerer/bidder is restricted from making contacts from the earliest notice of the solicitation through final award and approval of the Procurement Contract by NYSED and, if applicable, Office of the State Comptroller (“restricted period”) to other than designated staff unless it is a contact that is included among certain statutory exceptions set forth in State Finance Law §139-j(3)(a). Designated staff, as of the date hereof, is identified below. NYSED employees are also required to obtain certain information when contacted during the restricted period and make a determination of the responsibility of the Offerer/bidder pursuant to these two statutes. Certain findings of non-responsibility can result in rejection for contract award and in the event of two findings within a four-year period, the Offerer/bidder is debarred from obtaining governmental Procurement Contracts. Further information about these requirements can be found at NYSED's Procurement Lobbying Law Policy Guidelines webpage.

Designated Contacts for NYSEDProgram Office – Victoria MaharContract Administration Unit – Jessica HartjenM/WBE – Joan Ramsey

Consultant Disclosure Legislation

Effective June 19, 2006, new reporting requirements became effective for State contractors, as the result of an amendment to State Finance Law §§ 8 and 163. As a result of these changes in law, State contractors will be required to disclose, by employment category, the number of persons employed to provide services under a contract for consulting services, the number of hours worked and the amount paid to the contractor by the State as compensation for work performed by these employees. This will include information on any persons working under any subcontracts with the State contractor.

111

http://www.oms.nysed.gov/fiscal/cau/PLL/procurementpolicy.htm

http://www.osc.state.ny.us/vendrep


RFP #20-004Chapter 10 of the Laws of 2006 expands the definition of contracts for consulting services to include any contract entered into by a State agency for analysis, evaluation, research, training, data processing, computer programming, engineering, environmental, health, and mental health services, accounting, auditing, paralegal, legal, or similar services.

To enable compliance with the law, State agencies must include in the Procurement Record submitted to OSC for new consultant contracts, the State Consultant Services Contractor’s Planned Employment From Contract Start Date Through the End of the Contract Term (Form A - see link below). The completed form must include information for all employees providing service under the contract whether employed by the contractor or a subcontractor. Please note that the form captures the necessary planned employment information prospectively from the start date of the contract through the end of the contract term.

Form A is available on OSC’s website.

Please note that although this form is not required as part of the bid submissions, NYSED encourages bidders to include them in their bid submission to expedite contract execution if the bidder is awarded the contract. Note also that only the form listed above is acceptable.Chapter 10 of the Laws of 2006 mandates that State agencies must now require State contractors to report annually on the employment information described above, including work performed by subcontractors. The legislation mandates that the annual employment reports are to be submitted by the contractor to the contracting agency, to OSC and to the Department of Civil Service. State Consultant Services Contractor’s Annual Employment Report (Form B - see link below) is to be used to report the information for all procurement contracts above $15,000. Please note that, in contrast to the information to be included on Form A, which is a one-time report of planned employment data for the entire term of a consulting contract on a projected basis, Form B will be submitted each year the contract is in effect and will capture historical information, detailing actual employment data for the most recently concluded State fiscal year (April 1 – March 31).

Form B is available on OSC’s website.

For more information, please visit OSC Guide to Financial Operations.

Public Officer’s Law Section 73 All bidders must comply with Public Officer’s Law Section 73 (4)(a), as follows:

4. (a) No statewide elected official, state officer or employee, member of the legislature, legislative employee or political party chairman or firm or association of which such person is a member, or corporation, ten per centum or more of the stock of which is owned or controlled directly or indirectly by such person, shall (i) sell any goods or services having a value in excess of twenty-five dollars to any state agency, or (ii) contract for or provide such goods or services with or to any private entity where the power to contract, appoint or retain on behalf of such private entity is exercised, directly or indirectly, by a state agency or officer thereof, unless such goods or services are provided pursuant to an award or contract let after public notice and competitive bidding. This paragraph shall not apply to the publication of resolutions, advertisements or other legal propositions or notices in newspapers designated pursuant to law for such purpose and for which the rates are fixed pursuant to law.

(i) The term "state officer or employee" shall mean:

112

http://www.osc.state.ny.us/agencies/guide/MyWebHelp/

http://www.osc.state.ny.us/agencies/forms/ac3272s.doc

http://www.osc.state.ny.us/agencies/gbull/g226form%20b.pdf

http://www.osc.state.ny.us/agencies/forms/ac3271s.doc

http://www.osc.state.ny.us/agencies/gbull/g226forma.pdf

RFP #20-004(i) heads of state departments and their deputies and assistants other than members of the board

of regents of the university of the state of New York who receive no compensation or are compensated on a per diem basis;

(ii) officers and employees of statewide elected officials;(iii) officers and employees of state departments, boards, bureaus, divisions, commissions,

councils or other state agencies other than officers of such boards, commissions or councils who receive no compensation or are compensated on a per diem basis; and

(iv) members or directors of public authorities, other than multistate authorities, public benefit corporations and commissions at least one of whose members is appointed by the governor, who receive compensation other than on a per diem basis, and employees of such authorities, corporations and commissions.

Review Public Officer’s Law Section 73.

NYSED Substitute Form W-9

Any payee/vendor/organization receiving Federal and/or State payments from NYSED must complete the NYSED Substitute Form W-9 if they are not yet registered in the Statewide Financial System centralized vendor file.

The NYS Education Department (NYSED) is using the NYSED Substitute Form W-9 to obtain certification of a vendor’s Tax Identification Number in order to facilitate a vendor’s registration with the SFS centralized vendor file and to ensure accuracy of information contained therein. We ask for the information on the NYSED Substitute Form W-9 to carry out the Internal Revenue laws of the United States.

Workers’ Compensation Coverage and Debarment

New York State Workers’ Compensation Law (WCL) has specific coverage requirements for businesses contracting with New York State and additional requirements which provide for the debarment of vendors that violate certain sections of WCL. The WCL requires, and has required since introduction of the law in 1922, the heads of all municipal and State entities to ensure that businesses have appropriate workers’ compensation and disability benefits insurance coverage prior to issuing any permits or licenses, or prior to entering into contracts.

Workers’ compensation requirements are covered by WCL Section 57, while disability benefits are covered by WCL Section 220(8). The Workers’ Compensation Benefits clause in Appendix A – STANDARD CLAUSES FOR NEW YORK STATE CONTRACTS states that in accordance with Section 142 of the State Finance Law, a contract shall be void and of no force and effect unless the contractor provides and maintains coverage during the life of the contract for the benefit of such employees as are required to be covered by the provisions of the WCL.

Under provisions of the 2007 Workers’ Compensation Reform Legislation (WCL Section 141-b), any person, or entity substantially owned by that person: subject to a final assessment of civil fines or penalties, subject to a stop-work order, or convicted of a misdemeanor for violation of Workers’ Compensation laws Section 52 or 131, is barred from bidding on, or being awarded, any public work contract or subcontract with the State, any municipal corporation or public body for one year for each violation. The ban is five years for each felony conviction.

113

http://www.jcope.ny.gov/about/ethc/PUBLIC%20OFFICERS%20LAW%2073%20JCOPE.pdf

RFP #20-004

Proof of Coverage Requirements The Workers’ Compensation Board has developed several forms to assist State contracting entities in ensuring that businesses have the appropriate workers’ compensation and disability insurance coverage as required by Sections 57 and 220(8) of the WCL.

Please note – an ACORD form is not acceptable proof of New York State workers’ compensation or disability benefits insurance coverage.

Proof of Workers’ Compensation Coverage

To comply with coverage provisions of the WCL, the Workers’ Compensation Board requires that a business seeking to enter into a State contract submit appropriate proof of coverage to the State contracting entity issuing the contract. For each new contract or contract renewal, the contracting entity must obtain ONE of the following forms from the contractor and submit to OSC to prove the contractor has appropriate workers’ compensation insurance coverage:

Form C-105.2 – Certificate of Workers’ Compensation Insurance issued by private insurance carriers, or Form U-26.3 issued by the State Insurance Fund; or

Form SI-12– Certificate of Workers’ Compensation Self-Insurance; or Form GSI-105.2 Certificate of Participation in Workers’ Compensation Group Self-Insurance; or

CE-200– Certificate of Attestation of Exemption from NYS Workers’ Compensation and/or Disability Benefits Coverage.

Proof of Disability Benefits Coverage

To comply with coverage provisions of the WCL regarding disability benefits, the Workers’ Compensation Board requires that a business seeking to enter into a State contract must submit appropriate proof of coverage to the State contracting entity issuing the contract. For each new contract or contract renewal, the contracting entity must obtain ONE of the following forms from the contractor and submit to OSC to prove the contractor has appropriate disability benefits insurance coverage:

Form DB-120.1 - Certificate of Disability Benefits Insurance; or

Form DB-155- Certificate of Disability Benefits Self-Insurance; or

CE-200– Certificate of Attestation of Exemption from New York State Workers’ Compensation and/or Disability Benefits Coverage.

For additional information regarding workers’ compensation and disability benefits requirements, please refer to the New York State Workers’ Compensation Board website. Alternatively, questions relating to either workers’ compensation or disability benefits coverage should be directed to the NYS Workers’ Compensation Board, Bureau of Compliance at (518) 486-6307.

Please note that although these forms are not required as part of the bid submissions, NYSED encourages bidders to include them in their bid submission to expedite contract execution if the bidder is awarded the contract. Note also that only the forms listed above are acceptable. 114

http://wcb.ny.gov/content/main/Employers/busPermits.jsp

RFP #20-004

Sales and Compensating Use Tax Certification (Tax Law, § 5-a)

Tax Law § 5-a requires contractors awarded State contracts for commodities or services valued at more than $100,000 over the full term of the contract to certify to the New York State Department of Taxation and Finance (“DTF”) that they are registered to collect New York State and local sales and compensating use taxes, if they made sales delivered by any means to locations within New York State of tangible personal property or taxable services having a cumulative value in excess of $300,000, measured over a specific period of time. The registration requirement applies if the contractor made a cumulative total of more than $300,000 in sales during the four completed sales tax quarters which immediately precede the sales tax quarter in with the certification is made. Sales tax quarters are June – August, September – November, December – February, and March – May. In addition, contractors must certify to DTF that each affiliate and subcontractor of such contractor exceeding such sales threshold during a specified period is registered to collect New York State and local sales and compensating use taxes. Contractors must also certify to the procuring State entity that they filed the certification with the DTF and that it is correct and complete.

The selected bidder must file a properly completed Form ST-220-CA (with NYSED as the Contracting Agency) and Form ST-220-TD (with the DTF). These requirements must be met before a contract may take effect. Further information can be found at the New York State Department of Taxation and Finance’s website . Forms are available through these links:• ST-220 CA• ST-220 TD

Please note that although these forms are not required as part of the bid submissions, NYSED encourages bidders to include them with their bid submissions to expedite contract execution if the bidder is awarded the contract.

115

http://www.tax.ny.gov/pdf/current_forms/st/st220ca_fill_in.pdf

RFP #20-0044.) ASSURANCES

The State of New York Agreement, Appendix A – Standard Clause for all New York State Contracts, Appendix A-1– Agency-Specific Clauses, Appendix R – Data Security and Privacy Plan Provisions, Appendix S – Parents’ Bill of Rights for Data Privacy and Security, and Appendix S-1 – Attachment to Parents’ Bill of Rights for Contracts Involving Disclosure of Certain Personally Identifiable Information WILL BE INCLUDED in the contract that results from this RFP. Vendors who are unable to complete or abide by these assurances should not respond to this request.

The documents listed below are included in 6.) Submission Documents, which must be signed by the Chief Administrative Officer. Please review the terms and conditions. Certain documents will become part of the resulting contract that will be executed between the successful bidder and the NYS Education Department.

1. Non-Collusion Certification

2. MacBride Certification

3. Certification-Omnibus Procurement Act of 1992

4. Certification Regarding Lobbying; Debarment and Suspension; and Drug-Free Workplace Requirements

5. Offerer Disclosure of Prior Non-Responsibility Determinations

6. NYSED Substitute Form W-9 (If bidder is not yet registered in the SFS centralized vendor file.)

7. Iran Divestment Act Certification

M/WBE Documents – (the forms below are included in 6.) Submission Documents) Please return the documents listed for the compliance method bidder has achieved:

Full Participation-No Request for Waiver Signatures Required1. M/WBE Cover Letter2. M/WBE 100 Utilization Plan3. M/WBE 102 Notice of Intent to Participate 4. EEO 100 Staffing Plan

Partial Participation-Partial Request for Waiver Signature Required1. M/WBE Cover Letter2. M/WBE 100 Utilization Plan3. M/WBE 102 Notice of Intent to Participate 4. EEO 100 Staffing Plan5. M/WBE 101 Request for Waiver6. M/WBE 105 Contractor’s Good Faith Efforts

No Participation-Request for Complete Waiver Signature Required1. M/WBE Cover Letter2. M/WBE 101 Request for Waiver3. M/WBE 105 Contractor’s Good Faith Efforts116

RFP #20-004

State of New York Agreement

This AGREEMENT is hereby made by and between the People of the State of New York, acting through MaryEllen Elia, Commissioner of Education of the State of New York, party of the first part, hereinafter referred to as the (STATE) and the public or private agency (CONTRACTOR) identified on the face page hereof.

WITNESSETH:WHEREAS, the STATE has the authority to regulate and provide funding for the establishment and operation of

program services and desires to contract with skilled parties possessing the necessary resources to provide such services; and

WHEREAS, the CONTRACTOR is ready, willing and able to provide such program services and possesses or can make available all necessary qualified personnel, licenses, facilities and expertise to perform or have performed the services required pursuant to the terms of this AGREEMENT;

NOW THEREFORE, in consideration of the promises, responsibilities and covenants herein, the STATE and the CONTRACTOR agree as follows:

I. Conditions of Agreement

A. This AGREEMENT may consist of successive periods (PERIOD), as specified within the AGREEMENT or within a subsequent Modification Agreement(s) (Appendix X). Each additional or superseding PERIOD shall be on the forms specified by the particular State agency, and shall be incorporated into this AGREEMENT.

B. Funding for the first PERIOD shall not exceed the funding amount specified on the face page hereof. Funding for each subsequent PERIOD, if any, shall not exceed the amount specified in the appropriate appendix for that PERIOD.

C. This AGREEMENT incorporates the face pages attached and all of the marked appendices identified on the face page hereof.

D. For each succeeding PERIOD of this AGREEMENT, the parties shall prepare new appendices, to the extent that any require modification, and a Modification Agreement (The attached Appendix X is the blank form to be used). Any terms of this AGREEMENT not modified shall remain in effect for each PERIOD of the AGREEMENT.

To modify the AGREEMENT within an existing PERIOD, the parties shall revise or complete the appropriate appendix form(s). Any change in the amount of consideration to be paid, or change in the term, is subject to the approval of the Office of the State Comptroller. Any other modifications shall be processed in accordance with agency guidelines as stated in Appendix A1.

E. The CONTRACTOR shall perform all services to the satisfaction of the STATE. The CONTRACTOR shall provide services and meet the program objectives summarized in the Program Workplan (Appendix D) in accordance with: provisions of the AGREEMENT; relevant laws, rules and regulations, administrative and fiscal guidelines; and where applicable, operating certificates for facilities or licenses for an activity or program.

F. If the CONTRACTOR enters into subcontracts for the performance of work pursuant to this AGREEMENT, the CONTRACTOR shall take full responsibility for the acts and omissions of its subcontractors. Nothing in the subcontract shall impair the rights of the STATE under this AGREEMENT. No contractual relationship shall be deemed to exist between the subcontractor and the STATE.

G. Appendix A (Standard Clauses as required by the Attorney General for all State contracts) takes precedence over all other parts of the AGREEMENT.

II. Payment and Reporting

A. The CONTRACTOR, to be eligible for payment, shall submit to the STATE's designated payment office (identified in Appendix C) any appropriate documentation as required by the Payment and Reporting Schedule (Appendix C) and by agency fiscal guidelines, in a manner acceptable to the STATE.

B. The STATE shall make payments and any reconciliations in accordance with the Payment and Reporting Schedule (Appendix C). The STATE shall pay the CONTRACTOR, in consideration of contract services for a given PERIOD, a sum not to

117

RFP #20-004

exceed the amount noted on the face page hereof or in the respective Appendix designating the payment amount for that given PERIOD. This sum shall not duplicate reimbursement from other sources for CONTRACTOR costs and services provided pursuant to this AGREEMENT.

C. The CONTRACTOR shall meet the audit requirements specified by the STATE.

III. Terminations

A. This AGREEMENT may be terminated at any time upon mutual written consent of the STATE and the CONTRACTOR.

B. The STATE may terminate the AGREEMENT immediately, upon written notice of termination to the CONTRACTOR, if the CONTRACTOR fails to comply with the terms and conditions of this AGREEMENT and/or with any laws, rules, regulations, policies or procedures affecting this AGREEMENT.

C. The STATE may also terminate this AGREEMENT for any reason in accordance with provisions set forth in Appendix A1.

D. Written notice of termination, where required, shall be sent by personal messenger service or by certified mail, return receipt requested. The termination shall be effective in accordance with the terms of the notice.

E. Upon receipt of notice of termination, the CONTRACTOR agrees to cancel, prior to the effective date of any prospective termination, as many outstanding obligations as possible, and agrees not to incur any new obligations after receipt of the notice without approval by the STATE.

F. The STATE shall be responsible for payment on claims pursuant to services provided and costs incurred pursuant to terms of the AGREEMENT. In no event shall the STATE be liable for expenses and obligations arising from the program(s) in this AGREEMENT after the termination date.

IV. Indemnification

A. The CONTRACTOR shall be solely responsible and answerable in damages for any and all accidents and/or injuries to persons (including death) or property arising out of or related to the services to be rendered by the CONTRACTOR or its subcontractors pursuant to this AGREEMENT. The CONTRACTOR shall indemnify and hold harmless the STATE and its officers and employees from claims, suits, actions, damages and costs of every nature arising out of the provision of services pursuant to this AGREEMENT.

B. The CONTRACTOR is an independent contractor and may neither hold itself out nor claim to be an officer, employee or subdivision of the STATE nor make any claim, demand or application to or for any right based upon any different status.

V. Property

Any equipment, furniture, supplies or other property purchased pursuant to this AGREEMENT is deemed to be the property of the STATE except as may otherwise be governed by Federal or State laws, rules or regulations, or as stated in Appendix Al.

VI. Safeguards for Services and Confidentiality

A. Services performed pursuant to this AGREEMENT are secular in nature and shall be performed in a manner that does not discriminate on the basis of religious belief, or promote or discourage adherence to religion in general or particular religious beliefs.

B. Funds provided pursuant to this AGREEMENT shall not be used for any partisan political activity, or for activities that may influence legislation or the election or defeat of any candidate for public office.

C. Information relating to individuals who may receive services pursuant to this AGREEMENT shall be maintained and used only for the purposes intended under the contract and in conformity with applicable provisions of laws and regulations, or specified in Appendix A1.

118

Appendix A: Standard Clauses for NYS Contracts

The parties to the attached contract, license, lease, amendment or other agreement of any kind (hereinafter, "the contract" or "this contract") agree to be bound by the following clauses which are hereby made a part of the contract (the word "Contractor" herein refers to any party other than the State, whether a contractor, licenser, licensee, lessor, lessee or any other party):

1. EXECUTORY CLAUSE. In accordance with Section 41 of the State Finance Law, the State shall have no liability under this contract to the Contractor or to anyone else beyond funds appro-priated and available for this contract.

2. NON-ASSIGNMENT CLAUSE. In accordance with Section 138 of the State Finance Law, this contract may not be assigned by the Contractor or its right, title or interest therein assigned, transferred, conveyed, sublet or otherwise disposed of without the State’s previous written consent, and attempts to do so are null and void. Notwithstanding the foregoing, such prior written consent of an assignment of a contract let pursuant to Article XI of the State Finance Law may be waived at the discretion of the contracting agency and with the concurrence of the State Comptroller where the original contract was subject to the State Comptroller’s approval, where the assignment is due to a reorganization, merger or consolidation of the Contractor’s business entity or enterprise. The State retains its right to approve an assignment and to require that any Contractor demonstrate its responsibility to do business with the State. The Contractor may, however, assign its right to receive payments without the State’s prior written consent unless this contract concerns Certificates of Participation pursuant to Article 5-A of the State Finance Law.

3. COMPTROLLER'S APPROVAL. In accordance with Section 112 of the State Finance Law (or, if this contract is with the State University or City University of New York, Section 355 or Section 6218 of the Education Law), if this contract exceeds $50,000 (or the minimum thresholds agreed to by the Office of the State Comptroller for certain S.U.N.Y. and C.U.N.Y. contracts), or if this is an amendment for any amount to a contract which, as so amended, exceeds said statutory amount, or if, by this contract, the State agrees to give something other than money when the value or reasonably estimated value of such consideration exceeds $25,000, it shall not be valid, effective or binding upon the State until it has been approved by the State Comptroller and filed in his office. Comptroller's approval of contracts let by the Office of General Services is required when such contracts exceed $85,000 (State Finance Law § 163.6-a). However, such pre-approval shall not be required for any contract established as a centralized contract through the Office of General Services or for a purchase order or other transaction issued under such centralized contract.

4. WORKERS' COMPENSATION BENEFITS. In accordance with Section 142 of the State Finance Law, this contract shall be void and of no force and effect unless the Contractor shall provide and maintain coverage during the life of this contract for the benefit of such employees as are required to be covered by the provisions of the Workers' Compensation Law.

5. NON-DISCRIMINATION REQUIREMENTS. To the extent required by Article 15 of the Executive Law (also known

as the Human Rights Law) and all other State and Federal statutory and constitutional non-discrimination provisions, the Contractor will not discriminate against any employee or applicant for employment, nor subject any individual to harassment, because of age, race, creed, color, national origin, sexual orientation, gender identity or expression, military status, sex, disability, predisposing genetic characteristics, familial status, marital status, or domestic violence victim status or because the individual has opposed any practices forbidden under the Human Rights Law or has filed a complaint, testified, or assisted in any proceeding under the Human Rights Law. Furthermore, in accordance with Section 220-e of the Labor Law, if this is a contract for the construction, alteration or repair of any public building or public work or for the manufacture, sale or distribution of materials, equipment or supplies, and to the extent that this contract shall be performed within the State of New York, Contractor agrees that neither it nor its subcontractors shall, by reason of race, creed, color, disability, sex, or national origin: (a) discriminate in hiring against any New York State citizen who is qualified and available to perform the work; or (b) discriminate against or intimidate any employee hired for the performance of work under this contract. If this is a building service contract as defined in Section 230 of the Labor Law, then, in accordance with Section 239 thereof, Contractor agrees that neither it nor its subcontractors shall by reason of race, creed, color, national origin, age, sex or disability: (a) discriminate in hiring against any New York State citizen who is qualified and available to perform the work; or (b) discriminate against or intimidate any employee hired for the performance of work under this contract. Contractor is subject to fines of $50.00 per person per day for any violation of Section 220-e or Section 239 as well as possible termination of this contract and forfeiture of all moneys due hereunder for a second or subsequent violation.

6. WAGE AND HOURS PROVISIONS. If this is a public work contract covered by Article 8 of the Labor Law or a building service contract covered by Article 9 thereof, neither Contractor's employees nor the employees of its subcontractors may be required or permitted to work more than the number of hours or days stated in said statutes, except as otherwise provided in the Labor Law and as set forth in prevailing wage and supplement schedules issued by the State Labor Department. Furthermore, Contractor and its subcontractors must pay at least the prevailing wage rate and pay or provide the prevailing supplements, including the premium rates for overtime pay, as determined by the State Labor Department in accordance with the Labor Law. Additionally, effective April 28, 2008, if this is a public work contract covered by Article 8 of the Labor Law, the Contractor understands and agrees that the filing of payrolls in a manner consistent with Subdivision 3-a of Section 220 of the Labor Law shall be a condition precedent to payment by the State of any State approved sums due and owing for work done upon the project.

7. NON-COLLUSIVE BIDDING CERTIFICATION. In accordance with Section 139-d of the State Finance Law, if this contract was awarded based upon the submission of bids, Contractor affirms, under penalty of perjury, that its bid was arrived at independently and without collusion aimed at restricting competition. Contractor further affirms that, at the time Contractor submitted its bid, an authorized and responsible

119

person executed and delivered to the State a non-collusive bidding certification on Contractor's behalf.

8. INTERNATIONAL BOYCOTT PROHIBITION. In accordance with Section 220-f of the Labor Law and Section 139-h of the State Finance Law, if this contract exceeds $5,000, the Contractor agrees, as a material condition of the contract, that neither the Contractor nor any substantially owned or affiliated person, firm, partnership or corporation has participated, is participating, or shall participate in an international boycott in violation of the federal Export Administration Act of 1979 (50 USC App. Sections 2401 et seq.) or regulations thereunder. If such Contractor, or any of the aforesaid affiliates of Contractor, is convicted or is otherwise found to have violated said laws or regulations upon the final determination of the United States Commerce Department or any other appropriate agency of the United States subsequent to the contract's execution, such contract, amendment or modification thereto shall be rendered forfeit and void. The Contractor shall so notify the State Comptroller within five (5) business days of such conviction, determination or disposition of appeal (2 NYCRR § 105.4).

9. SET-OFF RIGHTS. The State shall have all of its common law, equitable and statutory rights of set-off. These rights shall include, but not be limited to, the State's option to withhold for the purposes of set-off any moneys due to the Contractor under this contract up to any amounts due and owing to the State with regard to this contract, any other contract with any State department or agency, including any contract for a term commencing prior to the term of this contract, plus any amounts due and owing to the State for any other reason including, without limitation, tax delinquencies, fee delinquencies or monetary penalties relative thereto. The State shall exercise its set-off rights in accordance with normal State practices including, in cases of set-off pursuant to an audit, the finalization of such audit by the State agency, its representatives, or the State Comptroller.

10. RECORDS. The Contractor shall establish and maintain complete and accurate books, records, documents, accounts and other evidence directly pertinent to performance under this contract (hereinafter, collectively, the "Records"). The Records must be kept for the balance of the calendar year in which they were made and for six (6) additional years thereafter. The State Comptroller, the Attorney General and any other person or entity authorized to conduct an examination, as well as the agency or agencies involved in this contract, shall have access to the Records during normal business hours at an office of the Contractor within the State of New York or, if no such office is available, at a mutually agreeable and reasonable venue within the State, for the term specified above for the purposes of inspection, auditing and copying. The State shall take reasonable steps to protect from public disclosure any of the Records which are exempt from disclosure under Section 87 of the Public Officers Law (the "Statute") provided that: (i) the Contractor shall timely inform an appropriate State official, in writing, that said records should not be disclosed; and (ii) said records shall be sufficiently identified; and (iii) designation of said records as exempt under the Statute is reasonable. Nothing contained herein shall diminish, or in any way adversely affect, the State's right to discovery in any pending or future litigation.

11. IDENTIFYING INFORMATION AND PRIVACY NOTIFICATION. (a) Identification Number(s). Every invoice or New York State Claim for Payment submitted to a New York State agency by a payee, for payment for the sale of goods or services or for transactions (e.g., leases, easements, licenses, etc.) related to real or personal property must include the payee's identification number. The number is any or all of the following: (i) the payee’s Federal employer identification number, (ii) the payee’s Federal social security number, and/or (iii) the payee’s Vendor Identification Number assigned by the Statewide Financial System. Failure to include such number or numbers may delay payment. Where the payee does not have such number or numbers, the payee, on its invoice or Claim for Payment, must give the reason or reasons why the payee does not have such number or numbers.

(b) Privacy Notification. (1) The authority to request the above personal information from a seller of goods or services or a lessor of real or personal property, and the authority to maintain such information, is found in Section 5 of the State Tax Law. Disclosure of this information by the seller or lessor to the State is mandatory. The principal purpose for which the information is collected is to enable the State to identify individuals, businesses and others who have been delinquent in filing tax returns or may have understated their tax liabilities and to generally identify persons affected by the taxes administered by the Commissioner of Taxation and Finance. The information will be used for tax administration purposes and for any other purpose authorized by law. (2) The personal information is requested by the purchasing unit of the agency contracting to purchase the goods or services or lease the real or personal property covered by this contract or lease. The information is maintained in the Statewide Financial System by the Vendor Management Unit within the Bureau of State Expenditures, Office of the State Comptroller, 110 State Street, Albany, New York 12236.

12. EQUAL EMPLOYMENT OPPORTUNITIES FOR MINORITIES AND WOMEN. In accordance with Section 312 of the Executive Law and 5 NYCRR Part 143, if this contract is: (i) a written agreement or purchase order instrument, providing for a total expenditure in excess of $25,000.00, whereby a contracting agency is committed to expend or does expend funds in return for labor, services, supplies, equipment, materials or any combination of the foregoing, to be performed for, or rendered or furnished to the contracting agency; or (ii) a written agreement in excess of $100,000.00 whereby a contracting agency is committed to expend or does expend funds for the acquisition, construction, demolition, replacement, major repair or renovation of real property and improvements thereon; or (iii) a written agreement in excess of $100,000.00 whereby the owner of a State assisted housing project is committed to expend or does expend funds for the acquisition, construction, demolition, replacement, major repair or renovation of real property and improvements thereon for such project, then the following shall apply and by signing this agreement the Contractor certifies and affirms that it is Contractor’s equal employment opportunity policy that:

(a) The Contractor will not discriminate against employees or applicants for employment because of race, creed, color, national origin, sex, age, disability or marital status, shall make and document its conscientious and active efforts to employ and

120

utilize minority group members and women in its work force on State contracts and will undertake or continue existing programs of affirmative action to ensure that minority group members and women are afforded equal employment opportunities without discrimination. Affirmative action shall mean recruitment, employment, job assignment, promotion, upgradings, demotion, transfer, layoff, or termination and rates of pay or other forms of compensation;

(b) at the request of the contracting agency, the Contractor shall request each employment agency, labor union, or authorized representative of workers with which it has a collective bargaining or other agreement or understanding, to furnish a written statement that such employment agency, labor union or representative will not discriminate on the basis of race, creed, color, national origin, sex, age, disability or marital status and that such union or representative will affirmatively cooperate in the implementation of the Contractor's obligations herein; and

(c) the Contractor shall state, in all solicitations or advertisements for employees, that, in the performance of the State contract, all qualified applicants will be afforded equal employment opportunities without discrimination because of race, creed, color, national origin, sex, age, disability or marital status.

Contractor will include the provisions of "a," "b," and "c" above, in every subcontract over $25,000.00 for the construction, demolition, replacement, major repair, renovation, planning or design of real property and improvements thereon (the "Work") except where the Work is for the beneficial use of the Contractor. Section 312 does not apply to: (i) work, goods or services unrelated to this contract; or (ii) employment outside New York State. The State shall consider compliance by a contractor or subcontractor with the requirements of any federal law concerning equal employment opportunity which effectuates the purpose of this clause. The contracting agency shall determine whether the imposition of the requirements of the provisions hereof duplicate or conflict with any such federal law and if such duplication or conflict exists, the contracting agency shall waive the applicability of Section 312 to the extent of such duplication or conflict. Contractor will comply with all duly promulgated and lawful rules and regulations of the Department of Economic Development’s Division of Minority and Women's Business Development pertaining hereto.

13. CONFLICTING TERMS. In the event of a conflict between the terms of the contract (including any and all attachments thereto and amendments thereof) and the terms of this Appendix A, the terms of this Appendix A shall control.

14. GOVERNING LAW. This contract shall be governed by the laws of the State of New York except where the Federal supremacy clause requires otherwise.

15. LATE PAYMENT. Timeliness of payment and any interest to be paid to Contractor for late payment shall be governed by Article 11-A of the State Finance Law to the extent required by law.

16. NO ARBITRATION. Disputes involving this contract, including the breach or alleged breach thereof, may not be submitted to binding arbitration (except where statutorily

authorized), but must, instead, be heard in a court of competent jurisdiction of the State of New York.

17. SERVICE OF PROCESS. In addition to the methods of service allowed by the State Civil Practice Law & Rules ("CPLR"), Contractor hereby consents to service of process upon it by registered or certified mail, return receipt requested. Service hereunder shall be complete upon Contractor's actual receipt of process or upon the State's receipt of the return thereof by the United States Postal Service as refused or undeliverable. Contractor must promptly notify the State, in writing, of each and every change of address to which service of process can be made. Service by the State to the last known address shall be sufficient. Contractor will have thirty (30) calendar days after service hereunder is complete in which to respond.

18. PROHIBITION ON PURCHASE OF TROPICAL HARDWOODS. The Contractor certifies and warrants that all wood products to be used under this contract award will be in accordance with, but not limited to, the specifications and provisions of Section 165 of the State Finance Law, (Use of Tropical Hardwoods) which prohibits purchase and use of tropical hardwoods, unless specifically exempted, by the State or any governmental agency or political subdivision or public benefit corporation. Qualification for an exemption under this law will be the responsibility of the contractor to establish to meet with the approval of the State.

In addition, when any portion of this contract involving the use of woods, whether supply or installation, is to be performed by any subcontractor, the prime Contractor will indicate and certify in the submitted bid proposal that the subcontractor has been informed and is in compliance with specifications and provisions regarding use of tropical hardwoods as detailed in § 165 State Finance Law. Any such use must meet with the approval of the State; otherwise, the bid may not be considered responsive. Under bidder certifications, proof of qualification for exemption will be the responsibility of the Contractor to meet with the approval of the State.

19. MACBRIDE FAIR EMPLOYMENT PRINCIPLES. In accordance with the MacBride Fair Employment Principles (Chapter 807 of the Laws of 1992), the Contractor hereby stipulates that the Contractor either (a) has no business operations in Northern Ireland, or (b) shall take lawful steps in good faith to conduct any business operations in Northern Ireland in accordance with the MacBride Fair Employment Principles (as described in Section 165 of the New York State Finance Law), and shall permit independent monitoring of compliance with such principles.

20. OMNIBUS PROCUREMENT ACT OF 1992. It is the policy of New York State to maximize opportunities for the participation of New York State business enterprises, including minority- and women-owned business enterprises as bidders, subcontractors and suppliers on its procurement contracts.

Information on the availability of New York State subcontractors and suppliers is available from:

NYS Department of Economic DevelopmentDivision for Small Business

121

Albany, New York 12245Telephone: 518-292-5100Fax: 518-292-5884email: [email protected]

A directory of certified minority- and women-owned business enterprises is available from:

NYS Department of Economic DevelopmentDivision of Minority and Women's Business Development633 Third AvenueNew York, NY 10017212-803-2414email: [email protected] M/WBE Directory

The Omnibus Procurement Act of 1992 (Chapter 844 of the Laws of 1992, codified in State Finance Law § 139-i and Public Authorities Law § 2879(3)(n)–(p)) requires that by signing this bid proposal or contract, as applicable, Contractors certify that whenever the total bid amount is greater than $1 million:

(a) The Contractor has made reasonable efforts to encourage the participation of New York State Business Enterprises as suppliers and subcontractors, including certified minority- and women-owned business enterprises, on this project, and has retained the documentation of these efforts to be provided upon request to the State;

(b) The Contractor has complied with the Federal Equal Opportunity Act of 1972 (P.L. 92-261), as amended;

(c) The Contractor agrees to make reasonable efforts to provide notification to New York State residents of employment opportunities on this project through listing any such positions with the Job Service Division of the New York State Department of Labor, or providing such notification in such manner as is consistent with existing collective bargaining contracts or agreements. The Contractor agrees to document these efforts and to provide said documentation to the State upon request; and

(d) The Contractor acknowledges notice that the State may seek to obtain offset credits from foreign countries as a result of this contract and agrees to cooperate with the State in these efforts.

21. RECIPROCITY AND SANCTIONS PROVISIONS. Bidders are hereby notified that if their principal place of business is located in a country, nation, province, state or political subdivision that penalizes New York State vendors, and if the goods or services they offer will be substantially produced or performed outside New York State, the Omnibus Procurement Act 1994 and 2000 amendments (Chapter 684 and Chapter 383, respectively, codified in State Finance Law § 165(6) and Public Authorities Law § 2879(5)) ) require that they be denied contracts which they would otherwise obtain. NOTE: As of October 2019, the list of discriminatory jurisdictions subject to this provision includes the states of South Carolina, Alaska, West Virginia, Wyoming, Louisiana and Hawaii.

22. COMPLIANCE WITH BREACH NOTIFICATION AND DATA SECURITY LAWS. Contractor shall comply with the provisions of the New York State Information Security

Breach and Notification Act (General Business Law § 899-aa and State Technology Law § 208) and commencing March 21, 2020 shall also comply with General Business Law § 899-bb.

23. COMPLIANCE WITH CONSULTANT DISCLOSURE LAW. If this is a contract for consulting services, defined for purposes of this requirement to include analysis, evaluation, research, training, data processing, computer programming, engineering, environmental, health, and mental health services, accounting, auditing, paralegal, legal or similar services, then, in accordance with Section 163 (4)(g) of the State Finance Law (as amended by Chapter 10 of the Laws of 2006), the Contractor shall timely, accurately and properly comply with the requirement to submit an annual employment report for the contract to the agency that awarded the contract, the Department of Civil Service and the State Comptroller.

24. PROCUREMENT LOBBYING. To the extent this agreement is a "procurement contract" as defined by State Finance Law §§ 139-j and 139-k, by signing this agreement the contractor certifies and affirms that all disclosures made in accordance with State Finance Law §§ 139-j and 139-k are complete, true and accurate. In the event such certification is found to be intentionally false or intentionally incomplete, the State may terminate the agreement by providing written notification to the Contractor in accordance with the terms of the agreement.

25. CERTIFICATION OF REGISTRATION TO COLLECT SALES AND COMPENSATING USE TAX BY CERTAIN STATE CONTRACTORS, AFFILIATES AND SUBCONTRACTORS. To the extent this agreement is a contract as defined by Tax Law § 5-a, if the contractor fails to make the certification required by Tax Law § 5-a or if during the term of the contract, the Department of Taxation and Finance or the covered agency, as defined by Tax Law § 5-a, discovers that the certification, made under penalty of perjury, is false, then such failure to file or false certification shall be a material breach of this contract and this contract may be terminated, by providing written notification to the Contractor in accordance with the terms of the agreement, if the covered agency determines that such action is in the best interest of the State.

26. IRAN DIVESTMENT ACT. By entering into this Agreement, Contractor certifies in accordance with State Finance Law § 165-a that it is not on the “Entities Determined to be Non-Responsive Bidders/Offerers pursuant to the New York State Iran Divestment Act of 2012” (“Prohibited Entities List”).

Contractor further certifies that it will not utilize on this Contract any subcontractor that is identified on the Prohibited Entities List. Contractor agrees that should it seek to renew or extend this Contract; it must provide the same certification at the time the Contract is renewed or extended. Contractor also agrees that any proposed Assignee of this Contract will be required to certify that it is not on the Prohibited Entities List before the contract assignment will be approved by the State.

During the term of the Contract, should the state agency receive information that a person (as defined in State Finance Law § 165-a) is in violation of the above-referenced certifications, the state agency will review such information and offer the person an

122

https://ogs.ny.gov/list-entities-determined-be-non-responsive-biddersofferers-pursuant-nys-iran-divestment-act-2012

opportunity to respond. If the person fails to demonstrate that it has ceased its engagement in the investment activity which is in violation of the Act within 90 days after the determination of such violation, then the state agency shall take such action as may be appropriate and provided for by law, rule, or contract, including, but not limited to, imposing sanctions, seeking compliance, recovering damages, or declaring the Contractor in default.

The state agency reserves the right to reject any bid, request for assignment, renewal or extension for an entity that appears on the Prohibited Entities List prior to the award, assignment, renewal or extension of a contract, and to pursue a responsibility review with respect to any entity that is awarded a contract and appears on the Prohibited Entities list after contract award.

27. ADMISSIBILITY OF REPRODUCTION OF CONTRACT. Notwithstanding the best evidence rule or any other legal principle or rule of evidence to the contrary, the Contractor acknowledges and agrees that it waives any and all objections to the admissibility into evidence at any court proceeding or to the use at any examination before trial of an electronic reproduction of this contract, in the form approved by the State Comptroller, if such approval was required, regardless of whether the original of said contract is in existence.

(October 2019)

123

Appendix A-1

Payment and Reporting

A. In the event that Contractor shall receive, from any source whatsoever, sums the payment of which is in consideration for the same costs and services provided to the State, the monetary obligation of the State hereunder shall be reduced by an equivalent amount provided, however, that nothing contained herein shall require such reimbursement where additional similar services are provided and no duplicative payments are received.

B. For each individual for whom costs are claimed under this agreement, the contractor warrants that the

individual has been classified as an employee or as an independent contractor in accordance with 2 NYCRR 315 and all applicable laws including, but not limited to, the Internal Revenue Code, the New York Retirement and Social Security Law, the New York Education Law, the New York Labor Law, and the New York Tax Law. Furthermore, the contractor warrants that all project funds allocated to the proposed budget for Employee Benefits, represent costs for employees of the contractor only and that such funds will not be expended on any individual classified as an independent contractor.

Terminations

A. The State may terminate this Agreement without cause by thirty (30) days prior written notice. In the event of such termination, the parties will adjust the accounts due and the Contractor will undertake no additional expenditures not already required. Upon any such termination, the parties shall endeavor in an orderly manner to wind down activities hereunder.

B. SED reserves the right to terminate this Agreement in the event it is found that the certification by the Contractor in accordance with New York State Finance Law §139-k was intentionally false or intentionally incomplete. Upon such finding, SED may exercise its termination right by providing written notification to the Contractor in accordance with the written notification terms of this Agreement.

Responsibility Provisions

A. General Responsibility LanguageThe Contractor shall at all times during the Contract term remain responsible. The Contractor agrees, if requested by the Commissioner of Education or his or her designee, to present evidence of its continuing legal authority to do business in New York State, integrity, experience, ability, prior performance, and organizational and financial capacity.

B. Suspension of Work (for Non-Responsibility)The Commissioner of Education or his or her designee, in his or her sole discretion, reserves the right to suspend any or all activities under this Contract, at any time, when he or she discovers information that calls into question the responsibility of the Contractor. In the event of such suspension, the Contractor will be given written notice outlining the particulars of such suspension. Upon issuance of such notice, the Contractor must comply with the terms of the suspension order. Contract activity may resume at such time as the Commissioner of Education or his or her designee issues a written notice authorizing a resumption of performance under the Contract.

C. Termination (for Non-Responsibility)Upon written notice to the Contractor, and a reasonable opportunity to be heard with appropriate SED officials or staff, the Contract may be terminated by the Commissioner of Education or his or her designee at the Contractor’s expense where the Contractor is determined by the Commissioner of Education or his

124

or her designee to be non-responsible. In such event, the Commissioner or his or her designee may complete the contractual requirements in any manner he or she may deem advisable and pursue available legal or equitable remedies for breach.

Property

A. The Contractor shall maintain a complete inventory of all realty, equipment and other non-expendable assets including, but not limited to, books, paintings, artifacts, rare coins, antiques and other collectible items purchased, improved or developed under this agreement. The Contractor shall submit a copy of the inventory in a form identical to or essentially similar to, Exhibit A annexed hereto. The term "non-expendable assets" shall mean for the purposes of this agreement any and all assets which are not consumed during the term of this agreement and which have a cost of One Thousand Dollars ($1,000) or more.

Inventories for non-expendable assets must be submitted with the final expenditure report. In addition to or as part of whatever rights the State may have with respect to the inspection of the Contractor, the State shall have the right to inspect the inventory without notice to the Contractor.

The Contractor shall not at any time sell, trade, convey or otherwise dispose of any non-expendable assets having a market value in excess of Two Thousand Dollars ($2,000) at the time of the desired disposition without the express permission of the State. The Contractor may seek permission in writing by certified mail to the State.

The Contractor shall not at any time use or allow to be used any non-expendable assets in a manner inconsistent with the purposes of this agreement.

B. If the Contractor wishes to continue to use any of the non-expendable assets purchased with the funds available under this agreement upon the termination of this agreement, it shall request permission from the State in writing for such continued use within twenty-five (25) days of the termination of this agreement. The Contractor's request shall itemize the non-expendable assets for which continued use is sought. The State may accept, reject or accept in part such request. If the request for continued use is allowed to any degree, it shall be conditioned upon the fact that said equipment shall continue to be used in accordance with the purposes of this agreement.

If after the State grants permission to the Contractor for "continued use" as set forth above the non-expendable assets are not used in accordance with the purposes of this agreement, the State in its discretion may elect to take title to such assets and may assert its right to possession upon thirty (30) days prior written notice by certified mail to the Contractor. The State upon obtaining such non-expendable assets may arrange for their further use in the public interest as it in its discretion may decide.

C. Upon termination of this agreement, the State in its discretion may elect to take title and may assert its right to possession of any non-expendable assets upon thirty (30) days prior written notice by certified mail to the Contractor. The State's option to elect to take title shall be triggered by the termination of this agreement or by the State's rejection of continued use of non-expendable assets by the Contractor as set forth herein. The State upon obtaining such non-expendable assets may arrange for their further use in the public interest as it in its discretion may decide.

D. The terms and conditions set forth herein regarding non-expendable assets shall survive the expiration or termination, for whatever reason, of this agreement.

Safeguards for Services and Confidentiality

125

A. Any copyrightable work produced pursuant to said agreement shall be the sole and exclusive property of the New York State Education Department. The material prepared under the terms of this agreement by the Contractor shall be prepared by the Contractor in a form so that it will be ready for copyright in the name of the New York State Education Department. Should the Contractor use the services of consultants or other organizations or individuals who are not regular employees of the Contractor, the Contractor and such organization or individual shall, prior to the performance of any work pursuant to this agreement, enter into a written agreement, duly executed, which shall set forth the services to be provided by such organization or individual and the consideration therefor. Such agreement shall provide that any copyrightable work produced pursuant to said agreement shall be the sole and exclusive property of the New York State Education Department and that such work shall be prepared in a form ready for copyright by the New York State Education Department. A copy of such agreement shall be provided to the State.

B. Required Web Accessibility of Delivered Documents and Applications. If applicable, all documentation, applications development, or programming delivered pursuant to the contract or procurement, will comply with New York State Education Department IT Policy NYSED-WEBACC-001, Web Accessibility Policy, which requires that documents, web-based information and applications are accessible to persons with disabilities. All delivered documentation and applications must conform to NYSED-WEBACC-001 as determined by quality assurance testing. Such quality assurance testing will be conducted by NYSED employee or contractor and the results of such testing must be satisfactory to NYSED before documents and applications will be considered a qualified deliverable under the contract or procurement.

C. All reports of research, studies, publications, workshops, announcements, and other activities funded as a result of this proposal will acknowledge the support provided by the State of New York.

D. This agreement cannot be modified, amended, or otherwise changed except by a writing signed by all parties to this contract.

E. No failure to assert any rights or remedies available to the State under this agreement shall be considered a waiver of such right or remedy or any other right or remedy unless such waiver is contained in a writing signed by the party alleged to have waived its right or remedy.

F. Expenses for travel, lodging, and subsistence shall be reimbursed at the per diem rate in effect at the time for New York State Management/Confidential employees.

G. No fees shall be charged by the Contractor for training provided under this agreement.

H. Partisan Political Activity and Lobbying. Funds provided pursuant to this Agreement shall not be used for any partisan political activity or for activities that may influence legislation or the election or defeat of any candidate for public office.

I. Nothing herein shall require the State to adopt the curriculum developed pursuant to this agreement.

J. This agreement, including all appendices, is, upon signature of the parties and the approval of the Attorney General and the State Comptroller, a legally enforceable contract. Therefore, a signature on behalf of the Contractor will bind the Contractor to all the terms and conditions stated therein.

The parties to this agreement intend the foregoing writing to be the final, complete, and exclusive expression of all the terms of their agreement.

Certifications

126

A. Contractor certifies that it has met the disclosure requirements of State Finance Law §139-k and that all information provided to the State Education Department with respect to State Finance Law §139-k is complete, true and accurate.

B. Contractor certifies that it has not knowingly and willfully violated the prohibitions against impermissible contacts found in State Finance Law §139-j.

C. Contractor certifies that no governmental entity has made a finding of nonresponsibility regarding the Contractor in the previous four years.

D. Contractor certifies that no governmental entity or other governmental agency has terminated or withheld a procurement contract with the Contractor due to the intentional provision of false or incomplete information.

E. Contractor affirms that it understands and agrees to comply with the procedures of the STATE relative to permissible contacts as required by State Finance Law §139-j (3) and §139-j (6)(b).

F. Contractor certifies that it is in compliance with NYS Public Officers Law, including but not limited to, §73(4)(a).

Notices

Any written notice or delivery under any provision of this AGREEMENT shall be deemed to have been properly made if sent by certified mail, return receipt requested to the address(es) set forth in this Agreement, except as such address(es) may be changed by notice in writing. Notice shall be considered to have been provided as of the date of receipt of the notice by the receiving party.

Miscellaneous

A. Contractor shall comply with the provisions of the New York State Information Security Breach and Notification Act (General Business Law Section 899-aa; State Technology Law Section 208). Contractor shall be liable for the costs associated with such breach if caused by Contractor’s negligent or willful acts or omissions, or the negligent or willful acts or omissions of Contractor’s agents, officers, employees or subcontractors.

B. If required by the Office of State Comptroller (“OSC”) Bulletin G-226 and State Finance Law §§ 8 and 163, Contractor agrees to submit an initial planned employment data report on Form A and an annual employment report on Form B. State will furnish Form A and Form B to Contractor if required.

The initial planned employment report must be submitted at the time of approval of this Agreement. The annual employment report on Form B is due by May 15th of each year and covers actual employment data performed during the prior period of April 1st to March 31st. Copies of the report will be submitted to the NYS Education Department, OSC and the NYS Department of Civil Service at the addresses below.

By mail: NYS Office of the State ComptrollerBureau of Contracts110 State Street, 11th FloorAlbany, NY 12236Attn: Consultant Reporting

By fax: (518) 474-8030 or (518) 473-8808

127

Reports to DCS are to be transmitted as follows:

By mail: NYS Department of Civil ServiceOffice of CounselAlfred E. Smith Office BuildingAlbany, NY 12239

Reports to NYSED are to be transmitted as follows:

By mail: NYS Education DepartmentContract Administration UnitRoom 505 W EBAlbany, NY 12234

By fax: (518) 408-1716

C. Consultant Staff Changes. If this is a contract for consulting services, Contractor will maintain continuity of the consultant team staff throughout the course of the contract. All changes in staff will be subject to STATE approval. The replacement consultant(s) with comparable skills will be provided at the same or lower hourly rate.

D. Order of Precedence. In the event of any discrepancy, disagreement, conflict or ambiguity between the various documents, attachments and appendices comprising this contract, they shall be given preference in the following order to resolve any such discrepancy, disagreement, conflict or ambiguity:

1. Appendix A – Standard Clauses for all State Contracts 2. State of New York Agreement3. Appendix A-1 - Agency Specific Clauses4. Appendix X - Sample Modification Agreement Form (where applicable)5. Appendix A-3 - Minority/Women-owned Business Enterprise Requirements (where applicable)6. Appendix B - Budget7. Appendix C – Payment and Reporting Schedule8. Appendix R – Data Security and Privacy Plan (where applicable)9. Appendix S – Parents’ Bill of Rights for Data Privacy and Security (where applicable)10. Appendix S-1 - Attachment to Parents’ Bill of Rights (where applicable)11. Appendix D – Program Workplan

(Revised 6/12/17)

128

Appendix R: Data Security and Privacy Plan Provisions

1. The individually identifiable data provided to or stored by the Contractor pursuant to this agreement (the "Data") are sensitive, requiring appropriate levels of security to prevent unauthorized disclosure or modification. The Contractor shall take all reasonable measures to protect the confidentiality of the Data as required by federal and state laws and regulations applicable to the Contractor. These may include but are not limited to the New York State Social Services Law, Personal Privacy Protection Law and Education Law §2-d; the federal Social Security Act and Family Educational Rights and Privacy Act; internet security laws; and any regulations promulgated thereunder.

2. The Contractor has full and final responsibility for the security of the Data. The Contractor agrees to implement reasonable technical and physical security measures to ensure the confidentiality, integrity and availability of the Data. Such security measures may be reviewed by the State, both through an informal audit of policies and procedures and/or through inspection of security methods used within the Contractor's infrastructure, storage, and other physical security. The Contractor should review its implementation and maintenance of its security review periodically to protect the data in strict compliance with statutory and regulatory requirements.

3. The Contractor's security measures must also include:

a. Provision that access to the Data is restricted solely to staff who need such access to carry out the responsibilities of the Contractor under this agreement, and that such staff will not release such Data to any unauthorized party;b. All confidential Data are stored on computer and storage facilities maintained within Contractor's computer networks, behind appropriate firewalls;c. Access to computer applications and Data are managed through appropriate user ID/password procedures;d. Contractor's computer network storing the Data is scanned for inappropriate access through an intrusion detection system. NYSED has the right to perform a site visit to review the vendor’s security practices if NYSED feels it is necessary;e. That Contractor has a disaster recovery plan that is acceptable to the State;f. Satisfactory redundant and uninterruptible power and fiber infrastructure provisions; andg. A copy of the Contractor's security review evidencing compliance with these requirements must be submitted to NYSED for review and approval within 6 months of the signing of the contract or before the first certification test is performed, whichever occurs first.

4. The Data must be returned to NYSED upon termination or expiration of this Agreement, or at such point that the Data are no longer needed for the purpose referenced in this Agreement, or, at the sole discretion of NYSED, securely destroyed. All hard copies of personally identifiable Data in the possession of the Contractor must be securely destroyed, and all electronic Data must be purged from the network in a manner that does not permit retrieval of the data. The contractor is specifically prohibited from commingling

129

any data from outside sources into the Data received from NYSED, except as specifically authorized by NYSED.

5. If personally identifiable data of students, teachers or building principals will be disclosed to the Contractor by NYSED for purposes of the Contractor providing services to NYSED, the Contractor must comply with the following requirements of Education Law §2-d (Chapter 56, Subpart L of the Laws of 2014) and any implementing regulations:

a. Any officers or employees of the third-party contractor and its assignees who have access to student data or teacher or principal data have received or will receive training on the federal and state law governing confidentiality of such data prior to receiving access;b. limits internal access to education records to those individuals that are determined to have legitimate educational interests;c. not use the education records for any other purposes than those explicitly authorized in its contract;d. except for authorized representatives of the third-party contractor to the extent they are carrying out the contract, not disclose any personally identifiable information to any other party:(i) without the prior written consent of the parent or eligible student; or(ii) unless required by statute or court order and the party provides a notice of the disclosure to the department, district board of education, or institution that provided the information no later than the time the information is disclosed, unless providing notice of the disclosure is expressly prohibited by the statute or court order;e. maintains reasonable administrative, technical and physical safeguards to protect the security, confidentiality and integrity of personally identifiable student information in its custody; andf. uses encryption technology consistent with Education Law §2-d and any implementing regulations.

6. If requested by NYSED to make any disclosure of aggregated data using the Data provided to or stored by the Contractor, Contractor must ensure that the disclosed aggregated data cannot reasonably be used to identify a particular individual. Aggregated data will be considered identifiable if the disclosure has less than five (5) data elements per cell or the data elements per cell comprise 100% of the subject population.

7. Contractor agrees that all Data shall remain at all times the property of the State and may not be used for any purpose other than the purpose outlined in this Agreement without the express written permission of NYSED. The Contractor has no ownership of or licensing rights to the Data except as provided in this Agreement, and Contractor specifically agrees that it will not sell, give or otherwise transfer the Data to any third party without NYSED’s express prior approval.

8. The Contractor must ensure that these confidentiality and security provisions apply to any subcontractor engaged by the Contractor for the work under this agreement. The Contractor shall take full responsibility for the acts and omissions of its subcontractors, and

130

the use of subcontractors shall not impair the rights of NYSED against the Contractor in accordance with this Agreement.

9. Hardware, software and services acquired by the Contractor under this Agreement may not be used for other activities beyond those described in the scope of the contract unless authorized in advance by NYSED.

10. Security of Location - Server room will remain a restricted access, locked room with access via security cards. The list of staff with access to the server room will continue to be reviewed quarterly against the number of times each staff actually gained access to the server room.

11. Breach Notification.

a. Contractor that receives student data or teacher or principal data pursuant to a contract or other written agreement with an educational agency shall be required to notify such educational agency of any breach of security resulting in an unauthorized release of such data in accordance with Education Law §2-d and any implementing regulations. Upon such notification, the educational agency shall take appropriate action in accordance with Education Law §2-d and any implementing regulations.

b. In the event that the State is required, pursuant to Education Law §2-d(6)(b), to notify one or more parent, eligible student, teacher or principal of an unauthorized release of student data by the Contractor or its assignee, the Contractor shall promptly reimburse the State for the full cost of such notification.

c. Contractor acknowledges that it may be subject to penalties under Education Law §§2-d(6)and 2-d(7) for unauthorized disclosure of personally identifiable student, teacher or principal data.

d. Contractor agrees that it will cooperate and promptly comply with any inquiries from the State based upon the State’s receipt of a complaint or other information indicating that an improper or unauthorized disclosure of personally identifiable information may have occurred. Contractor will permit on-site examination and inspection and will provide at its own cost necessary documentation or testimony of any employee, representative or assignee of Contractor relating to the alleged improper disclosure of data.

* The Laws of New York State: http://public.leginfo.state.ny.us/menugetf.cgi?COMMONQUERY=LAWS: Regulations for New York State Social Services : Social Services Law §§ 136, 372, 390(3)(c)(iv), 409-f, 422, 444 and 460-e; 18 NYCRR Part 339, Part 357, § 414.15(a)(5), § 416.15(a)(7), § 417.15 (a)(7), § 418-1.15 (a) (5), § 418-2.15(a) (7), Part 445 and Part 466 New York State Personal Privacy Laws (PPPL): 21 NYCRR Chapter XXV and 19 NYCRR Part 81SED-specific PPPL regulations: 8 NYCRR Part 187

131

Appendix S: Parents’ Bill of Rights for Data Privacy and Security

To satisfy their responsibilities regarding the provision of education to students in pre-kindergarten through grade twelve, “educational agencies” (as defined below) in the State of New York collect and maintain certain personally identifiable information from the education records of their students. As part of the Common Core Implementation Reform Act, Education Law §2-d requires that each educational agency in the State of New York must develop a Parents’ Bill of Rights for Data Privacy and Security (Parents’ Bill of Rights). The Parents’ Bill of Rights must be published on the website of each educational agency, and must be included with every contract the educational agency enters into with a “third party contractor” (as defined below) where the third party contractor receives student data, or certain protected teacher/principal data related to Annual Professional Performance Reviews that is designated as confidential pursuant to Education Law §3012-c (“APPR data”).

The purpose of the Parents’ Bill of Rights is to inform parents (which also include legal guardians or persons in parental relation to a student, but generally not the parents of a student who is age eighteen or over) of the legal requirements regarding privacy, security and use of student data. In addition to the federal Family Educational Rights and Privacy Act (FERPA), Education Law §2-d provides important new protections for student data, and new remedies for breaches of the responsibility to maintain the security and confidentiality of such data.

A. What are the essential parents’ rights under the Family Educational Rights and Privacy Act (FERPA) relating to personally identifiable information in their child’s student records?

The rights of parents under FERPA are summarized in the Model Notification of Rights prepared by the United States Department of Education for use by schools in providing annual notification of rights to parents. It can be accessed at http://www2.ed.gov/policy/gen/guid/fpco/ferpa/lea-officials.html, and a copy is attached to this Parents’ Bill of Rights. Complete student records are maintained by schools and school districts, and not at the New York State Education Department (NYSED). Further, NYSED would need to establish and implement a means to verify a parent’s identity and right of access to records before processing a request for records to the school or school district. Therefore, requests to access student records will be most efficiently managed at the school or school district level.

Parents’ rights under FERPA include:

1. The right to inspect and review the student's education records within 45 days after the day the school or school district receives a request for access.

2. The right to request amendment of the student’s education records that the parent or eligible student believes are inaccurate, misleading, or otherwise in violation of the student’s privacy rights under FERPA. Complete student records are maintained by schools and school districts and not at NYSED, which is the

132

secondary repository of data, and NYSED make amendments to school or school district records. Schools and school districts are in the best position to make corrections to students’ education records.

3. The right to provide written consent before the school discloses personally identifiable information (PII) from the student's education records, except to the extent that FERPA authorizes disclosure without consent (including but not limited to disclosure under specified conditions to: (i) school officials within the school or school district with legitimate educational interests; (ii) officials of another school for purposes of enrollment or transfer; (iii) third party contractors providing services to, or performing functions for an educational agency; (iv) authorized representatives of the U. S. Comptroller General, the U. S. Attorney General, the U.S. Secretary of Education, or State and local educational authorities, such as NYSED; (iv) (v) organizations conducting studies for or on behalf of educational agencies) and (vi) the public where the school or school district has designated certain student data as “directory information” (described below). The attached FERPA Model Notification of Rights more fully describes the exceptions to the consent requirement under FERPA).

4. Where a school or school district has a policy of releasing “directory information” from student records, the parent has a right to refuse to let the school or school district designate any all of such information as directory information. Directory information, as defined in federal regulations, includes: the student’s name, address, telephone number, email address, photograph, date and place of birth, major field of study, grade level, enrollment status, dates of attendance, participation in officially recognized activities and sports, weight and height of members of athletic teams, degrees, honors and awards received, and the most recent educational agency or institution attended. Where disclosure without consent is otherwise authorized under FERPA, however, a parent’s refusal to permit disclosure of directory information does not prevent disclosure pursuant to such separate authorization.

5. The right to file a complaint with the U.S. Department of Education concerning alleged failures by the School to comply with the requirements of FERPA.

B. What are parents’ rights under the Personal Privacy Protection Law (PPPL), Article 6-A of the Public Officers Law relating to records held by State agencies?The PPPL (Public Officers Law §§91-99) applies to all records of State agencies and is not specific to student records or to parents. It does not apply to school districts or other local educational agencies. It imposes duties on State agencies to have procedures in place to protect from disclosure of “personal information,” defined as information which because of a name, number, symbol, mark or other identifier, can be used to identify a “data subject” (in this case the student or the student’s parent). Like FERPA, the PPPL confers a right on the data subject (student or the student’s parent) to access to State agency records relating to them and requires State agencies to have procedures for correction or amendment of records.

133

A more detailed description of the PPPL is available from the Committee on Open Government of the New York Department of State. Guidance on what you should know about the PPPL can be accessed at http://www.dos.ny.gov/coog/shldno1.html. The Committee on Open Government’s address is Committee on Open Government, Department of State, One Commerce Plaza, 99 Washington Avenue, suite 650, Albany, NY 12231, their email address is [email protected], and their telephone number is (518) 474-2518.

C. Parents’ Rights Under Education Law §2-d relating to Unauthorized Release of Personally Identifiable Information

1. What “educational agencies” are included in the requirements of Education Law §2-d?

The New York State Education Department (“NYSED”); Each public-school district; Each Board of Cooperative Educational Services or BOCES; and All schools that are:

o a public elementary or secondary school; o a universal pre-kindergarten program authorized pursuant to Education Law

§3602-e;o an approved provider of preschool special education services;o any other publicly funded pre-kindergarten program; o a school serving children in a special act school district as defined in

Education Law 4001; oro certain schools for the education of students with disabilities - an approved

private school, a state-supported school subject to the provisions of Education Law Article 85, or a state-operated school subject to Education Law Article 87 or 88.

2. What kind of student data is subject to the confidentiality and security requirements of Education Law §2-d?

The law applies to personally identifiable information contained in student records of an educational agency listed above. The term “student” refers to any person attending or seeking to enroll in an educational agency, and the term “personally identifiable information” (“PII”) uses the definition provided in FERPA. Under FERPA, personally identifiable information or PII includes, but is not limited to:

(a) The student’s name; (b) The name of the student’s parent or other family members; (c) The address of the student or student’s family; (d) A personal identifier, such as the student’s social security number, student number, or biometric record;

134

(e) Other indirect identifiers, such as the student’s date of birth, place of birth, and Mother’s Maiden Name4; (f) Other information that, alone or in combination, is linked or linkable to a specific student that would allow a reasonable person in the school community, who does not have personal knowledge of the relevant circumstances, to identify the student with reasonable certainty; or (g) Information requested by a person who the educational agency or institution reasonably believes knows the identity of the student to whom the education record relates.

3. What kind of student data is not subject to the confidentiality and security requirements of Education Law §2-d?

The confidentiality and privacy provisions of Education Law §2-d and FERPA extend only to PII, and not to student data that is not personally identifiable. Therefore, de-identified data (e.g., data regarding students that uses random identifiers), aggregated data (e.g., data reported at the school district level) or anonymized data that could not be used to identify a particular student is not considered to be PII and is not within the purview of Education Law §2-d or within the scope of this Parents’ Bill of Rights.

4. What are my rights under Education Law § 2-d as a parent regarding my student’s PII?

Education Law §2-d ensures that, in addition to all of the protections and rights of parents under the federal FERPA law, certain rights will also be provided under the Education Law. These rights include, but are not limited to, the following elements:

(A) A student's PII cannot be sold or released by the educational agency for any commercial or marketing purposes.

o PII may be used for purposes of a contract that provides payment to a vendor for providing services to an educational agency as permitted by law.

o However, sale of PII to a third party solely for commercial purposes or receipt of payment by an educational agency, or disclosure of PII that is not related to a service being provided to the educational agency, is strictly prohibited.

(B) Parents have the right to inspect and review the complete contents of their child's education record including any student data stored or maintained by an educational agency.o This right of inspection is consistent with the requirements of FERPA. In

addition to the right of inspection of the educational record, Education Law §2-d provides a specific right for parents to inspect or receive copies of any data in the student’s educational record.

4 Please note that NYSED does not collect certain information defined in FERPA, such as students’ social security numbers, biometric records, mother’s maiden name (unless used as the mother’s legal name).

135

o NYSED will develop policies for annual notification by educational agencies to parents regarding the right to request student data. Such policies will specify a reasonable time for the educational agency to comply with such requests.

o The policies will also require security measures when providing student data to parents, to ensure that only authorized individuals receive such data. A parent may be asked for information or verifications reasonably necessary to ensure that he or she is in fact the student’s parent and is authorized to receive such information pursuant to law.

(C) State and federal laws protect the confidentiality of PII, and safeguards associated with industry standards and best practices, including, but not limited to, encryption, firewalls, and password protection, must be in place when data is stored or transferred.

Education Law §2-d also specifically provides certain limitations on the collection of data by educational agencies, including, but not limited to:

(A) A mandate that, except as otherwise specifically authorized by law, NYSED shall only collect PII relating to an educational purpose;

(B) NYSED may only require districts to submit PII, including data on disability status and student suspensions, where such release is required by law or otherwise authorized under FERPA and/or the New York State Personal Privacy Law; and

(C) Except as required by law or in the case of educational enrollment data, school districts shall not report to NYSED student data regarding juvenile delinquency records, criminal records, medical and health records or student biometric information.

(D) Parents may access the NYSED Student Data Elements List, a complete list of all student data elements collected by NYSED, at http://www.p12.nysed.gov/irs/sirs/, or may obtain a copy of this list by writing to the Office of Information & Reporting Services, New York State Education Department, Room 863 EBA, 89 Washington Avenue, Albany, NY 12234; and

(E) Parents have the right to file complaints with an educational agency about possible breaches of student data by that educational agency’s third-party contractors or their employees, officers, or assignees, or with NYSED. Complaints to NYSED should be directed in writing to the Chief Privacy Officer, New York State Education Department, 89 Washington Avenue, Albany NY 12234, email to [email protected]. The complaint process is under development and will be established through regulations to be proposed by NYSED’s Chief Privacy Officer, who has not yet been appointed. o Specifically, the Commissioner of Education, after consultation with the

Chief Privacy Officer, will promulgate regulations establishing procedures for the submission of complaints from parents, classroom teachers or

136


http://www.p12.nysed.gov/irs/sirs/

building principals, or other staff of an educational agency, making allegations of improper disclosure of student data and/or teacher or principal APPR data by a third-party contractor or its officers, employees or assignees.

o When appointed, the Chief Privacy Officer of NYSED will also provide a procedure within NYSED whereby parents, students, teachers, superintendents, school board members, principals, and other persons or entities may request information pertaining to student data or teacher or principal APPR data in a timely and efficient manner.

5. Must additional elements be included in the Parents’ Bill of Rights.?Yes. For purposes of further ensuring confidentiality and security of student data, as an appendix to the Parents’ Bill of Rights each contract an educational agency enters into with a third-party contractor shall include the following supplemental information:(A) the exclusive purposes for which the student data, or teacher or principal data,

will be used;

(B) how the third-party contractor will ensure that the subcontractors, persons or entities that the third-party contractor will share the student data or teacher or principal data with, if any, will abide by data protection and security requirements;

(C)when the agreement with the third-party contractor expires and what happens to the student data or teacher or principal data upon expiration of the agreement;

(D) if and how a parent, student, eligible student, teacher or principal may challenge the accuracy of the student data or teacher or principal data that is collected; and

(E) where the student data or teacher or principal data will be stored (described in such a manner as to protect data security), and the security protections taken to ensure such data will be protected, including whether such data will be encrypted.

a. In addition, the Chief Privacy Officer, with input from parents and other education and expert stakeholders, is required to develop additional elements of the Parents’ Bill of Rights to be prescribed in Regulations of the Commissioner.

6. What protections are required to be in place if an educational agency contracts with a third-party contractor to provide services, and the contract requires the disclosure of PII to the third-party contractor?Education Law §2-d provides very specific protections for contracts with “third party contractors”, defined as any person or entity, other than an educational agency, that receives student data or teacher or principal data from an educational agency pursuant to a contract or other written agreement for purposes of providing services to such educational agency. The term “third party contractor” also includes an educational partnership organization that receives student and/or teacher or principal APPR data from a school district to carry out its responsibilities pursuant to Education

137

Law §211-e, and a not-for-profit corporation or other non-profit organization, which are not themselves covered by the definition of an “educational agency.”Services of a third-party contractor covered under Education Law §2-d include, but not limited to, data management or storage services, conducting studies for or on behalf of the educational agency, or audit or evaluation of publicly funded programs. When an educational agency enters into a contract with a third party contractor, under which the third party contractor will receive student data, the contract or agreement must include a data security and privacy plan that outlines how all state, federal, and local data security and privacy contract requirements will be implemented over the life of the contract, consistent with the educational agency's policy on data security and privacy. However, the standards for an educational agency’s policy on data security and privacy must be prescribed in Regulations of the Commissioner that have not yet been promulgated. A signed copy of the Parents’ Bill of Rights must be included, as well as a requirement that any officers or employees of the third-party contractor and its assignees who have access to student data or teacher or principal data have received or will receive training on the federal and state law governing confidentiality of such data prior to receiving access.Each third-party contractor that enters into a contract or other written agreement with an educational agency under which the third party contractor will receive student data or teacher or principal data shall:

o limit internal access to education records to those individuals that are determined to have legitimate educational interests

o not use the education records for any other purposes than those explicitly authorized in its contract;

o except for authorized representatives of the third party contractor to the extent they are carrying out the contract, not disclose any PII to any other party (i) without the prior written consent of the parent or eligible student; or (ii) unless required by statute or court order and the party provides a notice of the disclosure to NYSED, district board of education, or institution that provided the information no later than the time the information is disclosed, unless providing notice of the disclosure is expressly prohibited by the statute or court order;

o maintain reasonable administrative, technical and physical safeguards to protect the security, confidentiality and integrity of PII in its custody; and

o use encryption technology to protect data while in motion or in its custody from unauthorized disclosure.

7. What steps can and must be taken in the event of a breach of confidentiality or security?

Upon receipt of a complaint or other information indicating that a third party contractor may have improperly disclosed student data, or teacher or principal APPR data, NYSED’s Chief Privacy Officer is authorized to investigate, visit, examine and inspect

138

the third party contractor's facilities and records and obtain documentation from, or require the testimony of, any party relating to the alleged improper disclosure of student data or teacher or principal APPR data.Where there is a breach and unauthorized release of PII by a by a third party contractor or its assignees (e.g., a subcontractor): (i) the third party contractor must notify the educational agency of the breach in the most expedient way possible and without unreasonable delay; (ii) the educational agency must notify the parent in the most expedient way possible and without unreasonable delay; and (iii) the third party contractor may be subject to certain penalties including, but not limited to, a monetary fine; mandatory training regarding federal and state law governing the confidentiality of student data, or teacher or principal APPR data; and preclusion from accessing any student data, or teacher or principal APPR data, from an educational agency for a fixed period up to five years.8. Data Security and Privacy StandardsUpon appointment, NYSED’s Chief Privacy Officer will be required to develop, with input from experts, standards for educational agency data security and privacy policies. The Commissioner will then promulgate regulations implementing these data security and privacy standards.

9. No Private Right of Action

Please note that Education Law §2-d explicitly states that it does not create a private right of action against NYSED or any other educational agency, such as a school, school district or BOCES.

139

Appendix S1: Attachment to Parents’ Bill Of Rights For Contracts Involving Disclosure of Certain Personally Identifiable Information

Education Law §2-d, added by Ch. 56 of the Laws of 2014, requires that a Parents’ Bill of Rights be attached to every contract with a third-party contractor (as defined in the law) which involves the disclosure of personally identifiable information (PII) derived from student education records (“Student Data”), or certain teacher/principal information regarding annual professional performance evaluations that is confidential pursuant to Education Law §30212-c (“APPR Data”). Each such Contract must include this completed Attachment to provide specific information about the use of such data by the Contractor.

1. Specify whether this Contract involves disclosure to the Contractor of Student Data, APPR Data, or both.

Disclosure of Student Data

Disclosure of APPR Data

2. Describe the exclusive purposes for which the Student Data or APPR Data will be used in the performance of this contract.

3. Identify any subcontractors or other persons/entities with whom the Contractor will share the Student Data or APPR in the performance of this Contract and describe how the Contractor will ensure that such persons/entities will abide by the data protection and security requirements of the Contract.

Subcontractors or other entities with whom the Contractor will share data:

In the event the Contractor engages a Subcontractor or otherwise shares Student Data or APPR Data with any other entity, Contractor acknowledges and agrees that before any such data is shared with a Contractor or another entity, such party must agree in writing to be bound by the confidentiality and data protection provisions set forth in this Contract including, but not limited to, the “Data Security and Privacy Plan” set forth in Appendix R.

140

Student demographic data will be provided to the Contractor for the purpose of it performing the following tasks for NYSED: conducting field testing and operational testing, analyzing field test and operational test results, and conducting standard setting. The Contractor will also be gathering student test result data in scoring students’ responses to multiple-choice questions and determining students’ scale score and

Bidder should specifically list in this section any/all subcontractors that will/may receive data.

Upon termination of the agreement between the Contractor and a Subcontractor or other entity, Contractor acknowledges and agrees that it is responsible for ensuring that all Student Data or APPR Data shared by the Contractor must be returned to Contractor or otherwise destroyed as provided in Paragraph 4 of the “Data Security and Privacy Plan” set forth in Appendix R.

4. Specify the expiration date of the Contract and explain what will happen to the Student Data or APPR Data in the Contractor’s possession, or the possession of any person/entity described in response to Paragraph 3, upon the expiration or earlier termination of the Contract.

Contract expiration date: 11/30/25

Contractor agrees to return the Student Data or APPR Data to NYSED consistent with the protocols set forth in Paragraph 4 of the “Data Security and Privacy Plan” set forth in Appendix R.

Contractor agrees to securely destroy the Student Data or APPR Data consistent with the protocols set forth in Paragraph 4 of the “Data Security and Privacy Plan” set forth in Appendix R.

5. State whether the Contractor will be collecting any data from or pertaining to students derived from the student’s education record, or pertaining to teachers or principals’ annual professional performance evaluation pursuant to the Contract, and explain if and how a parent, student, eligible student (a student eighteen years or older), teacher or principal may challenge the accuracy of the Student Data or APPR data that is collected.

Student Data

APPR Data

Any challenges to the accuracy of any of the Student Data or APPR Data shared pursuant to this Contract should be addressed to the school, educational agency or entity which produced, generated or otherwise created such data.

6. Describe where the Student Data or APPR Data will be stored (in a manner that does not jeopardize data security), and the security protections taken to ensure that the data will be protected, including whether such data will be encrypted.

141

Bidder should detail in this section where data will be stored, what security measures will be in place, and whether electronic data is encrypted in motion and/or at rest.

5.) ATTACHMENTS

Attachment A: Minimum Technical Requirements for Computer Devices to be Used by Schools for CBT

At the time of the development of this RFP, the minimum device specifications for New York State schools are provided below. The contractor shall ensure the test administration and test delivery platforms are compatible with the existing computers and infrastructure likely to be available in New York State’s schools.

General System Requirements:

System Memory/Hard Disk Spaceo Minimum 512MB Free RAMo Recommended 1GB Free RAMo Minimum 1 GB Free Storage Space

LAN Networko Recommended available LAN bandwidth at each workstation 2Mbps

Internet Speedo Minimum per device: 150Kbpso Recommended: 300Kbps

Connectivityo Wiredo Wireless

Input Device Requirementso Keyboardo Mouseo Touchpado Touchscreeno Styluso Digital Pens

Headphones/Earphones/Microphones Required

External Keyboards

o Due to the onscreen space occupied by a tablet’s virtual keyboard, external keyboards are required for test takers using tablets so as not to limit or obscure the view of test item content and related functionalities when text input is required.

Thin Client Environmento Each computer operating in a thin client environment must meet or exceed minimum

hardware specifications, as well as, bandwidth and security requirements.

142

OS Specific System Requirements:

Operating System

OS Version Processor Screen Size Resolution

Windows 8.1+ Intel Core 2 Duo 1.6 GHz equivalent or higher performing CPU

Minimum 10” -11.6+ screen sizeMinimum 1024-768 screen resolutionRecommended 11.6 or larger screen size

Mac Mac OS X10.10-10.14 Need to be able to support mass installations

Intel Core 2 Duo 1.6 GHz equivalent or higher performing CPU

Minimum 10: screen sizeMinimum 1024 X 768 screen resolutionRecommended 11.6 or larger screen size

Apple iOSiPad 4 or newer

10.3, 11.2.5 to 12.1.4

1.0 GHz dual core equivalent or higher

Minimum 9.7 screen sizeMinimum 1024 X 768 screen resolution

Chrome V59-72 + 1.6 GHz dual core equivalent or higher

Minimum 10” screen sizeMinimum 1024 X 768 screen resolution

Windows Tablets 8.1+ 1.0 GHz dual core equivalent or higher


Android Tablets 4.2+ Intel Atom® Processor Z2520 1.2 GHz, or faster processor


Linux Intel Core 2 Duo 1.6 GHz equivalent or higher performing CPU

Minimum 10inch -11.6+ screen sizeMinimum 1024 X 768 screen resolutionRecommended 11.6 or larger screen size

143

Attachment B: Grades 3-8 ELA and Math Test Development Process

Major Events Process Step Time FrameFi

eld

Test

Ban

k D

evel

opm

ent

Passage Development

1. Passage Selection Summer2. Content Review Early Fall3. Educator Passage Review

MeetingLate Fall

Item Development

4. Four Regional Item Writing Workshops

Early Winter

5. Item Editing Winter6. NYSED Item Pre-Review Late Winter – Early

Spring7. Internal Item Review Early Summer8. First Copy Edit/Content QC Early Summer9. External Quality Control Early Summer10. Educator Item Review Meeting Summer11. Reconciliation Summer


12. Formal Copy Edit* Early Fall13. Stand-Alone Field Test Form

Construction Fall – Late Fall

14. Field Test Form NYSED-Review

Rangefinding15. Rangefinding Meeting Mid-July16. Rangefinding Reconciliation Mid-July17. SAFT CR Scoring Late August

Ope

ratio

nal F

orm

s D

evel

opm

ent

Forms Construction/ Forms Finalization

18. First Draw (Contractor) Late Summer19. Form Pre-Review (NYSED) Late Summer/Early

Fall20. Content Review/Risk Assessment Fall21. Forms Construction Meeting November22. Final Eyes Meeting/Reconciliation November23. Form Copy Edit 1 Late Fall24. Form Copy Edit 2 Late Fall/Early

Winter25. Final Executive Review Early Winter

Scoring Ancillaries

26. Proposed Operational Scoring Materials Sets

Late Fall

27. OSA Review of Scoring Materials Sets

Late Fall/Early Winter

28. Scoring Copy Edit and Senior Management Approval

Early Winter

Item Release 29. Preparation of Selected Items for Public Release

June

All Activities Requiring Educators

a. Educator Committee Selection Ongoing

* The last step in process where item editing is permitted

144

Attachment C: Passage Development

Time Frame: Late Summer – Late Fall

Distinct Steps:

1. Passage Selection (Contractor)2. Content Review 3. Educator Passage Review Meeting

Guiding Documents:

Passage Selection Guidelines for Assessing ELA Passage Review Criteria

o Quantitativeo Qualitative

145

Step 1: Passage SelectionOwner: Contractor

Goal/Outcome:1. To present to OSA passages in defined batches that meet expectations set

forth in Passage Selection Guidelines for Assessing ELA and Test Specifications.

2. Contractor will provide clear timeline around batch delivery.

Process: Contractor Defined

QA within Process: Contractor Defined

QC: Contractor Defined

Documentation: Quantitative Criteria Qualitative Criteria

Metrics: Tracking of Outcomes Data (i.e., accept, reject, moved to another grade, etc.)

146

Step 2: Content Review

Owner: OSA

Participants: OSA Content Team – content review and manage batch

review/documentation. Education Specialists – content review, as needed. Supervisor – monitor the progress by OSA Content Team/Education

Specialists.

Goal/Outcome: To ensure that passages presented meet the guidelines set out in the Passage Selection Guidelines for Assessing ELA. As such, OSA ensures that each passage is:

capable of supporting questions to specified standards/item types, appropriately complex for the grade, appropriately complex for the item format, and of sufficient quality.

Process: 1. OSA Content Team will run and verify whether quantitative criteria are met

(with Education Specialist support, if needed).2. OSA Content Team checks whether qualitative criteria are met.3. OSA Content Team fills out criteria sheets, including their judgement to

accept, reject or move to higher grade, and sends to Contractor.

QA within Process: OSA Content Team must verify that quantitative, qualitative, and characteristic

guidelines are followed.

Documentation: Signed off criteria sheets, including quantitative, qualitative, judgement, and

meta-data- OSA Content Team

Metrics: Tracking of rate of approval from OSA (and type of reject)

147

Step 3: Educator Passage Review MeetingOwner: Contractor

Participants/Roles: Educator Panelists – 6 from each grade- review passages and render the final

judgement. Contractor – facilitates, keeps master record, note-takers. OSA Content Team – participates in the discussion. OSA Management/Supervisor – review for sensitivity, grade appropriateness, and

quality.

Goal/Outcome: To gain actionable feedback on all passages from field educators. Educators provide comment on the following:

Passage:o The passage readability is appropriate for grade level.o The passage is appropriate and fair for grade level. o The passage can be read in the time allotted for grade level.o The passage contains attributes that will support items aligned to certain

standards at grade level.

Process:1. Training on test development overview, passage review process, and review criteria.2. Educators read passages, then discuss;o Work through passages with meta-data availableo Verify alignmento Record answers to questions listed in goals section above/educator

discussion/majority vote to accept, reject, or moved to other grade level

QA within Process: Questions listed in goals section

QC: Reconciliation External Review

Documentation: Judgement Forms (or electronic documentation) Contractor records

Metrics: Tracking % responses to questions listed in goals section

148

Attachment D: Item Development

Time Frame: Early Winter – Late Summer

Distinct Steps:

4. Item Writing Workshops5. Item Editing6. OSA Item Pre-Review7. Internal Item Review Meeting8. First Copy Edit/Content QC9. External Quality Control10.Educator Review Meeting11.Reconciliation

Guiding Documents:

Universal Design (UD) for Learning Guidelines Standards for Educational and Psychological Testing (American Educational

Research Association, American Psychological Association, and the National Council of Educational Measurement in Education, 2014)

Enhanced Specifications Style Guide External Review Sheet Student Math Reference Sheet Standards for ELA and Math ELA and Math scoring rubrics

149

Step 4: Four Regional Item Writing Workshops

Owner: Contractor

Participants: Contractor – facilitates and coaches Educators – draft items based on specified assignments from Contractor

Goal/Outcome: Conduct four regional item writing workshops to generate item drafts that meet expectations set forth in item criteria documents and UD guidelines.

Process:1. Contractor will train on item writing principles.2. Practice writing sample set. 3. Contractor will train on item authoring system.4. Write items under the guidance of facilitator/coach.

QA within Process: Coaching

QC: Item Editing (Step 5)

Documentation: Item Writer Assignment Sheet (ELA & Math) Reference Sheet (Math) Passages (ELA) Enhanced Specs (ELA & Math) Standards (ELA & Math) Rubrics

150

Step 5: Item Editing

Owner: Contractor

Participants: Contractor

Goal/Outcome: Fully edited, NYSED-ready items.



QC: OSA Item Pre-Review (Step 6)

Metrics: % of items surviving editing process

151

Step 6: OSA Item Pre-Review

Owner: OSA/Contractor

Participants: OSA Content Team – review items and suggest revisions. Supervisor – monitor OSA Content Team progress, record keeping, manage

timeline. Receive delivery confirmations from Contractor. Contractor – provide training and guidance on the tools used for recording

feedback. Deliver items for review.

Goal/Outcome: To make judgements and suggest revisions based on guiding documents to

provide feedback to Contractor.

Process: 1. Contractor provides OSA Content Team with NYSED-ready items and notifies

Supervisor of delivery.2. Supervisor informs OSA Content Team of delivery.3. OSA Content Team reviews items, verifying;

i. Item conforms to Specs.ii. Item is aligned to standard.

4. Verify; the key for MC items. for CR items-

i. Will the wording of the item elicit the desired response?ii. Does the item allow for a variety of ways for a student to respond?iii. Is the sample response/Rationale provided appropriate?iv. Can students answer similarly on paper and computer?

5. OSA Content Team makes judgement, accept/revise/reject. If the judgment is to revise, they should provide suggestions for revision. If rejected, OSA Content Team provides rationale.

6. Supervisor monitors process and provides feedback to OSA Content Team. Supervisor provides updates to Contractor as required.

QA within Process:

OSA Pre-Review Item Criteria Collaboration with the OSA Content Team Training on criteria prior to review

QC:

152

Internal Item Review (Step 7)

Documentation: Recorded comments in the CMS (export Summary from CMS)

Metrics: Tracking % accept/revise/reject Tracking % agreement of alignment Tracking % agreement on key

153

Step 7: Internal Item Review Meeting


Participants/Roles: Supervisor – observe and monitor progress, assist as needed, facilitate the process and

revise items, record results OSA Content Team – discuss pre-review comments and make suggestions

Goal/Outcome: To review, edit, and approve: test items, key, meta-data, and sample response/rationale.

Process: 1. Training as needed on use of tools/software for recording feedback2. Round table review of each question as follows;

o Display Passage and Questiono Read comments from pre-review and discuss o Does passage support question? Can it be fixed? (if not, reject)o Does the item measure the intended standard? (if not, fix)o Is the item worded clearly? (if not, fix)o Is there one and only one correct answer?o Are meta-data correct?o Is sample response/rationale correct?

3. Discuss and revise as appropriate QA within Process:

Results of item pre-review Meta-data check

QC: Educator Item Review (Step 10)

Documentation: Final judgement, as recorded by facilitator (export summary to Supervisor)

Metrics: Tracking % accept/revise/reject of items

154

Step 8: First Copy Edit/Content QC

Owner: Contractor

Participants: Contractor

Goal/Outcome: To Copy (not content) Edit; Finalize draft items for Content QC Copy editing:

o Formattingo Spellingo Paragraph # checkso Graphics

Check against style guide


QA within Process: Verify against Style Guide

QC: Contractor Defined

155

Step 9: External Quality Control

Owner: Contractor

Participants/Roles: Contractor – manages the process. External Third-Party Contractor – reviews and provides comments.

Goal/Outcome: To verify that items going to Educator Review meet the following criteria: Item aligns to the standards. Item is clearly worded. Item has one and only one correct answer. Item is Mathematically correct. Review passages- are passages appropriate for grade level?

Process: 1. External Third-Party Contractor reviews each item, using the criteria in goals section.

If necessary, suggest revisions and record comments. 2. External Third-Party Contractor attends Educator Review as note-takers.3. External Third-Party Contractor participates in Reconciliation.

QA within Process: Checking items against criteria listed in the goals section

QC: Educator Item Review (Step 10)

Documentation: Comments/Summary Report (export to Supervisor).

Metrics: Tracking % of items flagged for each criterion in goals section Tracking % of items flagged for edit

156

Step 10: Educator Item Review Meeting

Owner: Contractor

Participants/Roles: Educator Panelists – located in 12 rooms (7 educators per room)- reviews items and

provides comments. Contractor – facilitates External Third-Party Contractor – note-takers, record themes OSA Content Staff – in each room to observe and address questions from educators.

Goal/Outcome: To gain actionable feedback on all items from educators in the field. Educators provide comment on the following:

Passage:o The passage readability is appropriate for grade level.o The passage is appropriate and fair for grade level. o The passage can be read in the time allotted for grade level.o The passage contains attributes that will support items aligned to certain

standards at grade level. Items:

o Item aligns to the standards.o Item is clearly worded.o Item has one and only one correct answer.o Item is Mathematically correct.o Review passages- are passages appropriate for grade level?

Process: 1. Training – Test development overview, item development process, applying the

criteria shown in goals section. Provide any necessary training on tools being used for recording feedback.

2. Read and answer the questions: educators work through 15-20 items, independently.3. Educators verify key, alignment, etc.4. Educators make comments, render judgment on each item.5. Group discusses and comes to a common resolution. If a resolution is unattainable,

take to Reconciliation.

QA within Process: Verify against criteria in the goals section Group discussion by the educators Draw upon educators’ experience and expertise

157

External Third-Party Contractor independent review and participationQC:

Reconciliation

Documentation: Educator comments (export to Supervisor) Group judgement recorded by facilitator in CMS (export to Supervisor) External Third-Party Contractor notes in CMS

Metrics: Tracking % agreement with items on each criterion in the goals section Tracking % items revised/rejected/accepted

158

Step 11: Reconciliation


Participants/Roles: Supervisor – content guidance and decision making Bureau Chief – decision making, as needed Contractor – facilitator, decision making External Third-Party Contractor – content guidance

Goal/Outcome: To verify that the OSA notes match the Contractor notes, and make any final polishing edits. Render final judgment on any items not brought to resolution at Educator Review.

Process: 1. Review Contractor and OSA Content Team notes and final judgments for each item.

QA within Process: Discussion at Educator Review Reconciliation of notes

QC: Field Testing

Documentation: Comments and reports (export to Supervisor)

159


Time Frame: Fall – Late Fall

Distinct Steps:

12.Formal Copy Edit13.Stand-Alone Field Test Form Construction14.Field Test Form NYSED-Review

160

Step 12: Formal Copy Edit

Owner: OSA

Participants: OSA Editor OSA Supervisor/Bureau Chief

Goal/Outcome: Full Copy Edit of field test item bank.

Process: Editor will review grammar/style, make notes/revisions in the CMS. Editor/Bureau Chief/Supervisor then reviews notes and decides which edits to

accept. OSA sends edits back to Contractor. Contractor implements edits. OSA verifies appropriate edits made.

QA within Process: Review findings discussed by Supervisor/Bureau Chief

QC: Verification by OSA

Documentation: Recorded copy editing

FROM THIS POINT ON, THERE SHOULD BE NO FURTHER ITEM EDITING

161

Step 13: Stand-Alone Field Test Form Construction

Owner: Contractor

Participants/Roles: Contractor Content Team Contractor Psychometric Team OSA Content Team

Goal/Outcome: Create proposed field test forms that:

oAllow for item bank replenishment

Process: 1. Item Bank Inventory2. Consult Blueprint3. Develop sampling plan4. Determine Number of Forms and receive OSA approval5. Contractor Item Selection

Verify that all Math CR items can be answered by students testing on both paper and computer

6. OSA Content Team approval of item selection Verify that all Math CR items can be answered by students testing on both

paper and computer7. Contractor then Builds Forms


QC: OSA Review

Documentation: Test Form Planners

162

Step 14: Field Test Form NYSED-Review

Owner: OSA

Participants/Roles: OSA Content Team – reviews forms Supervisor – facilitates, manages process

Goal/Outcome: Review forms for

o item placement problems/conflictso time concernso REVIEW ONLY: NO ITEM EDITING

Process: 1. OSA Content Team review forms.2. OSA Content Team identifies and records potential problems.

o Contractor considers; confirms or disputeso Supervisor resolves

QA within Process: OSA Content Team/Contractor Collaboration

QC: N/A

163

Rangefinding

Time Frame: Mid-July – Late August

Distinct Steps: 15.Rangefinding Meeting16.Rangefinding Reconciliation17.SAFT CR Scoring

Guiding Documents: Rangefinding Facilitator Log

Step 15: Rangefinding Meeting

Owner: Contractor

Participants/Roles: Educators: make decisions about scores and rubric application Contractor: facilitates, training, record-keeping Supervisor: Monitoring and providing clarification/interpretation OSA Content Team: Note-takers, participate in discussions as possible,

provide clarification on policy questions

Goal/Outcome: Educators rate field test responses without OSA/Contractor influence. Educators provide comments to be used for future operational scoring

materials development.

Process: 1. Contractor provides general session introduction to Range Finding and to holistic

rubric and scoring.2. Educators move to ELA and Math breakout rooms:

Contractor provides additional training in breakout rooms using ELA and Math Grounding Sets (sets that were used to score the previous operational administration) and Training Sets (one per each item type)

Contractor provides training for reviewing responses and recording scores and comments

3. Rating: Use key words to characterize each 0,1,2,3,4 score (key words from rubric)

164

Facilitator encourages educators to highlight key adjectives from each score point on the rubrics provided.

4. Panel discusses individual ratings as a group and tries to come to consensus5. Facilitator records final ratings and comments6. Facilitator/OSA Content Team informs Supervisor of problematic trends and

Supervisor provides corrective action7. Supervisor keeps track of participant involvement/focus/capacity

QA within Process: Training Exemplars (Grounding Sets and Practice Sets) Contractor/OSA daily debrief sessions

QC: Reconciliation

Documentation: Educator Participant Logs Facilitator Logs

Metrics: % of ratings agreement among panelists for final scores (majority vs.

consensus)

165

Step 16: Rangefinding Reconciliation

Owner: Contractor

Participants/Roles: Contractor Content Leads – organize OSA Content Team – review/adjust/approve panels’ final scores OSA Supervisor – monitors Content Teams’ progress

Goal/Outcome: Approval of final panel ratings/scores

Process: 1. OSA Supervisor and OSA Content Team review/adjust/approve panels’ final

scores in Facilitators’ Logs

QA within Process: Contractor/OSA daily debriefs

Documentation: Facilitators’ Logs with final OSA approvals/decisions

Metrics: % of changed ratings

166

Step 17: SAFT CR Scoring

Owner: Contractor

Participants/Roles: Contractor Scoring Directors – Train Team Leaders and Scorers, monitor scoring,

conduct read-behinds Contractor Team Leaders – monitor scoring and conduct read-behinds Contractor Scorers – Conduct scoring for SAFT CR items

Goal/Outcome: Conduct scoring for SAFT CR items accurately and consistently

Process: 1. Contractor Scoring Directors train the Team Leaders.2. Contractor Scoring Directors train Scorers.3. Contractor Scoring Directors and Team Leaders monitor CR scoring.4. Contractor Scoring Directors address concerns, as needed, regarding Scorer

accuracy and consistency.QA within Process:

Monitoring Reports for computer-based scoring platformo System-generated reports on scoring metrics for monitoring and internal use.

Reports can be filtered using different parameters to monitor scorers, such as by teams, individuals, and individual items. The Scoring Director uses these reports to monitor each team and the group as a whole to ensure consistent scoring across all teams. Team Leaders use these reports to closely monitor the scorers on their team, both in terms of productivity and reliability.

Read-behindso Random read behinds are done throughout the day for all scorers,

regardless of whether an issue or concern has been noted. Random read behinds are part of the ongoing monitoring process.

o Prescribed read behinds represent an increased number of read behinds due to some issue that may have come to the attention of the scoring leader through the computer-based scoring platform’s system monitoring report, a comment or question from the scorer, or during a random read behind.

One-on-One Discussions o A one-on-one discussion may be held with a scorer in the context of a score

changed in a read behind. A discussion may also take place to address questions or issues brought up by the scorer, or as a training tool using specific exemplar responses from scoring to point out problems or scoring tendencies a scorer may exhibit. *Note: If one-on-one discussions are required and performance does not improve, the scorer will be removed from scoring that item based on qualitative and quantitative data.

167

Dismissal from Scoring the Project o A scorer will be dismissed if retraining does not elicit satisfactory results and it

is determined that a scorer cannot accurately score student responses. *Note: Should a scorer be removed from scoring an item or dismissed from scoring the project, their scored responses would be reviewed and potentially rescored based on the monitoring process.

QC: SAFT Equating Operational Form Construction

Documentation: Training Materials – Grounding Set specific to each grade and content area (which

includes a Training Set and Practice Set for a short response and extended response), and a Training Set developed from responses scored at Rangefinding.

Computer-based scoring platform Monitoring Reports

Metrics: % of changed ratings

168

Forms Construction/Forms Finalization

ELA Time Frame: Fall – Late Fall

Distinct Steps:

18.First Draw (Contractor)19.Form Pre-Review (OSA) 20.Content Review/Risk Assessment21.Forms Construction Meeting22.Final Eyes/Reconciliation23.Form Copy Edit 124.Form Copy Edit 225.Final Executive Review

Guiding Documents:

Specs, Actual Blueprint (how they met the specs) and Stats Projected Raw Cuts Judgement Sheet for committee members Forms Construction notes External Third-Party Contractor notes (pre and post) Final Eyes educator comments

Run 3 grades on each day: (Day 1 – Math 3, 5, 7 and ELA 3, 5, 7; Day 2 – Math 4, 6, 8 and ELA 4, 6, 8). Reconciliation 3 days (Math 5, 7 and ELA 3, 5; Math 6, 8 and ELA 4, 6; Math 3, 4 and ELA 7, 8)

169

Step 18: First Draw

Owner: Contractor

Participants/Roles: Contractor Content Team Contractor Psychometric Team

Goal/Outcome: Create proposed forms that

o meet content specifications (blueprint)o meet technical specifications (statistics)

Provide OSA with;o initial Operational Blueprint.o initial Projected Stats.

Process: Contractor Defined Verify that all Math CR items can be answered in the TDS


QC: Form Pre-Review (step 19)

Documentation: Specs, Test Design Blueprint (how they met the specs) and stats. Projected Raw Cuts

170

Step 19: Form Pre-Review

Owner: OSA

Participants/Roles: OSA Content Team – review first draw form and recommend changes OSA Psychometrics Team – review stats

Goal/Outcome: Review first draw to check the following;o Does question have one and only one correct answer?o Does question clue within form?o Does form meet content and psychometric specifications?o Are passages and items fair and appropriate for students in grades 3 – 8?o Does form adequately measure each concept?

i. No clear themes across passages ii. Diverse voice across passagesiii. Math questions display a variety of contexts, formats, and number

choices Review documentation showingo Blueprint is meto Statistical targets are met

Step 20: Content Review/Risk Assessment

Owner: Contractor

Participants/Roles: Contractor: facilitates External Third-Party Contractor – review

Goal/Outcome: Ensure that first draw form is defensible and meets all criteria:

oDoes question have one and only one correct answer?oDoes question clue within form?o Does form meet content and psychometric specifications?o Are passages and items fair and appropriate for students in

grades 3 – 8?o Does form adequately measure each concept?

o No clear themes across passages

171

o Diverse voice across passageso Math questions display a variety of, contexts, formats and

number choices

Process: 1. Contractor provides first draw forms and criteria to External Third-Party

Contractor.2. External Third-Party Contractor reviews forms against criteria and

provides feedback.3. External Third-Party Contractor attends Forms Construction Meeting as

note-taker.

QA within Process: Review criteria

QC: Forms Construction

Documentation: Recorded notes (export to Supervisor) Judgement Spreadsheet

Metrics: % of items flagged % of flagged items acted upon

172

Step 21: Forms Construction Meeting

Owner: Contractor

Participants: Contractor Content Team – facilitates Contractor Psychometrician – advises on impact of proposed changes. OSA Content Team – observes and takes notes for reconciliation. OSA Research/Psychometrics – review any requested changes. Educators – Review forms and select alternative Passage/Item if necessary.

Goal/Outcome: Review proposed form to ensure that:

ii. Does question have one and only one correct answer?iii. Does question clue within form?iv. Are passages and items fair and appropriate for students in grades 3 – 8?v. Does form adequately measure each concept?

o No clear themes across passages o Diverse voice across passageso Math questions display a variety of contexts, formats and

number choices

Process:

1. Contractor supplies form A for each grade.2. Contractor displays passages and questions for ELA, and questions for Math.3. Educators review without item keys, verifying the criteria in the goals section.4. External Third-Party Contractor takes notes. 5. OSA Content Team/Supervisor/Bureau Chief discuss resulting forms with the

Contractor.

QA within Process: OSA and Contractor Psychometric Advisement

QC: Final Eyes

Documentation: Educator Judgement Forms Forms Construction Notes Specs/Stats

Metrics: % of items swapped

173

Step 22: Final Eyes Meeting/Reconciliation

Owner: Contractor

Participants/Roles: Educators –: take test, record comments, and participate in discussions. OSA Content Team – be present and take notes. Supervisor – reconcile via meeting afterwards with Contractor. Director – reconcile Contractors – facilitate and record-keep

Goal/Outcome: Have educators’ QC print or CBT forms by completing the test and evaluating

that there is one and only one defensible key.

Process: Contractor provides overview of process;

Role of Final Eyes in test development process Charge to educators Use of Workshop Tool

Educators take one grade level test, all sessions. For ELA- provide outline of a full credit CR For Math- completely work the problem, with use of Workshop Tool

(containing equation editor) Educators will be split into two groups within the room, some will review

computer and some paper (optional: Groups review both paper and CBT, if time permits)

While taking test, they record Key Any issues/comments

Discuss notes after completion of each session. Contractor records discussion via note taking.

Contractor aggregates results Reconciliation

Supervisor, Senior Management and Contractor review flagged items;i. Items that are decided to be double keys are removed/edited ii. Grammatical issues are fixed

QA within Process: Key Check

QC:

174

Final Executive Review

Documentation: Form Judgement Sheet

Metrics: % of items with key flags % of items flagged for discussion % of items altered for content issues % of items edited for mechanical issues, as a result of review

175

Step 23: Form Copy Edit 1

Owner: OSA

Participants/Roles: Editor – edit, check, and reconcile Supervisor/Bureau Chief – reconcile/approve

Goal/Outcome: Full Review of Operational Form Language and Formatting to prepare for

approval.

Process: 1. Two Editors copy edit independently.2. Editor compiles and aggregates.3. Supervisor and Bureau Chief review with editor to decide which edits to make.4. Send edits back to Contractor.5. Contractor enacts.

QA within Process: Copy Edit 2 Double review

QC: Supervisor and Bureau Chief Sign-off

Step 24: Form Copy Edit 2

Owner: OSA

Participants/Roles: Editor – edit, check, and reconcile Supervisor/Bureau Chief – reconcile/approve

Goal/Outcome: Full Review of Language and Formatting to prepare for approval

Process: 1. Two Editors copy edit independently.2. Editor compiles and aggregates.3. Supervisor and Bureau Chief review with Editor.

176

QA within Process: Check to see if all edits requested were made Double review Review findings discussed by Senior Management

QC: Final Executive Review

Step 25: Final Executive Review

Owner: OSA

Participants/Roles: Senior Management – reviews and approves forms. Bureau Chief, Supervisor – advises Senior Management on review results.

Goal/Outcome: Final Approval of Form

Process: 1. Senior Management reviews forms2. Senior Management discusses review findings with Bureau Chief & Supervisor

QA within Process: N/A

QC: N/A

Documentation: Final Approval

177

Scoring Ancillaries

Time Frame: Fall-Winter

Distinct Steps:

26.Proposed Operational Scoring Materials Sets 27.OSA Review of Scoring Materials Sets28.Scoring Copy Edit and Senior Management Approval

Guiding Documents:

Scoring Materials Sets Annotation Template Scoring Review Criteria Copy Edit Review Criteria

Step 26: Proposed Operational Scoring Materials Sets

Owner: Contractor

Participants/Roles: Contractor Scoring Services and Publishing Teams create proposed Training Sets, Practice Sets, and Consistency Assurance Sets

Goal/Outcome:Provide proposed Training Sets, Practice Sets, and Consistency Assurance Sets for OSA review and approval.Process: Contractor selects and annotates student responses identified during Rangefinding and builds draft Scoring Materials Sets.

QA within Process: Contractor Publishing Team review of Scoring Materials Sets prior to OSA review.

QC: OSA review of Scoring Materials Sets

Documentation: Scoring Materials Sets

Step 27: OSA Review of Scoring Materials Sets

Owner: OSA

178

Participants/Roles: OSA Content Team – reviews Supervisor – manages documentation and approved recommended edits

Goal/Outcome: Review and approve proposed student responses, placement of student responses,

and assigned scores. Review annotations for accuracy in relation to the student response and rubric. Ensure consistency of annotations across items, grades, and sets.

Process: 1. Contractor provides proposed Scoring Materials Sets with annotations.2. OSA Content Team reviews Scoring Materials Sets using the following criteria;

i. Annotation template1. Rubric language which supports the score point assigned 2. Uses examples from student response to support the

rubric requirementsii. Ensure a range of student performance for each score point is represented in

the Training Sets.3. For annotations not fitting template or criteria;

i. Content Team provide edits in the format agreed upon by OSA and the Contractor.

4. Supervisor reviews and approves all recommended edits with an emphasis on replacing student responses in lieu of changing score points.

QA within Process: Supervisor review of changes Criteria/Template

QC: Contractor reviews and implements OSA’s edits.

Documentation: Criteria/Template Revised Scoring Materials based on OSA’s edits

179

Step 28: Scoring Copy Edit and Senior Management Approval

Owner: OSA

Participants/Roles: OSA Editor – edit Supervisor – reconcile/approve Senior Management Sign-off

Goal/Outcome: Full Review of Language and Formatting to prepare for approval

Process: 1. Editor performs copy edit and records recommended edits.2. Supervisor/Senior Management review with editor to decide which edits to

make.3. OSA send edits back to Contractor in the format agreed upon by OSA and the

Contractor.4. Contractor enacts 5. Repeat process6. Senior Management manager sign-off

QA within Process: Check to see if all edits requested were made Review findings discussed by Senior Management

QC: Senior Management Sign-off

180

Item Release

Time Frame: Late Spring (Deadline: June 1st)

Distinct Steps:

29. Proposed Release Items

Guiding Documents:

Release Item Criteria Release Item Template Editing Checklist

181

Step 29: Preparation of Selected Items for Public Release


Participants/Roles: Bureau Chief – selects items for release. Supervisor – assists Bureau Chief with selection process. OSA Editor – performs redactions. Contractor – Provides the selected items to OSA in text format suitable for

posting on the web by NYSED on the NYSED web site.

Goal/Outcome: Release Items and associated data to public.

Process: 1. Contractor provides test items in an agreed upon text format that allows for

manipulation/reproduction once posted on the website.2. Bureau Chief & Supervisor select items for release.3. Contractor provides spreadsheet with data (Standards alignment, keys, and P

values) and Text Complexity Matrix for all test items.4. OSA Editor performs redactions/removes any items not to be released 5. Supervisor verifies redacted materials prerelease.6. Bureau Chief shares data with RICs.7. Released items and scoring materials posted on EngageNY/NYSED website.**Released items must be provided in a text format that enables schools to manipulate them for use in the classroom once they are posted on the website; released items cannot be provided to NYSED as images.

QA within Process: Supervisor Verification

QC: Contractor review of items to be released and all associated data

Documentation: Released Items, scoring materials, and associated data.

182

Educator Committee Selection

Owner: OSA

Participants/Roles: OSA – Identify educators to participate. Supervisor – facilitates Contractor – requests, schedules meetings, makes contact

Goal/Outcome: Recruit and select educators who;

o have appropriate experience to conduct specific tasks.o are nominated or otherwise vetted.o represent a diverse cross section of the NYS Educators in terms of

geography demographics, and student populations.

Process: 1. Consult Application Database housing educator information obtained from

ongoing recruitment.2. Multiple candidates (i.e., tiers) are identified for each seat on the

committee based on diversity criteria above. Provided to Contractor with explicit instructions on how to contact

candidates3. Contractor contacts candidates in order by tier (e.g., A1, A2…) and

informs NYSED of responses. Contractor consults with OSA for placement of candidates in a

different grade and/or subject from initial selection,4. OSA identifies and sources additional candidates as needed.5. Contractor provides post-meeting feedback on educators to OSA.

QA within Process: Recommendation Process/Application Process Contractor-provided feedback

Documentation: Application database

183

Attachment E: Educator Committees for ELA and Math ELA

Date Committee

Number of Days for

Each Teacher

Participant

Number of Teachers

Honoraria Paid by Contractor to Teacher Participants

Travel Expense Reimbursement

Paid by Contractor to

Teacher Participants

Purpose Product

November 2020, 2021, 2022, 2023, 2024

Four Regional Item Writer Workshops

3 250 - 350 $200* per day No, participants in these events are responsible for their own travel expenses.

NYS certified teachers will draft test items at workshops facilitated by contractor

Draft Field Test Items

Late August 2020, 2021, 2022, 2023, 2024

Review of ELA Field Test Passages

3–5 36(approx. 6 per grade level)

$200 per day Yes Review passages Review passages for Field Tests

July 2021, 2022, 2023, 2024, 2025

ELA Item Review

3 36(approx. 6 per grade per grade)

$200 per day Yes Review and edit draft items

Field Test Bank

July 2021, 2022, 2023, 2024, 2025

ELA RangefindingGrades 3-5Grades 6-8

10 18 (approx. 3 per grade)

$200 per day Yes Choose score papers

Scoring Materials

Late August 2020, 2021, 2022, 2021, 2024

ELA Forms Construction

1 36 (6 per grade)

$200 Yes To select the passages and items that will make up the operational tests

Draft Operational Test Forms

October 2020, 2021,

ELA Final Eyes Grades 3–5

11

1530 (15 per

$200 per day Yes Final review of operational tests

Print-ready Operational Tests

184

Date Committee

Number of Days for

Each Teacher

Participant

Number of Teachers





Purpose Product

2022, 2023, 2024

Grades 6–8 grade band, 3-5 and 6-8)

March 2021 ELA Test Performance Level Descriptions Review

1 24 (8 per grade band)

$200 per day Yes Develop/revise performance level descriptions for each of the four performance levels

Performance Level Descriptions

June 2021 ELA Test Standard Setting Grades 3-4, Grades 5-6, Grades 7-8

3 36(approx. 12 per grade band)

$200 per day Yes Determine cut scores

Cut Scores

*Teacher Participants in the Item Writer Workshops receive an additional honorarium of $125 for engaging in prerequisite training developed by the contractor in item writing fundamentals, done prior to attending the workshop. NYSED estimates that this activity will take approximately 5 hours.

185

Math

Date Committee

Number of Days for

Each Teacher

Participant

Number of Teachers





Purpose Product

November 2020, 2021, 2022, 2023, 2024

Four Regional Item Writer Workshops

3 250 – 350 $200* per day No, participants in these workshops are responsible for their own travel expenses

NYS certified teachers will draft test items at workshops facilitated by contractor

Draft Field Test Items

July 2021, 2022, 2023, 2024, 2025

Math Item Review

3 36(approx. 6 per grade per grade)

$200 per day Yes Review and edit draft items

Field Test Bank

July 2021, 2022, 2023, 2024, 2025

Math RangefindingGrades 3-5Grades 6-8

10 18 (approx. 3 per grade)

$200 per day Yes Choose score papers

Scoring Materials

Late August 2020, 2021, 2022, 2021, 2024

Math Forms Construction

1 36 (6 per grade)

$200 Yes To select the passages and items that will make up the operational tests

Draft Operational Test Forms

October 2020, 2021, 2022, 2023, 2024

Math Final Eyes Grades 3–5 Grades 6–8

11

1530 (15 per grade band, 3-5 and 6-8)

$200 per day Yes Final review of operational tests

Print-ready Operational Tests

March 2021 Math Performance

1 24 (8 per grade band)

$200 Yes Develop/revise performance

Performance Level Descriptions

186

Date Committee

Number of Days for

Each Teacher

Participant

Number of Teachers





Purpose Product

Level Descriptions Grades 3-4, 5-6, 7-8

level descriptions for each of the four performance levels

June 2021 Math Standard Setting Grades 3-4, Grades 5-6, Grades 7-8

3 36(approx. 12 per grade band)

$200 per day Yes Determine cut scores

Cut Scores

*Teacher Participants in the Item Writer Workshops receive an additional honorarium of $125 for engaging in prerequisite training developed by the contractor in item writing fundamentals, done prior to attending the workshop. NYSED estimates that this activity will take approximately 5 hours.

187

Attachment F: External Staffing Requirements for Test Development Process

Test Development Process Step Number

Test Development

EventExternal Staffing Requirement

2 Content Review

NYSED approved ELA consultants – threeNYSED approved math consultants - three

6 Item Review

NYSED approved ELA consultants – threeNYSED approved math consultants - threeNYS ELA educators - sixNYS Math educators - six

10External Quality Control

NYSED approved ELA consultants – one to twoNYSED approved math consultants – one to two

20Content

Review/Risk Assessment

NYSED approved ELA consultants – twoNYSED approved math consultants - two

32Approval of Release Set

Choices

NYSED approved ELA consultants – three

NYSED approved math consultants - three

33 Content Review

NYSED approved ELA consultants – three

NYSED approved math consultants - three

188

Attachment G: Grades 3-8 ELA Tests Guidelines for Passages

Selecting passages of appropriate length and complexity for use in assessments is integral in support of standards that emphasize the development of skills for comprehending and analyzing literary and informational texts. At the same time, selected passages must be appropriate for large-scale assessment format.Below are some general guidelines that the contractor will be expected to follow when identifying passages for the Grades 3-8 ELA Tests. The comprehensive NYS requirements for passages will be shared with the selected contractor.

1. Text Complexity and Grade Level VocabularySelecting passages of appropriate complexity is essential for assessing the comprehension skills at each developmental level. Passages too basic for a given level will not possess the necessary vocabulary, syntax, structures, and content development to assess the grade-level skills. Passages too complex for a given grade level will contain characteristics that interfere with the assessment of the grade-level skills. Text complexity is generally determined using quantitative and qualitative measures.

2. Specified Word Counts and ReadabilityAll passages must adhere to NYSED’s word count and readability scores for each grade level.

3. Grade-Level Characters and TopicsCharacters and topics of passages must be appropriate for the grade-level from both qualitative and quantitative guidelines.

4. VariationIt is important that passages represent a range of texts that students should be reading at each grade level. In considering this range, one important consideration is a need to balance texts written by authors with diverse backgrounds, including a balance of authors by gender and ethnicity. In addition, passages selected should appeal to a wide range of student audiences. Passage variation allows students to demonstrate their ability to read and comprehend a range of complex texts. Passage selections must include both literary and informational texts. When using excerpts for literary passages, careful consideration to maintaining the unity and sense of story development is essential.

5. Bias and sensitivity guidelinesNYSED’s goal is to have high-quality passages that are interesting and thought provoking, while avoiding troubling content. NYSED has strict guidelines with regard to content that is biased or insensitive. Language must be familiar to most students regardless of socioeconomic status, region, language proficiency, cultural background, or gender. Content must also not give students of a particular group, background, or region an advantage or a disadvantage.

6. Testing the NYS StandardsA thorough analysis of a potential text’s purposes, meanings, structure and language lays the foundation for determining its suitability for assessing specific standards. To ensure accuracy and efficacy of assessment, passages should be chosen for assessing specific skills. Passage finders should have the NYS Learning Standards clearly in mind when selecting texts. Preliminary analysis will chart the specific standards the passage will support. Passage finders should have a strong understanding of the text characteristics necessary for assessing each

189

standard. They also should have a sense of how standards can be bundled together to craft a comprehensive set of items for particular passage types.

7. Permissions and Sources The contractor must adhere to all NYSED guidelines for obtaining permissions for the use of sources (see the section of this RFP “Requests for Permissions”) and must follow guidelines for citations.

NY 3-8 ELA Passage ReviewPassage Criteria Checklist

Is the passage reading level appropriate for the grade?Is the passage appropriate and fair for the grade?Can the passage be read in the time allotted? (5-7 min)

Does the passage contain attributes that will support items aligned to the required standards?Are the passage graphics accurate and appropriate?

190

Attachment H: Assessment Specialist Content Verification Checklist

Content Area: Grade: Assessment Specialist:

Check the boxes for which you agree with the provided statement. For unchecked boxes, provide a brief explanation in the comments section using the checklist number and letter (e.g., “1.c” if the reading load of the passages is inappropriate).

General Test Design Considerations

1. Test Blueprint

☐ a. The test meets the provided test blueprint. For Math, the cluster range supersedes the standard target.

☐ b. ELA Only : The number of passages follows the requirements in the test blueprint.

☐ c. ELA Only : The reading load of the passages is appropriate.

2. Sequence of the test

☐ a. Easy items or difficult items are not clustered together but are mixed throughout the books.

☐ b. Items associated with the same stimulus are sequenced together on the test.

3. Key

☐ a. Keys of A–D (i.e., 1–4) are similarly represented on the entire test.

☐ b. The same key does not appear more than three times in a row.

Content Considerations

4. Item-Level Content Considerations

☐ a. Identified key is correct.

☐ b. No other correct response options exist.

☐ c. No duplicate items appear on the form.

☐ d. No clueing/cuing among items.

☐ e. Items are free from typographical, spelling and punctuation, and grammatical errors.

☐ f. Items meet style specifications (e.g., bolding and italics).

☐ g. No released or Do Not Use (DNU) items have been selected on the forms.

☐ h. The correct versions of items are used.

☐ i. The item may be administered on both paper and by computer.

☐ j. Mathematics Only : Only grade-level or post-standards items from the immediate lower grade are selected for operational testing.

☒ k. ELA Only : The anchor set contains one literary and one informational passage.

5. General Test-Level Content Considerations

191

☐ a. ELA Only : There is a good mix of urban and suburban passages, in addition to a balanced representation of gender and ethnicities.

☐ b. ELA Only : The passages included on the test are solid reads, have appropriate levels of text complexity, and are free of bias.

☐ c. To the extent possible, multiple perspectives of the standards are addressed by the selected items.

Comments:

______________________________________________________________________

Assessment Specialist Sign-Off: Date:

192

Item Review CriteriaNY 3-8 ELA Educator Review

Item Review Criteria Checklist

Is the passage reading level appropriate for the grade?Is the passage appropriate and fair for the grade?Are the passage graphics accurate and appropriate?Is the question aligned to the intended standard?Is there one and only one key?Are the distractors plausible?Is the question clearly worded?Is the item free of bias and sensitivity concerns?

193

NY 3-8 Math Educator ReviewItem Review Criteria Checklist

Is the question aligned to the intended standard?Are the mathematics correct?Is there one and only one key?Are the distractors plausible?Is the question clearly worded?Is the item free of bias and sensitivity concerns?Are the item graphics accurate and appropriate?

194

Attachment I : Specifications for Contractor/Subcontractor Transcribing Braille

1. The contractor/subcontractor must be certified by the National Library Service for the Blind and Physically Handicapped, Library of Congress (NLS), in literary braille transcribing, under the rules of Unified English Braille (UEB).

2. The contractor is required to provide UEB for literary braille transcriptions and mathematic transcriptions under the rules of Nemeth Code for Mathematics and Scientific Notation, adhering to all current braille codes, guidelines, and standards provided by the Braille Authority of North America (BANA) for all countries using English Braille.

3. Original transcriptions must be checked thoroughly and proofread by a reviewer, and discrepancies resolved before submission to NYSED.

4. The contractor must provide a method for producing tactile graphics that are well suited for the sense of touch for students who use Braille editions.

5. Tests and related materials must be transcribed into Braille text and raised-line graphics for illustrations in their entirety, including all labeled items on tactile graphics (maps, graphs, diagrams, charts, and tables).

6. If the contractor/subcontractor is going to be using interpoint (Braille on both sides of the paper), the contractor/subcontractor must use paper of a sufficient quality that will prevent the Braille from pressing through and allow for maximum embossing quality.

7. If a tactile graphic (maps, graphs, diagrams, charts, and tables) contains information that does not fit within the constraints of a standard Braille page (11 inches by 11.5 inches), then foldouts must be used to create these graphics.

8. If a tactile graphic has information that cannot fit in the constraints of the BANA codes for tactile graphics due to the amount of space Braille takes up, the contractor/subcontractor must obtain NYSED’s prior approval to change, modify, or eliminate information contained in the printed copy.

9. The contractor must adhere to the Security Guidelines for the New York State Assessment Program (Attachment M) in working with its Braille subcontractor.

10.All materials will be transcribed into grade II (contracted) Braille, unless otherwise specified by NYSED.

11.Since the items on the tests are often graphics based, some adaptations and adjustments to various items may be required. This tends to occur more often with Mathematics tests than with English Language Arts tests. Any suggestions for adapting these items to make them more accessible and able to be transcribed into Braille more efficiently should be discussed with NYSED prior to beginning transcription.

12.The exclusion and/or prorating of items from the Braille edition of these tests are prohibited.

195

Attachment J : Guidelines for Large-Type Print Editions

1. The typeface, size and spacing should be standardized on all tests, including print contained in charts, graphs, maps, tables and other graphics and visual stimuli, using a serif-free typeface.

2. Base text size will be 18 pt., with proportionally headings and subheadings that are proportionately larger.

3. A sans serif font, such as Helvetica, should be used (New Century Schoolbook, New Caledonia).

4. Minimal use of parentheses, italics, underlining, and boldface in text.

5. Spacing between lines of print is at least 1.25 spaces.

6. Paragraphs are block-style with no indentation. The left margin should be justified, and the right-hand margin should not be justified (“flush left”/” ragged right”).

7. Pages should be single-column, with one-inch margins.

8. Words should not be allowed to break across lines.

9. Type should be solid black with no shades of gray printed on white, ivory, cream or yellow paper with a dull finish so as not to promote glare.

10. Avoid text used over a background design or other graphical material.

11. Graphics should not only be enlarged but maintain strong contrast and clarity. Graphics should be modified to eliminate or minimize gray shades and fills, increase contrast, and enlarge type.

12. Graphics for Math items that require measurement with a ruler cannot be enlarged.

13. High-quality black line art should be used instead of gray-scale or shaded drawings.

14. Page layouts should keep any referenced art together on a spread with the related questions (to avoid flipping).

15. Artist credits and other copyright information that typically appear right below graphics, illustrations and other visual stimuli shall be moved to a clearly delineated section at the bottom of the page or to an appendix whenever it is determined that such material is not relevant to the student’s understanding of the item itself.

16. If staple binding is utilized, staples shall be placed along the spine side (instead of at the upper left corner). This will make it easier for students to manipulate the pages and to maintain continuity.

196

Attachment K: Technical Manual Outline for New York State Assessments

Section 1: Introduction and Overviewa. Introductionb. Test Purposec. Target populationd. Test Use and Decisions Based on Assessmente. Testing Accommodationsf. Test Transcriptionsg. Test Translations

Section 2: Test Design and Developmenta. Test Descriptionsb. Test Configurationc. New York State Educators’ Involvement in Test Developmentd. Test Blueprintse. Passage Selection and Item Criteria Documentsf. Passage Findingg. Item Developmenth. Educator Item Reviewi. Field Testingj. Rangefindingk. Item Selection and Test Creation (Criteria and Process)l. Educator Form Constructionm. Test Form Productionn. Final Eyes Committeeso. Proficiency and Performance Standards

Section 3: Validitya. Content Validityb. Construct (Internal Structure) Validity

Section 4: Test Administration and Scoringa. Test Administrationb. Scoring Procedures of Operational Testsc. Scoring Modelsd. Scoring of Constructed-Response Itemse. Scorer Qualifications and Trainingf. Quality Control Process

Section 5: Operational Test Data Collection and Classical Analysisa. Data Collectionb. Data Processingc. Classical Analysis and Calibration Sample Characteristics

197

d. Classical Data AnalysisSection 6: IRT Calibration and Linking

a. IRT Models and Rationale for Useb. Calibration Samplec. Item-Model Fitd. Local Independencee. Scalingf. Test Characteristic Curvesg. Scoring Procedure

Section 7: Reliability and Standard Error of Measurementa. Test Reliabilityb. Standard Error of Measurement (SEM)c. Performance Level Classification Consistency and Accuracy

Section 8: Standards ReviewSection 9: Summary of Operational Test Results

a. Scale Score Distribution Summaryb. Performance Level Distribution Summary

198

Attachment L: Print Quantities

See separate Excel Spreadsheet

199

Attachment M: Security Guidelines for the New York State Assessment Program

1. Staff of NYSED and of contractors working with the Office of State Assessment who are responsible for transporting, receiving, or handling secure test materials or confidential data, distributing such materials to consultants, work groups, and committees, and/or coordinating and overseeing related activities with consultants must be provided appropriate guidance to convey the importance of maintaining the security of materials.

2. Secure test materials and data must either be under the direct physical control of authorized NYSED personnel or their designated consultants or contractor personnel or in a secure storage area, which is inaccessible to other than authorized staff, always. Materials may never be left at a vacant desk, in an unattended conference room, or in an unattended hotel conference room.

3. Contractors must discuss with their NYSED contact person and receive prior approval of arrangements for delivery and storage of secure materials to locations other than NYSED or the contractor’s place of business.

4. Contractors must plan for the secure destruction of any secure materials used during the contract process of which they wish to dispose.

5. The contractor’s security procedures will include shipment of all secure materials needed for test development activities to test development sites in NYSED’s locked boxes. The locked containers will be provided to the contractor by NYSED. When shipping of all non-secure test materials, the contractor must use a carrier with ground-tracking capability, to test centers and to NYSED, whenever shipment of printed materials is necessary. The contractor may not utilize electronic transfer to ship individual student information or any secure test materials, unless as authorized by NYSED on a case-by-case basis. Electronic transfer includes transfer via e-mail, Internet, or facsimile (FAX).

6. Secure materials may never be emailed or faxed. If there appears to be a compelling reason to do so, prior approval must be obtained in writing from the NYSED Office of State Assessment.

7. The contractor will host and maintain a secure file transfer protocol (SFTP) site as a means of file transfer. Access to the New York State test information on the site must be limited to the contractor and NYSED unless further sharing with other parties is authorized in writing by NYSED.

8. Photocopying of secure or confidential material must be undertaken with care. Paper jams may result in paper containing secure or confidential materials being lodged in the copier and later discovered by another user. Extreme caution must be used. The contractor’s plan for photocopying must be approved by NYSED.

9. The contractor will require all staff and consultants who review secure materials, including but not limited to secure test questions, scoring materials, and related materials that reference secure test questions, or confidential data, must sign a Non-Disclosure Agreement (NDA) to be provided by NYSED. The contractor must retain the signed forms for at least one year beyond the end date of the contract, and promptly submit the signed forms to NYSED upon request.

200