How the Scoring Process Works Webinar...@EducateIN How the Scoring Process Works Webinar. School Year 2019-2020. Welcome to the How the Scoring Process Works Webinar. This presentation
22
@EducateIN How the Scoring Process Works Webinar School Year 2019-2020
How the Scoring Process WorksWebinarSchool Year 2019-2020
Presenter
Presentation Notes
Welcome to the How the Scoring Process Works Webinar. This presentation is designed to help educators understand the scoring process for the ILEARN assessments.
Objectives:• Increase educator knowledge about what hand-
scoring is for the upcoming ILEARN assessments.
• Review the overall process involved to train and qualify to become a Scorer.
Agenda:• Background on Hand-scoring • Scorer Training and Qualification• Scorer Requirements
Webinar Objectives
Presenter
Presentation Notes
The purpose of this module is to increase educator knowledge about the ILEARN hand-scoring process. This training will explain how human scorers are trained and meet qualification requirements for scoring student responses to open-ended ILEARN test items.
• Prior to ILEARN legislation, vendors were solely responsible for recruitment and oversight of scorers• Maintained qualifications, but Indiana educator
field was not primary source of scorers
• ILEARN legislation requires IN to recruit educators to assist in the hand-scoring process for open-ended items
IN Hand-Scoring Philosophy Change
Presenter
Presentation Notes
In accordance with Indiana legislation, the Spring 2019 ILEARN test administration marked the first time vendors recruited Indiana educators to assist in the scoring of student responses. Beginning with the 2019-2020 school year, Indiana educators will be recruited to help score student responses to open-ended items appearing on ILEARN tests during the Fall, Winter, and Spring Biology ECA windows as well as the Spring 3-8 window.
Definition of Hand-Scoring:• What is hand-scoring?
• Hand-scoring on ILEARN
• Information on which assessments have hand-scored items can be confirmed on the public facing blueprints which are located here: https://www.doe.in.gov/assessment/ilearn-test-design
• ILEARN-approved hand-scoring guidelines.
• Hand-scoring philosophy founded on comprehensive training and consistent scoring quality.
Background on Hand-Scoring
Presenter
Presentation Notes
Hand-scoring is the process in which scorers use an application of scoring tools and resources to assign scores to student assessments. While some assessments have only machine-scored items, most assessments (including ILEARN) contain some �hand-scored items that are scored by humans. Open ended items, which often lend themselves to hand-scoring, typically allow for higher depths of knowledge to ensure the cognitive complexity of the standards can be fully attained on the assessment. Where do educators find this information? There is a link to the ILEARN blueprints on this slide. The additional information section of each blueprint highlights this information. The scoring vendor is responsible for ensuring that qualified Scorers are hired and properly trained to hand-score responses according to ILEARN-approved scoring guidelines. The scoring vendor’s hand-scoring philosophy is founded on ensuring that all Scorers receive comprehensive training on the specific items they are scoring and that their scoring quality remains consistent. This criteria is extremely important to the Indiana Department of Education (IDOE) as well.
• MIRA securely displays all of the required materials
What is the difference between scoring and grading?
Scoring Training and Qualification
VS.
Presenter
Presentation Notes
Scorers will access the remote scoring application (MIRA) online training interface remotely. MIRA can be accessed from home or work. MIRA securely displays all of the materials required to successfully train and qualify. Scorers must sign non-disclosure agreements to ensure the integrity of all assessment information is maintained. These materials include scoring rubrics, anchor sets, training sets, and qualification sets. Grading often involves looking at each response as an independent entity. This process can lend itself to subjectivity. Hand-scoring involves defining consistent logic to be utilized across all responses. This process requires objectivity.
Definition of a Scorer:• What is a Scorer?
• Scorer training – in-person and/or online/remote• The same training is provided for in-person and remote Scorers.
• All Indiana educators who participate in hand-scoring will score remotely.
• Scorers are assigned specific items or item groups.
• All Scorers must successfully complete the training and qualification process.
Scoring Training and Qualification
Presenter
Presentation Notes
A Scorer is employed by the scoring vendor to learn scoring criteria through item- or prompt-specific training and evaluate student performance-assessment (constructed) responses either onsite or remotely. Scorers maintain a high level of consistency and production through feedback from scoring leadership. Scorers are trained by the Scoring Director (in-person) or using scripted videos (online remotely). The same training is provided for in-person and remote Scorers. This ensures comparability of all Scorers who are scoring the same test item. More information on how to apply to become a Scorer will be shared in a later slide.
@EducateIN
Questions?
Presenter
Presentation Notes
Definition of a Scoring Rubric:• What is a scoring rubric?
• Accurate scoring requires that each Scorer learns how to apply the scoring criteria.
Scoring Training and Qualification
Presenter
Presentation Notes
Definition of a Scoring Rubric: A rubric is an evaluation tool or set of criteria used to promote the consistent application of scoring expectations, scoring objectives, or scoring standards in a hand-scoring setting. Rubrics are used as scoring instruments to ensure the objectivity is maintained throughout the evaluation process. Accurate scoring requires that each Scorer internalizes how to apply the scoring criteria for the specific item(s) he or she is scoring. These criteria are outlined in a scoring rubric. The Scorer training will focus on a single item at a time.
Example Scoring Rubric: • Suppose a Mathematics item has a score point
range of 0 to 2 points and asks the students to provide the correct answer with an explanation.
• If the correct answer is 25, then the scoring rubric may be as follows:• Score point 2: Student provides the correct answer (25)
with an explanation that 5 x 5 = 25.• Score point 1: Student provides the correct answer (25),
but the explanation is unclear or incorrect.• Score point 0: Student does not provide the correct
answer (25) with an explanation.
Scoring Training and Qualification
Presenter
Presentation Notes
Example Scoring Rubric: Suppose a Mathematics item has a score point range of 0 to 2 points and asks the students to provide the correct answer with an explanation. If the correct answer is 25, then the scoring rubric may be as follows: Score point 2: Student provides the correct answer (25) with an explanation that 5 x 5 = 25. Score point 1: Student provides the correct answer (25), but the explanation is unclear or incorrect. Score point 0: Student does not provide the correct answer (25) with an explanation.
Definition of an Anchor Set:• What is an anchor set?
• Each response in the anchor set includes an annotation.
• Scoring Directors use anchor sets to elucidate how scoring criteria or logic are applied.
Scoring Training and Qualification
Presenter
Presentation Notes
Definition of an Anchor Set: An anchor set is used as a hand-scoring management tool comprised of exemplar responses with annotations and other associated scoring documents to be used by Scorers during initial training and then to self-monitor their scoring for calibration throughout the scoring session. Each response in the anchor set includes an annotation that identifies the approved score the response received and explains the reasoning or logic behind the score given, noting how the response contains certain elements detailed in the scoring rubric. Scoring Directors use anchor sets to elucidate how scoring criteria are applied to each response’s specific features, thereby training Scorers how to recognize the differences between score points.
Example of an Anchor Set:• For the Mathematics item we previously discussed, an
anchor set may provide the following types of responses:• Some examples of 2-point responses that show various
correct ways for a student to explain how the correct answer, 25, was determined.
• Some examples of 1-point responses that show the correct answer (25) along with various explanations that are not clear enough to earn the highest score point.
• Some examples of 0-point responses that show various incorrect answers, common misconceptions, and/or responses too vague to receive any points.
• Educators can access the practice scoring module that is available on the IDOE Moodle Course Page https://www.doe.in.gov/assessment/ilearn-sample-items-and-scoring
Scoring Training and Qualification
Presenter
Presentation Notes
To see an example of an anchor set educators can access the practice scoring module that is available on the IDOE Moodle Course Page https://www.doe.in.gov/assessment/ilearn-sample-items-and-scoring
Definition of a Practice Test:• What is a practice set?
• The practice responses are responses from actual students.
• Review of correct score for each response and the rationale for the score.
Scoring Training and Qualification
Presenter
Presentation Notes
Definition of a Practice Set: A practice set consists of 10 student responses designed to help the Scorers learn to apply the criteria illustrated in the anchor set and ensure that they are familiar with the process of scoring student responses and assess their understanding of the scoring criteria. After Scorers have reviewed the anchor set, they review a set of practice responses. The practice responses are responses from actual students and contain examples of all score points arranged randomly. After Scorers have scored the practice set responses in the practice set, they are provided with the correct score for each response and the rationale for the score.
Definition of a Qualification Set:• What is a qualification set?
• Taking qualification sets
• Qualification standard
Scoring Training and Qualification
Presenter
Presentation Notes
Definition of a Qualification Set: Much like a practice set, a qualification set consists of student responses arranged in random score point order used to assess the Scorers’ understanding of the scoring criteria before they are allowed to begin live scoring of student responses. After Scorers have reviewed and discussed the practice set, Scorers take two qualification sets. In order to score actual student responses, a Scorer must meet the standard on one qualification set.
Sample Qualification Criteria:• For example, if an item has a score point range of 0 to 2
points, then the qualification criteria may be 80% correct.
• In this example, the Scorer has qualified by correctly assigning 90% of the scores correctly, only missing response # 5.
Scoring Training and Qualification
Response # 1 2 3 4 5 6 7 8 9 10
Assigned Score 2 0 1 0 2 1 2 0 1 2
Correct Score 2 0 1 0 1 1 2 0 1 2
Presenter
Presentation Notes
Sample Qualification Criteria: For example, if an item has a score point range of 0 to 2 points, then the qualification criteria may be 80% correct. In this example, the Scorer has qualified by correctly assigning 90% of the scores correctly, only missing response #5.
@EducateIN
Questions?
Presenter
Presentation Notes
• Organization by teams
• Scorers utilize all training materials during live scoring
Live Scoring and Scorer Monitoring
Presenter
Presentation Notes
Once a Scorer has qualified to score an item, he or she begins to score actual student responses. During scoring, Scorers are grouped into teams, with each team supervised by a team leader. Scorers utilize all training materials during live scoring: scoring rubrics, anchor sets, practice sets, qualification sets, and any other examples of responses that are determined useful during the live scoring window.
• Training is an ongoing and continuous process.
• The team leader role
• What is Scorer monitoring?• Read Behinds• Validity Checks• Interrater Reliability
Live Scoring and Scorer Monitoring
Presenter
Presentation Notes
Training is an ongoing and continuous process that does not end once a Scorer qualifies. The Team Leader assists in monitoring the scoring performance of Scorers. Monitoring includes reviewing, or “reading behind,” scored responses to check Scorers’ accuracy and reviewing statistical data of how well Scorers perform. By focusing on specific items, Scorers develop expertise in the nuances of the scoring criteria and how that criteria applies to the item(s) being scored. This ensures consistency of scoring within and across test administrations. Monitoring of Scorers During Live Scoring: Read Behinds: Team Leaders review responses that Scorers have scored in order to provide feedback to Scorers and to ensure accuracy. Validity Checks: Pre-scored responses are distributed to scorers blindly throughout live scoring to further assess Scorer performance and accuracy. Interrater Reliability Data: This data provides the number of responses scored by each Scorer and the percentage of those responses scored in exact agreement with other Scorers.
Live Scoring and Scorer Monitoring
Retrain Release Rescore
Presenter
Presentation Notes
Monitoring of Scorers During Live Scoring: If a Scorer does not score with acceptable accuracy and production, he or she is retrained and monitored by scoring leadership. If retraining is unsuccessful, the Scorer is released from scoring the assessment. When possible, scoring leadership will look for other areas where the Scorer might be a better fit before releasing the Scorer. All responses scored by the released Scorer are reset and resubmitted to qualified Scorers to access and score.
Scorer Requirements
• 4-year bachelor’s degree from an accredited college or university
• Currently employed as an educator at an accredited Indiana school
• Currently licensed by the State of Indiana• Access to a computer/online device that meets the
minimum technical requirements• Consistency in applying scoring consistency during
process
Checklist
Presenter
Presentation Notes
A 4-year bachelor’s degree from an accredited college or university is required for scoring. Educational licenses include the following - instructional, administrative, and school services. Minimum technical requirements includes high speed internet and a secure testing environment. Scorers must demonstrate the ability to apply scoring criteria as directed and maintain consistent and reliable scoring throughout the scoring process while employed as a temporary Scorer. Scorers must be willing to engage in a repetitive task situation and maintain confidentiality.
Any questions about the Indiana educator scoring opportunity can be sent to the AIR Help Desk at [email protected] or via telephone (866) 298-4256.
Become a Paid Scorer!
• January 13, 2020
Hiring Process
• January 13, 2020
Educator FAQ
• April 22 – June 5, 2020
Scoring Window
Presenter
Presentation Notes
Educators will have the opportunity to officially apply on January 13, 2020.
• A Practice Scoring Module is currently available on IDOE’s Moodle page for educator access -https://www.doe.in.gov/assessment/ilearn-sample-items-and-scoring
• The Practice Scoring Module contains a few Mathematics and English/Language Arts items.
• Supplemental scoring materials will be available to support these items.
Online Practice Scoring Module
Presenter
Presentation Notes
The scoring module will contain items that will be similar to items educators will score on the ILEARN assessments in Spring 2020.
