Slide 1

Evaluating the Validity of English Language Proficiency Assessments (EVEA): A Consortium Approach National Conference on Student Assessment Detroit, Michigan June 22, 2010 8:30-10:00 am Slide 2 Overview of the Project Ellen Forte The Role of the Research Partner Alison Bailey The State Perspective Joe Willhoft Questions Slide 3 Overview of the Project Project Partners, Tasks, and Timelines The Argument-Based Approach to Validity Evaluation ELPA Common Interpretive Argument Moving Forward Slide 4 The Evaluating the Validity of English Language Proficiency Assessments (EVEA) project: is a federally funded, Enhanced Assessment Grant spans October 2009 through March 2011 brings together a consortium of five states with a team of researchers and a panel of experts to develop a coherent validity argument for English language proficiency (ELP) assessments will build a common interpretive argument, and design a set of studies and instruments to support and test the claims within the argument Builds off of previous, federally-funded projects (NHEAI- EAG, NAAC-GSEG), which focused on technical documentation and validity arguments for AA-AAS. Slide 5 Five states Washington (lead) Oregon Montana Idaho Indiana Five organizations edCount, LLC National Center for the Improvement of Educational Assessment University of California Los Angeles Synergy Enterprises, Inc. Pacific Institute for Research and Evaluation Partners Slide 6 Alison Bailey, UCLA Co-Principal Investigator Ellen Forte, edCount Co-Principal Investigator Management Team Liz Towles-Reeves Project Director Marianne Perie, NCIEA Co-Principal Investigator Sara Waring Deputy Project Director Research Partnerships Coordinator: Sara Waring Expert Panel Coordinator: Marianne Perie Administration and Partnerships Coordinator: Liz Towles-Reeves Idaho RP: Sara Waring State Lead: Wendy St. Michell Montana RP: Sara Waring State Lead: Judy Snow Oregon RP: Laura Kuti/Ellen Forte State Leads: Phyllis Guile and Tony Alpert Indiana RP: Allison Kerbel State Lead: Wes Bruce Jamal Abedi, UC-Davis, CRESST Derek Briggs, U of CO Frances Butler, Language Testing Consultant Laurene Christiansen, NCEO Charlie DePascale, NCIEA Scott Marion, NCIEA Teddi Predaris, Fairfax Co. Public Schools Katherine Ryan, U of IL edCount NCIEA UCLA Brent Garrett, PIRE External Evaluator SEI, Inc. Joe Willhoft, Washington OSPI Senior Advisor and State Lead Washington RP: Alison Bailey State Lead: Joe Willhoft Organization Chart Slide 7 Task 200920102011 O1O1 N2N2 D3D3 J4J4 F5F5 M6M6 A7A7 M8M8 J9J9 J 10 A 11 S 12 O 13 N 14 D 15 J 16 F 17 M 18 Task 1 Convene Expert Panel Task 2 Management Meetings Task 3 Develop Validity Plan Task 4 Develop Study Designs Task 5 Instrumentation Task 6 Collect and Analyze Data Task 7 Reporting to ED Task 8 External Reporting Task 9 Grant/Contract Management Project Partners, Tasks, and Timelines Slide 8 Kane, M. (2006). Validity. In R. L. Linn (Ed.), Educational Measurement (4th ed., pp. 1764). New York: American Council on Education, Macmillan Publishing. Builds on previous validity perspectives The argument-based framework described by Kane: assumes that the proposed interpretations and uses [of test scores] will be explicitly stated as an argument, or network of inferences and supporting assumptions, leading from observations to the conclusions and decisions. Validation involves an appraisal of the coherence of this argument and of the plausibility of its inferences and assumptions (Kane, 2006, p. 17) Slide text adapted from Perie, M. (2009). Evaluating the Validity of English Language Proficiency Assessments (EVEA). Washington, DC: edCount. The Argument-based Approach Slide 9 Arguments An interpretative argument specifies the proposed interpretations and uses of test results by laying out the network of inferences and assumptions leading from the observed performances to the conclusions and decisions based on the performances. The validity argument provides an evaluation of the interpretative argument (Kane, 2006). The Argument-based Framework Slide text adapted from Perie, M. (2009). Evaluating the Validity of English Language Proficiency Assessments (EVEA). Washington, DC: edCount. Slide 10 Interpretive Argument Illustration of the theory that underlies test score interpretation and use Identifies claims about the test and the assumptions underlying those claims Allows for the prioritization of evidence needs for claims and assumptions Used as a logic model; for example: IF the assessment adequately represents the construct IF teachers administer the instrument correctly IF the tasks are scored accurately THEN we can make claims about a students knowledge of that construct based on his score The Argument-based Framework Slide text adapted from Perie, M. (2009). Evaluating the Validity of English Language Proficiency Assessments (EVEA). Washington, DC: edCount. Slide 11 The outcome: tested claims from the interpretive argument Claims that survive the evaluation are included in the final validity argument. Claims can be deleted or modified and additional claims can be added based on findings in the final validity argument. The validity argument includes evidence. The Argument-based Framework Slide text adapted from Perie, M. (2009). Evaluating the Validity of English Language Proficiency Assessments (EVEA). Washington, DC: edCount. Slide 12 Teachers have the knowledge, skills, and orientation necessary to provide instruction in support of academic English language acquisition Programs successfully move more ELL students into the Proficient category and exit them out of the ELP programs The ELD/P standards have been developed to support the acquisition of English language proficiency necessary to achieve academic content and performance expectations. The ELP assessment has been designed to yield scores that reflect students knowledge and skills in relation to academic English language expectations defined in the ELD/P standards. ELP assessment scores/ performance levels are used appropriately to inform decisions about progress in attaining English language proficiency ELP assessment scores/ performance levels accurately reflect students English language proficiency The ELP assessment is administered as intended ELL students become proficient in English, acquiring the academic language skills necessary to participate fully in instructional discourse conducted in English. ELPA Common Interpretive Argument Students have been appropriately identified to participate in the ELP program and assessment. Slide 13 A Claim Teachers have the knowledge, skills, and orientation necessary to provide instruction in support of academic English language acquisition Programs successfully move more ELL students into the Proficient category and exit them out of the ELP programs The ELD/P standards have been developed to support the acquisition of English language proficiency necessary to achieve academic content and performance expectations. The ELP assessment has been designed to yield scores that reflect students knowledge and skills in relation to academic English language expectations defined in the ELD/P standards. ELP assessment scores/ performance levels are used appropriately to inform decisions about progress in attaining English language proficiency ELP assessment scores/ performance levels accurately reflect students English language proficiency The ELP assessment is administered as intended ELL students become proficient in English, acquiring the academic language skills necessary to participate fully in instructional discourse conducted in English. Students have been appropriately identified to participate in the ELP program and assessment. Slide 14 Assumptions Underlying a Claim The ELP assessment is administered as intended Teachers/administrators have sufficient training to administer the assessment correctly Teachers/administrators have sufficient training AND other knowledge and skills necessary to conduct live scoring, if that is required Teachers/administrators have access to the materials and other resources (e.g., time, space, help) they need to administer the assessment correctly Slide 15 Moving Forward Common Interpretive Argument to Individual State Interpretive Arguments Individual State Validity Evaluation Plans Prioritization of Claims and Assumptions Research Studies Designed to Test the Claims and Assumptions Targeting common issues to develop methodologies for investigation through pilot studies Slide 16 Dissemination of products CIA Study Methodologies (piloted) Instrumentation (piloted) Generic Foundations Document Home Language Survey White Paper Table of Contents for an ELPA Technical Manual Everything will be freely available to the public at www.eveaproject.com www.eveaproject.com Moving Forward Slide 17 The Research Partner Perspective Slide 18 The State Perspective Slide 19 Benefits for States EVEAs benefits for partner states All states in EVEA use a different ELP test Opportunity to develop across-state analyses Mutual learning opportunities distribution of common issues Learn from other states of approaches to common issues Slide 20 Benefits for States EVEAs benefits for partner states Support provided by the Research Partner Experienced and knowledgeable research partners provide assistance with state-specific research and analysis issues for each partner state Opportunity to collaborate with external expert on development of a validity framework Slide 21 Benefits for States EVEAs benefits for partner states Development of a state-specific validity framework Hands-on experience with elaboration of a validity framework Expanding sophistication of state-level assessment staff with the argument-based approach to validity Able to work in an assessment context with a standard set of operational criteria and a manageable theory of action Slide 22 Contact Information Ellen Forte, Ph.D. President, edCount, LLC 5335 Wisconsin Avenue, NW, Suite 440 Washington, DC 20015 eforte@edCount.com 202/895-1502 Alison Bailey, Ph.D. Faculty Associate Researcher (CRESST) and Associate Professor, UCLA Department of Education CRESST/UCLA abailey@gseis.ucla.edu 310/825-1731 Slide 23 Contact Information Joe Willhoft, Ph.D. Assistant Superintendent for Assessment and Student Information Washington State Office of Superintendent of Public Instruction 600 Washington St., SE Olympia, WA 98501 Joe.Willhoft@k12.wa.us 360/725-6334 Project Website: www.eveaproject.comwww.eveaproject.com