ccsso 2015 symposium: future of science assessment · ccsso 2015 symposium: future of science...
TRANSCRIPT
![Page 1: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/1.jpg)
CCSSO 2015 Symposium: Future of Science Assessment
Lei Liu, Kathleen Scalise, Madeleine Keehner and Cindy Ziker Examples and demonstrations from the new U.S. National Assessment for Educational Progress (NAEP): Virtual science lab; scenario-based tasks; domain modeling; and process data in hands-on and virtual objects tasks.
![Page 2: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/2.jpg)
CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering scenario-based tasks
Kathleen Scalise Director, NAEP Science, ETS 6/22/15
![Page 3: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/3.jpg)
Technology-Enhanced Assessments
6/23/15
• Innovation is a central component for the future of educational assessment. New claims about student reasoning, behavior, and mental processes in context, along with new data sources, new scoring methods, and new performance assessment tasks are driving the next generation of science, mathematics, engineering and technology assessments.
3
![Page 4: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/4.jpg)
4
Measurement Technology
Information Technology
Source of Concept: Wilson, M. (2003). The technologies of assessment, Invited PresentaDon at the AEL NaDonal Research Symposium, Toward a NaDonal Research Agenda for Improving the Intelligence of Assessment Through Technology. Chicago.
Two Types Assessment Technology Innovations
![Page 5: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/5.jpg)
Information Technology Innovations
6/23/15
• NAEP Pilot 2015 employs science “scenarios” and simulators in rich tasks.
• NAEP also uses “hybridized” hands-on science tasks, and blocks of discretes (single) items.
• The tasks offer tools and animations to elicit what students know and can do through virtual and hands-on investigations.
• U.S. National Assessment of Educational Progress
5
![Page 6: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/6.jpg)
Task: Community water well in a rural village. Students investigate problems, query avatars, explore data, and provide explanations (Carr, 2013).
Carr, P. (2013). Presentations Of Selected Items/Tasks by Developers Of Those Assessments: NAEP. Presented at the Invitational Research Symposium on Science Assessment, Washington, DC.
NOTE: TEL Wells movie to be played.
Simulations: TEL Wells Task
![Page 7: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/7.jpg)
Task: Community water well in a rural village. Students investigate problems, query avatars, explore data, and provide explanations (Carr, 2013).
Carr, P. (2013). Presentations Of Selected Items/Tasks by Developers Of Those Assessments: NAEP. Presented at the Invitational Research Symposium on Science Assessment, Washington, DC.
NOTE: TEL Wells movie to be played.
Simulations: TEL Wells Task
![Page 8: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/8.jpg)
Engagement & Access Results: TEL
6/23/15
National Assessment Governing Board, May 2014 • NCES shared information from students and
school staff after the 2014 TEL administration, including discussion of three positive themes that emerged: • High levels of student engagement in TEL tasks (“now I think I might like to be an engineer”); • High levels of student completion of TEL additional supplemental block; • Supportive reactions to TEL administration and to task types in schools from school staff.
8
![Page 9: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/9.jpg)
Pump Troubleshooting Activity TEL Wells task is about process –
All students will (eventually) fix the pump.
We are interested in whether the process is: • Efficient: solves problem without unnecessary steps. • Systematic: solves problem methodically, with a logical sequence of steps.
9 Source: NCES, Sept. 2013
![Page 10: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/10.jpg)
10
1
2
5
3
4
We capture process data: • What is clicked (decisions/selections) • Order of clicks (sequences) • Number of clicks (frequencies) • Timing of clicks (timestamps)
Provides a trail of actions so we can: • Reconstruct problem-solving process • Characterize different strategies • Infer underlying cognition
Source: NCES, Sept. 2013
![Page 11: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/11.jpg)
Characterizing “Efficient Actions”
11
What does an “efficient” pattern
look like?
- WHICH choices you make
5
4
Source: NCES, Sept. 2013
![Page 12: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/12.jpg)
Characterizing “Systematic Actions”
12
What does a “systematic”
pattern look like?
- HOW you order your choices
Source: NCES, Sept. 2013
![Page 13: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/13.jpg)
Games-based Assessment
13
Source: GlassLab, May 2015
![Page 14: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/14.jpg)
Conversation-based Assessment
6/23/15 14
Source: J. Gorin, CERA, Dec. 2014
![Page 15: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/15.jpg)
Collaborative Tasks
Source: J. Gorin, CERA, Dec. 2014
![Page 16: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/16.jpg)
Multimodal Assessment: Live Performance
16
Source: J. Gorin, CERA, Dec. 2014
![Page 17: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/17.jpg)
Measurement Technology Innovations
6/23/15
• Adaptivity is one example of measurement technology innovation from NAEP.
• In NAEP multistage adaptive tests (MST), the test adaptation occurs based on student cumulative performance on a block of items. Multistage testing (MST) can be highly suitable because it can help better meet the needs of all students.
• Also, NAEP doing a special study on the use of adaptivity within the simulation tasks – “responsive” scenario-based tasks (RSBTs).
17
![Page 18: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/18.jpg)
Source: ETS, Nov. 2014
0
1
2
3
4
5
6
7
-5 -4 -3 -2 -1 0 1 2 3 4 5 6
Series1
Series2
Series3
Series4
Series5
Measurement Technology Innovation
![Page 19: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/19.jpg)
19
Examples of UDL tools available
• Available Only in Discrete (Single) Items 1. Elimination Tool (multiple-choice questions only) 2. Highlighter Tool 3. Zoom 4. Word Definition (some items only)
• Available in Discrete Items and SBTs (and Survey Questions) 5. Text to Speech (TTS) 6. Hide/Show Timer
![Page 20: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/20.jpg)
20
Use Example: Text-to-Speech (TTS) Use on TEL Cognitive Items
• TEL Discrete Items: Text-to-Speech (TTS) use ranged from about 6% to 30%.
• TEL Scenario-based Tasks (SBTs): In SBTs, TTS use ranged from 16% to 50% per task.
• At the student level, 53% of students used TTS at least once (either discrete or SBT).
Result of UDL tool use: TTS example
![Page 21: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/21.jpg)
Wrap-Up: Potential new directions for Science assessments • Tasks: Open-ended, more free-form
• Authentically reflect real science and engineering practices
• Evidence: Includes rich process data, assistive tools • Pathways, sequences, timing of actions, tool choices
• Reporting: Beyond scaled scores • Insights into process, strategy, cognition
We have more research to do, but what we are learning can contribute to the development of more authentic, rich, and informative approaches to STEM assessment and reporting.
21 Source: NCES, Sept. 2013
![Page 22: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/22.jpg)
NRC report describes that a “system” of assessment is needed: 1. Assessment tasks should allow students to engage in science
pracDces in the context of disciplinary core ideas and crosscuTng concepts.
2. MulD-‐component tasks that make use of a variety of response formats will be best suited for this.
3. Selected-‐response quesDons, short and extended constructed response quesDons, and performance tasks can all be used, but should be carefully designed to ensure that they measure the intended construct and support the intended inference.
4. Students will need mulDple and varied assessment opportuniDes to demonstrate their proficiencies with the NGSS performance expectaDons. 22
NRC Report on Assessing NGSS
![Page 23: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/23.jpg)
Discussion & Questions: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering scenario-based tasks
Contact Kathleen Scalise, 6/22/15 [email protected],
![Page 24: CCSSO 2015 Symposium: Future of Science Assessment · CCSSO 2015 Symposium: Future of Science Assessment Accessible, engaging assessment for all students in the NAEP science and engineering](https://reader033.vdocuments.site/reader033/viewer/2022042419/5f35e1d1f5fbf5651e633db4/html5/thumbnails/24.jpg)
24
U.S. National Assessment of Educational Progress:
• Largest nationally representative and continuing assessment of what America's students know and can do in various subjects.
• Provides the U.S. national and state “Report Cards” and trend assessments, as well as many publications, products, and data tools, see http://nces.ed.gov/nationsreportcard/
What is NAEP?