hr lsa trends

Advancements in Large-‐scale Assessment

Heiko Rölke & Krisz:na Toth DIPF -‐ Deutsches Ins:tut für Interna:onale

Pädagogische Forschung Frankfurt am Main

TAO Days | 10.09.2012 Seite 1

Overview

•  Trends in LSA •  Closer Look at Trends •  Example: Making Use of Data •  Challenges in LSA

TAO Days| 10.09.2012 Seite 2

Trends

Main trend: CBA

Sub-‐trends/effects of CBA: •  CAT

•  towards forma:ve assessment •  Complex items

•  Simula:on, interconnec:on •  Interweaving

•  different tests, ques:onnaires, etc. •  Big data


Trend: CBA

Computer-‐based Assessment has to serve a purpose •  Costs? •  Time! •  Validity!


CAT

•  Also not prac:ced for its own sake •  Not as important in pure summa:ve assessment •  But:

•  Strict :me limits (e.g. PIAAC) •  Growing demand for forma:ve aspects in summa:ve assessment


Complex Items

•  Closer at reality •  Simula:on Most important: •  Ability to assess new domains


Examples of Complex Items §  Simula+on of an “Email / Web Scenario”

The test person receives an email and should book cinema :ckets online.

§  Dynamic Model (MicroDYN)

The test person should explore and master a dynamic system with input (exogenous) variables influencing output (endogenous) variables.

§  Automaton with Finite State Machine (MicroFIN) The test person should interact with a mobile phone and set :me to summer:me.

Interweaving of Tests and Ques:onnaires •  Framing

•  test only if precondi:ons are fulfilled •  Double-‐check

•  findings of ques:onnaire


Big Data

•  Log data, not only results •  Find out what is going on •  Make use of data

•  E.g. par:al scoring

•  -‐> Examples from PIAAC PS-‐TRE


Simula:on-‐based assessment

•  Modern assessments: •  real life situa:ons: e.g. web environments •  complexity of instrument – various ways

•  Monitoring students‘ ac:ons -‐ log files •  Analysis of test-‐taking paths is a rela:vely new field in educa:on

(reason: printed vs. computer-‐based data collec:on) •  Process data requires to consider and evaluate methods

Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 10

Aim •  to inves:gate how log data can be integrated into the process

of educa+onal assessment and evalua:on to support researchers and prac::oners in making use of the data assembled in computer-‐based test delivery


Aim •  to inves:gate how log data can be integrated into the process

of educa+onal assessment and evalua:on to support researchers and prac++oners in making use of the data assembled in computer-‐based test delivery


Hypertext Item

13

15

829709 15 {"sender":"15","type":"loading","data":""} 833463 15 {"sender":"15","type":"loaded","data":""} 848153 15 {"sender":"15","type":"user_interac+on","":"<?xml version=\"1.0\">

\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntry xmlns:cbaloggingmodel=\"htp://www.soucon.de/cba/cbaloggingmodel\" id=\„-‐ b576759f1:-‐7fed\" sourcePageId=\"Item15_linklist\" targetPageId=\"Item15_website1\" textFieldId=\"cbaTextField_71"/>\u000d\u000a"} 849094 15 {"sender":"15","type":"variable_change","data":{"name":

"snapshot_url„,"value":"htp://localhost:8101/cba-‐run:me/itemjsessionid= LB1?custom_servicehandler=downloadService&file= C:\\...snapshot5537437125227002523.xml"}}

855852 15 {"sender":"15","type":"user_interac+on","data":"<?xmlversion= \"1.0\“encoding=\"UTF8\"?>\u000d\u000a <cbaloggingmodel:BuRonLogEntryxmlns:cbaloggingmodel=\"htp://www.sou con.de/cba/cbaloggingmodel\id=\"cbaBackBuRon_14_13454937565947\"/> \u000d\u000a„}":"15","type":"unloaded","data":""}

Example log

Promising methods •  Sta:s:cs and visualisa:on •  Clustering •  Classifica:on


Process Measures


1. Number of page visits 2. Number of different page visits 3.  Visit of relevant page 4.  Time spent on the relevant page 5.  Ra:o of :me spent on the relevant page 6.  Ra:o of :me spent on the opening screen 7.  Comple:on :me

Sta:s:cs

Process measures Mean Standard Devia+on

Number of page visits 5.18 3.13 Number of different page visits 2.38 1.38 Time spent on the relevant page 10.87 9.77 Ra:o of :me spent on the relevant page .14 .11 Ra:o of :me spent on the opening screen .64 .18 Comple:on :me 67.74 27.49


Distribu:on of different page visits


Visualisa:on

21 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-‐Scale Assessments

Visualisa:on

22 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-‐Scale Assessments

Cluster Analysis

Features Cluster 1 Cluster 2 Cluster 3 Cluster 4

Nr. of page visits

Nr. of different page visits

Comple:on :me

Relevant page visited (Y/N)

Ra:o of :me on relevant page

Ra:o of :me on star:ng page

Distribu:on of sequences (%) 23% 16% 34% 27%

Ra:o of correct responses

23 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments

Cluster Analysis


Nr. of page visits

Nr. of different page visits

Comple:on :me

Relevant page visited (Y/N)

Ra:o of :me on relevant page

Ra:o of :me on star:ng page


Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%


Cluster Analysis


Nr. of page visits .72 5.41

Nr. of different page visits .36 2.54

Comple:on :me 28.38 62.08

Relevant page visited (Y/N) No No

Ra:o of :me on relevant page 0 0

Ra:o of :me on star:ng page .93 .60

Distribu:on of sequences (%) 23% 16%

Ra:o of correct responses 25.0% 1.7%


Cluster Analysis


Nr. of page visits 4.73 9.57

Nr. of different page visits 2.29 4.53


Relevant page visited (Y/N) Yes Yes

Ra:o of :me on relevant page .20 .17





Cluster Analysis


Nr. of page visits 5.41 4.73

Nr. of different page visits 2.54 2.29


Relevant page visited (Y/N) No Yes

Ra:o of :me on relevant page 0 .20





Cluster Analysis


Nr. of page visits .72 5.41 4.73 9.57

Nr. of different page visits .36 2.54 2.29 4.53

Comple:on :me 28.38 62.08 59.60 91.60

Relevant page visited (Y/N) No No Yes Yes

Ra:o of :me on relevant page 0 0 .20 .17

Ra:o of :me on star:ng page .93 .60 .62 .45




Summary and Conclusions


Summary and Conclusions


Nr. of page visits .72 5.41 4.73 9.57

Nr. of different page visits .36 2.54 2.29 4.53

Comple:on :me 28.38 62.08 59.60 91.60

Relevant page visited (Y/N) No No Yes Yes

Ra:o of :me on relevant page 0 0 .20 .17

Ra:o of :me on star:ng page .93 .60 .62 .45




Future work •  Pilot study – sample size •  Valida:on

•  Other types of items require new process measures


Future work •  Pilot study – sample size •  Valida:on

•  Other types of items require new process measures •  We have a lot of work to do:

•  Souware developer •  Test developer •  Psychometricans


(Selected) Challenges

•  Authoring and Management •  Delivery •  Re-‐use and exchange


hr lsa trends

Documents

big data tao days

tth tao days

ve assessment tao days

main tao days

lsa tao days

onnaire tao days

process data

big data log data