.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Model Selection For Adaptive Testing

Martin Plajner, Jirka Vomlel

The Institute of Information Theory and AutomationCzech Academy of Sciences

16.7.2015Bayesian Applications Workshop

UAIAmsterdam

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 1 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Adaptive testing

Selecting subset of questionsShorter test versionsIndividual sets of questions

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 2 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Data set

Paper test of mathematical knowledge (domain of functions)

4 grammar schools

281 test subjects (students)

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 3 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Test description

29 questions (mathematical problems) - 53 sub questions

Example question

Decide which of the following functions

f (x) = x2 − 2x − 8

g(x) = −x2 + 2x + 8

is decreasing in the interval (−∞,−1].

Rated correct(0) / incorrect(1)or graded 0 - 4

Total maximum of 120 points

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 4 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Test results, correlations

Table: Average test scores of the four grammar schools.

GS1 GS2 GS3 GS4

42.76 46.68 46.35 43.65 44.53

Table: Correlation of the grades and the test total score.

Mathematics Physics Chemistry

-0.60 -0.42 -0.41

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 5 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Bayesian network models

Each model consist of

A set of n variables {S1, . . . , Sn} - skills.

S will denote the multi variable (S1, . . . , Sn) taking statess = (s1, . . . sn).

A set of m variables {X1, . . . ,Xm} - questions (mathematicalproblems)

X will denote the multi variable (X1, . . . ,Xm) taking statesx = (x1, . . . , xm).

A set of arcs between variables

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 6 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Models’ options

Options summary

Question grading scales Boolean numericNumber of skill nodes 1 7Skill nodes states 2 3Additional information NO YES

Bayesian network models7 models for Boolean scale7 models for numeric scale

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 7 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Network model with one skill node and additionalinformation

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 8 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Expert network model

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 9 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Adaptive test

Select a next question

Ask the question

Update the network

Next question selection

Selection based on information gaini.e. reduction of expected entropy

Cumulative Shannon entropy

H(e) =n∑

i=1

∑si

−P(Si = si |e) · logP(Si = si |e)

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 10 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Next question selection

Expected entropy for the selection of X ′

EH(X ′, e) =

p∑j=1

P(X ′ = x ′j |e) · H(e ∪ {X ′ = x ′j})

Question X ∗ minimizing the expected entropy

X ∗ = argminX ′∈Xs

EH(X ′, e)

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 11 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Model evaluation

10 fold cross-validation

For every (281) test repeat:

Select a questionAsk the selected questionInserted answer as evidence into the BNDistribute and collect evidenceEstimate subsequent answers

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 12 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Model evaluation

We measure the success rate of a test

⇒ By comparing the estimated value to the observed value

Success rate

SRts =

∑Xi∈Xs+1

I (x∗i = x ′i )

|Xs+1|, where

I (expr) =

{1 if expr is true0 otherwise

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 13 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Boolean graded question answers

0 10 20 30 40

0.65

0.75

0.85

0.95

step

succ

ess

rate

b2+b3b2e

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 14 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Numeric graded question answers

0 10 20 30 40

0.65

0.70

0.75

0.80

0.85

0.90

0.95

step

succ

ess

rate

n2+n3n2e

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 15 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Question selection order

0 5 10 15 20 25 30

010

2030

40

steps

vars

Step

Que

stio

n ID

0 5 10 15 20 25 30

steps

vars

Step

Que

stio

n ID

0 5 10 15 20 25 30

stepsva

rs

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 16 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Conclusions

Proposed models provide reasonable results

3-stated skill variables provide better results

Expert model does not score as good as expected

Additional information is not as important

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 17 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Future research

Explore more options of parameters selection

Comparison with other model types

standard Rash, IRT, other methods (neural network)

Practical issues (fairness, constraints,. . . )

−4 −2 0 2 4

0.2

0.6

1.0

x

f

−4 −2 0 2 40e+

003e

−10

x

f

−4 −2 0 2 40e+

004e

−21

x

f

−4 −2 0 2 40e+

006e

−26

x

f

−4 −2 0 2 40e+

004e

−30

8e−

30

x

f

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 18 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Future research

Explore more options of parameters selection

Comparison with other model types

standard Rash, IRT, other methods (neural network)

Practical issues (fairness, constraints,. . . )

−4 −2 0 2 4

0.2

0.6

1.0

x

f

−4 −2 0 2 40e+

003e

−10

x

f

−4 −2 0 2 40e+

004e

−21

x

f

−4 −2 0 2 40e+

006e

−26

x

f

−4 −2 0 2 40e+

004e

−30

8e−

30

x

f

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 18 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Future research

Explore more options of parameters selection

Comparison with other model types

standard Rash, IRT, other methods (neural network)

Practical issues (fairness, constraints,. . . )

−4 −2 0 2 4

0.2

0.6

1.0

x

f

−4 −2 0 2 40e+

003e

−10

x

f

−4 −2 0 2 40e+

004e

−21

x

f

−4 −2 0 2 40e+

006e

−26

x

f

−4 −2 0 2 40e+

004e

−30

8e−

30

x

f

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 18 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Future research

Explore more options of parameters selection

Comparison with other model types

standard Rash, IRT, other methods (neural network)

Practical issues (fairness, constraints,. . . )

−4 −2 0 2 4

0.2

0.6

1.0

x

f

−4 −2 0 2 40e+

003e

−10

x

f

−4 −2 0 2 40e+

004e

−21

x

f

−4 −2 0 2 40e+

006e

−26

x

f

−4 −2 0 2 40e+

004e

−30

8e−

30

x

f

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 18 / 18

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Future research

Explore more options of parameters selection

Comparison with other model types

standard Rash, IRT, other methods (neural network)

Practical issues (fairness, constraints,. . . )

−4 −2 0 2 4

0.2

0.6

1.0

x

f

−4 −2 0 2 40e+

003e

−10

x

f

−4 −2 0 2 40e+

004e

−21

x

f

−4 −2 0 2 40e+

006e

−26

x

f

−4 −2 0 2 40e+

004e

−30

8e−

30

x

f

Martin Plajner, Jirka Vomlel Model Selection For Adaptive Testing 16.7.2015 18 / 18