NCME April 11, 2007

Assessing Knowledge in a Learning Space:

Validity and/or Reliability

Jean-Claude Falmagne

University of California, Irvine

Eric CosynChris DobleNicolas ThieryHasan Uzun

Content:

Assessing Knowledge in a Learning Space What is a Learning Space? The Fringe Theorem The Projection Theorem Uncovering a Knowledge State Validity/Reliability of the Assessment Summary and Discussion

Content:

Assessing Knowledge in a Learning Space What is a Learning Space? The Fringe Theorem The Projection Theorem Uncovering a Knowledge State Validity/Reliability of the Assessment Summary and Discussion

Content:

Assessing Knowledge in a Learning Space What is a Learning Space? The Fringe Theorem The Projection Theorem Uncovering a Knowledge State Validity/Reliability of the Assessment Summary and Discussion

Assessing Knowledge in a Learning Space

In its principle, a learning space enables a very accurate assessment engine capable of uncovering the knowledge state of a student among

many possible ones.

In beginning algebra, for example, there are around 250 different types of problems (concepts, skills) that a student must master.

What do we mean by a “type of problem”?

Here is an example of a type of word problem.

One story line--out of 5--goes as follows:

"Abdul works mowing lawns and raking. He earns $5.40 anhour for mowing and $4.40 an hour for raking. How much will he earn for 5 hours of mowing and 1 hour of raking?"

Here is an example of a type of word problem.

One story line--out of 5--goes as follows:

"Abdul works mowing lawns and raking. He earns $5.40 anhour for mowing and $4.40 an hour for raking. How much will he earn for 5 hours of mowing and 1 hour of raking?”

Another story line for the same type of word problem is:

"In the past month Felipe rented 1 video cassette and 7 DVDs.The rental price for the video cassette was $2.80 . The rental price for each DVD was $3.20 . What is the total amount that Felipe spent on video cassette and DVD rentals in the past month?"

Let us say that there are five such story lines for that problem type. When selecting an instance of that word problem, one of those five story lines has to bechosen randomly, and the particular numbers involved(dollar amounts and whole numbers), must also be chosen. (Integer dollar amounts are excluded.)

Overall, there are 28,125 instances to choose from, randomly, for that particular word problem.

In other words, in this particular case, one type ofword problem corresponds to 28,125 items in the usual sense of a standardized test.

A knowledge state is a feasible subset of thatset of 250 problem types.

The number of feasible knowledge states is typically very large. In beginning algebra, this number is on the order of 107.

Despite this very large number of states, it is nevertheless possible to assess a student’sknowledge state in the topic, accurately, in the span of 25-35 questions.

Content:

Assessing knowledge in a learning space What is a Learning Space?

A learning space is a collection K of knowledge states.Each knowledge state is a subset of a set Q of problem types. Any knowledge state contains all the types of problems that some student in the population of referencemight be capable of solving.

forming the collection K.

The set Q

(Not a partition)

There are serious constraints on this concept. We suppose that a learning space always contains the empty set and the full set Q of problem types.

Thus, it is in principle possible for a student not to know anything, and for some other student to know everything.

Moreover, any learning space K satisfies two pedagogically reasonable principles:

First Principle:

If a knowledge state K is included in some otherknowledge state K’

K

K’

First Principle:

If a knowledge state K is included in some otherknowledge state K’, then

K

K’

First Principle:

If a knowledge state K is included in some otherknowledge state K’, then

K

K’

First Principle:

If a knowledge state K is included in some otherknowledge state K’, then

K

K’

First Principle:

If a knowledge state K is included in some otherknowledge state K’, then

K

K’

First Principle:

If a knowledge state K is included in some otherknowledge state K’, then

K

K’

First Principle:

If a knowledge state K is included in some otherknowledge state K’, then

K

K’

there exists at least one chain of states going from K to K’, each differing from the previous one by a single problem, so that learning can proceed from K to K’ by mastering the concepts one at a time.

In other words, the material must belearnable one step at a time, no matterwhere you are, and where you want to go beyond that.

Second Principle:

If a problem type c is learnable from someknowledge state K

K

Second Principle:

If a problem type c is learnable from someknowledge state K

c

K

Second Principle:

If a problem type c is learnable from someknowledge state K and if that knowledge state K is included in some knowledge state K’,

c

K

K’

Second Principle:

If a problem type c is learnable from someknowledge state K and if that knowledge state K is included in some knowledge state K’,

c

K’

then c should also be learnable from thestate K’.

In other words:

Knowing more can only help to learn something new.

These axioms have a Guttman’s scale flavor, but thatintuition would be misleading: if a Guttman’s scale interpretation of a learning space was attempted for any realistic learning space, literally billions ofGuttman’s scales would ensue. Such a construction would not be helpful.

Also, a formal concept of skill maps has been developed within this framework (Doignon and Falmagne, 1999). So far, we do not use it.

A learning space onthe set of problems{a,b,c,d,e,f }

Content:

Assessing Knowledge in a learning space What is a Learning Space? The Fringe Theorem

Except for the empty state (the empty set) and the full set Q, each knowledge state has two fringes: the outer fringe and the inner fringe.

Except for the empty state (the empty set) and the full set Q, each knowledge state has two fringes: the outer fringe and the inner fringe.

The outer fringe of a state K in a learning space is theset of all problem types q such that K + {q} is also astate.

In other words, intuitively: the outer fringe of a state K is the set of all problem types that are learnable from K.

Outer fringe

Except for the empty state (the empty set) and the full set Q, each knowledge state has two fringes: the outer fringe and the inner fringe.

The inner fringe of a state K in a learning space is theset of all problem types q such that K - {q} is also astate.

In other words, intuitively: the inner fringe of a state K is the set of all problem types that could have beenlearned last.

Inner fringe

Outer fringe of state K set of problem types learnable from K

Inner fringe of state K set of most recently learned problem types in K

EXAMPLE OF AN INNER AND OUTER FRINGE OF A STATE IN ARITHMETIC

(High Points)

The Fringe Theorem:In a learning space, any state is characterized (or can be accurately summarized) by its inner and outer fringe.

A FUNDAMENTAL RESULT

The Fringe Theorem:In a learning space, any state is characterized (or can be accurately summarized) by its inner and outer fringe.

A FUNDAMENTAL RESULT

The two fringes provide a very good summary of an assessment, immediately giving an entry intolearning: the student may be offered the choiceof studying any problem type in the outer fringe ofhis or her state.

The Fringe Theorem:In a learning space, any state is characterized (or can be accurately summarized) by its inner and outer fringe.

A FUNDAMENTAL RESULT

The two fringes provide a very good summary of an assessment, immediately giving an entry intolearning: the student may be offered the choiceof studying any problem type in the outer fringe ofhis or her state. One can obviously also give a total score, or a score in any of the subtopics.

Content:

What is ALEKS? What is a Learning Space? What are the fringes? The Fringe Theorem The Projection Theorem

What is a Projection of a Learning Space?

Intuitively, it is a smaller learning space giving a macroscopic view of the bigger one, a concept useful for a placementtest or similar tests.

A learning space onthe set of problems{a,b,c,d,e,f }

We begin by selecting a subsetof {a,b,c,d,e,f}. We first choose Problem type b.This choice determines a bipartition of the set of states.

Projection Theorem (Informal Statement.) If K is a learning space on Q, then any subset T of Q defines a partition of Q such that each class of that partition corresponds to a subset of T forming a knowledge state of a new learning space U on T. This learning space has fewer states and fewer problem types if T is a proper subset of Q.

The learning space U is called the projection of thelearning space K under T.

Projection Theorem (Informal Statement.) If K is a learning space on Q, then any subset T of Q defines a partition of Q such that each class of that partition corresponds to a subset of T forming a knowledge state of a new learning space U on T. This learning space has fewer states and fewer problem types if T is a proper subset of Q. Moreover, each class of the partition is a collectionsatisfying all the conditions of a learning space, except that the empty set is not necessarily a state. (Dan Cavagnaro, 2006, submitted to the JMP.)

The learning space U is called the projection of thelearning space K under T.

Why a projection?

Think about the learning space for the complete mathematics curriculum for all of K-12.

We have such a learning space, based on around900 problems.

Using a projection, based on selecting a subsetof problem types, a placement test can be manufactured in a matter of a couple of days.

Using a projection, based on selecting a subsetof problem types, a placement test can be manufactured in a matter of a couple of days.

With a little more programming, it would be in acouple of hours.

Content:

Assessing knowledge in a learning space What is a Learning Space? The Fringe Theorem The Projection Theorem Uncovering a Knowledge State

Content:

What is ALEKS? What is a Learning Space? What are the fringes? The Fringe Theorem The Projection Theorem Uncovering a Knowledge State

NOTE THAT I COMPLETELY BYPASS THE VERY DIFFICULT PROBLEM OF HOW WE CAN, IN PRACTICE, CONSTRUCT A LEARNING SPACE.

Content:

What is ALEKS? What is a Learning Space? What are the fringes? The Fringe Theorem The Projection Theorem Uncovering a Knowledge State

Schema of an assessment algorithm

Schema of an assessment algorithm

The most informativequestion is asked

ALSO, BEFORE I FORGET: NO MULTIPLE CHOICE.

Schema of an assessment algorithm

Bayesian Rule

Content:

What is ALEKS? What is a Learning Space? What are the fringes? The Fringe Theorem The Projection Theorem Uncovering a knowledge state Is the assessment valid/reliable?

Beginning Algebra

Basic idea: in each assessment, an additional,randomly selected problem is asked, the response to which is not taken into account to gauge the knowledge state. However, the assessed state can be used to predict the student’s response.

Beginning Algebra

210,102 assessments204 problem types, out of 250

Beginning Algebra

210,102 assessments204 problem types, out of 250

46 problems were discarded because the relevant data were too meager.

Beginning Algebra

210,102 assessments204 problem types, out of 250.

85% college students15% high school students.

So, we have on the average210,102/250 = 1,030… assessmentsper problem.

204 beginning algebra problems

204 beginning algebra problems

On the suggestion of Brian Junker (Thanks!)we also used the log odds ratio to evaluate thecorrelation.

Based on 185 problems(19 problems were discardedbecause one of the cellscontained a zero)

However, the tetrachoric coefficient and thelog odds ratio tell essentially the same story.

Finally, still another, very important way of evaluating the accuracy of the assessment is to look at the efficiency of the learning process guided by the assessment.

The assessments are taking place at well chosen moments determined by the system or by the teacher.But, in addition to those assessments, the system also updates the student’s knowledge states continuously.

This is what some of you call “a system that teaches as it assesses.”

When a student has successfully learned a new problem type, the system adds that problem type to the student’s state, and displays the new statewith its outer fringe on the computer screen.

The student then picks, in that outer fringe, a newproblem type to learn.

A natural questions is: how successful is the systemat predicting what the student is ready to learn?

Distribution of the conditional probabilities that a student successfully learnsa problem type chosen in the outer fringe of his or her knowledge state.Data based on 1,564,296 learning occasions (for the 250 problem types).

Distribution of the conditional probabilities that a student successfully learnsa problem type chosen in the outer fringe of his or her knowledge state.Data based on 1,564,296 learning occasions (for the 250 problem types).

Major differences with a psychometric test

1. The objectives and the philosophy.

Major differences with a psychometric test

1. The objectives and the philosophy.2. The theories.

Major differences with a psychometric test

1. The objectives and the philosophy.2. The theories.3. The result of the assessment or the test.

Major differences with a psychometric test

1. The objectives and the philosophy.2. The theories.3. The result of the assessment or the test.4. The principles underlying the construction of the two instruments.

THANK YOU!