Upload File

introduction to item response theory · classical test theory vs. latent trait models test level is...

  • Home
  • Documents
  • Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended
45
Introduction to Item Response Theory www.psychometrics.cam.ac.uk Prof John Rust, [email protected] David Stillwell, [email protected] Aiden Loe, [email protected] Luning Sun, [email protected]

Upload: others

Post on 20-May-2020

10 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Introduction to Item Response Theory

www.psychometrics.cam.ac.uk

Prof John Rust, [email protected] Stillwell, [email protected]

Aiden Loe, [email protected] Sun, [email protected]

Page 2: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Goals

� Build your own basic tests using Concerto

� General understanding of CTT, IRT and CAT

concepts

� No equations!

Page 3: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Understanding the Latent Variable

3

Page 4: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

What is a construct?� Personality scales?

� Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism

� Intelligence scales? � Numerical Reasoning, Digit Span

� Ability scales? � Find egs. Of ability

4

Page 5: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

What is a construct?� Measures and items are created in order to

measure/tap a construct

� A construct is an underlying phenomenon within a scale - referred to as the latent variable (LV)� Latent: not directly observable� Variables: aspects of it such as strength or magnitude,

change

� Magnitude of the LV measured by a scale at the time and place of measurement is the true score

5

Page 6: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Latent variable as the cause of item values

� LV is regarded as a cause of the item score

� i.e. Strength of LV (its true score) causes items to take on a certain score.

� Cannot directly assess the true score

� Therefore look at correlations between the items measuring the same construct

� Invoke the LV as cause of these correlations

� Infer how strongly each item correlated with LV

6

Page 7: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Classical Measurement Assumptions

X = T + e

�X = observed score

�T = true score

�e = error7

Page 8: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Classical Measurement Assumptions

1. Items’ means unaffected by error if have a large number of respondents

2. One item’s error is not correlated with another item’s error

3. Error terms are not correlated with the true score of the latent variable

8

Page 9: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Classical Test Theory� Observed Test Score = True Score + random error

� Item difficulty and discrimination

� Reliability

� Limitations:� Single reliability value for the entire test and all

participants� Scores are item dependent

� Item stats are sample dependent� Bias towards average difficulty in test construction

Page 10: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Classical Test Theory vs.Latent Trait Models

� Test level is the basis for CTT (not the item).

� While the statistics outcome are often extended to similar students taking a similar test; they only really apply to those students taking thattest (norm based)

� Latent trait models go further and investigate the underlying traits which are producing the test performance.

Page 11: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Introduction to IRT

Some materials and examples come from the ESRC RDI in Applied Psychometrics run by:Anna Brown (University of Cambridge)

Jan Böhnke (University of Trier)Tim Croudace (University of Cambridge)

Page 12: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended
Page 13: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Item Response Theory

� Item Response Theory (IRT) – refers to a family of latent trait models.

� They are used to establish psychometric properties of items and scales

� IRT has many advantages over CTT that have brought IRT into more frequent use

Page 14: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

3 Basics Components of IRT� Item Response Function (IRF) – Mathematical function

that relates the latent trait to the probability of endorsing an item

� Item Information Function – an indication of item quality; an item’s ability to differentiate among respondents

� Invariance – position on the latent trait can be estimated by any items with known IRFs. The ICCs are sample independent (If you spilt by groups, the item parameters should be the same).

Page 15: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Ratio of correct responses to items on different level of total score

Measured concept (ability)

Probability of getting item right

1

Page 16: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Item Response FunctionBinary items

Parameters:• Difficulty• Discriminati

on• Guessing• Inattention

Measured concept (theta)

Probability of getting item right1

Models:• 1

Parameter• 2

Parameter• 3

Parameter• 4

Parameter• unfolding

Difficulty

Guessing

Inattention

Page 17: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

One-Parameter Logistic Model/Rasch Model (1PL)

7 items of varying difficulty (b)

Page 18: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Rasch vs IRT philosophy (roughly)

� Rasch –Model primacy. Data should fit the model

� IRT – Data primacy. Model should fit the data

Different fields, constructs, and purposes demand different approaches.

� The jury is out as to whether this has any impact on measurement in the real world.

Page 19: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Two-Parameter Logistic Model (2PL)

5 items of varying difficulty (b) and discrimination (a)

Page 20: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Three-Parameter Model (3PL)

One item showing the guessing parameter (c)

Page 21: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Option Response Function

Binary items

0.0

0.2

0.4

0.6

0.8

1.0

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0

Pro

bab

ility

Theta

Probability of Correct + Probability of Incorrect = 1

Correct response

Incorrect response

Page 22: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Graded Model (example of a model with polytomous items – e.g. Likert Scales)

“I experience dizziness when I first wake up in the morning”(0) “never”(1) “rarely”(2) “some of the time”(3) “most of the time”(4) “almost always”

Category Response Curves for an item representing the probability of responding in a particular category conditional on trait level

Page 23: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

0.0

0.2

0.4

0.6

0.8

1.0

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0

Prob

ability

Theta

Fisher Information Function

Page 24: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Test Information Function� Test Information Function (TIF) – The IIFs are

additive so that we can judge the test as a whole and see at which part of the trait range it is working the best.

Page 25: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

0.0

0.2

0.4

0.6

0.8

1.0

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0

Prob

ability

Theta

(Fisher) Test Information Function

Three items

*Not to scale

Page 26: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

TIF and Standard Error (SE)

� Error of measurement inversely related to information

� Standard error (SE) is an estimate of measurement precision at a given theta

Page 27: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Most likely score

Scoring Test:1. Normal

distribution 2. q1 – Correct3. q2 – Correct4. q3 - Incorrect

Most likely score

0.0

0.2

0.4

0.6

0.8

1.0

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0

Prob

ability

Theta

Most likely score

Page 28: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Quiz !

Page 29: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Quiz ! • Which of the following is not an IRT parameter?

- Unfolding- Discrimination- Difficulty - Probability - Guessing

• [HARD ONE!] Name the parameters from their shorthand

a -

b -

c -

Page 30: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Quiz ! • Which of the following is not an IRT parameter?

- Unfolding- Discrimination- Difficulty - Probability - Guessing

• [HARD ONE!] Name the parameters from their shorthand

a - discrimination

b – difficulty

c - guessing

Page 31: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

IRT assumptions� Unidimensionality – All items are assumed to

measure a single common factor.

� Local independence – item responses are uncorrelated after controlling for the latent trait.

� Homogenous population – No differential item functioning (DIF) between groups.

Page 32: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Classical Test Theory vs. Item Response Theory

Classical IRT

Modelling / Interpretation Total score

Individual items (questions)

Accuracy / Information Same for all participants and scores

Estimated for each score/ participant

Adaptivity Virtually not possible Possible

Score Depends on the items Item independent

Item Parameters Sample dependent Sample independent

Preferred items Average difficulty Any difficulty

Page 33: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

IRT in R

ltm package

Suggested Resource:Computerised Adaptive Testing: The State of the Art (November 2010)

Dr Philipp Doebler of the University of Munster describes the latest thinking on adaptivity in psychometric testing to an audience of psychologists.

Page 34: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

“Mobility” Survey

� A rural subsample of 8445 women from the Bangladesh Fertility Survey of 1989 (HuqandCleland, 1990).

� The dimension of interest is women’s mobility and social freedom.

� Described in: Bartholomew, D., Steel, F., Moustaki, I. and Galbraith, J. (2002) The Analysis and Interpretation of Multivariate Data for Social Scientists. London: Chapman and Hall.

� Data is available within R software package “ltm”

Data we are using today:

Page 35: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

“Mobility” SurveyWomen were asked whether they could engage in the following activities alone (1 = yes, 0 = no):

1. Go to any part of the village/town/city.

2. Go outside the village/town/city.

3. Talk to a man you do not know.

4. Go to a cinema/cultural show.

5. Go shopping.

6. Go to a cooperative/mothers' club/other club.

7. Attend a political meeting.

8. Go to a health centre/hospital.

Page 36: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

ltm packageinstall.packages("ltm")

require(ltm)

help(ltm)

head(Mobility)

my1pl<-rasch(Mobility)

my1pl

summary(my1pl)

plot(my1pl, type = "ICC")

plot(my1pl, type = "IIC", items=0)

Page 37: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

ltm package## rasch

myrasch<-rasch(Mobility, cbind(9,1))

my2pl <- ltm(Mobility ~ z1)

anova(my1pl, my2pl)

(the smaller the better!)

Now plot ICC and IIC for 2pl model.

Page 38: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

ltm package – scoring

resp<-matrix(c(1,1,1,1,0,1,0,1), nrow=1)

factor.scores(my2pl, method="EAP", resp.patterns=resp)

EXPLAIN: “$” addressing

theta = dataCAT$score.dat$z1

sem = dataCAT$score.dat$se.z1

mobIRT <- factor.scores(my2pl, resp.patterns=Mobility, method="EAP")

head(mobIRT$score.dat)

Page 39: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Compare IRT and CTT scores

CTT_scores <- rowSums(Mobility)

IRT_scores <- mobIRT$score.dat$z1

plot(IRT_scores, CTT_scores)

#Plot the standard error and scores

IRT_errors <- mobIRT$score.dat$se.z1

plot(IRT_scores, IRT_errors, type="p")

Page 40: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Why use Item Response Theory?

� Reliability for each examinee / latent trait level

� Modelling on the item level

� Examinee / Item parameters on the same scale (so you can compare the outcomes of two different tests measuring the same latent trait)

� Examinee / Item parameters invariance (so if you test a latent trait on a subsection of the population you can get parameter estimates for the whole population)

� Score is item independent

� Adaptive testing

� Also, test development is: cheaper and faster! (no need to re-norm every time you change an item)

Page 41: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Introduction to CATVery brief

Page 42: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Computerized Adaptive Testing

� Standard test is likely to contain questions that are too easy and/or too difficult

� Adaptively adjusting to the level of the test to this of participant:� Increases the accuracy� Saves time / money� Prevents frustration

Page 43: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

0.0

0.2

0.4

0.6

0.8

1.0

-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0

Prob

ability

Theta

Example of CATStart the test:1. Ask first question, e.g. of

medium difficulty2. Correct!3. Score it4. Select next item with a

difficulty around the most likely score (or with the max information)

5. And so on…. Until the stopping rule is reached

Most likely score

Difficulty

Correct responseIncorrect response

Normal distribution

Page 44: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

Elements of CAT� IRT model

� Item bank and calibration

� Starting point

� Item selection algorithm (CAT algorithm)

� Scoring-on-the-fly method

� Termination rules

� Item bank protection / overexposure

� Content Balancing

Page 45: Introduction to Item Response Theory · Classical Test Theory vs. Latent Trait Models Test level is the basis for CTT ( not the item). While the statistics outcome are often extended

CAT Demos

Suggested Resource:Computerised Adaptive Testing: The State of the Art (November 2010)

Dr Philipp Doebler of the University of Munster describes the latest thinking on adaptivity in psychometric testing to an audience of psychologists.

http://planning.e-psychometrics.com/test/cat-demo

http://planning.e-psychometrics.com/test/cat-dna

Demos created by Aiden Loe


Statistical Approaches to Analyzing Well-being Data … · Statistical Approaches to Analyzing Well-being Data ... item response theory, ... item response theory, classical test theory,

UNIT IV ITEM ANALYSIS IN TEST DEVELOPMENT CHAP 14: ITEM ANALYSIS CHAP 15: INTRODUCTION TO ITEM RESPONSE THEORY CHAP 16: DETECTING ITEM BIAS 1

Test Item Specification

CLASSICAL TEST THEORY vs. ITEM RESPONSE THEORY · Classical Test Theory vs. Item Response Theory An evaluation of the theory test in the Swedish driving-license test Marie Wiberg

Online placement test based on Item Response Theory … · Online placement test based on Item Response Theory and IMS Global ... (NTIC) to continuously ... The theory aims to expose

Classical Test Theory as a First- Order Item Response ...Paul W. Holland Machteld Hoskens Research & Development Division Princeton, NJ 08541 Classical Test Theory as a First-Order

Linear Item Response Theory, Nonlinear Item Response ... irt, non... · Linear Item Response Theory, Nonlinear Item Response Theory, and Factor Analysis: A Unified Framework Albert

An Item Response Theory Analysis for the Test of ...parrise.elte.hu/tpi-15/slides/Rakkapao_Suttida.pdf · 3 TUV is a 20-item 5-choice research-based conceptual test of vectors (Barniol

Item Response Theory: What It Is and How You Can Use the IRT …€¦ · Item response theory (IRT) is concerned with accurate test scoring and development of test items. You design

JMETRIK: Classical Test Theory and Item Response Theory

The Effects of Multiple-Choice Item Formats on Grammar ......Keywords: grammar item format, multiple-choice items, test methods, placement test, diagnostic test, generalizability theory,

Assessing the discriminating power of item and test … the discriminating power of item and test ... used in classical test theory ... Hofstee et al., 1998; McDonald,

Comparing Test Sets with Item Response Theory

Test Item Writing

Classical Test Theory as a First- Order Item Response ... · Classical Test Theory as a First-Order Item Response Theory: Application to True-Score Prediction From a Possibly Nonparallel

Item response theory 101

Advances in Clinical Personality Measurement: An Item ... Classes/Spring 08... · Advances in Clinical Personality Measurement: An Item Response Theory ... classical test theory

Item Response Theory and Rasch Models · Item Response Theory and Rasch Models I tem response theory (IRT) is a second contemporary alternative to classical test theory (CTT). Although

Menuliskan Item-Item Test Untuk Berbagai

analysis test item

Item Response Theory. What’s wrong with the old approach? Classical test theory –Sample dependent –Parallel test form issue Comparing examinee scores

Item Response Theory

Classical Test Theory vs. Item Response Theory

Silva’s Management Competency Theory: A Factor-Item Analytic Approach …€¦ ·  · 2014-10-29Abstract - Classical test theory is a body of psychometric ... was developed, following

Item Test Analysis

Implementing Item Response Theory

DEVELOPMENT OF CHINESE COMPUTERIZED ADAPTIVE TEST SYSTEM … · erized adaptive test (HCAT) system based on higher-order Item Response Theory (HO-IRT) for a Chinese proficiency test

The Congeneric Test Theory and The Congeneric Item Analysis:

Item Response Theory and Computerized Adaptive Testing CAT Gershon 6_10_2014.pdf · Item Response Theory and Computerized Adaptive Testing. Outline Item Response Theory versus Classical

Chapter 8 The “New Psychometrics” – Item Response Theory · PDF fileChapter 8 The “New Psychometrics” – Item Response Theory Classical test theory is concerned with the

A visual guide to item response theory - metheval - … 2: Some basic ideas 5 2. Some basic ideas The ultimate aim of both classical test theory (CTT) and item response theory (IRT)

Modern Test Theory Item Response Theory (IRT). Limitations of classical test theory An examinee’s ability is defined in terms of a particular test The

Item Response Theory Analysis of the Recoded Internet ... - Item response theory... · The IGD-20 test is an extended measure to assess IGD that is aligned with Griffiths’s(2005)

An item response theory analysis of sex differences with ... · An item response theory analysis of sex differences with the Miller Forensic Assessment of Symptoms Test ... these

USING MULTIDIMENSIONAL ITEM RESPONSE THEORY TO …3920/... · 2016-09-14 · USING MULTIDIMENSIONAL ITEM RESPONSE THEORY TO REPORT SUBSCORES ACROSS MULTIPLE TEST FORMS By Jing-Ru