CAA 2004 1

Thinking Skills Tests forUniversity Admission

Assisting Admissions Decisions

Robert HardingDirector

TSAT Project and ITAL UnitInteractive Technologies in Assessment and Learning

http://tsa.ucles.org.uk/

CAA 2004 2

Thinking Skills Assessment :Recent history in Cambridge

Need for additional information to select applicants BMAT: BioMedical Admissions Test:

now used also by Oxford and UCL, Royal Vet College About 4000 candidates in 2003, expecting 6000 in 2004

TSA numbers (used in some Cambridge Colleges): 289 in 2001 478 in 2002 Over 1500 in 2003

There is evidence that in the Cambridge context the test is a better predictor of performance than either A-Levels or interview scores.

CAA 2004 3

Some admissions tests in ROW

The United States Medical Licensing Examination (USMLE)http://www.nbme.org/about/about.asp

The Law School Admission Test (LSAT)http://www.lsac.org/

Graduate Medical School Admissions Test (GAMSAT)http://www.acer.edu.au/tests/university/gamsatuk/intro.html

“The SAT is a three-hour test that measures verbal and mathematical reasoning skills students have developed over time and skills they need to be successful academically.”http://www.collegeboard.com/student/testing/sat/about/SATI.html

CAA 2004 4

BMAT :what is tested?

Section 1 Thinking Skills: 40 multiple choice questions, 1 hour

Section 2: Science and Maths core knowledge:30 questions, 30 mins

Section 3 essay: capacity to develop ideas and to communicate them effectively in writing30 mins.

CAA 2004 5

Thinking Skills Assessment :what is tested?

Critical Thinking and Problem Solving 50 multiple choice questions, online or on paper One ‘paper’: test lasts 90 minutes Origins in academic research around mid 1980’s MENO 1993/4 tested six ‘thinking skills’ Today:

OCR’s AS Level in Critical Thinking CIE’s AS Level in Thinking Skills

Evidence that these skills are highly relevant to successful study.

CAA 2004 6

Thinking Skills Assessment :Is it useful?

“The TSA Project has been running on a pilot basis since 2001 in Cambridge. Its results are used carefully and conditionally by Colleges as additional information and never replace any traditional criteria on which admission decisions are based. More research and evaluation is being done in this pilot project to establish its value for admissions purposes. Although this research is not yet complete, the use of the TSA test within Cambridge has grown each year as confidence in its usefulness grows.”

CAA 2004 7

Thinking Skills Assessment :what does this do for ‘access’?

Provides a uniform measure of readiness for Higher Education

Can be self-tested; encourage applicantsLow cost support materials widely

available, eg via wwwWorth learning these skillsFacilitates access research.

R = 0.33

0

50

100

150

200

250

300

350

0 10 20 30 40 50 60 70 80 90

TSA score

Trip

os m

ark

Data for 48 candidates who took the TSA in December 2001 and their first University examination (the Tripos) in the summer of 2003, all Computer Science.

2001/2003 data for 48 candidates, showing the top 12 TSA scorers and the bottom 12 TSA scorers.

R = 0.33

0

50

100

150

200

250

300

350

0 10 20 30 40 50 60 70 80 90

TSA score

Trip

os m

ark

2001/2003 data for 48 candidates, but Tripos marks replaced by randomised values.

R = -0.002

0

50

100

150

200

250

300

350

0 10 20 30 40 50 60 70 80 90

TSA score

Trip

os m

ark

Scatter plot of 2002 TSA score vs 2004 Tripos score:478 candidates took the TSA, 83 went on to 1st year exams in Natural Sciences, Computer Science, and Mathematics courses.

0 10 20 30 40 50 60 70 80 90 100

Overall Mark

0

50

100

150

200Fr

eque

ncy

Mean = 60.62Std. Dev. = 8.216N = 1,551

TSA 2003 total scores - distribution

0 10 20 30 40 50 60 70 80 90 100

Critical Thinking

0

50

100

150

200

Freq

uenc

y

Mean = 59.23Std. Dev. = 10.005N = 1,551

0 10 20 30 40 50 60 70 80 90 100

Problem Solving

0

50

100

150

Freq

uenc

y

Mean = 62.8Std. Dev. = 10.227N = 1,551

TSA 2003 scores distribution

Critical Thinking Problem solving

CAA 2004 14

What’s the CAA perspective?

Whole process is computer-assisted: Item banking & test construction Administration and registration Answer sheet scanning (ICR) Results return and reporting System developed by Cambridge’s CARET

(Centre for Applied Research in Educational Technology) To be made Open Source, QTI compliant

MCQ’s, so can be taken on paper or on-line: Can compare on-line & paper performance.

CAA 2004 19

What’s the CAA perspective?

Whole process is computer-assisted: Item banking & test construction Administration and registration Answer sheet scanning (ICR) Results return and reporting System developed by Cambridge’s CARET

(Centre for Applied Research in Educational Technology) To be made Open Source, QTI compliant

MCQ’s, so can be taken on paper or on-line: Can compare on-line & paper performance.

Delivery Mode

Score type N Mean Std Dev

Paper Total 1114 60.4 8.3

Prob. Solv. 1114 62.5 10.2

Crit. Think. 1114 59.0 10.1

Online Total 437 61.0 8.1

Prob. Solv. 437 63.4 10.2

Crit. Think. 437 59.5 9.6

CAA 2004 21

TSAT in 2004 and Beyond? UCLES is planning to continue pilot testing the TSA in Cambridge, on

a similar basis to 2003

The BMAT continues in 2004 with additional universities We will continue to consult interested parties and stakeholders

We will continue to develop the technology

We may look for interest from other Universities, and will consider

overseas testing We will look at issues of timing and location We will continue to research admissions issues.

CAA 2004 22

Some URL’s:

http://www.ucles.org.uk/http://tsa.ucles.org.uk/

http://bmat.org.u/http://www.caret.cam.ac.uk/

Thinking Skills Admissions Testing

Acknowledgements and thanks …

To many colleagues in UCLES and to our partners in CARET who played very full parts in this work.