developing the aptis testing system -...

25
Developing the APTIS Testing System Professor Barry O’Sullivan Senior Advisor, English Language Assessment

Upload: hoangtuong

Post on 01-Feb-2018

271 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Developing the APTIS Testing System

Professor Barry O’SullivanSenior Advisor, English Language Assessment

Page 2: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Overview

Localisation

Developing theories

Computerisation

Conclusions

Page 3: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

DEVELOPING THEORIESconceptualising the test

Page 4: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

System Design

Page 5: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Thinking about Test Performance

PERFORMANCETask Language

Target Language

Task Performance Parameters

Social Factors

Personal Characteristics

Cognitive Characteristics

Page 6: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Then There’s Scoring…

Machine or Human

Answer Key or Rating Scale

Selection & Training Consistency

Accuracy

Interpretation

Page 7: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Validation Theory

Page 8: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Decisions, Decisions…

Why are we testing?

Page 9: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

How Many Decisions?

TEST SCORE

Task Language•Vocabulary &

Grammar•Genre, Mode, etc.

Target Language•Vocabulary &

Grammar•Function, Mode etc.

Task Performance Parameters•Time & Support•Knowing Assessment

Criteria•Physical Conditions, etc.

Social Factors•Interlocutor [who;

number]•Relative Age, gender

etc.•Familiarity & Status•Etc.

Personal Characteristics•Physical•Psychological•Experiential

Cognitive Characteristics•Maturity•Ability•Etc.

Scoring System•Raters & Rating•Scales & Keys•Analysis & Grading•Etc.

Page 10: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Why all the Fuss?

Curriculumexplicitimplicit

Deliveryroom design

materialsteachers

Assessmentjudgemental

developmental

Decisions

Page 11: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

FOCUS ON READINGtheories of language progression

Page 12: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

The Reading Model

Page 13: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Word/Sentence

Paragraph

Text [within]

Text [external]

Inter Text [full text level]

Inter Text [paragraph level]

A

B

C

How it Works

Page 14: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Introducing

CORE[Grammar & Vocabulary]

Speaking

Writing

Listening

Reading

Page 15: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

LOCALISATIONthe british council approach

Page 16: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Theory to Practice – Test Design

Scoring System

Appropriate Content

Appropriate Task

Appropriate Language

Appropriate Scoring

Appropriate Topic

Test Taker

Test Task

LOCALISATION

Page 17: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Defining Localisation

The process of ensuring that all construct-irrelevant

variance related to the test taking population is

controlled for in the design and delivery of a test

Testing a well defined population within a well defined context in order to make decisions that will apply only to that context

Page 18: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Localisation – Case Study

Curriculum

Delivery Assessment

Page 19: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Case Study - Approach

Context

Population 15 & 17 yrsSecondary SchoolMixed ability, GenderVarious L1 [multilingual]Similar CultureSimilar CurriculumSimilar Delivery

Localisation

Language Reviewed[inc. instructions]

Reading TopicsListening TopicsWriting Topics/TasksSpeaking TopicsSpeaking Images

Page 20: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

COMPUTERISATIONthe growing importance of the machine

Page 21: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

The Computer & Test Practice

Item DevelopmentSpecificationWritingTriallingBankingArchiving

Test ScoringRater TrainingTest Break UpScoringAnalysing/AwardingScore Reporting

Test TakingKnowledgeReceptive SkillsProductive Skills

ResearchLanguageData AnalysisTask PerformanceRating Performance

System ManagementBack OfficeReporting to Clients

Test DeliveryComputerTabletOther…

Page 22: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

The Computer & Test Practice

Item DevelopmentSpecificationWritingTriallingBankingArchiving

Page 23: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

CONCLUSIONSwhat’s the point?

Page 24: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

The Message?

Every decision impacts on ALL stakeholders, though especially on the test taker

Without a clearly defined ABILITY MODEL the test is suspect

Without a clearly defined VALIDATION MODEL the test is suspect

It is close to impossible to deliver any test without technology

Page 25: Developing the APTIS Testing System - piap.clpiap.cl/.../barry-osullivan/Developing-The-APTIS-Testing-System.pdf · Developing the APTIS Testing System Professor Barry O’Sullivan

Professor Barry O’Sullivan

barry.o’[email protected]