1

Task-BasedLanguageAssessmentandTes1ngforProficiency:WhereDotheTwainMeet?ErwinTschirnerL.E.A.R.N.WorkshopSeptember20-21,2016UniversiAesatShadyGrove

Overview

•  OriginsofTask-BasedLanguageAssessment•  Task-Based:ShadesofMeaning•  Evidence-CenteredDesign(ECD)•  RelaAonshipBetweenECDandProficiencyAssessment•  ConstructoftheOPI•  AssessmentUseArgument•  FutureofProficiencyAssessment

2

Task-BasedLanguageAssessment(TBLA)

3

TheprocessofevaluaAng,inrelaAontoasetofexplicitlystatedcriteria,thequalityofthecommunica1veperformanceselicitedfromlearnersaspartofgoal-directed,

meaning-focusedlanguageuserequiringtheintegra1onofskillsand

knowledge”(Brindley,1994:74).

4

“TheelicitaAonandevaluaAonoflanguageuse(acrossallmodaliAes)forexpressingandinterpreAngmeaningwithinawell-definedcommunica1vecontext(andaudience),foraclearpurpose,towardsavaluedgoalor

outcome”(Norris,2014:25).

5

EarlyConsideraAons(Mislevyetal,2002)

•  Disenchantmentwithdiscrete-skillsassessmentsfocusingongrammarandvocabularywithlargelydecontextualizedtestitems–  RealizaAonthatlinguisAccompetence(grammarandvocabulary)

isnotsufficienttousealanguagetoachieveendsinsocialsituaAons

•  Alignmentofassessmentwith(task-based,proficiency-oriented)communicaAveinstrucAon

•  PosiAvewashbackofassessmentpracAcesoninstrucAon

6

Advantages

•  AssessestheabilitytouselanguagetoachieveendsinsocialsituaAons(Mislevyetal.,2002)

•  AssessestheintegraAonoftopical,social,andpragmaAcknowledgewithknowledgeoftheformalelementsoflanguage

•  AlignswithCommunicaAveLanguageTeachingandTask-basedLanguageInstrucAoninForeignLanguageEducaAon

•  EnablesmeaningfulintegraAonofteaching,learning,andassessment(Wiggins1990)

7

EarlyProponents

•  ForeignServiceIns1tuteOralInterview(1950s)•  GrantWiggins(1990):AuthenAcAssessment•  GeoffBrindley(1994):Task-centeredAssessment

–  Outcomes-basedassessment(1998andlater)

•  LyleBachman&AdrianPalmer(1996):LanguageTesAnginPracAce

•  TimMcNamara(1996):PerformanceTesAng•  MichaelLong&JohnNorris(2000):Task-basedAssessment•  RobertMislevy(2002):Evidence-basedDesign

8

ShadesofMeaning

•  Theinfluenceoftaskfeaturesontesttakers’cogniAveprocessesandresulAngperformance(Skehan,1998)

•  SharedcharacterisAcssuchaslearner-centeredness,contextualizaAon,andauthenAcitybetweenassessmentandinstrucAon(Chalhoub-Deville,2001)

•  AbilitytoaccomplishspecifictargettasksinparAcularcommunicaAonsegngs(Long&Norris,2000)

•  AbilitytoengageinthekindsofacAviAescharacterisAcallyencounteredincommunicaAvelanguageteaching(Wiggins,1990)

9

Evidence-CenteredDesign

10

‘Data’become‘evidence’onlywhentheirrelevancetosomehypothesis,someinference,

someclaimisestablished.Intask-basedlanguageassessment,thismeansthatwhatwereallyneedtounderstandfirstandforemostistheinferen1al

argumentassociatedwiththeassessment.Whatisitspurpose?Whatdowewanttoknow,aboutwhatstudentsknoworcando,inwhatkindsof

situa1ons?(Mislevyetal.,2002:492-3).

11

TheMeaningofTestResults

•  Databecomeevidenceonlywhen:=TestResultsaiainmeaningonlywhen:•  Theirrelevancetoahypothesis,aninference,aCLAIMhas

beenestablished.

Ø  Toulmin’sArgumentModelØ  Bachman/Palmer’s(2010)AssessmentUseArgumentØ Mislevyetal’sEvidence-CenteredDesign

12

Evidence-CenteredDesign

Whatisthepurposeoftheassessment?

TaskAnalysis

Whatdowewanttoknowaboutwhatstudentsknoworcando?

Ability

Evidence

InwhatkindsofsituaAons? Task

13

Mislevy,Steinberg,&Almond(2002)

ExamineeModelAbility

KnowledgeSkills

EvidenceModelScoring

Measuring

TaskModelSchemastoelicit

evidence

AssemblyModelCombiningtaskstoproduceanassessment

ExamineeModel

•  ComplexofknowledgeandabiliAestobeassessed•  Unobservable•  Construct-driven(basedonSLAtheory)

–  CommunicaAveCompetence(e.g.Bachman/Palmer1996)–  ProficiencyLevelDescripAons(ILR,ACTFL)–  Can-dostatements(ILR,ACTFL,CEFR)

•  Developmentalstages/ImplicaAonalscaling

16

EvidenceModel•  Observablebehaviorsthatprovidesufficientevidenceaboutthe

knowledgeorabiliAeswewishtomeasure•  EvaluaAonComponent

–  Extractsthesalientcharacteris1csofwhatstudentssayordo(Keyaspectsoftheperformance)

–  BasedonevaluaAonrules(rubricsforraAngscales)–  Whatisvalued,andhowisitevaluated?

•  MeasurementComponent–  Accumula1onofinformaAontoupdatebeliefsaboutexaminees:What

dotheobservablevariablestellusabouttheexaminee’sabiliAes?–  HowisevidencesynthesizedacrossmulApletasksanddifferent

performances?(Whendoyouknow,e.g.,whenyouhaveenough?)

17

TaskModel•  TasksorsituaAonsthatelicitthebehaviorsweneedas

evidence•  SchemaforconstrucAnganddescribingthesituaAonsinwhich

examineesact•  Task-driven(basedontask/domain/needsanalysis)

–  Salientfeaturesoftasks–  Howdotasksinfluence(andconstrain)performance?–  Whatarekeyfeaturesoftargetlanguageuses(TLUs)intasks

(authenAcity)?•  Developmentalstages/ImplicaAonalscaling

–  Hierarchyoftasks

18

AssemblyModel

•  Mixandnumberoftasksthataresufficientforanassessment–  Determinestherangeofcircumstancesthatneedtobecovered

(tobeabletogeneralize)–  Controlsthedifficultyoftasks–  ManageswhatinformaAonaccumulatesordoesnotaccumulate

19

ExamineeModelPragmaAcComp.Socio-ling.Comp.TextCompetenceGram./Lex.Comp.

EvidenceModelSuccess/Failure

Quality:Howwell?QuanAty:Howmuch?

TaskModelGlobalTasks

Context/ContentRoleofInterlocutor

AssemblyModelChecks/ProbesRangeofTopicsRangeofTasksKindsofTasks

SpeechSample

•  Providescumula1veevidenceoftheexaminee’slanguageability

•  RepresentaAveoftheexaminee’slanguageability–  FuncAons,TextualorganizaAon,Register,Culturalcompetence–  Varietyoftopicsandcontexts–  ConcernforaffecAveschemata(warm-upetc.)

•  IndicaAveoftheexaminee’scontrolovertheabove–  Random,emerging,developing,sustained

21

Task-centeredandConstruct-centered

Task-centeredperspecAve•  Featuresoflanguage-usesituaAonsthatreveal:•  Thelanguage-usecompetencesthatareofinterest;and•  Thekindsofperformancesthatcontainevidenceabout

language-usecompetencesConstruct-centeredperspecAve•  WhatperformancesinwhatsituaAonstelluswhatabout

studentabiliAes?

22

TheConstructoftheOPICommunicaAveCompetence

23

LanguageAbility(Bachman/Palmer1996)

•  OrganizaAonalability–  GrammaAcalability(sentencelevel)–  Textualability(textlevel)

•  PragmaAcability–  FuncAonalability

•  IdeaAonal,instrumental,heurisAc,imaginaAve–  SociolinguisAcability

•  ConvenAons,register,culturalreferences

24

LanguageAbilityinOPIs

•  FuncAonalability–  Instrumental(negoAaAngdailyinteracAons)–  IdeaAonal(describingandreporAng)–  ImaginaAve(narraAngandhypothesizing)–  HeurisAc(explainingandarguingapoint)

•  Textualability–  Word-andsentence-lengthuierances–  Connectedtextsofvariouscomplexity

•  SociolinguisAcability–  Register,culturalcompetence

25

DevelopmentalHierarchies(ILR/ACTFL)

•  FuncAonalability–  AskingforandprovidingsimplepersonalinformaAon–  DescribingandreporAng–  NarraAngandexplaining–  Arguingandhypothesizing

•  SociolinguisAcability–  None–  Oneregister–  Twoormoreregisters–  HighlevelofacculturaAon(controlofconvenAons)

26

DevelopmentalHierarchies(ILR/ACTFL)

•  OrganizaAonalability•  UseoforganizaAonaltypes(texttypes)

–  Stringsofwords–  Sentence-lengthuierances–  Stringsofsentences–  Connectedspeechofvariouslengths(cohesion)

27

DevelopmentoforganizaAonalabilityI

•  GillianBrown(1994):Modesofunderstanding–  HierarchiesofcogniAveload

•  IdenAfying:words•  Proceduralunderstanding:onesentenceataAme

–  Fullysupportedbytheexternalworld;ParAalunderstandingissufficient;Immediatefeedback

•  NarraAveunderstanding:texts•  Understandingofargument:complextexts

28

DevelopmentoforganizaAonalabilityII

•  ManfredPienemann(1998):Processabilitytheory–  GrammaAcalmemory:featureunificaAon

•  Lemmaaccess:singlewords,nofeatures Words–  LisAngsinglewordsorexpressions;nosenseforsyntaxor

morphology

•  Categoryprocedure•  Phrasalprocedure•  Sentenceprocedure Sentences•  Subordinateclauseprocedure Texts

29

TheBiggerPictureTheAssessmentUseArgument

30

Performance

Task

Consequences

Decisions

InterpretaAons

AssessmentRecords(Scores,Descriptors)

TestDevelop

men

t

InterpretaAo

nandUse

ClaimTeresawilldowellinherfinalexam.

RebuialTeresahastest

anxiety.

WarrantGoodstudentsdowellinfinalexams.

BackingStaAsAcalrelaAonshipbetweenGPAandfinal

exams.Data

Teresaisagoodstudent.

RefutaAonTeresahasn‘thadtestanxietyinanyfinals.

Onaccountof

Because

Unless

ThereforeExcept

ArgumentApproach

Warrants•  Explicit•  Generalizable•  ProvideiniAalsupportthattheconnecAonbetweenthedata

andtheclaimisappropriateandlegiAmate

Backing•  Providestheevidenceforthewarrant•  Laws,rules,principles,facts•  WidelyacceptedassumpAonsorsharedexperience

33

Claim4

Claim3

Claim2

Claim1

Data

Data

Data

Data

WarrantsRebuials

WarrantsRebuials

WarrantsRebuials

WarrantsRebuials

Consequences:Beneficial

Decisions:ValuessensiAve,Equitable

InterpretaAons:Meaningful,Generalizable,ImparAal,Relevant,Sufficient

AssessmentRecords:Consistent

Performance

Task

Example:InterpretaAons•  Generalwarrant:Theassessmentrecordcanbeinterpreted

meaningfully.•  Specificwarrant:TheraAngreflectswhattheexamineeisable

andnotabletodolinguisAcally.•  Backing:Theconstructoftheassessment,i.e.proficiency,is

describedincomprehensivecan-dostatements.•  Rebuial:TheconstructwasdevelopedexperienAallyandisnot

theory-based.•  RepudiaAon:TheconstructalignswellwithcurrentSLA

theories.

35

Performance

Task

Consequences

Decisions

InterpretaAons

AssessmentRecords(Scores,Descriptors)

TestDevelop

men

t

Impact

Reliability

Validity

TheEvoluAonofthe(ACTFL)OPI

HighReliability/QualityAssurance•  SubstanAveanddetailedTesAngProtocol•  SubstanAveanddetailedRaAngProtocol•  ExtensivetestertrainingandcerAficaAonprocedures•  Extensiveongoingtesterandraternorming•  BlinddoubleraAngsandarbitraAon•  Quality-controlledoperaAonaltesAngandraAng(LTI)

37

What’sMissing?

•  …theevaluaAonoflanguageuseforexpressingmeaningwithinawell-definedcommunicaAvecontextandaudience,foraclearpurpose…(Norris,2014)

•  ...meaningfulintegraAonofteaching,learning,andassessment...(Wiggins,1998);...posiAvewashbackofassessmentpracAcesoninstrucAon...(Mislevyetal.,2002)

38

TaskAnalysisDomainAnalysisInterpretaAon

CurricularTie-inConsequences

SomeBestPracAceExamples

•  RevisedFSISpeakingTest(2014)–  TaskAnalysis

•  OPI+(NSW)(2016) MarlaFedere(2016)–  TaskAnalysis–  CurricularTie-in

39

KeyTake-Aways•  Assessmentsneedtobeboth,task-basedandconstruct-based

toyieldevidenceaboutanexaminee’sabiliAes•  Currentvalidity/validaAontheoriesputindoubtthenoAonofa

generalproficiencytest•  Currentvalidity/validaAontheoriescallfor:

–  Domain-specificproficiencytests•  Evenforgenerallanguageprogramsinschools/universiAes

–  SubstanAalcurricularAe-in•  Thefutureofproficiencyassessmentmaybetheassessmentof

Proficiency+–  Proficiencyforamission/jobplusmoregeneraldomains

40

Bibliography•  Bachman,L.F.,&Palmer,A.S.(1996).LanguageTes*nginPrac*ce:Designingand

DevelopingUsefulLanguageTests.Oxford:OxfordUniversityPress.•  Bachman,L.F.&Palmer,A.S.(2010).LanguageAssessmentinPrac*ce:Developing

LanguageAssessmentsandJus*fyingtheirUseintheRealWorld.Oxford:OxfordUniversityPress.

•  Brindley,G.(1994).Task-centredassessmentinlanguagelearning:Thepromiseandthechallenge.InBird,N.,Falvey,P.,Tsui,A.,Allison,D.&McNeill,A.,eds.,LanguageandLearning:PapersPresentedattheAnnualInterna*onalLanguageinEduca*onConference.HongKong:HongKongEducaAonDepartment,73–94.

•  Brown,G.(1994).Modesofunderstanding.InG.Brown,K.Malmkjær,A.Pollii&J.Williams,eds.,LanguageandUnderstanding.Oxford:OxfordUniversityPress,9-20.

•  Chalhoub-Deville,M.(2001).Task-basedassessments:CharacterisAcsandvalidityevidence.InBygate,M.,Skehan,P.andSwain,M.,eds.,ResearchingPedagogicTasks:SecondLanguageLearning,TeachingandTesAng.NewYork:PearsonEducaAon,210–228.

•  Federe,M.(2016,September).AuthenAcity:FromTheorytoReality.InvitedPaper.2016LEARNTaskBasedAssessment.UniversiAesatShadyGrove,Rockville,MD.

•  Long,M.H.andNorris,J.M.(2000).Task-basedlanguageteachingandassessment.InByram,M.,ed.,EncyclopediaofLanguageTeaching.London:Routledge,597–603.

41

Bibliography•  McNamara,T.(1996).MeasuringSecondLanguagePerformance.NewYork:Longman.•  Mislevy,R.J.,Steinberg,L.S.,&Almond,R.G.(2002).Designandanalysisintask-based

languageassessment.Languagetes*ng,19,477-496.•  Norris,J.M.(2014,April).Howdoweassesstask-basedperformance?InvitedLARC/

CALPERtesAngandassessmentwebinar.Online:hips://larc.sdsu.edu/testassesswebinar/jnorris/Norris_Task-BasedAssessment_PPT.pdf

•  Pienemann,Manfred(1998).LanguageProcessingandSecondLanguageDevelopment:ProcessabilityTheory.Amsterdam:JohnBenjamins.

•  Skehan,P.(1998).ACogni*veApproachtoLanguageLearning.Oxford:OxfordUniversityPress.

•  Toulmin,S.(1958).TheUsesofArgument.Cambridge:CambridgeUniversityPress.•  Wiggins,G.(1990).TheCaseforAuthen*cAssessment.ERICDigest.Online:hip://

files.eric.ed.gov/fulltext/ED328611.pdf.

42