TheACTFLListeningandReadingBenchmarkingStudy

ErwinTschirner,UniversityofLeipzigElviraSwender,ACTFL

ILRPlenary

March18,2016NaLonalForeignLanguageCenter

Overview

• WhytheInterpreLveModes?•  ACTFLReadingandListeningProficiencyTests•  ACTFLListeningandReadingBenchmarkStudy•  Findings•  ImplicaLonsforCurriculumandInstrucLon•  QuesLons

2

TimespentcommunicaLngingeneral

Listening 45% Speaking 30% Reading 16% WriLng 9%

(Wilt1950)

3

TimespentcommunicaLngonthejob

Listening 38% Speaking 33% Reading 19% WriLng 9.5%

Evans,G./Pastor,E.(1972).CommunicaLon12½.Fieldsurveyoflanguageskillsandrealjobneeds.Stockholm.

4

Listening

•  ProvidesafoundaLonforalllanguageandcogniLvedevelopment•  Playsalife-longroleintheprocessesoflearningandcommunicaLonessenLaltoproducLveparLcipaLoninlife•  Drivesforeignlanguagelearning(inputasengine)

5

Reading

•  Twiceasfastaslistening(300vs.150words/minute)•  MoreinformaLonandlearninginlessLme

•  EffecLveatlowerlevelsofproficiency•  Requireslowerlanguageabilitythanlistening

•  Buildsvocabularyfasterthananyotherskill•  Buildstherightkindofvocabulary

•  Relatedtohigher-orderthinkingskillsandsubjectareas

•  BuildsgrammaLcalcompetence(grammarinuse)•  Drivesforeignlanguagelearning(inputasengine)

6

7

ReadingandListening•  Firstrevisionsincetheoriginal

publicaLon•  Progressionofabilityto

comprehendwridenandspokentexts

•  DivisionofAdvancedintoHigh,Mid,andLowsublevels

•  DescribewhatreadersandlistenersCANDOatonemajorlevelandCANNOTdoatthenexthighermajorlevel

•  OnlinepublicaLon•  AddiLonofsamplewriden

andspokentextsasexamplesofthemajorlevels

ACTFLProficiencyGuidelines2012

QuesLonsandChallenges:Reading

•  Defininglevelsofproficiency•  Kindsoftextsstudentscanreadwithwhatkindofunderstandingatwhatlevel•  Theroleofreading

•  Influenceofreadingproficiencyon-andhowitisinfluencedby-listening,speaking,andwriLngproficiency

•  Designingcurriculatopromotethedevelopmentofreadingproficiency

8

QuesLonsandChallenges:Listening

•  Defininglevelsofproficiency•  Kindsofauralandvisual-auraltextsstudentscanunderstandwithwhatkindofunderstandingatwhatlevel•  Theroleoflistening

•  Influenceoflisteningproficiencyon-andhowitisinfluencedby-reading,speaking,andwriLngproficiency

•  Designingcurriculatopromotethedevelopmentoflisteningproficiency

9

AssessmentforSejngProficiencyTargets

•  Assessmentprovidesfeedbackforstudents,instructors,anddepartments•  AssessmenthelpstosetrealisLcgoals•  AssessmentoutcomeshelptoaligncurriculaandinstrucLonwithprogramgoals

10

©ACTFL2015

ACTFLReadingProficiencyTest(RPT)

•  Internet-basedtesttakenonacomputerinaproctoredsituaLon•  EvaluatesreadingproficiencyaccordingtotheACTFLProficiencyGuidelines2012–Reading•  Machinescored

12

ReadingHighlights

•  Real-lifepassages,tasks,andprocedures•  QuesLonscanbeseenbeforereading:Providesreaderintent•  TesttakerscangobackandforthbetweenquesLonsforthesametext:Allowsflexibility•  10passagesattwosublevels:ProvidestesttakerswithenoughopportuniLestoobtainareliableraLng•  4differentquesLonstypes(global,detail,selecLve,inference):Coversabroadrangeoflisteninggoals

13

Items

•  EachtexthasthreequesLonswithfourmulLplechoiceopLonsaboutthetext.•  QuesLonsaskformainideas,supporLngdetailandtheinferencesandconnecLonsthecandidatecanmakefromthecontentandorganizaLonofthetext.•  QuesLonsareinthetargetlanguage.

14

ACTFLListeningProficiencyTest(LPT)

•  Internet-basedtesttakenonacomputerinaproctoredsituaLon•  EvaluateslisteningproficiencyaccordingtotheACTFLProficiencyGuidelines2012–Listening•  Machinescored

15

ListeningHighlights

•  Real-lifepassages,tasks,andprocedures•  QuesLonsareaskedbeforelistening:Provideslistenerintent•  Testtakersareabletotakenotes:Preventshavingtorelysolelyonmemory•  Passagesareheardonce:AppropriateforinterpreLvelistening•  10passagesattwosublevels:ProvidestesttakerswithenoughopportuniLestoobtainareliableraLng•  4differentquesLonstypes(global,detail,selecLve,inference):Coversabroadrangeoflisteninggoals

16

Items

•  EachpassagehasthreequesLonswithfourmulLplechoiceopLonsaboutthetext.•  QuesLonsaskformainideas,supporLngdetail,inferencesandconnecLonsthecandidatecanmakefromthecontent,tone,paceandorganizaLonofthediscoursetheyhear.•  QuesLonsareinthetargetlanguageforalphabetlanguages•  Fornon-alphabetlanguages,quesLonsandopLonsmaybeinthetargetlanguageorinEnglishattheIntermediatelevel•  AttheAdvancedandSuperiorlevels,theyareinthetargetlanguage

17

TheACTFLListeningandReadingProficiencyBenchmarkingStudy

18

Acknowledgments•  LanguageTesLngInternaLonal(LTI)•  TheLanguageFlagship•  UniversityofLeipzig•  InsLtuteforTestResearchandTestDevelopment(ITT)•  AllparLcipaLngschools,especially:

•  MichiganStateUniversity•  UniversityofMinnesota•  UniversityofUtah

19

TheACTFL/ILRProficiencyFrameworkCanDoandWillbeAbletoDoNext

DevelopmentalPerspecLve

“floor”â

“ceiling”á

Background

•  LidledataonrealisLcproficiencylevelexpectaLonsintheinterpreLvemodesincollegeFLprograms

MainQuesLons:•  Areprofessional(advanced)levelsofproficiencyintheinterpreLvemodespossibleincollegeFLprograms?• WhatistherelaLonshipbetweenlisteningandreadinginlanguagelearning?• Whatroledoeslanguagedistanceandorthographicdepthplay?

21

ParLcipants:22CollegesandUniversiLes

•  UofMinnesota,UofUtah,MichiganStateU•  UCBerkeley,UofSouthernCalifornia,YaleU,UofDelaware,UofMaryland-CollegePark•  UofPidsburgh,UofWisconsin-EauClaire,SUNYPladsburgh•  MiddleburyCollege,HunterCollege,BowdoinCollege,LorasCollege•  SanDiegoStateU,GeorgiaSouthernU,LeeU•  NorthCarolinaStateU,EasternWashingtonU,GrandValleyStateU,OldDominionU

22

Languages

Listening ReadingSpanish ca.1700 ca.1800French ca.750 ca.800German ca.250 ca.350Russian ca.150 ca.150Italian ca.100 ca.150Portuguese ca.50 ca.50Japanese ca.100Total ca.3000 ca.3400

Results

NL=1,NM=2,NH=3,IL=4,IM=5,IH=6AL=7,AM=8,AH=9,S=10

24

25

0

1

2

3

4

5

6

7

8

Sem2(111) Sem3(129) Sem4(240) Sem5(244) Sem6(334) Year4(359)

ACTFLRa

4ng

MedianSpanishandFrenchProficiencyRa4ngs

LPTFrench

RPTFrench

LPTSpanish

RPTSpanish

ReadingSemester2(111h)N RaLngMedian RaLng85thPercenLle

Spanish 242 NH(3)

IM(5)

French 120 NH(3)

IL(4)

Italian 73 IL(4)

IM(4.9)

ReadingSemester4(240h)N RaLngMedian RaLng85thPercenLle

Spanish 338 IL(4)

IM(5)

French 215 IM(5)

IH(6)

Italian 24 IL(4)

IM(5)

German 178 IL(4)

IM(5.15)

Russian 52 NH/IL(3.5)

IM(5)

ReadingSemester6(339h)N RaLngMedian RaLng85thPercenLle

Spanish 208 AL(7)

AM(8)

French 62 AL(7)

AM(8)

German 18 IH(6)

AL(7)

Russian 20 IM(5)

AL(7)

ReadingYear4(359h)N RaLngMedian RaLng85thPercenLle

Spanish 242 AL(7)

AM(8)

French 124 AL(7)

AM(8)

German 38 IH(6)

AL(7)

Russian 45 BR AH(9.1)

SpanishReading

N RaLngMedian RaLng85thPercenLle

Year4 242 AL(7)

AM(8)

Sem6 208 AL(7)

AM(8)

Sem5 432 IH(6)

AL(7)

Sem4 338 IL(4)

IM(5)

Sem3 222 IL(4)

IM(5)

Sem2 242 NH(3)

IM(5)

FrenchReading

N RaLngMedian RaLng85thPercenLle

Year4 124 AL(7)

AM(8)

Sem6 62 AL(7)

AM(8)

Sem5 166 IH(6)

AL(7)

Sem4 215 IM(5)

IH(6)

Sem3 86 IL(4)

IM(5)

Sem2 120 NH(3)

IL(4)

ReasonableGoals:ReadingProficiency

CollegeYear ReadingProficiency

EndofSeniorYear AdvancedHigh

EndofJuniorYear AdvancedMid

EndofSophomoreYear IntermediateHigh

EndofFreshmanYear IntermediateMid

SpanishListening

N RaLngMedian RaLng85thPercenLle

400 233 AL(7.0)

AL(7)

302 187 IH(6)

AL(7)

301 392 IM(5)

AL(7)

202 317 NH(3)

IL(4)

201 209 NM(2)

IL(4)

102 235 NM(2)

IL(4)

FrenchListening

N RaLngMedian RaLng85thPercenLle

400 121 IM(5)

AL(7)

302 89 IM(5)

AL(7)

301 141 IL(4)

IH(6)

202 203 NH(3)

IL(4)

201 88 NM(2)

IL(4)

102 111 NM(2)

NH(3)

ReasonableGoals:ListeningProficiency

?

ListeningandReading

36

CorrelaLons

Language N Spearman‘srho p

French 539 .766 .01

German 179 .719 .01

Italian 62 .739 .01

Portuguese 53 .736 .01

Russian 99 .842 .01

Spanish 1403 .786 .01

All 2339 .777 .01

37

Listeningvs.Reading(Median)

SPA ITA FRE POR GER RUS

ReadingSem2 3 4 3 4 1

ListeningSem2 2 4 2 1 1

ReadingSem4 4 4 5 6 4 3.5

ListeningSem4 3 4 3 4 3 3

ReadingSem5 6 7 6 7 6

ListeningSem5 5 7 4 5 5

MeanHoursofInstrucLon

FR(M) SD GE(M) SD SP(M) SDR IL 187 71 219 45 182 68

IM 231 61 233 66 233 80IH 277 56 274 93 279 69AL 309 63 306 81 312 80AM 319 54 342 75

L IL 238 63 242 31 210 76IM 291 62 289 56 273 73IH 304 61 317 110AL 330 68 373 153 398 164AM 444 176

OrthographicDepth

Katz,L.&Frost,R.(1992).Thereadingprocessisdifferentfordifferentorthographies:Theorthographicdepthhypothesis.InFrost,R.&Katz,L.,(Eds.).Orthography,Phonology,Morphology,andMeaning,pp.67-84.Amsterdam:ElsevierNorthHollandPress.

40

ResultsandImplicaLons

•  AdvancedlevelsofreadingproficiencyappeartobeveryrealisLcgoalsforCategoryIlanguagesatgraduaLon•  Advancedinreading:Semester5:top15%;Semester6:median

•  Principledapproachestoreading(Years1-4)mayhelpstudentsachieveSuperiorlevels

•  ListeningproficiencyneedsextraadenLon–itappearstodevelopmoreslowlythanreading•  ItneedssuperextraadenLonindeeporthographylanguagessuchasFrench

•  Notmuchgaininlisteningproficiencyinthefirsttwoyears(Median=NH)

•  Evenlessgainindeeporthographylanguages•  Principledapproachestolistening(Years1-4)appeartobehighlydesirable

41

ResultsandImplicaLons2

•  Listeningproficiencylevelsappeartobesimilartooralproficiencylevels•  ForeignlanguagemajorsdonotusuallyachieveAdvancedlevelsoforalproficiencyunlesstheyspentasignificantamountofLmeabroad(1yearormore)

•  IsthefeebledevelopmentoflisteningproficiencythemainreasonfortheAdvancedbarrierinspeaking?• Doeslisteningproficiencyneedtobetaughtdifferentlyfordeeporthographylanguages?

42

PreviewofPhase2

• AddiLonallanguages• Effectoflisteningproficiencyonspeakingproficiency• Effectofvocabularysizeonreadingandlisteningproficiency

43

AddiLonalLanguages(TotalNasofMarch2016)

German 370Russian 243

Portuguese 143Italian 141Chinese 125Japanese 96Korean 12Arabic

PreviewofPhase2

Effectoflisteningproficiencyonspeakingproficiency•  ca.3,600studentswithLPTandOPIcresults(CollaboraLonwithMSU,UofUtah,UofMinnesota)

Effectofvocabularysizeonreadingproficiency•  Spanish,Russian,German,Chinese•  CorrelaLonsbetweenRPTsandVocabularySizeTests(CollaboraLonwiththeUofUtah)

45

http://www.itt-leipzig.de/static/startseite.html

47