Strengths & Weaknesses of IALS Strengths & Weaknesses of IALS as an Assessment Framework:as an Assessment Framework:

A Longitudinal PerspectiveA Longitudinal Perspective

Stephen RederStephen RederPortland State UniversityPortland State University

I

IALS, Its Meaning and Impact for Policy and Practice

The Centre for Literacy Fall Institute

Banff, Alberta - October 24, 2011

Where I’m Coming FromWhere I’m Coming From

Literacy researcher interested in how the IALS Literacy researcher interested in how the IALS Framework affects policies and programs in Framework affects policies and programs in adult basic educationadult basic education

Some of my researcher and practitioner Some of my researcher and practitioner colleagues believe that IALS is not relevant to colleagues believe that IALS is not relevant to their programs and studentstheir programs and students

Although I believe there Although I believe there isis a disconnect between a disconnect between the IALS Framework and the needs of adult the IALS Framework and the needs of adult education, we can fix iteducation, we can fix it! !

Strengths of IALS FrameworkStrengths of IALS Framework (for adult education)(for adult education)

Contextualizes literacy in terms of important Contextualizes literacy in terms of important social & economic indicators social & economic indicators Provides compelling Provides compelling snapshotssnapshots of adult literacy of adult literacy at given points in timeat given points in timeBuilds technically sound assessment Builds technically sound assessment instruments and proceduresinstruments and proceduresPays careful attention to maintaining Pays careful attention to maintaining comparability of large-scale assessments over comparability of large-scale assessments over time and spacetime and space

Weaknesses of IALS FrameworkWeaknesses of IALS Framework(for adult education)(for adult education)

Supports increased literacy proficiency as a Supports increased literacy proficiency as a policy goal but…policy goal but…

……does not provide useful tools for assessing does not provide useful tools for assessing program impact on literacyprogram impact on literacy

……is of limited use for short-term accountability is of limited use for short-term accountability regimes & program improvement effortsregimes & program improvement efforts

The Longitudinal Study of Adult Learning The Longitudinal Study of Adult Learning (LSAL)(LSAL)

funded by

U.S. Department of Education

and

National Institute for Literacy

Portland State University

Longitudinal Study of Adult LearningLongitudinal Study of Adult Learning

Decade-long panel study of Portland high school dropouts, Decade-long panel study of Portland high school dropouts, age 18-44 at the beginning of the studyage 18-44 at the beginning of the studyRepresentative sample of ~1,000 drawn from local rather than Representative sample of ~1,000 drawn from local rather than national population of dropoutsnational population of dropoutsPeriodic in-home interviews and literacy assessments and Periodic in-home interviews and literacy assessments and SSN-linked administrative data (with individuals’ permission)SSN-linked administrative data (with individuals’ permission)Examines program participation and other learning activities, Examines program participation and other learning activities, social and economic changes, and changes in literacy skills, social and economic changes, and changes in literacy skills, literacy practices & technology use over timeliteracy practices & technology use over timeSmaller-scale qualitative componentsSmaller-scale qualitative components

LSAL TimelineLSAL Timeline wavewave 1 1 1998 – 19991998 – 1999

wave 2wave 2 1999 – 20001999 – 2000

wave 3wave 3 2000 – 20012000 – 2001

wave 4wave 4 2002 – 20032002 – 2003

wave 5wave 5 2004 – 20052004 – 2005

wavewave 66 2006 – 20072006 – 2007

Literacy Measures in LSALLiteracy Measures in LSAL

Measures of ProficiencyMeasures of ProficiencyRepeated measures of TALS Document LiteracyRepeated measures of TALS Document LiteracySSN-matched GED test scoresSSN-matched GED test scores

Measures of PracticesMeasures of PracticesRepeated measures of literacy practicesRepeated measures of literacy practicesRepeated self-reported changes from wave-to-wave in Repeated self-reported changes from wave-to-wave in reading, writing and mathreading, writing and math

Measures of Component SkillsMeasures of Component SkillsOral vocabularyOral vocabulary Word recognition Word recognition Fluency Fluency Holistic writingHolistic writing

Some LSAL DemographicsSome LSAL Demographics

Average age is 28 (at Wave 1)Average age is 28 (at Wave 1)

50 % female and male50 % female and male

35 % minority35 % minority

9 % foreign-born9 % foreign-born

34 % live in poverty34 % live in poverty

29 % report a learning disability29 % report a learning disability

34 % took special education34 % took special education

Broad range of assessed basic skillsBroad range of assessed basic skills

Up Close and PersonalUp Close and Personal

Looking at Things Seen in IALS Looking at Things Seen in IALS in the LSAL Panel Studyin the LSAL Panel Study

• AgeingAgeing

• Proficiency & economic indicatorsProficiency & economic indicators

NALS Document Literacy by AgeNALS Document Literacy by Age(1992 national data corrected for education & disabilities)(1992 national data corrected for education & disabilities)

200

225

250

275

300

16-24 25-34 35-44 45-54 55-64 65+

Age

Pro

ficie

ncy

UK Document Literacy by AgeUK Document Literacy by Age(IALS data, corrected for education)(IALS data, corrected for education)

200

225

250

275

300

16-25 26-35 36-45 46-55 56-65

Age

Pro

ficie

ncy

Employment and Earnings DataEmployment and Earnings Data

Using individuals’ quarterly administrative Using individuals’ quarterly administrative employment & wage data reported by employment & wage data reported by employers for unemployment insuranceemployers for unemployment insurance

Covers more than 90% of labor activityCovers more than 90% of labor activity

Examining 1997-2003 (7 years / 28 qrtrs)Examining 1997-2003 (7 years / 28 qrtrs)

Major recession in Portland 2001-2003Major recession in Portland 2001-2003

Wages by Proficiency, 1997-2003Wages by Proficiency, 1997-2003

Earnings by Proficiency GrowthEarnings by Proficiency Growth

So What’s The Problem?So What’s The Problem?

Longitudinal data seem consistent with IALS snapshots Longitudinal data seem consistent with IALS snapshots and allow us to look more closely at the individual level of and allow us to look more closely at the individual level of the large-scale phenomenathe large-scale phenomena

Controlled comparisons of adults who participate in basic Controlled comparisons of adults who participate in basic skills programs with similar adults who do not participate skills programs with similar adults who do not participate show there is no significant impact of programs on show there is no significant impact of programs on proficiency one or two years following participationproficiency one or two years following participation

Does this mean programs are not effective? NO! But it Does this mean programs are not effective? NO! But it illustrates the practical disconnect of IALS from programsillustrates the practical disconnect of IALS from programs

0% 20% 40% 60% 80% 100%

read instructions

do financial math

use ATM

read street maps

read entertain,tv guides

read news section

read fiction

read non-fiction

read mags, comic bks

write diary,story,poem

write note, letter, email

read manuals, ref mats

do math

use comp at home

never rarely < weekly weekly > weekly daily

Repeated Measures of Engagement in Literacy Practices

Program ImpactProgram Impact

Literacy proficiency growth over relatively short periods of time is Literacy proficiency growth over relatively short periods of time is not not affected by program participationaffected by program participation

Accountability data show proficiency gains but do not contrast Accountability data show proficiency gains but do not contrast participants’ gains with those of comparable non-participants; LSAL participants’ gains with those of comparable non-participants; LSAL indicates their gains are equivalentindicates their gains are equivalent

Growth in literacy practices over short time periods Growth in literacy practices over short time periods is, is, on the other on the other handhand,, directly impacted by programs directly impacted by programs

These findings are consistent with cross-sectional research (e.g., These findings are consistent with cross-sectional research (e.g., Smith & Sheehan-Holt) and classroom studies (e.g., Purcell-Gates, Smith & Sheehan-Holt) and classroom studies (e.g., Purcell-Gates, Jacobson & Degener)Jacobson & Degener)

DisconnectDisconnect: Proficiency measures do not reflect the impact that : Proficiency measures do not reflect the impact that programs have in the short-term nor can support evidence-based programs have in the short-term nor can support evidence-based program improvement processesprogram improvement processes

Repairing the Disconnect:Repairing the Disconnect:Practice-Engagement TheoryPractice-Engagement Theory

Predicts that engagement in literacy practices over time leads Predicts that engagement in literacy practices over time leads to proficiency growthto proficiency growth

Proposed by Reder from ethnographic work and by Smith & Proposed by Reder from ethnographic work and by Smith & colleagues in analyses of NALScolleagues in analyses of NALS

LSAL confirms through longitudinal modeling:LSAL confirms through longitudinal modeling:

Adults with the same proficiency level and a higher level of Adults with the same proficiency level and a higher level of engagement in literacy practices at one point in time have engagement in literacy practices at one point in time have significantly higher levels of literacy proficiency 6 years latersignificantly higher levels of literacy proficiency 6 years later

What Should We Do?What Should We Do?

Stop relying on only IALS (or other) proficiency Stop relying on only IALS (or other) proficiency measures for measures for short-termshort-term accountability & accountability & program improvementprogram improvementUse literacy practices measures along with Use literacy practices measures along with proficiency measures in large-scale proficiency measures in large-scale assessments and for assessments and for short-termshort-term program program accountability and improvementaccountability and improvementUse proficiency measures as Use proficiency measures as long-termlong-term (5-6 (5-6 year) follow-up outcome measuresyear) follow-up outcome measures

Contact informationContact information

Steve RederSteve RederProfessor, Chair – Department of Applied LinguisticsProfessor, Chair – Department of Applied LinguisticsPortland State UniversityPortland State University(503) 725-3999(503) 725-3999reders@pdx.edureders@pdx.edu

For more information and references to the research covered in the Powerpoint For more information and references to the research covered in the Powerpoint presentation, please see:presentation, please see:

Longitudinal Study of Adult Learning Longitudinal Study of Adult Learning www.lsal.pdx.eduwww.lsal.pdx.edu

Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Policy Development Policy Development www.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdfwww.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdf