IALS Measurement Validity, Program Impact IALS Measurement Validity, Program Impact and Logic Models for Policy Developmentand Logic Models for Policy Development

Stephen RederStephen RederPortland State UniversityPortland State University

I

IALS, Its Meaning and Impact for Policy and Practice

The Centre for Literacy Fall InstituteBanff, Alberta – October 25, 2011

The Longitudinal Study of Adult Learning The Longitudinal Study of Adult Learning (LSAL)(LSAL)

funded by

U.S. Department of Education

and

National Institute for Literacy

Portland State University

Periods of Participation in Basic Skills Periods of Participation in Basic Skills Programs Since Leaving SchoolPrograms Since Leaving School

None, 40%

One, 21%

Two, 17%

Three or four, 13%

5 or more, 9%

Self-Study in the LSAL

Defined as “studying on your own to improve Defined as “studying on your own to improve your reading, writing or math skills or prepare for your reading, writing or math skills or prepare for the GED”the GED”

Probes distinguished such self-study from Probes distinguished such self-study from activity conducted in school or as part of a basic activity conducted in school or as part of a basic skills or GED classskills or GED class

In-depth qualitative interviews confirmed the In-depth qualitative interviews confirmed the validity of these self-reportsvalidity of these self-reports

Self-Study by Literacy ProficiencySelf-Study by Literacy Proficiency(wave 1)(wave 1)

05

1015202530354045

Level 1, 0-225 Level 2, 226-275 Level 3, 276-325 Level 4&5, 326-

500

PR

op

ort

ion

self

-stu

dy

Self-Study: Skills “Practiced”Self-Study: Skills “Practiced”(Wave 3)(Wave 3)

0

5

10

15

20

25

30

35

40

45

50

% P

ract

ice

Vocab, spelling

Punctuation, grammar

Business writing

Creative writing

Reading compr.

Reading speed

Arithmetic

Algebra, geometry

Advanced math

Self-Study: Materials UsedSelf-Study: Materials Used(Wave 3)(Wave 3)

0

10

20

30

40

50

60

% U

sin

g

Vocab, spelling,writing workbooks

Math workbooks

GED workbooks

Computer

Educ TV/video

Other

Audio cassettes

By Wave 5, computer use for self-study rose to 51%

44%

20%20%

16%

Self-Study and ParticipationSelf-Study and Participation

(between leaving school and Wave 4)

self-study & adult ed

self-study only

adult ed only

neither

64% have self-studied

60% have taken adult ed

Self-Reported Change in Literacy Self-Reported Change in Literacy Practices by Participation & Self-StudyPractices by Participation & Self-Study

Percent GED Attainment by Percent GED Attainment by Participation & Self-StudyParticipation & Self-Study

0%

5%

10%

15%

20%

25%

Neither Program Only Self-Study Only Both

Some ImplicationsSome Implications

We need a broader construct of participation We need a broader construct of participation thatthat Includes programs, self-study & other learning Includes programs, self-study & other learning

activitiesactivities Sees self-study in a continuum of provision Sees self-study in a continuum of provision

strategies along with various types of classroom strategies along with various types of classroom and tutoring programsand tutoring programs

Conceives of learners as actively deploying Conceives of learners as actively deploying resources resources as well as programs delivering as well as programs delivering servicesservices

Some Implications (con’t)Some Implications (con’t)

We need broader programmatic structures We need broader programmatic structures that support & coordinate different modes that support & coordinate different modes of participationof participation

Technologies may be particularly helpful Technologies may be particularly helpful here but need to provide more than just here but need to provide more than just direct instruction (e.g., support goal setting direct instruction (e.g., support goal setting & plan development, counselling, etc.)& plan development, counselling, etc.)

Modeling Lifespan Literacy DevelopmentModeling Lifespan Literacy Development

Program participation and self-study have positive, short-term Program participation and self-study have positive, short-term effects on engagement in effects on engagement in literacy practicesliteracy practices

Engagement in literacy practices over time has cumulative Engagement in literacy practices over time has cumulative effects on the growth of effects on the growth of literacyliteracy proficiencyproficiency

This leads to a broader concept of participation that involes This leads to a broader concept of participation that involes both both programsprograms and and self-studyself-study

Self-study bridges between periods of program participation Self-study bridges between periods of program participation and helps broaden and helps broaden persistence of attendance persistence of attendance into into persistence persistence of learningof learning

Programs as Practice-BuildingPrograms as Practice-BuildingContextsContexts

Programs that build engagement in Programs that build engagement in authentic literacy practices may lead to authentic literacy practices may lead to long-term proficiency growth that is not long-term proficiency growth that is not evident in short-term pre-post testingevident in short-term pre-post testing

Programs as Facilitated LearningPrograms as Facilitated LearningContextsContexts

Programs often blend time in classroom instruction with Programs often blend time in classroom instruction with time in learning centerstime in learning centers

There may be a continuum of facilitated learning with There may be a continuum of facilitated learning with classroom activities on one end and self-study on the classroom activities on one end and self-study on the other end, with facilitated use of learning centers and other end, with facilitated use of learning centers and tutoring in-betweentutoring in-between

Redefining Program, Participation Redefining Program, Participation and Persistence and Persistence

Focus on engagement in literacy practices rather than on persistence in programs

Broaden definition of program to include facilitated engagement in literacy practices

Technology has broader role here than just delivering distance learning

Effects of InstructionEffects of Instruction

Strongest evidence of program impact: Controlled Strongest evidence of program impact: Controlled comparison of participants’ and non-participants’ gainscomparison of participants’ and non-participants’ gains

Next strongest: Covariation of hours of instruction with Next strongest: Covariation of hours of instruction with observed gainsobserved gains

My analysis of CASAS proficiency gains in several My analysis of CASAS proficiency gains in several states’ NRS data indicates the gains attributable to states’ NRS data indicates the gains attributable to instructional hours are greatly attenuated (if present at instructional hours are greatly attenuated (if present at all) when test experience is controlledall) when test experience is controlled

Program Participation..Program Participation..

..is often fragmented by life circumstances..is often fragmented by life circumstances

..is better thought of as a busy intersection ..is better thought of as a busy intersection among life and learning trajectories rather than among life and learning trajectories rather than as a parking lotas a parking lot..should support students’ trajectories not just ..should support students’ trajectories not just their activities in the intersectionstheir activities in the intersections..is expanded by a ..is expanded by a learning planlearning plan that learners that learners can follow and that services can be wrapped can follow and that services can be wrapped aroundaround

Learning Support SystemsLearning Support SystemsAdults need Adults need learning support systemslearning support systems that provide portable, that provide portable, personalized personalized learning planslearning plans they can follow to reach their goals they can follow to reach their goals

Learning support systems include a focus on building Learning support systems include a focus on building engagement in literacy practicesengagement in literacy practices

Learning support systems provide flexible blends of online and Learning support systems provide flexible blends of online and offline local resources to support learningoffline local resources to support learning

The technology supports learners directly as they follow a The technology supports learners directly as they follow a learning plan and indirectly by coordinating the work of those who learning plan and indirectly by coordinating the work of those who support themsupport them

Communities design and implement local learning support Communities design and implement local learning support systems, utilizing the technology to facilitate collaboration and systems, utilizing the technology to facilitate collaboration and information-sharing among the various organizations working with information-sharing among the various organizations working with learnerslearners

A New Logic Model A New Logic Model for Program Impactfor Program Impact

Practice-engagement theoryPractice-engagement theory

Busy intersection vs. parking lotBusy intersection vs. parking lot

Learning support systemLearning support system

Research RecommendationsResearch RecommendationsImprove ways of measuring engagement in literacy Improve ways of measuring engagement in literacy practicespracticesConduct interventions that stimulate increased practice-Conduct interventions that stimulate increased practice-engagementengagement Need long-term longitudinal studies with multiple Need long-term longitudinal studies with multiple

repeated literacy outcome measuresrepeated literacy outcome measures Note that National Reading Panel found little evidence Note that National Reading Panel found little evidence

indicating that programs which aim to increase indicating that programs which aim to increase independent reading in schoolchildren lead to independent reading in schoolchildren lead to increased proficiencyincreased proficiency

Examine literacy changes (including proficiency loss) in Examine literacy changes (including proficiency loss) in older adults and interventions to maintain literacy older adults and interventions to maintain literacy capabilities in older adultscapabilities in older adults

Contact informationContact information

Steve RederSteve RederProfessor, Chair – Department of Applied LinguisticsProfessor, Chair – Department of Applied LinguisticsPortland State UniversityPortland State University(503) 725-3999(503) 725-3999reders@pdx.edureders@pdx.edu

For more information and references to the research covered in the Powerpoint For more information and references to the research covered in the Powerpoint presentation, please see:presentation, please see:

Longitudinal Study of Adult Learning Longitudinal Study of Adult Learning www.lsal.pdx.eduwww.lsal.pdx.edu

Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Some Thoughts on IALS Measurement Validity, Program Impact, and Logic Models for Policy Development Policy Development www.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdfwww.centreforliteracy.qc.ca/sites/default/files/Rederthinkpiece.pdf