s. white 8-04 1 the health literacy component (hlc) of the 2003 naal the health literacy component...
TRANSCRIPT
1 S. White 8-04
The Health Literacy
Component (HLC)
of the 2003 NAAL
The Health Literacy
Component (HLC)
of the 2003 NAAL
Sheida White
Project Officer
National Assessment of Adult Literacy
(NAAL)
Sheida White
Project Officer
National Assessment of Adult Literacy
(NAAL)
An Overview of the
2003 National
Assessment of Adult
Literacy (NAAL)
An Overview of the
2003 National
Assessment of Adult
Literacy (NAAL)
Sheida White
Project Officer
National Assessment of Adult Literacy
(NAAL)
Sheida White
Project Officer
National Assessment of Adult Literacy
(NAAL)
A webcast presentationSeptember 8, 2004
2 S. White 8-04
Background, key features, and major goals
Performance of literacy tasks
Skills required to perform literacy tasks
Assessment design and administration
Data analysis and reporting
Topics covered by this presentationTopics covered by this presentation
4 S. White 8-04
In 1985 and 1992, the National Center for Education Statistics (NCES) conducted adult literacy assessments that focused on real-world tasks
The 2003 National Assessment of Adult Literacy (NAAL) builds on and expands previous NCES adult literacy assessments
NAAL is the first assessment of the nation’s progress in adult literacy since the 1992 National Adult Literacy Survey (NALS)
NAAL backgroundNAAL background
5 S. White 8-04
Is based on in-person interviews of approximately 18,000 Americans age 16 and older
Provides data for the United States as a whole, individual states choosing to participate in a state-level assessment, and the nation’s prison population
Emphasizes the use of printed everyday materials (such as bills, prescriptions, newspapers) needed to function adequately in one’s environment
Provides data on background characteristics, performance of everyday literacy tasks, and skills underlying task performance
Key features of NAALKey features of NAAL
6 S. White 8-04
Describe the status of adult literacy
Report on trends between 1992 and 2003
Identify relationships between literacy and selected characteristics of adults
Provide new information about skills underlying adult literacy, including basic reading skills of the least-literate adults
Facilitate use of NAAL data by diverse audiences, including policymakers, researchers, and educators
Major goalsMajor goals
8 S. White 8-04
NAAL’s task-based literacy definitionNAAL’s task-based literacy definition
Literacy is the ability to use
printed and written infor-
mation [prose, document,
and quantitative] to function
in society, to achieve one’s
goals, and to develop one’s
knowledge and potential
S. White 8-04
10 S. White 8-04
Measurement of prose, document, and quantitative literacy
Measurement of prose, document, and quantitative literacy
NAAL’s prose literacy, document literacy, and quantitative literacy scale scores are comparable to those of the 1992 NALS
Scores are based on completion of items that are all Classified as prose, document, or quantitative Open-ended to represent everyday tasks
Placed before stimulus materials to set a purpose for reading
Most items require Short written responses
Searching the text for specific information
12 S. White 8-04
Sample prose item used in 1992Sample prose item used in 1992
Vitamin E (tocopherol)—helps protect red blood cells.May aid the circulatory system and counteract the agingprocess. Best sources: wheat germ, whole grains, eggs, pea-nuts, organ meats, margarine, vegetable oils, green leafy veg-etables.
S. White 8-04
14 S. White 8-04
Sample quantitative item used in 1992Sample quantitative item used in 1992
S. White 8-04
15 S. White 8-04
NAAL includes 28 health-related items (out of 153 items)
All participants receive some health-related items
All health-related items are also classified as prose, document, or quantitative items
The health-related items
Are included with other items when calculating the prose, document, and quantitative scale scores
Are used to calculate a separate health literacy score, based solely on health-related items
A new health literacy score based on health-related items
A new health literacy score based on health-related items
16 S. White 8-04
NAAL’s Health Literacy Component was developed in response to a request by the U.S. Department of Health and Human Services (HHS)
NAAL focuses on one key aspect of health literacy—the ability to use printed and written materials to accomplish a broad range of health-related tasks
NAAL’s health-related tasks include Clinical tasks, having to do with the health care itself Preventive tasks, having to do with healthy habits and
prevention of illness
Navigation tasks, having to do with bureaucratic demands to get to the health care
Development and focus of the NAAL Health Literacy Component
Development and focus of the NAAL Health Literacy Component
18 S. White 8-04
Purpose of the Adult Literacy Supplemental Assessment (ALSA)
Purpose of the Adult Literacy Supplemental Assessment (ALSA)
The 1992 NALS provided little information about the literacy abilities of the least-literate adults, who were not able to complete the assessment
The purpose of the new Adult Literacy Supplemental Assessment (ALSA) is to get a clearer picture of the skills of the least-literate adults
ALSA is intended only for adults who would not be capable of meaningful participation in the “main NAAL” assessment, because they lack the skills to perform even the easiest tasks with a high degree of success
These adults take ALSA instead of the main NAAL
19 S. White 8-04
Selection of ALSA participantsSelection of ALSA participants
All NAAL participants take seven easy core screening items
Interviewers score participants’ answers using the scoring guides provided on their computers
Computer selects ALSA participants using an empirically derived algorithm that predicts very low performance
The algorithm predicts which participants would get a total of five items or fewer correct (including the core items)
Five items, spread across the three NAAL scales, would not be sufficient to allow accurate estimates of performance on any of the scales
20 S. White 8-04
Important features of ALSAImportant features of ALSA
ALSA instructions and responses are given orally and can be in either English or Spanish
ALSA materials are
Contextualized (with logos, pictures, etc.)
Familiar (as measured by background questions)
Tangible (e.g., food boxes, medicine bottles, utility bills)
ALSA tasks, although simple and contextualized, still require participants to read at least parts of the words
21 S. White 8-04
The need for additional data about skill deficitsThe need for additional data about skill deficits
In addition to the 6% of adults who lacked the skills to complete the 1992 NALS, an additional 15% of adults were able to perform only the easiest NALS tasks (classified as “Level 1” tasks)
For adults with low scores, more information is needed about the nature of the skill deficits
Lack of basic (word-level) reading skills?
Lack of fluency in basic skills?
Lack of higher level literacy skills?
22 S. White 8-04
Range of skill deficits and their potential implications
Range of skill deficits and their potential implications
23 S. White 8-04
New data on the skills of all adultsNew data on the skills of all adults
The Fluency Addition to NAAL (FAN) provides new data on the skills of all adults
After completing either the main NAAL or ALSA, all participants take FAN, which
Consists of timed oral reading tasks
Describes the oral reading skills of adults (e.g., words read correctly per minute)
Provides a measure of basic (word-level) reading skills
FAN scores can be compared with main NAAL scores at various performance levels, ALSA scores, and health literacy scores
24 S. White 8-04
FAN oral reading materials and associated measures
FAN oral reading materials and associated measures
Pseudoword lists, consisting of possible but nonoccurring English forms (e.g., “wike”), provide a measure of adults’ ability to “decode” (or identify the sounds of) words with which they are not familiar
Word lists, consisting of English words arranged in increasing order of difficulty, provide a measure of adults’ ability to recognize familiar words (often referred to as “sight words”) as well as to decode
Text passages, consisting of 150–200 words each, provide a measure of adults’ ability to read words in connected texts
25 S. White 8-04
Initial measures and scoring challenges for FANInitial measures and scoring challenges for FAN
Initial FAN target measures are
Total words read aloud—whether correctly or not—per unit of time*
Words read correctly per unit of time*
Words read correctly as a percentage of total words read
Challenges in scoring FAN data include ensuring
That correctness can be measured reliably
That speakers of nonstandard varieties of English are not unfairly penalized
*Unit of time: 60 seconds for text passages or 20 seconds for lists
26 S. White 8-04
Key similarities and differences between FAN and ALSA
Key similarities and differences between FAN and ALSA
Similarities include the following:
Both measure basic reading skills
Both have oral instructions in either English or Spanish
Both require reading words and connected text
Differences include the following:
Only ALSA permits the use of compensatory strategies, while FAN does not provide any nonlinguistic clues (such as pictures)
ALSA connected texts are fewer, easier, and shorter (12 sentences each in ALSA, compared with 150200 words each in FAN)
27 S. White 8-04
The functionality of word-level skills and higher level literacy skills
The functionality of word-level skills and higher level literacy skills
NAAL defines literacy as the ability to use printed and written information
As defined by NAAL, all literacy is functional; therefore, NAAL does not differentiate between “literacy” and “functional literacy”
Certain skills, including word-level reading skills, are needed to successfully perform NAAL tasks
The ability to read words is a functional skill, but not the only one
In addition to word-level skills, NAAL recognizes six types of literacy skills
28 S. White 8-04
The six types of literacy skillsThe six types of literacy skills
NAAL literacy skillsNAAL literacy skills
Language comprehension
skills
Understanding the structure and meaning of sentences
Text search skills
Searching textefficiently
Inferential skills
Drawing appropriate text-based inferences
Application skills
Identifying the calculations required to solve quantitative problems
Computation identification
skills
Computation performance
skills
Performing any required calculations
Using newly searched, inferred, or computed information to accomplish a variety of goals
29 S. White 8-04
Analysis of main NAAL data in terms of underlying skills
Analysis of main NAAL data in terms of underlying skills
Recently developed multidimensional analysis methods will be applied to 2003 main NAAL data in order to
Characterize the proficiency of adults in terms of the NAAL literacy skills
Characterize effects of task demands and written materials on the difficulty of literacy tasks
31 S. White 8-04
Selection of the national sample for NAAL Selection of the national sample for NAAL
32 S. White 8-04
Key features of the NAALKey features of the NAAL sampleKey features of the NAALKey features of the NAAL sample
Adults age 16 and over living primarily in households
Nationally representative sample of 10,006 adults, augmented by
Samples from the six states that participated at the state level (an additional 7,166 adults)
Prison sample of inmates (about 1,000 more adults)
Incentive payment of $30 to increase representativeness of the sample and response rate
Oversampling of Blacks and Hispanics
33 S. White 8-04
Matrix samplingMatrix sampling
The main NAAL has a total of 153 items (including the 7 core screening items that are given to all participants and used to select ALSA participants)
Matrix sampling—administering only a portion of the non-core items to each respondent—ensures broad coverage while limiting respondent burden
Each respondent takes 1 of 26 booklets
Each booklet includes 3 of 13 blocks of items (for about 40 items per booklet, including the core items)
Each block is spiraled in the 26 booklets (i.e., paired with every other block)
34 S. White 8-04
Allowing comparisons with the 1992 NALS while introducing new features
Allowing comparisons with the 1992 NALS while introducing new features
1992 assessment
13 blocks total
6 from 1985
7 new for 1992
2003 assessment
13 blocks total
6 from 1992
7 new for 2003
Six of the blocks from 1992 are used again in 2003.
Seven new (2003) blocks replace all six of the blocks from 1985 plus one of the blocks from 1992.
S. White 8-04
35 S. White 8-04
The seven blocks that were newly created for 2003
Are similar to the 1992 blocks in terms of the distribution of skills required by tasks
Have approximately the same average task difficulty as the 1992 blocks
Are spiraled with the 1992 blocks in the 26 booklets
Item response theory will be used to link the 1992 and 2003 scales using the tasks common to both assessments
Allowing comparisons with the 1992 NALS while introducing new features (continued)
Allowing comparisons with the 1992 NALS while introducing new features (continued)
36 S. White 8-04
New performance levels for 2003New performance levels for 2003
In 1992, five performance levels were developed by grouping together tasks of similar difficulty
The 1992 levels proved to have some limitations Scores that are the cut points between levels vary
depending on the measure of task difficulty used No distinctions are made within the lowest level of
performance (Level 1)
For 2003, new performance levels are being developed for the main NAAL by the National Academy of Sciences
Goals of the new levels include increased clarity and better differentiation among adults at the lower end of the scale
37 S. White 8-04
Main NAAL ALSA
Background questionnaire 28 min 28 min
Core screening items 8 min 8 min
Main NAAL or ALSA assessment 39 min 25 min
FAN 14 min 12 min
Total 89 min 73 min
Respondents may take as long as they need to perform main NAAL or ALSA tasks, but are encouraged to move to a new task when stuck or frustrated
NAAL total average assessment timeNAAL total average assessment time
38 S. White 8-04
The assessment is conducted in the participant’s home
The assessment is administered one on one
All participants receive additional time to complete the main NAAL or ALSA if they need it
The background questionnaire is administered orally in either English or Spanish
Instructions for the core screening items and for ALSA are given orally in either English or Spanish
Participants with a native language other than English or Spanish may take the core screening items even if they cannot complete the background questionnaire
Accommodations for adults with special needsAccommodations for adults with special needs
39 S. White 8-04
Expanded NAAL background questionnaireExpanded NAAL background questionnaire
For 2003, the background questionnaire was expanded to better reflect
Economic and technological developments in the past decade
The background of low-performing adults
Health-related activities of adults
Literacy-related activities of parents and their children
40 S. White 8-04
Characteristics of the NAAL scoring rubrics:Statistical considerations
Characteristics of the NAAL scoring rubrics:Statistical considerations
Responses to each item are analyzed for
Frequency with which a particular response is given
Interrater reliability in scoring responses
P-value (percentage of the population responding correctly)
Biserial (and polyserial) correlation between each response (and partially correct response) and the respondent’s overall score
Range findings for delineation of the range of responses received
Differential item functioning (DIF) in favor of certain groups
41 S. White 8-04
Seek evidence that adults can indeed use printed materials
Allow for partially correct responses if the information provided is still useful in accomplishing the task
Score most responses as either correct or incorrect, since often a partially completed task is not an accomplished task
Allow for writing errors as long as the overall meaning of the response is correct
Distinguish responses providing specific information from those providing generalizations
Development of the NAAL scoring rubrics:Content considerations
Development of the NAAL scoring rubrics:Content considerations
43 S. White 8-04
AM software has been developed by the American Institutes for Research (AIR), with NCES funding
The new software will reduce the amount of time required to analyze data by several months
The software will allow the technical report and the Data File Users Manual to be produced on a more timely schedule than in 1992
The software will also make it easier for states and researchers to do their own data analyses to supplement the information in the state and national reports
New software for data analysis New software for data analysis--
44 S. White 8-04
Average scores and distributionsAverage scores and distributions
Average scale scores and score distributions will beprovided
For the population as a whole
For population groups
But not for individuals
45 S. White 8-04
Inappropriateness of individual scoresInappropriateness of individual scoresInappropriateness of individual scoresInappropriateness of individual scores
No respondent takes all of the assessment
It would not be appropriate to calculate individual scores based on part of the assessment because
The individual scores would be unreliable (a reliable or consistent measure requires many items)
The individual scores would not be a valid representa-tion of the domain of adult literacy (valid representation requires many types of tasks, genres, etc.)
46 S. White 8-04
Trend analysisTrend analysis
Results from the 1992 NALS and the 2003 main NAAL will be compared in terms of
Average scale scores for prose, document, and quantitative literacy
Performance levels set by the National Academy of Sciences
P-values (the percentage of the population giving the correct answer to each item)
47 S. White 8-04
New release of items and p-valuesNew release of items and p-values
About 90 assessment items used in 1992 assessment will soon be released
All released items will be available on the NAAL website (http://nces.ed.gov/naal)
Each item’s p-value will be published
This will be the first publication of p-values for 1992 assessment items
P-values will be published for all items used in 1992—even items that are not released
48 S. White 8-04
Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #1
Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #1
49 S. White 8-04
Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #2
Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #2
50 S. White 8-04
New types of results in 2003New types of results in 2003
Health literacy scale scores measuring the performance of American adults on health-related tasks
Skill-based analyses inferring the literacy skills associated with the ability to perform main NAAL tasks
ALSA scores providing information on the literacy skills and deficits of America’s least-literate adults
FAN oral reading and basic skills data providing information about the skills of all adults
51 S. White 8-04
If you would like to receive NAAL information tailored to your interests, simply fill out a short online form at http://www.edpubs.org/list/NAAL
On the form, you can indicate up to 16 areas of interest Adult basic education Adult secondary education Citizenship/civics education College prep Computer literacy Correctional education Early childhood/children’s literacy ESL/ESOL (English as a second
language)
NAAL mailing listNAAL mailing list
Family literacy Health literacy Homeless education Learning disabilities Mathematical literacy Native language literacy (i.e., literacy in Spanish) Policy and legislation Workplace literacy
52 S. White 8-04
Sheida WhiteNAAL Project Officer
(202) [email protected]
Andrew KolstadSenior Technical Adviser
(202) [email protected]
Websitehttp://nces.ed.gov/naal
Sheida WhiteNAAL Project Officer
(202) [email protected]
Andrew KolstadSenior Technical Adviser
(202) [email protected]
Websitehttp://nces.ed.gov/naal
National Center for Education StatisticsNational Center for Education Statistics