pencils down - technology's answers to testing.pdf

7/29/2019 Pencils Down - Technology's Answers to Testing.pdf

1/76

With Support From the William and Flora Hewlett Foundation

TECHNOLOGY COUNTS 2003

Pencils Down:Technologys Answer

To Testing

American Educations Newspaper of Record Volume XXII Number 35 May 8, 2003 2003 Editorial Projects in Education / $6.00

EDUCATIONWEEK


2/76

Techs AnswerTo TestingSchool districts and states are turning to computer-

based assessment for quicker results and to meet

new federal accountability requirements.

Legal Twists,Digital TurnsThe No Child Left Behind Act of 2001 is having the

unusual effect of both encouraging and discouraging

schools use of computerized testing.

12 State Initiatives

15 A Test-Takers Perspective

A Question

Of DirectionComputer adaptive testing, which adjusts questionslevel of difficulty based on how well a student

answers them, is the source of much debate.

Prepping for theBig TestStudents turn to test-preparation Web sites

to help them prepare for high-stakes state

assessments and college-entrance exams.

Spec. Ed. TechSparks IdeasTesting tools for children with disabilities, such as

digital video cameras and computer text readers,

are attracting mainstream attention.

The TeachersNew TestThe use of computerized quizzes and exams for

classroom testing is on the rise as teachers look

for more efficient ways to evaluate students.

MarketingTo the TestTraditional test publishers and start-up businesses

have developed a host of technology-based testing

programs in the hope of cornering this new market.

Essay GradingGoes DigitalSoftware that evaluates student essays has its

share of critics, but some teachers say it has made

it easier to add more writing exercises.

Tracking TechTrendsStudent access to the Internet continues to improve,

but schools are spending only a small percentage

of technology dollars on professional development.

50 Analyzing the Tech Effect

State DataTables

54 Access to Technology

58 Capacity to Use Technology

60 Use of Technology

92 Sources and Notes

State of theStatesSnapshots of what each state is doing in the area

of computer-based testing, and how the states

are using technology to improve schools.

Index toAdvertisers

Contents8

11 39

43

54

62

94

17

23

27

31

35

COVER BY LAURA BAKER AND ALLISON SHELLEY/PHOTO ILLUSTRATIONS BY ALLISON SHELLEY


3/76

Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003

TECHNOLOGY COUNTS 2003

Pencils Down:Technologys

Answer to Testing

EDITOR & PUBLISHER

Virginia B. Edwards

MANAGING EDITOR

Gregory Chronister

PROJECT EDITORKevin Bushweller

PROJECT DIRECTOR

Ronald A. Skinner

EDITORIAL PRODUCTION CHIEF

Darcia Harris Bowman

RESEARCH ASSOCIATES

Susan E. Ansell

Melissa McCabe

Jennifer Park

Lisa N. Staresina

CONTRIBUTING WRITERS

Rhea R. Borja

Michelle Galley

Lisa Fine Goldstein

Kathleen Kennedy Manzo

Lynn Olson

Andrew Trotter

Mark Walsh

DESIGN DIRECTOR

Laura Baker

ASSOCIATE DESIGN D IRECTOR

Gina Tomko

DESIGN ASSOCIATE

Alyson Salon

PHOTO EDITOR

Allison Shelley

Technology Counts 2003was produced with support

from the William and Flora

Hewlett Foundation.


4/76


EDITORIAL & BUSINESS OFFICES:

6935 Arlington Road, Suite 100, Bethesda, MD 20814-5233

(301) 280-3100 E-mail address: [email protected]

FAX: Editorial (301) 280-3200 Business (301) 280-3250

EDUCATION WEEK is available on the World Wide Web atwww.edweek.org.

For New Subscriptions, Subscription Problems,

and Changes of Address:

EDUCATION WEEK, Post Office Box 2083, Marion, OH 43305

1-800-728-2790 E-mail address: [email protected]

EDITOR & PUBLISHER Virginia B.Edwards

MANAGING EDITOR Gregory Chronister

DEPUTY MANAGING EDITOR Karen Diegmueller

SENIOR EDITORS Caroline Hendrie, Lynn Olson (Special Projects),M. Sandra Reeves (Commentary)

ASSISTANT MANAGING EDITORS

Ann Bradley, Kevin Bushweller, Robert C.Johnston, Ben Wear

CONTRIBUTING EDITORS Mary-Ellen Phelps Deil y, Sandra Graziano, Scott W.Wright

ASSOCIATE EDITORS Jeff Archer, Kathleen Kennedy Manzo, Debra Viadero, M ark Walsh

ASSISTANT EDITORS Catherine Gewertz, David J. Hoff, Linda Jacobson,Bess Keller, Mary Ann Zehr

ASSISTANT COMMENTARY EDITOR Amy Conrad

STAFF WRITERS Julie Blair, Rhea R. Borja, Darcia Harris Bowman, Sean Cavanagh,Michelle R. Davis, Michelle Galley, John Gehring, Catherine Gewertz, Lisa Fine Goldstein,

Karla Scoon Reid, Alan Richard, Erik W.Robelen, Joetta L.S ack, Andrew Trotter

EDITORIALASSISTANT Marianne D.Hurst

EDITORIALINTERN Hattie Brown

DIRECTOR OF SPECIAL RESEARCH PROJECTS Kathryn M.Doherty

RESEARCH DIRECTOR Ronald A. Skinner

RESEARCH ASSOCIATES Susan E.Ansell, Melissa McCabe, Jennifer Park, Lisa N. Staresina

CONTRIBUTING WRITERS Rich Shea, Samantha Stainburn, Mark Toner

DESIGN DIRECTOR Laura Baker

ASSISTANT DESIGN DIRECTOR Gina Tomko

DESIGN ASSOCIATE Alyson Salon

PHOTO EDITOR Allison Shelley

ASSISTANT PHOTO EDITOR James W.Prichard

SYSTEMS MANAGER Hunter McCleary

ASSISTANT SYSTEMS MANAGER Bill Moyer

DIRECTOR OF NEW MEDIA Jeanne McCann

SENIOR ONLINE EDITOR Anthony Rebora

ONLINE EDITOR Rachell Deutsch

ASSISTANT ONLINE EDITOR Craig Stone

ONLINE INTERNS Patrick Byrnett, Alex Kobulsky, Michele Owens

LIBRARY DIRECTOR Kathryn Dorko

ASSISTANT LIBRARIAN Claire Guimbert

GENERAL MANAGER Michele J.Givens

CHIEF FINANCIAL OFFICER Eric W.Arnold

ACCOUNTING MANAGER Janice Brunson

GENERAL LEDGER ACCOUNTANT Kiki Aboaba

ACCOUNTS RECEIVABLE Barbara M.Guinn

ACCOUNTS PAYABLE Sage Wicinski

ONLINE BUSINESS MANAGER Angela Lunter

MARKETING DIRECTOR Karen Creedon Dunn

MARKETING ASSOCIATE Cheryl Staab

CIRCULATION DIRECTOR Patrick Haller

ASSOCIATE CIRCULATION MANAGER Christopher Klipple

CIRCULATION ASSOCIATE Alicia Osgood

FULFILLMENT MANAGER Laurie Breathnach

CUSTOMER SERVICE REPRESENTATIVE Joel I.Schwartz

PRODUCTION MANAGER Melayne Cohen (301) 280-3176, [email protected]

PRODUCTION ASSISTANT Armando Calonje Jr.

ASSOCIATE PUBLISHER Sam Bonfante, Advertising Sales(301) 280-3100

SENIOR ADVERTISING SALES EXECUTIVE Irving Carpenter

REGIONAL ADVERTISING MANAGERS Julie Fagan, Norlie Lin

MIDWEST ADVERTISING MANAGER Brian McGoldrick

WESTERN ADVERTISING MANAGER Sharon Makowka

ADVERTISING SALES ASSOCIATE Jennifer S. Haley

ADVERTISING ASSISTANT Carolina Calonje

CLASSIFIED MANAGER Titilayo Ellis

CLASSIFIED ACCOUNT REPRESENTATIVE Sean McLaughlin

CLASSIFIED ADVERTISING ASSISTANT Rose Wilson

EXECUTIVE ASSISTANT Shaiy E.Knowles

RECEPTIONIST Jackie Briceo

ADMINISTRATIVE ASSISTANTS Catherine Carroll, Maria Shaffer

EDUCATION WEEK (ISSN 0277-4232) is published 43 times per year (weekly on Wednesdays)

except for December 18, and December 25, 2002, and January 1, June 25, July 2, July 16, July23, July 30, August 13, August 20, and August 27, 2003, and issues will also be publishedThursday, January 9, 2003, and Thursday, May 8, 2003, by Editorial Projects in Education Inc.Editorial and advertising-sales offices are located at 6935 Arlington Road, Suite 100, Bethesda,MD 20814-5233;(301) 280-3100.Subscriptions: U.S.:$79.94 for one year (43 issues).Canada:$135.94, Foreign $208.94.Orders must be paid in U.S.dollars. Periodicals postage paid atBethesda, MD, and at additional mailing offices.POSTMASTER: Send address changes toEDUCATION WEEK, PO Box 2083, Marion, OH 43306.Volume XXII, Issue 35 . (USPS#599-870)

Back issues are available at $3.50 each, except the May 9, 2002 (Technology Counts) issue at $6.00each, and the January 10, 2002 (Quality Counts) issue at $10.00 each.Price includes first-classpostage and handling. Send order to:B ack Issues, EDUCATION WEEK, 6935 Arlington Road,Suite 100, Bethesda, MD 20814-5233.

Copies of the newspaper on microfilm can be ordered from Bell & Howell Information & Learning,Periodicals, PO Box 1346, Ann Arbor, MI., 48106-1346 .

EDUCATIONWEEK


5/76


Education and technology forces have converged

this year to vault computer-based testing into

the headlines, raising important questions

about whether this new mode of assessment is more

useful than traditional paper-and-pencil exams.

To begin with, the increased testing requirements

imposed by the No Child Left Behind Act of 2001a

far-reaching overhaul of federal education policy signed

into law by President Bush in January 2002have set

schools scrambling to find more efficient ways to

assess academic skills and get children ready for high-

stakes state exams. Unlike traditional standardized

tests on paper, which can take weeks or even months to

score and return to schools, computer-based assess-

ments can provide almost immediate feedback. That is

arguably one of the biggest draws for educators.

Already, 12 states and the District of Columbia have

a computerized exam or a pilot project under way to eval-

uate the effectiveness of computer-based testing, accord-

ing to a new Education Week survey of state depart-

ments of education.All of these testsexcept for one in

North Carolina and the District of Columbia examare

administered via the Internet. In five states, officials re-

port that computerized testing was designed to partially

meet requirements under the new federal law.

Eventually, experts predict, technology could change

the face of testing itself, enabling states to mesh the use

of tests for instructional and accountability purposes.

Youve got the potential that technology could be a

solution, says Wesley D. Bruce, the director of school

assessment for the Indiana Department of Education,

but there is, right now, just a huge set of issues.

Chief among them is a simple question: Do schools

have enough computers to test children in this new

manner? The answer in many places is no. And with

most states struggling with budget deficits, its unlikely

they are going to use their limited resources to fill

that void.

Yet researchers point out that computer-based test-

ing has the potential to be far cheaper than its printed

counterpart.

Richard Swartz, a senior research director at the Edu-

cational Testing Service, in Princeton, N.J., estimates that

the actual costs of putting a test online and building a cus-

tomized scoring model are comparable to those of develop-

ing a good paper-and-pencil exam.Once the tests are im-

plemented, he adds, the difference in scoring costs is

enormously in favor of the computer.

Still, other problems with computerized assessment

have emerged.

Techs AnswerTo TestingSchools turn to computerized

exams to meet new demands


6/76

One prickly issue involves the use of what is calledadaptive testing, in which the computer adjusts thelevel of difficulty of questions based on how well a stu-dent is answering them. Proponents of this form oftesting argue that it provides a more individualized

and accurate assessment of a students ability.But the No Child Left Behind law, a revision of the

Elementary and Secondary Education Act that puts ahigher premium than ever on schools accountabilityfor student achievement, continues to mandate thatstates measure student performance against the ex-

pectations for a students grade level.With adaptive testing, a 7th grader, for instance,

might be bumped up to questions at the 8th gradelevelor dropped down to the 6th grade level. As aconsequence, debate is growing about whether adap-tive testing can meet the purposes of the federal law,and if it doesnt, how the technology should be modi-fied to meet the requirements.

To give educators a head start on understanding

computer-based testing, Technology Counts 2003thesixth edition ofEducation Weeks annual report on ed-ucational technology in the 50 states and the Districtof Columbiaexamines these new developments

from a host of angles, beginning with an analysis ofthe impact of the No Child Left Behind law. Surpris-ingly, perhaps, the story points out that the law ishaving the effect of both encouraging and discourag-ing the use of computerized assessments.

As another part of this years focus on computer-based testing, Technology Counts 2003 takes a closelook at adaptive testing, with analysis from propo-nents and critics, and a description of how it works.The upshot of the adaptive-testing debate is that de-velopers of such assessments are worried that theymay be left out of what could be the greatest precol-legiate testing boom in history.

Computerized assessment may turn out to have itsbiggest impact in the area of online test preparation,observers of the field say. Last year, for instance, morethan 200,000 students in 60 countries signed up forthe Princeton Reviews online demonstrations of suchtests as the SAT and state exit exams. TechnologyCounts 2003 tracks the online test prep trend.As educators face the new federal requirement to

test all 3rd through 8th graders annually in readingand mathematics, states are experimenting with new

ways of using technology to evaluate the abilities ofspecial education students. Testing experts say thatwhat educators learn from how to tailor assessmentsto the needs of special education students could alsoshape how they test other students, who may havemore subtle individual needs. This years report ex-amines those lessons.

Technology Counts 2003 also includes a story aboutteachers who are using computer-based testing togive classroom quizzes and tests, an examination ofthe benefits and drawbacks of essay-grading soft-ware, an analysis of the growing business of com-puter-based testing, and a look at national trends ineducational technology.

Snapshots of the steps each state has taken to usecomputer-based testingor simply to use educationaltechnology more effectivelyare also included in thereport, as are data tables with state-by-state statis-

tics on technology use in schools.We hope youll find information here that will help

you understand computer-based testing and its evolv-ing role in education. THE EDITORS


10

Techs

Answer

to

Testing

"There is ...justa huge set of

issues" with thisnew way of testing.

WESLEY D. BRUCE, Director of School Assessment, Indiana Department of Education

COMPUTER-BASED STATE TESTS

Twelve states and the District of Columbia are administering computer-based state exams in 2002-03.

Six of these tests are pilots. Georgias online testing was temporarily suspended for spring 2003. Florida

and Oklahoma are planning pilots for 2003-04.

SOURCE: Education Weeksurvey of state departments of education, 2003

State has a computer-based state test (13)


7/76

Educat ion Week ' s Techno logy Counts/May 8,20 03

Computerized testing

feels the impact of

No Child Left Behind

ByBy Lynn Olson

South Dakota officials announced this past

winter that they were making the states

much-touted online testing program

voluntary for districts, and instead

requiring new paper-and-pencil

tests to meet the require-

ments of the No Child

Left BehindAct of 2001.

Meanwhile, officials in Idaho are

forging ahead with a statewide online

testing system, which they have modified

to meet the requirements of the federal law.

The contradictory moves illustrate the push

and pull that the law, a wide-ranging revision of

the Elementary and Secondary Education Act,

could have on the growth of technology-based

assessments in K-12 schools. While many

experts predict the law could lead to a

moderate slowing-down of computer-

based testing at the state level, they

anticipate a boom in low-stakes, tech-

nology-based assessments at the dis-

trict level to help students prepare for the

state exams.

I think No Child Left Behind poses an in-

teresting situation, because it could be both an

impediment and an impetus to the use of technology-

Legal Twists,

DigitalTurns


8/76

based tests, says Randy Bennett, an expert on tech-nology and assessment at the Educational TestingService in Princeton, N.J.

Technology can affect virtually every aspect ofassessment, from test design and administration to

the scoring and reporting of results. Advocates arguethat technology could make it easier for states anddistricts to meet some of the federal laws require-ments, by providing for cheaper and more efficienttest delivery, a quicker turnaround of scores, and theability to analyze and report test data in new ways.

Technological innovations also could make statetests more accessible to special populations of stu-dents, such as those with disabilities or limited flu-ency in English, who must be included in state test-ing systems under the federal legislation.

Eventually, experts predict, technology could changethe face of testing itself, enabling states to mesh the useof tests for instructional and accountability purposes.

Youve got the potential that technology could be asolution, says Wesley D. Bruce, the director of schoolassessment for the Indiana Department of Education,but there is, right now, just a huge set of issues.

An Immediate Concern

One of the most immediate issues for states thathave not yet ventured into computer-based testing iscost, particularly given ballooning state deficits.

Many states dont believe that theyre going to beable to meet the federal government mandate for as-sessments under No Child Left Behind with thefunds that the act is providing, Bennett of the ETSsays. So, given that, to then go out and try to buildan assessment system thats going to require evenmore upfront investment is just not going to be veryattractive to anyone.

In addition,Bennett notes, the federal laws aggressivetimelinesstates must administer reading and mathtests in grades 3-8 and at least once in high school by

2005-06work against the multiyear ramp-up re-quired to do high-stakes testing on a computer.

For that reason, I think its going to slow thingsdown from the point of technology-based testing, hesays, at least on the accountability end.

Not everyone shares that view.Scott Elliot, the chief operating officer for Vantage

Learninga for-profit company based in Yardley, Pa.,that provides online testing and scoring services for bothprecollegiate and higher educationsays his companyexpects to give some 17 million tests online this year.Although its not atypical for a state to spend $8 to

$10 per student to administer a paper-and-penciltest, he estimates the same test could be given on acomputer for $5 to $6 a student. And while the cost ofhuman scorers for essay and open-ended questionskeeps rising, the cost of automated scoring for suchquestions will probably stay the same or decrease.

While Elliot initially was concerned that the weakeconomy would deter states from venturing into tech-nology-based assessment, he now argues otherwise. Ijust think its going to break wide open in the nextfew months with state interest, he asserts. Demand,

he reasons, will be driven largely by the need to pro-duce test results faster, cheaper, and more efficiently.

Im getting more interest than ever before, Elliotsays, and I think it makes sense.

Richard Swartz, a senior research director at theETS, estimates that the actual costs of putting a testonline and building a customized scoring model arecomparable to those of developing a good paper-and-pencil exam.

But, he adds, once the tests are implemented, thedifference in scoring costs is enormously in favor ofthe computer, particularly when it comes to the com-puterized scoring of essays and open-ended ques-tionsan area in which both the ETS and Vantagehave worked extensively.

Last year, Indiana used the ETSs automated-scoringtechnologies for an online pilot of an end-of-courseEnglish exam for high school students, which in-


12

L

egal

Tw

ists,

Dig

ital

Tur

ns

Name of statecomputer-based test

Subject(s)tested3

State InitiativesTwelve states and the District of Columbia are administering computer-based assessments in 2002-03. Six of the tests are pilots. Of the states

with computer-based tests this year, five have reported that they designed the exams to partially meet federal requirements under the No Child Left

Behind Act of 2001, which requires testing of students in English and math in grades 3-8 and once in high school. Georgias online testing was

suspended for spring 2003 because of problems with test security. Florida and Oklahoma have planned computer-based testing pilots for the

2003-04 school year.

5

12

4

2

V

E, M

E, M

E, M

E, M

M

E, M, H, S,

E, M

T

E, M

E, M

E, M

E, M, S

E, M, H, S

8-12

5, 6, 8, 9

2

2-10

8-12

7

4, 5, 7, 8, 10-12

6-12

8

3-8, 10

3-12

2-12

3-12

high school

Vocational Student Competency Testing Program

Delaware Student Testing Program1

Measure of Academic Progress1

Idaho Standards Achievement Test

Indiana End-of-Course Assessments1

Kansas Mathematics Assessment1

Kentucky Core Content Assessment1, 2

Maryland Functional Reading/Math Tests

Computer Skills Test

North Carolina Computerized Adaptive Testing System1, 2

Technology Enhanced Student Assessment

Dakota Assessment of Content Standards

Utah Performance Assessment System for Students

Standards of Learning End-of-Course Test

Arkansas

Delaware

District of Columbia

Idaho

Indiana

Kansas

Kentucky

Maryland

North Carolina

Oregon

South Dakota

Utah

Virginia

U.S.

Grade(s) in whichcomputer-based

tests are available

All districtsand/or schoolsare required to

participate incomputer-based

testing

Test isadministered

over the Internet

Test is adaptive(questions

change based

on prior answers)

1Testing is in the pilot phase.

2Test serves as an accommodation option for students with disabilities to take the state assessments. Test is only available for students with disabilities.

3E-English/language arts, M-Mathematics, H-Social Studies/history,

S-Science, T-Technology, V-Vocational studies.4Any district or school chosen by the state for the pilot phase of testing must participate. However, there is no statewide participation requirement. State does not count in U.S. total.



9/76

cluded both essay and short-answer questions.We compared the cost of administering paper-and-

pencil tests and scoring all of the open-ended re-sponses by hand to administering the tests on com-puter and scoring everything with computer, says

Swartz, and the computer-administered and com-puter-scored version cost about a quarter of what thepaper-and-pencil version costs.

Whats more, students received the results in a mat-ter of days, rather than months.

The ETS official says more states and districts arenow open to technological solutions to their testingneeds, where we never saw that before.

Before, it was strictly, We want paper-and-pencil,Swartz says. Now, theyre saying, If youve got a tech-nology solution to propose, feel free.

Tight budgets definitely have put a crimp in stateplans, however.

Indiana had hoped to create an online item bank forteachers to craft classroom assessments linked to statestandards. But the planwhich would have cost about$800,000 a year for two years to generate a deepenough pool of test questionsis on hold.

We certainly havent given up on that, says Bruce,thestate testing director,but its going to be slightly delayed.

Adapting to Roadblocks

In Oregon, work on an online writing assessment hascome to a halt because of budget cuts. Schools alreadycan opt to administer state reading and math tests ingrades 3, 5, 8, and 10 either electronically or on paperthrough the Technology Enhanced Student Assess-ment program, or TESA, and the state will pilot addi-tional online tests in other grades this spring.

Were able to implement tests at new grades morequickly through TESAbecause essentially all the infra-structure is right there, says Bill Auty, the associatesuperintendent for the state education departmentsoffice of assessment and evaluation.

But he adds: Were fortunate that we started whenwe did. We could not start this up now. Our state fundsfor assessment have been cut considerably.

States that have ventured into online testing have al-ready confronted one roadblock under the federal law:

a mandate that states measure student performanceagainst the expectations for a students grade level.

In 2002, that requirement forced Idaho officials to mod-ify their plans for an online adaptive testing system, de-vised by the Portland, Ore.-based Northwest EvaluationAssociation.The system would have permitted studentsto take tests harder or easier than their actual grade lev-els, based on their ability to answer an initial set of testitems. As a compromise, state officials agreed to give allstudents a uniform, grade-level examination in additionto a more individualized set of test questions that couldprovide better diagnostic information.

Then, in January, South Dakota officials announcedthat they were making their states adaptive, onlinetesting programunder contract to the Scantron Corp.of Irvine, Calif.voluntary for districts and instead re-quiring new paper-and-pencil tests to meet the federalrequirements.

Wade Pogany, the director of education services forthe state education department, says the fact that theonline teststhe Dakota Assessment of Content Stan-dards, or DACSwere adaptive was not the primaryreason for moving to a new testing program.

I dont ever want to give the impression that SouthDakota does not like computer-adaptive tests, he says.Were a big supporter of that.But there are some issueswith the new legislation that caused us to look at a fixed-form test in relation to our standards, including the re-quirement that students be tested at grade level.

South Dakota, Pogany adds, is trying to use the bestof both worlds by continuing to finance the DACS fordistricts that want to use it, while coming up with anew set of exams that will allow the state to measurestudent performance against both state standards andnational norms.

Those experiences have clearly caused states to think


13

Test was designedto meet testingrequirements

under the federalNo Child Left Behind

law

L

egal

Tw

ists,

Dig

ital

Tur

ns

Test providesdiagnosticinformationabout thetest-taker

State uses anoutside vendor

to designand/or

administerthe test

State administerscomputer-based

tests withspecial

accommodationsfor ELL or

disabled students

Test includesopen-ended questions

Answers toopen-endedquestions are

machine-graded

Test includesmultimedia features

(sound orvideo clips)

Arkansas

Delaware

District of Columbia

Idaho

Indiana

Kansas

Kentucky

Maryland

North Carolina

Oregon

South Dakota

Utah

Virginia

U.S.

1

4

5

2

6

10

5

8

5Tests are both machine-graded and graded by hand.


10/76

twice about venturing into adaptive, online assess-ments. Oregon had planned to add an adaptive fea-ture to its Web-based assessment program this schoolyear, but delayed it in part because of budget cutsand development work.

We will implement that next year, says Auty, theassociate superintendent for assessment and evalua-tion. Its something were looking into.

He adds, though: We want to be very clear, giventhe controversy, were talking about adaptive testingin the grade level.

Others argue that adaptive testing could meet therequirements under the federal law, but that it wouldtake more extended discussion and explanation fromtesting experts.

A Major, Major Issue

A far bigger impediment for state testing systems isthe infrastructure needed to deliver a secure com-puter-based exam to thousands of students under thesame conditions at the same time.

The major, major issue is crystal clear, and thats theinfrastructure, argues David J. Harmon,the director oftesting for the Georgia Department of Education.

Georgia had been planning to give districts the op-tion of offering the states Criterion-Referenced Com-petency Tests in grades 1-8 in a paper-and-pencil orWeb-delivered format this spring. But the state wasforced to suspend online testing this year after discov-ering that some 270 actual test questions were pub-licly available on an Internet site for students, parents,

and teachers. The state had developed the Web-basedtest-item bank to help schools prepare for the exams.

Kathy Cox, the state superintendent of education,says a miniscule amount of school districts evenwanted that online option, about half a dozen out of

181 districts.But in Oregon, where tests are available online,

Auty estimates about one-third of schools use theTechnology Enhanced Student Assessment programelectronically.

In Virginia, where districts can choose to offer mosthigh school end-of-course tests online or on paper,Mark J.Schaefermeyer, the associate director of Web-based assessment in the state department of educa-tion, estimates that up to 85 of 135 districts willchoose the Web-based version this spring.

Heres the challenge: High-stakes tests, such asthose in Virginia, which are used to make key deci-sions about individual students or schools, must begiven under secure conditions so that there are fewopportunities for cheating.Theyre typically adminis-tered during a limited time window. And they aresupposed to be given under the same conditions toevery student.

So how many computers does it take? asks Har-mon. I think thats the biggest problem.

In addition to the upfront and ongoing costs of buy-ing and maintaining hardware and software systems,computers and Internet connections dont alwaysfunction dependably. And testing sessions may be in-terrupted or proceed so slowly that the conditions in-terfere with student performance.

Capacity is the number-one issue; compatibility isprobably number two; and keeping your lines open isnumber three, says Bruce of Indiana.

This year, Indiana plans to pilot both end-of-courseEnglish and algebra exams exclusively online for

high school students. One of the things we ran intoin one of the largest high schools in the state wherethey wanted to do this, Bruce says, is that in orderto give the algebra test, they had to close down fourcomputer labs for two weeks. It took that capacity.

Eventually, Georgias Harmon predicts, the solutionmay lie with small, handheld, wireless devices thatlook like GameBoys, but with larger screens. Givensomething like that, he says, then there will be verysubstantial savings compared with the printing,shipping, and storage costs of a traditional exam.

The Wrong Question

On the measurement side, the biggest challenge forstate testing programs is the comparability of testresults. To compare performance fairly, each student

should be tested under the same conditions. If thereare any variations, they shouldnt affect performance.

But Bennett of the ETS points out that, in reality,the equipment used in statewide computerized-test-ing programs often varies from one school to the next,and sometimes even from one machine to the nextwithin the same building. Researchers dont yet knowhow to adjust for such variations in scoring results.

In states such as Virginia, where some students willtake a test online while others take a paper-and-pen-cil version, policymakers need assurances that nei-ther group is particularly advantaged or disadvan-taged by the mode of delivery.A study by researchers at Boston College, for in-

stance, found that students accustomed to writingon computers scored higher on test questions an-swered electronically than on those answered byhand, while those who were used to writing in long-

hand scored worse than similar students who tookthe exam on paper.Although several test ing companies report they

have conducted comparability studies, few have beenpublished. Moreover, observes Wayne Ostler, the di-rector of eMeasurement Services for Pearson Educa-tional Measurement, based in Iowa City, Iowa, youreally have to perform comparability studies on everyprogram. You cant just say, Oh, I tested math in Vir-ginia, and it turned out to be comparable, thereforeeverything is comparable.

But Elliot of Vantage Learning suggests the con-cerns are off target. When the car was introduced,can you imagine them conducting a car-horse compa-rability study? he says.Its the wrong question. Thecar is coming. Ive got news for you. If theyre notcomparable, so be it.

Not surprisingly, though, such concerns are leading

some states to focus their initial online efforts on low-stakes examsones that arent used to rate schoolsor decide students futures.

Low-Stakes Testing Boom

In contrast to some of the reticence about givinghigh-stakes tests electronically, experts predicta boom in the use of technology for instructionaland diagnostic purposes spurred by the new federallegislation.

What I think No Child Left Behind may speed upis the development of low-stakes, computer-basedtests that are designed to help kids do better on thehigh-stakes ones,Bennett of the Educational TestingService says. I think thats where were really goingto see a lot of movement.

In December 2002, Texas unveiled the first phase ofthe Texas Math Diagnostic System, a roughly $2 million,online practice assessment program for grades 4-8.Thisspring, the state is adding a 3,400-question item bankthat teachers can use to come up with their own tests,quizzes, and homework assignments.And it is translat-ing the test items in grades 4-6 into Spanish.

Texas also is working to add open-ended questions to


14

Legal

Tw

ists,

Dig

ital

Tur

ns

"How many computers

doesit take? I thinkthats the biggestproblem."

DAVID J. HARMON, Director of Testing, Georgia Department of Education

Continued on Page 16


11/76


By Jennifer Park

A

n informal survey of students about Oregons online test-ing system shows they find computer-based testing fasterand more enjoyable than the paper-and-pencil variety, and

they report feeling that they perform better on computer-ized assessments than on traditional tests.

While some states are just beginning to contemplate the possi-bility of computerized testing, Oregon piloted a program in spring2001 that is now being expanded across the state.The system de-livers the Oregon Statewide Assessment over the Internet.

The main goal of the Technology Enhanced Student Assessment,or TESA, program is to provide test resultsquickly, so teachers can more effectively deter-mine how well students are meeting state stan-dards. The program also aims to offer more flexi-bility: Teachers can administer the online testswhenever they deem appropriate, whether to anindividual student, a group of students, or an en-tire class.

To assess the pilot phase of the online testingprogram, Oregon state education officials surveyed740 3rd graders and 730 high school students from

around the state about their experiences takingthe state assessment online compared with takingthe hard-copy version. About 2,500 students hadtaken the online tests. State officials also held de-briefing sessions and hired evaluators to conductinterviews with teachers.

Third graders were especially positive aboutWeb-based testing. Seventy-nine percent of thosepupils surveyed believed they had done theirbest work on the computer-based reading test,compared with a little more than 5 percent of3rd graders who felt they either had done betteron paper or were frustrated using the computer-ized test.

Results were similar for the online math test.Onthat assessment, more than 94 percent of the 3rdgraders said they felt that they had done theirbest work on the computer test or that they had

done equally well on both versions. Only 6 percentbelieved they had performed better on paper orhad felt frustrated with the computer-based exam.A majority of the 3rd graders enjoyed taking the

Web-based version more than they did the papertest: Sixty-two percent reported that the onlinereading test was easier to use and more enjoyablethan the paper test, and 58 percent said the sameabout the Web-based test in mathematics.

Not as Positive

High school students were slightly less positiveabout their performance on the Internet-basedtests.

On the reading test, 78 percent of the highschool students believed they had either donebetter on the computerized version or equallywell on the Web and paper tests. About 22 per-cent of the high schoolers felt they had eitherdone better on paper or were frustrated using thecomputer-based exams.

In math, about 76 percent of the high school students either feltthat they had done their best work on the computerized test orthat they had done equally well on both versions. Fewer than aquarter of the high school test-takers felt they had done better onthe paper exam or were frustrated using the Web assessment.

Most of the high school students reported finding that the Inter-net version had taken less time or the same amount of time as thepaper test.

But a considerable number of the students disagreed. Thirty-onepercent of the high school students tested said the Web version ofthe reading test had taken more time than the paper test. Thesame percentage reported that the computerized math test hadtaken longer to complete than the paper test.

Thirty-seven percent of the high school students rated the com-puterized version of the reading test easier to use and more enjoy-able to take than the paper version; 38 percent found the Web-based math test easier to use and more enjoyable to take.

Twenty percent of the high schoolers surveyed in Oregon saidthat the Web-based reading assessment was harder to use thanthe paper-and-pencil test.

A Test-Takers PerspectiveSurvey asks Oregon students

what they think about the states

computerized assessments

15

Legal

Tw

ists,

Dig

ital

Tur

ns

COMPUTER VS. PAPER TESTING

Third graders responded positively about their performance on a Web-based test,

compared with a paper-and-pencil version of the Oregon state reading exam. While

high school students were less positive, most still felt they either did better on the

Web-based test or equally well on the Web and paper versions of the test.

SOURCE: Oregon Department of Education, Technology Enhanced Student Assessment (TESA), student survey, 2001

A majority of 3rd graders found the online version of the Oregon state math test

easier to use than the paper-and-pencil version. Thirty-eight percent of high school

students found the Web version easier to use and more enjoyable.

3rd Grade Math High School Math

Web test was easier to use and more enjoyable

About the same as paper-and-pencil test

It was harder than paper-and-pencil test

More difficult at first, then easier

0 20 40 60 80 100

Percent of students

I feel sure I did my

best work on the Web

I did equally well

on Web and paper

I didnt do as well

as I did on paper

I was too frustrated

using the Web

79

39

40

16

16

6

4

23rd grade reading

High school reading

15%

58%

22%

5%

38%

39%

17%

6%

NOTE: Percentages do not add up to 100 because of rounding.


12/76


16

L

egal

Tw

ists,

Dig

ital

Tur

ns

the system, as well as an adaptive-testing component.And in Florida, teachers can use the FCAT Ex-

plorer, a free, Web-based program that provides a

series of test prompts and skills packages designedaround the states academic-content standards,which guide the Florida Comprehensive Assess-ment Test.

We are currently getting about 11.5 million hits aday, says policy consultant Don Griesheimer, and Ithink the company thats hosting this has said wehave about 4,000 simultaneous users a second. Soits taken off big guns here in the state of Florida.

Other states have offered schools access to commer-cially developed assessment-and-diagnostic packagesthrough state Web portals.Indeed, the greatest activityis taking place at the district level, where companiesare offering schools a range of products to help monitorstudent progress and provide teachers with instantfeedback on what to do next.

This indirect market, generated by the No ChildLeft Behind Act, I see as the real impetus for how

technology will be used in schools, says Michael T.Nesterak, the director of product management forCTB/McGraw-Hill, one of the countrys biggest com-mercial test-makers.

Untapped Power

Despite such inroads, most observers agree thatthe potential power of technology to redesign as-sessment remains largely untapped.

The powers of the computer have not been fullyused yet in any of the state testing programs, saysNeil Kingston of Measured Progress, a Dover, N.H.-based company that has worked on online testingprograms in Georgia and Utah.

Most computer-based programs at the state level re-main exclusively multiple-choice.Thats not surprising,

observers say. The reality is we have to start whereschools and districts are, says John E. Laramy, thepresident of Riverside Publishing, one of the nationslargest test companies.He believes that in time, youwill see companies introducing more innovative, inter-active assessments.

But with the federal laws emphasis on providingmore tests, at more grade levels, faster and moreefficiently, no one expects that breakthrough any-time soon.

No Child Left Behind might be pushing states,says Doris Redfield, a technology expert at the Ap-palachian Education Laboratory in Charleston,W.Va., but its pushing them in the direction thattheyre already going of getting quicker results andensuring that their assessments are aligned withtheir standards.

Is it really pushing them to create assessments

that are more true-to-life or out of the ordinary?Redfield wonders. I dont know that it is.At least one observer, Ed Roeber, the vice presi-

dent of external relations for Measured Progress,worries that the current emphasis on cheaper,faster, better is leading states to eliminate or re-duce the number of open-ended responses in theirtesting programs.After an initial burst of steam,experts admit,the ex-

pansion of computer-based testing at the state level ishappening more slowly than they had expected.

Its kind of frustrating, says Greg Nadeau, the di-rector of the U.S.Open e-Learning Consortium,a col-laboration among 14 states for sharing online educa-tional tools. The economic contraction is taking someof the wind out of the sails of the [information tech-nology] industry and of state and local and federalprojects.

It takes years to develop online testing,Pogany ofSouth Dakota says. So you cant come in and say:OK.We have a new test. Put it online. Do it.

Until I know that we have a solid and reliabletest, he adds of South Dakotas new assessmentsystem, were not going to put kids at a computer.Now, will that happen next year? I dont know. It willhappen when were ready, and were not going to doit before.

Continued from Page 14


13/76


C

omputer adaptive testing is used to test re-

cruits to the U.S. military, for licensing nurses

and computer technicians, for entrance tests

to graduate school, and for a popular place-

ment test used by community collegesbut not for aca-

demic testing in all but a handful of K-12 schools.

Most notably, computer adaptive testing has been

left out of nearly all the large-scale testing programs

that states are ramping up to meet the requirements

of the federal No Child Left Behind Act of 2001.

A prime reason:The U.S.Department of Educa-

tion interprets the laws test-driven accountability rules

as excluding so-called out-of-level testing. Federal offi-

cials have said the adaptive tests are not grade-level

tests, a requirement of the law.

Psychometricians regard that decision as humor-

ous, Robert Dolan, a testing expert at the nonprofit

Center for Applied Special Technology in Wakefield,

Mass., says of the departments stance.

Adaptive tests deliver harder or easier items, depending

on how well the individual test-taker is doing.They are con-

sidered out-of-level because the difficulty range could in-

clude skills and content offered in higher and lower grades.

Dolan and other test experts concede states may have

reason to say no to computer

adaptive testing, because of cost, uneven

technology levels in schools, and even educators

unfamiliarity with the methodbut not because of

grade-level testing.

The span of [test item] difficulty from easiest to

A QuestionOfDirection

ByBy Andrew Trotter

Adaptive testing puts federal

officials and experts at odds


14/76

hardest is entirely under the control of the test devel-oper,says Tim Davey, the senior research director of theEducational Testing Service, based in Princeton, N.J.

Some experts say adaptive tests give schools a bet-ter return on the time and money devoted to test-

ingincluding more accurate measurement of theproficiency of students who are above and below av-erage, and speedier access to the test results.

But Education Department officials say their handsare tied. The regulations are very clear in saying all stu-dents have to be held to the same standard as the foun-dation for school accountability,says Sue Rigney, an ed-ucation specialist in the department.The focus here isvery explicitly on the grade level the state has defined.

Federal officials worry that out-of-level testing mightlead to lower expectations for below-average students.

They also note that states are free to use computer-adaptive tests outside the accountability purposes ofthe No Child Left Behind law, which requires yearlyassessments in reading and mathematics of studentsin grades 3-8.

But the upshot, for now, is that computer adaptivetests are left out of the federal law, along with the pub-

lic attention and federal money for test developmentthat come with it.And the developers of adaptive testsfeel they are missing out on what may be the greatestprecollegiate testing boom in history.

Made Us a Pariah

[The Education Departments] decision made us apariah, says Allan L. Olson, the president of theNorthwest Evaluation Association, a nonprofit test-ing organization in Portland, Ore.The group was de-veloping a computer adaptive test for Idahos assess-ment when the department ruled its method out justover a year ago.

Federal officials gave the same message to SouthDakota and Oregon. South Dakota subsequentlymade voluntary its once-required computer adaptive

test, and has adopted a conventional paper-and-pen-cil test for its statewide program. Oregon has post-poned for a year the addition of a computer adaptivefeature to its online test.

I think the [departments] interpretation in the caseof South Dakota was based on a sort of misunder-standing of what adaptive testing does, says Davey ofthe ETS. He says computer adaptive tests typicallyspan more than a single grade levela diagnostic ben-

efitbut they dont have to, and in any case, grade-level information is recorded for each test item.

Researchers express puzzlement because the federalgovernment has been deeply involved in the develop-ment of computer adaptive testing,starting with sem-inal research at the U.S. Office of Naval Research inthe 1970s and 1980s. A decade later, Education De-partment grants paid for new computer adaptive read-ing tests in foreign languages,and department officialslauded the methods potential for school-based testing.

David J. Weiss, one of the original leaders of theNavy research, says there is no reason why com-puter adaptive testing is not appropriate for K-12.

Now the director of the psychometric-methods pro-gram at the University of Minnesota, Twin Cities,Weiss notes that a study of children who took suchtests in Oregon for several years produced beautifuldata on improvements in math and reading.

Federal officials say they would consider the useof a computer adaptive test if it tested within thegrade level.

But other test experts say the federal governmentis right to be wary of computer adaptive testing.

The technology is not ready for prime time, con-tends Robert A. Schaeffer, the public education direc-tor for the National Center for Fair & Opening Testing,or FairTest, a Cambridge,Mass.-based advocacy groupthat opposed the No Child Left Behind Act because ofits testing mandates.

He says the computer adaptive version of the Grad-uate Record Examination launched at ETS testingcenters in 1994 was initially flawed because it had apool of test items that was too small, and there wereinsufficient facilities for the number of test-takers.

ETS spokesman Tom Ewing acknowledges thoseproblems occurred but says they were quickly re-

solved through enlarging the pool of questions andimproving test scheduling.

But Schaeffer warns that schools could face arougher transition, considering their budget limita-


18

A

Question

of

Direction


15/76

tions and the high stakes involved in testing.W. James Popham, a professor emeritus and educa-

tional testing authority at the University of Califor-nia, Los Angeles, says the theoretical accuracy ofcomputer adaptive testing does not necessarily trans-

late into reality: Even though [such testing] makesmeasurement types quite merry, they can play gameswith numbers and it doesnt help kids.

Popham, a former president of the American Educa-tional Research Association, contends that the testingtechnology is opaqueto the public and policymakers.

He says federal officials may believe the testingmethod could introduce loopholes into the education law.

They fear educational con artists who have histori-cally frustrated congressional attempts to safeguard dis-advantaged youngsters, Popham says, referring to edu-cators who wish to avoid accountability. The fear is,theyll pull a fast one and downgrade expectations.

Zeroing In on Skills

But proponents of adaptive, computer-based testing

fear that schools may wait decades for access to amajor improvement over conventional, linear stan-dardized tests, which present each student with thesame set of test items.

The logic of the new tests is that of a coach pitching to ayoung batter: If the youngster is missing,the coach easesup a little; if not, he increases the challenge. Sooner orlater, the coach zeroes in on the batters skill level.

Some testing experts argue that the adjustment im-proves test accuracy.

In paper-and-pencil tests, items tend to be groupedaround average kids.Those in the tails of distributionwe dont get as much information about those kids,saysMichael L. Nering, the senior psychometrician at Mea-sured Progress,a testing company in Dover,N.H.

The great thing about adaptive testing is that it hasequal precision, meaning the results are accurate at allproficiency levels, says Nering, who helped design two

state assessments and developed computer adaptivetests for ACT Inc. No matter what your ability is,

whether youre really smart or not, the test will stop ad-ministering items when equal precision is reached.

By contrast, most of the items on conventionaltestson paper or computerare aimed at the aver-age student in the target population.

If Im a very low-performing student, there may beonly two or three items on the [conventional] test thatare appropriate to my level of performance, Davey ofthe ETS says, adding that the same is true for high-performing students.

Inside the IRT

Computer adaptive tests often use the same types ofquestions as conventional tests, though with adjust-ments for display on a screen. Other features are dis-tinctive, such as the order of items being irreversible.Students are not allowed to recheck or change answers.

This one-way street is necessary because of theprocess that takes place after each answer: A centralcomputer recalculates the test-takers ability level,then selects the next item, based on the individuals

success to that point.As the student completes more items, the computer

tracks the statistical accuracy of the score until a setaccuracy level is reached. Then the test moves to an-other skill or content area. Reaching that level mayrequire fewer items if the student answers with con-sistent proficiencyor many more items, if the stu-dent answers inconsistently.

Adaptive testing doesnt waste the examinees time byasking questions that were already pretty sure we knowhow the student is going to answer, says Davey.

To make the crucial decisions about which items topresent, the test is outfitted with an item responsetheory modelessentially its brains and the part ofthe system that some critics consider opaque.

The IRT model governs the interaction between thetest-taker and the test items. It weighs the studentsrecord of right and wrong answers against several

known characteristics of the test itemssuch as diffi-culty, the ability to discriminate between higher- and


20

A

Question

of

Direction


16/76

lower-ability students, the degree to which guessingmay succeed,and coverage of academic content.

By solving the complex algorithm written into theIRT model, the computer determines which test itemshould be presented to the student next.

Test developers concede that IRT models are un-fathomable to lay people and even challenge the in-tellects of experts unfamiliar with a given test.

Schaeffer of FairTest calls the IRT model the pig ina poke that makes computer adaptive testing hardfor policymakers to accept.

Who knows what the algorithm is for test deliv-ery? he asks. You have to accept the test manufac-turers claims about whether the test is equivalentfor each student.

Scott Elliot, the chief executive officer of VantageLearning, a major maker of computer-based tests lo-cated in Yardley, Pa., says, There are many technicalnuances under the IRT; some differences [betweenIRTs] are sort of like religion.

Davey of the ETS agrees that the IRT resists attemptsto explain it, but adds that the apparent simplicity ofconventional testing is based largely on oversimplifi-

cation of how paper testing typically is done.In fact, he says, virtually identical IRT models are

used with some conventional state tests to ensurethat the same score in different years represents ap-proximately the same proficiency level on the testavital issue for accountability.

Breaking With the Past

Because of technology hurdles and spotty acceptanceof computer adaptive testing, experts generally predictthat the field will struggle for the next five or 10 years,but that schools will eventually turn to the approach.

Davey believes educators will be persuaded by thegreater amount of diagnostic information the tests pro-duce from fewer school days spent testing.

Thats not to overlook other formidable problems

that computer-based testing poses for schoolsno-tably, the difficulty of providing technology that is re-liable and consistent for all students, so the playingfield is kept level. The tests must be delivered over arobust infrastructure to avoid processing and com-munications delays that would leave students wait-ing for their next test items.

Computer adaptive tests also require larger banksof test items than conventional tests do. Yet theadaptive method gives items a longer useful life be-cause its harder for test-takers to predict whichitems they will encounter.

Finally, adaptive tests are subject to some of thesame well-documented problems as other standard-ized tests, such as cultural biases, says FairTestsSchaeffer. Automating test items that are used in-appropriately, in many ways makes matters worseyou add technical problems and dissemination-of-in-

formation problems, he says.Referring to the ETS adaptive Graduate Record Ex-

amination, he adds, The GRE, in spite of all the hoopla,is the same lame questions put out using a hidden algo-rithm, instead of linearly on a sheet of paper.

Ewing of the ETScounters that its test items are whatthe graduate deans have said are the math and verbalskills that they want students to be able to handle.

Meanwhile, researchers are working on new kindsof adaptations that could be applied in computeradaptive testsincluding presenting items usingmultimedia or computer simulations and catering toan individuals preferred learning style. Already,some tests present items in different languages.

Those changes highlight another potential pitfall.Today, policymakers insist on having new testsdemonstrate comparability with old tests, a taskthat Davey says becomes vastly more difficult astesting methods change.

Benefiting from many promising innovations willrequire letting go of comparability, Davey maintains.

Its like when we moved away from strictly essaysand constructed-response items 100 years ago andintroduced multiple-choice items, he says. For testsbased on simulations, theres no paper-and-pencilequivalent anymore.You have to make a clean breakwith the past.


21

A

Question

of

Direction


17/76


A

s test-preparation materials leap off the

printed page and onto the Web, an increas-

ing number of states and districts are

turning to online test-prep programs

to help raise student scores on high-stakes

assessments, Advanced Placement tests,

and college-entrance exams.

California, Florida, and Massa-

chusetts are among the states

signing up for Web-based remedial

offerings and other academic help,

spurred on by stiffer state and federal

accountability requirements.

Soon, experts predict, most states and districts

will use online test prep because of its flexibility, its

reasonable cost, and some promising, though very

preliminary, results on its effects.

This is definitely not a fadyoull see Web-based

assessments broadly adopted in the next several years,

says Adam J. Newman, the vice president of research for

Eduventures Inc., a Boston-based research firm that

tracks the K-12 market.

Print will never disappear, but institutions will have

more cost-effective solutions with the Web.Also, the ability

for educators to

customize their assessments

will really enhance what they already do

[for students].

The many companies that have jumped into the test-

prep market range from the well-established, New York

City-based Princeton Review to small, aggressive

upstarts such as Test University, also in New York, and

the Washington-based Smarthinking Inc.

The money has followed. The Princeton Reviews

PreppingforThe BigTestStudents turn to the Web to get

ready for high-stakes exams

ByBy Rhea R. Borja


18/76

K-12 revenue shot up 43 percent last year, boostedin large part by a $2.5 million boom in online test-prep subscriptions and other supplemental programs,according to the Heller Reports, an educationalnewsletter and market-research publisher based in

New York.Last year, more than 200,000 students in 60 coun-

tries signed up for the Princeton Reviews onlinedemonstrations of such tests as the SAT and state exitexams, according to Drew Deutsch, an assistant vicepresident of the company.

Our online test prep is growing by leaps andbounds, he says.

Not for Everybody

Eduventures Newman and other experts caution,though, that online test preparation is only as good asa students motivation to use it and a teachers abilityto analyze the data gleaned from it. Web-based testprep is a tool to improve learning, they emphasize, nota magic bullet.

Teachers need to know how to use this resource andapply it to their classroom pedagogy, Newman says.Otherwise, its useless.

Online test prep isnt for everybody, adds BurckSmith, the chief executive officer of Smarthinking,which serves 29,000 high school and college studentsin at least 15 states. He says students who use themethod must be highly self-motivated or have teachersmonitoring them. He found that out with students whohad failed their state assessments the first time anddidnt seem especially impelled on their own to im-prove their scores.

With any online test-prep program, youre not goingto see a lot of success unless theres someone theremaking sure the students are doing what theyresupposed to be doing, Smith says. Motivation is ahuge factor.

And critics of online test preparation wonder whetherits approach of drill and repetition is supplanting morelively, creative teacher-led instruction.Are the studentstaking online test prep getting only the chunks of infor-mation they need to know to improve their scores, in-

stead of a more balanced education?Web-based, online computerized coaching pretty

much focuses on the basics, so the kids pretty muchlose out on everything else, says Robert A. Schaeffer,the public education director for the National Centerfor Fair & Open Testing, or FairTest, a test-reformadvocacy group based in Cambridge, Mass. What istested is what is taught.

Its a Conversation

But supporters of online test prep point out that theprograms can be customized with lessons and practicetests to fit each students academic needs, can providea wealth of student data, and can be used anytimeinclass during the day, and at home at night and onweekends.

The Princeton Reviews Homeroom.com, for example,has a 130,000-question bank for grades 3-12, which dis-tricts and states can mine to produce online practice testsand courses specifically aligned to state exams.Its onlineSAT-prep program offers 185 interactive, multimedialessons and drills, and an online coach to answer ques-tions 24 hours a day.

Test Universitys online SATcourse has interactive micro-courses covering 145 basic skills, 18 practice tests, a2,000-word vocabulary lab, and two full-length SAT testswith immediate feedback. TestU also features lessonscovering 55 test-taking strategies.

Smarthinking, meanwhile, uses e-tutors, or online in-structors, and digital whiteboards to give students moreintensive tutoring help. Students can communicate withe-tutors in real time, as well as submit questions or as-signments to the instructors online and receive feedback


24

Prepping

for

the

Big

Test

COMPUTER-BASED PRACTICE

Several states are using technology to help their students prepare for state tests by offering computer-based

practice exams. These practice exams range from tutorials that mainly serve to guide students through an

online testing format to those that test students knowledge of content related to state standards.


Computer-based practice exams (12)


19/76

within 24 hours,the company says.The virtual whiteboards act much like real chalk-

boards: A student can write out or draw a multisteptrigonometry problem using an online toolbox of math-ematical symbols. The e-tutor sees the students work,

and can comment on the whiteboard and help correctany missteps.

The 12,000-student Whitfield County school systemin rural northern Georgia uses Smarthinking in partto help its 164 Spanish-speaking, English-as-a-second-language students pass the Georgia Graduation Test.For two hours a day, ESL-trained, Spanish-speakingmath tutors answer students questions online, saysLorijo Calhoun, the districts ESL instructor.

Its a conversation, she says. Its literally backand forth.

In Marylands 107,400-student Baltimore Countysystem, teenagers preparing for the PSAT, the SAT, orthe ACT can log on to TestUs Web site and get extrahelp on their vocabulary, for example, or more in-depthinformation on algebraic algorithms.

Students first take a 30-minute diagnostic pretest, orbrainscan, that TestU uses as a baseline for the on-line curriculum. If a student does poorly on grammarbut well on comprehension, for example, TestU offersmore intensive work on the former. Like many otherschool districts and states, the Baltimore County sys-tem is making TestU available to all students, but es-pecially pupils from low-income families.

We want to ensure equity of access, says BarbaraDezmon, the assistant to the superintendent of theBaltimore County district, which does not include thecity of Baltimore. It started as an effort to assistschools that had significant minority student popu-lations, significant poor populations, and it spreadfrom there.

Diane Young, an assistant principal at the 1,350-stu-dent Eastern Technical High School in Baltimore

County, says that online test preparation is more help-ful than the schools traditional SAT-prep class becauseinstruction is customized for each studentsomethinga teacher with 25 students cannot do.

It allows a teacher to say to a student, Youre hav-ing problems with fractions, and I cant spend anymore time in class, so go on TestU and work on it, andIll check your answers, she says.

I Felt More Prepared

But online test prep is not a substitute for a teacher,Young and other educators stress. Instead, its an-other tool that can enhance the effectiveness of in-class instruction.

Glenn Finnerty, a test-prep teacher at the 3,200-stu-dent Timber Creek High School in Orlando, Fla., sayshis students use both a workbook and individualizedonline mathematics and reading courses by TestU tohelp them improve their scores on the Florida Com-prehensive Assessment Test, or FCAT.

During class, Finnerty walks around his room andmonitors students as they work on the computercourse and practice tests. He can also view a variety ofcomputer reports that analyze student performanceand participation rates.

One is a group diagnostic report, which aggregatesand analyzes students responses to test questions,while a group learning report shows the skills andstrategies studied on a course, and evaluates the stu-dents quiz performance. A group performance report

shows student participation and performance rates,the frequency of tests taken, the average percent ofquestions answered correctly, and the distribution ofstudent responses to each question.

The combination of in-class instruction, onlinecoursework, and teacher monitoring is essential in try-ing to help low-scoring and perhaps poorly motivatedstudents improve their test scores, Finnerty says.

Its a complement to what we already have with ourbook work, he says. If teachers just say to the kids,Go at it, and leave them at the computers by them-selves, thats not good.

One of his students, 15-year-old Davis Ho, recentlytook the FCAT, and says that online test prep did itsjob. In fact, he says, many of the questions in his test-prep program were similar to the questions on thestate assessment.

I felt more confident, more prepared, says the 9th

grader.Ho says that in some ways, online test preparation

is more helpful than a teacher because it allows him tomove at his own pace. You can go back and review thestuff, he says. You can take your time and read it overand over again without having to frustrate anybody.

He and others suggest that the main difficulty inusing Web-based test prep is the tendency of comput-ers to freeze up. Technical glitches greatly hamperedTimber Creek Highs ability to use the TestU programlast year, school staff members say, but emphasizethere are fewer such problems this year.

Ho replies that while that may be so, glitches arestill not uncommon. Sometimes, the program kicksme off, he says.

Often, glitches occur because of a miscommunicationbetween TestU and the school districts filtering andsecurity software, says Nicole Marshall, TestUs vice

president for educational research. Thats what hap-pened to Timber Creek.

So county technology administrators readjusted thesettings in the security software to accommodateTestU. If the school is still experiencing glitches, itcould be that a lot of students are using the programand other technology at peak times, slowing down thesystem, Marshall adds.

Buying in Bulk

Web-based test prep tends to be cheaper for schooldistricts than face-to-face preparation classes are.Costs vary widely, but they fall well under the hun-dreds of dollars a test-prep course would cost an indi-vidual student.

Massachusetts, for example, paid $200,000 to the

Princeton Review this school year to help up to 80,000high school students statewide prepare for the Massa-chussetts Comprehensive Assessment System,or MCAS.

On the west coast, the University of California sys-tem signed a three-year, $125,000 contract with Num-ber2.com, a division of Xap Corp., an education in-formation-management-systems company based inLos Angeles. Number2.com provides online SAT andACT preparation for all California high school studentsfor free.

Research on the results of online test preparation issparse, but preliminary results of one analysis showonline test prep can be one factor that helps improvestudent test scores.

For instance,75 percent of high school students who hadpreviously failed the Texas Assessment of Academic Skillsin the 211,000-student Houston school district improvedtheir TAAS reading scores by 29 percent on average in apilot test-prep program, according to TestU, whose ser-vices the Houston district uses.

Jo Beth Harris, the director of Houstons VirtualSchool, who administers the online test-prep program,says that last summer 50 seniors who previously failedthe TAAS spent time with the online test-prep programfor two hours a day.

The result the second time they took the TAAS?They all passed, Harris says.


26

Prepping

for

the

Big

Test

"You can take yourtime...withouthaving to frustrateanybody."

DAVIS HO, 15, Timber Creek High School, Orlando, Fla.


20/76


I

ndiana special education student Marvin

Stuller couldnt believe the boy reading

aloud on the TV set was him.xxxxxxxxxx

His teacher had videotaped

him at the beginning of 5th grade and at

the midway point.The before and

after scenes of the then-12-year-

old with speech difficulties were

dramatic: His reading level had

risen,and his voice had even lowered

and deepened by the later video.

Mom, I really improved, he told

his mother when he got home that night,

recalls his mother, Jenny Stuller of Elkhart,

Ind. She was impressed with how the video gave

her an intimate glimpse into her sons academic progress.

As educators face new federal requirements to test all

students annually in reading and mathematics in grades

3-8, states are experimenting with new ways of using

technology to evaluate the abilities of special education

students. Testing experts say that what educators learn

from tailoring assessments to the needs of special educa-

tion students could shape how they test regular students,

who have more subtle individual needs.

Special education students are likely to be the ca-

naries in the coal mine as educators experiment with

new technological assessments, says David Rose, the co-

executive director of the Center for Applied Special Tech-

nology, or CAST. The nonprofit group,based in Wakefield,

Mass., seeks to expand educational opportunities for stu-

dents with disabilities through the use of technology.

[Special education students] are the first to experience

ByBy Lisa Fine Goldstein

Spec Ed TechSparks IdeasTesting tools for children with disabilities

attract mainstream attention

. .


21/76

an oxygen deficit if a test doesnt work, Rose ex-plains. Its like when you test-drive a new car: Youdo it on rough terrain. You take it out to where it willbe tested.

Testing in the Margins

Many districts and an increasing number of statesare ramping up to launch computerized assessments,which experts say can be more efficient, use less labor,and provide instant scoring. The tests will offer thepotential for more sophisticated accommodations,such as the use of spell-checkers or audio featuresthat read questions aloud.

There was a time when schools would try to deteras many special education students from taking thetests as possibletell them to stay home on test dayso their [schoolwide] scores wouldnt be lowered,saysRobert Dolan, a senior research analyst at CAST.Now, all of those students have to be included, sostates want to do whatever they can to help themscore as high as possible.

At the same time, those students diff icul ties onsuch tests could bring to light broader problems:whether the tests are really evaluating the right ma-terial; if administration of the tests is biased; andwhether the technology presents more problems thanit overcomes.

For example, if a student is uncomfortable in frontof a camera, the videotaping approach may be coun-terproductive. Or if a student is a poor typist, com-puterized assessments might be more difficult thanpaper-and-pencil versions.

The [testing] revolution begins in the margins,Rose says. Special education students are being in-creasingly pushed toward the general education cur-riculum. How they adjust will teach us a lot about as-sessments.

In Indiana, for instance, educators use electronicportfolios to measure the progress of students with

disabilities. To do that, teachers collect videotape, au-diotape, and written materials on their students and

then scan that information onto CD-ROMs. Teachersreview the electronic portfolios to determine if childrenare ready to advance to the next grade.

Many general educators have looked on enviouslyat those alternative assessment methods, and some

have even adopted the practices themselves, saysRobert Marra, the state director of special educationfor the Indiana Department of Education.

General education teachers who have seen thetapes want to be able to do this for their children,Marra says.There are some who are using the video-tapes and other techniques already.

Whats most important, Marra says, is that theportfolios are helping teachers prepare students withdisabilities for regular assessments. The teachershave feedback right away on how a student is pro-gressing. Unlike with conventional testing methods,there is no lag time while a test is being scored.

The electronic portfolios also cut down on reamsof paperwork, one of the chief complaints of specialeducators, because the digital records help teachersdocument information on how a student is progress-ing as required by the federal Individuals with Dis-

abilities Education Act, Marra says.Whats more, teachers can use the electronic portfo-

lios to carry out IDEA-required obligations to give par-ents evidence of a childs academic progress.Teacherscan simply download the work from a computer ande-mail files to parents.

In the past, teachers and parents may have as-sumed that students with disabilities werent ableto take assessments, because they performed badlyon classroom tests that measure standards. But elec-tronic portfolios have changed that impression,Marra says.

Seeing proof of a childs abilities helps teachers andparents feel more comfortable with the prospect ofthe student being tested. Or, in some cases, a portfo-lio can be used in place of testing to show a studenthas met the standards for graduation.

If a parent says, I dont think my child can do that,

a teacher can turn around and show it to them onvideotape, Marra says. Also, the parents could use it


28

Spec.

Ed.

Tech

Sparks

Idea

s

SPECIAL PREPARATIONS

Although special education teachers are more likely than regular teachers to report that students use computers

in their classes, those same teachers, on average, are less likely to report that they have had professional

development on how to use computers in the classroom. New special education teachers feel less prepared

to use computers for teaching than do other novice teachers in their first year of teaching.

100

80

60

40

20

0

Percentofteachers

79

Teachers whose

students

use computers

in their classes

Teachers who have had

professional development

on the use of computers

for instruction in the

past 12 months

Novice teachers who

felt well or very well

prepared to use

computers for instruction

in their first year

Special education teachers

Non-special-education teachers

68 67

71

37

43

SOURCE: Education Weekanalysis of U.S. Department of Education, National Center for Education Statistics, Schools and Staffing Survey,1999-2000


22/76

to videotape their children at home doing somethingthe teacher didnt know the student could do.

It works both ways, he says.

World of Glitches

With states preparing to test so many more special ed-ucation students who would otherwise have been ex-empt from the process, some states are already comingup with innovative technological accommodations.

Oregon is devising a test that would allow hearing-impaired students to use American Sign Language.By clicking on either English or ASL, the studentscould choose to read the problems in English text orsee a pair of hands signing the questionsor evensplit the screen with both English and ASL.

The same concept could potentially be used for, say,Spanish-speaking students for whom English is asecond language.

Oregon is also working on a version of the test thatwould allow certain students with disabilities to hearthe questions read aloud by the computer. In some

cases, that feature could benefit students with read-ing disabilities or visual impairments.

Though the state has no plans to offer the option tostudents without disabilities, Pat Almond, the as-sessment project director, says one day the audio fea-ture could be used on tests taken by all students.Sheemphasizes that would not be the case, of course, ifthe test were designed to assess reading ability.

The prospect of having an audio capability on testsfor all students isnt financially out of sight, Almondadds, because the technology to do so is becomingmore widely available and affordable.

Meanwhile, Massachusetts has prepared versionsof its state assessments for students with disabilitiesthat would allow the use of text readers.About 1,000 Massachusetts students with reading

disabilities or visual impairments use the text read-ers for their classroom work; the accommodation is

allowed in their individualized education plans, orIEPs, which are required by federal law. Only thosestudents would be eligible to take large-scale assess-ments using the text readers.

Careful piloting of such programs is essential, saysDaniel J.Wiener,the states assessment coordinator forspecial populations. For example, text readers have atough time presenting math problems that includeGreek symbols,or equations within parentheses.

Theres just a world of glitches out there, Wienercautions. Its expensive to test these students. Youdont want to go down the wrong path.

Because of the high cost and other potential prob-lems, he doesnt foresee that text readers will ever beused by general education students.And, he says, it would not be appropriate to pro-

vide use of an electronic reader to a s tudent whodoes not have a disability that prevents him or her

from learning to read without use of a reader.

Show What They Know

For states gearing up to offer assessments online, theact of transferring a paper-and-pencil test to a comput-erized format would be like designing a building with nowheelchair ramps, wide hallways, or elevators, saysDolan ofCAST. Undoubtedly, retrofitting the traditionaltests would prove expensive and complicated,he says.

Still, Oregon is developing computerized tests inwhich teachers would set individualized parametersfor special education students, according to GeraldTindal, the director of behavioral research and teach-ing at the University of Oregons college of education.For example, students with disabilities who are poorreaders could take a math test with the reading leveladjusted to their own needs.

It seems to Tindal that such a strategy could oneday be used with all students to make sure theirknowledge of academic content isnt held hostage toan unrelated skill.

It would be great if we could increase the opportu-nity for students to show what they know, Tindalsays. Its an interesting time to proceed in the con-text of high-stakes testing.


30

Spec.

Ed.

Tech

Sparks

Idea

s


23/76


S

tudents in Carole Givens U.S. history class last

year took a quiz at the beginning of every

period.But they didnt line up at the

pencil sharpener to get ready. In-

stead,they fired up their laptops.

Givens, a 34-year teaching veteran who

teaches at the 1,500-student Varina High

School in suburban Richmond,Va., used online

quizzes as a way of reviewing each unit with her

students. A computer graded the quizzes and gave

her a running total on how each student was performing.

It was an incredible feature, says Givens, who knew

right away if she needed to review a lesson, based on how

her students had performed on the latest quiz.

Computerized testing is starting to catch on in class-

rooms across the country as teachers look for more efficient

ways to evaluate students. Some say their use of computer-

based quizzes or tests has saved them untold hours that

would otherwise have been spent grading by hand.

Yet a host of barriers exist that could prevent many

teachers from going down the path to computerized tests.

And even the teachers who are already using such tests

say the method has its limitations.

To begin with, many teachers dont know how to use

software to give tests and quizzes, says Rosemary Skeele,

an education professor at Seton Hall University, in South

Orange, N.J., who trains teachers to use technology in

their classrooms.

Educators also point out that computerized exams

could underestimate the knowledge of some students who

ByBy Michelle Galley

TheTeachersNewTestComputerized exams finding a niche

in classroom assessment


24/76

are not particularly skilled at using computers.Beyond those concerns, Skeele says most class-

rooms have only one or two computers, which is notenough to give every student a test. And in manycases, the computers that teachers use are so old that

they cannot support new test-writing programs.The systems in K-12 schools are just not sophisti-

cated enough, Skeele says.Those problems are not issues for Givens. Still, she

doesnt use computers to give regular exams.Computer access is not the reason: Givens district,

Virginias 42,000-student Henrico County publ icschools, supplies each student and teacher with his orher own laptop.

Training is not a problem, either: Each middle andhigh school in the district has a technology-supportperson on staff to train and assist teachers. This year,Givens fills that position

pencils down - technology's answers to testing.pdf

Documents