Development Development Research: Research:

A Key Strategy A Key Strategy for Improving for Improving

Teacher Teacher Education Education through through

TechnologyTechnology

SITE 2004SITE 2004

Tom Reeves 12 years of

Catholic school 1 year public high

school U.S. Army Vet School teacher in

Atlanta in 1970s The University of

Georgia since 1982

My earliest experiences in “learning” with technology began in 1953.

Goals Critique the state-of-

the-art of educational research.

Describe an application of “development research” to teacher education.

Encourage new thinking about why and how we do research.

Bad News

Most educational research has little impact on practitioners and yields few discernable benefits.

Oh...no!

You’ll find The Impact of Educational Research in Fiction.

The Failure of Educational Research

– Vast resources going into education research are wasted.

– They [educational researchers] employ weak research methods, write turgid prose, and issue contradictory findings.

The Failure of Educational Research

– Too much useless work is done under the banner of qualitative research.

– Qualitative research…. [yields] ….little that can be generalized beyond the classrooms in which it is conducted.

Research Methods

Case Study Ethnography Critical

Deconstruction Discourse Analysis Narrative Inquiry Correlation Analysis

of New Instrument

Research Methods Program Description Evaluation - Multiple

Methods Program Description Evaluation- Surveys Phenomenological

Case Study Ethnographic Case

Study

College of EducationThe University of Georgia

Ranked 27th of 187 education colleges in the USA

240 faculty members in 20 departments

4,700 students in 33,000 student university

Research Productivity 1997-2001Refereed Journal Articles (in-cites.com

U. of Wisconsin - 202 U. of Georgia - 201 U. of Michigan - 164 Indiana U. - 161 U. of Maryland - 146

Georgia vs. Wisconsin

$7,824

44,073

51%

49th

$8,604

42,232

78%

7th

Per pupil

Salary

HS Grad.

Ranking

Isn’t it time we put the PUBLIC back in publication?

U.S. Dept. of Education’s Position

“There’s been no improvement in education over the last 30 years, despite a 90 percent increase in real public spending per pupil.”

Promotes randomized controlled trials as used in medical research. Secretary of Education

Ron Paige

“What Works” Position “Once we have dozens or

hundreds of randomized or carefully matched experiments going on each year on all aspects of educational practice, we will begin to make steady, irreversible progress.”

NCLB funds “scientifically based research.” Robert Slavin

“It Won’t Work” Position Double blind experiments

impossible in education Implementation variance

reduces treatment differences

Causal agents are unspecified in education

Goals, beliefs, and intentions of students and teacher affect treatments

Medical knowledge is not applied in many cases David Olson

Randomized controlled trials are the only way

we’ll ever be able to prove “what works” in education!

Randomized controlled trials promotes

pseudoscience and will limit effective change!

American Evaluation Association

The priority given to randomized controlled trials “manifests fundamental misunderstandings about 1) the types of studies capable of determining causality, 2) the methods capable of achieving scientific rigor, and 3) the types of studies that support policy and program decisions. We would like to help avoid the political, ethical, and financial disaster that could well attend implementation of the proposed priority.”

Ellen Lageman argues that educational researchers, in a misguided effort to be “scientific,” have turned away from the pragmatic vision of John Dewey.

She attacks the excessive emphasis on quantitative measurement.

Kieran Egan argues that progressive ideas from Herbert Spencer, John Dewey, and Jean Piaget are responsible for the “general ineffectiveness” of our schools.

He also assails the notion that education can be improved through research as traditionally conceived.

Thomas Kuhn The Structure of Scientific Revolutions

"I'm not sure that there can now be such a thing as really productive educational research. It is not clear that one yet has the conceptual research categories, research tools, and properly selected problems that will lead to increased understanding of the educational process. There is a general assumption that if you've got a big problem, the way to solve it is by the application of science. All you have to do is call on the right people and put enough money in and in a matter of a few years, you will have it. But it doesn't work that way, and it never will."

Complexity of Interactions We cannot store up

generalizations and constructs for ultimate assembly into a network.

When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion.

Lee Cronbach

If Sisyphus were a scholar, his field would be educational research.

- David Laberee

Educational technology researchers are not doing much better than other educational researchers.

NCLB Requirements "every student is

technologically literate by the time the student finishes the eighth grade," and

"that technology will be fully integrated into the curricula and instruction of the schools by December 31, 2006."

Abundant technology has not led to extensive use of computers for “tradition-altering classroom instruction.”

The small percentage of computer-using instructors only use it to maintain existing classroom practices.

Teachers have legitimate concerns. Is it simple enough for

me to learn quickly? It it versatile? Will it motivate students? Is it aligned with skills I’m

expected to teach. Is it reliable? It it breaks, who will

help? Will it weaken my

classroom authority?

Ed. Tech Research Reality Isolated researchers conduct

individual studies rarely linked to a research agenda or concerned with any relationship to practice.

Studies are presented at conferences attended by other researchers and published in journals few people read.

Occasional literature reviews and meta-analyses are published.

Ed. Tech Research Reality Many instructional technology

studies claim to have predictive goals (testing theories) and use quasi-experimental designs with quantitative measures.

Research reviewers usually must reject 75 percent or more of the published studies to find the few worthy of further review or inclusion in meta-analyses.

Ed. Tech Research Reality Dillon & Gabbard’s 1998 literature

review of “Hypermedia as an Educational Technology” highlights problems with IT research.

Major conclusion: “Clearly, the benefits gained from the use of hypermedia technology in learning scenarios appear to be very limited and not in keeping with the generally euphoric reaction to this technology in the professional arena.”

Ed. Tech Research Reality Fabos & Young 1999 literature review

of “Telecommunications in the Classroom: Rhetoric Versus Reality” is another bad sign.

Major conclusion: “…many of the expected benefits of telecommunications [enhancing writing, multicultural awareness, and economic possibilities] are inconclusive, optimistic, and even contradictory.”

CSLP 2003 Meta-analysis: “How Does Distance Education Compare to Classroom Instruction?”.

a very small but positive mean effect size for interactive distance education over traditional classroom instruction on student achievement

small negative effect for retention rate

Distribution of Effect

Sizes

-3

-2

-1

0

1

2

3

1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321

Distribution of Effect Sizes for Achievement Outcomes

Magnit

ude o

f Eff

ect

Siz

e

Hedges’g

Effect Sizes Ordered by Magnitude

325 independent outcomes (achievement)Hedges’ g = +0.0122, p < .001

Sir John Daniel - UNESCO… the futile tradition of comparing test performances of students using new learning technologies with those who study in more conventional ways…is a pointless endeavor because any teaching and learning system, old or new, is a complex reality. Comparing the impact of changes to small parts of the system is unlikely to reveal much effect and indeed, “no significant difference” is the usual result of such research.

Todd OppenheimerOur [American] desperation for objective information [is] illustrated nowhere more gorgeously than in the field of education. I am speaking of our tendency to promote any new concept by invoking volumes of quantitative “research” that ostensibly proves its value. …technology advocates have played it expertly when it comes to claims about what computers will do for student achievement. As it turns out, the vast bulk of their research is surprisingly questionable.

Chewing Gum More Effective than Interactive Multimedia CD-ROM

Dr. Ken Allen at NYU wanted to compare CD-ROM with lectures

Wrigley’s wanted to fund chewing gum study

Combined study Gum chewers = B-

Abstainers = C+ CD-ROM no better

Media comparison studies are akin to comparing copper bracelets with voodoo dolls as medical cures.

Experimental approaches to educational research won’t work!

MedicalCures

GeneticsResearch

Basic Applied

Te

c h n o l o g y

“Pasteur’s Quadrant” approach to research is needed (Stokes, 1997).

Considerations of use?

No Yes

Quest for fundamental understanding?

No

Yes

Bohr

Research is inspired by:

Edison

Pasteur

Good News There are new strategies for conducting “development research” that can improve our research so that it can become a socially responsible enterprise.

Thankgoodness!