Scaffolding Students' Reflection for Science Learning

by

Elizabeth Anna Davis

B.S.E. (Princeton University) 1989M.A. (University of California, Berkeley) 1994

A dissertation submitted in partial satisfaction of the

requirements for the degree of

Doctor of Philosophyi n

Education

in the

GRADUATE DIVISION

of the

UNIVERSITY OF CALIFORNIA, BERKELEY

Committee in charge:Marcia C. Linn, Chair

Barbara Y. WhiteMichael J. Clancy

Spring 1998

The dissertation of Elizabeth Anna Davis is approved:

Chair Date

Date

Date

University of California, Berkeley

Spring 1998

Table of Contents

List of Figures......................................................................................................................List of Tables........................................................................................................................Acknowledgments.............................................................................................................

PART ONEChapter 1: IntroductionIntroduction and Design ..................................................................................................

Knowledge Integration.........................................................................................Knowledge Integration and Conceptual Change............................................Rationale for Reflection Prompts.......................................................................The Scaffolded Knowledge Integration Framework and Prompting.........

Hypotheses and Results....................................................................................................Investigating Prompts for Reflection ................................................................Investigating Focus of Reflection.......................................................................Investigating Students' Beliefs ...........................................................................

Structure of the Dissertation ...........................................................................................

Chapter 2: Research FrameworkIntroduction and Rationale.............................................................................................Research on Knowledge Integration .............................................................................

Research in the CLP/KIE Classroom .................................................................Research in Programming Classes.....................................................................The Dissertation Research ...................................................................................

Background .........................................................................................................................Reflection and Reflection Prompts....................................................................Students' Beliefs about Science and Learning.................................................Design of Guidance in Technology-Based Learning Environments ..........

Chapter 3: Preliminary StudiesIntroduction and Rationale.............................................................................................Study 1..................................................................................................................................

Methods ...................................................................................................................Results: Justification and Synergy......................................................................

Study 2..................................................................................................................................Methods ...................................................................................................................Results: Project Completion and Principled Knowledge Integration ........

Study 3..................................................................................................................................Methods ...................................................................................................................Results: Prompt Response Characterization....................................................

Discussion............................................................................................................................

Chapter 4: MethodsIntroduction........................................................................................................................Research Context................................................................................................................

The Research Paradigm........................................................................................The Students...........................................................................................................The Curriculum.....................................................................................................The Software...........................................................................................................The Assessments....................................................................................................

Study Design .......................................................................................................................Prompting Conditions..........................................................................................Pairing Students.....................................................................................................Data Sources and Outcome Measures ...............................................................

Investigations and Analyses............................................................................................Effects of Reflection Prompts...............................................................................Focus of Reflection ................................................................................................Beliefs about Science and Learning Science.....................................................Synthesis..................................................................................................................

PART TWOChapter 5: Effects of Reflection PromptsIntroduction and Rationale.............................................................................................Methods ...............................................................................................................................

Data Sources............................................................................................................Outcome Measures................................................................................................Analyses...................................................................................................................

Project Quality Results......................................................................................................General Project Quality Measures......................................................................Critique Measures..................................................................................................Knowledge Integration Measures ......................................................................

Quiz Results........................................................................................................................Relationships among Measures and Productivity of Experiences..........................Summary and Implications.............................................................................................

Chapter 6: The Focus of ReflectionIntroduction and Rationale.............................................................................................Methods ...............................................................................................................................

Data Sources and Outcome Measures ...............................................................Analyses...................................................................................................................

Reflection Prompts and Project Quality Results.........................................................Reflection in Response to Reflection Prompts................................................Prompt Responses and Project Characteristics................................................

Summary and Implications.............................................................................................

Chapter 7: The Role of Students' Beliefs about Science and LearningIntroduction and Rationale.............................................................................................Methods ...............................................................................................................................

Data Sources............................................................................................................Outcome Measures................................................................................................Analyses...................................................................................................................

Results..................................................................................................................................Characterizing "Productive" and "Less Productive" Beliefs........................Comparing Pre- and Post-test Results ...............................................................Relating Epistemological Dimensions..............................................................Relating Beliefs to Performance .........................................................................Relating Beliefs and Reflection...........................................................................Comparing Males' and Females' Beliefs ..........................................................

Discussion............................................................................................................................Summary and Implications.............................................................................................

Chapter 8: Synthesizing the Roles of Prompts, Reflection, Cites, and BeliefsIntroduction and Rationale.............................................................................................Methods ...............................................................................................................................Results..................................................................................................................................

Predicting the Quality Measures.........................................................................Identifying Groups of Students...........................................................................Investigating Groups of Students.......................................................................

Discussion and Implications ...........................................................................................Overview of Reflection Prompts' Effects..........................................................Relating Autonomy, Principle-citing, and Elaboration.................................Considering Prompt Condition and Lack of Reflection................................Investigating the Complex Role of Reflecting on Content...........................

PART THREEChapter 9: Discussion and ConclusionsIntroduction and Rationale.............................................................................................Discussion............................................................................................................................

Expanding the Repertoire of Ideas .....................................................................Identifying Weaknesses in Knowledge.............................................................Autonomous Reflection ......................................................................................Synthesis..................................................................................................................General Findings....................................................................................................

Implications ........................................................................................................................Future Directions...............................................................................................................

Prompting, Reflection, and Knowledge Integration......................................Beliefs .......................................................................................................................Technology..............................................................................................................

References............................................................................................................................

APPENDIXESAppendix A: The "All The News" Article (Study 2)..................................................Appendix B: Activity Prompts "Letter" Document (Study 2) ..................................Appendix C: Self-Monitoring Prompts "Letter" Document (Study 2) ...................Appendix D: Prompts Used in "All The News" (Study 3).......................................Appendix E: All The News Article (Dissertation Study)...........................................Appendix F: Evidence (Dissertation Study) .................................................................Appendix G: Student Work.............................................................................................Appendix H: Relevant Questions on Beliefs Test ......................................................Appendix I: Prompts in All The News (Dissertation Study)....................................Appendix J: Interview 1....................................................................................................Appendix K: Interview 2..................................................................................................Appendix L: Interview 3...................................................................................................Appendix M: Questions from Pre-, Post-Project Quiz (Dissertation Study)..........Appendix N: Examples of Levels of Coherence..........................................................Appendix O: Trends in the Data.....................................................................................

List of Figures

Figure 2Ð1: Guidance in the Knowledge Integration Environment ......................Figure 3Ð1: Instructions for Clothing Design for Two Conditions in Study 1......Figure 3Ð2: Examples of Prompts ...................................................................................Figure 3Ð3: Instructions for Letter for the Two Conditions in Study 2..................Figure 3Ð4: Principled Knowledge Integration............................................................Figure 6Ð1: Reflection Focus for Directed and Generic Prompts.............................Figure 6Ð2: Reflection Focus for Planning, Monitoring, Generic Prompts...........Figure 7Ð1: Mean Dimensions Scores on Pre- and Post-Tests..................................Figure 7Ð2: Mean Change in Autonomy Scores for each Autonomy Group.......Figure 7Ð3: Mean Change in Strategy Scores for each Strategy Group...................Figure 7Ð4: Mean Change in Process Scores for each Process Group......................Figure 8Ð1: Mean Values of Coherence for Autonomy Groups..............................Figure 8Ð2: Mean Values of Coherence for Autonomy Groups by Condition.....Figure 8Ð3: Mean Values of Coherence for Principle-Citing Groups.....................Figure 8Ð4: Mean Values of Overall Critique Quality for Elaboration Groups

by Condition.....................................................................................................................Figure 8Ð5: Mean Values of Guidelines Quality for Uncodable Groups by

Condition..........................................................................................................................Figure 8Ð6: Mean Values of Overall Critique Quality for Non-Reflective

Groups by Condition......................................................................................................Figure OÐ1: Mean Values of Coherence for Principle-Citing Groups by

Condition..........................................................................................................................Figure OÐ2: Mean Values of Guidelines Quality for Principle-Citing Groups by

Condition..........................................................................................................................Figure OÐ3: Mean Values of Critique Quality for Uncodable Groups by

Condition..........................................................................................................................

List of Tables

Table 2Ð1: Dimensions of Student Beliefs...................................................................Table 3Ð1: Coding Prompt Responses............................................................................Table 3Ð2: Characteristics of Comments in Prompt Responses...............................Table 4Ð1: The Directed and Generic Prompt Conditions.........................................Table 4Ð2: Pairing Students based on Beliefs ...............................................................Table 5Ð1: Outcome Measures for Student Projects ...................................................Table 5Ð2: Class Periods' Project Scores.........................................................................Table 5Ð3: Significant Differences in Class Periods' Project Scores.........................Table 5Ð4: Quality of Guidelines.....................................................................................Table 5Ð5: Mean Number of Cites in Claim Notes (Claims 1 and 3)......................Table 5Ð6: Mean Number of Cites in Letters................................................................Table 5Ð7: Coherence of Students' Ideas.......................................................................Table 5Ð8: Correlations among Major Project Quality Measures............................Table 5Ð9: Correlations between Principle Cites, Project Quality Measures .........Table 6Ð1: Coding Reflection in Prompt Responses...................................................Table 6Ð2: Exemplars of Reflection Types.....................................................................Table 6Ð3: Degree of Elaboration of Reflection Prompt Responses.........................Table 6Ð4: Mean Proportion of each Focus of Reflection..........................................Table 7Ð1: Dimensions of Beliefs Investigated............................................................Table 7Ð2: Descriptive Statistics for the Dimensions .................................................Table 7Ð3: Correlations for Pre- and Post-Test Scores ................................................Table 7Ð4: Correlations between Dimensions .............................................................Table 7Ð5: Correlations for Dimensions and Performance.......................................Table 7Ð6: Correlations for Pairs' Beliefs and Project Quality Measures ...............Table 8Ð1: Regression Coefficients for Predicting Overall Critique Quality for

Directed Prompt Condition ..........................................................................................Table 8Ð2: Regression Coefficients for Predicting Overall Critique Quality for

Generic Prompt Condition ...........................................................................................Table 8Ð3: Predicting Overall Critique Quality for Directed and Generic

Prompts.............................................................................................................................Table 8Ð4: Regression Coefficients for Predicting Coherence for Directed

Prompt Condition...........................................................................................................Table 8Ð5: Regression Coefficients for Predicting Coherence for Generic

Prompt Condition...........................................................................................................Table 8Ð6: Predicting Coherence for Directed and Generic Prompts ......................Table 8Ð7: Regression Coefficients for Predicting Guidelines Quality for

Directed Prompt Condition ..........................................................................................Table 8Ð8: Regression Coefficients for Predicting Guidelines Quality for

Generic Prompt Condition ...........................................................................................Table 8Ð9: Predicting Guidelines Quality for Directed and Generic Prompts.......Table 8Ð10: Mean Differences in Coherence for Autonomy Groups......................

Table 8Ð11: Mean Differences in Coherence for Principle-Citing Groups.............Table 8Ð12: Mean Differences in Guidelines Quality for Principle-Citing

Groups...............................................................................................................................Table 8Ð13: Mean Differences in Overall Critique Quality for Elaboration

Groups...............................................................................................................................Table 8Ð14: Mean Values of Overall Critique Quality for Elaboration Groups

by Condition.....................................................................................................................Table 8Ð15: Mean Values of Guidelines Quality for Uncodable Groups by

Condition..........................................................................................................................Table 8Ð16: Mean Values of Overall Critique Quality for Non-Reflective

Groups by Condition......................................................................................................Table OÐ1: Boys' and Girls' Autonomy Beliefs, by Pre-test Scores..........................Table OÐ2: Boys' and Girls' Process Beliefs, by Pre-test Scores.................................Table OÐ3: Mean Values of Guidelines Quality for Principle-Citing Groups by

Condition..........................................................................................................................Table OÐ4: Mean Values of Critique Quality for Uncodable Groups by

Condition..........................................................................................................................

Acknowledgments

I would like to thank the eighth grade students who participated in this

research. Without them, this dissertation would not exist. Their teacher,

Doug Kirkpatrick, has helped me in more ways than I could ever

nameÑespecially by being such a wonderful teacher and role model. I am

grateful to my advisor, Marcia Linn, whose insight and experience have been

invaluable throughout my years of working with her. Because of these two

exceptional mentors, I leave graduate school far more able to help change the

field of education. I also want to thank my other committee members,

Barbara White and Mike Clancy, for their interest, guidance, and support over

the years I've been at Berkeley. In different ways, they both have helped me

remember to notice both the forest and the trees. My colleague and friend

Philip Bell has helped me with this research at every step of the way, and for

his help in thinking about this work I am more than thankful. I also

appreciate the ongoing help of my other dissertation group colleagues. In

particular, thank you to Dawn Rickey for providing excellent suggestions at

all levels of this dissertation and the research that has preceded it. I've also

been lucky to be a part of three different research groups over the last six

years, and I appreciate the ways in which the members of these groups have

helped me develop my ideas. Last, I want to thank my family and nearly-

family. To Garrett Scott and the rest of the book/camping club, to Tom and

Betsy Davis, and to Dick and Kathy Davis: Thank you all for the support

you've given me while I've worked on this research. You've helped me

financially, emotionally, psychologically, intellectually, and physically, and I

can't imagine doing this without you.

Scaffolding Students' Reflection for Science Learning

Copyright 1998

by

Elizabeth Anna Davis

Abstract

Scaffolding Students' Reflection for Science Learningby

Elizabeth Anna Davis

Doctor of Philosophy in EducationUniversity of California, Berkeley

Professor Marcia C. Linn, Chair

Research in recent decades has emphasized the importance of reflection forstudents learning science, but educators have not reached consensus on themost effective ways to promote reflection, nor has a mechanism explainingthe effects of reflection been accepted. Furthermore, many have put forthtechnology as a vehicle for improving student learning, yet others discountits ability to facilitate real reflection. This research determines whetherreflection prompts promote knowledge integration for students working onscience projects and what level of prompt specificity best supports students inthat endeavor. The Knowledge Integration Environment (KIE) affordsinvestigation of computer-delivered prompts for students completingcomplex projects. This research takes place in the context of the KIE softwareand curriculum as used in an eighth grade physical science class.

Pilot research on prompts indicated that focusing students on reflectionsignificantly increased knowledge integration. A basic question unansweredby the pilot research was: As students work on projects like those used in KIE,do they merely need to be prompted to reflect, or do they need guidance indetermining what to reflect about? The prompts contrasted in this researchdiffer in their specificity. Some students received directed prompts aimed atfostering planning and self-monitoring, while others received generic 'stopand think' prompts.

The investigations describe the gross effects of reflection prompts, thenattempt to identify a mechanism behind those effects through characterizingthe kinds of reflection they elicit and the beliefs about science and learningscience of individuals using the prompts. I argue that by engaging inreflection, students identify weaknesses in their knowledge and then aremore ready and able to link and distinguish their ideas. Generic prompts aremore effective than directed prompts at engaging students in theseknowledge integration processes. Autonomous students benefit most fromgeneric prompts for reflection.

This research contributes to teaching practice, technology design, and theeducational and cognitive research literature. The success of generic prompts,in particular, indicates that instructional designers should concentrate onbuilding learning environments that provide opportunities for students toreflect, and allow students to take responsibility for directing their ownreflection autonomously.

Chair

1

Chapter 1: Introduction

Introduction and Design

This research is designed to determine (a) if reflection prompts promoteknowledge integration for students working on science projects and (b) whatlevel of prompt specificity best supports students in that endeavor. Here, I usethe term "reflection" to refer to both metacognition and sense-making. Forexample, reflection can focus on one's own thinking or goals, or on contentitself. I look at reflection facilitated by sentence-starter prompts that explicitlycall for this special kind of thinking. For example, a reflection prompt mightsay, "In thinking about how it all fits together, we're confused aboutÉ"; atypical response in a science class studying energy conversion would be, "whyblack gets hotter than white." My goals for this research are twofold. Not onlydo I hope to develop ways to encourage knowledge integration throughfacilitating reflection, but I also hope to improve educational practice andinform the design of instruction through a synergy of technology andpedagogy.

This work is situated in the perspective of knowledge integration (Linn, 1995;Linn & Eylon, 1996). When I say "knowledge integration," I mean, at its mostbasic level, the process of linking scientific ideas together to develop a robust,coherent, conceptual understanding. Knowledge integration represents aview of how students learn. Specifically, in the current work, I am interestedin how students learn science. However, the knowledge integrationperspective can be applied to learning in any domain. The knowledgeintegration research is situated in the larger literature on conceptual changeand has evolved over the past decade. Reflection provides one method forfostering knowledge integration by helping students to expand theirrepertoire of ideas, differentiate among them, and make connections betweenthem. With reflection prompts, I hope to foster the knowledge integrationprocesses that happen naturally for some students and help students learn toengage in these processes autonomously over time. Most students need manyopportunities to reflect to build cohesive, coherent accounts of new material.

The Knowledge Integration Environment (KIE) affords investigation ofcomputer-delivered prompts for students completing complex projects. Thisresearch takes place in the context of the KIE software and curriculum, in aneighth grade physical science classroom using the Computer as LearningPartner (CLP) software and curriculum, as well. KIE projects scaffold students

2

in making sense of complex information from the World Wide Web.Students think critically about "evidence" and use it to build arguments. Inmore traditional science classes, reflection is less important because studentscan succeed without it; in the CLP/KIE classroom, though, reflection plays aparamount role, because it helps students set goals and monitor (andimprove) their understanding. (CLP and KIE will be described in greater detailin Chapter 2.)

This research on reflection prompts has grown out of the extensive history ofthe idea of reflection. Today's educational researchers generally accept theneed for reflection in students' learning, although it is not alwaysincorporated into instructional practice. My work builds on research showingthat when students reflect on their ideas they generally produce betterproducts. I postulate that prompting students to reflect can set this process inmotion. The process of reflecting on ideas may help students identifyweaknesses with their current understanding and thus motivate them torevisit, test, and reformulate the links and connections among their ideas,leading to more coherent, robust, and integrated understanding.

This research describes effects of prompts, identifying features of particularlypromising ones. I ask, Do students merely need to be prompted to reflect, ordo they need guidance in determining what to reflect about? This researchalso delineates mechanisms that explain how the prompts facilitate reflection,and how reflection, in turn, is linked to knowledge integration. I ask, Do allfocuses of reflection lead to the same results? I also investigate the role ofindividual students' beliefs about science and learning science in theirreflection and learning. I ask, Do all students benefit equally from the sametype of prompts for reflection?

To investigate these questions I contrast two types of reflection prompts. Thefirst type, called generic prompts, represents a view that asking students to"stop and think" will encourage reflection. The second type, called directedprompts, assumes that a generic request for reflection is insufficient, and thatstudents should instead be provided with hints indicating potentiallyproductive directions for their reflection. An example of a generic prompt is,"Right now we're thinkingÉ." An example of a directed prompt is, "To do agood job on this project, we need toÉ." The rationale for each of these types ofprompts is given in Chapter 2.

Knowledge Integration

I view science learning as a process of integrating ideas (diSessa, 1988; Linn &Eylon, 1996). To integrate ideas, students add information, reorganizeinformation, promote some ideas, and demote other ideas. The ideas students

3

bring to science class get linked to new ideas, combined with each other, andreorganized. Through the process of knowledge integration, students developa coherent, robust understanding of science concepts. In this process, studentsexpand their repertoire of ideas, discriminate between ideas, and reorganizethe links among them (Linn & Eylon, 1996). Expansion of the repertoire ofideas is necessary but not sufficient (Hsi, 1997). Some students are content toaccept any idea presented, without consideration of whether that idea makessense on its own or of how it fits with other ideas. These students are lesssuccessful at developing an understanding of scientific phenomena than arestudents who reflect on new ideas, working to understand each concept;distinguish between ideas; determine places where links can be made amongthe ideas, thus improving their knowledge integration; and identifyweaknesses in their current knowledge. Let us consider each of theseprocesses in turn.

First, when students expand their repertoire of ideas, it is more useful if theywork to understand a new concept before they add it. For example, if a studentis exposed to a new idea (say, that black absorbs light), the student mustdevelop an understanding of that concept before it can be linked appropriatelyto other ideas.

Students must distinguish between ideas. For example, students may start outconsidering the ideas "black attracts heat" and "black absorbs light" asequivalent. Later, they may come to recognize the importance of the nuancesof scientific language, and may distinguish between attracting and absorbing.They may now consider "black absorbs heat" and "black absorbs light" asequivalent, though. Eventually, they may distinguish between the concepts(and the words) "light" and "heat," and may come to believe that black objectsactually absorb light.

Students also need to make connections among ideas. For example, they maybelieve that black absorbs light. They might also know that black objects gethotter than white objects. Connecting these ideas, though, can be difficult. Auseful connection in this case would be, "Absorbed light is converted intoheat energy." Through a connection like this students can come to have amore robust understanding of scientific phenomena.

To reach that point, though, they must identify a weakness in theirknowledge. Many students are perfectly content with "knowing" that aphenomenon is "true." To them, knowing that black objects get hotter issufficient. However, in the CLP/KIE classroom, students are required to makescientific explanations; descriptions of phenomena are not enough. Oftenthrough one-on-one conversations with students, we can help them identify

4

where their knowledge has weaknessesÑthat is, where links, distinctions, orreorganizations could be made or where new ideas could be added.

The weakness might simply be a place where a new idea can be added to therepertoire. For example, a student may notice for the first time that one getswarmer wearing black during the day, but experiences no temperature changeat night. They might add the idea "Black objects do not get hotter when theyare in the dark." Or, the student might identify where ideas should bedistinguished from one another, as in the distinction between light and heatdiscussed above. The weakness might alternatively be a place where a linkcan be made between two ideas. A direct link might be made between theideas that black T-shirts and black asphalt both get hot on a summer day, forexample. A more sophisticated link might connect one or both of those ideaswith the knowledge that dark colored objects are harder to see at night.Through identifying where links like these could be made, students candevelop new models or explanations for phenomena. For example, thestudent might link the two principles "black absorbs light" and "absorbedlight is converted to heat energy" or the student might link the idea "blackgets hotter" and the principle "black absorbs light." Identifying where theseweaknesses are in the current knowledge will help the student develop acoherent, integrated understanding of energy conversion by illuminating theneed to engage in more thought.

The research presented in this dissertation will investigate ways in whichreflection can help students engage in these knowledge integration processes.

Knowledge Integration and Conceptual Change

How does knowledge integration relate to conceptual change? Through theprocesses of knowledge integrationÑdistinguishing between ideas, linkingideas, and identifying weaknesses in one's knowledgeÑlearners revise theirunderstanding of scientific concepts and develop a coherent, integratedunderstanding. Knowledge integration, then, is a mechanism of conceptualchange.

Students often have non-normative understandings of science concepts.These conceptions are generally grounded in their phenomenologicalexperience (Inhelder & Piaget, 1958). For example, we have all had theexperience of sitting down on a metal chair and having it feel cold. Studentsmay believe that the chair is, in fact, colder than, say, a wooden table, even ifthe two objects have been in the same surroundings for several years. Theliterature would suggest many different ways in which students might cometo understand that "objects in the same surroundings are at thermalequilibrium," and that "some objects feel colder than others in part because of

5

differences in the rate of heat flow through the materials." Accompanyingthese different views of how conceptual change happens cognitively aredifferent instructional approaches to achieve conceptual change. Here Icontrast the theoretical and instructional implications of many of these viewswith the implications of the knowledge integration view.

Replacement of Misconceptions

Some researchers would consider students' non-normative beliefs as"misconceptions" to be replaced or eradicated. Much of traditional instructiontakes the approach that telling the students more scientifically normativeideasÑpresenting them with an explicitly conflicting "correct" ideaÑwouldhelp them develop an improved understanding (McCloskey, 1983). Studentsmight be "induced" to give up their intuitive beliefs and adopt instead thescientifically accepted ideas.

In the thermal equilibrium example described above, a misconceptionsapproach would view the "problem" as lying with students' understanding ofenergy at a fairly global level, and instruction might involve telling studentsthat objects in the same surroundings are at thermal equilibrium. As a resultof instruction like this, students might adopt the scientific explanation, atleast temporarilyÑbut their understanding would not be likely to be robust,because they would not link the new information with their existing ideas.

Conflict

Findings regarding conflict as a mechanism for conceptual change have beeninconclusive, though Chan and colleagues identify maximal conflict as mostsuccessful for those students who have a propensity toward knowledge-building, as opposed to assimilation (see Chan, Burtis, & Bereiter, 1997, for areview of this literature, as well). Piaget maintained that children learn whentheir understanding becomes disequilibriatedÑthough he did notrecommend conflict as a method for encouraging learning (Piaget, 1952).Constructivist theory holds students' experience-based conceptions assomething to build upon, rather than something to replace. In this view,learners should struggle with ideas and build upon their existing knowledge.(See Smith, diSessa, and Roschelle [1993] for a discussion of these issues.)

Dissatisfaction with Ideas

Strike and Posner (1992) claim that for conceptual change to take place,students must first be dissatisfied with their current idea. They must also findthe new idea intelligible; under this constructivist approach (unlike, perhaps,

6

the misconceptions approach), students will not adopt a new idea if they donot understand it. But furthermore, they must see the new idea as bothplausible and as a fruitful replacement for their old idea (Strike & Posner,1992). In the knowledge integration approach, on the other hand, the old ideawould not be replaced but instead would be applied less and less frequently.White and Gunstone (1989) extend Strike and Posner's view, adding explicit,sustained reflection to the necessary conditions for conceptual change.

How would this very rational approach play out in the thermal equilibriumexample? Students would first need to decide, for some reason, that materialscannot in fact be "naturally cold." They might, for example, be asked to thinkof situations in which that explanation is inadequate. They would need at thesame time to have under consideration some alternative explanationÑforexample, that rate of heat flow might be an explanation. But in order todetermine that rate of heat flow is a plausible and fruitful explanation, theymust first make sense of the concept of rate of heat flow itself; it must be anintelligible idea to them. They might make predictions about the rate of heatflow through different materials, doing a laboratory experiment where theyattach balls of wax to the ends of bars of different materials, and then heat thebars. Through experiments like this they might come to consider rate of heatflow as intelligible; the next step would be to identify it as a plausible andfruitful alternative. Rate of heat flow might be a plausible explanation ifstudents understand that humans are heat sources and that a bench mightfeel cold if heat is flowing quickly from the body into the material. And, theidea might be adopted as fruitful when students come to recognize both theintractability of their intuitive ideas and the appeal of new ones.

Theory Change

Kuhn's (1970) ground-breaking work on the development of new knowledgein science provides an over-arching analogy of a "revolutionary" rather than"evolutionary" stance. Carey (1988) postulates that students have theoriesabout science and that those theories need to be revised. Unlike themisconceptions camp, though, which anticipates theories being replacedthrough didactic instruction, Carey argues that theories are changed through aseries of differentiations and coalescences. Instructionally, this would meanthat students would be engaged in activities designed to help themconceptually understand the topic at hand. For example, Carey and hercolleagues have engaged students in a several-week inquiry-based curriculumdesigned to help students revise their ideas about the nature of science, in thecontext of thinking about whether yeast is alive (Carey, Evans, Honda, Jay, &Unger, 1989).

7

Analogously, we could imagine engaging students in a series of lab activitiesto help them recognize that objects in the same surround are at the sametemperature. They might also do labs designed to help them understand rateof heat flow as a possible explanation, and to identify themselves as heatsources. The activities would not necessarily differ very much from thoseused in the CLP/KIE classroom. However, we view students as linking ideasrather than revising theories (diSessa, 1988; Linn, 1995). Driver and hercolleagues argued against the theory revision stance toward conceptualchange, claiming that students' knowledge is too tacit and situated toconstitute theories (Driver, Asoko, Leach, Mortimer, & Scott, 1994).

Category Change

Chi and her colleagues postulate that students' ontological categories (nottheir ideas or theories themselves) need improvement (Chi, 1992, 1993). Chiholds that people have ontological categories into which they put ideas andconcepts. Sometimes, learning is relatively easy because it does not require ashift in ontological category. An example would be when young childrenreorganize their knowledge in recognition that humans are animals ratherthan representing a separate form of life. Chi would not consider this kind oflearning to be conceptual change. Other times, though, students need eitherto move an idea into a different existing category or to develop a new categoryinto which to put the idea. An example of a shift in ontological category likethis is given in Chi and Slotta's work in electricity. They found that studentscould learn about the topic of electricity much more easily if they were firsttrained in the category of "constraint-based interactions" (Slotta & Chi, 1996).In learning about heat, as with electricity, students are likely to have a"material substance" view that might cause difficulty with learning.

From this perspective, in the thermal equilibrium example students might betrained in constraint-based interactions to help them move away fromthinking of heat as a substance and toward recognizing equilibrium as one ofmany examples of an "equilibration process." Isomorphic problems might beused in this training. However, our work in CLP indicates that some ideaslike this are actually strong foundations for students to build on, even whenthey are not scientifically normative (Linn & Muilenburg, 1996).

Restructuring Ideas

All of these represent potentially useful ways of thinking about conceptualchange. My work, however, is based in the knowledge integration view, inwhich students need to distinguish between ideas, link ideas, restructureideas, and identify weaknesses in their knowledge to develop a truly robust,

8

coherent, integrated understanding of science concepts. In knowledgeintegration, students' initial conceptions are viewed as productive ideas to bebuilt upon, rather than misconceptions to be eradicated. We address students'existing conceptions through scaffolding them. Sometimes ideas need to bedistinguished from one another, and sometimes new links need to be madebetween existing ideas. New ideas also, of course, need to be added to therepertoire. Instructionally, students should be exposed to a repertoire ofalternative models (Linn, diSessa, Pea, & Songer, 1994). This gives themaccess to intermediate (often qualitative) models that could more easily relateto their alternative conceptions, which might then be refined and evolvedinto more sophisticated models (White, 1993). Sometimes old, non-normative ideas fall out of use when new ideas are added to the repertoireand ideas are linked in new ways; this does not mean that the old, less fruitfulor satisfying ideas are gone from the students' repertoire. Often even expertscientists rely on non-normative ideas in certain contexts (Linn &Muilenburg, 1996).

In the knowledge integration approach, learners are not viewed as havingtheories about science concepts nor is the learning process viewed as one inwhich categorical shifts are at the basis of learning. Students' understandingsare instead viewed as webs of loosely- and tightly-linked ideas, whichundergo a process of integration and restructuring (Linn et al., 1994; Smith etal., 1993).

This dissertation will investigate how reflection can help foster knowledgeintegration to help students develop a more coherent understanding ofscience ideas.

Rationale for Reflection Prompts

How does reflection help students integrate their knowledge? Recall that I usethe term "reflection" to refer to something with both metacognitive andsense-making components. Metacognition has been defined as "knowledge ofcognition" and as "regulation of cognition" (Brown, Bransford, Ferrara, &Campione, 1983). I break metacognition into three differentiable pieceshaving to do with both self-monitoring and self-regulation: reflection onone's thinking, learning goals, and behavioral goals. For students, one'slearning goals may concern the product of instruction while one's behavioralgoals may concern the activities inherent in that instructional contextÑthatis, the process of participating in the instruction. (Students who reflect ontheir behavior might plan to "work hard" or "listen to the teacher," forexample, while those who reflect on their learning goals might plan to "do agood report." Students who instead focus on their thinking might anticipate

9

identifying "how this relates to the lab we just did." Of course, these focusesgenerally occur in combinations, rather than in isolation.)

Experts also reflect on their content knowledge and engage in sense-making.(Students reflecting on content might wonder why black absorbs light, forexample; they might then work to understand this question.) People whoreflect on content integrate their knowledge and are able to apply it. However,metacognitive activity is also necessary for knowledge integration, sincelearners must identify weaknesses in their current knowledge before theybegin to distinguish between and link ideas. Reflection on cognition andreflection on content are closely tied.

My hypothesis is that reflection, used generally, can help students undergothe processes of distinguishing between ideas, making links between them,and identifying weaknesses in their current knowledge. Reflecting on one'sthinking, one's learning goals, and (to some extent) one's behavioral goalsprimes students for engaging in these processes; some reflection is moreproductive than other reflection because it is more likely to foster knowledgeintegration. A student who ponders his or her level of understanding, whoplans for the day's work, and who considers ways in which the current ideasare or are not well-understood will be more apt to engage in knowledgeintegration than a student who merely forges full-speed ahead withoutengaging in these metacognitive activities or who only reflects on how theybehave, rather than how they think. In particular, students who recognizeweaknesses in their current understanding are more apt to engage in theother knowledge integration processes of linking and distinguishing ideas.But, being primed for knowledge integration is not the same thing asengaging in it. When students take the opportunity to reflect on theirunderstanding of science concepts and ideas, they identify places where newideas could be added and links and distinctions should be made.

Sentence-starter reflection prompts in KIE are used to promote explicitly thismetacognitive and sense-making reflection. An interesting feature ofreflection prompts is that students can interpret them in any way they wish.Because of this characteristic, reflection prompts are intrinsicallymetacognitive and only potentially do they provide a locus for sense-making:Students must interpret them in a certain way for them to be used as a sense-making opportunity. Reflection prompts represent the only explicit call formetacognition that students receive in KIE projects, though sense-making isan expectation throughout the projects.

A more specific rationale for the reflection prompts contrasted in thisresearch will be given in Chapter 2.

10

The Scaffolded Knowledge Integration Framework and Prompting

KIE projects require students to engage in sustained reasoning. Our goal withKIE projects is to help students develop an improved conceptualunderstanding and an improved ability to think critically about evidence anduse it effectively. Proper scaffolds are necessary to help students succeed atthese difficult tasks. Reflection prompts represent one of these scaffolds.

Reflection prompts fit into the scaffolded knowledge integration framework,which frames the instruction for CLP and KIE. Scaffolded knowledgeintegration is the instructional approach used to support knowledgeintegration: Students are scaffolded, or supported, as they make links amongtheir ideas. The framework has grown out of 12 years of design studies andinvolves four elements (Linn, 1995). First, instruction should "make thinkingvisible" to students by illustrating how links and connections are made.Teachers and students reasoning about scientific phenomena need to revealtheir own thinking to themselves and their peers. Second, instruction shouldidentify models for scientific phenomena that make sense to students so theycan connect new information to existing knowledge and to problems that areboth familiar and relevant. Third, instruction should provide social supportsso all students learn new links and connections for their ideas from theirpeers. Finally, students should be encouraged to become autonomouslearners so they can regularly revisit their ideas and continue to engage inknowledge integration. These four elements jointly promote knowledgeintegration.

Although reflection prompts contribute to each of the tenets of the scaffoldedknowledge integration framework, they particularly encourage autonomy byproviding an explicit place for reflection. By learning how to engage inreflectionÑrather than just memorization, as is often encouraged by theirother experiences with scienceÑstudents may begin to take responsibility fortheir own learning. Reflection prompts scaffold this process. We also hopethey help students to develop the propensity to continue linking ideas andevaluating views autonomously. To promote autonomy, I designed promptsto encourage reflection with the idea that regularly engaging in reflectionwould illustrate the advantages of reflection and lead to autonomousreflection in the future. Prompts may help students develop the dispositiontoward regular reflection (cf. Perkins, Jay, & Tishman, 1993; Resnick, 1987).

Reflection prompts also help to make thinking visible. Prompts can modelsome of experts' thinking practices. Directed prompts, in particular, model theplanning and monitoring in which experts engage. Reflection prompts alsohelp students make their own thinking visible by providing an explicit placefor reflection. As with thinking aloud (e.g., Bereiter & Bird, 1985; Collins &

11

Smith, 1982; Inhelder & Piaget, 1958; see Kucan & Beck, 1997, for a review ofthis literature) and self-explaining (Bielaczyc, Pirolli, & Brown, 1995; Chi,Bassok, Lewis, Reimann, & Glaser, 1989; Chi, deLeeuw, Chiu, & LaVancher,1994; Webb, 1983), prompts make explicit students' own thinking. However,written prompt responses make their thinking truly visibleÑto them, totheir teachers, and to researchersÑrather than being spoken without record.

Hypotheses and Results

In this research, I contrast two conditions. During a single semester, three ofthe KIE/CLP eighth grade physical science class periods received directedprompts, and three received generic prompts as they worked on a KIE critiqueproject. Having reviewed the theoretical framework that provides thefoundation for this research, we turn now to a rationale for this design, andan overview of the kinds of outcome measures investigated. I then discussthe predicted outcomes for each of the major investigations and brieflyoutline the actual findings.

First, what are the broad implications of this work? This research contributesto both the educational and cognitive research literature. By identifying thesuccess rates of generic and directed prompts, we can make informeddecisions about how best to help students succeed in science. We will learnmore about facilitating reflection and about facilitating knowledgeintegration. Furthermore, by investigating the cognitive mechanismsexplaining how reflection prompts facilitate knowledge integration, wecontinue to refine our understanding of how reflection and learning arerelated. And finally, by investigating the relationships among students'beliefs, their reactions to different instructional approaches (here, prompts),and their ability to integrate their knowledge, we are better able to improvelearning for students with particular characteristics. Understanding the effectsof prompts and the mechanisms behind those effects will help us supportstudents individually and collectively as we develop curriculum materialsand technologies for learning.

Investigating Prompts for Reflection

Technology affords us the opportunity to offer students prompts tailored tostudent and instructional characteristics. The pilot research discussed inChapter 3 indicates that students benefit from prompts encouragingreflection. What remains unclear is whether that benefit is a result of theimplicit instruction to reflect on X, or whether the mere act of stopping toreflect is the cause of the benefit. Thus, this dissertation compares promptsthat vary in specificity: directed and generic reflection prompts.

12

In investigating the effects of directed and generic prompts on studentlearning, we must first identify some measures of those effects. The measuresused in this research include the kinds of cites students make and link to oneanother (i.e., do they cite principles, labs, experiences, etc. in explainingscientific phenomena?) and the overall quality of their projects themselves.The KIE project investigated in this research is intended to improve bothstudents' conceptual understanding of certain science ideas and their ability tocritique scientific evidence and claims. Thus, the measures of project qualityinclude the coherence of their ideas, the quality of their critiques, and theirability to abstract out from the act of critiquing to develop guidelines forcritiquing. These measures depend in part on the students' ability to identifyweaknesses in ideasÑtheir own or those of others. One question thisdissertation will address is, which kind of prompt is most helpful at helpingstudents see these weaknesses?

One potential benefit of generic prompts is that they are unintrusive. Suchprompts are less likely than directed prompts to interrupt productiveactivities with requests that are potentially irrelevant to the students. Wemight hypothesize that generic prompts would be best for students who arealready likely to succeedÑfor example, for students who realize that learninginvolves the revision of ideas and that they are responsible for seeing wheretheir knowledge is problematic. Directed prompts, on the other hand, aremore supportive. They model productive kinds of reflection for students.Some students may need such guidance to help them reflect effectively.Furthermore, the pilot work indicates that prompts focused on self-monitoringÑthe prompts on which directed prompts are basedÑaresuccessful at eliciting knowledge integration. It may be the case that studentsneed directed prompts until they realize that reflection is an important part ofworking on a complex project. After that point, they may benefit just as muchfrom generic prompts. We might hypothesize that directed prompts would inparticular be more useful, overall, in helping students develop good critiques;in response to directed prompts students might create plans emphasizing thesalient aspects of critiquing.

The effects of the two types of prompts on students' work on the project arediscussed in Chapters 5 and 8. I find that the effects of prompts propagatethrough the rest of the project. Specifically, I find that generic prompts elicitmore cites of principles and evidence, and more cites in general, whenstudents work on the project itself. Citing principles, in particular, is found tobe positively related to developing a coherent, integrated understanding and,to a lesser degree, to being able to critique. Students in the generic promptcondition who cited principles also developed high-quality guidelines forcritiquing, and students in the generic prompt condition who cited manyideas overall did better at critiquing. Students who cited few principles

13

developed less coherent ideas and did worse at writing guidelines forcritiquing.

Generic prompts thus appear to elicit a broad range of productive ideas,helping students expand their repertoires of ideas. There were also differencesin the conceptual understanding of students in the two conditions. Studentswho received generic prompts developed a better understanding of theindividual ideas of energy conversion and thermal equilibrium more oftenthan did students who received directed prompts. Students who receivedgeneric prompts developed a more coherent, integrated understanding of thescience overall. I hypothesize that this stems in part from the broader range ofideas elicited by the generic prompts.

However, we also see in Chapter 5 that students in the two conditions did notdiffer in their overall project scores, the quality of the guidelines theydevelop, or the quality of their critiques. We see that abstracting andcritiquing, in particular, are hard for students to do well, though studentswho were successful at one quality measure tended to be successful at others,as well.

Investigating Focus of Reflection

Also under investigation is how the focus of the reflection in response toprompts is related to students' success on their science projects. Whatweaknesses do they detect in their knowledge? What else do they choose toconsider? Pilot research identifies distinct differences in students'interpretations of even very directed prompts. For example, when asked toassess their understanding, some students focus on science concepts, whileothers focus on logistical aspects of the project.

The responses to the reflection prompts themselves are investigated here toinvestigate the nature of knowledge integration. Each response is coded for itsdegree of elaboration and the focus of its reflection. Students' focuses ofreflection are then linked to their success on the project, to identify possiblerelationships between the focus of reflection and knowledge integration. Iinvestigate differences in how students reflect as a result of the two levels ofspecificity of prompts.

Content, cognitive, and goal-oriented focuses might be reactions to varioustypes of weaknesses; these focuses can all be useful. If a prompt encouragesstudents to think about the "big picture"Ñfor example, what their goal is forthe project or how they should accomplish that goalÑthey may work moreproductively than if they simply plow ahead. On the other hand, if a promptencourages students to contemplate their own conceptual understanding,

14

they should also benefit. Students who are content-focused may developmore integrated knowledge, while students who are more goal-focused maydo a better job of critiquing evidence. If students focus instead on theirbehavior, they may do unremarkable work.

Generic prompts leave the context to the students to provide. One mightpredict that many students' focus will be on instructional goals since that maybe considered a more manageable task. Directed prompts are more likely topoint students in one direction or the other, but the prompts leave room forinterpretation, as shown by the range of student responses in the pilot work. Ihypothesize that certain orientations might be beneficial in response to onetype of prompt, but less useful when made in response to the other type,because the benefit of reflection might depend in part on the context in whichthe reflection takes place.

In Chapters 6 and 8 I outline the results of these analyses of students'responses to prompts. I find that elaboration in response to directed promptsmay help students develop better critiques. I find that for both types ofprompts, students most often focus on specific instructional goals (e.g.,identifying criteria for critiquing evidence). I also find that generic promptswere less likely to enable a lack of reflection than were directed prompts, andelicited more reflection on science content. However, while focusing oncontent in response to directed prompts may improve the coherence ofstudents' ideas seen in their projects, reflection on content in other instancesmay have negative effects. For example, focusing on content in response togeneric prompts may reduce students' performance in all three majormeasures of project quality: coherence of ideas, overall critique quality, andguidelines quality.

Pairs who are cryptic in response to directed prompts tend to have poorperformance overall, and those who are non-reflective in response to theseprompts (i.e., who say they understand everything) tend to have poorcritiques. This finding leads us to the other aspect explaining generic prompts'effectiveness: Students who do not take advantage of the opportunitiesprompts afford for reflection are less likely to identify weaknesses in theircurrent knowledge, and as a result do not integrate their knowledge as well.Since generic prompts are less likely to elicit this minimalist approach, theireffectiveness is improved.

Overall, we see that taking specific actions in response to directed prompts canhave both strong positive and strong negative effects on students' work onthe project, depending on the action. Generic prompts, on the other hand,seem to have more generally positive effects. When are the seeminglyproductive activities promoted by generic promptsÑmost notably, reflecting

15

on contentÑtruly productive? The lack of an evaluative context in genericprompts may reduce the usefulness of certain kinds of reflection in responsesto them.

Investigating Students' Beliefs

A third area of interest is the relationship between students' beliefs and theiruse of prompts. Others' research (e.g., Eylon & Linn, in press; Songer & Linn,1991) indicates that student beliefs about learning science are related to theirknowledge integration in science class; I also investigate the relationshipbetween those beliefs and students' use of prompts.

Students' beliefs are assessed using a written instrument which has beenrefined over the last several years. In-depth interviews with a subset ofstudents confirm the beliefs assessments. Students' views of the nature ofscience and the nature of learning science are then related to their reflectionand their performance on the project, bringing an aspect of individualstudent characteristics to the fore of the work. In particular, I investigatestudents' beliefs about the dynamic nature of the process of science, the valueof understanding for learning science, and the value of taking personalresponsibility for learning.

We might hypothesize that students' beliefs about the value ofunderstanding might be particularly important in predicting theirperformance in the class, since they may expect to revise and revisit theirideas. We might also hypothesize that students with productive beliefs in onearea would be likely to have productive beliefs in the other areas investigated,as well. Students who believe science represents a collection of facts, forexample, may be more likely to memorize in science class.

In Chapters 7 and 8 I characterize students' beliefs and discuss the apparenteffects of those beliefs. I show that students in the CLP/KIE classroomundergo a significant improvement in their beliefs about learning scienceover the course of the semester, and that this change is especially pronouncedfor students who start out in the low and middle groups. I also show thatstudents' beliefs about the process of science and the strategy for learningscience are highly correlatedÑbut that beliefs about strategy and autonomyare not correlated. Students can have highly productive beliefs about oneaspect of learning science while they have less productive beliefs aboutanother aspect.

Students who believe understanding is the best strategy for learning scienceare likely to do best on the final exam. Highly autonomous studentsdeveloped more coherent ideas in the project regardless of which prompt type

16

they received, and generic prompts helped middle-autonomy studentsachieve high levels of coherence. Students with some propensity towardautonomy may be more able to identify weaknesses on their own and thusbenefit more from generic prompts, while less autonomous students mayexpect others to identify weaknesses for them. Low autonomy students,however, did not appear to benefit from either type of prompt and mayrequire a different instructional approach. (Since these students developedmore sophisticated ideas about learning science over the course of thesemester, the instructional approach best suited for them may involve longerexposure to a curriculum like that used in CLP/KIE.)

Structure of the Dissertation

In this chapter I have reviewed the theoretical framework for this research. Ihave given a rationale for the research and foreshadowed possibleimplications of the work. This and the following three chapters comprise PartOne of the dissertation, which provides salient background information. Inthe next chapter, I outline in greater detail the framework for this research. InChapter 3 I provide further background, presenting the pilot work on whichthis dissertation work is based. I outline the basic methods used for this studyin Chapter 4; more specific methods discussions will occur in the resultschapters as appropriate.

In Part Two I present the results of the dissertation's investigations. The grosseffects of reflection prompts on student performance are discussed in Chapter5. I describe the kinds of reflection students engage in in response to theprompts in Chapter 6. And, in Chapter 7 I discuss students' beliefs and therole those characteristics of individual students may play in students'reflection and learning. In Chapter 8 I summarize and synthesize the resultspresented in the previous chapters, in part through statistical analyses thatlink the three investigations.

Part Three is a single chapter. In Chapter 9 I discuss the broader implicationsof this work.