model-based inquiry: epistemology, modeling skills, assessment, & research

Model-based Inquiry: Epistemology, Modeling Skills, Assessment, & Research

Janice GobertThe Concord Consortium

mac.concord.org mtv.concord.org

Based on work from 1) Making Thinking Visible (NSF #9980600) and 2) Modeling Across the Curriculum (IERI #0115699).

All opinions expressed are those of the author and do not

necessarily reflect the views of the granting agencies.

How are you defining “scientific practice” in your design and empirical work? The Scientific Practice is Modeling, this includes model-based

reasoning, model-based inquiry, etc.

• MBL is a theory of science learning that integrates research in cognitive psychology and science education (Gobert & Buckley, 2000).

• Its tenets are that understanding requires the construction of mental models and all subsequent problem-solving, inferencing, or reasoning are done by means of manipulating or ‘running’ these mental models (Johnson-Laird, 1983).

• Model-based reasoning also involves the testing, and subsequent reinforcement, revision, or rejection of mental models.

• Modeling research at the Concord Consortium organizes learning activities, assessment, and research around model-based learning.

MBR Involves both internal and external

models

External Models,i.e., hypermodels

Mental model

cognitive processes act on mental model

Assumes students’ epistemologies influences model-based reasoning;Gobert & Discenna, 1997; Gobert & Pallant, 2004).

Other research literature….

In addition to students’ pre-instruction models in designing the unit, we (J. Gobert, Jim Slotta, Amy Pallant) drew on current findings from:

• causal models (White, 1993; Schauble et al, 1991; Raghavan & Glaser, 1995),

• model-based teaching and learning (Gilbert, S., 1991; Gilbert, J. 1993);

• model revising (Clement, 1989; 1993; Stewart & Hafner, 1991);

• diagram generation and comprehension (Gobert, 1994; Gobert & Frederiksen, 1988; Kindfield, 1993; Larkin & Simon, 1987; Lowe, 1989; 1993),

• the integration of text and diagrams (Hegarty & Just, 1993), and

• text comprehension (van Dijk & Kintsch, 1983; Kintsch, 1998).

How is it being supported?from Making Thinking Visible Project

(mtv.concord.org)

• Scaffold drawing of their own models of plate tectonics phenomena based on progressive model-building principles (model pieces acquisition).

• Scaffold on-line “field trip” to explore differences between the East and West coast in terms of earthquakes, volcanoes, mountains (beginning with the most salient differences to support knowledge building around the driving question; model-pieces acquisition).

• Posing a question about their current model (to model pieces integration and model-building).

• Learn about location of earth’s plates (to scaffold relationship between plate boundaries anf plate tectonic phenomena as model pieces integration).

• Reify important spatial and dynamic knowledge (model pieces integration) about transform, divergent, collisional, and convergent boundaries.

• Learn about causal mechanisms involved in plate tectonics, i.e., convection & subduction (scaffolded by reflection activities to integrate spatial, causal, dynamic, and temporal aspects of the domain- model pieces integration).

Pedagogical support (cont’d) from Making Thinking Visible Project

(mtv.concord.org)• Students are scaffolded to critique learning partners’ models using prompts in

WISE (reconstruct, reify, & reflect). Prompts include:1. Are the most important features in terms of what causes this geologic process

depicted in this model? 2. Would this model be useful to teach someone who had never studied this geologic

process before? 3. What important features are included in this model? Explain why you gave the model

this rating.4. What do you think should be added to this model in order to make it better for

someone who had never studied this geologic process before?

• Reflect on how their model was changed and what it now helps explain (reconstruct, reify, & reflect), e.g.,: prompts include:

“I changed my original model of.... because it did not explain or include....” “My model is now more useful for someone to learn from because it now includes….”

• Transfer what they have learned in the unit to answer intriguing points (reconstruct, reify, & reflect):

Why are there mountains on the East coast when there is no plate boundary there? How will the coast of California look in the future?

How do you know when you see it?

• Examples to follow….

Comments on example 1….

• Original model- focus on crustal layer, no causal mechanisms for what causes mountain formation.

• W. coast partners’ critique requested labels.

• Revised model-includes labels, and a cut away view of the interior of the earth which includes convection in the mantle.

Comments on example 2…

• Original model- cross section, no causal mechanisms for what causes mountain formation.

• W. coast partners’ critique requested information about direction of plate movement.

• Revised model-includes a cross section with plate movement, added the mantle as an interior layer.

Does it effect students’

epistemologies? *• Data to follow….

* Acknowledging the problem with assessing epistemologies with surveys.

WISE Period 1 - sig. Epistemological gains

02468

10121416

Cell

Mea

n

preMtot postMtotCell

TSA

Interaction Bar Plot for modelgain Effect: Category for modelgain * teacher

2 22.442 11.221 .692 .5044 1.384 .15761 988.926 16.212

1 115.697 115.697 16.046 .0002 16.046 .9872 83.882 41.941 5.817 .0049 11.633 .866

61 439.837 7.210

DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelgainCategory for modelgain * teacherCategory for modelgain * Subject(Group)

ANOVA Table for modelgain

1.012 1.571 .2047.511 1.543 .5139

-.502 1.739 .5692

Mean Diff. Crit. Diff. P-ValueA, SA, TS, T

Fisher's PLSD for modelgain Effect: teacher Significance Level: 5 %


0

2

4

6

8

10

12

14

Cell

Mea

n


TSA

Interaction Bar Plot for modelgain Effect: Category for modelgain * teacher

2 2.335 1.167 .079 .9244 .158 .06159 874.504 14.822

1 311.401 311.401 40.945 <.0001 40.945 1.0002 56.782 28.391 3.733 .0297 7.466 .659

59 448.710 7.605

DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelgainCategory for modelgain * teacherCategory for modelgain * Subject(Group)

ANOVA Table for modelgain

.064 1.632 .9380-.289 1.632 .7268-.353 1.827 .7028


Fisher's PLSD for modelgain Effect: teacher Significance Level: 5 %


0

2

4

6

8

10

12

14

Cell

Mea

n


TSA

Interaction Bar Plot for modelchange Effect: Category for modelchange * teacher

2 47.195 23.597 1.433 .2464 2.866 .28562 1021.132 16.470

1 366.531 366.531 54.803 <.0001 54.803 1.0002 106.362 53.181 7.952 .0008 15.903 .958

62 414.665 6.688

DF Sum of Squares Mean Square F-Value P-Value Lambda PowerteacherSubject(Group)Category for modelchangeCategory for modelchange * teacherCategory for modelchange * Subject(Group)

ANOVA Table for modelchange

-.809 1.684 .3437.833 1.654 .3207

1.642 1.876 .0857


Fisher's PLSD for modelchange Effect: teacher Significance Level: 5 %


0

2

4

6

8

10

12

14

Cell

Mea

n


TSA


2 63.678 31.839 2.807 .0681 5.614 .52362 703.214 11.342

1 190.437 190.437 35.768 <.0001 35.768 1.0002 65.833 32.917 6.182 .0036 12.365 .889

62 330.098 5.324



-.073 1.392 .9180-1.589 1.367 .0231 S-1.516 1.551 .0552




0

2

4

6

8

10

12

14

Cell

Mea

n


TSA


2 26.202 13.101 .840 .4368 1.680 .18160 936.016 15.600

1 444.676 444.676 75.513 <.0001 75.513 1.0002 90.227 45.113 7.661 .0011 15.322 .950

60 353.325 5.889



.701 1.631 .3970-.531 1.758 .5510

-1.232 1.909 .2040



Modeling Across the Curriculum Team

Principal & Co-Principal InvestigatorsPaul Horwitz, Concord Consortium, Principal InvestigatorJanice Gobert, Concord Consortium, Co-PI & Research DirectorRobert Tinker, Concord Consortium, Co-PIUri Wilensky, Northwestern University, Co-PI

Other senior personnel Barbara Buckley, Concord Consortium Chris Dede, Harvard University Ken Bell, Concord Consortium Sharona Levy, University of HaifaTrudi Lord, Concord Consortium Jaclyn Scobo (intern), Northeastern University

mac.concord.org; IERI #0115699

www.concord.org

http://ccl.northwestern.edu

http://www.concord.org/

Design of Activities, Scaffolding, & Research are based on…

Model-based learning (Gobert & Buckley, 2000) as well as other literature….

• cognitive and perceptual affordances of learning with technology-based representations (Gobert, 2005; Larkin & Simon, 1987)

• progressive model-building (White & Frederiksen, 1990; Raghavan & Glaser, 1995)

• students’ difficulties in learning with models (Sweller, et al, 1990; Gobert, 1994; Lowe, 1989; Head, 1984).

Thus, scaffolding is designed to… • guide search, supports perceptual cues, and inference-making from

perceptual cues (Larkin & Simon, 1987). • elicit prior knowledge, support integration with new knowledge, and

support reification & reflection of knowledge. Theory driving our analyses is based on…• expert problem-solving for estimating solutions (Paige & Simon, 1966)• experts vs. novices search and knowledge acquisition strategies (Gobert,

1994, 1999; Thorndyke & Stasz,1980).

Model-Based Learning in situ

Intrinsic Learner Factors

Epistemology of models(SUMS, Treagust et al, 2002)

….because students’ epistemologies influence both knowledge integration (Songer & Linn, 1991) and model-based reasoning (Gobert & Discenna, 1997),

Intrinsic Teacher Factors

Epistemology of models(adapted from

Grosslight et al, 1991)

Teaching experienceBackground

(adapted from Fishman, 1999)

Classroom FactorsImplementation of MAC activity use

(logged)Teacher practices

(reported via Classroom Communique)

Hypermodels*simulationsdiagrams

explanationsinstructionsdata tables

graphs

model reinforcementmodel revision

model rejection

Learner'sMentalModels

model evaluation

prior knowledge new information

model formationInteracting with

understandingreasoninggenerating

Phenomenaexperiencesexperiments

model use

+ MetacognitiveSelectingDirecting Monitoring

What is the model for the pedagogical support of the practice? What kinds

of designs put this model into effect?

Scaffolds from the MAC project include:

• Representational Competence: view and understand a representation or representational features of the domain.

• Model pieces acquisition: understand & reason with pieces of models (spatial, causal, functional, temporal).

• Model pieces integration: combine model components in order to come to a deeper understanding of how they work together as a causal system.

• Model based reasoning: reasoning with models or pieces of models. • Reconstruct, Reify, & Reflect: reify knowledge and transfer it to

another context or level of understanding.

Technology & Affordances for Research & Assessment with

ModelsTechnology:

Log files on students’ interactions with models capturing students’…

– Data on duration and sequence – Actions and choices with models– What info or help they seek– Responses to questions

Embedded & Performance Assessments with models & questions…

– Generate profile for students at

“pivotal” points in curriculum– Responses to questions

AffordancesImplementation data -- which activities

were used, pattern of use (consecutive or intermittent days) at classroom level & student level.

Finer-grained log data can be used for– Measure of students’

systematicity and inquiry skills.

– Test for interactions with prior knowledge & epistemology.

These data are used to derive student reports….

¯ Formative assessments ¯ Summative assessments¯ Performance assessments

Drill down to performance assessment

with logsCurrently we are focusing on log files as indices of:1) Domain-specific model based reasoning by investigating

“hot spots”2) Domain- General Inquiry skills (DoGI spots, similar to

NSES inquiry strands).3) This allows us to assess inquiry development

– both within (hot spots) and across domains (DoGI spots). – assess transfer from one domain to another– assess how a student’s inquiry skills are progressing

“independent” of content learning. Since our activities are enacted over multiple days and in

three domains, we avoid the problems faced by earlier studies of inquiry in which there were not enough data to get at students’ inquiry skills (Shavelson et al, 1999).

Inquiry “Hot Spots”Tasks or parts of tasks that contain multiple

components of model-based inquiry these, by definition, require deep reasoning.

MAC supports 5 strands of model-based inquiry. These are more specific than the NSES (1996) inquiry standards which were are not specific to current technology-based learning nor are the NSES strands specific to modeling tasks.

• Representational Competence: view and understand a representation or features of the domain.

• Model pieces acquisition: understand & reason with pieces of models (spatial, causal, functional, temporal).

• Model pieces integration: combine model components in order to come to a deeper understanding of how they work together as a causal system.

• Model based reasoning: reasoning with models or pieces of models. • Reconstruct, Reify, & Reflect: reify knowledge and transfer it to another

context or level of understanding.Fine-grained analysis, one hot spot at a time, is necessary

in order for us to code the various process variables we plan to aggregate and focus on.

Hot spot from Collisions task 5: Student sets mass of two balls

• The challenge: adjust the masses of the two balls to make the orange ball move as fast as possible after the collision.

Strategies for InquiryPreliminary analysis based on human

coding identified 2 different inquiry patterns:

1. haphazard2. systematic(Also, there are students who got it correct on first

trial, sometimes with explicit test).

These are consistent with literature: ~ experts vs. novices search and knowledge acquisition

strategies (Thorndyke & Stasz,1980; Gobert, 1994, 1999).~ expert problem-solving for estimating solutions (Paige &

Simon, 1966).

Examples …

Haphazard Strategy- this student obtained the correct answer (11.0; 1.0) on trials 2,10,(& 15) but did

not know it!Student 12116 made 15 trials:Blue Ball Orange ball

11.0 11.0 11.0 1.011.0 3.011.0 4.0 1.0 1.0 1.0 11.0 8.0 7.0 11.0 2.0 11.0 11.0 11.0 1.011.0 5.03.0 5.01.0 5.01.0 8.011.0 1.0

Systematic Strategy, e.g.,vary one ball at a time (a good strategy in the absence of prior knowledge).

Student 18115 had a plan:Blue Ball Orange ball

11.0 11.0 5.0 11.0

10.0 11.011.0 1.0

Another Hot Spot from Dynamica: “What settings cause the blue ball to stop when

it collides with the orange ball?”

Input sliders

Numerical data from runConstructed

text response

• Track students’ iterations of this as index of systematicity in inquiry.

TL3 time

TL3 RdTsk

T3 trials T3 values T3 Vx

T3 #rtPr

T3 success Q10A

T3 %vary1

T3 %rpt

T3 #eqPr

T3 #extrem

PrT3

%clgT3

%frg

T3 %goal Flips

T3 CAT

2.5 73 2 2.0 v 5.0 5.0 v 5.0

-1.7, 0.0,

1 1 that they must have have equal masses

1 0 1 0 1 0 0 B1

2.9 34 8 2.0 v 5.0, 4.0 v 11.0, 1.0 v 4.0,

11.0 v 11.0, 6.0 v 7.0, 5.0 v 7.0, 3.0 v 7.0, 7.0 v 7.0,

-1.71, -1.87, -2.4, 0.0, -0.31, -0.67, -1.6, 0.0,

2 1 they have to both have to be the same size

0.43 0 2 0 0.29 0.71 0.43 B2

2.7 130 1 5.0 v 5.0, 0.0, 1 1 match the orange 0 0 1 0 0 0 0 A

2 13 10 2.0 v 5.0, 1.0 v 5.0,

11.0 v 5.0, 8.0 v 5.0,

8.0 v 11.0, 7.0 v 11.0, 6.0 v 11.0, 7.0 v 10.0, 11.0 v 10.0, 10.0 v 10.0,

-1.71, -2.67, 1.5,

0.92, -0.63, -0.89, -1.18, -0.71, 0.19, 0.0,

1 1 The mass must be almost as big as the other ball

0.89 0 1 0 0.67 0.33 0.33 B2

CC’s approach for Task 3- Additional Categories for coding & 4

students’ data.

Additional categories (in addition to CMU) are % of trials in which~ set the masses as equal~ set the masses as extremes~ closer the the goal, further from the goal,~ goal flips.

Hot Spot from BioLogica: Monohybrid (Task 3):

produce only 2-legged offspring

Arrow toolCross toolSnip tool

Chromosome tool

Dragon genome chartPunnett square pad

The task

Requires changing both Legs alleles of one parent

Monohybrid Task 3 Subtasks & Data

collected• Predict whether a pair of dragons can have only 2-legged

offspring– Multiple choice question

• Describe the necessary parental genotypes.– Full text response

• Change alleles of one parent to homozygous recessive

• Cross parents– Success = making right cross– Number of crosses made– List of crosses made

Data Monohybrid performances

• Student performance is scored by computer based on– Prediction– Success– Number of attempts– Whether they repeated any crosses (an indication of haphazard behavior).

• Performances can be grouped into – Systematic & correct– Systematic & incorrect– Haphazard & correct– Haphazard & incorrect

Systematic vs. haphazard performance and Pre/Post

gainsDependent Var iable: Total Score PostT3CATSYS2NUM Mean Std. Error 95% Confidence Interval

Lower Bound Upper BoundHaphazard 19.980 .740 18.518 21.442Systematic 22.868 .651 21.582 24.154

Table below indicates that Pre-test covariate is significant, as is the two-category predictorvariable (S, H). Together, the covariate and this variable account for 25.2% of the variance in thePost-test scores.

Data based on 649 students in 10 member schools; (54.2%) in ‘regular’ classes.

ANCOVA with pre test score as covariate indicates pre-test is significant (p ≤ .001), as is the two-level predictor variable (Systematic versus Haphazard).

Together, the pretest scores and the systematicity variable account for 25.2% of the variance in the Post-test scores.

Students who are systematic at this task outperform students on the post-test who are not, irrespective of whether they succeeded at the inquiry task.

Thus the systematic inquiry is facilitating knowledge building (as measured by the post-test).

Overview of Data Analysis with Hot spots

We are aggregating hot spots and testing their relationship:

• conceptual learning measurements, i.e., pre-post content tests• measures of students’ epistemologies of models and views of science since

students’ epistemologies influence learning (Songer & Linn, 1991; Gobert & Discenna, 1997).

With these data, we can:• track students’ systematicity in learning with models as one important facet

of inquiry skills and conceptual learning. To us, inquiry skills co-evolve with content learning but each can be measured separately (sort of).

• test for development of inquiry strategies across time and across domains ~ complicated by task difficulty increasing over time ~ complicated by the co-evolution of the development between domain-knowledge and inquiry strategies ~ complicated by the likelihood that students build knowledge in small, conceptual pieces, I.e., about acceleration or velocity).

In the future, using log files we seek to identify at risk students- i.e., students whose inquiry strategies are buggy.

Domain-General Inquiry Spots (“DoGI”

spots)1- Making predictions with models2- Interpreting data from a representation (i.e.,

model/graph, pedigree, etc). 3- Making explanations (about models, etc)4- Mathematizing with models- Filling in an

equation/solve an equation; reasoning with an equation.

5- Designing and/or conducting an experiment with models.

Thus, if a student can do these types of tasks, they are doing model-based inquiry.

model-based inquiry: epistemology, modeling skills, assessment, & research

Documents

current model

modelbased inquiry

progressive model

modelbased teaching

rejection of mental

construction of mental

learning partners models

modeling research