assessment of the modeling competence: a systematic review and synthesis of empirical research

Educational Research Review 13 (2014) 52–73

Contents lists available at ScienceDirect

Educational Research Review

journal homepage: www.elsevier .com/locate /EDUREV

Review

Assessment of the modeling competence: A systematic reviewand synthesis of empirical research

http://dx.doi.org/10.1016/j.edurev.2014.10.0011747-938X/� 2014 Elsevier Ltd. All rights reserved.

⇑ Corresponding author.E-mail addresses: [email protected] (Chr.Th. Nicolaou), [email protected] (C.P. Constantinou).

Chr.Th. Nicolaou, C.P. Constantinou ⇑Learning in Science Group, Department of Educational Sciences, University of Cyprus, Cyprus

a r t i c l e i n f o

Article history:Received 8 February 2014Revised 11 October 2014Accepted 13 October 2014Available online 22 October 2014

Keywords:Science educationModeling competenceAssessment approaches

a b s t r a c t

We provide an analysis of the existing literature on the assessment of modeling as a scien-tific competence focusing on empirical research findings. Out of 802 searched citations, atotal of 23 publications from science teaching and learning met the inclusion criteria.The analysis was based on the types of instruments used (interviews, questionnaires, vid-eos) to assess the different aspects of scientific modeling (e.g. modeling practices; model-ing product; meta-knowledge; cognitive processes during modeling). The results indicatethat specific aspects of the modeling competence tend to be evaluated by specific typesof assessment instruments and that assessment of other important aspects of the modelingcompetence is scarce. We suggest that this may be occurring due to the lack of a unifyingframework for conceptualizing the modeling competence. In addition, these findings pro-vide insights into certain challenges and confounding factors involved in designing newassessment instruments for each aspect of the modeling competence.

� 2014 Elsevier Ltd. All rights reserved.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532. Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

2.1. Working definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542.2. Literature search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.3. Study selection criteria and procedures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.4. Coding study characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

2.4.1. Name of the study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.4.2. Modeling aspect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.4.3. Modeling software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.4.4. Phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572.4.5. Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572.4.6. Subjects (and level of instruction). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572.4.7. Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.4.8. Tools or other means of data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.4.9. Data analysis process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.4.10. Principal outcomes of the research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.4.11. Country . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

http://crossmark.crossref.org/dialog/?doi=10.1016/j.edurev.2014.10.001&domain=pdf

http://dx.doi.org/10.1016/j.edurev.2014.10.001

mailto:[email protected]

mailto:[email protected]


http://www.sciencedirect.com/science/journal/1747938X

http://www.elsevier.com/locate/EDUREV

Chr.Th. Nicolaou, C.P. Constantinou / Educational Research Review 13 (2014) 52–73 53

2.4.12. Study type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582.4.13. Instruction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2.5. Data synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3. Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.1. Search results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.2. Aspects of the modeling competence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.3. Types of assessment instruments or other means of data collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

3.3.1. Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.3.2. Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603.3.3. Concept map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.3.4. Student constructed models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.3.5. Student work and discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

3.4. Instruments quality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673.5. Educational context of the studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673.6. Participants age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693.7. Types of modeling tools used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4. Discussion and conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.1. What do we know and what needs to achieved through future research. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.2. The need for a unifying framework of the modeling competence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.3. Strengths and limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Appendix A. Supplementary data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

1. Introduction

Modeling, the process of constructing and deploying scientific models, has received widespread attention as a compe-tence whose development also facilitates student learning of science concepts, methodological processes and the develop-ment of an awareness of how science operates (Hodson, 1993). Modeling natural phenomena is known to be challenging forboth students and teachers (Schwarz et al., 2009; Sins, Savelsbergh, & van Joolingen, 2005). However, its potential benefitsare thought to make it a worthwhile activity to include in science education, particularly as a context for developing anawareness of the value of epistemological objects that lead to evidence-based predictions, as a means of understanding com-plex dynamic systems, as well as a process of acquisition of conceptual knowledge and learning of scientific reasoningprocesses.

Efforts to design modeling-based learning instruction have relied on a theoretical framework about the modeling com-petence, which analyzes the constituent components into two broad categories, namely modeling practices and meta-knowl-edge (Nicolaou, 2010; Papaevripidou, 2012; Papaevripidou, Nicolaou, & Constantinou, 2014) (Fig. 1). Attempts to validatesuch designs have led to the claim that student modeling competence can emerge as a result of active participation in spe-cific modeling practices, and is shaped by meta-knowledge about models and modeling. Model construction (Stratford,Krajcik, & Soloway, 1998); model use (NRC, 2012); comparison between models (Penner, Giles, Lehrer, & Schauble, 1997);model revision (Schwarz & White, 2005) and model validation have been identified as the main practices in which studentsare engaged during modeling. Meta-knowledge is analyzed into the metacognitive knowledge about the modeling process,which refers to student ability to explicitly describe and reflect on the actual process of modeling, and meta-modeling knowl-edge (Schwarz & White, 2005), i.e. the epistemological awareness about the nature and the purpose of models.

Assessment is considered, along with curriculum, instruction and teacher development, as one of the key components ofscience education (NRC, 2012). It is a vital part of classroom life and as such it should be at the focus of any educational effort(Pellegrino, 2012). Despite the growing research interest in modeling-based learning, research on possible approaches forassessing the modeling competence would appear, from a first glance, somehow fragmented in that they typically focuson student meta-modeling knowledge, or on specific constituent components of modeling ability (e.g. model constructionor model comparison) without presenting a comprehensive perspective of modeling. Evaluation of the cognitive processesenacted during modeling and evaluation of the constructed models is particularly scarce (Louca, Zacharia, Michael, &Constantinou, 2011).

This fragmented view of modeling assessment and the need for additional modeling implementations in teaching practicehighlight the need for a systematic review of the assessment of the modeling competence in science teaching and learning.To our knowledge, such a review has not been conducted so far. Most of the relevant research is composed of single empiricalor design studies investigating learners understanding of epistemological aspects of models and modeling prior to or afterthe implementation of interventions and seeking to analyze the process or the end product of modeling. We identified onlytwo extensive reports on modeling (Louca & Zacharia, 2011; Stratford, 1997). Stratford’s review covered research conductedon the topic of using computer models to aid science instruction at the precollege level. The review investigated researchabout students (a) running simulations, (b) creating dynamic models using modeling environments, and (c) using program-

Fig. 1. The modeling competence framework.

54 Chr.Th. Nicolaou, C.P. Constantinou / Educational Research Review 13 (2014) 52–73

ming environments to create simulations (Stratford, 1997). Louca and Zacharia (2011) reviewed the research concerningmodeling-based learning with an aim to systematize the accumulated knowledge. Their review sought to describe the cog-nitive, metacognitive, social, material and epistemological aspects of modeling-based learning. The effort to assess modelinginvolves identifying the various dimensions that theoretically comprise the modeling competence, operationally definingeach of these dimensions and developing assessment instruments for them. In this way, our review of assessment methodsand approaches of the modeling competence can potentially influence the discourse on the nature and facets of the modelingcompetence.

This review study aims to contribute to the debate on the value of scientific modeling activities for teaching and learningby systematically compiling the available empirical evidence on the various approaches to assessing the modeling compe-tence as published in journal articles. In particular, we set out to address the following research question: What differentapproaches to the assessment of learners modeling competence can be identified in the existing literature and how is ourunderstanding of learners modeling competence shaped by evidence derived from the use of different forms of assessment?

2. Methods

2.1. Working definitions

For communication purposes we provide some definitions of the terms used in this review paper.First, we provide the definition of the core idea of this paper, assessment and then two interrelated concepts, summative

and formative assessment. We do so, because through our review we sought to identify if these two different assessmentprocedures were implemented by the researchers when assessing the modeling competence.

Assessment is a judgment or a set of judgments which can be justified according to specific weighted set goals, yieldingeither comparative or numerical ratings. The process of assessment are the steps (or the mechanics) required to effectuate ajudgement (Taras, 2005).

Summative assessment is the result of the process of assessment, in other words, a judgement which encapsulates all theevidence up to a given point (Taras, 2005). As such, summative assessment is criterion referenced or norm referenced(Harlen & James, 1997).

Formative assessment is a form of summative assessment that requires feedback, which indicates the existence of a ‘gap’between the actual level of the work being assessed, and the required standard. It also requires an indication of how the workcan be improved to reach the required standard (Taras, 2005). It is formative in that the evidence is actually used to adapt theteaching work to meet learning needs (Black & Wiliam, 1998) and therefore, it is always made in relation to where pupils arein their learning in terms of a specific content or skill. As such it is, by definition, criterion referenced (Harlen & James, 1997).

Second, we provide some working definitions with regards to the other core component of this paper, models and modeling.In the literature about modeling, researchers elaborate on three different types of models: mental, conceptual and scientific:

Mental models (Gentner & Stevens, 1983), a construct of cognitive psychology, refer to those ‘‘transient representationsthat are activated usually when one is exposed to a new situation and act as structural analogies to situations or processes’’(Greca & Moreira, 2001, p. 108).

Conceptual models (Greca & Moreira, 2000), a construct of science education research, are coherent conceptual structuresdeveloped with a view to achieve permanence, to offer facility for understanding phenomena or situations, and to be used foranalysis and predictions. They are created by researchers, teachers or engineers to facilitate aspects of understanding or theteaching of systems and are, therefore, consistent with scientifically accepted knowledge.

Scientific models (Crawford & Cullin, 2004; Nicolaou, Nicolaidou, & Constantinou, 2009), epistemological constructs of thenatural sciences, are interpretive representations, usually in symbolic form, with predictive power, which are used as tools inknowledge development or theory testing in a certain discipline. As epistemological entities, scientific models meet threedistinct requirements; they represent the characteristics of the phenomenon, they provide a mechanism that accounts for


how the phenomenon operates, and they can be used to formulate predictions about the observable aspects of the phenom-enon. For the purpose of this review paper we were interested in studies which relate to scientific models and how they areused in teaching–learning situations.

Modeling refers to the ability to construct and improve a model of a physical object, a process or a phenomenon (Halloun& Hestenes, 1987). Whenever a learner is purposefully engaged in the complex processes entailed in modeling, it is assumedthat the intent is to develop understanding of a phenomenon through modeling-based learning.

Modeling-based learning refers to learning through construction and refinement of scientific models by students. It is dis-tinguished from model-based learning, which often refers to how teaching–learning processes relate to the construction ofmental models.

2.2. Literature search

A search of international peer-reviewed published literature was undertaken with a focus on empirical studies thatemphasized modeling-based learning. We searched keywords, titles and abstracts in the Scopus database (www.scopus.-com) using the following keyword combinations: (modeling OR modelling) AND (learning) AND (teaching) AND (assess-ment) AND (science education) within the subject area of social sciences. A detailed description of the electronic databasesearches can be obtained upon request from the authors. We manually searched reference lists of pertinent articles and man-ually searched for articles from key authors in the field.

2.3. Study selection criteria and procedures

The following criteria guided the study selection process. We selected papers that:

� investigated some aspect of the student modeling competence in accordance to our definition. The modeling compe-tence is considered an amalgam of students modeling practices and their meta-knowledge about models andmodeling;

� used quantitative or qualitative empirical measures for assessing the modeling competence;� described research involving students of any age in any educational system, who attended private or public schools or

universities;� were written in English;� satisfied pre-determined methodological criteria, namely: the assessment approaches were described in adequate

detail, any instruments used were accessible and there was information about the process of administering data col-lection procedures; the raw data were illustrated adequately to create a sense of the actual evidence used to supportclaims; the structure, content and timeline of any interventions were described in adequate detail.

The first author reviewed the abstracts and relevant full-text articles. 10% of the papers were reviewed by the secondauthor. Inter-rater reliability reached 88% and any disagreement was resolved through discussion. A study was excludedif both reviewers agreed that it did not meet the eligibility criteria. In the same vein, the first author followed this procedureto exclude any of the remaining papers that did not meet these criteria. Book chapters were excluded because of limitationson time, human and financial resources. However, we feel that this limitation does not reduce the value of our review,mainly because, based on the information provided in chapter synopses, these texts did not describe specific assessmentapproaches.

2.4. Coding study characteristics

Following the methodology paradigm established in other review papers (Dochy, Segers, & Buehl, 1999), we defined thecharacteristics central to our review and analyzed the articles we selected on this basis. Specifically, the following informa-tion was recorded in tables (Table 1 includes part of the results of this coding procedure):

2.4.1. Name of the studyThe first author and the year of publication are stated to denote the name of the study. A letter follows the year when

more than one paper of the same author is in the list. A complete reference list of the studies under review is presentedin the Appendix.

2.4.2. Modeling aspectThis category refers to the specific modeling aspect on which each study focuses: (a) modeling practices, (b) metacogni-

tive knowledge about the modeling process, (c) meta-modeling knowledge, (d) the modeling product (models) and (e) thecognitive processes during modeling.

2.4.3. Modeling softwareThe software used as a tool by the students to construct and revise models.

http://www.scopus.com

http://www.scopus.com

Table 1Characteristics of the included studies.

Name of thestudy

Modeling aspect Modelingsoftware

Phenomenon Subjectsb Case # Tools or other meansof data collection

Countryd Studytypec/Instruction

1 Bamberger(2013)

Product – Smell n = 65, 6thgrade

Case 1 Drawings of 3scientific models(smell, evaporation,and friction)

USA MixedYes

2 Crawford(2004)

Metamodelingknowledge

Model-it Stream ecosystem n = 14,studentteachers

Case 2 Open-endedquestionnaires

USA Qual.Yes

Case 3 Interview (n = 6)3 Danusso

(2010)Product,metamodelingknowledge

– N/A n = 388,studentteachersa

Case 4 Open-endedquestionnaires aboutmodels

IT MixedYes

Case 5 Multiple-choicequestionnaire aboutmodels

Case 6 Drawings of studentsconstructed models

4 Dori (2012) Modeling practices – Chemical reactionsin several differenttopics (e.g. energy,acid-base reactions,sedimentation)

n = 614,12th gradestudents

Case 7 Open-endedquestionnaire formodeling practices

IS MixedYes

5 Ergazaki(2007)

Cognitive strategies ModelsCreator Plant growth n = 6(3 dyads)14 years old

Case 8 Videos of studentsduring modeling

GR Qual.Yes

6 Everett (2009) Metamodelingknowledge

– N/A n = 200,studentteachers

Case 9 Yes/No questionnaire USA MixedYes

Case 10 Concept mapCase 11 Open-ended

questionnaire aboutnature and role ofmodels

Case 12 SUMS test7 Gobert (2004) Metamodeling

knowledge– Plate tectonics n = 360

(15 classes),middle &high school

Case 13 Open-endedquestionnaire fornature and role ofmodels

USA MixedYes

8 Grosslight(1991)


– N/A 7th graders(N = 33)11thgraders(N = 22)Experts(N = 4)

Case 14 Interview USA Qual.No

9 Justi (2002) Metamodeling &metacognitiveknowledge

– N/A n = 39,science andstudentteachers

Case 15 Interview BR Qual.No

10 Justi (2003) Metamodelingknowledge

– N/A n = 39,science andstudentteachers

Case 16 Interview BR MixedNo

11 Löhner (2003) Product SimQuest The temperatureregulation inside ahouse

n = 42(18 dyadsand 2triads), 11thgrade

Case 17 Studentsconversations andactions to run eitherthe system simulationor their own models

NL MixedYes

Case 18 Students models12 Löhner (2005) Product, cognitive

strategiesSimQuest The temperature

regulation inside ahouse

n = 42 (18dyads and 2triads), 11thgrade

Case 19 Studentsconversations andactions to run eitherthe system simulationor their own models

NL MixedYes

Case 20 Students models13 Louca (2008) Cognitive strategies Stagecast

Creator,Microworlds

Several phenomena n = 19(in smallgroups), 5th

Case 21 Videotaped students’group work,Videotaped class

USA Qual.Yes


Table 1 (continued)

Name of thestudy

Modeling aspect Modelingsoftware

Phenomenon Subjectsb Case # Tools or other meansof data collection

Countryd Studytypec/Instruction

Logo grade discussions14 Louca (2011) Cognitive strategies,

productStagecastCreator

Accelerated motion,relative motion,diffusion

n = 38(2 groups),

Case 22 Transcribed wholegroup video tapedconversations

CY Qual.Yes

11–12years old

Case 23 Students models

15 Louca (2011) Product StagecastCreator,MicroworldsLogo

Several phenomena(students decision)

n = 40(in smallgroups), 6thgrade

Case 24 Students constructedmodels (80 modelversions percondition)

CY MixedYes

16 Papaevripidou(2007)

Modeling practices StagecastCreator

Marine ecosystem n = 33(2 groups),5th grade

Case 25 Assessment tasks formodeling practices

CY MixedYes

17 Pluta (2011) Metamodelingknowledge

– Volcanoese n = 324(4 classes),7th grade

Case 26 Open-endedquestionnaire fornature and role ofmodels

USA Qual.No

18 Schwarz(2005)

Metacognitiveknowledge,metamodelingknowledge

ThinkerTools Force and motion n = 4classes, 7thgrade

Case 27 Modeling Test USA MixedYes

Case 28 Interview (n = 12)

19 Sins (2009) Metamodelingknowledge,cognitivestrategies

Powersim Motion andacceleration

n = 26(13 dyads),11th-grade

Case 29 Open-endedquestionnaire aboutmetamodelingknowledge

NL MixedYes

Case 30 Videos of Students on-screen actions andverbal communicationin dyads

20 Stratford(1998)

Cognitive strategies Model-it Stream ecosystems n = 16(8 pairs),9th grade

Case 31 Videos of studentsduring modeling

USA Qual.Yes

21 Treagust(2002)


– N/A n = 228, 8th,9th, 10thgrades

Case 32 StudentsUnderstanding ofModels in Science(SUMS) Test.

AU Quant.No

22 van Borkulo(2012)

Cognitive strategies Co-lab Global warming n = 74, 11thgrade

Case 33 2 paper-and-penciltests, 1 domain-independent and 1specific for the domainof energy of the Earth

NL Quant.Yes

23 Van Driel(1999)

Metamodeling andmetacognitiveknowledge

– N/A n = 86,scienceteachers

Case 34 Open-endedquestionnaire onmodels and modeling

NL MixedNo

Case 35 Likert-type scalequestionnaire onmodels and modeling

a The research was implemented in 3 phases: (a) Survey phase: n = 180 prospective teachers, (b) 1st: trial: n = 115 prospective teachers, (c) 2nd trial:n = 93 prospective teachers.

b Where the number of students (n) is stated in the ‘‘tools or other means of data collection’’ column, the tool was implemented in a subsample of thewhole, which is stated in the brackets.

c Quant. = studies using quantitative methodology, Qual. = studies using qualitative methodology, Mixed = studies using mixed methodologies.d IT = Italy, IS = Israel, GR = Greece, BR = Brasil, NL = The Netherlands, CY = Cyprus, AU = Australia.e This study did not implement any instruction, but the assessment instrument was extensive and its subject was volcanoes.


2.4.4. PhenomenonWhen modeling-based learning instruction was implemented, the phenomenon under study was reported (e.g. smell,

ecosystems, volcanoes, etc.).

2.4.5. Research questionsThe research questions and/or purpose of the study were reported.

2.4.6. Subjects (and level of instruction)In the studies where intervention was implemented, the level of instruction was recorded as a specific level (e.g. grade 5

or 5th graders) or as an age range (e.g. 10–11 years old). Details about the number of students comprising the sample and theway they were organized during instruction were noted down.


2.4.7. CasePlease see Section 3 for details on this.

2.4.8. Tools or other means of data collectionThe data collection tools used in each study was recorded, and in cases where the number of subjects using each tool was

not constant, the corresponding sample is presented in a parenthesis.

2.4.9. Data analysis processThe data analysis process was identified for each research study with regards to each data collection tool or group of tools.

2.4.10. Principal outcomes of the researchThe main findings of the research were summarized in relation to each of the research questions.

2.4.11. CountryThe country where the research took place is stated as an abbreviation.

2.4.12. Study typeStudies that used qualitative or quantitative methodology to answer the research questions were classified as ‘‘qualita-

tive’’ or ‘‘quantitative’’, respectively. In cases where both methodologies were used, the term ‘‘mixed’’ was employed (firstword of column 10 of Table 1).

2.4.13. InstructionTo identify classroom intervention studies the word ‘‘yes’’ is stated at the end of the last column of Table 1. There were no

studies in our list that included other types of instruction.

2.5. Data synthesis

Our goal pertained to integrating results from various types of primary research (quantitative, qualitative and mixedmethodologies) into a map describing assessment of the modeling competence. We conducted a synthesis, which summa-rized data from the studies and did not try to develop new theories out of them. For this purpose, we used a qualitative con-tent analysis procedure. The categories of our analysis were either pre-determined (a priori coding) or emerged inductivelyfrom the data (emergent coding) (Strijbos, Martens, Prins, & Jochems, 2006).

3. Results

In the course of the data analysis, we realized that many of the included publications concerned the assessment of more thanone modeling aspect (7 studies) and provided the effects of more than one type of assessment instruments (9 studies). To distin-guish between these effects, we differentiated between ‘‘cases’’ within publications in order to separately analyze the distincteffects of different types of assessment. In this way, within the 23 included publications, we identified 35 ‘‘cases’’ of differentassessment types (Table 1, column 7). In the remainder paper, the results will be described with reference to both studies and cases.

Fig. 2. Study selection process.

Table 2Modeling aspect assessed by each study.

Modeling aspect Number of studies

Modeling practices 2Metaknowledge Metamodeling knowledge 12

Metacognitive knowledge about the modeling process 3Modeling product 6Cognitive processes during modeling 7


3.1. Search results

After screening 803 titles and abstracts and assessing 46 full texts for eligibility, we identified 23 publications that metthe inclusion criteria (Fig. 2).

3.2. Aspects of the modeling competence

Each of the reviewed studies assessed one or more aspects of the modeling competence (Section 1). Table 2 presents thefrequency of the studies evaluating each modeling aspect.

Sixteen of the 23 studies assess only one aspect of the modeling competence (i.e. the modeling skills: 2 studies, meta-modeling knowledge: 7 studies, modeling product: 3 studies, cognitive processes during modeling: 4 studies). The remaining7 studies assessed a combination of the aforementioned modeling aspects.

Metacognitive knowledge about the modeling process was never assessed as a separate construct, but always in combi-nation with student meta-modeling knowledge. From these studies, only one explored the development of the metacogni-tive knowledge (Schwarz & White, 2005) and found no statistically significant effects. The other two studies identifiedteachers metacognitive knowledge about the modeling process before any instruction and found that it was limited (Justi& Van Driel, 2005; Van Driel & Verloop, 1999).

3.3. Types of assessment instruments or other means of data collection

Five main categories of modeling competence assessment tools have been identified: (a) interview, (b) questionnaire, (c)student work and discussions, (d) concept map, and (e) models. Table 3 presents a detailed description of these assessmenttools.

The first type of assessment is the semi-structured clinical interview (5 cases). Next, the questionnaire category includes 2main subcategories; closed questionnaire (5 cases) and open-ended questionnaire (11 cases). Closed questionnaires includequestions which provide respondents limited choices, from which they have to choose one as their answer to the question(‘‘yes or no’’, ‘‘multiple choices’’, ‘‘Likert-type’’ questions). Open-ended questionnaires allow the respondent to generate theinformation that seems appropriate as an answer to the question. Some studies assessed the modeling competence usingvideos. Videos capture students working together to construct and revise models, students discussing within their groupor with other students in the class or student actions/moves on the computer (screen capture) through computerlogs (7 cases). Transcripts of students work/discussions were generated from the videos and these constituted the meansof data collection. The concept map was also used as an assessment tool (1 case). It is widely used to represent concepts heldby learners and provide an observable statement of student conceptual schemata (Novak, 1990). Finally, student constructedmodels were used to assess their modeling competence (6 cases). Models were either expressed in form of a drawing (papermodel) or a computer model. What follows is an analysis of the results according to the abovementioned categories ofinstruments.

3.3.1. InterviewsThe five studies (cases 3, 14, 15, 16, 28) that used interviews focused on student meta-modeling knowledge.Grosslight, Unger, and Jay (1991) (case 14) used an interview to gain insights on how groups of students think about the

nature of scientific knowledge (and models) and how it is acquired. Their work is fundamental to the area of meta-modelingknowledge and many researchers subsequently used elements from both their interview and/or coding scheme in theirresearch studies about modeling. Grosslight et al. (1991) identified three general levels of thinking about models. At Level1, models are thought of as either toys or simple copies of reality and are useful as copies of actual phenomena. At Level2, students realize that models have a specific and explicit purpose and must not exactly correspond with the phenomenon.However, the main focus is still on the model and the reality, not the ideas behind it. Level 3 is characterized by three impor-tant factors; the model is constructed in the service of developing and testing ideas, the modeler takes an active role in con-structing the model, which can be manipulated and subjected to tests in the service of informing ideas. General levels wereassigned to student ideas based on six separately scored dimensions: role of ideas, use of symbols, role of the modeler, com-munication, testing, and multiplicity in model building. Students were scored as providing a level 1, 2, or 3 answers for eachdimension. Seventh and eleventh graders were assigned a 1 or 2 level score while only experts were categorized as having ageneral level 3 understanding.

Table 3Description of types of the assessment tools used.

Types of assessment Description Cases

1. Semi-structured interview A semi-structured interview is flexible, allowing new questions to be brought up during theinterview as a result of what the interviewee says.

3, 14, 15, 16, 28

2. Questionnaire2.1. Close-ended Limits respondents with a list of choices from which they must choose to answer the

question.2.1.1. Multiple choices Respondents are asked to select the best possible answer (or answers) out of the choices

from a list.5

2.1.2. Scale format(Likert questions)

Respondents specify their level of agreement or disagreement on a symmetric agree-disagreescale for a series of statements. Thus, the range captures the intensity of their feelings for agiven item.

12, 32, 35

2.1.3. Yes/no When responding to a yes/no question respondents have to choose one of the two as ananswer.

9

2.2. Open-ended question An open-ended question allows the respondents to answer the question and give theinformation that seems to them to be appropriate.

2, 4, 7, 11, 13, 25, 26,27, 29, 33, 34

3.Transcripts of student work3.1. Transcribed student orclass conversations

Videotaped conversations and actions while working in a group with an aim to construct,deploy or assess a model of a phenomenon

8, 21, 22, 31

3.2. Students actions Computer logs of students actions while using a computer software with an aim to construct,deploy or assess a model of a phenomenon

17, 19, 30

4. Concept map A diagram showing the relationships among concepts. It is a graphical tool for organizing andrepresenting the respondent knowledge.

10

5. Artifact – model5.1. Computer models Scientific models constructed by respondents through the use of a modeling software 18, 20, 23, 24,5.2. Drawings Scientific models constructed by respondents on paper (drawings) 1, 6


Two studies (cases 3, 28) (Crawford & Cullin, 2004; Schwarz & White, 2005) used post-instruction interviews to evaluatestudent meta-modeling knowledge. For these two studies, the interviews were obtained to get an in-depth view of studentknowledge, which was initially reported via open-ended questionnaires. Details about the questionnaires used can be foundin Section 3.3.2. Both the questionnaires and the interviews were based on the work of Grosslight et al. (1991).

For studies 9 and 10 (Justi & Gilbert, 2002, 2003), the interview was the main instrument for collecting data pertaining toteacher meta-modeling knowledge and metacognitive knowledge about the modeling process. Both studies refer to the sameinterview and participants (39 science teachers). Data analysis indicated the existence of seven ‘aspects’ of teachers notionsof a model; the nature of a model, its use, the entities of which it consists, its relative uniqueness, the time span over which itis used, its status in the making of predictions, and the basis for the accreditation of its existence and use. These researcherssuggest, however, that the teachers do not hold coherent ontological and epistemological views of models and modeling, asthey were not able to identify ‘profiles of understanding’ for individuals that cut completely across the seven aspects (Justi &Gilbert, 2003). The same researchers reported the knowledge and skills that science teachers think are needed to produce amodel successfully (Justi & Gilbert, 2002). According to those teachers, the purpose of modeling is to construct a model,which will be almost identical to the phenomenon, based on the audience needs. The phenomenon should be known inadvance and a modeler needs to have good manual and observation skills, abstract thinking and logical thought.

In summary, all studies using interviews as assessment instruments evaluated meta-modeling knowledge and used thework of Grosslight et al. (1991) as a foundation for their methodology both with regards to the means of data collection andthe analysis procedures that followed.

3.3.2. Questionnaires3.3.2.1. Open-ended questionnaires. Open-ended questionnaires were used in 11 studies (cases 2, 4, 7, 11, 13, 25, 26, 27, 29,33, 34). Most of them (8 cases) investigated student meta-modeling knowledge, two of them examined student modelingpractices and one assessed student cognitive processes during modeling. Additionally, the questionnaires were used forthe assessment of the effectiveness of specific instruction units in 5 studies (cases 2, 4, 11, 13, 27).

Schwarz and White (2005) (case 27) administered an open-ended questionnaire on the nature of models and modeling andreported that after instruction, students were able to identify abstract models and understand that a model is a representationthat predicts and explains. Most students understood that there can be multiple models for the same phenomenon, there canbe incorrect models, and models are estimates of the physical world. They learned that scientific models are useful in a widevariety of ways including visualization, testing theories, predicting phenomena and helping people understand science.

Crawford and Cullin (2004) (case 2) implemented an intervention that included designing open investigations, followedby building and testing dynamic computer models. This implementation proved to be successful, as students developedmore articulate and robust ways to talk about scientific models. These researchers report, however, that when all questionswere taken as a whole for each participant, none of the participants jumped completely from a level 2 to a level 3 under-standing of models as suggested by Grosslight et al. (1991).

Danusso, Testa, and Vicentini (2010) (case 4) also implemented an intervention about models and modeling, based on theidea of ‘‘teachers as learners in a learning community’’. The data extracted from an open-ended questionnaire helped them


form the alternatives provided in a closed-type questionnaire (see Section 3.3.2.2). The data were coded into five modelingclusters (MCs): MC1 included students with good understanding of the model nature, components, and functions, and MC5included those with poor understanding of scientific models.

Gobert and Pallant (2004) (case 13) implemented modeling-based instruction, which was evaluated as successful sincestatistically significant changes were identified when comparing student answers to the pre and post open-ended question-naire. When correlating these results to content knowledge, they reported that students who had a more sophisticatedunderstanding of the nature of models were better able to use this knowledge to impact their content learning as comparedto their epistemologically less-sophisticated peers.

Finally, in the research of Everett, Otto, and Luera (2009) (case 11), student meta-modeling knowledge was improved sig-nificantly when comparing their performance prior to and after the implementation of the ‘‘Science Capstone Course’’ focus-ing on models as they are used in multiple science disciplines. Student knowledge was improved with regards to their abilityto: (a) identify models and report their types from specific examples (b) identify models as multiple representations and asexplanatory tools (c) understand the uses of models, and (d) capture the changing nature of models.

The remaining three studies that used open-ended questionnaires to investigate student meta-modeling knowledge didnot use the data to examine the effectiveness of an intervention, but rather handled them as a tool to identify student ideason the nature and role of models prior to (cases 26, 34) (Pluta, Chinn, & Duncan, 2011; Van Driel & Verloop, 2002) or after(case 29) (Sins, Savelsbergh, van Joolingen, & van Hout-Wolters, 2009) instruction.

Pluta et al. (2011) (case 26) developed an open-ended questionnaire to investigate middle school students initial ideasabout models and modeling and concluded that the majority of them demonstrated some understanding of multiple modelgoals. They also noted that students in all classes collectively identified that models can have a wide range of goals. Thediversity of student responses concerning the nature and the role of models contradicts previous research (Grosslightet al., 1991; Treagust, Chittleborough, & Mamiala, 2002). While previous research conclude that most of the students primar-ily conceive of models as direct replicas of phenomena, only 8% of Pluta et al.’s (2011) sample share this idea. Students in thisstudy seemed to understand models as something that is not just a scale of the phenomenon.

Van Driel and Verloop (1999) (case 34) conducted an exploratory study to identify teacher content knowledge of modelsand modeling through the use of an open-ended questionnaire. Their results indicate that teachers share the same generaldefinition of models and that their content knowledge was limited and diverse. Teachers did identify the explanatory anddescriptive functions of models, but other important functions (e.g. prediction) were rarely mentioned. The researchers con-clude that most of the teachers displayed a constructivist orientation, indicating, for instance, that different models can co-exist for the same target, but, a minority of them, however, reasoned in terms of logical positivism.

Case 29 (Sins et al., 2009) included analysis of student epistemological understanding of models and modeling afterinstruction. Student knowledge was categorized at about Level 2 according to Grosslight et al.’s (1991) coding scheme.Sins et al. (2009) concluded that in comparison with studies with similar students reported in the literature, the studentsin their sample (11th-graders) hold relatively advanced epistemological viewpoints after modeling-based instruction.

Studies 4 and 16 (Dori & Kaberman, 2012; Papaevripidou, Constantinou, & Zacharia, 2007) focused on the improvement ofstudent modeling practices after implementing modeling-based instruction. Both studies used open-ended questionnairesand reported statistically significant improvement of students modeling practices1; which were, however, defined differently.Dori and Kaberman (2012) measured modeling practices as the sum of specific modeling skills which mainly apply for chem-istry models and require (a) understanding of the symbolic level and molecular structure in order to perform transformationamong the various chemical representations and (b) transferring between symbols or models on the one hand and the micro-scopic, macroscopic, and process levels of understanding in chemistry on the other hand. Papaevripidou et al. (2007) measuredmodeling practices as a sum of six different sub-categories: (i) model formulation; (ii) extraction of information from a givenmodel; (iii) model revision through comparison with the corresponding phenomenon and formulation of ideas for improve-ment; (iv) comparative evaluation of models of the same phenomenon; (v) appreciation of the purpose and utility of models;and (vi) reflection on the process of model development and refinement. The definition of modeling practices provided byPapaevripidou et al. (2007) is proposed for all domains of the natural sciences and relates to a broad range of phenomena.

Finally, van Borkulo, van Joolingen, Savelsbergh, and de Jong (2012) (study 22) is the only study in our review that used aquestionnaire to examine the effectiveness of an intervention with regards to student cognitive processes during modeling.The questionnaire was developed to assess student complex or simple higher order skills; declarative knowledge, applica-tion, construction, and evaluation. They used the results of the questionnaire to compare a modeling-based interventionto an expository-based intervention on the domain of the Earth’s Energy. They found that for complex tasks, the modelinggroup outperformed the expository group on declarative knowledge and on evaluating complex models and data. However,no differences were obtained in the application of knowledge or the creation of models.

3.3.2.2. Close-ended questionnaires. 3.3.2.2.1. Likert-type scale questionnaires. Cases 12 and 32 (Everett et al., 2009; Treagustet al.,2002) refer to the same assessment instrument; the Student Understanding of Models in Science Test (SUMS test).Treagust et al. (2002) developed the SUMS test, which includes 27 Likert-type questions, aiming to gain insight into student

1 In both studies the term modeling skills is used. However, for the sake of uniformity, we will use the term modeling practices as defined in the introductionof this paper, which is in line with what these researchers call modeling skills.


understanding of the role of scientific models in science. Analysis of the results confirmed the existence of five scales formeta-modeling knowledge, which were similar for different age levels (years 8, 9 and 10). The data suggested that students(a) conceive the model as a mere representation of an entity and that each representation displays a particular perspective oremphasis (scale 1), (b) understand models as exact replicas of the phenomenon (scale 2), (c) are aware of the value of thevisual representation that many scientific models provide (scale 3), (d) understand models as tools for making predictions,formulating theories and showing how information is used (scale 4), and (e) appreciate that models support scientific the-ories and that they change according to changes in scientific thinking. Everett et al. (2009) (case 12) used the SUMS test tocheck the effectiveness of a course. The results showed statistically significant improvement in four scales. The factor of mod-els as exact replicas did not show a statistically significant difference although the mean decreased in the post-test.

Van Driel and Verloop (1999) used a Likert-type scale questionnaire (case 35) in combination with an open-ended ques-tionnaire to identify teachers content knowledge on models and modeling. The results were used to form scales of under-standing models and modeling; the first scale grouped statements referring to the relations between models and targetsin a positivist way; the second scale referred to the physical appearance of models; and the third scale grouped items thatreferred to the use and the construction of models in social contexts. Based on this analysis, they concluded that experiencedscience teachers, though they share the general notion of a model as a simplified representation of reality, may have quitedifferent cognitions about models and modeling in science.3.3.2.2.2. Other closed-type questionnaires. Case 9 (Everett et al., 2009) used a questionnaire comprised of yes/no questions totrack student ideas on different types of models through specific examples. Student ability to identify types of models wasimproved after instruction. Moreover, case 5 (Danusso et al., 2010) used a five-item closed form questionnaire to check theeffectiveness of an intervention (second trial phase). The results indicated a statistically significant improvement of studentmeta-modeling knowledge which was attributed to the effects of the instruction units.

3.3.2.3. Summary and thoughts on the studies that assessed students meta-modeling knowledge. All studies that used question-naires to explore students meta-modeling knowledge, borrowed or adapted the questions and the analysis procedures ofGrosslight et al. (1991). Studies 2 and 6 (Crawford & Cullin, 2004; Pluta et al., 2011) however question the functionalityof this coding scheme. These researchers believe that the three levels of understanding are too broad to be useful as mea-sures of growth in knowledge of models and that these levels are not sufficiently focused on the important aspects of mod-eling such as evaluation of models for teaching, specific examples of models, and types of models. For instance, Pluta et al.(2011) report that the diversity of student responses (in their study) surprised them, given that previous research on studentunderstanding of modeling has reported that most students see models only as direct replications of reality. To overcomethis problem, Crawford and Cullin (2004) proposed a more fine-grained analysis to determine changes in views of modeling.

The critique of this researchers on the coding scheme of Grosslight et al. (1991) imply that new assessment techniques areneeded to evaluate students meta-modeling knowledge. We elaborate on this issue in the discussion section.

3.3.3. Concept mapA concept map was used to trace students understanding of the terms employed during instruction and their knowledge

of models in case 10. Student responses on the concept maps showed a statistically significant improvement from pre- topost evaluation, a fact that supplements the results of the other three types of assessment used in the study of Everettet al. (2009) and was claimed to indicate the effectiveness of the implemented instruction.

3.3.4. Student constructed models3.3.4.1. Paper models. Danusso et al. (2010) (Case 6) asked students to design/draw a model as well as identify the compo-nents and functions of two given paper models (drawings) of a real phenomenon. The data were analyzed using contentanalysis techniques based on the MC (see Section 3.3.2.1 for details) categorization to investigate if the nature, components,and functions of the models had been correctly recognized. The researchers compared the results of the open-ended ques-tionnaire and the data derived from student constructed paper models (both coded using MCs) and found that the interven-tion (first trial phase) was not successful in promoting student meta-modeling knowledge. Therefore, a second trial phasefollowed in order to overcome this problem. It is important to state that this study is the only one among the 23 underreview that reports use of assessment data to modify the teaching approach. Even though it does not meet the requirementsof our definition of formative assessment, we note that it is the only one that makes use of some kind of formative assess-ment techniques.

Students constructed paper models were the subject of the study of Bamberger and Davis (2013) (case 1). They analyzedpre- and post-instruction student efforts to draw models of smell, evaporation and friction. The modeling performances wereevaluated based on the four elements of modeling: explanation, comparativeness, abstraction, and labeling. For the smellmodel (same content as instruction), student performance was improved significantly with respect to content understandingas well as for most of the modeling performances. For the evaporation model (near transfer from instruction), performancewas improved in a statistically significant way with respect to the content knowledge. After instruction student models werebetter in a statistically significant way only for the explanation domain. For the friction model (far transfer from instruction),student performance was not better after instruction as far as the content knowledge, but their models were better withrespect to the modeling performances and specifically for the explanation, abstraction, and labeling domains. These findings


point to relative independence between content knowledge and modeling performances, when modeling performances aredefined separately from the content accuracy. The researchers claim that scientific concept-process models can serve as use-ful instructional tools in middle-school science, and can be used as a means for students to develop understanding of not justcontent, but also the modeling competence.

3.3.4.2. Computer models. Cases 18 and 20 (Löhner, van Joolingen, & Savelsbergh, 2003; Löhner, van Joolingen, Savelsbergh, &van Hout-Wolters, 2005) include the analysis of student constructed models which emerged from modeling-based learningwith the use of a graphical or a textual modeling representation tool. The quality of the models was determined by takinginto account the number of correct and incorrect relations, relation being the influence one variable on another. In otherwords, they compared the models to the ‘‘correct’’ model of the phenomenon and counted the number of correct minusthe incorrect included relations. Löhner et al.’s (2003) (case 18) results indicated that the pairs of students working in thegraphical representation tool constructed more complex and higher quality models and ran simulations of more differentmodels than those working in the textual representation. Additionally, the groups of students in the graphical condition pro-duced better models than those in the textual condition (Löhner et al., 2005) (case 20). They also found a significant positivecorrelation between the amount of time spent on model evaluation and the highest model score and a significant negativecorrelation between the amount of time spent on off task communication and the model score. A significant positive corre-lation between the time spent on formulation of hypotheses and the model score was also identified. However, all those sig-nificant correlations disappeared, except the negative correlation between the amount of time spent on off taskcommunication and the model score, when looking at the separate conditions (graphical and textual condition).

Louca, Zacharia, and Constantinou (2011) and Louca, Zacharia, Michael, et al. (2011) (cases 23 and 24) used the same meth-odology to analyze student constructed computer models. Artifact analysis was implemented to examine the ways that stu-dents represented different elements in their models: physical objects (characters), physical entities (variables), physicalprocesses (procedures), and physical interactions. Unlike case 1 (Bamberger & Davis, 2013), in cases 23 and 24, a fifth elementwas introduced in the analysis; coding for whether a model adequately depicted the surface structure of the phenomenon.Louca, Zacharia, and Constantinou (2011) (case 23) described a framework for evaluating student constructed models andfor monitoring the progress of these models over the course of the modeling procedure. The methodology used in study 15(case 24) was based on the framework developed by Louca, Zacharia, and Constantinou (2011). Their analytical frameworkappears to: (a) provide a coherent system for analyzing and evaluating student constructed models based on the aforemen-tioned structural components; (b) capture the model’s sophistication across the different structural components; (c) provide apicture of model progress over time; and (d) be sensitive enough in terms of capturing the differences between models con-structed with different types of computer-based modeling media. Louca, Zacharia, and Constantinou (2011) described threedistinct modeling frames that students worked in during modeling-based learning in the context of constructing models ofphysical phenomena with the modeling tool Stagecast Creator. Models created by students during modeling frame I (InitialPhenomenological Description) simply showed scenes from the phenomenon in temporal sequence, without any referenceto an underlying causal agent that brought about a certain change. Modeling frame II (Operationalization of the Physical Sys-tem’s Story) led students to the construction of models of physical phenomena including both causal and non-causal repre-sentations, possibly indicating that this was a transitional frame of student work within the micro-context of constructingmodels of physical phenomena. Some of the student constructed model features represented the causal mechanism thatunderlined the phenomenon, whereas some others continued to represent simple descriptive features of the overall phenom-enon. Models constructed during or after modeling frame III (Construction of Algorithms) were, in all cases, causal models inthe sense that they contained representations of physical entities represented in the form of variables, operationally definedphysical processes and relationships between physical objects, physical processes, and physical entities.

3.3.4.3. Summary and thoughts on the studies that assessed student models. Assessment of the modeling product, i.e. studentconstructed models, seem to be performed by a range of methods which are indeed very different when compared. Themethod used by Löhner et al. (2003, 2005) (Cases 18 and 20) is based on a norm model, which is considered correct, asthe basis for the comparison with the learner constructed models. The comparison of learner constructed models to normmodels limits the possibility of assessment, as there is often natural (philosophical or structural) variation in the models cre-ated in a approach. Moreover, teachers have to develop norm models for each part of the curriculum that is taught throughmodeling. The method used by Louca, Zacharia, and Constantinou (2011) and Louca, Zacharia, Michael, et al. (2011) to ana-lyze student constructed models does not include a norm model but is rather based on the analysis of the constituent com-ponents of each student model, which are then evaluated one by one. This allows for identifying the details of each modeland providing a score.

Danusso et al. (2010) categorized paper models (drawings) made by learners according to five modeling clusters (MCs).Each modeling cluster is described and characterized by a short description of ten words accompanied by a short example.It is therefore unspecified how the researchers matched the student models to the particular clusters. Louca, Zacharia, andConstantinou (2011), Louca, Zacharia, Michael, et al. (2011) and Bamberger and Davis (2013) move beyond this approachby being more fine-grained. They focus on the model details based on several parameters (Louca and colleagues: physicalobjects, physical entities, physical processes, and physical interactions; Bamberger and Davis: explanation, comparativeness,abstraction, and labeling), which are also further divided into subcategories. This more complex system of assessment safe-guards that assessment identifies the possible differences between models, and clusters them into the correct final categories.


3.3.5. Student work and discussions3.3.5.1. Student or class conversations (videos). Stratford et al. (1998) (case 31) used qualitative video analysis to identify thecognitive strategies for modeling in which ninth graders are engaged when constructing dynamic models of stream ecosys-tem phenomena. They developed the Cognitive Strategies for Modeling Framework (analyzing, reasoning, synthesizing, test-ing/debugging, and explaining) to analyze their data. Their analysis indicated that most students: (a) engaged in an analysisof appropriate objects and factors for their model (b) engaged in relational reasoning about model factors, (c) were able tosynthesize a working model, employing a range of strategies; (d) attempted to articulate explanations for their relationships,but sometimes explanations were shallow or even non-existent, and (e) tested their model, though some tested their modelmuch more substantially and thoroughly than others. However, only a few persisted in their debugging to fine-tune thebehavior of the model to match their expectations.

Ergazaki, Zogza, and Komis (2007) (case 8) adapted the Cognitive Strategies for Modeling Framework developed byStratford et al. (1998) to analyze students cognitive processes during modeling. Their results indicate that a wide rangeof modeling ‘operations’ is activated in the context of the three major modeling processes (analysis, synthesis, testing-interpreting). Specifically, peers spent a significant part of the activity engaged in analysis without encountering seriousdifficulties. Synthesis seemed to be much more demanding. It required most of the facilitator’s cognitive support and alsopeers needed to be highly involved and critical as well. Testing and interpreting held the smallest part of the activity. Thecognitive support provided within this action seemed limited but it was significant and it was usually triggered by thefacilitator.

Louca and Zacharia (2008) (case 21) analyzed videotaped student group work with Microworlds Logo and Stagecast Cre-ator as well as videotaped discussions in the whole class about the phenomena under study to describe the ways that fifth-graders use these as modeling tools in science. They identified differences in approaches to planning, writing and debuggingcode and using code as a representation of the phenomenon. They concluded that the type of programming language hasimplications on (a) the programming process-textual language systems are more open-ended environments, enabling usersto create many kinds of routines with limited scaffolding, whereas graphical language systems restrict users to pre-definedscaffolding for creating programs; and (b) the modeling process-Microworlds Logo, a textual language system, seemed tomore easily trigger causal accounts of natural phenomena, whereas Stagecast Creator, a graphical language system, seemedto better support narrative accounts.

In addition, Louca, Zacharia, and Constantinou (2011) used contextual inquiry and analysis of student conversations togain better insight in student activity and conversation patterns while working with two modeling tools (Microworlds Logoand Stagecast Creator). Transcripts of videotaped conversations were separately coded for (a) activity and (b) communica-tion patterns. The main findings of this study appeared to show that differences in the modeling tool influence the ‘‘mode ofwork’’ that learners enter when using the tool, pushing their learning experiences into different directions with an effect onboth the programming and modeling processes.

3.3.5.2. Student actions (computer logs). Löhner et al. (2005) (case 19) compared the effects of two different modeling tools(graphical and textual versions of SimQuest) on the modeling process with a focus on differences in the way the collaborat-ing students communicate about the models they construct. The protocols of student actions and discussions were tran-scribed and analyzed with a coding scheme describing the reasoning processes during inquiry modeling. The codingscheme viewed scientific reasoning in terms of orientation, hypothesizing, experimenting, model implementation, andmodel evaluation. These researchers report that students working with the graphical representation designed more exper-iments with their own model, formulated more qualitative hypotheses, and spent more time evaluating their own modelthan students working with the textual representation. Results also indicated that many students encountered difficultiesperforming this task in a systematic manner (see Section 3.7 for details on the differences of the two tools).

Sins et al. (2009) (case 30) investigated the relationship between the level of student epistemological understanding ofmodels and modeling and the level of their cognitive processing during modeling. They coded the emerged episodes of stu-dent work in terms of evaluation, explanation, quantification, inductive reasoning and analyzing, with the depth of cognitiveprocesses being another parameter of their analysis. The study found positive correlations between epistemological under-standing and deep processing and negative correlations between epistemological understanding and surface processes. Stu-dents with higher meta-modeling knowledge were found to evaluate and elaborate on an element of their model, explain toeach other how elements within their model work, specify a quantity, elaborate upon elements within their model, and talkabout modeling elements at a deeper level as compared to those processing these modeling practices superficially.

3.3.5.3. Summary and thoughts on the studies that assessed students work and discussion. Students cognitive processes wereassessed mainly by coding schemes that have their origins in the work of Stratford et al. (1998) (study 20). Ergazaki et al.(2007) adapted this scheme and used it in their study. Löhner et al. (2005) presented an overview of the reasoning processesfound in studies of computer-based modeling, one of which was the study of Stratford et al. (1998). The work of Sins et al.(2009) was based on the framework developed by Löhner et al. (2005).

Louca, Zacharia, and Constantinou (2011) did not base their analysis on a pre-existing coding scheme, but rather devel-oped a new one, which was finalized through the data analysis. Their coding scheme was related to physical objects, physicalprocesses, and physical entities of the model under construction, which was the subject of their discussions. Similarly, Louca

Table 4Quality of tools or other means of data collection.

Name of thestudy

Tools or othermeans of datacollection

Case Based on priorwork?

Method of analysis Inter-raterreliabilitya

InstrumentValidity

Triangulation

1 Bamberger(2013)

Drawings of 3scientific models(smell,evaporation, andfriction)

1 No 1. Content analysis basedon a coding scheme(comparativeness,abstraction, and labeling)2. Statistics (paired t-test)

Explanation: 80%Comparativeness:100%Abstraction: 100%,Labeling: 100%

– No

2 Crawford(2004)

Open-endedquestionnaires

2 Grosslightet al. (1991)

Coding scheme ofGrosslight et al. (1991)

Agreement in allcases, (except for2)

– Yes

Interview (n = 6) 3 No Supportive toquestionnaire findings

– –

3 Danusso(2010)

Open-endedquestionnairesabout models

4 Pinto (2005) Content analysis ?development of modelingclusters (MCs)Statistics (Pre-postcomparison)

– Facevalidity bypriorresearchers

Yes

Multiple-choicequestionnaireabout models

5 No Analysis based on MCsStatistics (Pre-postcomparison)

– –

Drawings ofstudentsconstructedmodels

6 No Analysis based on MCsStatistics (Pre-postcomparison)

– –

4 Dori (2012) Open-endedquestionnaire formodelingpractices

7 No Content analysis (Rubric ofeach skill/Scoring)Statistics (Pre-postcomparison)

– –

5 Ergazaki(2007)

Videos of studentsduring modeling

8 Stratford(1998)

Revised coding scheme ofStratford (1998)

0.94 –

6 Everett (2009) Yes/noquestionnaire

9 Chittleboroug(2004)

Confidence intervals todetermine pre- and post-test statistical differences

– – Yes

Concept map 10 No Score for correct linksbetween given concepts

97.5% –

Open-endedquestionnaireabout nature androle of models

11 Grosslight(1991),Crawford(2004)

Coding scheme ofGrosslight et al. (1991)Paired samples t-test wasapplied

80% –

SUMS test 12 Treagust(2002)

Confidence intervals todetermine pre and posttest statistical differences

– –

7 Gobert (2004) Open-endedquestionnaire fornature and role ofmodels

13 Gobert (1997) Questions scored forsophisticationStatistics: ANOVA todetermine pre- and postdifferences

– – No

8 Grosslight(1991)

Interview 14 No Application of 3 codingschemes

– – No

9 Justi (2002) Interview 15 Grosslightet al. (1991)

No use of Grosslight et al.(1991) coding scheme.

– – No

10 Justi (2003) Interview 16 Grosslightet al. (1991)

No use of Grosslight et al.(1991) coding scheme.

– – No

11 Löhner (2003) Studentsconversations andactions to runeither the systemsimulation ortheir own models

17 No Supportive evidence toverify the conclusions ofthe quantitative data

– – Yes

Students models 18 No Scoring according to thecorrect relationshipsNon parametric statisticsapplied

– –

12 Löhner (2005) Studentsconversations andactions to runeither the systemsimulation ortheir own models

19 No Coding scheme forreasoning categories:(Orientation.Hypothesizing.Experimenting. Modelimplementation. Modelevaluation)

Main categories:0.75Subcategories(0.34 & 0.49)

– No

(continued on next page)


Table 4 (continued)

Name of thestudy




InstrumentValidity

Triangulation

Students models 20 No Scoring according to thecorrect relationshipsNon parametric statisticsapplied

– –

13 Louca (2008) Videotapedstudents groupwork, Videotapedclass discussions

21 No Contextual inquiry andconversation analysis fortriangulation purposes.Activity andcommunication patterswere identified

0.82 – Yes

14 Louca (2011) Transcribed wholegroup video tapedconversations

22 No Coding scheme: utterance-by-utterance analysis formodeling

0.87 – Yes

Students models 23 No Artifact analysis based onmodel’s elements: physicalobjects (characters),physical entities(variables), physicalprocesses (procedures),and physical interactions

0.867 –

15 Louca (2011) Studentsconstructedmodels (80 modelversions percondition)

24 No Artifact analysis based onmodel’s elements: physicalobjects, physical entities,physical processes, andphysical interactions.Statistical procedures tocompare pre- and postmodels of the twoconditions

0.867 – No


Assessment tasksfor modelingpractices

25 No Criteria for scoring thetestsMancova to determine thedifferences between thetwo conditions (post-scores = dependentvariables)

0.92 – No

17 Pluta (2011) Open-endedquestionnaire fornature and role ofmodels

26 No They developed candidatecategories based onliterature, implementedand revised them (contentanalysis)

85% – No

18 Schwarz(2005)

Modeling test 27 No Scored students responsesto pre and posttestStatistics: Paired t tests todetermine pre- and postdifferences.Differences of individualitems were determinedusing McNemar chi-squaretests

– – Yes

Interview (n = 12) 28 Schwarz(1998)

Supportive to the modelingtest results

– –

19 Sins (2009) Open-endedquestionnaireaboutmetamodelingknowledge


Adaptation of the codingscheme of Grosslight et al.(1991)

0.70 – No

Videos of Studentson-screen actionsand verbalcommunication indyads

30 No Coding scheme: (a) forreasoning categories:(evaluate, explain,quantify, inductivereasoning, analyse), (b) fordeep or surface processing]Non parametric statisticsto determine relationshipsbetween epistemologicalunderstanding andcognitive processing

0.75 –


Table 4 (continued)

Name of thestudy




InstrumentValidity

Triangulation

20 Stratford(1998)

Videos of studentsduring modeling

31 No Coding scheme: (a) forreasoning categories:(analyzing, relationalreasoning, synthesizing,testing/debugging, andexplaining

– – No

21 Treagust(2002)

StudentsUnderstanding ofModels in Science(SUMS) Test.

32 No Factor analysis identified 5scales for modelingunderstanding

– Factoranalysis forconstructvalidity

No


2 paper-and-pencil tests, 1domain-independent and1 specific for thedomain of energyof the Earth

33 No Scored students (1,0)responses to pretests andposttests)Statistical analysis ofvariance with the pretestsub-score as a covariate.

– – No

23 Van Driel(1999)

Open-endedquestionnaire onmodels andmodeling


Analysis based on theinterpretativephenomenologicalapproach

– – Yes

Likert-type scalequestionnaire onmodels andmodeling

35 No It was based on the resultsof the open endedquestionnaire.Several statisticalprocedures wereimplemented

– –

a Agreement (%) or Cohen’s kappa value.


and Zacharia (2008) developed a coding scheme through the analysis of the data, which described students conversation pat-terns and activities during modeling.

The main difference between these two groups of researchers lies in that the first group investigated students reasoningprocesses during modeling (based on pre-determined codes) while the second group used open coding to identify studentsactivities and conversation patterns during modeling.

3.4. Instruments quality

Table 4 presents information about the assessment instruments or processes used in each study. This table reports onwhether the instruments have been developed by the authors or have been adapted from previous studies and presentsthe methods and the steps followed to analyze the data deriving from these tools. The last two columns of the table referto the validity measures taken for the development of the instruments and the inter-rater reliability measures taken.

In 12 out of the 35 cases (10 studies) presented in this review paper, the assessment instruments have been borrowed oradapted from previous studies. For two of those 12 cases, the instrument derived from unpublished work (Everett et al.,2009; Schwarz & White, 2005). Six of those are based on the work of Grosslight et al. (1991), but only two of those six usedthe coding scheme of these researchers. For the rest 22 cases, instruments have been developed for the purposes of theresearch presented in the papers under review.

Inter-rater reliability of the coding procedure was calculated in 12 studies. Table 4 shows that when calculated, the CohenKappa Value for reliability was high in almost all cases. However, validity of the instruments was not calculated with theexception of case 4 (Danusso et al., 2010) where the authors mention that face validity of the instrument has been calculatedin a previous study. Additionally, construct validity was estimated by Treagust et al. (2002) in case 32 where Factor analysiswas conducted to confirm the existence of five scales for the modeling ability. The low percentage of the studies measuringinstruments validity is somehow justified by the fact all but two studies (Treagust et al., 2002; van Borkulo et al., 2012) werequalitative in nature. Qualitative researchers are often not interested in identifying validity of the instruments they are usingand usually pay attention to triangulating their results through several sources. In eight out of the 23 qualitative or mixed-type studies of our sample triangulation of the results was conducted through comparing the data of different instrumentsmeasuring the same constructs (last column of Table 4).

3.5. Educational context of the studies

Seventeen studies described a modeling-based instruction and implemented assessment prior, during or after the inter-vention (see Table 1). Table 5 presents core information about the educational context of these studies. The duration of the

Table 5The educational context of the studies.

# Name of thestudy

Instructor Type of instruction & curriculum Duration School/Institution

1 Bamberger(2013)

ClassTeacher

Project based science curriculum-authentic inquiry andscientific modeling

29 periods (45 min)(modeling) & 32periods (Smell unit)

Middle urban school

2 Crawford(2004)

Authors Design experiment-authentic inquiry and scientificmodeling

9 h (of 4 three hourmeetings)

Cornell University,Teacher Education

3 Danusso(2010)

Universityteacher

Model of teacher education in which teachers arethemselves learners who work in learning community.Teachers share their own professional growth by means ofreflective participation (Schon, 1983) to the intervention

About 12 h Post GraduateSpecialization Schoolin Secondary Teaching(2 Italian universities)

4 Dori (2012) ClassTeachers

CCL: Case based Computerized LaboratoriesCCM: Computerized Molecular ModelingBy the Technion, Israel Institute of Technology in the frameof national reform in chemical education for high schoolstudents

2 months for CCL5 months for CLL

High schools in Israel

5 Ergazaki(2007)

Researchers Part 1: software demonstration, Part 2: model construction(environmental factors interfering with plant growth)

45 min Experiment at theUniversity of Patras

6 Everett (2009) Authors Science Capstone Course: In depth examination of a majorunifying theme in science: models (5E learning cycle:Engagement, Exploration, Explanation, Elaboration,Evaluation)

1 semester University ofMichigan–Dearborn

7 Gobert (2004) 3 classTeachers

‘‘What’s on your plate’’ Unit No information aboutduration

California andMassach. High schools

8 Grosslight(1991)

– – – –

9 Justi (2002) – – – –10 Justi (2003) – – – –11 Löhner (2003) Class

teachersPart 1: software familiarization, Part 2: construction of amodel that fits the results of empirical data (simulationbased)

3 h experiment Dutch high School

12 Löhner (2005) Classteachers

Part 1: software familiarization, Part 2: construction of amodel that fits the results of empirical data (simulationbased)

3 h experiment Dutch high School

13 Louca (2008) Author Part 1: software familiarization, Part 2: construction of arepresentation of a natural phenomenon

1½ h per week(almost 2 months)

Afternoon computer/science club

14 Louca (2011) Classteacher

Part 1: software familiarizationPart 2: construction several models of natural phenomena

80 min per week for7 months

Public elementaryschool

15 Louca (2011) Classteacher

Part 1: software familiarizationPart 2: construction several models of natural phenomena

80 min per week for7 months



Author Control group: traditional worksheet-based instructionabout ecosystems, Experimental group: reconstructedmodeling unit

8 teaching hours perweek for 2 months


17 Pluta (2011) – – – –18 Schwarz

(2005)Classteachera

Model enhanced version of the ThinkerTools InquiryCurriculum

45 min per day for10.5 weeks

Public middle school

19 Sins (2009) No info Part 1: software familiarizationPart 2: Revision of a model (model-data-revision)

2 ½ h Public high school

20 Stratford(1998)

Classteacher

Part 1: software familiarizationPart 2: construction of a model about an ecosystem

6–8 daily 50 minschool class periods

Public high school

21 Treagust(2002)

– – – –


Classteacher

Part 1: software familiarization, Part 2: Group 1: model-based instruction about global warming, Group 2:expository instruction to write a report on global warming

200 min Two upper tracksecondary schools

23 Van Driel(1999)

– – – –

a Except from 1 class where the first author co-taught with class teacher.


studies varies from 45 min, an experiment with high school students held at the University of Patras (Ergazaki et al., 2007), toabout 50 h (Bamberger & Davis, 2013), a study which included the implementation of a unit to teach the modeling compe-tence (29 periods) and a unit about Smell, which included model construction (32 periods).

Additionally, all but two studies pertained to interventions implemented in a formal education context at elementary (3studies), middle (9 studies) or university (3 studies) level. As mentioned before Ergazaki et al. (2007) performed an exper-iment with high school students at the university campus, while Louca and Zacharia (2008) implemented modeling-basedinstruction in an afternoon computer/science club.

The fourth column of Table 5 provides information about the curriculum or the units of instruction and the last column ofthe table reports on the institution in which the instruction was implemented. Almost all studies implemented modeling-


based curriculum which was developed by the researchers for the purpose of their study. On the contrary, the curriculumimplemented in the study of Dori and Kaberman (2012) was developed at the Technion Israel Institute of Technology inthe frame of national reform in chemical education for high school in Israel. Therefore, the two units, the ‘‘Case based Com-puterized Laboratories’’ and the ‘‘Computerized Molecular Modeling’’, were part of the national curriculum of the country.

Class teachers taught the curriculum implemented in 11 studies (Table 5, column 3). In five studies the authors or otherresearchers played the role of instructor and for study 19 (Sins et al., 2009) no information is provided concerning the iden-tity of the instructor.

3.6. Participants age

The studies investigated a broad range of ages and educational levels. Details on the age levels are provided in Table 6. Thesample of some studies belonged to more than one categories, and this is the reason that the total number of studies in thelast column of Table 6 does not correspond to the total number of the studies under review.

When comparing the results of the different studies, we conclude that similar outcomes regarding the five aspects of themodeling competence (Table 2) are reported for both younger and older students. For example, in study 8 (Grosslight et al.,1991), which compared student understanding of the nature of models in different grades, the ideas of all students were cat-egorized as level 1 or 2 and nobody exhibited a general level 3 understanding. The results of study 21 lie along a similar vein.Treagust et al. (2002) confirmed the existence of five scales for the modeling ability (see Section 3.3.2.2.1) but statistical anal-ysis showed no significant differences for any of the scales between year levels (years 8, 9 and 10).

Additionally, other studies report improvement of student modeling practices (study 4: 12th graders; study 16: 5th grad-ers), student meta-modeling knowledge (studies 3 and 6: university students; study 18: 7th graders; study 19: 11th graders)and student models (studies 1 and 15: 6th graders; studies 12 and 13: 11th graders) after implementing modeling-basedinstruction at several and different student ages.

3.7. Types of modeling tools used

Most of the reviewed research studies (n = 17) included modeling-based instruction. Thirteen studies used one or moremodeling software as a means to using or constructing models. The software information for all reported studies is presentedin Table 1. It seems that there is no dominant software used for research studies aiming to develop the modeling compe-tence. Löhner (2005) suggests the taxonomy shown in Table 7 for categorizing computer-based modeling media. Followingthis taxonomy, it seems that eight of the studies presented in this review paper employed a diagram oriented medium, twostudies implemented computer modeling through the use of an emergence-based modeling tool, two of the studies usedboth an equation oriented medium and an emergence-based modeling tool and two studies used a combination of diagramoriented medium (i.e. Colab-simulation-based modeling tool) and an equation oriented medium (i.e. Colab-text-based mod-eling tool.).

Löhner et al. (2005) (study 12) highlighted the need for introducing a semi-quantitative diagram-based tool to novicemodelers as they found that students using this tool (graphical condition) designed more experiments with their own model;formulated more qualitative hypotheses; spent more time evaluating their own model, and supported their hypotheses moreoften by reasoning with a mechanism than students in the textual condition. Moreover, learners constructed more complexand of higher quality models and run simulations of more different models than those working in the textual representation.These researchers conclude, however, that after building more modeling experience, using equation oriented tools shouldalso lead to the normative model. To provide this experience, modeling should be incorporated in the regular science lessonsfor a longer period of time, instead of being presented as a separate activity for a very limited amount of time as is the case inmost methods used in schools.

Louca and Zacharia (2008) and Louca, Zacharia, and Constantinou (2011) compared the ways in which students used twodifferent tools, an equation oriented medium (i.e. Microworlds logo) and an emergence-based modeling tool (i.e. Stagecast

Table 6Frequencies of student educational level and ages.

Educational level Age Studies

Elementary level 10–12 years old 1, 13, 14, 15, 16 (n = 5)Lower secondary (middle school) level 12–15 years old 5, 7, 8, 17, 18, 20, 21 (n = 7)Higher secondary (high school) level 15–18 years old 4, 7, 8, 11, 12, 19, 21, 22 (n = 8)Teacher education Pre-service Elementary education Varies 6, (n = 1)

Secondary education 2, 3, 9,10 (n = 4)In-service Elementary education 9,10 (n = 2)

Secondary education 9,10,23 (n = 3)Other studies University teachers Chemistry Varies 9,10, (n = 4)

Experts A science museum directorA high school physics teacherA professor of engineering and educationA researcher

Varies 8 (n = 1)

Table 7Classification of computer modeling tools (Löhner, 2005).

Computer modeling tools

Systems modeling tools Emergence based modeling tools

Equation oriented tools Diagram/structure oriented tools

Programmingenvironments

Equation basedmodeling tools

Quantitative diagrambased modeling tools

Semi-quantitative diagrambased modeling tools

Object based modelingtools

Cellular automata

A certain amount ofprogramming isrequired

Noprogrammingrequired

Precise (quantitative) Imprecise/intuitive (semiquantitative/qualitative)

Describe interaction ofsmall grained units

Describeinteraction of cellson a grid

e.g. LOGO, Boxer e.g. DMS,Modellus,Dynamo

e.g. StellaPowersimDynasisVensimFunctional Blocks

e.g. IQONVISQLinkItModelIt

StarlogoOOTL’s

AgentsheetsStagecast Creator


Creator). In case 21 (Louca & Zacharia, 2008) differences in approaches to planning, writing and debugging the model andusing the model as a representation for the phenomenon were identified. More specifically, the researchers recorded thatthe type of the modeling medium has implications on the modeling process, as equation oriented media seemed to moreeasily trigger causal accounts of natural phenomena, whereas emergence-based oriented media seemed to better supportnarrative accounts. In case 24 (Louca, Zacharia, Michael, et al., 2011), a framework for evaluating student constructed modelsand for monitoring the progress of these models over the course of the modeling process is described. The researchers exam-ined whether this framework could capture differences between models that are created by using the two aforementionedmodeling media. Their findings indicate that the models developed with the emergence-based modeling tool included moreinteractions among physical objects and more representations of object behavior in a causal manner comparing to thosedeveloped by the equation oriented medium. Moreover, more models in the latter medium did not have any interactionsamong physical objects and included semi-causal representation of object behavior.

4. Discussion and conclusions

With this review and synthesis of empirical research we have presented a map of the diverse assessment techniques andinstruments used to evaluate different aspects of the modeling competence. Assessment is important in science educationfor monitoring student learning, for formulating feedback and also for evaluating tools and designed instructionalapproaches. As a result of this synthesis, various issues become pertinent about the assessment of the modeling competence,all of which provide insights for future research. For better clarity we have organized our discussion into three separate sub-sections. In the first section, we have highlighted where the domain is at i.e. what we have learnt but also what needs to beachieved through future research. Next, we discuss the need for developing a unifying framework of the modeling compe-tence; finally, we present the strengths and limitations of the study.

4.1. What do we know and what needs to achieved through future research

When reviewing these 23 studies for the purposes of this paper, we identified some important questions that need to beaddressed by research on assessment of competences: What is valuable to assess? What are the analytical dimensions ofeach construct that needs to be addressed? What instruments and what methods can be developed to assess these dimen-sions? How can these instruments be used productively in the classroom (formative and summative assessment) to supportteaching and learning? What relationships can research explore, given the availability of valid instrumentation? Below wetry to attend to these questions.

The modeling competence is one of the core competences that need to be fostered at all educational levels (NRC, 2012).Assessment of the modeling competence and therefore its constituent components should also be considered as an impor-tant domain of research in science teaching and learning and also of science education. We have carried out our review fol-lowing a framework which analyzes the modeling competence into two broad categories, namely modeling practices andmeta-knowledge (Nicolaou, 2010; Papaevripidou, 2012; Papaevripidou et al., 2014) (Fig. 1). Each of these categories is furtheranalyzed. Model construction, model use, model comparison, model validation and model evaluation and revision are prac-tices which along with the metacognitive knowledge of the modeling process and the epistemological awareness about thenature and role of models, have been proposed as important dimensions of the competence. In our review we have collectedthe studies that sought to analyze modeling and we have coded them based on these dimensions. The outcome reveals aneed to rethink what these dimensions of the modeling competence are and how we can assess them coherently as wellas holistically. The available research has not progressed to the stage where individual scales have been developed and val-idated. There are various proposals on the dimensions of the modeling competence but this review clearly demonstrates thatadditional research work is necessary before we can have a valid scale for each of these dimensions.


This review indicated that the most commonly used tools for assessing the modeling competence are (a) questionnaires(open-ended, closed, Likert-type, other), (b) interviews (mostly after intervention interviews), (c) concept maps, (d) studentconstructed models (paper of computer-based), and (e) coding schemes for analyzing students ongoing work (through com-puter log files and/or videos).

Typically, these tools have been used as summative assessment instruments, prior to and after instruction, mainly forresearch purposes. Clearly there is a need to further elaborate the theoretical framework for assessing the modeling compe-tence so as to also inform and support attempts to undertake formative assessment. For example, based on the modelingcompetence framework, the results of open-ended questionnaires which address each modeling competence componentcould lead to the formulation of rubrics of students level with regards to each component of the modeling competence(e.g. student attainment levels in constructing models, student attainment levels in using models etc). These rubrics canbe used both for summative and formative assessment purposes. Using these rubrics one can identify students level for eachmodeling component prior to and after a teaching intervention. This can provide teachers with information about the var-iability within their class in terms of the levels of competence. This, in turn, can be used to:

1. Adjust instruction so as to better address the needs of the students;2. compare students levels prior to and after instruction with the aim to evaluate its effectiveness and identify aspects of

the teaching intervention that could be improved;3. provide feedback to students about what they can do well and what to work on in order to further improve their ability

but also in order to become more autonomous and gradually learn to self-regulate.4. inform students about their (final) performance.

These rubrics can also be used formatively by the teacher or by peers (in cases where peer assessment and feedback isimplemented) during instruction. For example, when students work on constructing successive models of a specific phenom-enon, e.g. the day–night cycle, the resulting models could be evaluated and included in one of these levels for:

1. Comparison purposes. Model 1 could be compared with model 2 in a form of artifact analysis to track student perfor-mance during the intervention. Additionally, students performance could be compared with her performance prior tothe instruction (questionnaire).

2. Scaffolding purposes. Teacher or peers who evaluate the constructed model could indicate to the modeler how hermodel could be improved in order to reach a more advanced level.

The modeling-based learning cycle is a cyclical procedure which produces feedback that can be used by learners as a basisto deploy their next model. Feedback can be a result of

1. Students own assessment (comparison of the model with the data or with their own experiences).2. Peer assessment (evaluation of the model by another group of students).3. Teacher assessment.

These feedback processes could be enhanced when students and teacher’s work is scaffolded by specific rubrics. However,the process of providing feedback can only be productive and meaningful if the assessment is situated in a theoretical per-spective that enables interpreting where students are, with respect to what is assessed, and what is the correspondingdesired learning goal. Finding a mechanism to make more coherent connections between theoretical understanding of mod-eling, the assessment of instruments and the interpretation of the evidence of student learning that is collected with theseinstruments can lead to more powerful feedback for scaffolding learning and autonomy.

Additionally, there seems to be a lack of research examining the impact of interventions using modeling-basedinstruction on objective assessments of knowledge, on student motivation and on student autonomy and self-regulationin learning.

4.2. The need for a unifying framework of the modeling competence

The data presented in this paper corroborate the claim made in the literature that different studies use different defini-tions of the modeling competence and its constituent aspects. More specifically, in the reviewed papers the modeling com-petence was neither defined nor assessed in a unified manner. Usually, each study defined and assessed only one part ofwhat can be conceptualized as the modeling competence, based on the available theoretical frames (NRC, 2012; Penneret al., 1997; Schwarz et al., 2009; Stratford et al., 1998). Even in cases where one particular aspect of modeling was underinvestigation, researchers often used different definitions, and, consequently, different assessment approaches. For instance,the two studies assessing student modeling practices (Dori & Kaberman, 2012; Papaevripidou et al., 2007) defined them in adissimilar manner. The same applies for the cognitive processes and the modeling product, which are also defined differentlyby different researchers. For example, explanation, comparativeness, abstraction and labeling constituted the coding schemeused by Bamberger and Davis (2013), the number of correct relations is the way Löhner et al. (2003; 2005) assess studentmodels and the ways that students represented physical objects, physical entities, physical processes and physical interac-


tions in their models comprised the coding scheme of Louca, Zacharia, and Constantinou (2011) and Louca, Zacharia,Michael, et al. (2011).

We suggest that the development of a more explicit modeling-based learning framework could productively serve as ameans to overcoming this fragmented diversity gap arising from the way the reviewed studies present and assess the mod-eling competence. In addition, when the studies under review are examined in light of the complete spectrum of knowledgeand skills that such a framework needs to cover, it is evident that the specific aspect of meta-modeling knowledge is over-evaluated (12 studies). It is also worth noting that the aspect metacognitive knowledge for the modeling process is alwaysassessed in combination with meta-modeling knowledge. This reinforces, those theoretical proposals which favor thesetwo aspects of the modeling competence being framed under the concept of meta-knowledge.

To conclude, an important contribution of this review paper is the demonstrated need for a more explicit and more coher-ent theoretical framework for assessing knowledge, practices and processes related to the modeling competence with an aimto not only contribute to this research domain, but also to lend support to efforts to promote modeling-based learning ineducational practice. From our review, it seems that there is no completely coherent way to conceptualize or to assess mod-eling. Therefore, a more coherent framework could help to define what is being assessed as well as the sub-dimensions ofmodeling.

4.3. Strengths and limitations

This review has several strengths and limitations. First, we conducted a broad search of more than 800 key articles in thefield. The selection criteria excluded studies that did not meet the methodological criteria. One limitation of our review couldbe that we did not extend our search to ‘‘gray literature’’ and book chapters because time, human and financial constraintsprecluded our taking this step. However, we feel that this limitation does not reduce the value of our review, mainly because,based on the information provided in chapter synopses and abstracts, these texts did not describe specific assessmentapproaches. Additionally, through a thorough search of the bibliographies of many of the included articles, we did identifysingle publications not indexed in electronic databases. Second, the review covered several types of modeling assessment;however, because of our stringent definition of modeling competence, we may have not seen assessment of other types ofmodels.

Furthermore, the articles included in this review covered broad and frequently overlapping age ranges of participants. Acrude analysis of the effects examined according to student age groups and the countries and educational systems understudy did not yield conclusive insights. The potential of reviews in mapping and exploring in depth what is known abouta particular field lies in identifying methodological and theoretical perspectives that have been useful in the topic under con-sideration, and clearly identifying gaps. From this perspective, the most important contribution of our review lies in identi-fying gaps in existing research on the assessment of the modeling competence and in revealing the need for developing amore coherent theoretical framework for the modeling competence, on which future research can be based.

Acknowledgements

This work was co-funded by (a) the European Regional Development Fund and the Republic of Cyprus through theResearch Promotion Foundation (Project Model Frame, Contract Number: DIDAKTOR/0311/92); and (b) by the EuropeanCommission, FP7 Science in Society Program, project ASSIST-ME, Contract Number: SIS-2012-2.2.1.1-CSA-321428.

Appendix A. Supplementary data

Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.edurev.2014.10.001.

References

Bamberger, Y. M., & Davis, E. A. (2013). Middle-school science students’ scientific modelling performances across content areas and within a learningprogression. International Journal of Science Education, 35(2), 213–238.

Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 39–148.Crawford, B., & Cullin, M. (2004). Supporting prospective teachers’ conceptions of modelling in science. International Journal of Science Education, 26(11),

1379–1401.Danusso, L., Testa, I., & Vicentini, M. (2010). Improving prospective teachers knowledge about scientific models and modelling: Design and evaluation of a

teacher education intervention. International Journal of Science Education, 32(7), 871–905.Dochy, F., Segers, M., & Buehl, M. M. (1999). The relation between assessment practices and outcomes of studies: The case of research on prior knowledge.

Review of Educational Research, 69(2), 145–186.Dori, Y., & Kaberman, Z. (2012). Assessing high school chemistry students’ modeling sub-skills in a computerized molecular modeling learning environment.

Instructional Science, 40(1), 69–91.Ergazaki, M., Zogza, V., & Komis, V. (2007). Analysing students’ shared activity while modeling a biological process in a computer-supported educational

environment. Journal of Computer Assisted Learning, 23(2), 158–168.Everett, S., Otto, C., & Luera, G. (2009). Preservice elementary teachers’ growth in knowledge of models in a science capstone course. International Journal of

Science and Mathematics Education, 7(6), 1201–1225.Gentner, D., & Stevens, A. L. (1983). Mental models. Hillsdale, NJ: Lawrence Erlbaum Associates.



http://refhub.elsevier.com/S1747-938X(14)00029-3/h0005

















Gobert, J., & Pallant, A. (2004). Fostering students’ epistemologies of models via authentic model-based tasks. Journal of Science Education and Technology,13(1), 7–22.

Greca, I. M., & Moreira, M. A. (2000). Mental models, conceptual models, and modelling. International Journal of Science Education, 22(1), 1–11.Greca, I. M., & Moreira, M. A. (2001). Mental, physical, and mathematical models in the teaching and learning of physics. Science Education, 86(1), 106–121.Grosslight, L., Unger, C., & Jay, E. (1991). Understanding models and their use in science: Conceptions of middle and high school students and experts. Journal

of Research in Science Teaching, 28(9), 799–822.Halloun, I., & Hestenes, D. (1987). Modeling instruction in mechanics. American Journal of Physics, 55, 455–462.Harlen, W., & James, M. (1997). Assessment and learning: Differences and relationships between formative and summative assessment. Assessment in

Education, 4(3), 365–379.Hodson, D. (1993). Re-thinking old ways: Towards a more critical approach to practical work in school science. Studies in Science Education, 22(1), 85–142.Justi, R., & Gilbert, J. (2002). Modelling, teachers’ views on the nature of modelling, and implications for the education of modellers. International Journal of

Science Education, 24(4), 369–387.Justi, R., & Gilbert, J. (2003). Teachers’ views on the nature of models. International Journal of Science Education, 25(11), 1369–1386.Justi, R., & Van Driel, J. H. (2005). The development of science teachers’ knowledge on models and modelling: Promoting, characterizing, and understanding

the process. International Journal of Science Education, 27(5), 549–573.Löhner, S. (2005). Computer based modeling tasks: The role of external representation. Amsterdam: Unpublished Doctoral Dissertation, University of

Amsterdam.Löhner, S., van Joolingen, W. R., & Savelsbergh, E. R. (2003). The effect of external representation on constructing computer models of complex phenomena.

Instructional Science, 31(6), 395–418.Löhner, S., van Joolingen, W. R., Savelsbergh, E. R., & van Hout-Wolters, B. (2005). Students’ reasoning during modeling in an inquiry learning environment.

Computers in Human Behavior, 21(3), 441.Louca, L. T., & Zacharia, Z. C. (2008). The use of computer-based programming environments as computer modelling tools in early science education: The

cases of textual and graphical program languages. International Journal of Science Education, 30(3), 287–323.Louca, L. T., & Zacharia, Z. C. (2011). Modeling-based learning in science education: Cognitive, metacognitive, social, material and epistemological

contributions. Educational Review, 1–22.Louca, L. T., Zacharia, Z. C., & Constantinou, C. P. (2011). In quest of productive modeling-based learning discourse in elementary school science. Journal of

Research in Science Teaching, 48(8), 919–951.Louca, L. T., Zacharia, Z. C., Michael, M., & Constantinou, C. P. (2011). Objects, entities, behaviors, and interactions: A typology of student-constructed

computer-based models of physical phenomena. Journal of Educational Computing Research, 44(2), 173–201.National Research Council (2012). A framework for k-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a conceptual framework

for new k-12 science education standards. Board on science education, division of behavioral and social sciences and education. Washington, DC: The NationalAcademies Press.

Nicolaou, Chr.Th. (2010). Modelling and collaboration in learning environments (in greek). Nicosia: Doctoral dissertation, University of Cyprus (ISBN: 978-9963-689-84-2).

Nicolaou, Chr. Th., Nicolaidou, I. A., & Constantinou, C. P. (2009). Scientific model construction by pre-service teachers using stagecast creator. In P.Blumstein, W. Hung, D. Jonassen, & J. Strobel (Eds.), Model-based approaches to learning: Using systems models and simulations to improve understandingand problem solving in complex domains (pp. 215–236). Rotterdam: Sense Publishers.

Novak, J. D. (1990). Concept mapping: A useful tool for science education. Journal of Research in Science Teaching, 27(10), 937–949.Papaevripidou, M. (2012). Teachers as learners and curriculum designers in the context of modeling-centered scientific inquiry. Nicosia: Doctoral dissertation,

University of Cyprus (ISBN: 978-9963-700-56-1).Papaevripidou, M., Constantinou, C. P., & Zacharia, Z. C. (2007). Modeling complex marine ecosystems: An investigation of two teaching approaches with

fifth graders. Journal of Computer Assisted Learning, 23(2), 145–157.Papaevripidou, M., Nicolaou, Chr.Th., & Constantinou, C. P. (2014). On defining and assessing learners’ modeling competence in science teaching and

learning. In Paper presented at the annual meeting of American Educational Research Association (AERA).Pellegrino, J. W. (2012). Assessment of science learning: Living in interesting times. Journal of Research in Science Teaching, 49(6), 831–841.Penner, D. E., Giles, N. D., Lehrer, R., & Schauble, L. (1997). Building functional models: Designing an elbow. Journal of Research in Science Teaching, 34(2),

125–143.Pluta, W. J., Chinn, C. A., & Duncan, R. G. (2011). Learners’ epistemic criteria for good scientific models. Journal of Research in Science Teaching, 48(5), 486–511.Schwarz, C., Reiser, B. J., Davis, E. A., Kenyon, L., Achér, A., & Fortus (2009). Developing a learning progression for scientific modeling: Making scientific

modeling accessible and meaningful for learners. Journal of Research in Science Teaching, 46(6), 632–654.Schwarz, C., & White, B. (2005). Metamodeling knowledge: Developing students’ understanding of scientific modeling. Cognition and Instruction, 23(2),

165–205.Sins, P. H. M., Savelsbergh, E. R., & van Joolingen, W. R. (2005). The difficult process of scientific modelling: An analysis of novices’ reasoning during

computer-based modelling. International Journal of Science Education, 27(14), 1695–1721.Sins, P. H. M., Savelsbergh, E. R., van Joolingen, W. R., & van Hout-Wolters, B. H. A. M. (2009). The relation between students’ epistemological understanding

of computer models and their cognitive processing on a modelling task. International Journal of Science Education, 31(9), 1205–1229.Stratford, S. J. (1997). A review of computer-based model research in precollege science classrooms. Journal of Computers in Mathematics and Science

Teaching, 16(1), 3–23.Stratford, S. J., Krajcik, J., & Soloway, E. (1998). Secondary students’ dynamic modeling processes: Analyzing, reasoning about, synthesizing, and testing

models of stream ecosystems. Journal of Science Education and Technology, 7(3), 215–234.Strijbos, J. W., Martens, R. L., Prins, F. J., & Jochems, W. M. G. (2006). Content analysis: What are they talking about? Computers & Education, 46(1), 29.Taras, M. (2005). Assessment – Summative and formative – Some theoretical reflections. British Journal of Educational Studies, 53(4), 466–478.Treagust, D. F., Chittleborough, G., & Mamiala, T. L. (2002). Students’ understanding of the role of scientific models in learning science. International Journal of

Science Education, 24(4), 357–368.van Borkulo, S., van Joolingen, W., Savelsbergh, E., & de Jong, T. (2012). What can be learned from computer modeling? Comparing expository and modeling

approaches to teaching dynamic systems behavior. Journal of Science Education and Technology, 1–9.Van Driel, J. H., & Verloop, N. (1999). Teachers’ knowledge of models and modelling in science. International Journal of Science Education, 21(11), 1141–1153.Van Driel, J. H., & Verloop, N. (2002). Experienced teachers’ knowledge of teaching and learning of models and modelling in science education. International

Journal of Science Education, 24(12), 1255–1272.




































































assessment of the modeling competence: a systematic review and synthesis of empirical research

Documents