concept maps about chemical equilibrium and students' achievement scores

Research in Science Education, 1996, 26(2), 169-185

Concept Maps about Chemical Equilibrium and Students' Achievement Scores

Jan Wilson Griffith University

Abstract

The purpose of this study was to examine relationships between structural characteristics of students' concept maps about chemical equilibrium and independent measures of their achievement in chemistry. Fifty students in 1991 and seventy students in 1992 completed a concept-mapping task using twenty-four specified concepts. Using similarities in concept map structure, based on the presence or absence of linked pairs of concepts, non-metric multidimensional scaling (MDS) was used to plot the location of the concept maps in coordinate space. The distribution of maps was based on differences in their structure, but also reflected levels of student achievement on independent tests. The relationship between the coordinate location of each student's maps and his or her test scores on independent chemistry achievement tests was sought by canonical correlation analysis of the 1991 data set. This revealed significant relationships between the MDS coordinates and test scores of recall of knowledge and its application. Multiple regression analysis of sixty-one students' maps from the 1992 data set against their percentile rank scores on a national chemistry quiz revealed significant relationships. The results axe interpreted as revealing structural differences in conceptual organisation about chemical equilibrium among students with different levels of achievement and thus relative expertise in the domain. The significant relationship between map structure and cognitive process scores in chemistry also supports the view that the organisation of declarative knowledge influences its accessibility in cognitive tasks.

Major reviews of secondary school science education in the USA (Linn, 1986; Walberg, 1991) have drawn attention to a conventional emphasis in secondary science teaching on the recall of domain-related declarative and procedural knowledge at the expense of acquisition of conceptual tmderstanding and reasoning skills. This raises a major issue for science education and many arguments have been put forward for a shift in emphasis from knowledge transmission and reproduction to the development of problem solving and complex, multi-step reasoning skills (Smith, Blakesee, & Anderson, 1993). Bereiter and Scardamalia (1992), in a review of the relevant research, questioned whether such reasoning skills can be explicitly taught or whether performance of higher-order skills is dependent on the acquisition of organised and integrated, domain-specific knowledge structures.

The role of domain-specific knowledge in the performance of higher-order cognitive tasks has been an area of intense research interest throughout the last decade (Voss, 1988; Chi, Glaser, & Farr, 1988; Ericsson & Smith, 1991; Bereiter, 1992; Voss, Wiley, & Carretero, 1995). Research has established that complex and well-organised knowledge structures provide the basis for successful problem solving in physics (Chi, Feltovich, & Glaser, 1981; Chi, Glaser, & Rees, 1982; Eylon & Reif, 1984; de Jong & Ferguson-Hessler, 1986; Reif& Allen, 1992; Chi & Slotta, 1993). DiSessa (1993) reported that "traditional physics instruction emphasizes concepts and problem solving (exercises) and neglects the more naive piecemeal knowledge structures upon which principles are built," and emphasised the importance of acquiring organised and integrated knowledge structures as the bases for higher-order cognitive processing. In contrast, the

170 WILSON

relationship between knowledge structures and higher-order thinking in chemistry has received little attention or emphasis to date.

Research outcomes which can identify the organisational characteristics of knowledge structures will be significant in that they can be used as the bases for the development of specific learning interventions designed to optimise knowledge acquisition and organisation. On this matter, Lavoie (1993, p. 767) suggested that "Knowledge of students' information-processing behaviours can be used to develop science teaching strategies that facilitate the construction of more effective cognitive networks-networks that connect procedural and declarative knowledge in ways most conducive to solving problems, achieving conceptual understanding, and facilitating subsequent learning."

A claim that researchers investigating the nature of expertise in a specific domain "want to know the content and organisation of knowledge in that domain, the concepts and their categories and their causal relationships" and also "how the organisation of concepts for an expert differs from that of a novice" was put forward by Olson and Biolsi (t991, p. 240). If cognitive skills, such as problem solving and complex reasoning are dependent on organised knowledge structures, then, it is necessary first to elucidate the nature and degree of variation in expert and novice smactures in that domain.

This article examines how senior chemistry students represent their understanding of concepts relating to the topic of chemical equilibrium and whether characteristics of those conceptual representations are statistically independent of the performance of students on separate measures of achievement in chemistry. It is assumed that differences in levels of achievement reveal variation in the relative levels of expertise of the participating students.

Research into expert/novice differences in cognitive performance has led to a resurgence of interest in the wide range of techniques used to represent such organisational attributes of knowledge structures in many areas (Olson & Biolsi, 1991; Royer, Cisero, & Carlo, 1993; Jonassen, Beissner, & Yacci, 1993). In contrast to the wide range ofknowledge representation and elicitation techniques applied in cognitive psychology research, few have been adopted within science education, although concept mapping (Novak & Gowin, 1984; Novak, 1990) has been the most frequently used knowledge elicitation technique to date.

Most research on concept mapping has focused on its efficacy as a metacognitive tool (Horton, McConney, Gallo, Woods, Senn, & Hamelin, 1993) although the technique has also been used to document conceptual change (Novak & Musonda, 1991). Wallace and Mintzes (1990) used concept maps to investigate qualitative differences in organisation of knowledge by different groups of students; Markham, Mintzes and Jones (1994) demonstrated differences in structural complexity between maps about mammals of advanced college biology majors and beginning non- majors, whereas Wilson (1994) using the Pathf'mder technique (Schvaneveldt, 1990) reported differences in the networks derived from concept maps, of groups of chemistry students with different levels of achievement.

The question of whether students' concept map scores correlate with measures of their achievement within the domain has been the focus of few investigations to date. Novak and Gowin (1984) proposed a scoring scale for the assessment of concept map quality using number of valid propositions, levels of hierarchy, and cross links that show valid relationships. Modifications of this approach have been used by Wallace and Mintzes (1990), Mason (1992), and Liu and Hinchley (1993). Positive correlations with achievement scores were reported by Fraser and Edwards (1985) and Liu and Hinchley (1993). However, Novak, Gowin and Johansen (1983, p. 34) reported finding "that concept map scores correlate poorly with standardised test scores for typical classroom tests, but substantially with tests of transfer problem solving."

CONCEPT MAPS AND ACHIEVEMENT 171

Whether the organisational structure of students' declarative knowledge (as revealed in concept maps) is statistically independent of traditional achievement scores on independently assessed performance measures in chemistry is the focus of this paper. In contrast to earlier studies of the use of concept maps, this study does not use traditional approaches to scoring (Novak & Gowin 1984) and examines only the presence or absence of links between pairs of concepts. Labels given to the links between concepts identify their propositional relationship. These relationships vary in levels of abstraction and in some few cases, indicate the presence of a naive concept. The label given to each link is not weighted in these analyses, and is thus independent of any external judgements of the relative merit of the links. In this way, differences in the conceptual structures of participants with different levels of "expertise" in the domain are open to scrutiny.

The study uses non-metric multidimensional scaling procedures, well-established in other fields, to represent the similarities and dissimilarities in concept map structure as distances between points in multi-dimensional space. Each map is identified by a set of spatial coordinates. Correlations between the coordinate locations of each map in the three dimensional solution space and the corresponding student's achievement scores on three performance scales are examined. This procedure assists the researcher to interpret the distribution of maps as points in the configuration.

Method

Participants

Concept maps about chemical equilibrium were prepared by a total of 120 Year 12 chemistry students (with average age of 17.5 years). Fifty students from two schools (A and B) in 1991 and a further 70 students from three schools (B, C and D) in 1992 participated. All four institutions were private secondary colleges with very high proportions of the Year 12 chemistry students proceeding to university entrance.

Concepts

Twenty-four concepts were chosen (Table I ) from those specified in the syllabus for Senior Chemistry by the Board of Senior Secondary School Studies (1987) that is responsible for accreditation of curriculum proem'ares and monitoring of school based assessment for Years 11 and 12. Twenty concepts were the same in both years with three of the original (as indicated in Table 1) replaced in the second year.

Knowledge Elicitation

All students were asked to prepare a concept map to represent their understanding of the topic chemical equilibrium. Each class had completed a three-week curriculum unit on the topic in the previous lesson. The procedure followed was based on that of Novak and Gowin (1984) and the researcher modelled the procedure using concepts drawn from an unrelated domain before participants commenced the task. Participants were given an envelope containing twenty four concept labels (Table I) related to the topic (each label typed on a small piece of paper). All maps were completed within a 60 minute lesson.

172 WILSON

Table 1 Concepts Used in the Concept-mapping Task

Concepts

chemical equilibrium steady state solubility (collisions)

condensation (kinetic) volume temperature

equilibrium constant phase change ions (energy)

endothermic vapour pressure reversibility

Le Chatelier's Principle evaporation pressure

exothermic molecules concentration

gas phase products direction

equation dynamic reactants

Note: Brackets indicate replacement concepts in 1992.

Students were asked to follow four directions: to place the most inclusive concept at the top of the map; to rearrange the concept labels until their placement best represented the participant's understanding of relationships between the nominated concepts; to glue the labels to a sheet of paper; and to draw labelled arrows to represent the relationship between pairs of concepts. Figure 1 illustrates a concept map about chemical equilibrium drawn by a student in the 1992 data set with very high achievement scores in all aspects of chemistry assessment.

Achievement Measures Used

School Assessment

The student achievement measures obtained in 1991 were based on a variant of criterion- based school assessment known as standards-based assessment that is monitored by the Board of Secondary School Studies (1987) in Queensland. Four performance dimensions were assessed and scores expressed as percentages. These were (I) content (recall of knowledge of content), (2) application process (understanding demonstrated by the ability to apply knowledge in simple and complex situations), (3) scientific process (use of scientific processes in dealing with experimental results and other data), and (4) laboratory skills.

The ability to apply recalled knowledge is tested through simple, single step numerical exercises and also through novel and complex multi-step problem solving tasks. An example of an application process test item used in one school was given by Wilson (1994). Laboratory skills were not considered in the analyses as the participating schools assessed student performance on these criteria as satisfactory or unsatisfactory.

Because of the school-based nature of assessment within the state, students' results were obtained using different test items; however, comparability of levels of difficulty is monitored by the state authority.


gas phase

system in- J the ~ a r a ~ e is rised I

by /

l and o17

which is a change in the number

of

& alters the net reaction's

direction

�9 ~ is

reversibility

i n a

in kthe

which is expressed as the

in a heterogeneous

system ~t e.g.

of liquid

with sufficient

leading to

~ _ _ . . ~ ~ I and librium a new c o n s t a n ~ n t

predicts changes in the

t ~ ~affected ~redicted Le Cha by ~ " ~ P r i n c i p l e b y j

ana alters the

�9 net reaction's ~reactioaf: ects which are /

Figure 1. A concept map drawn by a student with very high acheievement scores (1992 data set) (Achievement scores: recall of knowledge, 94%; ability to apply knowledge, 90%; ability to use scientific processes, 95%; Chem Quiz score, 99%).

174 WILSON

Despite the possible influence of differences in testing standards on the data, tests revealed no significant differences between the means of distributions of test scores in schools A and B. The level of difficulty of test items used in the two schools was judged to be very similar by the participating classroom teachers.

Percentile Rank Scores on a National Chemistry Quiz

To avoid the potential influence of differences in testing standards between schools referred to above, an independent measure of conceptual knowledge was applied in 1992. The measure of achievement used as the independent variable in multiple regression analysis was percentile rank score on the Australian National Chemistry Quiz. The Chem Quiz sponsored by the Royal Australian Chemical Institute has been described and discussed by Walding, Fogliani, Over and Bain (1994).

Data Analysis

Wilson (1994) showed how the distribution of concept maps from the 1991 data set reflected participants' exit levels of achievement. This study replicates and extends the initial study and investigates the nature and extent of the statistical relationship between the location of participants' maps in the non-metric multidimensional scaling (MDS) configuration and their percentage scores on independent tests of achievement. Relationships are sought between coordinate location of each of the 1991 maps and students' scores on three performance dimensions. Coordinate location and ChemQuiz scores are investigated for the 1992 data.

To examine differences in the ways participants represented their understanding of chemical equilibrium, MDS was used to represent the individual maps as points in three-dimensional space. The assumption was made that any similarities and differences in the way that participants constructed their concept maps would be apparent in the distribution of the set of concept maps as points in coordinate space.

MDS identified the location of each concept map by a set of spatial coordinates. The three- dimensional coordinates from each MDS analysis were used as the basis for the subsequent analyses, canonical correlation and multiple regression. These analyses were undertaken to aid interpretation of the distribution of points with respect to the dimensions as suggested by Kruskal and Wish (1978).

Non-metric Multidimensional Scaling (MDS)

MDS is a set of mathematical techniques that enable a researcher to uncover the "hidden structure" of data bases" (Kruskal & Wish, 1978, p. 7). The analysis plots concept maps as points in multidimensional Euclidean space so that the distances between maps represent the strength of the association (similarity) between pairs of maps. The relationship is strongest where distances between maps are small and weakest where distances between maps are large. The number of dimensions used to represent distances between points is increased until monotonic correspondence between similarity and distance is achieved. A measure of the degree of monotonicity in any solution of particular dimensionality can be assessed by Kruskal's stress value. A value of zero percent represents perfect monotonic correspondence, five percent is good and 20% fair (Kruskal, 1964). The number of dimensions in a particular solution is chosen to minimise the stress value. The location of an individual item, or concept map in this study, in multidimensional space is described by a set of coordinates.


Input to the analysis is based on measures of "proximity": a number which indicates how similar or different two items are or are perceived to be. Output is a spatial representation, consisting of a geometric configuration of points in a specified number of dimensions. The analysis used the SPSS for UNIX programs PROXIMITIES and ALSCAL (SPSS Inc., 1990).

The items for analysis were the participants' concept maps. The concept map drawn by each participant was coded in the form of a twenty-four by twenty-four cell matrix, indicating the presence or absence of a directed arrow between pairs of concepts. All links between pairs of concepts were included, regardless of the label given to the propositional relationship by the student. Data coding was cross-checked independently by the investigator and a research assistant.

The matrices analysed by MDS involved 50 (1991 data set) and 70 (1992 data set) cases. As there were 24 concepts in each map, the number of possible linked pairs of concepts was 276 (24 x 23/2). Data were entered as a row of 276 cells per case representing the presence (1) or absence (0) of a link between the two specified concepts in the map drawn by the individual. As a first step, a proximity matrix was constructed based on patterns of co-occurrence o f pairs of concepts. Using the indicators for presence and absence of linked pairs of concepts within each case, PROXIMITIES constructs a 2x2 contingency table for each pair of cases in turn. The binary matching coefficient, the Dice similarity measure, was used to compute proximity values between cases. The Dice coefficient takes into account the frequency with which two concepts were linked to each other and also to other concepts, thus giving a measure of similarity between any two maps. The resultant proximity matrix was read by ALSCAL, which then used a Euclidean model to derive stimulus coordinates in a three-dimensional space (SPSS Inc., 1990).

= , . a i ' u . ~ I ' ' ' ? I = �9 ~ i I t . . ~ ....

0 7 3

O 6 0 0 3 ~

O 4 8

0 7 ~

0 7 4 67 66 9"10 ~ 00"76 '7"10 85

O ~ 0 9 0

0 " 7 0

Oi tO

�9 85

O

0 7 2

O o �9

0 6 6 0 8 4

O ~

71(~ O o ~ " �9 7111

0 9 O

81 0 7 9 �9 o 6 0

0 O 7 2 08"1

68 O ~ 0"79

615

O 6 8 I w s n I t l I ' I , �9 . . I , , ~ ~ I . t . ,

Figure 2. Distribution of fifty concept maps (1991 data set) on the first two of three dimensions obtained from MDS. The associated number shows the participant's percentage score on a test of recall of knowledge.

176 WILSON

' ' ' " " I ~ ' ' ' I ' ' ' ' I ' ' ' ' t ' " ' '

04. '7

o.3F/' 0 4 1

0 1 8 , 0 0~38

O ~

'.am, 4 3 0 5 6

0 5 7

07"3

o 51 o

o ~ o

06E3,

o o 0 6 9

o ~ o ~

0 8 0

, o , ~

o ~ - ~ 0 7 4 -

o 5 8 0 "

O ~

o ~ �9 0 4 6 o 7 7

g O ~ O

0"7 '0 0 4 6

O 6 8

Figure 3. Distribution of fifty concept maps (1991 data set) on the fast two of three dimensions obtained from MDS. The associated number shows the participant's percentage score on a test of ability to apply knowledge.

�9

0 .C:~:~

o .40 o

i - - i = = 7 7 7O 0

0

86 7 1 0 7 3 76 7"4 ~5 0 0

0 0 Q 7 7 ,

O 134"0 86 ~ "70

87

�9 , ! , , , , I , , , , I , , , , ! ,

Figure 4. Distribution of fifty concepts maps (1991 data set) on the first two of three dimensions obtained from MDS. The associated number shows the participant's percentage score on a test of ability to use scientific processes.


,.. ' " ' ' I ' ' ' ' I ' ' ' ' I ' ' '" ' i . . . .

. �9 63 ~ 63

�9 63 0 3 8 ~ 0 ~

�9 5 5 2 1 0 �9 �9 8 0 9 �9 1 9

-; '-~o 6 3 , 0 �9 5 �9

- 33 �9 8 3 ~

0 : 3 8 0o809c3 . 0 47 �9 2 9 �9 8 2 ~ �9 _ ! ] 6

�9 ~ ~H:) ~p 4:-/ �9

�9 47 77 ~ 38 ~ ~ 0 6 3 o , , ~ o 90

, i L - 1 0 8 6 �9 �9 �9 7 7 ,4:'2'

�9

f 4 3 �9 ~o

~" �9 63 i � 9 77

. . . . I , , I t I I , I , I e ' ' , , I . . . .

Figure 5. Distribution of sixty-one concept maps (1992 data set) on the second and third dimensions obtained from MDS. The associated number shows the partcipant's percentage score on the RACI Chem Quiz.

Canonical Correlation Analysis

To test whether distribution of 50 maps in three-dimensional space in the 1991 MDS solution was statistically independent of students' achievement, canonical correlation analysis (Tabachnik & Fidetl, 1989) using the MANOVA program of SPSS was used to test for independence between the MDS coordinates on the ftrst three dimensions and the percentage test scores on recall of knowledge, application of knowledge and use of scientific processes obtained by the respective students.

Multiple Regression Analysis

Similarly, to test whether the spatial coordinates resulting from the second MDS analysis of 70 concept maps in 1992 were statistically independent of students' percentile scores on the national chemistry quiz, multiple regression (SPSS REGRESSION) was applied using quiz score as the dependent variable and coordinates on the first three dimensions as independent variables. As nine students had not participated in the quiz, they could not be included in the analysis, leaving a data set of 61.

178 WILSON

Results

Multidimensional Scaling Analysis of the Concept Maps

The distance between any two maps in Figures 2, 3, 4 (each map is represented as e) reflects the degree of similarity in their conceptual structure. Two maps in close location are similar in terms of the pairs of concepts linked by the student. Those at greatest distance are very different. The apparent clustering of the maps of students with high achievement scores shows that these maps have a high level of commonality in the pairs of linked concepts.

Figures 2, 3 and 4 show the distribution of the 50 participants' maps in 1991 as points on the first two dimensions obtained from a scaling solution of three-dimensions. The number shown adjacent to each point represents the corresponding student's percentage scores on each performance dimension: recall of content (M = 76.22, SD = 11.92) (Figure 2); application of knowledge (M = 59.76, SD = 14.71) (Figure 3); scientific process (M = 72.90, SD = 14.07) (Figure 4).

Similarly, Figure 5 shows the distribution of 61 of the 70 maps in the 1992 analysis for which a Chem Quiz score was available. The numbers identify the students' result on the quiz (M = 60.80, SD = 24.55). As nine students did not participate in the Chem Quiz the location of their maps is not shown.

Fifty per cent of the variance in the 1991 data is accounted for in three dimensions of the MDS solution and the Kruskal stress value was equal to .25. In the larger 1992 sample (n = 70) the explained variance was 37% with a stress value of .27, In concept mapping, where no two maps are identical, the number of possible combinations of linked pairs of concepts is high. In such a highly variable system the relatively low level of explained variance results from the large number of possible paired links that could be drawn. By increasing the dimensionality of the solutions the percentages of explained variance would also increase.

MDS produces a spatial configuration of maps as points, with those maps having a high proportion of linked pairs in common placed close to each other. The distribution of maps along a dimension can be interpreted in terms of differences in their structure. In Figures 2 and 3 distribution on the ftrst (horizontal) dimension reflects differences in the initial (upper) levels of the concept maps while variation along the second (vertical) dimension reflects hierarchical organisation and complexity. Any maps located in close proximity to each other were very similar in structure and identity of linked pairs of concepts. When the corresponding test scores on tests of recall of knowledge and application of knowledge are considered, there is an apparent gradation along the second (vertical) dimension. Figure 4 shows the distribution of students' scores on the scientific process items. No apparent pattern in the distribution is discernible.

To determine whether the distributions in Figures 2, 3 and 4 are independent of the students' test scores, the coordinates which identify the spatial location of each map are used in the subsequent analyses.

Canonical Correlation Analysis

To determine whether the trends apparent in Figures 2, 3 and 4 were statistically significant, relationships between the three test scores achieved by the students in the 1991 data set and the spatial coordinates locating their maps were analysed using canonical correlation. No transformations were applied to the data and there were no missing data among the 50 cases. The canonical correlation for the fin'st of canonical variate pair was .56 (90% of variance). The canonical correlation for the first canonical variate pair was statistically significant (F=2.31,


p=.02). The remaining two canonical correlations were less than .2 and were not statistically significant. Data on the first canonical vafiate are presented in Table 2.

The standardised coefficients show that the relative loading of application process against the canonical variate pair is greater than the other two scores. The negative value of the standardised coefficient with scientific process results from the relationship between application process and scientific process scores. Students with high scores on application process generally achieve high scores on scientific process but the converse it not necessarily true. Tests of scientific process examine the student's ability to deal with experimental results and other data. Local prescriptions require that marks given to this dimension are assessed independently of content knowledge or application of conceptual knowledge. Hence, some students with low scores on application process obtained high scores on scientific process. This phenomenon accounts for the apparent anomaly in the standardised coefficients.

Multivariate regression analysis for each of the dimension variables against each of the test score variables was statistically significant (t---2.58, p=.01) for test scores on application of knowledge with Dimension 2. Other variables were not significant.

These results support statistically the earlier interpretation of the MDS output as the spatial location of individual concept maps being apparently related to the students' achievement. In particular there is a significant relationship between position on Dimension 2 and test score on application o f knowledge.

Table 2

Canonical Correlation Analysis of Test Scores and MDS Coordinates (1991 data set) for the First Canonical Variate Pair*

Other Statistics Correlation Standardised Coefficients

Dimension set

Dimension 1 (horizontal)

Dimension 2 (vertical)

Dimension 3

Per cent of variance

Redundancy

33%

10%

.07 .12

- . 9 9 - . 9 9

- . 0 9 - . 0 9

Test score set

Knowledge of content

Application process

Scientific process

Per cent of variance

Redundancy

50%

15%

.76 .58

.91 .85

.33 - .61

* Canonical correlation is .56 (F=2.31, p=.02)

180 WILSON

Multiple Regression Analysis

For the 1992 data, the MDS plot of Dimension 2 against Dimension 3 reveals a concentration of maps drawn by students with lower percentile ranks on the chemistry quiz in the upper left quadrant, with the majority of the maps drawn by students with a percentile ranks of 80 or more located below a diagonal from top right to bottom left of Figure 5. This suggests a relationship between percentile rank on the Chem Quiz and Dimension 3.

To determine whether the apparent relationship in the data presented above was statistically significant, standard multiple regression was performed between the three spatial coordinates as independent variables against the percentile rank scores as the dependent variable for the 1992 data set of 61 cases. Standardised scatter plots revealed no correlation among independent variables. Results of the analysis using SPSS REGRESSION show that coordinate location on the three dimensions significantly predicted the students' individual percentile rank. Table 3 displays the correlations between the variables, the unstandardised regression coefficients (B) and intercept, the standardised regression coefficients (fl) and R, R 2, and adjusted R z . R for regression was significantly different from zero (F (3, 57)=2.809, p< .05). The means and standard deviations of the variables in this regression analysis were: Percentile Rank, M=61, SD=-24; Dimension 1, M = -.03, SD=-I.04; Dimension 2, M=.01, SD=I.01; and Dimension 3, M=.06, SD=.97.

The three dimensions in combination significantly predicted percentile rank. Examination of the beta values for individual variables indicates a significant negative relationship between Dimension 3 and the dependent variable (r = -.27). This result supports the trend apparent in the MDS plot of percentile rank scores on Dimensions 2 and 3 (Figure 5).

Table 3 Correlations and Standard Multiple Regression Statistics for MDS Coordinates on Percentile Ranks (1992 data set)

Correlations

Percentile Dimension Dimension Dimension B 13 rank 1 2 3

Dim l -.14 -3.235 -.138

Dim2 .20 .02 4.870 .201

Dim3 -.27 .02 -.02 -6.487* -.259

R2 =.13, Adjusted R 2= .08, R = .36*, Intercept = 61.589"**. * p< .05; ***p<.001.

Conclusions

Canonical correlation of the data sets which describe 1991 achievement scores and the corresponding spatial coordinates for students' concept maps reveals a significant relationship between location and score on a test of ability to apply knowledge in simple and complex situations, as well as a test of recall of knowledge. There was no statistically significant relationship with test scores of ability to deal with experimental results and other data. Similarly, multiple regression analysis of the 1992 data set showed that the set of coordinates which describe


a map location was a statistically significant predictor of percentile rank on the national chemistry quiz, a test of conceptual knowledge.

The distribution of points in the MDS solution showed that the maps of high achieving students were placed closer to each other than maps of students with low scores. As the MDS analyses used as input patterns of similarity between maps in relation to the presence and absence of linked pairs of concepts, this indicates that the high achieving students had drawn maps which were similar in structure to each other. Maps of high achieving students had more paired linkages in common and were more alike in their hierarchical structure than those of lower ability students. The identity of the paired links was the most significant aspect of the analyses. These results demonstrate that there were structural distinctions between the maps of students with different levels of achievement and understanding of the topic of chemical equilibrium.

Although the propositional labels given to paired relationships were not used in this study, the nature of the analysis is sensitive to unusual combinations. Pairs of concepts that were rarely or infrequently linked by the participants were sometimes also labelled by propositional links indicating the presence of a naive concept or "misconception."

It is important to note that the students who participated in this study had no experience of the technique of concept mapping prior to the activity described herein. The organisational structure of knowledge relating to the topic of chemical equilibrium revealed in the concept maps was obtained from highly motivated but naive students, uninfluenced by prior training in the technique.

Within the context of the upper secondary chemistry classroom, no student would have the status of an "expert" in the sense in which this terminology is used in the expert/novice literature (e.g., Ericsson & Smith, 1991). However, within the spectrum of achievement reached within a single year level, such as year twelve, there are relative differences in the levels of developing expertise. This study has provided evidence that the differences in structure of the students' concept maps reflect such differences in expertise as shown by the significant correlations with independent achievement scores. The results presented in this article support the claim that the structural properties of declarative knowledge organisation in memory can influence its accessibility in cognitive tasks and that concept mapping can reveal qualitative information about the structure and organisation of the declarative knowledge base of the individual,

Limitations

Wilson (1994) noted some limitations associated with the procedures used in the collection and initial analysis of the 1991 data set. The reliance on school based assessment data meant that students from different schools had been assessed using different items. For this reason, the data collection was repeated with a second set of participants in 1992 and the data were analysed using the results of the Chem Quiz, a standard test of conceptual knowledge used throughout Australia. As the results have demonstrated, the statistical relationship between scaling coordinates and achievement scores is significant in both years.

In Anderson's (1982, 1987) classification of types of knowledge associated with the acquisition of a cognitive skill, he identified declarative and procedural knowledge, that is, knowing "what" and knowing "how." Concept maps give a limited perspective on a student's knowledge as they elicit the constructed relationships between concepts, that is, declarative knowledge alone and do not access the procedural knowledge of the learner. A more extensive picture of students' knowledge bases would necessitate the use of alternative knowledge elicitation techniques.

182 WILSON

Discussion

VanLehn (1989, p. 565) reviewed the f'mdings of psychological research on associative structures in memory and their relationship to problem solving and suggested that the cited studies "showed that traditional methods for measuring semantic distance or connectedness succeeded in uncovering expert-novice differences in knowledge structure, and in most cases these differences are readily interpretable in terms of their utility in solving problems."

The results of this study have shown that differences in the gross structure of concept maps drawn by students correlate with independent tests of achievement, and, as a general case, support the results obtained by Fraser and Edwards (1985) and Liu and Hinchley (1993). More particularly, the results also support the conclusions ofNovak, Gowin and Johansen (1983, p. 34) who showed that concept map scores correlated poorly with standardised test scores for typical classroom tests, but substantially with tests of transfer problem solving. This claim is supported by the results of this study, in particular, the analysis of the first data set which shows a higher correlation of map coordinates with test scores on application of knowledge (in simple and complex situations), than with test scores on recall of knowledge. The results also support the proposition that the organisational structure of conceptual knowledge in memory influences its accessibility in higher-order cognitive tasks.

Implications for Chemistry Teaching

Proponents of concept mapping have long argued (Novak & Gowin, 1984; Novak, 1990) that the technique is effective as a metacognitive tool as it assists the learner to construct and organise an effective knowledge base. However, it has been asserted within the literature on concept mapping that the organisation is idiosyncratic and that no one organisational structure is superior to another (e.g., Novak, 1990).

In contrast, in cognitive psychology, the expert/novice paradigm has been applied for many years and has been elaborated in investigations of knowledge-based systems in many diverse domains, but most notably in the development of expert systems (Cooke, 1992; Gillan, Breedin, & Cooke, 1992). Claims have been made that the knowledge bases of experts are more alike in structure and organisation than those of novices or naive learners (Goldsmith & Johnson, 1990) and as Bereiter and Scardamalia (1986) have noted, multilevel knowledge structures with many connections within and between levels are characteristics of expertise.

This study has demonstrated that differences in levels of achievement of secondary chemistry students are not independent of differences in the structure of their concept maps on the topic of chemical equilibrium. If this f'mding can be demonstrated for other "knowledge-rich" domains of study in which higher-order cognitive skills, such as problem solving and complex reasoning are dependent on the construction of extensive, ordered and integrated knowledge structures, then significant implications will be raised for science education. The f'mdings suggest that the knowledge structures of experts in knowledge-rich domains may be more similar in structure to each other than those of novices and naive students. Research outcomes that can identify the characteristics of expert knowledge structures will be significant as they can be used as bases for the development of specific learning experiences and teaching strategies to assist and support students' construction of integrated and cohesive knowledge networks and thus, facilitate students' acquisition of higher-order skills.

These conclusions are at variance with the views of Novak (1990, p. 947), the originator of the concept mapping technique, who found little of value in the cognitive psychologists' approach


to knowledge representation. He stated that "such research methodologies (are) driven largely by empiricist/positivist epistemologies" and are inapplicable to concept mapping.

Future Research

In recognition of the limitations of techniques which elicit only declarative knowledge, recent studies using the narrative of clinical interviews (Anderson & Demetrius, 1993) or prediction problem solving (Lavoie, 1993) have given a broader andmore inclusive view of the knowledge structures of learners. Further research using multiple methodologies will provide a more complete picture of the relationships between declarative and procedural knowledge and procedural skill in the domains of science, and extend professional knowledge bases for teaching science through the development of new learning strategies and teaching interventions.

Correspondence: Jan Wilson, Faculty of Education, Griffith University, Mt Gravatt Campus, Qld, 4111, Australia. Internet email: [email protected]

References

Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369-406. Anderson, J. R. (1987). Methodologies for studying human knowledge. Behavioural and Brain

Sciences, 10, 467-505. Anderson, O. R., & Demetrius, O. J. (1993). A flow-map method of representing cognitive

structure based on respondent's narrative using science content. Journal of Research in Science Teaching, 30, 953-969.

Bereiter, C. (1992). Referent-centred and problem-centred knowledge: Elements of an educational epistemology. Interchange, 23/24, 337-361.

Bereiter, C., & Scardamalia, M. (1986). Educational relevance of the study of expertise. Interchange 17(2), 10-19.

Bereiter, C., & Scardamalia, M. (1992). Cognition and curriculum. In P. Jackson (Ed.), Handbook of research on curriculum (pp. 517-542). New York: Macmillan.

Board of Secondary School Studies. (1987). Senior syllabus in chemistry. Brisbane: Board of Secondary School Studies.

Chi, M. T. H., Feltovish, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.

Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.). (1988). The nature of expertise. Hillsdale, NJ: Erlbaum.

Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. Sternberg (Ed.), Advances in the psychology of human intelligence (Vol. 1) (pp. 7-75). Hillsdale, NJ: Erlbaum.

Chi, M. T. H., & Slotta, J. (1993). The ontological coherence of intuitive physics: Commentary on diSessa's "Toward an epistemology of physics." Cognition and Instruction, 10, 249-260.

Cooke, N. J. (1992). Eliciting semantic relations for empirically derived networks. International Journal of Man and Machine Studies, 3 7, 721-750.

De Jong, T., & Ferguson-Hesster, M. G. M. (1986). Cognitive structures of good and novice problem solvers in physics. Journal of Educational Psychology, 78, 279-288.

184 WILSON

DiSessa, A. A. (1993). Towards an epistemology of physics. Cognition andlnstruction, 10, 101- 104.

Ericsson, K. A., & J. Smith (Eds.). (1991). Toward a general theory of expertise. Cambridge: Cambridge University Press.

Eylon, B. S., & Reif, F. (1984). Effects of knowledge organisation on task performance. Cognition and lnstruction, 1, 5-44.

Fraser, K., & Edwards, J. (1985). The effects of training in concept mapping on student achievement in traditional tests. Research in Science Education, 15, 158-165.

Gillan, D. J., Breedin, S. D., & Cooke, N. J. (1992). Network and multidimensional representations of the declarative knowledge of human-computer interface design experts. International Journal of Man and Machine Studies, 36, 587-615.

Goldsmith, T. E., & Johnson, P. J. (1990). A structural assessment of classroom learning. In R. W. Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organisation. Norwood: Ablex Publishing Corporation.

Horton, P. B., McConney, A. A., GaUo, M., Woods, A- L., Senn, G. J., & Hamelin, D. (1993). An investigation of the effectiveness of concept mapping as an instructional tool. Science Education, 77, 95-111.

Jonassen, D. H., Beissner, K., & Yacci, M. (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge. Hillsdale, NJ : Erlbaum.

Kruskal, J. B. (1964). Multidimensional scaling by optimising goodness of fit to a non-metric hypothesis. Psychometrika, 29, 1-27.

Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Sage University Paper Series on Quantitative Applications in the Social Sciences (series no. 07-001). Beverly Hills and London: Sage Publications.

Lavoie, D. R. (1993). The development, theory and application of a cognitive-network model of prediction problem solving in biology. Journal of Research in Science Teaching, 30, 767-785.

Linn, M. (1986). Science. In R. Dillon & R. J. Steinberg (Eds.), Cognition and instruction. Orlando: Academic Press.

Liu, X., & Hinchley, M. (1993). The validity and reliability of concept mapping as an akernative science assessment. Proceedings (electronic) of the Third International Seminar on Misconceptions and Educational Strategies in Science and Mathematics. Ithaca, NY: Misconceptions Trust.

Markham, K. M., Mintzes, J. J., & Jones, M. G. (1994). The concept map as a research and evaluation tool: Further evidence of validity. Journal of Research in Science Teaching, 31, 91- 101.

Mason, C. L. (1992). Concept mapping: A tool to develop reflective science instruction. Science Education, 76, 51-63.

Novak, J. D. (1990). Concept mapping: A useful tool for science education. Journal o f Research in Science Teaching, 27, 937-949.

Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. New York: Cambridge University Press.

Novak, J. D, Gowin, D. B., & Johansen, G. T. (1983). The use of concept mapping and knowledge Vee mapping with junior high school science students. Science Education, 67, 625 - 645.

Novak, J. D., & Musonda, D. (1991). A twelve-year longitudinal study of science concept learning. American Educational Research Journal, 28, 117-153.

Olson, J. R., & Biolsi, K. J. (1991). Techniques for representing expert knowledge. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise (pp. 240-285). Cambridge: Cambridge University Press.


Reif, F., & Allen, S. (1992). Cognition for interpreting scientific concepts. Cognition and Instruction, 9, 1-44.

Royer, J. M., Cisero, C. A., & Carlo, M. S. (1993). Techniques and procedures for assessing cognitive skills. Review of Educational Research, 63, 201-243.

Schvaneveldt, R. W. (Ed.). (1990). Pathfinder associative networks: Studies in knowledge organisation. Norwood: Ablex Publishing Corporation.

Smith, E. L., Blakesee, T. D., & Anderson, C. W. (1993). Teaching strategies associated with conceptual change learning in science. Journal of Research in Science Teaching, 30, 111-127.

SPSS Inc. (1990). SPSS~Base system user's guide. Chicago: Author. Tabachnik, B. G., & Fidell, L. S. (1989). Using multivariate statistics (2nd ed.). New York:

Harper & Row. VanLehn, K. (1989). Problem solving and cognitive skill acquisition. In M. I. Posner (Ed.),

Foundations of cognitive science. Cambridge, MA: The M.I.T. Press. Voss, J. F. (1988). Problem solving and the educational process. In A. M. Lesgold & R. Glaser,

Foundations for a psychology of education. Hillsdale, N J: Erlbaum. Voss, J. F., Wiley, J., & Carretero, M. (1995). Acquiring intellectual skills. Annual Review of

Psychology, 46, 155-181. Walberg, H. J. (1991). Improving school science in advanced and developing countries. Review

of Educational Research, 61, 25-69. Walding, R., Fogliani, C., Over, R., & Bain, J. D. (1994). Gender differences in response to

questions on the Australian National Chemistry Quiz. Journal of Research in Science Teaching, 31, 833-846.

Wallace, J. D., & Mintzes, J. L. (1990). The concept map as a research tool: Exploring conceptual change in biology. Journal of Research in Science Teaching, 27, 1033-1052.

Wilson, J. M. (1994). Network representations of knowledge about chemical equilibrium: Variations with achievement. Journal of Research in Science Teaching, 31, 1133-1147.

concept maps about chemical equilibrium and students' achievement scores

Documents