Sim, no sim; sim, sim plus

(1) Simimlation vs no simulation and (2) simulation vs simulation plus

Two research questions:

Forty-eight articles were coded.Random control experiments and quasi-experimental studies were included. In addition, it was necessary for the articles to measure achievement gains and compare means under one of two possible control conditions:Simulation vs traditional instructional treatmentSimulation vs simulation plus modification

The coding of articles

Rejected articles

Liu (2006) Used a counterbalanced repeated measures design and did not report results after each intervention, therefore, there is no clear sim vs. no sim that can be included in the meta-analyisSon, Robert, Goldstone (2009) The experimental and control groups receive the same sim- what is being tested is the language used in the sim (content vs intuitive descriptors)Sheehy, Wylie & Orchard (2000) No control group-pre-experimental designLambAnnetta (2010)Sierra-Fernandez & Perales-Palacios (2003) no achievement gains were measured quantitativelyJimoyiannis & Komis (2003) Did not compare group means

|QUESTION 1: Sim vs. no sim|

How do simulation instructional treatments compare to non-simulation instructional treatments?Fixed effect size = .59 (truncated), df= k-1= (26), z-value 12.43, p = .00 (results are significant)q-value = 311.29, df = k-1 = (26), p = .00 (sum of squares of within study variance)i-squared = 91.65 (measure of heterogeneity of studies)*Using a random effect size is necessary because the percentage of between studies heterogenity is quite large: 92%; for example, in terms of technology choices, design and sample size.

Two kinds of variance

Within-study variance derives mainly from the sample size of the study as reflected in the standard deviation (SD). The within-study variance (V) is the SD squared.Large studies will have smaller within-study variances and small studies will have larger within-study variances.Between-study variance can only be calculated when the studies are synthesized. It is expressed as a sum of squares (SS of within study variance). When the V for each study is summed, the result is Q.

Slide authoredby Bob

Fixed effect model and random effects model

Fixed effect model assumes that there is one fixed average effect size in the population and that between-study variance is sampling error. A Q-value that exceeds chance (using the Chi-squared distribution with df = k -1) indicates that the distribution is heterogeneous.Random effects model does not assume homogeneity and applies when effect sizes are assumed to estimate different populations that may vary one from another.Between-study variance represents differences among populations, not between-study variation within populations.

Slide authoredby Bob

Question 1: Forest plot & effect sizes of individual studies

Large studies with negative results are skewing the fixed effect model.

Question 1: Funnel plot of standard error by Hedge's g

The funnel plot looks at publication bias. The scale on the bottom demonstrates the distribution of effect sizes and the scale on the side shows standard error (sample size). When you read this chart, you want to determine if the distribution is biased across effect size and sample size.In this instance, while there are too many positive studies, the balance for sample sizes is reasonable.

Tau -squared value

0.69 Tau squared value

Truncated valuesWe truncated three g-values to 2.5 because they were too high (one was 2.7, one was over 3, and one was over 4)fixed: .591 (.65 original)random: .859 (.99 original)

Random effect size:

Random effect size = .89, k = df-1 = (26), z-value 5.19, p = .00 (results are significant)*The unweighted average effect size is close to the random effect size (approx. 1.02)Note:Fixed effect size = .59, k= df-1= (26), z-value 12.43, p = .00 (results are significant)

Moderator variables:Grade level and research design

Moderating grade level

K-5, 2: 6-8, 3: 9-12, 4: (multiple ranges) * A mixed effects analysis q-value= 7.469, df =(2), p-value = 0.024 demonstrates that grade level is a significant moderatorFrom the data, one can conclude that simulations work better for younger learners than for older learners **

*The "4" multiple range was removed because there was only one study or effect and it wouldn't mean anything*NOTE: the younger learner category has only 2 main effects compared to 6 and 18 effect sizes for the other 2 ranges.

Moderating research design

RCT or Quasi-Experimental A mixed effects analysis, revealed a Q-between value = 0.01, df=1, p=.91 (results not significant)Research design was not a significant moderatorThese data are collapsed

Results and implications:

The meta-analysis of research studies comparing simulation instructional treatments to non-simulation instructional treatments rejects the null hypothesis that there is no difference in knowledge acquisition between the two treatments. The students in the bulk of the studies who received simulation training received significantly higher scores when tested for learning or mastery of the material.

Instructors who provide training through simulation will most likely improve learning outcomes over those who follow traditional instructional methods. Furthermore, grade level was found to be a moderating variable, however, the sample size in the elementary age group makes reaching any definitive conclusion unwarranted. Research design was found not to be a significant moderating variable.

QUESTION 2: Sim+modification vs. sim

How do simulation alone compare with simulations with instructional enhancements.Fixed effect size = .766, df= k-1= (16), z-value 14.63, p = .00 (results are significant)q-value (sum of squares of within study variance) = 380.87, df= k-1= (16), p =.00 (sum of squares of variance)i-squared = 95.80 (measure of heterogeneity of studies)**Using a random effect size is necessary because the percentage of between studies heterogenity is quite large: 96%; for example, in terms of technology choices, design and sample size.

Question 2: Forest plot and effect sizes of individual studies

Large studies are not skewing the results since they fall into the middle range of effect sizes.

Question 2: Funnel plot of standard error by Hedge's g

The funnel plot looks at publication bias. The scale on the bottom demonstrates the distribution of effect sizes and the scale on the side shows standard error (sample size). When you read this chart, you want to determine if the distribution is biased across effect size and sample size.In this instance, there is a balanced distribution of effect size and sample sizes.

Tau -squared value

1.09 Tau squared value

No truncationSome effect sizes were found with large weights but they were in the middle range.

Random effect size

Random effect size = 0.638, df = k-1 = (16), z-value 5.19, p =.00 (results are significant)Fixed effect size = .766, df= k-1= (16), z-value 14.63, p =.00 (results are significant) The fixed and random effect sizes are closer together than for Q1, with no large effect sizes skewing the fixed effect sizeSome effect sizes were found with large weights, but they were in the middle range of the forest plot and, therefore, did not dramatically skew the fixed effect size.

Moderator variables:

Grade level and research design

Moderating grade level

K-5, 2: 6-8, 3: 9-12, (multiple ranges) A mixed effects analysis reveals a q-value = 5.13, df=2, p=.08This results demonstrates that grade level is not a significant moderatorFrom the data, simulations plus modification does not work better for younger learners These data are collapsed

Moderating research design

RCT or Quasi-Experimental A mixed effects analysis reveals a Q-between value = 0.02, df=1, p=0.87Research design was not a significant moderatorThese data are collapsed

Results and implications:

Our meta-analysis of research studies comparing modified simulation, simulation combined with additional instructional treatment, to simulation only instruction rejected the null hypothesis that there is no difference in knowledge acquisition between the two treatment methods. The students in the bulk of the studies who received simulation plus supplemental training groups received significantly higher scores when tested for learning or mastery of the material over the simulation only groups.

Instructors who provide training through simulation and with additional modifications will most likely improve learning outcomes over those who follow simulation only instructional methods. Furthermore, no moderating variables were identified in terms of grade level or research design.

A extremidade, the end, FIN