manova (july 2014 updated)

ANOVA-BASED PROCEDURES July 2014 updated

Prepared by Michael Ling

QUANTITATIVE RESEARCH METHODS

SAMPLE OF

ANOVA-BASED PROCEDURES

Prepared by

Michael Ling



Problem

A consultant that develops training courses for users of a particular software wants to examine how

different characteristics of training courses influence knowledge gained about the software. Specifically,

the consultant is interested in the effects of two variables—(i) mode of delivery (face-to-face training

versus video-based training) and (ii) the provision of summary notes (provided versus not provided) on

knowledge about the software. The consultant recruits 160 individuals to participate in an experiment

designed to examine this issue. First, the consultant administers an intelligence (IQ) test to each

participant as intelligence is a factor that is known to influence how much knowledge individuals gain

from training courses. Each participant is then randomly allocated to one of four conditions (cells) that

are formed by crossing the two levels of the mode of delivery factor (i.e., whether the person gets face-

to-face training or video training; labelled “face-to-face” versus “video”) with the two levels of the

provision of summary notes factor (i.e., whether the person is provided with summary notes or not;

labelled “notes” versus “no notes”). The four conditions differ in the following way:

1. In the face-to-face/notes condition, participants spend three hours with an instructor who trains

them on how to use the software and then they receive notes that summarise the main points.

2. In the video/notes condition, participants spend three hours watching a video that trains them

how to use the software and then they receive notes that summarise the main points.

3. In the face-to-face/ no notes condition, participants spend three hours with an instructor who

trains them how to use the software however they do not receive any notes.

4. In the video/ no notes condition, participants spend three hours watching a video that trains

them how to use the software however they do not receive any notes.

The day following training each participant completes a knowledge test that assesses their

knowledge of how to use the software.

Dataset

The data from the experiment has been entered into the SPSS file SoftTrain.sav. The file contains the

following variables:

id = A code used to identify each participant

knowledge = The participants score on the knowledge test completed after training (out of 50).

mode_of_delivery = The mode of delivery condition that the participant was in; video = 0, face-to-

face = 1

provision_of_notes = Whether the participant received summary notes; no notes = 0, notes = 1

IQ = The participants score on the intelligence test completed prior to training (out of 1000).



Instructions:

The researcher is interested in examining the main effects and interaction effect of mode of delivery and

provision of summary notes on knowledge (as measured by the score on the knowledge test). Open the

file and conduct the appropriate data analysis. Once you have analysed the data, write a report (no

longer than 2-3 double-spaced pages of text with 12pt font) in which you:

1. Describe how you analysed the data

2. Report the results of your data analysis, including an interpretation of what it means.

Solution

A 2x2 ANOVA/ANCOVA procedure was selected for the analysis because the

experiment consisted of (i) two independent factors (‘mode of delivery’ and ‘provision of notes’)

each of which had 2 levels and (ii) a single dependent variable (knowledge). In the early stage of

our analysis, testing was carried out to find out whether the intelligence (IQ) factor could be

considered as a covariate.

Testing IQ as a covariate

Treating the intelligence factor (IQ) as if it were a covariate, we used the tests of

Between-subjects Effects to study any changes in F ratios and error variance (Table 1, 2). We

found the total error variance was reduced to 5899.813 from 6081.175, which indicated that IQ

was, to a certain extent, related to the dependent variable. However, the F ratios were both

reduced for provision of notes (from 22.547 to 20.937) and mode of delivery (from 11.259 to

10.391), which was often an indication that the covariate (IQ) was not only correlated with the

dependent variable, but also with the between-groups factors. The covariate (IQ) not only

partitioned variance away from the error variance, but also from the variance due to the between-

groups factor. By removing the effect of IQ from the analysis, we had removed the true effects of



the between-group factors on the dependent variable. The effect of IQ was not limited to the

dependent variable alone.

In addition, the relationship between the covariate and the dependent variable was tested

and found to be nonlinear, as shown by the scatterplots (Figure 1, 2). The violation of this

assumption of ANCOVA reduced the power of the ANCOVA to find significant differences. As

a result, the use of IQ as a covariate had been ruled out from subsequent analysis.

Testing the basic assumptions of ANOVA

Data were analyzed to determine whether the basic assumptions of ANOVA procedure

were satisfied. The descriptive statistics were as shown in Table 3. The independence

assumption was satisfied as the participants were randomly allocated to one of the four groups.

The scale of measurement assumption was satisfied as the dependent variable (knowledge) was

an interval score from 0 to 50. As Levene’s Test for equality of error variance was non-

significant (Sig > 0.05), thus the assumption of homogeneity of variance had not been violated

(Table 4).

The skewness and kurtosis of the two groups in the mode of delivery factor were shown

in Table 5. Likewise, the skewness and kurtosis of the two groups in the provision of notes

factor were shown in Table 6. Their values lied between -1 and +1 which indicated that each of

the four groups was approximately normally distributed. This was also supported by the non-

significance of the Shapiro-Wilk statistics in the two groups of mode of delivery (Table 7) and

the two groups of provision of notes (Table 8). The box plots showed that there were no

outliners in the four groups (Figure 3, 4). Normality of the sampling distribution was attained.



Main and interaction effects

To investigate the main and interaction effects, we used ANOVA to examine four

sources of variability: (i) variability due to main effect of mode of delivery; (ii) variability due to

main effect of provision of notes; (iii) variability due to interaction effect of mode of delivery

and provision of notes; and (iv) within group (or error) variability.

As shown in Table 3, participants provided with notes (n=80, M = 29.3375, SD =

6.31844) achieved significantly higher scores than participants provided with no notes (n=80, M

= 24.6500, SD = 6.67908). The main effect of provision of notes factor was statistically

significant, F(1, 156) = 22.547, p < .005 (Table 1). Partial eta-square of the provision of notes

factor showed that it accounted for 12.6% of the overall variability (Table 1), which was a

‘medium’ effect.

As shown in Table 3, participants in face-to-face delivery mode (n=80, M = 28.6500, SD

= 6.84937) achieved significantly higher scores than participants in video delivery mode (n=80,

M = 25.3375, SD = 6.57179). The main effect of mode of delivery factor was also statistically

significant, F(1, 156) = 11.529, p < .05 (Table 1). Partial eta-square of the delivery factor

showed that it accounted for 6.7% of the overall variability (Table 1), which was a ‘medium’

effect.

The interaction effect of the mode of delivery and the provision of notes was

statistically significant, F (1, 156) = 4.053, p < 0.05 (Table 1). Partial eta-square of the

interaction effect showed that it accounted for 2.5% of the overall variability (Table 1), which

was a ‘small’ effect.

Nevertheless, with respect to the effect of mode of delivery on knowledge test, the profile

plot showed that it was an ordinal interaction. The mode of delivery had a stronger effect on the

knowledge test of the participants when they were not provided with notes than when they were

provided with notes (Figure 5).



With respect to the effect of provision of notes on knowledge test, the profile plot showed

that it was an ordinal interaction. The provision of notes had a stronger effect on the knowledge

test of the participants when they were trained in video mode than when they were trained in

face-to-face mode (Figure 6).



Appendix

Table 1: Tests of Between-Subjects Effects (with no covariate)

Dependent Variable:knowledge test

Source

Type III Sum of

Squares df Mean Square F Sig.

Partial Eta

Squared

Corrected Model 1475.819a 3 491.940 12.620 .000 .195

Intercept 116586.006 1 116586.006 2990.773 .000 .950

provision_of_notes 878.906 1 878.906 22.547 .000 .126

mode_of_delivery 438.906 1 438.906 11.259 .001 .067

provision_of_notes *

mode_of_delivery

158.006 1 158.006 4.053 .046 .025

Error 6081.175 156 38.982

Total 124143.000 160

Corrected Total 7556.994 159

a. R Squared = .195 (Adjusted R Squared = .180)

Table 2: Tests of Between-Subjects Effects (with IQ as covariate)


Source

Type III Sum of

Squares df Mean Square F Sig.

Partial Eta

Squared

Corrected Model 1657.180a 4 414.295 10.884 .000 .219

Intercept 1897.936 1 1897.936 49.863 .000 .243

IQ 181.362 1 181.362 4.765 .031 .030

provision_of_notes 796.945 1 796.945 20.937 .000 .119

mode_of_delivery 395.530 1 395.530 10.391 .002 .063

provision_of_notes *

mode_of_delivery

203.173 1 203.173 5.338 .022 .033

Error 5899.813 155 38.063

Total 124143.000 160

Corrected Total 7556.994 159

a. R Squared = .219 (Adjusted R Squared = .199)



Table 3: Descriptive Statistics


provision of notes factor mode of delivery factor Mean Std. Deviation N

no notes video 22.0000 5.59762 40

face-to-face 27.3000 6.68024 40

Total 24.6500 6.67908 80

notes video 28.6750 5.77078 40

face-to-face 30.0000 6.83130 40

Total 29.3375 6.31844 80

Total video 25.3375 6.57179 80

face-to-face 28.6500 6.84937 80

Total 26.9938 6.89407 160

Table 4: Levene’s Test of Equality of Error Variancea


F df1 df2 Sig.

.527 3 156 .664

Tests the null hypothesis that the error variance

of the dependent variable is equal across

groups.



Table 5: Descriptives of ‘Mode of Delivery’

mode of delivery factor Statistic Std. Error

knowledge test video Mean 25.3375 .73475

95% Confidence Interval for

Mean

Lower Bound 23.8750

Upper Bound 26.8000

5% Trimmed Mean 25.3611

Median 25.0000

Variance 43.188

Std. Deviation 6.57179

Minimum 9.00

Maximum 42.00

Range 33.00

Interquartile Range 8.75

Skewness .075 .269

Kurtosis .026 .532

face-to-face Mean 28.6500 .76578


Mean

Lower Bound 27.1257

Upper Bound 30.1743


Median 29.0000

Variance 46.914


Minimum 12.00

Maximum 49.00

Range 37.00


Skewness .181 .269

Kurtosis .427 .532



Table 6: Descriptives of ‘Provision of notes’

provision of notes factor Statistic Std. Error

knowledge test no notes Mean 24.6500 .74674


Mean

Lower Bound 23.1636

Upper Bound 26.1364


Median 24.5000

Variance 44.610


Minimum 9.00

Maximum 46.00

Range 37.00


Skewness .410 .269

Kurtosis .762 .532

notes Mean 29.3375 .70642


Mean

Lower Bound 27.9314

Upper Bound 30.7436


Median 29.0000

Variance 39.923


Minimum 12.00

Maximum 49.00

Range 37.00


Skewness .040 .269

Kurtosis .554 .532



Table 7: Tests of Normality for ‘mode of delivery’

mode of delivery factor

Kolmogorov-Smirnova Shapiro-Wilk

Statistic df Sig. Statistic df Sig.

knowledge test video .100 80 .046 .990 80 .774

face-to-face .059 80 .200* .988 80 .684

a. Lilliefors Significance Correction

*. This is a lower bound of the true significance.

Table 8: Tests of Normality for ‘provision of notes’

provision of notes factor

Kolmogorov-Smirnova Shapiro-Wilk

Statistic df Sig. Statistic df Sig.

knowledge test no notes .092 80 .094 .983 80 .392

notes .066 80 .200* .990 80 .765

a. Lilliefors Significance Correction

*. This is a lower bound of the true significance.



Figure 1: Scatterplot between IQ and Knowledge test; mode of delivery

as independent variable

Figure 2: Scatterplot between IQ and Knowledge test; provision of notes

as independent variable.

.



Figure 3: Effect of mode of delivery on knowledge test

Figure 4: Effect of provision of notes on knowledge test

manova (july 2014 updated)

Education

provision of notes

labelled notes

video training

participant knowledge

covariate iq

videobased training

knowledge test

intelligence iq factor