The effects of progress reports and goal orientation on effort and achievement of students using an online individualized instructional program

Roy B. Clariana, Penn State University & Ray Bernardi, Wyoming Area School District

2 of 26

Overview

This investigation examined the effects of advisement on 8th grade students’ (n = 194) effort and achievement. Advisement consisted of a progress report sent to parents of their child’s work on online computer-delivered lessons.

We asked, “Does advisement interact with achievement goal orientation and gender to affect effort (the number of online lessons completed) and achievement (PSSA performance)?”

3 of 26

Achievement goal orientation

Achievement Goal Theory (Pintrich & Schunk, 1996) proposes two broad classes of achievement goal orientations, performance or mastery (either approach or avoidance).

Learners with a performance goal orientation are ego-centered and need to validate or prove their ability through open performance. They may avoid difficult tasks when failure is possible.

In contrast learners with a mastery goal orientation seek to increase their knowledge and ability -- learning for the sake of learning.

4 of 26

Achievement goal orientation Achievement goal orientation may relate to

personal epistemological belief systems (Schommer-Aikins & Easter, 2006) and thus may be relatively stable.

It may be a state and/or a trait; but the learning context likely affects immediate achievement goal orientation.

Also, achievement goal orientation may shift over time based on experiences in the learning context (i.e., consider the likely effects of NCLB on achievement goal orientation).

5 of 26

Advisement

Advisement is not praise (Henderlong & Lepper, 2002).

Advisement is information that relates to required, actual, or anticipated lesson performance, and according to Smith (1988), addresses questions such as, "How am I doing in this lesson?"

Progress reports are one very common form of advisement (e.g., report cards, or a detailed report from the teacher showing every graded test and assignment within a date range).

Advisement is assumed to support effective instructional decision making, thus it directly relates to learners’ achievement goals.

6 of 26

Advisement in computer-mediated learning contexts Milheim and Azbell (1988) report that advisement in

computer-mediated settings improves posttest scores relative to no-advisement groups; increases the number of students that reach mastery; increases time on task; and thus increases instructional efficiency.

Clariana (1993) considered the effects of advisement on effort and achievement of at-risk 11th grade students working online. Progress reports were given to students at the end of their 1st and 3rd sessions (max. 5 sessions). Analysis of the attendance data showed a significant

difference between the advisement (M = 4.33 out of 5) and no-advisement (M = 3.44 out of 5) groups.

Analysis of the achievement data indicated that the advisement group did a little better (nsd) than the no- advisement group and advisement had a slightly larger effect on females’ mathematics achievement relative to males (p = .08).

Similar gender differences for achievement goal orientation have also been observed (Dunn & Shapiro, 1999).

7 of 26

Voluntary homework

Providing students with structured learning opportunities outside school is a way to extend the school day.

The online lessons used in this investigation run in a browser delivered from a server on the internet. This means that the online lessons are available not only in school but anywhere that students can access the internet.

Which students (if any) will choose to do the online lessons as voluntary ‘homework’?

8 of 26

Participants

8th grade students (total N = 218) in the Wyoming Area School District in northeastern Pennsylvania 17 excluded because they did not participate in

the online lessons. 7 excluded because they did not have 2006 PSSA

data. Thus the final sample used in the data analysis

consisted of n = 194 students. The students attended classes in a facility that

houses all 7th through 12th grade students (facility N = 1,315).

In this facility, 98% of students are Caucasian and 24% are on free or reduced lunch.

9 of 26

Setting and Treatment

Online IPs – CompassLearning lessons delivered over the internet that are aligned to state standards and examination anchors in mathematics and reading, and are unique for each student based on past performance measures.

Participants were randomly assigned to the Advisement or No-Advisement treatment that consisted of a progress report sent to parents in Week Three (in addition, gender and achievement goal orientation were two other independent variables).

10 of 26

Effort and achievement dependent measures

Effort – Number of online lessons completed

Achievement – PSSA Reading and Math scaled scores

Other descriptive data homework completion data Self-report survey measuring achievement

goal orientation and internet access.

11 of 26

Achievement goal orientation survey A ten item self-report survey to measure

achievement goal orientation (from Hannah’s dissertation) even numbered items – performance goal odd numbered items – mastery goal

The self-report survey was completed by 164 students (84 females and 80 males), 30 students (14 females and 16 males) did not take the survey

Breakdown: performance oriented (43%; n = 71 with 33

females and 38 males) mastery oriented (41%; n = 67 with 32 females

and 35 males) neutral achievement goal orientation (15%; n =

26 with 19 females and 7 males).

See the survey in Appendix A

12 of 26

Achievement goal orientation analysis

Principal components analysis with varimax rotation obtained two factors.

The ten items aligned as they should into mastery and performance categories.

Cronbach alpha for the 10-item achievement goal orientation: All 10, α = 0.866 Odd numbered (m), α =

0.789 Even numbered (p), α =

0.837coefficients sorted by size

.485.320Item14

.489.305Item10

.663.442Item18

.871.178Item16

.901.066Item12

.197.665Item19

.235.705Item11

.223.779Item13

.260.795Item17

.184.801Item15

21

Component

.485.320Item14

.489.305Item10

.663.442Item18

.871.178Item16

.901.066Item12

.197.665Item19

.235.705Item11

.223.779Item13

.260.795Item17

.184.801Item15

21

Component

mastery

performance

13 of 26

PSSA data availableJan 15 Feb 15 Mar 15

August

Mail online IP progress report letter to parents of randomly selected group

Students work on online IPs

Jan 27

Print all finalprogress reports

Mar 20

Student and teacher interviewsStudent self-report survey

Week of Mar 13

Jan 9 Mar 17Mar 21 Mar 31

Letter to 8th grade parents

Week of Dec 19

Timeline of milestones…

PSSA makeup

Changed to the Week of Mar 31

14 of 26

Achievement: PSSA results

Reading Mathematics

Adv

ance

d

Pro

fici

ent

Bas

ic

Bel

ow B

asic

Adv

ance

d

Pro

fici

ent

Bas

ic

Bel

ow B

asic

no report - mastery 17 8 1 2 15 9 1 3 - neutral 10 3 0 0 7 2 3 1 - performance 29 11 2 0 27 10 5 0 - unknown * 9 6 2 1 6 8 3 1 report - mastery 25 13 1 0 19 14 5 1 - neutral 10 3 0 0 4 8 1 0 - performance 19 7 2 1 17 8 4 0 - unknown * 8 3 0 1 7 4 1 0 (N = 194)

Cut score >1472 >1279 >1145 ≤1145 >1445 >1283 >1170 ≤1170 Total (%) 65% 28% 4% 3% 53% 32% 12% 3%

* unknown, are those students who did not complete the self-report survey

93% reading 85% math

15 of 26

PSSA Analysis

The 2006 reading and mathematics PSSA scaled score data were analyzed by a 3-between x 2-within mixed ANOVA.

The between-subjects factors were Treatment (report or no report), Gender (female or male), and Achievement Goal (mastery, neutral, performance, and unknown).

The within-subjects repeated measure was Content (reading and mathematics).

16 of 26

Mixed ANOVA

Source SS df MS F Sig. Intercept 660169193.941 1 660169193.941 7875.147 0.000 Treatment (T) 21913.967 1 21913.967 0.261 0.610 Gender (G) 42090.836 1 42090.836 0.502 0.480 Achievement Goals (A) 70509.520 3 23503.173 0.280 0.840 T * G 72418.436 1 72418.436 0.864 0.354 T * A 245969.187 3 81989.729 0.978 0.404 G * A 133613.943 3 44537.981 0.531 0.661 T * G * A 177922.931 3 59307.644 0.707 0.549 Error 14921641.530 178 83829.447 PSSA Reading and Math within subjects repeated measure Content (C) 547982.057 1 547982.057 32.686 0.000 C * T 657.237 1 657.237 0.039 0.843 C * G 137810.678 1 137810.678 8.220 0.005 C * A 6710.917 3 2236.972 0.133 0.940 C * T * G 81137.649 1 81137.649 4.840 0.029 C * T * A 23576.936 3 7858.979 0.469 0.704 C * G * A 30043.814 3 10014.605 0.597 0.618 C * T * G * A 47549.357 3 15849.786 0.945 0.420 Error(Content) 2984210.930 178 16765.230

17 of 26

PSSA: Content x Gender interaction

1400

1450

1500

1550

1600

1650

math reading

female

male

PSSA

Sca

led

Sco

re

1400

1450

1500

1550

1600

1650

math reading

female

male

PSSA

Sca

led

Sco

re

reading

math

18 of 26

PSSA: Content x Treatment x Gender interaction

1,520

1,540

1,560

1,580

1,600

1,620

Female PSSA Reading

Male PSSA Reading

no report report

treatment

1,440

1,460

1,480

1,500PSSA

Sca

led

Sco

res

Female PSSA Mathematics

Male PSSA Mathematics

1,520

1,540

1,560

1,580

1,600

1,620

Female PSSA Reading

Male PSSA Reading

no report report

treatment

1,440

1,460

1,480

1,500PSSA

Sca

led

Sco

res

Female PSSA Mathematics

Male PSSA Mathematics

19 of 26

Effort: Analysis of the number of lessons completed The CompassLearning online IP total lessons

completed data (combined reading and mathematics) were analyzed by a three factor ANOVA that included: Treatment (report or no report), Gender (female or male), and Achievement Goal (mastery, neutral, performance, and unknown).

Results: None of the main factors or interactions were significant except for the 3-way interaction of Treatment, Gender, and Achievement Goal (T * G * A), F(3,178) = 2.680, MSE = 394.182, p = .048

20 of 26

Lessons completed: Treatment x Gender x Achievement Goal interaction

Male, Mastery orientation

Male, Performance orientation

no report report

30

35

40

45

50

55

IEP

Les

sons

Com

plet

ed

Female, Mastery orientation

Female, Performance orientation

Treatment

(possible ceiling effect)

note

21 of 26

Did they do Homework?

The online lessons were not checked by teachers, and no daily grade was given for completion, so there was little or no pressure to do the online lessons outside school.

5.7% of the students worked on the online IPs outside of school time. Most were females (9 females and 2 males).

ANOVA indicates that there was apparently an interaction (see next slide).

22 of 26

Possible interaction of Treatment x Gender for minutes of homework completed

no report report

treatment

0

5

10

15female

male

Hom

ewor

k m

inut

es

Caution, small sample n=11

23 of 26

Findings

In general terms, sending advisement home to parents … relates to increased achievement for males in

reading and math, for females in math, but a decrease (sig.) for females in PSSA reading scores.

did not positively influence females to do more online IP lessons. In fact, the presence of reports seemed to depress females’ online IP lesson completion (especially those with a performance achievement goal orientation).

This is the opposite of the trend observed by Clariana (1993) that females may benefit more than males from progress reports.

24 of 26

Regarding praise

Deci (1975) proposed that males tend to be more sensitive to the informational aspect of praise and so perceive praise as an informational message that they are competent in the relevant task. This tends to enhance their intrinsic motivation for the task.

In stark contrast, he proposed that females tend to be more interpersonally sensitive and concerned with others’ evaluations of them and so perceive praise as a controlling intervention that diminishes their autonomy, thus decreasing their intrinsic motivation for the task.

Also Katz, Assor, Kanat-Maymon, and Bereby-Meyer (2006) extended Deci’s hypothesis to include the effects of gender differences in reaction to advisement.

25 of 26

Possible explanations…

Katz et al. (2006) In 7th grade students with a moderate task interest, advisement had a positive effect on males’ and a negative effect on females’ self-report of intrinsic motivation.

Katz’s et al. findings are in line with the results of the present investigation for females with a performance orientation (see Figure on slide 20).

Similar to Deci’s hypothesis that females tend to be more interpersonally sensitive and concerned with others’ evaluations of them, performance-oriented females in the present investigation perhaps resented advisement as an attempt to control or manipulate their actions (which, of course, was exactly the case).

26 of 26

What would you do next time?

If you are the principal?

If you are the teacher?

If you are the investigator?

What effect did the age of the students have on the study?

27 of 26

Appendix A: Achievement Goal Orientation self-report items

28 of 26

Appendix B: Correlation table

PS

SA

Rea

ding

PS

SA

Mat

h

IP L

esso

ns c

ompl

eted

IP L

esso

n sc

ore

IP H

omew

ork

min

utes

Item

10 (

P)

Item

12 (

P)

Item

14 (

P)

Item

16 (

P)

Item

18 (

P)

Item

11 (

M)

Item

13 (

M)

Item

15 (

M)

Item

17 (

M)

Item

19 (

M)

PSSA Reading 1 0.66 0.22 0.56 0.20 0.35 0.15 0.17

PSSA Math 0.66 1 0.26 0.51 0.28 0.21 0.15 0.19 0.18

IP Lessons completed 0.22 0.26 1 0.18 0.23

IP Lesson score 0.56 0.51 0.18 1 0.33 0.21 0.16 0.24 0.16 0.19 0.18 0.27 0.20

IP Homework minutes 0.20 0.23 1

Item10 (P) 0.35 0.28 0.33 1 0.33 0.32 0.34 0.37 0.35 0.27 0.29 0.30 0.26

Item12 (P) 0.21 0.21 0.33 1 0.33 0.77 0.56 0.25 0.33 0.25 0.29 0.25

Item14 (P) 0.16 0.32 0.33 1 0.35 0.38 0.23 0.32 0.34 0.27 0.36

Item16 (P) 0.15 0.24 0.34 0.77 0.35 1 0.57 0.36 0.35 0.35 0.43 0.23

Item18 (P) 0.37 0.56 0.38 0.57 1 0.45 0.43 0.42 0.53 0.44

Item11 (M) 0.16 0.35 0.25 0.23 0.36 0.45 1 0.46 0.54 0.61 0.33

Item13 (M) 0.15 0.19 0.19 0.27 0.33 0.32 0.35 0.43 0.46 1 0.61 0.62 0.50

Item15 (M) 0.18 0.29 0.25 0.34 0.35 0.42 0.54 0.61 1 0.60 0.42

Item17 (M) 0.17 0.18 0.27 0.30 0.29 0.27 0.43 0.53 0.61 0.62 0.60 1 0.44

Item19 (M) 0.20 0.26 0.25 0.36 0.23 0.44 0.33 0.50 0.42 0.44 1