successmaker reading and mathematics 2014-15 efficacy...

SM Field Test Report Gatti Evaluation, Inc. 4-1-16

SuccessMaker Reading and Mathematics 2014-15 Efficacy Study

Final Report

This report and its contents, including all graphics, are proprietary information belonging to Gatti Evaluation Inc.

No part of this report may be reproduced or used for any purpose without permission from Gatti Evaluation Inc.


Principal Investigator

Guido G. Gatti

Gatti Evaluation Inc.

162 Fairfax Rd.

Pittsburgh, PA 15221

[email protected]

Primary Stakeholder

Funded By Pearson

For Information Please Contact:

Ann Vilcheck

Project Manager, Academic & Product Research

[email protected]


TABLE OF CONTENTS

EXECUTIVE SUMMARY __________________________________________ i

I. INTRODUCTION _________________________________________________ 1

Literature Review 1

Study Goals and Research Questions 1

II. METHODOLOGY ________________________________________________ 2-6

Student Outcome Measures 2

Teacher Measures 3

Site Recruitment and Selection 3

SuccessMaker Teacher Training 4

Participants 5

Data Analysis Procedures 6

III. RESULTS _______________________________________________________ 7-15

SuccessMaker Implementation 7

Student Achievement 8

Student Achievement for Subgroups 13

IV. DISCUSSION ____________________________________________________ 16

A.1 Study Site Descriptions 17-21

A.2 aimswebPlus TEL and TEN Descriptions 22-23

References 24

Table 1: SuccessMaker Training by School 4

Table 2: School Demographic Information 5

Table 3: Average SuccessMaker Usage Statistics 7

Table 4: Kindergarten Reading aimswebPlus TEL Results 8

Table 5: Kindergarten Math aimswebPlus TEN Results 10

Table 6: First Grade Reading aimswebPlus TEL Results 11

Table 7: First Grade Math aimswebPlus TEN Results 12

Table 8: Kindergarten Reading aimswebPlus TEL 13

Results for Free/Reduced Lunch Students

Table 19: Kindergarten Reading aimswebPlus TEL Results for ELL Students 13

Table 10: Kindergarten Math aimswebPlus TEN 14


Table 11: Kindergarten Math aimswebPlus TEN Results for ELL Students 14

Table 12: First Grade Reading aimswebPlus TEL 14


Table 13: First Grade Reading aimswebPlus TEL Results for ELL Students 15

Table 14: First Grade Math aimswebPlus TEN 15


Table 15: First Grade Math aimswebPlus TEN Results for ELL Students 15


Figure 1: Kindergarten aimswebPlus TEL Results Graphs 9

Figure 2: Kindergarten aimswebPlus TEN Results Graphs 10

Figure 3: First Grade aimswebPlus TEL Results Graphs 11

Figure 4: First Grade aimswebPlus TEN Results Graphs 12


i

EXECUTIVE SUMMARY

Gatti Evaluation conducted a study during the 2014-15 school year to examine whether students

using the SuccessMaker reading and mathematics program for the first time would experience

significant growth in achievement. The findings were analyzed for the entire sample of students,

and then disaggregated for English Language Learners (ELL) and students eligible for

Free/Reduced Priced Lunch. The researchers also documented program usage and implementation

to explore trends in student performance as related to SuccessMaker usage.

The final analytic sample was comprised of 719 students from six schools across four districts and

four states. More than half of the students identified as Hispanic and were eligible for free or

reduced priced lunch. Also, half of the study sample were designated English Language Learners.

All of these students made regular use of the SuccessMaker program 60 minutes a week for up to

6 months (i.e., December through April into early May). The average user logged 15 hours on the

program and gained 6.5 months in course level.

The results of the study indicate that SuccessMaker students across both grade levels and each

outcome measure experienced very large and statistically significant gains in reading and

mathematics as evidenced by performance on the aimswebPlus assessment. The achievement

gains for the kindergarten students ranged from 0.74 to 1.34 standard deviations for reading scales

and from 0.91 to 1.31 standard deviations for math scales. The achievement gains for first grade

students ranged from 0.70 to 1.11 standard deviations for reading scales and from 0.44 to 1.27

standard deviations for math scales.

Students who were classified as ELL or eligible for Free/Reduced Priced Lunch also experienced

very large and statistically significant gains for both grades on all the aimswebPlus reading and

mathematics scales (i.e., 0.57 to 1.68 standard deviations).


1

I. INTRODUCTION

Literature Review

Many districts are faced with pressure to improve student achievement for an increasingly diverse

group of learners. Districts are exploring the use instructional technology to support their teachers

and students based on research that indicates it is effective. For example, Tamin, et al (2011)

conducted a second-order meta-analysis of studies of a variety of types of learning technologies,

including 25 meta-analyses that encompassed 1,055 primary studies, and found a strong overall

positive effect (ES = 0.33) for classrooms that used technology compared to their face-to-face

counterparts.

As a result, educators are leaning more heavily on instructional technology in the classroom to

help provide assistance to their students. SuccessMaker is an adaptive, computer-based learning

program that offers an instructional management system, ongoing assessment, individualized

curriculum resources, and a reporting system to inform administrators and teachers as to student

progress.

This study sought to examine the performance of students using the SuccessMaker program, for

both reading and math.

Study Goals and Research Questions

Traditionally, the first year of implementation of any new program is the most challenging and is

often associated with a decrease in student achievement known as the “implementation dip”.

Michael Fullan (2001) defines the implementation dip as “literally a dip in performance and

confidence as one encounters an innovation that requires new skills and new understandings.”

The primary goal of this study was to examine whether students using the SuccessMaker reading

and mathematics programs achieved significant gains over the course of half a school year in

reading and mathematics achievement. The students in the study were using SuccessMaker for

the first time.

A secondary goal of the study was to examine the implementation fidelity of SuccessMaker usage.

Yap et al. (2000) have argued that, “no program – no matter how sound it is – can have an impact

if its essential elements are not used.” Although SuccessMaker is a computer-based program, there

are best practices for both the teacher and student that have been associated with higher student

performance. This study monitored and documented student program usage and implementation

to examine the growth patterns of student achievement with the associated implementation fidelity.


2

II. METHODOLOGY

The SuccessMaker program was evaluated using a one-group pretest-posttest design. Students in

kindergarten and first grade received SuccessMaker reading and mathematics as a supplement to

their core instruction. Classroom teachers were instructed to continue using the district-provided

curricula for core and intervention instruction (if assigned) and to supplement with

SuccessMaker for 60 minutes per week.

Gatti Evaluation provided participating schools all data collection materials, maintained

communication with study schools, and followed data collection procedures that were pre-

established with the schools throughout the study period. A complete description of each study

school is included in Appendix 1.

The following sections provide information on study procedures, including; student and teacher

level data collection, site recruitment and selection, the nature of core and intervention instruction

at each school, program training and implementation, demographic information for study

participants, and the statistical methodologies used to analyze outcomes.

Student Outcome Measures

The aimswebPlus assessment battery was used to measure student learning during the study period.

aimswebPlus is a standardized, norm-referenced assessment that can be used to screen students

and monitor progress, identify students at risk of failing, and continuously monitor struggling

students. The aimswebPlus is comprised of developmentally appropriate scales designed to

measure several English language and mathematics skills. See Appendix 2 for a list and

description of these scales along with their recommended administration periods.

For developmental reasons, all scales are not available during each benchmark session (i.e., Fall,

Winter, Spring). For example, Oral Reading Fluency is not administered until the Fall of 1st grade.

Other scales may be discontinued during kindergarten (ex., Print Concepts, Initial Sounds) or first

grade (ex., Phoneme Segmentation).

It should be noted that since baseline testing using the Fall scales was administered from the third

week of November into December, only newly introduced Winter measures were administered in

January for the middle-of-school-year testing. For example, Initial Sounds was administered to

the kindergarten students in the Fall but not administered again for the Winter testing period, so

soon after the baseline administration. Phoneme Segmentation, however, was administered to the

Kindergarten students in the Winter session and again for the Fall testing period in May. The

assessments were individually administered. Students viewed tasks presented from a stimulus

book while the administrator scored students’ performance during testing in a record book.


3

aimswebPlus Test of Early Literacy (TEL) and Test of Early Numeracy (TEN)

The National Reading Panel (Panel 2000) identified the following to be the best predictors of how

well children will learn to read in kindergarten and first grade: (1) phonemic awareness, and (2)

elements of phonics including letter names, letter sounds, and the ability to read nonsense words.

The TEL measures each of these constructs, with the addition of Oral fluency.

The TEN is based on research examining students’ development of informal mathematical

knowledge. Students are asked to orally count, identify numbers, compare numbers, apply basic

concepts, and perform mental computations. Each task is designed to represent a critical early

numeracy skill for kindergarten and first grade students.

Teacher Measures

Classroom and SuccessMaker Observations Teacher classrooms from each study school were observed in the Spring of 2015. Due to school

schedules and time constraints, not every classroom was observed. The teachers and students were

observed in their classrooms receiving either mathematics or reading instruction and using the

SuccessMaker program.

Observers used a classroom observation rubric to record information on each classroom lesson

observed, along with a debriefing interview. This rubric included: a description of the physical

classroom environment (i.e., available technology, desks/tables, instructional materials), summary

of the lesson taught, student comments, pacing, use of differentiated instruction, classroom control,

student engagement, and basic interview questions.

The observations also allowed researchers to observe SuccessMaker students using the program,

verify the ability and willingness of teachers and students to properly implement the program. A

second rubric was used to record information from these observations. This rubric included a

description of the adequacy of the computer lab and rated the engagement of the students, as well

as, the implementation from the teachers.

It should be noted that some teachers had to adjust their teaching schedule to accommodate the

observers’ visits. The observations provided only a snapshot of day-to-day instruction and

program implementation. Despite this, the observations are still considered worthwhile because

they are the only opportunity the research team has to directly observe the study’s students and

teachers.

Site Recruitment and Selection

Prior to the 2014-15 school year, potential research schools were identified by Pearson sales

representatives and via email blasts sent to districts that had specific demographics. Ensuring

ethnic and socio-economic diversity in the study sample were two important criteria that the

research team considered when recruiting study sites.


4

Schools that indicated interest were sent a study description that included responsibilities and

incentives. If the school indicated interest after reviewing the study description, they were asked

to complete a detailed questionnaire with follow-up phone interviews. The intent of the

questionnaire and interview process was for Gatti Evaluation to gather information to determine if

each school would be an appropriate candidate for the study.

When sites were deemed eligible for participation and approved by the principal investigator, the

school was invited to be a study participant. The principal investigator completed any research

applications required by each school. Both a district level administrator (i.e., curriculum director,

superintendent) and a school level administrator (i.e., principal) signed a memorandum of

understanding outlining the responsibilities of each stakeholder. All students were eligible to

participate in the study unless their parent or guardian elected for them to not participate.

Ultimately, six schools from four districts in four states (CA, CO, OH, and OR) participated in the

study. Appendix 1 provides details about the community, educational environment and

demographic breakdown for each study site. This information is important for understanding how

results from this study may be applicable to consumers of this report.

SuccessMaker Teacher Training

To initiate the study, Gatti Evaluation conducted study orientations for all teachers. The study

orientation introduced the teachers to the research team, explained the requirements and benefits

of participation in the study, and addressed questions or concerns about the study. All teachers

were required to read and sign informed consent forms. Pearson ensured that schools had full

access to the SuccessMaker program and provided training and funding to cover the costs of

substitute teachers (if needed) for the training sessions.

Two SuccessMaker training sessions were provided to each school (see Table 1 for timeline). The

first training session focused on the fundamentals of using SuccessMaker such as student login,

classroom/lab management, and program reporting systems. In addition, trainers provided

information on best practices associated with SuccessMaker implementation. The second training

session focused on a deeper understanding of reporting capabilities and report data uses, as well

as addressing questions or challenges that arose during the previous months of SuccessMaker

implementation.

Table 1 SuccessMaker Training by School

State School No. of Teachers Initial Training Second Training

CA 1 6 (3 K, 3 1st) Dec 2014 March 2015

CA 2 3 (2 K, 1 1st) Dec 2014 March 2015

CA 3 3 (3 K) Dec 2014 May 2015

CO 1 4 (2 K, 2 1st) Jan 2015 March 2015

OH 1 4 (2 K, 2 1st) Nov 2014 March 2015

OR 1 2 (2 1st) Nov 2014 April 2015


5

Participants

The final analytic sample is comprised of 719 students from six schools across four districts and

four states. The sample included 205 kindergarten reading students and 203 kindergarten math

students. There were 155 first grade reading students and 156 first grade math students. Half of

the student sample (i.e., 51%) was designated English Language Learner or not English proficient

and a majority of the students (1.e., 65%) were eligible for free or reduced priced lunch. Close to

half (i.e., 44%) of the study sample was Hispanic, a majority (i.e., 60%) if the students from the

3rd California school are included. A description of the study sample demographics broken out by

school can be found in Table 2.

Table 2 School Demographic Information

Grade Student

Count

Not

English

Proficient

Free/Reduced

Lunch

Percent

Caucasian

Percent

Hispanic

Percent African

American

Other or

No

Information

Whole Sample

K-Reading

1-Reading

205

155

54%

47%

59%

73%

28%

37%

39%

49%

0%

3%

33%

11%

K-Math

1-Math

203

156

54%

48%

61%

69%

29%

38%

39%

50%

0%

0%

32%

12%

California District-School One

K-Reading

1-Reading

50

67

78%

81%

98%

96%

0%

0%

92%

76%

0%

0%

8%

24%

K-Math

1-Math

50

68

78%

81%

98%

96%

0%

0%

92%

76%

0%

0%

8%

24%

California District-School Two

K-Reading

1-Reading

34

22

76%

82%

9%

86%

6%

0%

88%

100%

0%

0%

6%

0%

K-Math

1-Math

30

22

76%

82%

9%

86%

6%

0%

88%

100%

0%

0%

6%

0%

California District-School Three

K-Reading

K-Math

60

60

77%

77%

100%

100%

0%

0%

0%

0%

0%

0%

100%

100%

Colorado District

K-Reading

1-Reading

41

34

0%

0%

12%

12%

6%

0%

88%

88%

10%

6%

2%

6%

K-Math

1-Math

42

33

0%

0%

12%

13%

6%

0%

88%

88%

10%

6%

2%

6%


6

Ohio District

K-Reading

1-Reading

20

18

0%

0%

20%

67%

100%

94%

0%

0%

0%

6%

0%

0%

K-Math

1-Math

21

17

0%

0%

38%

18%

100%

94%

0%

0%

0%

0%

0%

6%

Oregon District

1-Reading

1-Math

14

16

7%

6%

100%

100%

71%

94%

7%

6%

23%

0%

0%

0%

Data Analysis Procedures

Statistical analyses were performed on the aimswebPlus raw gain scores. Gain scores are

calculated by subtracting the beginning of year raw score from the end of year raw score. The

results were calculated for the entire sample of students, and then disaggregated for English

language learners and students eligible for free or reduced priced lunch. An ordinary least squares

linear fixed-effects intercept-only model (i.e., H0: gain ≠ 0) was employed to statistically test mean

gain scores from zero. Specifically, SPSS’s Generalized Estimating Equations procedure was used

with a robust sandwich estimator for all standard errors with districts set as the independent level

of nesting, and a naïve independent working covariance structure

While students were the unit of analysis, the districts were the independent units. The hierarchical

nature of the data (i.e., students nested within classrooms, classrooms nested within schools,

schools nested within districts) has the effect of reducing the amount of independent information

available in the sample, therefore decreasing the precision of estimates and the power of hypothesis

tests to find these estimates statistically significant (Donnar & Klar, 2000). A naïve covariance

structure within a robust empirical standard error formulation was used to calculate confidence

intervals for estimated effects. This procedure results in estimates that are unbiased and statistical

hypothesis tests that are consistent (Liang & Zeger, 1986) despite the nested nature of the data.

All statistical significance tests are two-tailed (i.e., gain ≠ 0) with a Type I error rate of 0.05. Each

mean gain score has no better than a 1 in 20 chance of being found statistically significant when it

is in fact equal to zero. Statistical significance thus implies that the sample was likely drawn from

a population that saw a true increase in achievement. Coupled with the rigorous study design and

program implementation we may then hold these statistically significant gains as possible evidence

for the effectiveness of the intervention.

Standardized effect size estimates (i.e., effect size = mean gain / gain score standard deviation)

along with a percentile rank based effect size measure were computed for statistically significant

differences (Hedges & Olkin, 1985). The latter effect size measure indicates the percentile rank

for the average student’s gain in relation to the baseline distribution. If the students outperformed

their baseline score by 0.20 standard deviations the average score for the subsequent testing was

larger than 58% of the baseline scores, thus students outperformed their first attempt by 8

percentile points.


7

III. RESULTS

This section first describes the implementation of SuccessMaker followed by the impact of the

SuccessMaker program on student learning as evidenced by gains on the aimswebPlus

assessment battery.

SuccessMaker Implementation

The implementation of SuccessMaker was documented by the research team to provide context to

the student performance data. The research schools were advised to implement the program for

one hour per week, for both reading and math, with their students. In addition, the study teachers

were trained to examine the SuccessMaker Cumulative Performance reports to ensure students

were making progress in the program and trying their best. For example, the SuccessMaker 6

Teacher Reports Guide states that acceptable progress is achieved when students master 90% of

presented skills in mathematics and 75% mastery of skills in reading. If a student is not able to

maintain consistency in their mastery of skills, then the teacher may meet with them to discuss

their challenges with the program and provide additional intervening instruction.

Each school attempted to maintain a consistent implementation schedule for the duration of the

study, although there were some variances due to school events, testing schedules and other normal

school activities. Table 3 provides a summary of SuccessMaker implementation across the school

sites.

Table 3 Average SuccessMaker Usage Statistics

Kindergarten

Reading

Kindergarten

Mathematics

First Grade

Reading

First Grade

Mathematics

Sample Size 205 207 155 156

Hours (Standard Deviation) 16.3 (3.7) 14.6 (3.6) 14.7 (4.8) 14.5 (4.5)

Minimum Hours 8.0 7.1 5.3 6.3

Maximum Hours 26.4 24.7 25.3 28.4

Initial Placement Level 0.12 0.57 1.00 1.30

Final Course Level 0.69 1.30 1.60 2.00

No. of Sessions 58 55 45 47

% Exercises Correct 71.6% 63.3% 80.7% 67.5%

% Skills Mastered 72.1% 94.4% 87.4% 96.0%

The majority of research schools began their SuccessMaker implementation in December, 2014

and the aimswebPlus post-testing occurred in May, 2015. Taking into consideration the holiday

breaks, most schools were able to achieve one hour of implementation per week as evidenced by


8

a mean usage of 15 (first) or 16 (kindergarten) hours for reading and almost 15 hours (both kindergarten

and first) for mathematics.

Students had very high mastery of mathematics skills, averaging 94% mastery of skills

kindergarten and 96% in first grade. Both of these averages exceed the recommendations from

the Teacher Reports Guide. The percentage of skills mastered in reading was 72% at kindergarten

and 87% in first grade. It is not unusual to see lower reading skills mastery in kindergarten as

students develop their literacy skills. At first grade, students were again exceeding the

recommendations from the Teacher Reports Guide.

It is recommended that students consistently complete 60% to 80% exercises correctly as they

move through the program’s content and are constantly challenged with new and more difficult

skills. The study students met this recommendation well. The students in this study used

SuccessMaker for approximately six months of the 2014-15 school year. The average course level

growth of 6.5 months was consistent with this time frame across the grade levels and content areas.

Student Achievement

Tables 4 through 7 present, on each aimswebPlus achievement scale, the number of final

participating study students, mean scores with standard deviations for each assessment period,

mean score gains and their standard deviations, and effect sizes for the gains. The gains

experienced by SuccessMaker students across both grade levels and each outcome measure are

large and statistically significant. The achievement gains for the kindergarten students were very

large, ranging from 0.63 to 0.89 standard deviations for the reading scales and from 0.91 to 1.31

standard deviations on the math scales. The achievement gains for the first grade students were

also very large, ranging from 0.70 to 1.11 standard deviations for reading scales and from 0.44 to

1.27 standard deviations on the math scales.

Table 4 Kindergarten Reading aimswebPlus TEL Results

Scale Sample Size BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size

Initial Sounds 205 9.65 (3.13) NA NA NA NA

Letter Naming Fluency 205 36.64 (20.43) NA 48.62 (19.94) 11.98 (16.13) 0.74 (27)*

Auditory Vocabulary 205 18.93 (5.33) NA 21.72 (3.56) 2.79 (3.51) 0.79 (29)*

Letter Word Sounds 205 NA 29.80 (14.87) 37.69 (14.66) 7.89 (8.87) 0.89 (31)*

Phoneme Segmentation 205 NA 30.32 (17.52) 37.94 (15.30) 7.62 (12.16) 0.63 (23)*

High Frequency Words 205 NA NA 18.27 (16.38) NA NA

Mean gain indicates the gain (posttest minus baseline score) and (SD) indicates the sample standard deviation of gain scores

Gain Effect Size includes a Standardized Mean Gain = Mean Gain / SD and a Percentile Gain (-) indicating the increase in percentile rank for the mean

posttest score when the baseline score is set to median/50th percentile

An * and Green highlight indicates statistically significant gain in scores


9

Kindergarten students achieved statistically significant gains on the reading scales from beginning-

of-year to end-of-year (i.e., Letter Naming Fluency and Auditory Vocabulary) or middle-of-year

to end-of-year (i.e., Phoneme Segmentation and Letter Word Sounds). These gains are depicted

in Figure 1. Students on average were able to correctly say 8 more phonemes and 8 more letter

word sounds from Winter to Spring which equated to a gains of 23 and 31 percentile points

respectively. These students were also able to name 12 more letters and point out 3 new vocabulary

words over the course of 6 months which equated to gains of 27 and 29 percentile points

respectively.

Figure 1 Kindergarten aimswebPlus TEL Results Graphs

N/A

30.32

37.94

0

5

10

15

20

25

30

35

40

BOY MOY EOY

Phoneme Segmentation

36.64

N/A

48.62

0

10

20

30

40

50

60

BOY MOY EOY

Letter Naming Fluency

18.93

N/A

21.72

0

5

10

15

20

25

BOY MOY EOY

Auditory Vocabulary

N/A

29.80

37.69

0

5

10

15

20

25

30

35

40

BOY MOY EOY

Letter Word Sounds

We will next examine the performance of kindergarten students on the aimswebPlus TEN.

Kindergarten students achieved statistically significant gains in mathematics from beginning-of-

year to end-of-year (i.e., Number Identification, Quantity Total, and Concepts and Applications)

and middle-of-year to end-of-year (i.e., Quantity Match). These gains are depicted in Figure 2.

Students on average were able to identify 10 more numbers, 4 more quantities of dots, and increase

5 more correct mental math problems (i.e., of 25 total) over the course of 6 months. These

performances equated to gains of 32, 34 and 41 percentile points respectively.


10

Table 5 Kindergarten Math aimswebPlus TEN Results

Scale Sample

Size

BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size

Number Identification 202 35.38 (15.38) NA 45.86 (12.62) 10.48 (11.48) 0.91 (32)*

Quantity Total 206 12.31 (4.53) NA 16.23 (3.98) 3.92 (3.93) 1.00 (34)*

Concepts and Applications 206 12.87 (5.41) NA 18.04 (4.84) 5.17 (3.94) 1.31 (41)*

Quantity Match 206 NA 7.34 (4.52) 11.42 (5.89) 4.07 (4.04) 1.01 (34)*


Gain Effect Size includes a Standardized Mean Gain = Mean Gain / SD and a Percentile Gain (-) indicating the increase in percentile rank for the mean posttest

score when the baseline score is set to median/50th percentile


Figure 2 Kindergarten aimswebPlus TEN Results Graphs

35.44

N/A

46.24

0

5

10

15

20

25

30

35

40

45

50

BOY MOY EOY

Number Identification

12.35

N/A

16.23

0

2

4

6

8

10

12

14

16

18

BOY MOY EOY

Quantity Total

12.88

N/A

18.04

0

2

4

6

8

10

12

14

16

18

20

BOY MOY EOY

Concepts and Applications

N/A

7.35

11.42

0

2

4

6

8

10

12

BOY MOY EOY

Quantity Match

First grade students achieved statistically significant gains on the reading scales from beginning-

of-year to end-of-year (i.e., Oral Reading, High Frequency Words, and Auditory Vocabulary).

Students on average were able to correctly read 42 more words in 60 seconds, correctly read 15

more high frequency words, and point out 2 new vocabulary words over the course of six months.

This equated to a gain of 37, 35 and 26 percentile points respectively. See Figure 3 for a graphic

representations of the gains experienced by 1st grade students on the aimswebPlus TEL.


11

Table 6 First Grade Reading aimswebPlus TEL Results


Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size

Oral Reading 155 77.03 (63.17) NA 119.10 (80.26) 42.07 (37.92) 1.11 (37)*

Letter Word Sounds 155 45.10 (12.92) NA NA NA NA

Phoneme Segmentation 155 41.37 (10.56) NA NA NA NA

High Frequency Words 155 29.57 (22.32) NA 44.30 (26.12) 14.74 (14.51) 1.02 (35)*






Figure 3 First Grade aimswebPlus TEL Results Graphs

77.03

N/A

119.10

0

20

40

60

80

100

120

140

BOY MOY EOY

Oral Reading

29.57

N/A

44.30

0

5

10

15

20

25

30

35

40

45

50

BOY MOY EOY

High Frequency Words

21.66

N/A

23.21

0

5

10

15

20

25

BOY MOY EOY

Auditory Vocabulary

First grade students also achieved statistically significant gains on the mathematics scales from

beginning-of-year to end-of-year (i.e., Number Comparison, Concepts and Applications, and Math

Facts) and middle-of-year to end-of-year (i.e., Mental Computation). These gains are depicted in

Figure 4. Students on average able to correctly identify the larger 2-digit numeral 4 more times,

complete 2 additional math facts (i.e., addition and subtraction 0-10), complete 4 more


12

computation (i.e., addition and subtraction of 2-digit numbers), and increase 5 more correct mental

math problems (i.e., of 25 total) over the course of 5-6 months. This equated to performance gains

of 25, 17, 31, and 40 percentile points correspondingly.

Table 7 First Grade Math aimswebPlus TEN Results

Scale Sample

Size

BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size

Number Comparison 156 23.24 (8.16) NA 27.17 (7.51) 3.92 (5.85) 0.67 (25)*


Math Facts 156 14.96 (7.22) NA 17.43 (7.50) 2.47 (5.67) 0.44 (17)*

Mental Computation 82 NA 4.60 (4.56) 8.48 (6.32) 3.88 (4.43) 0.87 (31)*





Figure 4 First Grade aimswebPlus TEN Results Graphs

23.24

N/A

27.17

0

5

10

15

20

25

30

BOY MOY EOY

Number Comparison

12.92

N/A

17.43

0

2

4

6

8

10

12

14

16

18

20

BOY MOY EOY

Concepts and Applications

14.96

N/A

17.43

0

2

4

6

8

10

12

14

16

18

20

BOY MOY EOY

Math Facts

N/A

4.29

8.48

0

1

2

3

4

5

6

7

8

9

BOY MOY EOY

Mental Computation


13

Student Achievement for Subgroups

Students designated as English Language Learners (ELL) or eligible for Free or Reduced Priced

Lunch are often considered “at-risk” for being left behind academically. This study further

examined the impact of the SuccessMaker program specifically on these two groups of students.

Tables 8 through 15 provide the aimswebPlus results for these students.

The achievement gains on the aimswebPlus for the kindergarten at-risk students were very large,

ranging from 0.57 to 1.23 standard deviations for the reading scales (i.e., 22 to 39 percentile points)

and from 0.91 to 1.68 standard deviations (i.e., 32 to 45 percentile points) for the math scales.

Likewise, the achievement gains for first grade at-risk students were also very large, ranging from

0.79 to 1.04 standard deviations for the reading scales (i.e., 28 to 35 percentile points) and from

0.57 to 1.46 standard deviations (i.e., 22 to 43 percentile points) for the math scales.

Table 8 Kindergarten Reading aimswebPlus TEL Results for Free/Reduced Lunch Students


Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size











Table 9 Kindergarten Reading aimswebPlus TEL Results for ELL Students


Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size








14





Table 10 Kindergarten Math aimswebPlus TEN Results for Free/Reduced Lunch Students

Scale Sample

Size

BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size




Quantity Match 125 NA 6.38 (4.12) 10.41 (6.17) 4.03 (4.44) .91 (32)*





Table 11 Kindergarten Math aimswebPlus TEN Results for ELL Students

Scale Sample

Size

BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size




Quantity Match 111 NA 6.07 (4.13) 10.19 (5.72) 4.12 (4.03) 1.02 (35)*





Table 12 First Grade Reading aimswebPlus TEL Results for Free/Reduced Lunch Students


Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size

Oral Reading 113 67.17 (53.38) NA 108.32 (76.88) 41.15 (39.60) 1.04 (35)*

Letter Word Sounds 113 43.35 (12.42) NA NA NA NA









15

Table 13 First Grade Reading aimswebPlus TEL Results for ELL Students


Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size

Oral Reading 73 60.05 (54.02) NA 91.45 (72.65) 31.40 (36.59) 0.86 (31)*

Letter Word Sounds 73 41.25 (13.55) NA NA NA N








Table 14 First Grade Math aimswebPlus TEN Results for Free/Reduced Lunch Students

Scale Sample

Size

BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size



Math Facts 107 14.42 (7.38) NA 17.26 (8.01) 2.84 (5.00) 0.57 (21)*






Table 15 First Grade Math aimswebPlus TEN Results for ELL Students

Scale Sample

Size

BOY Mean

Score (SD)

MOY Mean

Score (SD)

EOY Mean

Score (SD)

Mean Gain

(SD)

Gain

Effect Size



Math Facts 74 14.36 (8.22) NA 16.96 (8.51) 2.59 (4.45) 0.58 (22)*







16

IV. DISCUSSION

A large sample of 719 kindergarten and first grade students from 22 classrooms participated in the

2014-15 SuccessMaker Math and Reading Efficacy Study. The student sample was quite diverse,

with more than half of the students identifying as Hispanic and eligible for free or reduced lunch.

Half of the study sample were designated English Language Learners.

The results indicate that students who consistently use the program (i.e., 15 hours over 6 months)

are able to have large achievement gains in reading and mathematics (i.e., ranging from .44 to 1.31

standard deviations). The gains are statistically significant, and are evident across a variety of

reading and mathematics constructs as measured by the aimswebPlus assessment.

An examination of performance for at-risk students, those classified as English Language Learners

or eligible for free or reduced priced lunch, indicates similar significant growth for these subgroups

as well. Once again, the growth was statistically significant and evident across both grades, as

well as, across all reading and mathematics scales.


17

A.1 SuccessMaker Study Site Descriptions

This appendix summarizes the educational environment for each study site as well as a

demographic breakdown. This information is important for determining how applicable results

from this study may be to the consumers of this report.

California District-School One

This school resides in a large suburb and students are expected to follow a strict dress code. This

school promotes student success by ensuring professional collaboration and communication

among staff, goal setting with both parents and students and making a strong effort to academically

use all time allotted for the school-day. In the 2010-11 school year, the district served a community

of over 43,000. The median household income is approximately $42,000, indicating a middle-

class community.

This is a medium size school serving close to 500 students in grades kindergarten through six. The

Kindergarten level is made up of approximately 80 students and first grade is made up of

approximately 70 students. This school is primarily Hispanic, which represents 82% of the

school’s population. Asian/Pacific Islanders make up the remaining 18%. This school falls into

the high range for participation in the nation’s lunch program with 93% of students eligible to

receive free or reduced-price lunch. The student/teacher ratio is approximately 27 to 1.

Three kindergarten and three first grade teachers agreed to be in the study. The district adopted a

widely published elementary basal reading and math curriculum. Daily English Language Arts

blocks for both kindergarten and first grade were anywhere up to 120 minutes in length, while

daily math blocks were anywhere up to 75 minutes in length.

All participating teachers used both the reading and mathematics portions of the program. During

the study period, kindergarten students received an average of 14 hours of SuccessMaker reading

instruction (standard deviation = 1.31 hours) and an average of 14 hours of SuccessMaker math

instruction (standard deviation = 1.40 hours). First grade students received an average of 16 hours

of SuccessMaker reading instruction (standard deviation = 1.44 hours) and an average of 15 hours

of SuccessMaker math instruction (standard deviation = 1.65 hours).

Teachers were trained on December 3rd, approximately three months after school began. They

received an additional training on March 4, 2015. The baseline testing for reading was

administered to kindergarten during the second week of December, 2014 and the math test was

administered to kindergarten the first week of January 2015. The baseline testing for reading and

math was administered to 1st grade during the third week of December, 2014. One first grade

classroom baseline tested for math during the first week of January, 2015. The middle of year

testing was administered to kindergarten during the third week of January, 2015. The middle of

school year testing for math was administered to 1st grade during the third week of January, 2015.

There was no middle of school year reading test administered to 1st grade. End of school year

testing was administered to Kindergarten during the second week of May, while first grade

completed end of school year testing for reading in math during the third week of May.


18

California District-School Two

This school resides in a large suburb. This is an economically disadvantaged site where most

students come from low-income families. In addition, teachers are challenged with large classes

and not enough staff. In the 2010-11 school year, the district served a community of over 29,000.

The median household income is approximately $39,000, indicating a middle-class community.

This is a medium size school serving approximately 550 students in grades kindergarten through

five. The Kindergarten level is made up of approximately 85 students and first grade is made up

of approximately 95 students. This school is primarily Hispanic, which represents 95% of the

school’s population. Asian/Pacific Islanders make up the remaining 5%. This school falls into

the high range for participation in the nation’s lunch program with 96% of students eligible to

receive free or reduced-price lunch. The student/teacher ratio is approximately 27 to 1

Two kindergarten and one first grade teacher agreed to be in the study. This district adopted a

widely published elementary basal reading and math curriculum. Daily English Language Arts

blocks for kindergarten were 80 minutes and for 1st grade were 100 minutes. Daily math blocks

for both kindergarten and first grade were 60 minutes.

All participating teachers used both the reading and mathematics portions of the program. During

the study period, kindergarten students received an average of 19 hours of SuccessMaker reading

instruction (standard deviation = 1.84 hours) and an average of 14 hours of SuccessMaker math

instruction (standard deviation = 1.70 hours). First grade students received an average of 12 hours

of SuccessMaker reading instruction (standard deviation = 2.55 hours) and an average of 11.50

hours of SuccessMaker math instruction (standard deviation = 2.69 hours).


received an additional training on March 2nd, 2014. The baseline testing was administered to

Kindergarten during the second and third weeks of December 2014 and to first grade during the

second week of December 2014. The middle of school year testing was administered to

Kindergarten and first grade during the last week of January 2015. End of school year testing was

administered to both Kindergarten and 1st grade during the last week of May and first week of June

2015.

California District-School Three

This school resides in a large suburb. This site values strong leadership, as evidenced by a high

level of involvement and support by the principal. Students at this school come from lower socio-

economic backgrounds and tend to struggle academically. In the 2010-11 school year, the district

served a community of over 43,000. The median household income is approximately $42,000,

indicating a middle-class community.

This is a medium size school serving just fewer than 600 students in grades kindergarten through

six. Kindergarten and first grade are both made up of approximately 80 students. This school is

primarily Hispanic, which represents 86% of the school’s population. Asian/Pacific Islanders

make up 13%, while Caucasian students make up the remaining 1%. This school falls into the

high range for participation in the nation’s lunch program with 96% of students eligible to receive

free or reduced-price lunch. The student/teacher ratio is approximately 28 to 1


19

Three kindergarten teachers agreed to be in the study. This site adopted a widely published

elementary basal reading and math curriculum. Daily English Language Arts blocks for

kindergarten were 150 minutes. Teachers used both the reading and mathematics portions of the

program with their students. Daily math blocks for kindergarten were 60 minutes. Students

received an average of approximately 20 hours of SuccessMaker reading (standard deviation =

2.31 hours) and an average of 19 hours of SuccessMaker math instruction (standard deviation =

2.05 hours) during the study period.


received an additional training on March 3rd, 2014. Baseline testing was administered during the

third week of December 2014, middle of school year testing during the last two weeks of January

2015, and end of school year testing during the second week of May 2015.

Colorado District This school resides in a rural area and students are expected to follow a dress code. This school

prides itself in being technologically advanced. This school strives to use the technology resources

available to them to help students realize their ultimate potential. In the 2010-11 school year, the

district served a community of approximately 11,400. The median household income is nearly

$83,000, indicating a middle-class community.

This is a small to medium size school serving approximately 450 students in grades kindergarten

through eight. Kindergarten and first grade are both made up of approximately 55 students. This

school is primarily Caucasian, which represents approximately 88% of the school population.

Hispanic students comprise 8% of the school’s population, with multi-ethnic and Asian/Pacific

Islander making up the remaining 4%. This school falls into the low range for participation in the

nation’s free or reduced-price lunch program with only 8% of students eligible to receive free or

reduced-price lunch. The student/teacher ratio is approximately 20 to 1.

Two kindergarten and two first grade teachers agreed to be in the study. This school recently

adopted an experimental system of curricula that includes math and reading instructional materials

and techniques. The school supplements these programs largely with digital programs. Daily

English Language Arts blocks ranged from 60-90 minutes, while daily math blocks were 60

minutes in length.

All participating teachers used the program for both reading and mathematics. During the study

period, kindergarten students received an average of 11 hours of SuccessMaker reading instruction

(standard deviation = 1.20 hours) and an average of 11 hours of SuccessMaker math instruction

(standard deviation = 1.25 hours). First grade students received an average of 8 hours of

SuccessMaker reading instruction (standard deviation = 1.52 hours) and an average of

approximately 9 hours of SuccessMaker math instruction (standard deviation = 1.49 hours).

Teachers were trained on January 20th, approximately five months after school began. They

received an additional training on March 10th 2015.

Kindergarten and first grade received baseline testing during the first week of December 2015.

Middle of school year testing was administered during the first two weeks of February 2015 to

both grades. The end of school year test was administered to both grades during the first two

weeks of May 2015.


20

Ohio District This school resides in a midsize suburb. This school encourages collaboration with the community

to help their students succeed in the 21st century. In the 2010-11 school year, the district served a

community of approximately 11,000. The median household income is approximately $45,000,

indicating a middle-class community.

This is a small to medium sized school serving approximately 450 students in grades kindergarten

through two. Kindergarten and first grade are both made up of approximately 140 students. This

school is primarily Caucasian, which represents 94% of the school population. Hispanic and multi-

ethnic students make up the remaining 6% of the population. This school falls into the mid-range

for participation in the nation’s free or reduced-price lunch program with 64% of students eligible

to receive free or reduced price lunch. The student/teacher ratio is approximately 15 to 1.

Two kindergarten and two first grade teachers agreed to be in the study. The district adopted a

widely published elementary basal reading and mathematics curriculum. Daily English Language

Arts and math blocks were 180-240 minutes in length. One teacher from each grade level used

the program for reading and the other teacher from each grade level used the math portion.

Teachers were trained on November 17th, approximately three months after school started. They

received an additional training on March 11th, 2015.

During the study period, kindergarten students received an average of approximately 18 hours of

SuccessMaker reading instruction (standard deviation = 1.04 hours) and an average of 13 hours of

SuccessMaker math instruction (standard deviation = 0.76 hours). 1st grade students received an

average of 18 hours of SuccessMaker reading instruction (standard deviation = 1.24 hours) and an

average of 18 hours of SuccessMaker math instruction (standard deviation = 1.70 hours).

The baseline testing was administered to Kindergarten during the last week of November and the

first week of December 2015 and to first grade during the first week of December. The middle of

school year testing was administered during the first week of February. End of school year testing

was given to Kindergarten during the last week of April 2015. The end of school year reading test

was given to first grade during the first week of May and the math test was given during the last

week of April 2015.

Oregon District This school is located approximately 25 miles outside of a large metropolitan area. This site is

small, but teachers and administration view their size as a strength to provide an individualized

educational experience for each student. Building strong relationships with families, community

and students promotes a collective responsibility for student success. In the 2010-11 school year,

the district served a community of approximately 28,000. The median household income is

approximately $59,000, indicating a middle-class community.

This is a small sized school serving fewer than 400 students in grades kindergarten through five.

Kindergarten and first grade are both made up of approximately 55 students. This school is

comprised of two primary ethnic groups, Caucasian and Hispanic, which represent a total of 70%

and 19% of the school population respectively. Asian/Pacific Islander, African American,

American Indian/Alaska Native, and multi-ethnic students make up the remaining 11% of the

student population. This school falls into the low range for participation in the nation’s free or


21

reduced-price lunch program with only 6% of students eligible to receive free or reduced-price

lunch. The student/teacher ratio is approximately 19 to 1.

Two first grade teachers agreed to be in the study, with one teacher using the program for reading

and the other math. This site adopted a widely published elementary basal reading and mathematics

curriculum. Daily English Language Arts blocks for 1st grade were 90 minutes in length and math

blocks were 80 minutes in length. During the study period, 1st grade students received an average

of 24 hours of SuccessMaker reading instruction (standard deviation = 0.97 hours) and an average

of 25 hours of SuccessMaker math instruction (standard deviation = 2.15 hours). Teachers were

trained on November 20th, approximately two months after school began. They received an

additional training on April 1, 2015.

The baseline testing was administered during the third week of November 2014, the middle of

school year testing during the last week of January 2015, and the end of school year testing during

the second week of May 2015.


22

A.2 aimswebPlus TEL and TEN Descriptions

aimswebPlus Test of Early Literacy (TEL) Description

Period(s) Given Measure What the Student Will Do What is Recorded Time Limit

Kindergarten

Fall

Print Concepts Show understanding of purpose,

use, and contents (letters,

pictures) of a book

Correct and

incorrect

responses

No time

limit

Kindergarten

Fall, Winter

Initial Sounds Look at 4 pictures and either

point to the one that begins

with a given letter sound

(receptive) or make the sound

that begins the word

(expressive)

Receptive: picture

selected

Expressive:

correct/incorrect

No time

limit

Kindergarten

Fall, Winter, Spring

Letter Naming

Fluency

Name letters Any errors made,

and how many

letters are named

in 1 minute

1 min.

Kindergarten


First Grade


Auditory Vocabulary Point to the picture that

matches the word said by the

examiner

Picture selected No time

limit

Kindergarten

Winter, Spring

First Grade Fall

Letter Word

Sounds

Say the sounds of letters,

syllables, and words for 1

minute

Any errors made,

and how far the

student reaches in

1 minute

1 min.

Kindergarten

Winter, Spring

First Grade Fall

Phoneme

Segmentation

Say the phonemes in orally

presented words

Correct and

incorrect

phonemes

No time

limit

Kindergarten

Spring

First Grade Fall,

Winter, Spring

High Frequency

Words

Read a list of words aloud for 1

minute

Any errors made,

and how far the

student reads in 1

minute

1 min.

First Grade Fall,

Winter, Spring

Oral Reading

Fluency

Read two stories aloud, each for

1 minute

Number of words

read correctly

1 min. each


23

aimswebPlus Test of Early Numeracy (TEN) Description

Period(s) Given Measure What the Student Will Do What is Recorded Time Limit

Kindergarten Fall,

Winter, Spring

Number

Identification (NID)

Verbally name numbers up to

20.

Put a slash through

items answered

incorrectly or

skipped, and circle

the last item

attempted.

1 minute

Kindergarten Fall,

Winter, Spring

Quantity Total (QT) Boxes containing blue dots are

presented. The student states

the total number of dots within

each box or each pair of boxes.

Put a slash through

items answered

incorrectly or

skipped, and circle

the last item

attempted.

1 minute

Kindergarten Fall,

Winter Spring

First Grade Fall,

Winter, Spring

Concepts and

Applications (CA)

Mentally solve various types of

math problems and state the

correct answers.

Circle 1 for

correct answers,

circle 0 for

incorrect answers.

No time

limit

Kindergarten

Winter, Spring

Quantity Match

(QM)

Pairs of boxes containing dots

(one with blue dots, one with

red dots) are presented. The

student states how many more

blue dots are needed to match

the number of red dots.

Put a slash through

items answered

incorrectly or

skipped, and circle

the last item

attempted.

1 minute

First Grade Fall,

Winter, Spring

Number

Comparison

Student identifies which of the

two numbers is larger.

1 minute

First Grade Fall,

Winter, Spring

Math Facts Student mentally solves simple

addition and subtraction

problems involving numbers 1

through 10 and states the

correct answer.

1 minute

First Grade Fall,

Winter, Spring

Mental Computation Student mentally solves

problems involving the addition

and subtraction of 10 and state

the correct answer.

1 minute


24

References

Donnar, A. & Klar, N. (2000) Design and analysis of cluster randomization trials in health research.

Arnold Publishers, London.

Fullan, M. (2001). Leading in a Culture of Change. Hoboken, NJ. Wiley Publishing.

Hedges, L. V. & Olkin, I. (1985). Statistics methods for meta-analysis. Academic Press, NY.

Liang, N. M. & and Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models.

Biometrika, 73, pp. 13-22.

National Reading Panel (2000). Teaching children to read: An evidence-based assessment of the scientific

research literature on reading and its implications for reading instruction [on-line]. Available:

http://www.nichd.nih.gov/publications/nrp/report.cfm

Tamin, R., Bernard, R., Borokhovski, E., Abrami, P. & Schmid, R. (2011). What forty years of

research says about the impact of technology on learning: A second-order meta-analysis and validation

study. Review of Educational Research, 81, 4–28.

Yap, K., Aldersebaes, I., Railsback, J., Shaughnessy, J., & Speth, T. (2000). Evaluating whole-school

reform efforts: A guide for district and school staff (2nd ed.). Portland, OR: Northwest Regional

Educational Laboratory.

http://www.nichd.nih.gov/publications/nrp/report.cfm

successmaker reading and mathematics 2014-15 efficacy...

Documents