finding what works in learning: simple ways to analyze education research studies

Finding What Works in Learning:

Simple Ways to Analyze Education Research Studies

Dr. Tim Hudson Vice President of Learning

DreamBox Learning@DocHudsonMath

Dr. Gina BurkhardtEducation & Organizational

Consultant, Former EVP at American Institutes for Research (AIR)

• Easily analyze research and ask probing questions to meaningfully and successfully review educational research about curricular programs

• Eight key questions to guide analysis of the “fine print” of a research study

• Using a simple rubric to quickly analyze the quality of any research study

• Ideas for conducting research in your own classroom, school, or district

“But where I’m flummoxed is how we are supposed to provide practicing educators with the tools to evaluate

these kinds of findings.”

-Justin Reich http://blogs.edweek.org/edweek/edtechresearcher/2013/08/the_headlines_versus_the_study_in_blended_algebra_program.html

EdWeek in 2013: The Headlines Versus The Study in Blended Algebra Program

Research-Based PLUS

Real-World Studies

What is the basis of the program in terms of the existing research and literature?

What is the research conducted in the real world that validates the program’s impact?

Visible Learning by John Hattie

http://visible-learning.org/glossary/#2_Piagetian_programs

Piagetian Programs“Focus on the thinking processes rather than the outcomes and do

not impose the adult thinking process on to children.”

Headlines and Studies

DreamBox Learning Drives Compelling Achievement Gains in Elementary Mathematics, Suggests Harvard University Personalized Learning Study

Study of 3,000 Students in Grades 3-5 Showed DreamBox Improved Math Achievement on NWEA MAP and PARCC Assessments

Key Findings

TechnicalAppendix

(2016). DreamBox Learning Achievement Growth in the Howard County Public School System and Rocketship Education . Center for Education Policy at Harvard University. http://cepr.harvard.edu/publications/dreambox-learning-achievement-growth

1

Conducted by the Center for Education Policy Research (CEPR) at Harvard University

3,000 Students in Grades 3-5

2 School Systems

100 Classrooms

2013-14 & 2014-15

NWEA MAP, PARCC, and State Assessments

Quasi-Experimental Analysis

SRI Study: Rocketship Students Experience Stronger Math Gains with More Online Instruction

DreamBox Intelligent Adaptive Learning™ Platform Significantly Improves Students’ Math Scores

SRI International Study of Rocketship Education’s Use of DreamBox Learning Validates Learning Gains

2

Conducted by SRI International

583 Students in Grades K-1

1 School System

3 Schools

2010-11

NWEA MAP Assessments

Randomized Controlled Trial

http://cepr.harvard.edu/publications/dreambox-learning-achievement-growth, Technical Appendix, page 13

Acme’s Algebra 2 ProgramQuadruples Growth for 11th Graders

Researchers from the prestigious University of Antarctica examined high schools where at least half of the students in Algebra 2 classes successfully completed 75 percent of the Acme Algebra 2 curriculum during the 2014–15 school year. Researchers found 12 schools in the region that fit these criteria, for a total of 984 eleventh-grade Algebra 2 students. For use as a comparison group, researchers also identified 12 demographically comparable schools in the area with 1,126 Algebra 2 students. For the baseline scores, researchers used the percentage of eleventh-grade students who scored proficient or advanced on the 2014 end-of-course Algebra 2 state exam at each of the 24 schools. In schools using Acme’s Algebra 2 online math program, the percentage of eleventh graders scoring at these top two levels on the state Algebra 2 exam in the spring of 2015 was 4 percent higher than the eleventh-grade spring 2014 baseline percentage. In the control-group schools, the percentage only increased by 1 percent (p < .01).

FAKE

Acme’s Algebra 2 ProgramQuadruples Growth for 11th Graders

(continued) Acme’s Algebra 2 online math program is a curriculum that allows students to move at their own pace as they view online videos and complete related practice problems. As reported in the study, students worked on the Acme program at school for 2 hours each week during the school year. They were also able to access the program at home. The Algebra 2 curriculum for the 12 schools was rewritten to align with the Acme Algebra 2 program. Therefore teachers received 5 days of training for the Acme online program before and during the school year. Teachers also used Acme’s reports throughout the school year to inform their instruction. Acme’s Algebra 2 online program costs $100 per student each year. Funding for this study was provided by Acme Education, Inc.

FAKE

Eight Key Questions to Ask of Every Study

The Rubric

1. Who conducted the study?

2. What was the research methodology?

3. Whose pre-test and post-test scores were compared?

4. Which students were included in (or excluded from) the treatment group?

Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance

milestone during the study time period. (Quasi-Experimental or Case Study)



Athlete Monday Tuesday Wednesday Thursday Friday Final (2 meters)

1 1.66 1.77 1.88 1.99 2.012 1.99 2.01 2.02 2.01 2.023 1.30 1.35 1.40 1.45 1.454 1.00 1.02 1.01 1.02 1.01



Athlete Monday Tuesday Wednesday Thursday Friday Final (2 meters)

1 1.66 1.77 1.88 1.99 2.012 1.99 2.01 2.02 2.01 2.023 1.30 1.35 1.40 1.45 1.454 1.00 1.02 1.01 1.02 1.01

Treatment Group


milestone during the study time period. (Quasi-Experimental or Case Study)2. School faculty or administration in the treatment group volunteered to use the

program, indicating there are other significant motivators & accountability systems.



program, indicating there are other significant motivators & accountability systems.3. Students assigned to the treatment group experienced significant classroom

curricular changes that coincided with the implementation of the program, which increases the difficulty of isolating the program’s impact



program, indicating there are other significant motivators & accountability systems.3. Students assigned to the treatment group experienced significant classroom

curricular changes that coincided with the implementation of the program, which increases the difficulty of isolating the program’s impact

4. Teachers in the treatment group participated in significant professional development (PD) that accompanied the implementation of the program that was different than the PD for teachers in the control group

4. Which students were included in (or excluded from) the treatment group?

5. What were the school support requirements during the study?

6. How large and representative was the researched population?

7. How significant were the findings?

http://fivethirtyeight.com/features/not-even-scientists-can-easily-explain-p-values/

7. How significant were the findings?

8. To what extent did learning and achievement improve?


Acme Algebra 2


Spring 2014 % Proficient Spring 2015 % Proficient

TreatmentControl

+ 4%

+ 1%

4x Growth

Acme Algebra 2

What was the baseline?


TreatmentControl

+ 4%

+ 1%

4x Growth

???

Acme Algebra 2



TreatmentControl

94%

91%

4x Growth

90%

Acme Algebra 2



TreatmentControl

54%

51%

4x Growth

50%

Acme Algebra 2



TreatmentControl

24%

21%

4x Growth

20%

Acme Algebra 2

The Rubric

Considerations for Designing & Conducting Your Own Studies• Randomized Controlled Trials are difficult and often inadvisable

in many education contexts• Quasi-Experimental Studies need valid comparison groups

(quasi-control and quasi-treatment groups)• Case Studies often have no comparison group, which must be

considered when interpreting any potentially causal impact• Create unbiased treatment and control groups (i.e., no minimum

restrictions on the amount of curriculum completed or only considering students who are “on grade level” in the program)

• Use same-student comparisons, not averages of a classroom, grade level, or school, nor different cohort comparisons

• Consider growth from each student’s baseline

Q & A

Dr. Tim Hudson Vice President of Learning

DreamBox Learning@DocHudsonMath

Dr. Gina BurkhardtEducation & Organizational

Consultant, Former EVP at American Institutes for Research (AIR)

Thank You!

DreamBox Learning® K–8 MathTransformative Learning Experiences available in English & Spanish

DreamBox Lessons & Virtual ManipulativesIntelligently adapt & individualize to:• Students’ own intuitive strategies• Kinds of mistakes• Efficiency of strategy• Scaffolding needed• Response time

“I’ve tried teaching this topic with pencil, paper, and plastic manipulatives for twenty years and students have never really gotten it. I can immediately see that DreamBox has invented new, digital manipulatives and games that are far more powerful for student learning.”– Urban District Math Leader, CA

AssignFocus™

Differentiated assignments for every student through your Insight Dashboard

To accelerate learning, offer remediation, and adjust classroom instruction

Prepare for the Next Unit Homework

Assignment Review for Assessment

Empowered Teaching with Data.Inspired Learning with Technology.

iNACOL is right around the corner… Catch us at @DreamBox_Learn

Learn more and see how it works:www.DreamBox.com/request-a-demo

Efficacy: Independent Validation from SRI and from CEPR at Harvard University

We value your feedback!Let us know how we’re doing:

www.surveymonkey.com/r/GC6ZCM7