how data mining contributes to efficacy studies and course redesign answering your questions: why?...

How Data Mining Contributes to Efficacy

Studies and Course Redesign

Answering Your Questions:Why? What? How?

Claire MassonDoug Paetzell Yun Jin Rho

Rasil Warnakulasooriya

24 Sept 2011

Tracking and Analyzing Student Data3

Your Comments from the pre-workshop survey…

1. Why are statistics useful in commenting on success – Show me some data on redesigned programs– How do I determine student success

2. What should we do after the redesign– How do I use the data more effectively– Show me ways to select the best data from pilot efforts

3. How should data be gathered to evaluate the redesign– What is the best way to collect data– What kind of data can be collected


How to Collect Data Doug & Claire

1. Setting Expectations– Determine goals for course redesign– Effects of increased rigor

2. Pilot the Program– Small program to full implementation– Pace of redesign– Gradual improvement

3. Review some Basic Statistics– Interchangeable learning aids– Why class size matters

4. Getting Started– Worksheets, Checklists, Templates


Setting Expectations

What are the specific goals of the course redesign?

1


First Step: Setting ExpectationsDetermine a Primary Goal

Set Quantifiable Expectations• set a specific goal to frame the redesign.

What is the problem we’re trying to solve?

Guiding questions– What percent do we want to increase student grades?– How many students do we want increase class size without

raising costs?– What qualitative effects do we want to see in our classroom as

a result of increased rigor?


The Flow of Redesign

Set

Goal

Evaluate

Resources

Design

Course

Select

Measurement

Tools

Implement

Course

Prepare

Data

Interpret

Data

Adjust

Course

Analyze

Data

Incr

ease

Lear

ning E

ffect S

ize b

y 0.5

Compute

r Lab A

vaila

ble? I

RB?

Emporiu

m M

odel

Compare

Fin

al Exa

m S

core

s

Compar

e His

toric

al /

Curren

t Exa

m S

core

s

Run Thru

Sem

este

r / G

ive

Last

Yea

r’s F

inal

Exa

m

Results: Statistically

Valid?

Results: Educationally Valid?

Apply Lessons Learned


Was this Redesign Successful?

TEACHER SAYS YES: “For probably the first time, all students are engaged in working on homework on a regular basis. The rigor of the homework assignment has increased, and even as we’ve implemented grading with no partial credit, success rates have increased in the course.”-Rebecca Muller, Mathematics Instructor

-Southeastern Louisiana University


Effects of Increased Rigor“The quality of the class experience has changed. Students come with constructive questions…Class time is more productive.”-Kathleen Almy, Associate Professor

Rock Valley College

“MyStatLab saves me from using class time to explain and re-explain how to solve problems. Because students are more prepared to learn and more proactive in their learning, I can convey more complicated, robust concepts to them. It makes teaching the course more fun to teach.”-Gwen Terwilliger, Ph.D., Professor Emeritus

-University of Toledo


Pilot the Program

The Gradual Process

2


The Road to SuccessPilot vs. Full Rollout

1. Importance of deadlines and benchmarks with assignments

— Teachers who monitor student participation have higher retention rates

2. Grade inflation may skew the effects

— Beginning phases of redesign may require remediation for students who were passed along previously

3. A good pilot often requires three phases of implementation to achieve success

Source: www.thencat.org


The Three Phases

Year 1: Develop and aggregate course material

Year 2:

— 1st Semester: Course development

— 2nd Semester: Campus pilots

— 2nd Semester: Course revision

Year 3:

— 1st Semester: Campus full implementation

— 2nd Semester: Convert course material for full campus

— 2nd Semester: Develop customized best practices plan and rework curriculum

www.thencat.org


Examples of Gradual ImprovementEffect of Course Redesign on Reducing Copying Over Time

—Decrease in copy rate over the four courses

—Decrease between Year 0 and Year 1 due to studio format redesign

—Decrease from Year 2 to Year 3 due to assigning to ABC grades instead of pass-no pass record

Traditional Year 1

Pilot

Year 2

Full Course

Year 3

Full Course

Palazzo/Lee/Warnakulasooriya/Pritchard “Patterns, Correlates, and Reduction of Homework Copying” Physics Education Research 6, 010104

(2010)


Examples of Gradual ImprovementEffect of Course Redesign on Improving Pass Rates Over Time

—Jackson State Community College conducted three pilots before full implementation

—Largest increased occurs after initial transition

—Continued improvement results from numerous adjustment


Review some Basic Statistics

Null Effect / Class Size / p-value

3


Similar Learning AidsStudy Design Yields Null Effect

Null Effects are NOT Negative.

Comparing one learning model to

another with the same intervention

goal, remediation, often yields

same results: null / no effect.


Same Content / Different PlatformStudy Design Yields Null Effect

Teacher assigned the same content, so there should be no expectation of improvement.

Mean exam scores with standard error bars for A&P (7 exams) using CC in 2010 vs. MAP in 2010 vs. MAP in 2011.

CourseCompass v Mastering for A&P


The Meaning Behind Class Size

MyITLab

233 124

p-value <0.05

statistically significant

But educationally significant?

Effect size: 0.44

p-value >0.05

not statistically significant

28 27 28 27 28 27

MyMathLab

67% 64%erro

r ba

rs o

verlap

erro

r ba

rs d

on’t o

verlap

# of students inside bars


Combining data and confounding factors

Combining data must be thoughtfully considered.1. Is it okay to combine your own sections of one class if the same

material is covered. (Consider student population v night / day classes)?

2. Is it okay to combine your student data with a colleague at a different institution (CC and 4-year research schools), administering different exams, etc.?

3. Is it okay to combine data with other instructors at your school teaching the same course?


Getting Started

Worksheets, Checklists, Templates, Examples

4

Handouts

Checklists Worksheets Templates

Learn more…

Case Studies: • www.mylab.com• www.masteringX.com

White Papers:• (Math): Making the Grade• (English): Vision in Action• (Sci/Eng): Make Learning Part of the Grade

Peer-Reviewed Journal Articles:• (MATH): Brewer/Becker “Online Homework Effectiveness for Underprepared and Repeating College Algebra Students,” Journal of Computers in Mathematics and Science Teaching 29(4), 353-371 (2010) • (BIO): Rayner: “Evaluation and Student Perception of MasteringBiology as a Learning and Formative Assessment Tool in a First Year Biology Subject ATN Assessment Conference (2008)• (PHYS): Palazzo/Lee/Warnakulasooriya/Pritchard “Patterns, Correlates, and Reduction of Homework Copying” Physics Education Research 6, 010104 (2010)

Thank you

how data mining contributes to efficacy studies and course redesign answering your questions: why?...

Documents

course year

student success

student grades

student data15

student participation

student data3

student data7

student data11