Scientifically Informed Web-Based Instruction

Financial and Intellectual Support:• The William and Flora Hewlett Foundation• Carnegie Mellon University through the Office of

Technology for Education and the Eberly Center for Teaching Excellence

Joel M. Smith, Vice Provost , CIO

Candace Thille, Director of OLI

“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”

Herb Simon,

Last Lecture Series, Carnegie Mellon, 1998

What Is a “Community Based Research Activity” in Teaching?

• Theory-based

• Feedback Loops

• Diversity of Perspective, Roles & Contexts

Open Learning Initiative

OLI Project Goals• Develop exemplars of scientifically based

online courses and course materials that enact instruction and support instructors

• Provide open access to these courses and materials

• Develop a community of use that contributes to the evaluation, iterative improvement, and ongoing growth of the courses and materials.

OLI is Currently a Small Scale Community Based Research Activity in

Teaching• Theory Based: Course designs based on

current theories in the learning sciences • Feedback Loops: Courses record student

activity for robust feedback mechanisms• Diversity of Perspectives, Roles and

Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers

Theory Based: Build on Prior/Informal Knowledge

What is a Cognitive Tutor?

• A computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutor—i.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.

Theory Based: Provide Immediate Feedback in the Problem Solving Context

Theory Based: Promote Coherence

Theory Based: Multiple Representations With Explicit Connections

Theory Based: Promote Authenticity, Flexibility & Applicability

• Learning environments with ambiguous problems that require flexible application of procedural knowledge

Feedback Loops in Learning

Evaluation• Chemistry: Post-test scores by treatment group show

significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out.

• Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students’ time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out

Evaluation

• Statistics 1st Study:

n Average % correct

Pre 488 43.3

Post 488 51.2

Increase: 7.9% Increase: 7.9% [t(487) = 13.8, p <.001][t(487) = 13.8, p <.001]

n Average % correct

Pre 24 55.8

Post 24 66.5

Increase: 11.7%Increase: 11.7%[t(23) = 4.7, p <.001][t(23) = 4.7, p <.001]

CAOS Sample:CAOS Sample:

CMU OLI Course Sample:CMU OLI Course Sample:

Evaluation

Measured learning Outcome

% correct CAOS % correct CMU

Pre Post Pre Post

Box plots provide accurate estimates of % data above & below only for quartiles

22.2 22.2 22.2 50.0

Correctly estimate and compare SD’s for different histograms.

31.5 46.4 66.7 83.3

41.8 46.4 59.3 75.0Correlation does not imply causation

51.9 49.4 48.1 70.8

Calculating appropriate conditional probabilities given table of data

49.6 47.4 70.4 70.8

Evaluation

Student Satisfaction

– End of course survey for online section:

• All students reported at an increase in their interest in statistics.

• 75% Definitely Recommend

25% Probably Recommend

0% Probably not Recommend

0% Definitely not Recommend

Student Quotes

“I have found it to be one of the fastest/most efficient ways to learn new material. Compared to reading the text, the interactivity makes it harder to skim/gloss over the material so I retain more at the end of the session.”

Student Quotes“I really like the way you can learn

individually and at your own pace. If I understand something, I can move through it quickly and take more time on challenging things.”

"This is so much better than reading a textbook or listening to a lecture! My mind didn’t wander, and I was not bored while doing the lessons. I actually learned something."

Feedback Loop – Current Research

Instructors can use such data to adjust their teaching to students’ needs.

The Vision

• Instructor assigns students to work through online instruction

• System collects data as students work • System automatically analyzes and

organizes the data to present instructor with the students’ current “learning state”

• Instructor reviews this data summary and adapts instruction accordingly

The Anticipated Benefits

• Instructors get a window onto students’ progress

• They can adapt their teaching accordingly

• Students get better feedback to monitor and adjust their learning

• Strengthens the student-instructor connection

Core OLI Community• Faculty Content Experts• Learning Scientists• Human Computer Interaction• Software Engineers• Evaluation/Assessment Specialists• Learners• A community of scholars from diverse

disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.

Extended Community: Ecology of Use and Reuse

• OLI Workshops in USA, Qatar, Mexico

• OLI Tools and support for adapting, localizing content

• Formal and informal evaluation

• Carnegie Foundation KEEP Toolkit - document scholarship of teaching

Feedback from Evaluation

“ The success of the implementation of the OLI course materials, in terms of learning gains and a renewed student enthusiasm for statistics, led the Department of Psychology at Universidad de los Andes to completely change the instructional sequence in research methodology .”

Extended Community: Ecology of Use and Reuse

• OLI Appliance in locations with poor internet access

• Collaborative Development with institutions in US, Chile, India, Qatar

• Developers workshop in US, Taiwan

The Challenge of Scale

• Theory Based : Learning Theories are like toothbrushes: Everyone has one and no one wants to use anyone else's.

• Feedback Loops: The nature of valuable feedback is contextual

• Ecology of Use and Reuse: Mix and mash-up of content in isolation.

“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”

Herbert Simon

www.cmu.edu/oli

cthille@cmu.edujoelms@cmu.edu