Scientifically Informed Web-Based Instruction
Financial and Intellectual Support:• The William and Flora Hewlett Foundation• Carnegie Mellon University through the Office of
Technology for Education and the Eberly Center for Teaching Excellence
Joel M. Smith, Vice Provost , CIO
Candace Thille, Director of OLI
“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”
Herb Simon,
Last Lecture Series, Carnegie Mellon, 1998
What Is a “Community Based Research Activity” in Teaching?
• Theory-based
• Feedback Loops
• Diversity of Perspective, Roles & Contexts
Open Learning Initiative
OLI Project Goals• Develop exemplars of scientifically based
online courses and course materials that enact instruction and support instructors
• Provide open access to these courses and materials
• Develop a community of use that contributes to the evaluation, iterative improvement, and ongoing growth of the courses and materials.
OLI is Currently a Small Scale Community Based Research Activity in
Teaching• Theory Based: Course designs based on
current theories in the learning sciences • Feedback Loops: Courses record student
activity for robust feedback mechanisms• Diversity of Perspectives, Roles and
Contexts: Courses developed and deployed by teams that include faculty content experts, learning scientists, software engineers
Theory Based: Build on Prior/Informal Knowledge
What is a Cognitive Tutor?
• A computerized learning environment whose design is based on cognitive principles and whose interaction with students is based on that of a (human) tutor—i.e., making comments when the student errs, answering questions about what to do next, and maintaining a low profile when the student is performing well.
Theory Based: Provide Immediate Feedback in the Problem Solving Context
Theory Based: Promote Coherence
Theory Based: Multiple Representations With Explicit Connections
Theory Based: Promote Authenticity, Flexibility & Applicability
• Learning environments with ambiguous problems that require flexible application of procedural knowledge
Feedback Loops in Learning
Evaluation• Chemistry: Post-test scores by treatment group show
significant positive correlation for the OLI treatment. Most significant indicator was time spent in Virtual Lab Activities – made all other variables drop out.
• Biology: End of the 3rd week showed an advantage for the OLI section. There was a positive and significant association between students’ time spent working on particular activities and performance on quiz questions testing the corresponding topics even after total time with OLI has been regressed out
Evaluation
• Statistics 1st Study:
n Average % correct
Pre 488 43.3
Post 488 51.2
Increase: 7.9% Increase: 7.9% [t(487) = 13.8, p <.001][t(487) = 13.8, p <.001]
n Average % correct
Pre 24 55.8
Post 24 66.5
Increase: 11.7%Increase: 11.7%[t(23) = 4.7, p <.001][t(23) = 4.7, p <.001]
CAOS Sample:CAOS Sample:
CMU OLI Course Sample:CMU OLI Course Sample:
Evaluation
Measured learning Outcome
% correct CAOS % correct CMU
Pre Post Pre Post
Box plots provide accurate estimates of % data above & below only for quartiles
22.2 22.2 22.2 50.0
Correctly estimate and compare SD’s for different histograms.
31.5 46.4 66.7 83.3
41.8 46.4 59.3 75.0Correlation does not imply causation
51.9 49.4 48.1 70.8
Calculating appropriate conditional probabilities given table of data
49.6 47.4 70.4 70.8
Evaluation
Student Satisfaction
– End of course survey for online section:
• All students reported at an increase in their interest in statistics.
• 75% Definitely Recommend
25% Probably Recommend
0% Probably not Recommend
0% Definitely not Recommend
Student Quotes
“I have found it to be one of the fastest/most efficient ways to learn new material. Compared to reading the text, the interactivity makes it harder to skim/gloss over the material so I retain more at the end of the session.”
Student Quotes“I really like the way you can learn
individually and at your own pace. If I understand something, I can move through it quickly and take more time on challenging things.”
"This is so much better than reading a textbook or listening to a lecture! My mind didn’t wander, and I was not bored while doing the lessons. I actually learned something."
Feedback Loop – Current Research
Instructors can use such data to adjust their teaching to students’ needs.
The Vision
• Instructor assigns students to work through online instruction
• System collects data as students work • System automatically analyzes and
organizes the data to present instructor with the students’ current “learning state”
• Instructor reviews this data summary and adapts instruction accordingly
The Anticipated Benefits
• Instructors get a window onto students’ progress
• They can adapt their teaching accordingly
• Students get better feedback to monitor and adjust their learning
• Strengthens the student-instructor connection
Core OLI Community• Faculty Content Experts• Learning Scientists• Human Computer Interaction• Software Engineers• Evaluation/Assessment Specialists• Learners• A community of scholars from diverse
disciplines who are committed to improving quality and access to instruction. The collaborative nature of the OLI course design process inspired participating faculty to rethink their approach to classroom teaching.
Extended Community: Ecology of Use and Reuse
• OLI Workshops in USA, Qatar, Mexico
• OLI Tools and support for adapting, localizing content
• Formal and informal evaluation
• Carnegie Foundation KEEP Toolkit - document scholarship of teaching
Feedback from Evaluation
“ The success of the implementation of the OLI course materials, in terms of learning gains and a renewed student enthusiasm for statistics, led the Department of Psychology at Universidad de los Andes to completely change the instructional sequence in research methodology .”
Extended Community: Ecology of Use and Reuse
• OLI Appliance in locations with poor internet access
• Collaborative Development with institutions in US, Chile, India, Qatar
• Developers workshop in US, Taiwan
The Challenge of Scale
• Theory Based : Learning Theories are like toothbrushes: Everyone has one and no one wants to use anyone else's.
• Feedback Loops: The nature of valuable feedback is contextual
• Ecology of Use and Reuse: Mix and mash-up of content in isolation.
“Improvement in post-secondary education will require converting teaching from a ‘solo sport’ to a community-based research activity”
Herbert Simon
www.cmu.edu/oli
[email protected]@cmu.edu