a framework for decision-making: developing an outcomes-based evaluation plan for adaptive learning
TRANSCRIPT
![Page 1: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/1.jpg)
Katrice A. Hawthorne, PhD
Karen Vignare, PhD
Cristi Ford, PhD
A Framework for Decision-Making:
Developing an Outcomes-Based
Evaluation Plan for Adaptive Learning
dLRN 2015 Conference
October 16 - 17, 2015
Stanford, CA
![Page 2: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/2.jpg)
◼Undergrad & Grad Enrollment: 87,000◼Demographics:
▪47% of the total enrollment comprised of minority students (34% African American)
▪20% of students are Pell eligible
◼Largest public online open access university ◼Primarily online, but serving 50% military
including on bases throughout world
University of Maryland University College
![Page 3: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/3.jpg)
◼74% of undergraduate students enrolled in UMUC’s stateside programs work full-time.◼54% of students are working parents.◼Median age for students enrolled in UMUC’s
stateside undergraduate programs is 31 years old.
University of Maryland University College
![Page 4: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/4.jpg)
“ An approach to instruction and remediation that uses technology and accumulated data to provide customized program adjustments based on an individual student’s level of demonstrated mastery.”
Waters, J. K. (2014, April 16). The great adaptive learning experience. Campus Technology. Retrieved from http://www.campustechnology.com
Defining Adaptive Learning
![Page 5: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/5.jpg)
◼2010, Carnegie grant with CMU for Open Learning Initiative (cognitive tutors)
◼2014, launched three vendor tools for piloting (Cogbooks, Realizeit, and EdReady)
◼2014, Next Gen Courseware subgrantee to Stanford OLI and Lumen Learning
UMUC History with Adaptive Learning
![Page 6: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/6.jpg)
6
UMUC Adaptive Research Scope
![Page 7: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/7.jpg)
“The systematic acquisition and assessment of information to provide useful feedback about some object” (program, policy, technology, etc.).
Defining Evaluation
Trochim, W. M. K. (2006). Introduction to Evaluation. Retrieved from http://www.socialresearchmethods.net
![Page 8: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/8.jpg)
Continuous Improvement Cycle
![Page 9: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/9.jpg)
Evaluation Logic Model
![Page 10: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/10.jpg)
What We THINK We Know
10
• Accurately predict persistence
• Influence persistence through interventions
• Predict which students need more help
• Non-cognitive factors are an important predictor of success/retention
![Page 11: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/11.jpg)
Results - Course Completion
![Page 12: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/12.jpg)
Results - Grade Distribution
![Page 13: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/13.jpg)
Study Time and Student Achievement
◼Moderate correlation between study time and score gain in solving equations, r(26) = .458, p = .019.
◼Moderate correlation between study time and score gain in percents and real numbers, r(25) = .472, p = .017.
◼Strong correlation between study time and score gain in whole numbers, fractions, and mixed numbers, r(26) = .501, p = .009.
![Page 14: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/14.jpg)
Score Gain
![Page 15: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/15.jpg)
Student Perceptions of Adaptive
![Page 16: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/16.jpg)
Student Comments
“I know so much more than I did before and this is an accomplishment for me in math. I have always stayed away from math until this class. This class helped me to move through math in a way I did not see me doing just weeks ago.”
“I really appreciated the A.I. It assesses my knowledge level as I complete questions and adjust accordingly allowing me to progress at my own pace.”
“It presented the lecture in bite size, easy to digest chunks of information and provided quizzes to assess comprehension with immediate feedback on answers.”
![Page 17: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/17.jpg)
What We Have Learned
◼Learning is not linear
◼Combine learning science and data science
◼Effective, flexible faculty and rich feedback are integral
◼Not enough content in some areas
◼Mapping and course organization are important
![Page 18: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/18.jpg)
What We Want to LEARN
![Page 19: A Framework for Decision-Making: Developing an Outcomes-Based Evaluation Plan for Adaptive Learning](https://reader035.vdocuments.site/reader035/viewer/2022070515/587a33661a28abdb1c8b51ab/html5/thumbnails/19.jpg)
Questions & Contact
Katrice A. Hawthorne, Associate Director, Evaluation and ResearchCenter for Innovation in Learning and Student SuccessUniversity of Maryland University [email protected]
Karen Vignare, Vice ProvostCenter for Innovation in Learning and Student SuccessUniversity of Maryland University [email protected]