researching for better o nline i nstructional m ethods using ab testing
DESCRIPTION
Researching for Better O nline I nstructional M ethods using AB Testing. --- Zhongzhou Chen, Christopher Chudzicki , Peter Bohacek , Neset Demirci and David Pritchard. Teaching is an old practice…. We enter the age of the Internet. Now we can teach like this. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/1.jpg)
Researching for Better Online Instructional
Methods using AB Testing
--- Zhongzhou Chen, Christopher Chudzicki, Peter Bohacek, Neset Demirci and David Pritchard
![Page 2: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/2.jpg)
Teaching is an old practice…
![Page 3: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/3.jpg)
We enter the age of the Internet
![Page 4: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/4.jpg)
Now we can teach like this
![Page 5: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/5.jpg)
Can we find better ways to teach in an online
environment?
![Page 6: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/6.jpg)
In a normal MOOC
Entire MOOC Population
Course Material
![Page 7: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/7.jpg)
A/B Testing
Traditional Teaching Method Experimental Teaching Method
![Page 8: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/8.jpg)
A/B Testing
Simple Design of an A/B Expriment
Treatment Control
Common Assessment
![Page 9: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/9.jpg)
Multiple A/B Experiments in a Course
Unit 1Experiment 1
![Page 10: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/10.jpg)
Multiple A/B Experiments in a Course
Unit 1 Unit 2
Independent Random Splits
Experiment 1 Experiment 2
![Page 11: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/11.jpg)
Multiple A/B Experiments in a Course
Unit 1Experiment 1 (part 1)
Unit 2Experiment 2
Unit 3Experiment 1 (part 2)
![Page 12: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/12.jpg)
Summary of A/B Experiment Features
• No limit on the number of groups, nor on the number of experiments in a course.
• Each group can receive a different number of materials.
• Group assignment is random.
• No student can "opt-out" of any of the experiments.
![Page 13: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/13.jpg)
Examples of A/B Experiments in our MOOC
![Page 14: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/14.jpg)
Our MOOC: 8.MReV
Content: College introductory level mechanics (calculus based)
Length: 14 Weeks (12 required, 2 optional) May 29th ------- Sep 14th
Initially Registered: 10709
Attempted Second Week’s Quiz: ~4100 (estimated)
![Page 15: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/15.jpg)
Our MOOC: 8.MReV
Total Number of AB experiments: 7
![Page 16: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/16.jpg)
Experiment 1: Does length of video make a difference?
![Page 17: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/17.jpg)
Does length of video make a difference?
Office hour videos: ~10+ minutes
Talk about a challenging homework problem
![Page 18: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/18.jpg)
Does length of video make a difference?
~2 minute video segments
Short Video
![Page 19: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/19.jpg)
Does length of video make a difference?
~2 minute video segments
Short Video
![Page 20: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/20.jpg)
Does length of video make a difference?
~2 minute video segments
Short Video
~10 minutes whole video
Long Video
![Page 21: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/21.jpg)
Does length of video make a difference?
~2 minute video segments
Short Video
~10 minutes whole video
Long Video
![Page 22: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/22.jpg)
Experiment Procedure
Challenging Homework Problem (Graded)
![Page 23: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/23.jpg)
Experiment Procedure
Long Video Short VideoChallenging Homework Problem (Graded)
![Page 24: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/24.jpg)
Experiment Procedure
Long Video Short VideoChallenging Homework Problem (Graded)
![Page 25: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/25.jpg)
Does length of video make a difference?
Research Questions:
• Difference in time spent watching video?
• Difference in number of questions answered?
• Difference in successfully solving the challenging homework problem?
![Page 26: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/26.jpg)
Increasing Interactivity of Course Material
![Page 27: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/27.jpg)
Interactivity of Course Material
Traditional problem vs. Direct measurement video
![Page 28: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/28.jpg)
Traditional Physics problem
Provides all necessary information
![Page 29: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/29.jpg)
Direct measurement video (by Peter Bohacek)
Time measured by counting frames
Player can play frame by frame
Distance measured by on screen ruler.
![Page 30: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/30.jpg)
Direct Measurement Video
Common Homework Problem
Traditional Homework
Direct Measurement Video
Survey question
![Page 31: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/31.jpg)
Direct Measurement Video
Research Questions:
• Does the extra interactivity result in more participation, or does the extra work lead to less participation?
• Can we identify any positive impact on students’ knowledge to justify the cost of creating these videos?
![Page 32: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/32.jpg)
Interactivity of Course Material
Multiple Choice vs. Drag and Drop Problem
![Page 33: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/33.jpg)
Multiple Choice
![Page 34: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/34.jpg)
Multiple Choice
![Page 35: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/35.jpg)
Multiple Choice
![Page 36: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/36.jpg)
Extraneous Cognitive Load
Traditional Multiple Choice question
![Page 37: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/37.jpg)
Drag and Drop Example
![Page 38: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/38.jpg)
Drag and Drop Example 2
![Page 39: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/39.jpg)
Drag and Drop and Deliberate Practice
Deliberate Practice: (Erickson)
![Page 40: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/40.jpg)
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
![Page 41: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/41.jpg)
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
![Page 42: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/42.jpg)
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
• Repetitive and highly focused training
![Page 43: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/43.jpg)
Drag and Drop and Deliberate Practice
Deliberate Practice:
• Focus on practicing one skill at a time
• Providing immediate specific feedback
• Repetitive and highly focused training
Domain Specific Problem Design
Problem Design/Auto Grading/ Customized Feedback
Drag and Drop format
![Page 44: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/44.jpg)
Experimental Setup
Entire MOOC population
![Page 45: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/45.jpg)
Experimental Setup
![Page 46: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/46.jpg)
Experimental Setup
![Page 47: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/47.jpg)
Multiple Choice vs. Deliberate Practice
Research Questions:
• Does increasing interactivity result in more engagement? (i.e. Are students more “deliberate” in their practice?)
• Compared to traditional problems, do Drag and Drop problems serve as better/worse assessments?
• Do “deliberate practice” result in any detectable learning gain?
![Page 48: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/48.jpg)
Thank you
Comments and Questions?
![Page 49: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/49.jpg)
Discussion Questions
• Other innovative instructional methods worth putting online and testing via AB experiment?
![Page 50: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/50.jpg)
Discussion Questions
• How should we design our experiments to suit the (unique) MOOC population?
• How can we transfer the experimental results obtained through a MOOC to a "flipped" classroom despite the difference in population?
![Page 51: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/51.jpg)
Discussion Questions
• What criteria should we use to judge whether one pedagogy is better than another? Learning? Engagement? More discussion? Less attrition?
![Page 52: Researching for Better O nline I nstructional M ethods using AB Testing](https://reader035.vdocuments.site/reader035/viewer/2022062309/56814e8d550346895dbc31c2/html5/thumbnails/52.jpg)
Discussion Questions
• How do we design effective assessments of learning when students have the freedom to access any external resources?