![Page 1: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/1.jpg)
Experiments in Educational Research & PracticeJoseph Jay Williams
Lytics Lab, Office of the Vice Provost of Online Education, Graduate School of Education, Stanford
![Page 2: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/2.jpg)
Experiments in Educational Research & Practice• Kinds of Experiments
– Laboratory– Design experiment (Brown, 1992)– In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)– Randomized Controlled Trial/Intervention
• Experiments & Internet, Online Education– Internet-facilitated Laboratory studies
• Mechanical Turk (tiny.cc/mechanicalturk )• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )• “A/B testing” more widespread
– Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)– In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)– Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
• REPEAT Approach to Experimental Research on Online/Blended Resources• Identifying a Practical Online Context for your Research Question
– Partner with platform *early* in research– “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)– Examples of Blended Learning Resources & MOOCs– Examples of resources that support Randomized Trials
• Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
– Interventions extended over time– Bridging studies
• Engaging with Practitioners– How to communicate value of experiments– Consulting
• Educational companies & publishers ([email protected] )• Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)• Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)• Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
– Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)– Virtual collaboration on practical resources (www.learnnetwork.net )
• Experiments & …– Computer Supported Collaborative Learning (Rose)– Cognitive Tutors (Aleven, Ritter, Koedinger)– EDM & AIED (Stamper, Gordon, Pardos)
• Conducting Experiments (tiny.cc/conductingexperiments)
![Page 3: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/3.jpg)
Kinds of Experiments
– Laboratory– Design experiment (Brown, 1992)– In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)– Randomized Controlled Trial/Intervention
![Page 4: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/4.jpg)
Experiments & Internet, Online Education
– Online Educational Resources– Internet-facilitated Laboratory studies
• Mechanical Turk (tiny.cc/mechanicalturk )• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )• “A/B testing” more widespread
– Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)
– In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)
– Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)
– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
![Page 5: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/5.jpg)
Online Education
Send questions to tiny.cc/apsonlinelearning
![Page 6: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/6.jpg)
A Novel Opportunity for Research
![Page 7: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/7.jpg)
Integrate Research & Practice
• Real-world environment• Authentic activities• Practical Challenges
• Randomized assignment• Experimental Control• Rich data
• Generalizable theories• “in vivo” experiments• Diverse populations
• Practical improvements• Disseminate research• Generate Funding
![Page 8: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/8.jpg)
Experiments & Internet, Online Education
– Internet-facilitated Laboratory studies• Mechanical Turk (tiny.cc/mechanicalturk )• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )• “A/B testing” more widespread
– Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)
– In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)
– Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)
– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
![Page 9: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/9.jpg)
Jascha Sohl-Dickstein
In-vivo motivation experiment on Khan Academy
• Williams, Paunesku, Haley, & Sohl-Dickstein, MOOCshop, AIED
• Embed messages in online Khan Academy exercises• Effect of Growth Mindset beyond encouragement?
![Page 10: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/10.jpg)
Implicit beliefs about Intelligence
• On a scale from 1 to 10, how much do you agree that?– Your intelligence is something very basic about you that you can’t
change very much.– No matter how much intelligence you have, you can always change it
quite a bit.
• Growth vs. Fixed Mindset (Dweck, 2007)
![Page 11: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/11.jpg)
Experimentally manipulate added messagesPractice-as-usual
Some of these problems are hard. Do your best!Remember, the more you practice the smarter you become!
Growth Mindset Message
Positive Message
![Page 12: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/12.jpg)
Design
• 50 000+ students per condition• Dependent measures:
– Number of problems completed– Number correct on first attempt– Accuracy
• Growth Mindset Message• "Remember, the more you practice the
smarter you become.”,• "Mistakes help you learn. Think hard to
learn from them.”
• Positive Message• "Some of these problems are hard. Just
do your best."• "This might be a tough problem, but we
know you can do it.”
Practice-as-usual
![Page 13: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/13.jpg)
R.E.P.E.A.T. Framework for Online Education Research
• Realistic• Experimental• Product• Evaluated• Accessible• Theoretically motivated
• REPEAT – iteratively improve through revision & collaboration
13
![Page 14: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/14.jpg)
Research & Practice in Online Education
• APS symposium: Williams, Saxberg, Means, Mitros. (2013). Online Learning and Psychological Science: Opportunities to integrate research and practice.
• Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society.
![Page 15: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/15.jpg)
Experiments in Educational Research & Practice• Kinds of Experiments
– Laboratory– Design experiment (Brown, 1992)– In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)– Randomized Controlled Trial/Intervention
• Experiments & Internet, Online Education– Internet-facilitated Laboratory studies
• Mechanical Turk (tiny.cc/mechanicalturk )• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )• “A/B testing” more widespread
– Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)– In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)– Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
• REPEAT Approach to Experimental Research on Online/Blended Resources• Identifying a Practical Online Context for your Research Question
– Partner with platform *early* in research– “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)– Examples of Blended Learning Resources & MOOCs– Examples of resources that support Randomized Trials
• Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
– Interventions extended over time– Bridging studies
• Engaging with Practitioners– How to communicate value of experiments– Consulting
• Educational companies & publishers ([email protected] )• Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)• Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)• Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
– Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)– Virtual collaboration on practical resources (www.learnnetwork.net )
• Experiments & …– Computer Supported Collaborative Learning (Rose)– Cognitive Tutors (Aleven, Ritter, Koedinger)– EDM & AIED (Stamper, Gordon, Pardos)
• Conducting Experiments (tiny.cc/conductingexperiments)
![Page 16: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/16.jpg)
Identifying a Practical Online Context for your Research Question
– Partner with platform *early* in research– “Customer Development”: Engaging with
Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)– Examples of Blended Learning Resources & MOOCs– Examples of resources that support Randomized Trials
• Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
– Interventions extended over time– Bridging studies
![Page 17: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/17.jpg)
Engaging with Practitioners
– How to communicate value of experiments– Consulting
• Educational companies & publishers ([email protected] )• Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will
Thalheimer)• Medical Education & Health Behavior (Medbiquitous, International Society for
Internet Interventions, cognitivescience.co/behaviorchange)• Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
– Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)– Virtual collaboration on practical resources (www.learnnetwork.net )
![Page 18: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/18.jpg)
Experiments & …
– Computer Supported Collaborative Learning (Rose)– Cognitive Tutors (Aleven, Ritter, Koedinger)– EDM & AIED (Stamper, Gordon, Pardos)
![Page 19: Experiments in Educational Research and Practice](https://reader034.vdocuments.site/reader034/viewer/2022052223/557bfab3d8b42a0a6a8b46e2/html5/thumbnails/19.jpg)
Experiments in Educational Research & Practice• Kinds of Experiments
– Laboratory– Design experiment (Brown, 1992)– In vivo embedded experiments (Koedinger, Corbett & Perfetti, 2010)– Randomized Controlled Trial/Intervention
• Experiments & Internet, Online Education– Internet-facilitated Laboratory studies
• Mechanical Turk (tiny.cc/mechanicalturk )• Qualtrics/Rapid authoring tools (cognitivescience.co/qualtrics )• “A/B testing” more widespread
– Iterative Design & Improvement (Amy Collier, Stanford Office of Online Learning)– In vivo motivation experiment on Khan Academy (Williams et al, 2013, MOOCshop)– Conducting Randomized Trials over the Internet (Paunesku, Greene, Williams)– Bridging Studies (Williams et al, 2012; Williams & Poldsam, 2013)
• REPEAT Approach to Experimental Research on Online/Blended Resources• Identifying a Practical Online Context for your Research Question
– Partner with platform *early* in research– “Customer Development”: Engaging with Products/Teachers/Districts/Curricula Designers (Blank & Reese, 2010)– Examples of Blended Learning Resources & MOOCs– Examples of resources that support Randomized Trials
• Go to the source: Empirically test value of instructional prescriptions & professional development (University of Virginia,
– Interventions extended over time– Bridging studies
• Engaging with Practitioners– How to communicate value of experiments– Consulting
• Educational companies & publishers ([email protected] )• Corporate Training & E-Learning (E-learning guild, Devlearn, Bror Saxberg, Will Thalheimer)• Medical Education & Health Behavior (Medbiquitous, International Society for Internet Interventions, cognitivescience.co/behaviorchange)• Synthesize literature: PSLC Wiki, www.josephjaywilliams.com/education
– Grant funding (IES, NSF, Gates, Spencer, moocresearch.com)– Virtual collaboration on practical resources (www.learnnetwork.net )
• Experiments & …– Computer Supported Collaborative Learning (Rose)– Cognitive Tutors (Aleven, Ritter, Koedinger)– EDM & AIED (Stamper, Gordon, Pardos)
• Conducting Experiments (tiny.cc/conductingexperiments)