1

Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning,

Financial Success, and Cognitive & Learning Sciences Research

Joseph Jay WilliamsLytics Lab, Graduate School of Education

Office of the Vice Provost for Online Learning, Stanfordjosephjaywilliams@stanford.edu

www.josephjaywilliams.com

[Originally from Trinidad & Tobago – apologies for difficulty in following accent.]

2

Topics

• I. Motivation• II. Learning through

Questions/Assessments• III. General Learning & Problem-

Solving Skills• IV. Performance Support Tools &

Checklists

3

Guiding Principles

• Multiple Constraints: Practical, Scientific, Financial

• Modularity in Product Design: “MOOClets”• Future-thinking but Actionable• Empirical and Integrative Work• Experiment-Focused Instructional Design• Matchmaking Relevant Researchers• Iterative Improvement• Collaborative Development

4

I. Motivation

• Beliefs about Intelligence• To what extent do you agree?

How smart you are can be significantly changed by your effort and hard work.You have a certain amount of intelligence, and what you do doesn’t really change that much.

• Growth Mindset vs. Fixed Mindset• Dweck, 2008; Yeager & Walton, 2012• Williams, Paunesku, Haley & Sohl-Dickstein, 2013

5

MOOClet 1: Exercise (Khan Academy)

1. Number of Problems Practiced

2. Accuracy

[TEXT SHOWN HERE – Randomize]

6

Embed motivational messages

Practice-as-usual

Remember, the more you practice the smarter you become!

Growth Mindset Message

7

Growth Mindset Message helpful?

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Practice-as-usual

Jascha Sohl-Dickstein

8

Results: More motivated?

• Growth Mindset Message > Practice-as-Usual

• reduction in dropout rate• increase in Number of problems practiced • increase in Number Correct

• Accuracy: Problems correct/Problems attempted

• Accuracy improvement

9

Interpreting results

• Obvious?• Skeptical? Anything’s statistically

significant with large N…• Second experiment (run simultaneously)

10

Practice-as-usualGrowth Mindset Message

Positive Message

Different motivational messages

Some of these problems are hard. Do your best!Remember, the more you practice the smarter you become!

11

Does any positive message work?

• Growth Mindset Message

• "Remember, the more you practice the smarter you become.”,

• "Mistakes help you learn. Think hard to learn from them.”

• Positive Message• "Some of these problems are

hard. Just do your best."• "This might be a tough problem,

but we know you can do it.”

• Practice-as-usual

12

Effects of Positive Messages?

• Positive Messages not better than Practice-as-Usual!

• Growth Mindset > Positive Message• How intuitive is this result?• Teachers?• Ongoing: After incorrect answer• www.assistments.org Neil Heffernan• Gates Grant for Modeling Research

Rapid Experimentation: tiny.cc/mechanicalturk

13

Guiding Principles

• Multiple Constraints: Practical, Scientific, Financial

• Modularity in Product Design: “MOOClets”• Future-thinking but Actionable• Empirical and Integrative Work• Experiment-Focused Instructional Design• Matchmaking Relevant Researchers• Iterative Improvement• Collaborative Development

14

Constraint Satisfaction

Practical ProductResearch

Financial Value

“MOOClets” or Modular Online Resources

• E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment

• www.josephjaywilliams.com/mooclets • MOOClets Group Tue 12-1 PST (remote)• tiny.cc/joinmooclets mailing list & updates• Williams, J.J. & Williams, B. A. (2013). Randomized

Experiments as a Methodological and Conceptual Tool for Improving Design of Online Learning. Data Driven Education, NIPS Workshop. tiny.cc/experimentsonlinelearning

15

16

Integrate Diverse Expertise & Innovations

17

Where is relevant research, best practice, technology, funding, market? tiny.cc/mapofspace, tiny.cc/stanfordmap

MOOCs (Coursera, EdX, Udacity)

www.KhanAcademy.org www.Assistments.org

K-12 Ed-Tech space (Bay Area, Boston...)(www.edsurge.com)

(www.edudemic.com )Higher-Ed Technology/LMS

E-Learning & Training forCorporate, State, Federal, Army

Psychological ScientistsComputer Scientists

Education

Corporate Training

Funding Agencies:IES, Dept. of Education, Dept. of Labor,

NIH, NSFPractical Reports:

What Works ClearinghouseIES Practice Guides

University Teaching & Learning CentersDepartment of Education reports

Think Tanks:SRI International

WestEd

Pittsburgh Science of Learning Center

Learning Sciences

Mooc Research Initiative

Open Learning Initiative

Distance LearningAthabasca, Western Governor’s University

Economics of Education

Habit & Behavior Change

Medical Education

Centers/Scholarship of Teaching & Learning

Assessment

PearsonETS

Open Education ResourcesCreative Commons

Foundations:Gates, Spencer, Hewlett, Kresge

Venture CapitalPrivate VCs

New SchoolsImpact Investing

Accelerators:Imagine K-12,

HCI, Design Thinking

Workforce Development

18

Relevant Resources

• www.josephjaywilliams.com/home/useful-software

• www.josephjaywilliams.com/education• www.josephjaywilliams.com/online-education-exp

eriments• tiny.cc/mapofspace• Applying Cognitive Science to Online

Learning. Paper presented at NIPS Workshop on Data Driven Education (tiny.cc/cognitivescienceinmoocs)

• Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science.

19

Guiding Principles

• Multiple Constraints: Practical, Scientific, Financial

• Modularity in Product Design: “MOOClets”• Future-thinking but Actionable• Empirical and Integrative Work• Experiment-Focused Instructional Design• Matchmaking Relevant Researchers• Iterative Improvement• Collaborative Development

Messages in Coursera?

20

Week 6 QuizWhether or not you’re correct on the first try, each attempt at a tough problem can make you better at statistics.

21

Motivational Introductions

Practice-as-usualHave fun in the course!

Anne Friedlander Growth MindsetText version of Mindset MessageMindset Message – Another InstructorPositive EncouragementBelonging

22

Instructor Guide to foster Growth Mindset• tiny.cc/idstrategymindset

Where do you think you will apply this?

Do you have any questions or want clarification?

Click to see any of the Case Studies below

Would you like to?

23

Guiding Principles

• Multiple Constraints: Practical, Scientific, Financial

• Modularity in Product Design: “MOOClets”• Future-thinking but Actionable• Empirical and Integrative Work• Experiment-Focused Instructional Design• Matchmaking Relevant Researchers• Iterative Improvement• Collaborative Development

24

Topics

• I. Motivation• II. Learning through

Questions/Assessments• III. General Learning & Problem-

Solving Skills• IV. Performance Support Tools &

Checklists

II. Learning through Questions/Assessments

• Discovery of patterns(Williams & Lombrozo, 2010, Cognitive Science)

• Use of prior knowledge(Williams & Lombrozo, 2013, Cognitive Psychology)

• Children’s causal learning (Walker, Williams, Lombrozo, Gopnik, 2012)

• Overgeneralize at expense of specific facts

(Williams et al, 2013, Journal of Experimental Psychology: General)

• Contradictions & belief revision (Williams, et al 2013; 2014; Cog Sci, CHI WiP)

• Comparison of examples (Edwards, Williams, Lombrozo, Gentner, 2013)

GLORP DRENTOnline/Blended

Educational Resources

26

Adding questions/activities

• tiny.cc/augmentedvideo• tiny.cc/augmentedexercise

27

III. General Learning & Problem-Solving Skills

Learning/Self-Questioning Strategies

• Specific Prompt VS.• General Strategy of Self-Questioning?• Teach strategies

(McNamara, 2004, Discourse Processes; Palinscar & Brown, 1984, Cognition and instruction; Williams & Lombrozo, 2010, Cognitive Science)

• Habit & Behavior Change • Theory & Technology: Embed & Remind• Methodology: “longitudinal”, large N

28e.g. Weinstein & Mayer, 1986 Rothman, Baldwin, & Hertel, 2004

29

Embedded “On-Request” Strategy Instruction

Clickable link.

[Click here to learn about the “What? Why? How?” strategy]

30

Self-questioning strategy: What? Why? How?

31

Cues – Reflect & Remind

32

Preliminary Results

N = ~ 800 per condition

• Prompted Problem-solving better than Practice-as-usual.

• Increased Accuracy on intervention and post-intervention problems.

33

Learning Assistant

• tiny.cc/learningassistant

34

Learning Assistant – e.g. 1

35

Guiding Principles

• Multiple Constraints: Practical, Scientific, Financial

• Modularity in Product Design: “MOOClets”• Future-thinking but Actionable• Empirical and Integrative Work• Experiment-Focused Instructional Design• Matchmaking Relevant Researchers• Iterative Improvement• Collaborative Development

36

Topics

• I. Motivation• II. Learning through

Questions/Assessments• III. General Learning & Problem-

Solving Skills• IV. Performance Support Tools &

Checklists

37

IV. Performance Support Tools & Checklists

38

Coach/Support Learning Behaviorstiny.cc/kalearningcoach

39

A Lesson: Concept + Digital Tool to apply Concept in Future

40

Tool for Difficult Conversations

41

Topics

• I. Motivation• II. Learning through

Questions/Assessments• III. General Learning & Problem-

Solving Skills• IV. Performance Support Tools &

Checklists

42

Guiding Principles

• Multiple Constraints: Practical, Scientific, Financial

• Modularity in Product Design: “MOOClets”• Future-thinking but Actionable• Empirical and Integrative Work• Experiment-Focused Instructional Design• Matchmaking Relevant Researchers• Iterative Improvement• Collaborative Development

43

Acknowledgements

• Tania Lombrozo & Tom Griffiths• John Mitchell & Candace Thille• Jascha Sohl-Dickstein• Jace Kohlmeier & Khan Academy• Sam Maldonado• Lytics Lab (lytics.stanford.edu)• VPOL (Vice Provost for Online

Learning, online.stanford.edu)