MOOCs in STEM Conference, June 2014

Elaine Collins, Ph.D., Associate Dean College of Science SJSU

Eva Schiorring, MPP, Senior Researcher, Research & Planning Group

SJSU Plus: SJSU-Udacity Partnership Spring 2013

Purpose2

• Share what was learned from SJSU-Udacity AOLE pilot project

• Discuss potential or “value added” of MOOCs/AOLEs

• Identify and facilitate conversation about questions raised by AOLE

Summary3

• Introduction to the SJSU AOLE

project

• Evaluation findings

• Lessons learned

SJSU Plus 2013

• SJSU Plus: Announcement and purpose

• First Iteration • Intro to College Algebra and Stats• Remedial Math

• Enrollment

Research Design and Implementation

Research Questions

• Who engaged and who did not engage in a sustained way and who passed or failed the AOLE courses?

• What student background and characteristics and use of online material and support services are associated with success and failure?

• What do key stakeholders (students,  online support services, faculty, coordinators, and leaders) tell us they have learned? 6

Data & Research Design/Implementation

• Exploration through contingency table analysis

• Logistic-regression models examining impact on pass/fail of 18 independent variables

• Grounding quantitative results in findings generated by qualitative research

7

Outcomes8

Course % Passed % Passed in Parallel FTF Courses for SJSU students

MATH 6L Matriculated 29.8% 34%-54% (2004-2009)MATH6L Non-matriculated 17.6%

MATH 8 Matriculated 50.0% 52%-74% (2010-2013)Spring 2013: 76%

MATH 8 Non-matriculated 11.9%

STAT 95 Matriculated 54.3% 71%-80% (2010-2013)Spring 2013: 75%

STAT 95 Non-matriculated 48.7%

9

Exploratory Example

0

20

40

60

80

100

< Median ? Median

Pass

Fail

≥

Percent Passing by Problems Submitted All Students

Problems Problems

10%

44%

56%

90%

Perc

en

t

PassedFailed

Model Findings10

Measured Variable

LatentVariable

Student Group(s)for which Variable

is Significant

Expected Improvement

in Odds of Pass over Fail per Unit of Variable Added

Strength of Net Effect

Confidence Effect is

not Random

Problems Done

Degreeof

Effort

High levels indicate

early effort & persistence

All MatriculatedMath 6LMath 8

30.5% –36.7%per problem done Strong 97.1% – 99.9%

Video TimeAll MatriculatedNon-matriculated

Stat 950.01% – 0.08%

per video minute

Extremely strong for Stat 95, strong

otherwise

99.1% - 99.9%

Weeks Active for at Least ½ Hour

Non-matriculated 34.2%per week Strong 98.6%

Number of SessionsLogged In

Math 8 3.3%per session Strong 98.1 %

Support Staff Characters Typed

Use of Support Non-matriculated -0.02%per character Negative 98.0%

Model Finding Example11

Expected Net Probability of Passingby Problems Done

All, Matriculated, Math 6, Math 8 Studentsholding Video Time & Sessions Logged In

constant at their means

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Problems Done

p(P

as

sin

g)

All Students w Video Time

Matriculated w Video Time

Math 6

Math 8 w Sessions

Expected Probability of Passing*All Students & Matriculated Students

Models of two groups: same results

p(P

ass

ing

)

* Holding Video Time constant

Problems Submitted

0.50

12

Expected Net Probability of Passingby Problems Done

All, Matriculated, Math 6, Math 8 Studentsholding Video Time & Sessions Logged In

constant at their means

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Problems Done

p(P

as

sin

g)

All Students w Video Time

Matriculated w Video Time

Math 6

Math 8 w Sessions

Expected Probability of Passing*All Students & Matriculated Students

Models of two groups: same results

p(P

ass

ing

)

Video Hours

* Holding Problems Submitted constant

Model Finding Example

0 50 100 150 200 250 300 350 400

Summary of Quantitative Analysis

13

• Student effort trumps all other variables in explaining outcomes

• Clearest predicators of passing were number of problem sets submitted; video time watched

• Nonlinear relationship – effect increases after baseline has been achieved

• Idiosyncratic finding regarding impact of support services

Qualitative Research Findings14

Students:

•did not understand what support was available to them

•expressed desire for more help with course content

•recommended to friends enrolling in the course: “Don’t fall behind.”

Qualitative Research Findings15

Udacity Support Providers:

•best way to use us is to get help becoming “unstuck.”

•most intense users of support in beginning: high school students with almost no chance of success

•noted potential to provide just-in-time support in response to evidence of where many students get stuck

Qualitative Research Findings16

Faculty:•More extensive planning would have been productive (decision-making, engagement of partners, etc)

•Enormous amount of work to develop the content

•Beneficial to have team-approach to teaching at least during first iterations

•Students almost never asked questions about content

•Potential to develop exciting content that could help different kinds of learners become successful

Lessons Learned, Potential Uncovered17

• Potential to deliver instruction in new ways not possible in classroom, potential to reach different kinds of learners

• Potential to deliver targeted, intrusive and just-in-time supports

• Potential to use MOOCs as content in flipped classrooms providing opportunities for more active learning

Lessons Learned, Challenges Encountered

18

• Faculty need to play the lead in designing and delivering the MOOC – they should also be able to make changes to the content over time

• MOOC content and design requires an enormous investment of time – intellectual property rights issues should be considered up front

• MOOC providers and universities operate very differently – time should be invested in developing the partnership, clarifying roles and responsibilities

Lessons Learned, Challenges Encountered

• MOOCs can generate very useful information in real time that can be used to target intrusive supports – but getting the information can be difficult

• Learning platforms need to be designed with student in mind and students need help navigating the online environment

Questions to Consider

• What is the purpose of the MOOC?

• Who is the intended audience?

• Can supports be scaled?

For More Information

Elaine Collins elaine.collins@sjsu.eduEva Schiorring eschiorring@rpgroup.org

NSF Project Page:

http://www.sjsu.edu/chemistry/People/Faculty/Collins_Research_Page/index.html

For More Information

Distance Education:Rob Firmin, Eva Schiorring, John Whitmer, Terrence Willett, Elaine D Collins, & Sutee Sujitparapitaya, Case study: using MOOCs for conventional college coursework

http://www.tandfonline.com/doi/full/10.1080/01587919.2014.917707#.U5CvZvldXzc