identifying & increasing stem student engagement through ... · john whitmer, ed.d. program...

John Whitmer, Ed.D. Program Manager, Academic Tech. & Analytics

California State University, Office of the Chancellor

Identifying & Increasing STEM Student Engagement through Learning Analytics

Webinar for Project Kaleidoscope

25 October 2013

Slides: http://bit.ly/19vY9aD


Outline 1.  Problem: STEM Attrition, Disengagement &

Academic Technologies

2.  Research Findings: 3 Learning Analytics Studies

3.  Dashboards for High Impact Practices

4.  Commercial Product Examples (time permitting)

5.  Q & A


PROBLEM: STEM ATTRITION, DISENGAGEMENT & ACADEMIC TECHNOLOGIES


STEM Student Attrition (2003-2009)

Source: National Center for Education Statistics http://nces.ed.gov/pubs2013/2013152.pdf

Slides: http://bit.ly/19vY9aD Source: National Center for Academic Transformation http://www.ncat.org


Student perceptions on value of academic technologies

Source: EDUCAUSE http://net.educause.edu/ir/library/pdf/ERS1302/ERS1302.pdf


Are students right? Do academic technologies make a difference in learning? What evidence do we have?


Tech Enhanced

Course

Conventional Course

Traditional Experimental Design to Assess Academic Technology


But there’s overwhelming variation within the treatment

Duration of access

Overall effort (frequency) put into online materials /

activities

Specific materials accessed

Time of access

(e.g. early or late)

interactions with peers


Quick Check: How familiar are you with Learning Analytics?

a)  Unfamiliar; Never heard of it b)  Somewhat familiar; I’ve seen a reference or two c)  Very familiar; I follow the literature and/or use it

in my practice d)  Expert; I’m very knowledgeable and actively

contributing to the field


Learner Analytics “ ... measurement, collection, analysis and

reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)


No Significant Difference: Framing Academic Technology

�  [academic technologies] are “mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition”. (Clark, 1983)

Image courtesy bsabarnowl @ Flickr

Index of studies: http://www.nosignificantdifference.org/


RESEARCH FINDINGS #1: CHICO STATE COURSE: “INTRO TO RELIGIOUS STUDIES”


Intro to Religious Studies Redesign •  Redesigned to hybrid delivery through

Academy eLearning

•  Enrollment: 373 students (54% increase on largest section)

•  Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)

•  Bimodal outcomes: •  10% increased SLO mastery •  7% & 11% increase in DWF

•  Why? Can’t tell with aggregated

reporting data

0.2

.4.6

.8D

ensi

ty

0 1 2 3 4Course Grade

0

54 F’s


Variables �� !��"�� #��$�� "��%�� &��%�� &��

��'��( ��"��)��'��( ��"��)�� *��)�� )��!��)�� )��


Correlation: LMS Use w/Final Grade

Assessment Tool Use Frequency vs.

Course Grade

FC

AN

/AB

D

0 20 40 60 80 100Assess LMS Hits

95% CI Fitted valuesCourse Grade

�� Total Hits 0.48 23% 0.0000 Assessment activity hits 0.47 22% 0.0000 Content activity hits 0.41 17% 0.0000 Engagement activity hits 0.40 16% 0.0000 Administrative activity hits 0.35 12% 0.0000 Mean value all significant variables 18%


Predict the trend � Is LMS use or student characteristics a better

predictor of final grade? How much better?

a)  Student char. are 200% larger b)  Student char. are 100% larger c)  they are close to the same d)  LMS use is 100% larger e)  LMS use is 400% larger


FD

CB

A

2 2.5 3 3.5 4High School GPA

95% CI Fitted valuesCourse Grade

Statistically Significant (strong to weak) r % Variance Sign. HS GPA 0.31 9% 0.0000 URM and Pell-Eligibility Interaction -0.26 7% 0.0001 Under-Represented Minority -0.21 4% 0.0001 Enrollment Status 0.19 3% 0.0003 URM and Gender Interaction -0.15 2% 0.0033 Pell Eligible -0.15 2% 0.0045 First in Family to Attend College -0.11 1% 0.0327 Mean value all significant variables 4%

Not Statistically Significant Gender 0.10 1% 0.0557 Major-College 0.06 0% 0.2522

Correlation: Student Char. w/Final Grade

Scatterplot of HS GPA vs. Course

Grade


Chart: LMS & Student Characteristics

SSSSSSSSSSliiiiiiiddddddddes: hhhhhhhhhhhhhhhhhtttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttppppppppppppppppppppppppppppppppppppppppppppppppp:::::::::::////////////////////////////////////////////////////////////bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbiiiiiiiiiiiiiiiiiiiiiiiittttttttttt.lllly/199999999999999vvvvvvvvvYYYYYYYYYYYYYYYYYYYYYYYYYYY999999999999999aDDDDDDDDDDD


At-Risk Students: “Over-Working Gap”

22


Activities by Pell and Grade

Extra effort in content-related activities


RESEARCH FINDINGS #2: AUGMENTED ONLINE LEARNING ENVIRONMENT @ SAN JOSE STATE UNIVERSITY


SJSU+ Course Descriptions

•  3 Math Courses (Remedial,

Algebra, Stats)

•  Developed by SJSU faculty with Udacity, MOOC Provider

•  Offered with “enhanced support services”

•  Offered for credit to matriculated and non-matriculated students


Pass Rates & Historical Grade Distribution


Predict the findings – which variables will be sign. related to passing course?

Use poll to select all variables that you predict will have a sign. relationship

a.  Gender b.  URM Status c.  Online problem sets completed d.  First week logged >30 minutes online e.  First week spoke with tutor f.  High School Student g.  Time spent on videos


Variables Predicting Pass Rates

Not Significant b) URM d) 1stWkLogged30Mins f) HS Student �  Age �  Pell �  FirstGen College

Student �  MOOC sessions �  and more ….

Significant a) Gender c) Online problem sets completed e) First week spoken with tutor g) Time spent on videos �  Weeks active >30 mins �  Total Time online


Expected Net Probability of Passingby Video Hours

All, Matriculated & Stat 95 Students holding Problems Done constant at its mean (except for Stat 95)

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 50 100 150 200 250 300 350 400 450

p(Pa

ssin

g) All Students w Problems Done

Matriculated wProblems Done

Stat 95

Slides: http://bit.ly/19vY9aD 30

0

20

40

60

80

100

< Median ? Median

Pass

Fail

≥

Passed Failed

Course Pass by Problem Sets Submitted

Problems Problems

10%

44%

56% 90%

Perc

ent


RESEARCH FINDINGS #3: ADAPTIVE LEARNING @ UNC WILMINGTON


Study Context & Objectives �  Does using “timed release” feature in Blackboard Learn

effect student learning?

�  Context: EDN 303 (Upper division, inst. tech, online) –  Experimental section: used “timed release”, releasing

materials only after “go live” date (n=20) –  Control section: allowed students to access all course

material whenever they wanted (n=18)

�  Hypothesis: controlling access time helps students focus attention on relevant materials and improve learning.


Grade Distribution (or not)

Grade Frequency Percent Cum. A 20 46.51 46.51A- 9 20.93 67.44B 3 6.98 74.42B+ 3 6.98 81.4B- 2 4.65 86.05C 1 2.33 88.37C+ 2 4.65 93.02C- 1 2.33 95.35D- 1 2.33 97.67F 1 2.33 100Total 43 100

67.4474.42

81 4

When did Students Access Course? Control Course

(no adaptive release) Experimental Course

01

02

03

0P

erc

en

t

-60.00 -40.00 -20.00 0.00 20.00 40.00 60.00 80.00Total_Access_Difference

05

10

15

20

Pe

rce

nt

0.00 20.00 40.00 60.00 80.00 100.00Total_Access_Difference

•  Larger distribution time in control course •  Few students access content early in control course •  More students access near “open time” in experimental

course


Findings from Analysis � Complete model explains 78% variance in final

grade! (included current GPA). Course type is a significant variable in equation, but low effect.

� Total items accessed and course grade was significant for all students (r=.47)

� Students with higher GPA accessed materials earlier (r=-.37), login more frequently (r=.33), and got higher grades (r=.738)

� Even with small variation in “outcomes”, can learn some interesting things about students and success.


Coming Soon . . . MOOC Research

Thanks to the leadership & support of:

“Patterns of Persistence: What Engages Students in a Remedial English Writing MOOC?”

•  John Whitmer, PI •  Eva Schorring, Co-PI •  Pat Hanz, Co-PI


DASHBOARDS FOR HIGH IMPACT PRACTICES


THE FRAMEWORK Advancing by Degrees: A Framework for Increasing College Completion by Offenstein, Moore & Schulock Institute for Higher Education Leadership and Policy and The Education Trust (http://bit.ly/10QtMXC)


Milestones� Leading Indicators�•  Year-to-year Retention •  Transition to college level coursework

(English and Math) •  Earn one year of college level credits •  Complete General Education •  Complete degree

Remediation •  Begin remedial coursework in the first term, if

needed. •  Complete needed remediation

Gateway Courses •  Complete college-level math and/or English in

the first or second year •  Complete a college-success course or other

first-year experience program

Credit Accumulation and Related Academic Behaviors •  Complete high percentage of courses

attempted (low rate of course dropping and/or failure)

•  Complete 20-30 credits in the first year •  Earn summer credits •  Enroll full time •  Enroll continuously, without stop-outs •  Register on-time for courses •  Maintain adequate academic progress�


Data Dashboard

ERS Data

CCA Data

LMS Data

Other Data

Sources

Data Dashboard


EXAMPLES OF PRODUCTS (TIME PERMITTING)


Blackboard Analytics for Learn


x-Ray Course Dashboard

Social network analysis

What are course participants talking

about?


Feedback? Questions?

John Whitmer � [email protected] � Twitter: johncwhitmer

� Slides: http://bit.ly/19vY9aD


For More Information � Learning Analytics Resources Googledoc:

http://goo.gl/Fwur6

� Society for Learning Analytics Research: http://goo.gl/bH9ts

� Educause Learning Analytics Library: http://goo.gl/UDRMx


Works Cited �  Chen, Xianglei, & Ho, Phoebe. (2012). STEM in Postsecondary Education:

Entrance, Attrition, and Coursetaking Among 2003-2004 Beginning Postsecondary Students. In N. C. f. E. Statistics (Ed.), Web Tables. Washington, D.C.: Institute for Education Statistics.

�  Clark, Richard E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53, 445-459. doi: 10.3102/00346543053004445

�  Dahlstrom, Eden, Walker, J.D., & Dziuban, Charles. (2013). ECAR Study of Undergraduate Students and Information Technology, 2013. In E. C. f. A. a. Research (Ed.). Louisville, CO.

�  Firmin, Rob, Schiorring, Eva, Whitmer, John, Willett, Terrence, & Sujitparapitaya, Sutee. (2013). SJSU+ AUGMENTED ONLINE LEARNING ENVIRONMENT PILOT PROJECT (pp. 48). San Jose, California: Research and Planning Group for California Community Colleges.

�  Offenstein, Jeremy, Moore, Colleen, & Shulock, Nancy (2011). Advancing by Degrees: A Framework for Increasing College Completion.

�  Siemens, George. (2011). Learning and Academic Analytics. Retrieved from http://www.learninganalytics.net/

�  Whitmer, J., Fernandes, Kathy, & Allen, Bill. (2012). Analytics in Progress: Technology Use, Student Characteristics, and Student Achievement. Educause Review Online(July 2012).

identifying & increasing stem student engagement through ... · john whitmer, ed.d. program...

Documents