using learner analytics to understand student achievement in a large enrollment hybrid course
DESCRIPTION
Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course. John Whitmer, Ed.D. & Kathy Fernandes Academic Technology Services California State University, Office of the Chancellor CATS 2013, Sonoma State University February 19, 2013. Outline. Context - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/1.jpg)
John Whitmer, Ed.D. & Kathy FernandesAcademic Technology Services
California State University, Office of the Chancellor
CATS 2013, Sonoma State UniversityFebruary 19, 2013
Using Learner Analytics to Understand Student Achievement in
a Large Enrollment Hybrid Course
![Page 2: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/2.jpg)
Outline1. Context
2. Learning Analytics, Methods & Tools
3. Findings
4. Conclusions & Next Steps
5. Becoming a Data Scientist (pending time & interest)
![Page 3: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/3.jpg)
Why are you here? 1. To increase my knowledge of “Learning
Analytics”
2. To get ideas about how I can become a “Data Scientist” (or practice analytics)
3. To learn about the results of this study
4. Because this was the closest room
![Page 4: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/4.jpg)
1. CONTEXT
![Page 5: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/5.jpg)
Case Study: Intro to Religious Studies• Undergraduate, introductory, high demand
• Redesigned to hybrid delivery format through “academy eLearning program”
• Enrollment: 373 students (54% increase on largest section)
• Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits)
• Bimodal outcomes:• 10% increase on final exam• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data
54 F’s
![Page 6: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/6.jpg)
Guiding Questions1. How is student LMS use related to academic
achievement in a single course section?
2. How does that finding compare to the relationship of achievement with traditional student characteristic variables?
3. How are these relationships different for “at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most useful to answer these questions?
![Page 7: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/7.jpg)
It takes a village … Chico “Graduation Initiative” endorsement Former CIO (Bill Post included in project portfolio) Registrar approval
Active participation: – Kathy Fernandes, Director Academic Technology– William A. Allen, Director of Institutional Research– Scott Kodai, Manager Distributed Learning
Technologies and Classroom Services
![Page 8: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/8.jpg)
Predict the trend: Grade w/Mean LMS Hits
What will the trend look like?
![Page 9: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/9.jpg)
Strong Trend: Grade w/Mean LMS Hits
![Page 10: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/10.jpg)
2. LEARNER ANALYTICS, METHODS & TOOLS
![Page 11: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/11.jpg)
Learner Analytics
“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
![Page 12: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/12.jpg)
How? Knowledge & Skills Involved
http://www.drewconway.com/zia/?p=2378
![Page 13: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/13.jpg)
Data Extracts & Sources LMS Data
– SQL queries on WebCT Vista web server log file
Student Data– Query against Peoplesoft data– ERSS data elements (common to CSU)– Required Institutional Research
Course Data– Query against Peoplesoft data
![Page 14: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/14.jpg)
Tools Used
App Function
Excel Early data exploration; simple sorting; tables for print/publication
Tableau Complex data summaries and explorations; complex charts; presentation chartsFinal/formal descriptive data; statistical analysis; some charts (scatterplots)Statistical analysis (factor analysis)
![Page 15: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/15.jpg)
Methods at a Glance1. Clean/filter/transform/reduce data (70% effort)2. Descriptive / exploratory analysis (20% effort)3. Statistical analysis (10% effort)
– Factor analysis– Correlation single variables– Regression multiple variables; partial & complete
![Page 16: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/16.jpg)
Final data set: 72,000 records (-73%)
![Page 17: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/17.jpg)
Variables
![Page 18: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/18.jpg)
3. FINDINGS
![Page 19: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/19.jpg)
Correlation: Student Char. w/Final Grade
Scatterplot of HS GPA vs. Course
Grade
![Page 20: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/20.jpg)
LMS Use Variables
Administrative Activities (calendar, announcements)
Assessment Activities (quiz, homework, assignments, grade center)
Content Activities (web hits, PDF, content pages)
Engagement Activities (discussion, mail)
Student Characteristic Variables Enrollment Status First in Family to Attend
College Gender HS GPA Major-College Pell Eligible URM and Pell-Eligibility
Interaction Under-Represented
Minority URM and Gender
Interaction
![Page 21: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/21.jpg)
Predict the trend LMS use and final grade is _______ compared to
student characteristics and final grade:
a) 50% smallerb) 25% smallerc) the samed) 200% largere) 400% larger
![Page 22: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/22.jpg)
Predict the trend LMS use and final grade is _______ compared to
student characteristics and final grade:
a) 50% smallerb) 25% smallerc) the samed) 200% largere) 400% larger
![Page 23: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/23.jpg)
Correlation LMS Use w/Final Grade
Scatterplot of Assessment Activity
Hits vs. Course Grade
![Page 24: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/24.jpg)
Conclusion: LMS Use Variables better Predictors than Student Characteristics
LMS Use
Variables
18% Average(r = 0.35–0.48)
Explanation of change in final grade
Student Characteristic
Variables
4% Average(r = -0.11–0.31)
Explanation of change in final grade
>
![Page 25: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/25.jpg)
Smallest LMS Use Variable
(Administrative Activities)
r = 0.35
Largest Student
Characteristic
(HS GPA)
r = 0.31
>
![Page 26: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/26.jpg)
Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables
LMS Use
Variables
25% (r2=0.25)
Explanation of change in final grade
Student Characteristic
Variables
+10%(r2=0.35)
Explanation of change in final grade
>
![Page 27: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/27.jpg)
Predict the trend LMS use and final grade is ______ for “at-risk”*
students?a) 50% smaller than not at-risk studentsb) 20% smaller than not at-risk studentsc) No difference than not at-risk studentsd) 20% larger than not at-risk studentse) 100% larger than not at-risk students
*at-risk = BOTH under-represented minority and Pell-eligible
![Page 28: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/28.jpg)
Predict the trend LMS use and final grade is ______ for “at-risk”*
students?a) 50% smaller than not at-risk studentsb) 20% smaller than not at-risk studentsc) No difference than not at-risk studentsd) 20% larger than not at-risk studentse) 100% larger than not at-risk students
*at-risk = BOTH under-represented minority and Pell-eligible
![Page 29: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/29.jpg)
At-Risk Students: “Over-Working Gap”
33
![Page 30: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/30.jpg)
Question 3 Results:Regression by “At Risk” Population Subsamples
![Page 31: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/31.jpg)
WHAT CAUSED THAT DIFFERENCE?
![Page 32: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/32.jpg)
Activities by Pell and Gradegrade / pelleligible
A B+ C C-
Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible0K
5K
10K
15K
20K
25K
30K
35K
Value
Content
Content
Engage
Engage
Assess
Assess
Admin
Admin
Content
Content
Engage
Engage
Assess
Assess
Admin
Content
Content
Engage
Engage
Assess
Assess
Content
Content Engage
Engage
Assess
Assess
Admin
Admin
Measure NamesAdmin
Assess
Engage
Content
Extra effort in content-related activities
![Page 33: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/33.jpg)
Previous Studies Relating LMS Use to Course Grade
![Page 34: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/34.jpg)
4. CONCLUSIONS & NEXT STEPS
![Page 35: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/35.jpg)
Conclusions1. At the course level, LMS use better predictor of
academic achievement than student demographics (what do, not who are).
2. Small strength magnitude of complete model demonstrates relevance of data, but suggests that better methods could produce stronger results.
3. LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.
![Page 36: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/36.jpg)
More Conclusions4. LMS use frequency is a proxy for effort. Not a
very complex indicator.
5. Student demographic measures need revision for utility in Postmodern era (importance to student, more frequent sampling, etc.).
6. LMS effectiveness for at-risk students may be caused by non-technical barriers. Need additional research!
![Page 37: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/37.jpg)
CSU Learner Analytics Collaborative Projects
Moodle: mCURL (Moodle Common Use Reporting &
Learning Analytics)
8 CSU & 2 UC Campuses
Building common SQL queries for accurate & comparative use metrics
2 rounds of data collection already completed and discussed
Blackboard: Analytics for Learn Pilot
3 CSU Campuses
Bb Learn Analytics product available “off the shelf”; defined and integrated with Peoplesoft
Building common queries & campus-specific reporting
![Page 38: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/38.jpg)
5. SO YOU WANNA BE A DATA SCIENTIST?
![Page 39: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/39.jpg)
Choose your domain(s)
http://www.drewconway.com/zia/?p=2378
![Page 40: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/40.jpg)
Potential Resources Math/Stats: campus classes
MOOC resources (search “Data”, “Statistics”, or “Analytics”)
Society for Learning Analytics Research (http://www.solaresearch.org/)
Learning Analytics Resources (http://johnwhitmer.net/resources/)
![Page 41: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/41.jpg)
Feedback? Questions?
Kathy [email protected]
John Whitmer [email protected]
Complete monographhttp://bit.ly/15ijySP
Twitter: johncwhitmer
![Page 42: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/42.jpg)
BONUS SLIDES
![Page 43: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/43.jpg)
Missing Data On Critical Indicators
![Page 44: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/44.jpg)
LMS Use Consistent across Categories
Factor Analysis of LMS Use Categories
![Page 45: Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course](https://reader036.vdocuments.site/reader036/viewer/2022070502/568132c4550346895d998a4e/html5/thumbnails/45.jpg)
Ideas & FeedbackPotential for improved LMS analysis methods: time series analysis social learning activity patterns discourse content analysis
Group students by broader identity, with unique variables: Continuing student (Current college GPA, URM, etc. First-time freshman (HS GPA, SAT/Act, etc)