using analytics to improve student success

USING ANALYTICS TO IMPROVE STUDENT SUCCESS:

A PRIMER ON LEVERAGING DATA TO ENHANCE STUDENT PERFORMANCEMarch 23, 2014 Matthew D. Pistilli, PhD

Plan for the day Introductions and Purpose Conceptual Overview Other Institutions’ Analytics Five Components of Analytics Individual/Group Work & Planning Managing Expectations in Next Steps

Who are we?Where are we from?Why are we here?

Introductions and Purpose

DefinitionsStudent Involvement Theory:

Astin’s Inputs-Environment-Output Model

Conceptual Overview

Definitions

Definitions of Learning Analytics The measurement, collection, analysis and

reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs (SoLAR)

Evaluating large data sets to provide decision makers with information that can help determine the best course of action for an organization, with a specific goal of improving learning outcomes (EDUCAUSE, 2011)

Definitions Continued Using analytic techniques to help target

instructional, curricular, and support resources to support the achievement of specific learning goals (van Bareneveld, Arnold, & Campbell, 2012)

the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data (Cooper, 2012)

Definitions Continued Using data to inform decision-making;

leveraging data to identify students in need of academic support; and allowing direct user interaction with a tool to engage in some form of sensemaking that supports a subsequent action (Krumm, Washington, Lonn, & Teasley)

The use of data, statistical analysis, and explanatory and predictive models to gain insights and act on complex issues (Bichsel, 2012)

Common Themes

Challenge: How do you find the student at risk?

http://www.youthareawesome.com/wp-content/uploads/2010/10/wheres-waldo1.jpg

http://www.youthareawesome.com/wp-content/uploads/2010/10/wheres-waldo1.jpg

Challenge: How do you find the student at risk?

Analytics is about…

Actionable intelligence Moving research to practice Basis for design, pedagogy, self-

awareness Changing institutional culture Understanding the limitations

and risks

Inputs-Environment-Output

Student Involvement Theory

Student Involvement Theory Alexander Astin - UCLA Involvement:

The amount of physical and psychological energy that the student devotes to the

academic experience. (1985, p. 134)

Exists on a continuum, with students investing varying levels of energy

Is both quantitative and qualitative Direct relationship between student learning and

student involvement Effectiveness of policy or practice directly related to

their capacity to increase student learning(Astin, 1999)

Inputs-Environment-Output Model

Inputs

Output

Environment

Inputs The personal, background, and

educational characteristics that students bring with them to postsecondary education that can influence educational outcomes (Astin, 1984).

Inputs Astin (1993) identified 146 characteristics, including

Demographics Citizenship Ethnicity Residency Sex Socioeconomic status

High school academic achievement Standardized test scores GPA Grades in specific courses

Previous experiences & self-perceptions Reasons for attending college Expectations Perceived ability

Outcomes Basic level

Academic Achievement Retention Graduation

More abstractly Skills Behaviors Knowledge

The things we are attempting to develop in students

Environment Where we have the most control Factors related to students’ experience

while in college Astin (1993) identified 192 variables

across 8 overarching classificationsInstitutional characteristics Financial AidPeer group characteristics Major Field ChoiceFaculty characteristics Place of residenceCurriculum Student involvement

… requires a shift in thought.

All this data…

Moving from…

DataDescribes

Decides

to…

Other Institutions’ Analytics

Austin Peay University

Degree Compass

Rio Salado College

Student Support Model

Open Learning Initiative

SNAPP

UMBC Purdue University

Check My Activity

Campbell & Pistilli, 2012

Analytics 5 Component Model

Five Components of Analytic Model

Gather

Predict

ActMonitor

Refine

Components are cyclical starting with gather but can be drawn upon at any point in the cycle.

Analytic Component 1: Gather

Gather Data In multiple formats From multiple sources With insights into students & their

success That can be analyzed & manipulated into

formulae

Data is the foundation for this work, and without good data, the effort may

be for naught.

Gather Before gathering, determine what will be

gathered. What question are you trying to answer?

To do so, consider… Where will your focus be? What data do you already have (or have access to)? What else do you need to collect?

Who owns that data? What will it take to get access to it?

What are the challenges associated with assembling all the data?

What are the funding implications for data collection and assembly?

GatherUltimately, answer the following questions:1. How will you describe this analytics

area to interested parties?2. Who are the key stakeholders that need

to be included in discussions?3. Who should serve as the lead for this

area at your institution?4. What other considerations are there?

Analytic Component 2: Predict

Predict Begins with the question asked in Gather:

What do you want to predict? How do you identify this as a focus area?

Prediction models built will be driven by Types of data gathered Question being answered

What’s currently being predicted? How? By whom? In what realms? Student success? How can you involve those persons in this effort?

Predict What makes a good model? Correlation vs. Causation Expertise required

Data analysis Statistical Content

Reliability & Validity Frequency of updating Challenges & obstacles

PredictUltimately, answer the following questions:1. How will you describe this analytics

area to interested parties?2. Who are the key stakeholders that need

to be included in discussions?3. Who should serve as the lead for this

area at your institution?4. What other considerations are there?

Analytic Component 3: Act

Act Harken back to journalism class…

Who? What? Where? When? Why? How?

Add: Available resources? Timing

Act Frequency – more is always better Funding the action Assessing the impact

What are you assessing? Were behaviors changed?

How do you know? Do different actions need to be:

Taken (on your end)? Suggested (on the students’ end)?

ActUltimately, answer the following questions:1. How will you describe this analytics

area to interested parties?2. Who are the key stakeholders that need

to be included in discussions?3. Who should serve as the lead for this

area at your institution?4. What other considerations are there?

Analytic Component 4: Monitor

Monitor Formative & summative in nature Can present challenges and obstacles It’s a process

Current process must be understood New/parallel processes developed as necessary

Involving others… to some extent, the more the merrier

Availability of resource (time, money, people) Timing of monitoring Ability to react

Monitor Review

Data collected and used… was it Necessary? Correct? Sufficient?

Predictions made… were they Accurate? Meaningful?

Actions taken… were they Useful? Sustainable?

Feedback received to date

MonitorUltimately, answer the following questions:1. How will you describe this analytics

area to interested parties?2. Who are the key stakeholders that need

to be included in discussions?3. Who should serve as the lead for this

area at your institution?4. What other considerations are there?

Analytic Component 5: Refine

Refine Self-improvement process for

Analytics at the institution The institution Enrolled students

Continual monitoring Small tweaks here and there Major changes after periods of time

Updating of algorithms and statistical models Outcome data important as

Assessment Additional components for inclusion in the model

Refine What was learned from this effort?

Where are the positives? Where are the deficiencies?

Was the goal realized? How does the goal/involvement in the

project help meet institutional goals? Who else needs to be involved to

improve/enhance the process, actions, and outcomes?

How can lessons learned be applied for future use?

RefineUltimately, answer the following questions:1. How will you describe this analytics

area to interested parties?2. Who are the key stakeholders that need

to be included in discussions?3. Who should serve as the lead for this

area at your institution?4. What other considerations are there?

Elevator Speech for ProjectDetermine/solidify Institutional GoalWork on Component Templates

Individual/Group Work

What is your goal for this project?What have you learned?What are your next steps?What questions do you still have?

Institution Reporting & Town Hall

Managing Expectations in Next Steps

http://i.imgur.com/nZArTnc.jpg

Expectations Reality Plug and Play Immediate results

Solve every problem – ever!

Universal adoption

Everyone would love it!

Fits, starts, reboots Mostly long term

outcomes Solve some

problems, create some new problems

Lackluster use Not everyone loved

it

Institutional Challenges

Data in many places, “owned” by many people/organizations

Different processes, procedures, and regulations depending on data owner

Everyone can see potential, but all want something slightly different

Sustainability – “can’t you just…” Faculty participation is essential Staffing is a challenge

New Possibilities

Using data that exists on campus Taking advantages of existing programs Bringing a “complete picture” beyond

academics Focusing on the “Action” in “Actionable

Intelligence”

Contact Information Email: mdpistilli@purdue.edu Phone: 765-494-6746 Twitter: @mdpistilli –

twitter.com/mdpistilli

ReferencesAstin, A. W. (1984). Student involvement: A developmental theory for higher education.

Journal of College Student Development, 24, 297-308.Astin, A. W. (1993). What matters in college? Liberal Education, 79(4).Astin, A. W. (1994). What matters in college: Four critical years revisited. San Francisco:

Jossey-Bass.Bichsel, J. (2012, August). Analytics in higher education: Benefits, barriers, progress,

and recommendations (Research Report). Louisville, CO: EDUCAUSE Center for Applied Research. Available: http://net.educause.edu/ir/library/pdf/ERS1207/ers1207.pdf

Cooper, A. (2012). What is Analytics? Definition and Essential Characteristics. CETIS Analytics Series, 1(5). Available: http://publications.cetis.ac.uk/2012/521

EDUCAUSE Learning Initiative. (2011). 7 things you should know about first-generation learning analytics. Louisville, CO: EDUCAUSE. Available: http://www.educause.edu/library/resources/7-things-youshould-know-about-first-generation-learning-analytics

Krumm, A. E., Waddington, R. J., Lonn, S., & Teasley, S. D. (n.d.). Increasing academic success in undergraduate engineering education using learning analytics: A design based research project. Available: https://ctools.umich.edu/access/content/group/research/papers/aera2012_krumm_learning_analytics.pdf

Oblinger, D. G. and Campbell, J. P. (2007). Academic Analytics, EDUCAUSE White Paper.Society of Learning Analytics Research. (n.d.) About. [Webpage] Available:

http://www.solaresearch.org/mission/about/