measuring and assessing the value of student … · 5/7/2013 · source: j. waldman, measuring the...

STRATEGIC ENROLMENT MANAGEMENT SUMMIT 2013

CALGARY, ALBERTA

TUESDAY, MAY 7, 2013

MEASURING AND ASSESSING THE VALUE OF STUDENT ENGAGEMENT

Informing the Future of Higher Education

Richard Wiggers, Executive Director, Research & Programs & Valerie Lopes, Research Consultant

WHAT IS HEQCO?

Higher Education Quality Council of Ontario

• HEQCO’s creation was recommended in the Rae Review (2005)

• It is an independent agency with a mandate to conduct research and give policy advice to the Minister of Training, Colleges and Universities (MTCU) on all aspects of PSE

• The primary focus is to employ evidence to influence policy

3


HEQCO has completed or launched over 250 research projects

4


Published Completed Underway Total

Projects

1. Accessibility/Participation 32 12 16 60

2. Accountability/System Design 18 2 2 22

3. Learning Quality 34 28 68 130

Internal Research (Various) 32 2 6 40

Manuscripts 3 0 0 3

Opportunities to Innovate Fund (OIF) 1 1 8 10

Total 120 45 100 265

Many current projects focus on teaching, learning and student outcomes

Student Services Large Classes Technology-

Assisted Learning

Professional Development

Graduate Studies

NSSE Collegiate Learning

Assessment

AHELO Tuning

Work Integrated Learning


5

Course–Based Interventions

Student Services

Teaching

Student success is impacted by many different factors

6


STUDENT SUCCESS

More recently HEQCO developed and released a national “how to” Guide


7

• Based on a similar document developed by McMaster University

• Endorsed by the Society for Teaching and Learning in Higher Education (STLHE) and the Canadian Association of College and University Student Services (CACUSS)

• September 18, 2012 release

Researching Teaching and Student Outcomes in

Postsecondary Education:

A Guide

Susan Elgie

With:

Ruth Childs

Nancy E. Fenton

Betty Ann Levy

Valerie Lopes

Karen Szala-Meneok

Richard Dominic Wiggers

STUDENT ENGAGEMENT

University and non-university attainment for main OECD countries, percentage of population aged 25-64, 2006

23 24

0

5

10

15

20

25

30

35

Tertiary-type B education (Non-university) Tertiary-type A education (University)

Canada already leads the OECD in PSE attainment


9

Ontario is achieving the goal of “mass education” in PSE

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

25-64 55-64 45-54 35-44 25-34

Hig

he

st e

du

cati

on

al a

ttai

nm

ent

Trades cert. & registered apprenticeship Total College or CEGEP Total University

67.5

61.4

Ontario Goal


10

The postsecondary student population is now much more varied

Dee

p

Str

ateg

ic

Su

rfac

e

En

titl

emen

t

%

sample Tentative Label

Group 1 2.62 2.67 3.49 3.15 7.4% Memorizer with expectations

Group 2 2.85 2.71 2.99 2.30 15.1% Just puttin’ in time

Group 3 2.90 3.50 2.80 2.80 14.5% Strategist

Group 4 3.22 3.28 2.41 1.88 14.5% Relaxed student

Group 5 3.43 3.03 3.05 2.86 12.4% Non-strategic worker bee

Group 6 3.49 3.52 3.31 3.65 6.9% Driven to succeed

Group 7 3.70 3.49 2.58 2.50 17.3% Inquiring student

Group 8 4.07 3.86 1.99 2.06 11.8% Student scholar

Source: Schoner, V., Andrey, J., Jayasundera, R., Joakim, E. & Hambly, D.

Students’ Learning Syles and Sense of Entitlement (2011).


11

Engaged students are curious and have a passion for learning

12


Source: metapreneurship.net/wp-content/uploads/2012/02/CollegeProfessorsMeme.jpg

I found this course fascinating and I’m thrilled I had the opportunity to take it.

I would love to have had this class all year and I would recommend it to anyone!.

This has been one of the best classes I’ve taken in four years.

Disengaged students don’t appreciate high academic standards

13


Source: metapreneurship.net/wp-content/uploads/2012/02/CollegeProfessorsMeme.jpg

Really needs a life!

Out to fail you to compensate for whatever he’s lacking.

He is literally ruining my university career!

He is a total ASS.

Millennials have a different view of the value of education


14

Source: Arthur Levine, Diane R. Dean, Generation on a Tightrope: A Portrait of

Today’s College Student (2012).

1969 1976 1993 2009

The chief benefit is that it increases one’s earning power --- 44 57 67

Essential goal: to get training and skills for an occupation 59 67 70 73

1969 1976 1993 2009

Essential goal: to formulate life values and goals for my life 71 62 52 50

Essential to learn to get along with people 76 66 47 38

One HEQCO study looked at the gap between expectations and reality

Source: M. Mancuso et al., Disappointment, Misunderstanding and Expectations: A Gap

Analysis of NSSE, BCSSE and FSSE (HEQCO: 2010).

-30 -20 -10 0 10 20 30 40 50

Gains in General Education

Gains in Practical Skills

Student Success Support

Diversity

Information Technology

Course Interaction

Out-of-Class Interaction

Collaborative Learning

Active Learning

Higher-Order Thinking Skills

Writing

Course Challenge

Disappointment GapBCSSE-NSSE

Misunderstanding GapNSSE-FSSE


15

16


Helicopter parents are raising more “bubble wrap kids”

Source: R. Finnie, S. Childs and T. Qiu, Patterns of Persistence in PSE (2011);

Academica, From Postsecondary Application to the Labour Market (2010);

Government of NB, High School Follow-Up Survey (2007). 17


Reasons for Leaving PSE Ontario Rest of Canada

Didn't like it/Not for me 35.7% 30.3%

To change schools or programs 35.2% 33.7%

Health, personal or other reason 17.3% 20.5%

Not enough money 3.1% 6.7%

Wanted to work 3.1% 5.0%

Marks too low 5.6% 3.9%

More PSE students are uncertain about their direction

Quite a few students arrive in PSE unprepared academically


18

Ontario College “A”

2006 2007 2008 2009 2010

Below College English 58.7% 59.3% 59.8% 59.3% 55.0%

College English 39.4% 38.4% 38.2% 36.5% 41.0%

Total Responses (Count) 6,150 6,409 6,380 6,834 7,179

PSE isn’t even the equivalent of full-time work for most students

14.0

15.2

15.9

17.0

27.3

28.4

29.3

31.7

0 10 20 30 40

below 70

70 to 79

80 to 89

90+

Hours per Week

Ove

rall

Un

ive

rsit

y G

rad

e

Total Time at School

work

Total Time Studying

Source: Lori McElroy and Alex Usher, HESA, Measuring Differences in Academic Challenges

and Grades at Canadian Universities (PowerPoint presentation for CIRPA 2010).


19

Being “networked” today means being connected/disconnected

20


Source: W. Drexler (2008)

http://teachweb2.blogspot.com/2008/11/networked-student-revision.html

The Paradox of Choice

21


HOW DO WE MEASURE IT?

What does student “engagement” look like?

23


• Volunteering

• Hours on campus outside of class

• Curious

• Enthusiastic

• Showing up for things

• Intentional

• Actively participating

• GPA


24


• Reflective

• Doing school work

• Empowered

• Clarity in institutional pledge to students

• Actively seeking learning

• Confident

• Active alumni

• Internally motivated


25


• Institutional pride

• Depends on the student

• Classes are full throughout

• Interest

• Asking questions

• Mental and physical health

• Time and effort students devote to their studies and related activities….

What variables and factors encourage student engagement?

26


• Passionate, engaged and well prepared faculty and staff

• Mentorship

• Activities that challenge

• Timeliness

• Student involvement

• Instruction that is relevant to where students are at

• Same level of commitment, etc. that we expect of students

What variables and factors encourage student engagement?

27


• Assuring that there is a fit when students come in

• Physical space

• Students have clear goals

• Outreach to K-12 about what it is all about

• Availability of learning resources (physcial space, etc.)

• Relevant ways to deliver services

• Mandatory voluntarism in the community

What are the barriers to student engagement?

28


• Time

• Money

• Collective agreements

• Tired and uninspired faculty and staff

• Conflicting commitments

• Tired and uninspired students

What measures of student engagement are you currently using on your campus?

29


• Attendance

• NSSE

• NCHA

• Participation in clubs and student leadership

• Required to withdraw rates

• Program satisfaction

• Retention

What measures of student engagement are you currently using on your campus?

30


• Course evaluations

• Student focus groups

• Feedback forms

• Graduation rates

• Undergraduate survey

What do these measures of student engagement actually measure?

31


• Satisfaction

• Academic performance

• Activity

• Extracurricular activities

• Impact of student supports

• Persistence rates

• Number of students engaged with peers

What should we be measuring in terms of student engagement?

32


• Why are they here and what do they want to study?

• What are their expectations?

• Satisfaction in exit surveys

• Expectations of program, faculty, etc.

• Perception of their own engagement

• Measure of student engagement and learning (transformation, desire to continue to learn?)

• How many resources are going to already engaged students

What should we be measuring in terms of student engagement?

33


Students are more satisfied with their PSE experience


34

Source: Arthur Levine, Diane R. Dean, Generation on a Tightrope: A Portrait of

Today’s College Student (2012).

1969 1976 1993 2009

I am satisfied overall 62 71 79 79

I would rather be going to university than anything else 69 69 75 83

I am satisfied with teaching at my institution 67 72 81 87

1969 1976 1993 2009

My grade point average is A- or higher 7 19 26 41

My grade point average is C or less 25 13 9 5

I have taken a basic skills or remedial course --- 29 32 45

I work hard at my studies --- 86 89 93

Some innovations increase satisfaction but not academic performance

35


Source: J. Waldman, Measuring the Impact of Hybrid College Programs (Presentation to

ISSOTL, 2012)

Student Success

(A, B or C) GPA

Student

Satisfaction

Withdrawal

Rates

University of Central

Florida High None to slightly

positive effect No effect Not measured

Sheridan (FOB, FT

Faculty, PY Control) High No effect No effect

Possibly small

negative effect

Sheridan (College-

wide, many controls) Not measured

Small negative

effect No effect

Small negative

effect

Course evaluations are a standard way to measure student “satisfaction”

Source: P. Gravestock and E. Gregor-Greenleaf, University of Toronto, Student

Course Evaluations: Research, Models and Trends (HEQCO: 2008).

Course evaluations typically include questions to students about the following:

Common uses for course evaluations include the

following:

Course content Teaching improvement (faculty)

Instructor’s communication skills Personnel decisions (PSE institutions)

Student-Teacher interaction Course selection (students)

Course difficulty and workload Teaching awards

Assessment practices in the course

Student self-assessment


36

NSSE has limitations as a means to measure student engagement

37


• Too blunt for institutional comparisons, and too broad for course-based assessments

• It can be useful for program level and sub-group analyses.

• “most student subgroups are not universally engaged or disengaged, but rather show more or less engagement across items within and across benchmarks in a given program, and more or less engagement for particular items across programs”

Source: C. Conway, H. Zhao and S. Montgomery, The NSSE National Data Project

Report (HEQCO: 2011; C. Conway and H. Zhao, The NSSE National Data Project: Phase

Two Report (HEQCO: 2012).

According to the CLA, students are studying less, and learning less

Source: Richard Arum and Josipa Roksa, Academically Adrift: Limited Learning on College

Campuses (2011)).


38

• Measures critical thinking, problem solving, writing, etc.

• 45% of students demonstrated no significant improvement in learning during the first 2 years of PSE

• 36% of students demonstrated no significant improvement over 4 years of PSE

• For students who show improvement, it is modest only

• Main culprit is lack of rigor in terms of reading requirements and study time (12-13 hours/week)

WHAT CAN PSE INSTITUTIONS DO?

Source: SSRC, 2012 from Learning to Earning, 2012, presentation by Brent

Herbert-Copley, SSHRC.

Unemployment By Student Performance (Sample Of Recent US Bachelor’s Graduates)

0

2

4

6

8

10

12

CLA scores Academic Engagement/Growth

LowMediumHigh


40

Students must be reminded that jobs require more than a credential

There are a number of different ways to introduce experiential learning

1 2 3 4

*Organize class visits to local business, government, or community workplaces

*Arrange job shadowing opportunities for students

*Invite business, government, or community guest speakers into the classroom

*Assign projects that require students to interact with local business, government or community organizations

*Invite students to share career goals with class

*Provide class time for students to reflect on their work experiences

*Design academic course content (readings, discussions) to integrate with students’ work experiences

*Use authentic assessment strategies to assess students’ ability to perform real-world tasks

*Use business/community/workplace case studies for student assignments

*Invite students to share their work experiences with class

*Talk to students individually about their work experiences

*Use business/community/workplace examples to illustrate concepts in class

*Talk to students individually about their career goals/concerns

University

College

Never 1-5 times 6-10 times 11-20 times

41 Source: Peters, J., Faculty Experiences with and Perceptions of Work-Integrated

Learning (WIL) in the Ontario Postsecondary Sector (Toronto: HEQCO, 2012).


Ontario students indicated what made a teacher effective

Factor % of Students

Selected

Clearly outlines expectations at the beginning of the semester 22.4%

Students often do well in his/her course 17.2%

A prominent researcher at the cutting edge of his/her field 13.2%

Integrates technology well into the learning environment 6.9%

Source: Ontario Undergraduate Student Alliance (OUSA), Ontario Results of the

Canadian Student Survey (PowerPoint presentation, 2010).


42

Factor % of Students

Selected

Delivers interesting, well prepared and organized lectures 83.7%

Enthusiastic, entertaining or motivating in the classroom 74.6%

Able to communicate the subject in multiple ways 52.4%

Able to meet regularly with students 25.8%

Clearly outlines expectations at the beginning of the semester 22.4%

Students often do well in his/her course 17.2%

A prominent researcher at the cutting edge of his/her field 13.2%

Integrates technology well into the learning environment 6.9%

Some ideas and advice for our faculty and programs


43

Integrate technology where appropriate

Find ways to involve students in their own learning

Set and maintain reasonable but high standards

Articulate clear learning outcomes

Utilize transparent assessment practices (rubrics, etc.)

Encourage students to utilize student services and supports

Remind students of the value and relevance of learning

Thank You!


Richard Dominic Wiggers Executive Director, Research and Programs

HEQCO

(416) 212-3881

[email protected]

Valerie Lopes [email protected]

Research Consultant

mailto:[email protected]

mailto:[email protected]

measuring and assessing the value of student … · 5/7/2013 · source: j. waldman, measuring the...

Documents