Models for Future Comparative Measurement of Higher Education Learning:

Lessons from the Collegiate Learning Assessment Longitudinal Study in the U.S.*

Richard Arum

New York University and

Social Science Research Council

* Josipa Roksa (University of Virginia) and Melissa Velez (NYU) collaborated on research findings presented here. We thank Ford and Lumina Foundations for their generous financial support and the Council for Aid to Education for assistance with data collection.

College Learning in the Spotlight (U.S. Policy Context)

“As other nations rapidly improve their higher education systems, we are disturbed by evidence that the quality of student learning at U.S. colleges and universities is inadequate, and in some cases, declining.”

A Test of LeadershipU.S. Secretary of Education’s Commissionon the Future of Higher Education (2006)

College Learning in the Spotlight (U.S. Policy Context)

“These shortcomings have real-world consequences. Employers report repeatedly that many new graduates they hire are not prepared to work, lacking the critical thinking, writing and problem-solving skills needed in today’s workplaces.”

A Test of LeadershipU.S. Secretary of Education’s Commissionon the Future of Higher Education (2006)

Measurement of Learning in U.S. Higher Education

Dearth of direct measures of higher education student learning that are comparable across institutions and/or states

Measuring Up 2008 – Assigned a grade of Incomplete to all states in the area of measuring learning: “All states receive an ‘incomplete’ in learning because there are not sufficient data to allow meaningful state-by-state comparisons.”

Measurement Challenges

Curriculum varies widely across fields of study and institutions – little consensus on what is to be learned

Practitioner resistance to “reductionist” approaches

Students are sorted by ability and other factors into different institutions

Collegiate Learning Assessment (CLA)

Dimensions of learning assessed critical thinking, analytical reasoning, and written

communication

Distinguishing characteristics Direct measures (as opposed to student reports) NOT multiple choice Holistic assessment based on open-ended prompts

representing “real-world” scenarios

Collegiate Learning Assessment (CLA)

Components Performance task Make an argument Break an argument

Performance Task (example)

You are the assistant to Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech’s sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235.

Performance Task (example, cont.)

Students are provided with a set of materials (e.g. newspaper articles, Federal Accident Report, e-mail exchanges, description and performance characteristics of AirSwift 235 and another model, etc.) and asked to prepare a memo that addresses several questions, including what data support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups, what other factors may have contributed to the accident and should be taken into account, and their overall recommendation about whether or not DynaTech should purchase the plane.

Determinants of College Learning Dataset

Longitudinal Design Fall 2005 and Spring 2007 (beginning of freshman and end

of sophomore years)

Large Scale 24 diverse four-year institutions; 2,341 students

Breath of Information Family background and high school information,

college experiences and contexts, college transcripts Collegiate Learning Assessment (CLA)

Sample Characteristics: Who are These Students?

CLA Analysis Sample

IPEDS – CLA Schools

IPEDS – All Schools

Demographics

Male 0.37 0.46 0.45

White 0.59 0.61 0.59

African-American 0.19 0.14 0.13

Hispanic 0.05 0.08 0.13

Asian 0.11 0.10 0.06

Test Scores

SAT, 25th percentile 1052.83 995.15 993.14

SAT, 75th percentile 1212.83 1219.02 1219.23

ACT, 25th percentile 22.05 20.86 20.33

ACT, 75th percentile 26.29 25.77 25.31

Research Questions

What individual, social and institutional factors are associated with learning in higher education?

How do disadvantaged groups of students fare in college in terms of measured learning?

To what extent do individual, social and institutional factors account for variation across disadvantaged groups?

Overview of the Conceptual Model

Employed in the Study

Measures of Disadvantage:

Race/Ethnicity Parental Education Parental Occupation

Racially Segregated High School (70+ % minority) Non-English Language

Control Variables:

2005 Test Score Gender Two Parent Household

Sibling Number Urbanicity Geographic Region

High School Academic Preparation:

GPA Number of AP Courses Taken

College Experiences:

Hours Spent Studying Alone Hours Spent Studying with Peers

Hours Spent in a Fraternity/Sorority Hours Worked On/Off Campus

Faculty Expectations Field of Study

College Fixed Effects

2007 Test Score

Analysis - Part I

Individual, Social and Institutional Factors Associated with Learning as Measured by Improvement in CLA Performance

High School Preparation

Figure 1. Predicted 2007 Test Score by Number of High School AP Courses

1110

1120

1130

1140

1150

1160

1170

1180

1190

1200

1210

1220

0 1 2 3 4 5+

Number of AP Courses

Test S

core

College Engagement and Learning

Figure 2. Predicted 2007 Score by College Engagement and Involvement Measures

1020

1040

1060

1080

1100

1120

1140

1160

1180

1200

0 5 10 15 20

Hours

Test S

core

Studying alone Studying w /peers Fraternity/sorority

College Employment and Learning

Figure 3. Predicted 2007 Test Score by Employment Measures

1110

1120

1130

1140

1150

1160

1170

1180

1190

0 5 10 15 20

Hours

Test S

core

On campus Off campus

Faculty Expectations and Learning

Figure 4. Predicted 2007 Test Score by Level of Faculty Expectations

1060

1080

1100

1120

1140

1160

1180

1200

1 2 3 4 5 6 7

Faculty Expectations

Test S

core

Fields of Study and Learning

Figure 5. Predicted 2007 Test Score by College Major

1100

1125

1150

1175

1200

Business Education andhuman services

Engineering,agriculture and

computerscience

Communications Health services Social sciencesand humanities

Science andmath

Other

Test

Scor

e

Analysis - Part II

Social Disadvantaged Group Differences in Learning as Measured by Improvement in CLA Performance

CLA Performance by Race

Figure 6. 2005 and 2007 Test Scores by Race

Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

950

1000

1050

1100

1150

1200

1250

1300

White African American Hispanic Asian

Race

Test

Sco

re

2005 Test Score 2007 Test Score

CLA Performance by Parental Education

Figure 7. 2005 and 2007 Test Scores by Parental Education

Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

950

1000

1050

1100

1150

1200

1250

1300

HS or Less Some College Bachelor'sDegree

Graduate orProfessional

Degree

Parental Education

Test

Sco

re

2005 Test Score 2007 Test Score

CLA Performance by High School Student Composition and Home Language

Figure 8. 2005 and 2007 Test Scores by Level of High School Student Composition and Home Language

Note: average growth=34.32; standard deviation=188 (Fall 05), 211 (Spring 07)

950

1000

1050

1100

1150

1200

1250

1300

High school<70% minority

High school70+% minority

English HomeLanguage

Non-EnglishHome Language

Level of High School Segregation and Home Language

Tes

t S

core

2005 Test Score 2007 Test Score

Analysis - Part III

Accounting for Variation in CLA Performance by Social Disadvantaged Groups

Accounting for Group Differences: H.S.-College Experiences and Institutional Differences

Figure 11. Test score gaps in baseline and full models with college institutional fixed effects.

Note: Baseline regression model predicts the 2007 score, controlling for the 2005 score and a range of background characteristics. Full model also includes measures of high school academic preparation and college experiences. Non-significant differences are shaded.

-120 -100 -80 -60 -40 -20 0

African American vs. White

Parental Education HS or Less vs. ParentalEducation More than BA

High school 70+ %minority vs. <70% minority

Non-English Home Language vs. English HomeLanguage

Test Score Gap

Baseline Full Model + Fixed Effects

Conclusions and Implications

Policy makers need to focus attention on improving individual student learning in higher education, not just access and retention.

Practitioners need to recognize the extent to which both student experiences as well as institutional differences are associated with variation in learning.

Additional systematic longitudinal research is necessary to improve understanding of these processes.

Measurement of learning across fields and institutions is possible with instruments such as the CLA.