Three Decades Later: A Day With Peter Ewell

Peter T. Ewell

National Center for Higher Education Management Systems (NCHEMS)

WASC Assessment Institute

June 16, 2011

What We Will Discuss Today

History of Assessment and Some of Its Major Philosophical Dilemmas

The Policy Environment for Assessment: Who Wants What and Why?

The Accreditation Connection and Implications for the WASC Region

Topic 1History of Assessment and Some

of Its Major Philosophical Dilemmas

Foundations and Forerunners of Assessment in the United States

Scholarly Research on College Student Development and Student Flow (Retention and Graduation)

Standardized Cognitive Testing: A Uniquely American Tradition

Evaluation and “Scientific” Management in Higher Education

Mastery Learning Approaches

Origins of the “Assessment Movement” in the 1980s

Undergraduate Reform Reports of 1985-86

Internal Stimulus: Call for More Coherent Teaching/Learning Approaches and Information to Improve Them

External Stimulus: Stakeholder Demands for Information on “Return on Investment”

Tensions in Motive and Message Ever Since

Episodes and Debates

What to Call It?

“Ineffability,” “Measurement,” and “Evidence”

The “Value-Added” Debate

The “TQM” Episode

Goals 2000 and an Attempt at National Assessment

FERPA and the Federal Student Unit Record System Proposal

Why Didn’t Assessment Go Away?

Pressure to Produce Evidence of Student Learning Outcomes Never Let Up

By Early 1990s, Accreditors Replace States as Primary External Stimulus to Get Started

Intermittent Federal Interest in Assessment as an Element of National Accountability

But Resulting Faculty Ambivalence About a Process Seen as “External” and “Administrative”

Looking Back: What’s Been Accomplished?

Assessment Is for the Most Part Perceived as Inevitable and Legitimate

Vast Majority of Institutions Have Statements of Learning Outcomes (General and Programmatic) and Most are “Doing Assessment”

A “Semi-Profession” of Folks Involved in Assessment, Conferences/Workshops, and an Implementation Literature

Steadily Growing Sophistication with Respect to Methods of Gathering Evidence

Looking Back: What Hasn’t Happened?

Authentic Integration of Assessment into Faculty Cultures and Behaviors

Assessment Activities Still Largely “Added On” to the Curriculum Instead of Being Embedded In It

Systematic and Widespread Use of Assessment Results for Institutional and Curricular Improvement

National Standards or Frameworks for Benchmarking Student Achievement and Assessment Practices Across Institutions

A Basic Dichotomy of Approaches

Accountability-Based: Assessments Intended to “Check Up” on the System in the Aggregate

Scholarship and Continuous Improvement: Assessments Intended to Assure Standards and Provide Feedback on Collective Performance for Improvement

Continuous Continuous ImprovementImprovement

AccountabilityAccountability

Strategic dimensions

Purpose Formative (improvement) Summative (judgment)

Orientation Internal External

Motivation Engagement Compliance

Implementation

Instrumentation Multiple/triangulation Standardized

Nature of evidence Quantitative and qualitative Quantitative

Reference points Over time, comparative, established goal

Comparative or fixed standard

Communication of results

Multiple internal channels Public communication, media

Use of results Multiple feedback loops Reporting

Some Details of the Dichotomy

Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

A Basic Dichotomy of Approaches

Accountability-Based: Assessments Intended to “Check Up” on the System in the Aggregate

Scholarship and Continuous Improvement: Assessments Intended to Assure Standards and Provide Feedback on Collective Performance for Improvement

Can Be Applied at Any Level of Analysis

A Taxonomy of Approaches to Assessment

Learning/Teaching (Formative)

Instruction

Individual TestsPortfolios

Placement

Diagnostic TestsAdvanced Placement Tests

Vocational Tests

Accountability (Summative)

“Gate-keeping”

Rising Junior ExamsComprehensive Exams

Certification Exams Capstone Performances

Program Enhancement

Individual assessment results may be aggregated to serve program evaluation purposes

Campus and ProgramEvaluation

Productivity ReviewsPerformance

Indicators

Individual

Group

As

ses

sm

en

t L

ev

elAssessment Focus or Purpose

A Cross-Cutting Dichotomy

Exo-Skeletal: Assessments Added onto Instruction as a Distinct Set of Entities

Embedded: Assessments Built into Instruction through the Regular Process of Grading and Assignments

Looking at the Dichotomies and Your Own Assessment Program

Where Would You Place Your Own Program on the Continuum of Each of the Two Dichotomies?

• External vs. Internal Focus or Direction

• Exo-skeletal vs. Embedded Assessment Approaches

In an Ideal World, Where Would You Like It To Be?

List a Couple of Steps that You Might Take in the Next Three Months to Move Your Program in the Desired Direction

Topic 2The Policy Environment for

Assessment: Who Wants What and Why?

Assessment and Public Policy

The Current Policy Imperative for Higher Education

The Policy Players: Their Motives and Basic Approaches to Assessment

Enduring Policy Issues and How They Are Typically Addressed

The Policy Imperative for Higher Education

Global Competitiveness in Collegiate Attainment

Differences in College Attainment (Associate & Higher) Between Younger & Older Adults—U.S. & OECD Countries, 2008

Source: Organisation for Economic Co-operation and Development (OECD), Education at a Glance 2010

0

20

40

60

Korea

Canada

JapanN

ew Z

ealandN

orway

IrelandD

enmark

Belgium

Australia

United S

tatesS

weden

France

Netherlands

Spain

Luxembourg

Sw

itzerlandU

nited Kingdom

Finland

Chile

IcelandP

olandG

reeceH

ungaryG

ermany

Portugal

ItalyM

exicoA

ustriaS

lovak Republic

Czech R

epublicT

urkey

25 to 34

45 to 54

slide 19

The Policy Imperative for Higher Education

Global Competitiveness in Collegiate Attainment

The “New Majority” and Demographic Achievement Gaps

Source: U.S. Census Bureau’s 2005 American Community Survey; OECD

U.S . E duc ational Attainm ent of Y oung Workforc e (Ag e 25-34) Indexed to the Mos t E duc ated C ountry, 2007

0.78

0.38

0.25 0.27

1.53

0.72

0.390.27 0.29

1.23

0.97

0.53

0.37 0.32

1.53

0.89

0.55

0.39 0.39

1.25

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

White B lack His panic NativeAmerican,AK Native

As ian/P ac ificIs lander

White B lack His panic NativeAmerican,AK Native

As ian/P ac ificIs lander

Male F emale

S ources : U .S . C ens us B ureau, 2007 Am erican C om m unity S urvey P UMS F ile. OE C D .

Norway C anada

B ac helor's Deg ree

As s oc iate's Deg ree

The Policy Imperative for Higher Education

Global Competitiveness in Collegiate Attainment

The “New Majority” and Demographic Achievement Gaps

Questionable Levels of Graduate Achievement

Prose Literacy Levels for College Level Populations

2

4

1

2

3

1

16

20

11

10

14

6

58

56

65

49

53

56

23

19

23

40

31

38

0 10 20 30 40 50 60 70 80 90 100

Associate Degree Holders 1992 (NALS)

Associate Degree Holders 2003 (NAAL)

Current 2-Year College Students (NSACS)

Bachelors Degree Holders 1992 (NALS)

Bachelors Degree Holders 2003 (NAAL)

Current 4-Year College Students (NSACS)

Below Basic Basic Intermediate Proficient

The Policy Imperative for Higher Education

Global Competitiveness in Collegiate Attainment

The “New Majority” and Demographic Achievement Gaps

Questionable Levels of Graduate Achievement

In an Environment of Continuing Fiscal Strain

Long-term Federal DebtFederal Debt as percent of GDP

0.0

50.0

100.0

150.0

200.0

250.0

300.0

1960 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 2020 2025 2030 2035 2040 2045 2050The 2008 CBO projection is the average of its "high" and "low" alternative estimates. The OMB projection was completed before the Administration had fully formulated its health reform proposals, and assumes no relief from the historic "excess" growth in health care costs.

OMB projection of March 2010

Recession and Stimulus: 2008-12

CBO Projection of December 2008

Recent Budget actions and proposals, together with the lasting effects of the recession on the size of the federal debt, have suddenly pushed the onset of serious debt sustainability problems twenty years closer.

The Policy Imperative for Higher Education

Global Competitiveness in Collegiate Attainment

The “New Majority” and Demographic Achievement Gaps

Questionable Levels of Graduate Achievement

In an Environment of Continuing Fiscal Strain

These are Now Urgent and It’s Not Just Our Conversation Any More

Assessment Policy: The Players

States

The Federal Government

Institutional Accrediting Organizations

“Third-Party” Actors

State Policy Approaches to Assessment Regulating Student Flow (SD “Rising Junior”)

Assessing Institutional Performance (WV CLA)

Performance Funding (TN Schedule)

“Institution-Centered” Mandate (VA)

Aligning Standards (“Tuning USA”)

Technical Assistance (WV Accreditation Review)

Assessing “Educational Capital”

Federal Interest in Assessment

Centered on Institutional “Stewardship” of Financial Aid Funds

Action Mostly Indirect Through Accreditation

Focus on Institutional Reporting for Consumer Information and Protection

Occasional Interest in National Testing

• Goals 2000

• “NAEP for College”

The Spellings Commission and Its Accountability Aftermath

Buildup: The Reports of 2005-2007

The Assault on Accreditation Around Learning Outcomes

• Spellings Testimony and Discussion

• “NegReg” and NACIQI

The Academy Responds

• The VSA and its Cousins

• The “New Leadership Alliance for Student Learning and Accountability”

Some “Third Party” Actors in the Quality Review Arena

U.S. News and World Report

Measuring Up and the Pew Forum on Undergraduate Learning

National Survey of Student Engagement (NSSE) and Community College Survey of Student Engagement (CCSSE)

National Student Clearinghouse

ETS Culture of Evidence Reviews

Enduring Issues of Assessment Policy

Balancing Accountability and Improvement

Standardized or Non-Comparable Measures

Absolute Achievement or “Value-Added”

Balancing Incentives and Consequences

Accounting for Differences in Context that May Affect Performance

Nature and Extent of Public Reporting

Looking at Your Own Approach in the Light of Two of These Issues…

Do You Look at “Value Added?”

• If You Do, Why Do You Do It?

• If You Don’t, Should You?

In Doing Program-Level Assessment, How Do You Handle Differing Circumstances Across Departments (e.g. Size, Student Preparation Levels, Demographics, etc.)?

Topic 3The Accreditation Connection and

Implications for the WASC Region

Accreditation and Assessment

The Evolution of Accreditation: Regional and Specialized

Taking Stock of Accreditation: Strengths and Challenges

Recent Changes in Accreditation

Potential Implications of these Changes for Assessment Practice and How to Address Them

Evolution of Regional Accreditation

Original Stimulus: “What Is a College?”

Original Focus on Resources and Processes Judged by Peer Evaluators

Mission-Centered Standards and Review (the “Golden Age”)

Second GI Bill and the Adoption of the Federal “Gatekeeper” Role

Mandatory Federal Focus on Student Learning Outcomes

Evolution of Specialized Accreditation

Original Stimulus: Flexner Report and the Rise of Professional Licensure and Identity

Steady Proliferation in Numbers to the Current Total of 61 Specialized Accreditors

Historically More Attention to Student Academic Performance than Regionals, Usually Through Performance on Licensure Examinations

Specific Attributes of a Graduate to be Assessed (e.g. ABET and AACSB International)

Taking Stock of Regional Accreditation

Strengths

• Widely Accepted “Signal” of Quality

• Opportunity for Self-Improvement

• Sharing Practices Through Mutual Visitation

Challenges

• Providing Information to the Public

• Consistency Across Reviews

• “All or Nothing” Outcomes

• Perceived Inefficiency and Institutional Burden

Accreditation and Assessment: The Current Situation with the Regionals

Steady Increase in Prominence of Assessment But Reluctance to Actually Sanction Institutions

Institutions Free to Choose Learning Goals and Ways to Gather Assessment Evidence

Focus on the Assessment Process, Not the Actual Results of Assessment

Focus on Institutional Transparency in Reporting the Results of Assessment

Trying Constantly to Increase Institutional Capacity

Recent Changes in Accreditation Practice (which WASC Pioneered)

Forces Driving Change in Accreditation

• Pressure from the DOE and Congress

• Demands from Institutions to “Add Value”

• Need to Respond to New Ways of Teaching

Resulting “New Looks” in Accreditation Practice

• Focus on Outcomes and Explicit Standards of Performance

• Presenting Evidence [e.g. “Institutional Portfolios”]

• Review Approaches [e.g. “Academic Audits”]

Potential New Developments for Assessment in Accreditation Context

Focus Review on Quality of Actual Learning Outcomes, Not Just the Adequacy of the Assessment Process

Look for External Benchmarks and Points of Comparison for Assessment Results: How Do You Know They Are Good Enough?

Focus More on the Use and Application of Assessment Results

Find Better Ways to Communicate Assessment Results (and Improvements) to Stakeholders

Focus on Assessment Results

What Are Particular Areas of Strength or Weakness

• Across Particular Student Populations?

• Across Particular Dimensions of Performance?

Connect Datasets to See What Experiences Drive Particular Outcomes (e.g. NSSE to CLA, Experience Inventory to Portfolio, etc.)

Be Prepared for a Widely-Participatory Conversation About These Things

Types of External Benchmarks

Nationally-Normed Assessments and Surveys (and Remember that Peer Comparisons are Best)

Industry Standards on Licensure Examinations

Consortia of Institutions with Similar Assessment Processes (e.g. Using Electronic Portfolios or the AACUI VALUE Rubrics)

External Reviews of Assessments and Assessment Results (e.g. External Examiners, Program Reviewers)

Using Assessment Results

Expectation Exercises

Package Results Around Real Problems

Link Assessment to Regularly-Occurring Processes and Decisions (e.g. Program Review, Budget Hearings)

Create Collective Opportunities to Review Assessment Results and Reflect on What They Mean

Avoid the “Perfect Data Fallacy”

Communicating Assessment Results

Websites and “Institutional Portfolios” [NILOA Website Review Tool]

Dashboards and Interactive Data Sites

Present Proposed Solutions and Improvements Together With Assessment Results

Create a Format that Shows Long-Term Trends and is Consistent Over Time

Keep It Simple

Looking at Your Own Assessment Program in the Light of Accreditation

How Does What WASC Requires with Respect to Assessment Compare with What Other Regions Require? [Draw on Your Homework]

What are the Implications for Your Own Assessment Program?

How Far Should WASC Move Along these Possible Paths? How Far Can It?