Contemplating the End: Contemplating the End: How Assessment Can How Assessment Can Help with RetentionHelp with Retention

Robert MundhenkAAACL Annual Conference8 April 2009

The Age of AccountabilityThe Age of Accountability

Once upon a time, colleges and universities were like doctors. . . .

Then tuition and greater consumer awareness rose

Then higher education become a political topic

The Age of AccountabilityThe Age of AccountabilityThe Spellings Commission

◦Early stirrings: Threatened regional accreditation Proposed standardized testing Argued that colleges were failing to serve

their public responsibility—and that they cost too much

◦Results: None of the above, but---

The Age of AccountablityThe Age of AccountablityThe charges remainedHigher Education Act required

increased reporting of outcomes, especially in teacher training

State legislatures and boards become increasingly interested in outcomes

“Voluntary” System of Accountability: Enrollment and graduation data Institutional climate data “Engagement” information Standardized test data on critical thinking

The Age of AccountabilityThe Age of Accountability

….but exactly what these “outcomes” are is still very murky

Thus “success” differs state to state and institution to institution

Context: AccreditationContext: Accreditation

Original purpose was to ensure quality

Peer Review Extreme confidentialityFocus traditionally on quantity

and processes rather than quality and results

Rarely concerned with learning

Context: Accreditation and Context: Accreditation and AssessmentAssessmentMid-80’s: Accreditors began to

require some attention to assessment of learning

Mid-90’s: Focus on structures and data collection

Mid-00’s: Focus of use of information to improve outcomes

Assessment and Assessment and Accountability: Serving Two Accountability: Serving Two MastersMastersNot always aligned values

institutionally; thus not always aligned activities

Assessment has moved toward improvement of learning outcomes, which tends to appeal to faculty while nonplussing external stakeholders

Assessment and Assessment and Accountability: Serving Two Accountability: Serving Two Masters?Masters?Accountability efforts seem

aimed at external stakeholders seeking “transparency” through broader, statistically based institutional outcomes—but these efforts seem pointless and bureaucratic to faculty

Why the Shift toward Why the Shift toward Accountability?Accountability?A need to determine degrees of

successCostAccountability data are relatively

easy to collect—and have been collected and reported for years—while learning outcome data are new and not particularly comparable

Can Assessment of Can Assessment of Learning and Learning and Accountability Efforts Be Accountability Efforts Be Compatible?Compatible?They ought to beServing two audiences but

essentially the same purpose, the success of students

Aligning Assessment and Aligning Assessment and Accountability: Data Set 1Accountability: Data Set 1Public Four-Year Totals  Re-en AnnRet CumGr Success 1992 8,929 64.5 72.2 0.1 72.31993 10,296 64.5 72.4 0 72.41994 9,804 63.4 72.4 0 72.41995 9,655 64.7 73.8 0 73.81996 9,624 65.7 74.7 0.1 74.81997 9,928 65.2 74.7 0.1 74.71998 10,406 66.6 75.5 0 75.61999 10,684 66.5 75.7 0 75.82000 10,480 68.6 77.3 0 77.32001 10,643 68.6 77.2 0.1 77.32002 10,861 68.4 76.7 0.1 76.82003 11,527 67.7 75.9 0.3 76.22004 11,497 67.5 75.5 0.1 75.62005 12,042 68.5 76.6 0.2 76.82006 12,083 68.3 77.0 0.2 77.12007 12,248 67.0 75.4 0.5 75.82008 12,781

Aligning Assessment and Aligning Assessment and Accountability: Data Set 2Accountability: Data Set 2Public Two-Year Totals  Re-en AnnRet CumGr Success1992 3,885 35.9 41.4 16.0 57.41993 3,785 38.8 44.7 12.3 57.01994 3,018 41.2 46.1 10.5 56.61995 3,498 44.3 49.2 6.5 55.71996 3,718 44.5 50.3 5.6 55.91997 4,133 45.5 51.4 4.0 55.41998 4,639 46.7 53.0 4.2 57.11999 5,041 48.2 53.9 3.5 57.42000 4,688 49.2 55.4 2.9 58.32001 5,336 51.4 56.7 3.2 59.82002 5,726 49.6 55.4 2.9 58.32003 6,107 48.0 53.2 5.3 58.42004 6,059 48.8 54.0 4.9 58.92005 6,179 46.9 52.3 4.8 57.12006 5,828 48.0 53.2 4.9 58.12007 6,678 49.4 54.0 5.7 59.72008 6,594

Aligning Assessment and Aligning Assessment and Accountability: Data Set 3Accountability: Data Set 3Total Public  Re-en AnnRet CumGr Success1992 12,814 55.8 62.9 4.9 67.81993 14,081 57.6 65.0 3.3 68.31994 12,822 58.2 66.2 2.5 68.71995 13,153 59.3 67.2 1.8 69.01996 13,342 59.8 67.9 1.6 69.51997 14,061 59.4 67.8 1.2 69.11998 15,045 60.4 68.6 1.3 69.91999 15,725 60.6 68.8 1.1 69.92000 15,168 62.6 70.5 0.9 71.52001 15,979 62.9 70.3 1.1 71.52002 16,587 61.9 69.3 1.0 70.42003 17,634 60.9 68.0 2.0 70.02004 17,556 61.0 68.1 1.8 69.8

Using Retention DataUsing Retention Data

What conclusions can we draw from these data?

What kinds of decisions can we make on the basis of these data?

How can these data help us improve learning?

Aligning Assessment and Aligning Assessment and Accountability: Data Set 4Accountability: Data Set 4 Graduation Rates at Arkansas Public Universities (Source:

U.S. Department of Education College Navigator)A 38%B 39%C 33%D 34%E 20%F 33%G 33%H 58%I 18%J 44%

(Data do not include transfer-out students)

Drawing InferencesDrawing Inferences

In what ways do the graduation data qualify the retention data?

What conclusions can be drawn from these data?

A Qualification:A Qualification:Alexander Astin (“How good is your

institution’s retention rate?,” 1993): Graduation rate depends on the characteristics of the students who initially enroll

“…some institutions with ‘high’ graduation rates should really have rates that are even higher….other institutions with graduation rates that appear to be very modest are actually retaining their students at a significantly higher rate…”

Finding AnswersFinding Answers

Retention is a worthy end in itselfRaw data on retention or

graduation rates give no clues about ways to improve or increase retention and graduation rates

Assessment data and strategies may be a way to process raw data

Using Assessment to Using Assessment to Improve RetentionImprove RetentionWhat data sets or assessment

assumptions currently in place might be used to help in understanding retention information?

What data sets or assessment assumptions might be employed to improve retention?

Thinking about the End: Why Thinking about the End: Why We AssessWe AssessSelf-justification (focus on data)Improvement of instruction

(focus on teaching)Improvement of learning (focus

on outcomes)

An AssertionAn AssertionWe assess the achievement of

student learning outcomes because we want to know the degree to which we have succeeded in helping students achieve the learning goals we have set for them—and caring about learning and its improvement may actually be a way to improve retention

A Validation of the A Validation of the AssertionAssertionFrank Heppner, “On the bottom

line, good teaching tops good research,” Chronicle of Higher Education, 13 March 2009:◦“Can faculty members be trained to

be more effective teachers and so have an impact of retention? Absolutely.”

A Validation of the A Validation of the AssertionAssertionIf better teaching produces better

retention, one way to improve teaching is to provide faculty with meaningful information that tells them the degree to which students are achieving student learning outcomes

Thus better information about learning can produce better teaching, which may improve retention

What about Other Ways in What about Other Ways in Which Assessment Which Assessment Information May Have a Information May Have a Positive Effect on Positive Effect on Retention?Retention?

How Assessment Work Can How Assessment Work Can Help RetentionHelp RetentionFocus on outcomes can improve:

◦Student understanding of expectations

◦Institutional articulation of expectations

◦Syllabi and planning of experiences so that they emphasize outcomes

◦Grading and/or prioritization of grades

How Assessment Work Can How Assessment Work Can Help RetentionHelp RetentionFormative assessment can help

students understand expectations and grading strategies—and to make progress toward important outcomes over time(Mantz Yorke, “Formative Assessment and Its Relevance to Retention,” Higher Education Research & Development, 2 (2001), 115-126.

How Assessment Work Can How Assessment Work Can Help RetentionHelp RetentionConcentrating on outcomes

focuses the conversation internally and externally:

Learning becomes center of improvement efforts

Learning (not numbers) becomes center of conversations about effectiveness

How Assessment Work Can How Assessment Work Can Help RetentionHelp Retention“Drilling Down”:

Assessment data can provide detailed information about the raw retention rates:--breakdown of cohorts by various characteristics--tracking of student experience--correlating predicted retention with actual retention

How Assessment Work Can How Assessment Work Can Help RetentionHelp RetentionCorrelating institutional mission,

outcomes, and resources with graduation and retention data

--to what degree are outcomes achieved?

--have students achieved their desired objectives?

Assessment and RetentionAssessment and Retention

Assessment data cannot directly affect retention rates, but they can clarify their meaning

Assessment data and strategies can affect retention indirectly by improving student and staff understanding of their common learning goals

Assessment and RetentionAssessment and Retention

Assessment data and strategies can help institutions understand why their retention rates are where they are—and to determine how best to deal with them

For Further Information:For Further Information:

Robert T. MundhenkConsultant, Higher Education

Assessmentasmt357@aol.com