john hayek jillian kinzie george kuh james moran sheeo higher education policy conference august 13,...
DESCRIPTION
Assessment for Improvement and Accountability: Examples from the States. John Hayek Jillian Kinzie George Kuh James Moran SHEEO Higher Education Policy Conference August 13, 2009 Denver Colorado. Context. Global Competitiveness in Degree Attainment The New Majority and Demographic Gaps - PowerPoint PPT PresentationTRANSCRIPT
John HayekJillian KinzieGeorge Kuh
James Moran
SHEEO Higher Education Policy Conference
August 13, 2009Denver Colorado
Assessment for Improvement and Accountability:
Examples from the States
Context Global Competitiveness in Degree
Attainment The New Majority and Demographic
Gaps Questionable Levels of Student
Performance In an Environment of Increasing Fiscal
Strain… We Need Higher Levels of Student
Achievement at an Affordable Price
Overview
Purposes of assessment First lessons from NILOA Examples of NSSE use
Advance OrganizersWhat are the most important
SLOs to measure in your state? What is being done to assess
these outcomes?What are the major obstacles or
challenges to SLO assessment?What do you need to further SLO
assessment in your state?
Continuous Improvement
Accountability
Strategic dimensions Purpose Formative (improvement) Summative (judgment)
Orientation Internal External
Motivation Engagement Compliance
Implementation Instrumentation Multiple/triangulation Standardized
Nature of evidence Quantitative and qualitative Quantitative
Reference points Over time, comparative, established goal
Comparative or fixed standard
Communication of results
Multiple internal channels Public communication, media
Use of results Multiple feedback loops Reporting
Two Paradigms of Assessment
Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.
NOLOA
Far too little is known about assessment practices on campuses around the country
NILOA’s mission is to document SLO assessment work, identify and disseminate best practices, and support institutions in their assessment efforts
www.learningoutcomesassessment.org
NOLOAFUNDERS
Lumina Foundation for EducationCarnegie Corporation of New YorkThe Teagle Foundation
NATIONAL ADVISORY PANEL
NILOA ActivitiesWeb scans: 700 institutionsInterviews & focus groups with key
actors: -- AAC&U -- ACE -- AIR
“Go to” Web site being developed (www.learningoutcomesassessment.org)
Case studiesWhite papers
NILOA White PapersBanta et al.: Authentic assessment
approachesBorden: “Measuring Quality II”Ewell: The improvement and
accountability purposes of assessment
Lenth & Hill: Assessment and state policy
Wellman: Assessment results and resource allocation
NILOA ActivitiesWeb scans: 700 institutionsInterviews & focus groups with key
actors“Go to” Web site being developedCase studiesWhite papersSurvey of institutional assessment
practices
NILOA 2009 Provost SurveyAll accredited, undergraduate
degree-granting 2- and 4-year public, private, and for-profit institutions in the US (n=2809)
53% response rate (n=1518)
Tentative ConclusionsPerhaps more assessment underway than
some acknowledge or wish to believeMore activity at the department/unit level
than institution levelAccreditation is a major force shaping
assessmentMore attention needed to using and
reporting assessment resultsInvolving faculty is a major challengeMore investment likely needed to move
from data to improvement
Genera
l Kno
wledge
Specia
lized
Kno
wledge
Additio
nal A
sses
smen
ts
Externa
l Jud
ges
Nation
al Surv
eys
Loca
l Surv
eys
Rubric
s
Portfol
ios
Studen
t Inter
views
Alumni
Survey
s
Alumni
Interv
iews
Employe
r Surv
eys
Employe
r Inte
rview
s0
20
40
60
80
100
Valid Institutional Samples
Examples from the States
South DakotaTexasTennesseeKentucky Pennsylvania
NSSE State and University System Participation 2000-
2009• California State
Universities• City University of
New York• Concordia
Universities• Connecticut State
Universities• Kentucky Council
on Postsecondary Education
• New Jersey Public Univ.
• North Dakota University System
• Ontario Universities• Penn State System• Pennsylvania
System of Higher Education
• South Dakota Public Universities
• State University of New York
• Tennessee Publics• Texas A&M System
• Texas Six• University System
of Georgia• West Virginia State
System• University Systems
of Hawaii, MaineMaryland, Massachusetts, Missouri, New Hampshire, North Carolina, Texas, Wisconsin
State System Use of NSSE Institutional improvement Research Accountability & public reporting Performance funding Decision-making models More specifically….
track student engagement overtime, set engagement performance goals, explore engagement patterns by student
characteristics merge NSSE data with other system data to
examine factors associated with student success
Examples: Student Engagement Data in State SystemsAssessment and Research Use data to examine student experience and in predictive models for retention, degree attainment
Institutional Improvement Data for institutional improvement initiatives, and to encourage collaboration among system & campuses to address common challenges
Accountability Monitor and demonstrate quality in undergraduate education
South Dakota Board of Regents Data from 5 NSSE administrations
since 2002 for all public institutions Longitudinal analysis shows upward
trend in student engagement
South Dakota Board of RegentsResearch project findings: Strong links between student
performance and student-faculty interaction
Student effort in-and-outside of class correlated with other Regents Assessments of general education goals, Collegiate Assessment of Academic Proficiency (CAAP)
Academic performance Critical thinking PersistenceSo. Dakota School of Mines & Technology
South Dakota Board of RegentsFindings Influenced Improvements and Policies:Expansion of universities’ research
capacity to foster collaborative projects between faculty and students
Salary competitiveness to retain high quality faculty
University of Texas System
NSSE Use for Accountability detailed in U.T. 2008 System Report: Inform Texas legislature, Board of
Regents, U.T. System Faculty Advisory Council and Student Advisory Council, and general public VSA U.T. System Accountability Report Legislatively mandated Consumer
Satisfaction requirements
University of Texas System
NSSE used for benchmarking:NSSE results considered useful for
benchmarking because items are strongly associated with student success
NSSE Benchmarks provides continuing source of comparisons to gauge U.T. System progress
Where significant differences occur, institutions are encouraged to identify areas for improvement
Student engagement assessment & research
Feedback for internal use Appropriate peer comparisons Focus for improvement Performance funding
Tennessee Higher Education SystemAll TN Public 4yr institutions participated in NSSE as required by TN performance funding program.
Research about the relationship between engagement and satisfaction at TN public institutions showed the importance of: Quality of academic advising Relationship with peers, faculty &
staff Institutional support for success
Tennessee Higher Education System
Interest in NSSE as a process measure… What happens to students
while attending higher education?
Identified State & Institutional Policy Implications:
More research to examine relationship between engagement, satisfaction and degree production (develop longitudinal database, track individual students surveyed)
Institutions encouraged to evaluate conditions of student services and improve advising; etc.
Tennessee Higher Education System
Using student engagement data in
Kentucky to promote student success
John HayekVice President
August 13, 2009
SHEEO conference
8 public universities, 16 community colleges and technical, 20 regionally accredited, nonprofit, independent colleges and universities, and numerous other postsecondary options.
212,000 undergraduate students and 27,000 graduate and professional students in 2007.
49,700 total degrees and certificates awarded; 18,600 bachelor’s degrees in 2007.
600,000 college and university alumni.
$183 million in federal Pell grants, $486 million in federal loans, and over $180 million in state financial aid in 2006.
$4.2 billion in total revenue generated by Kentucky’s public universities and colleges in 2007.
1,869 public postsecondary education buildings with more than 47 million gross square feet.
30,000 full-time faculty and staff employed at Kentucky’s public universities and colleges in 2008.
$327 million in extramural R&D expenditures combined at the University of Kentucky and University of Louisville in 2006.
Kentucky Postsecondary Education Enterprise
28
Net General Funds have been cut $78 million or 7.2% over the past 18 months, while K-12, Medicaid, and Corrections have been exempted.
Budget Update
Source: CPE Comprehensive Database
$1,084
$1,052$1,026
$1,006
$936
Enacted FY08 Revised FY08 After 3% Cut
Enacted FY09 After 2.4% Cut
Revised FY09 After 2% Cut
Revised FY10
Kentucky Public Postsecondary Education Net General Funds ($millions)
FY08 to FY10
$78 M Cut in Net General Funds
$70 MSFSF
Tuition increases (3% for community college, 4% for comprehensive universities, and 5% for research universities), lowest in a decade
Why it Matters
The 50 StatesRelationship between per capita income and college degrees
The single factorwith the greatest powerto explain differences in per capita income between statesis the percentage of college graduates.
Milken Institute, 2002
30
Context Postsecondary Reform Legislation in 1997 - Long-term goals
Public Agenda – Short-term strategic plan updated every four years
Accountability System – Key performance indicators and goal setting process
KY Action NSSE (2001, 2003, 2005, 2007, & 2009)
College-level learning pilot (2002)
CCSSE (2006, 2007, and 2008)
Regular reports to Council on progress
Faculty development meetings
Accreditation and quality enhancement plans
NSSE WorkshopKentucky was one of the first states to participate as a system in NSSE.
Outcomes Increased attention to quantity and quality
Improvement on 25% of benchmark scores between 2001 and 2007
FTFT retention rates improved from 71% (1998) to 74% (2007)
Graduation rates improved from 37% (1998) to 47% (2006)
Bachelor’s degrees increased from 14,600 (1998) to 18,600 (2007)
Key is to keep student engagement and student outcomes data on the radar screen.
Jim MoranVice Chancellor for Academic and
Student Affairs
August 2009
Performance Matters Pennsylvania State System of Higher Education
35
Overview
What is the Pennsylvania State System of Higher Education (PASSHE)?
How did we develop a culture of accountability and performance?
What results have we achieved?
36
PASSHE In BriefFourteen (14) regional public
universities Focus on baccalaureate and masters
programs (doctoral mission at one university)
Rural settingsOver 112,000 students90% undergraduate / 90% Pennsylvania
residentsSystem created in 1983
38
Measuring What Matters
Defining values and goals
Developing meaningful, reliable measures
Data driven decision-making
Discussion
Quantitative MeasuresQualitative MeasuresWhat have we learned?Where are we going?Evolving Use of the NSSE
40
Conclusions
Performance driven by values and goals
Solid data upon which to make decisions is everyone’s business
Accountability for results is shared
What other ways can NSSE and other assessment tools be used to stimulate improvement in state system institutions?
How can NSSE be used to advance state system concerns, or inform policy?
NSSE advises against using results in performance funding. What is an appropriate use?
What other uses and tools are you considering?
For Discussion…