leveraging comparative analysis - presentation by robert
DESCRIPTION
TRANSCRIPT
Leveraging Comparative Analysis for Institutional
Decision Making
NJAIR Annual ConferenceApril 17th, 2009
The College of New Jersey
Robert Miller, Centenary CollegeChad May, The Richard Stockton College of NJ
BENCHMARKING: WHAT IS IT AND WHO DO WE COMPARE TO?
Leveraging Comparative Analysis for Institutional
Decision Making
What Forces are Driving the Need for Using Data for Institutional Decision Making
Internal Finite Resources Competition for Students
External Increased Accountability Increased Call for Transparency Students as Consumers
What is Benchmarking?
Benchmarking is an ongoing, systematic process for measuring and comparing the work processes of one organization to those of another, by bringing an external focus to internal activities, functions, or operations (Kempner 1993).
Practitioners at colleges and universities have found that benchmarking helps overcome resistance to change, provides a structure for external evaluation, and creates new networks of communication between schools where valuable information and experiences can be shared (AACSB 1994).
Benchmarking is a positive process, and provides objective measurements for baselining (setting the initial values), goal-setting and improvement tracking, which can lead to dramatic innovations (Shafer & Coate 1992).
Why Benchmark?
Identification of “Best Practices” Academic Operational
Provide context for institutional dataGoal setting and measurementInstitutional Planning
Competitor Aspiration Predetermined Peer
Natural (i.e. athletic conference, regional group)
Traditional (i.e. based on history)
Jurisdictional (state or local jurisdiction)
Classification-based (i.e. Carnegie Commission, AAUP,
US News, etc.)
Adapted from (Teeter & Brinkman 2003 in The Primer for Institutional Research, AIR)
How Do We Select the Institutions?
Types of Peers
Definitional Have similar identifiers expressing the essential nature of the
institutionInformational
Hold practical knowledge of a desired process, outcome, accomplishment
Analytical Provide realistic and practical benchmarks for internal and
external reviewNonsensical
Have “…no meaning or [convey] no intelligible ideas”;”…absurd or contrary to good sense”
*Webster’s Seventh New Collegiate Dictionary. G.&C. Merriam Company, Springfield, Massachusetts. 1967.
Adapted from a presentation given at the NEAIR 2002 Conference
Reasons for PEER ANALYSIS ???
CompareComplain
AssessBoast
ImproveFund
Evaluate
Some Common Peer Characteristics
• Affiliation (Public vs. Private)• Carnegie Classification• Financials (endowment, tuition, assets, liabilities, expenses,
revenue)• Enrollment and Staffing Levels• Selectivity (SAT, Acceptance rates)• Academic Programs (majors and degrees)
• IPEDS PAS System can generate a comparison group automatically using the information above
Data & Statistics & Judgment (Hybrid approach)
Data & Statistics (Cluster Analysis)
Data & Judgment (Threshold Approach)
Judgment (Panel Review)
Strategies of Developing Peer/Aspirant List
(Adapted from Teeter & Brinkman 2003 in The Primer for Institutional Research, AIR)
Sources of Comparison DataTo Help Identify Peers
Carnegie Foundation
National Student Clearinghouse- StudentTracker
Integrated Postsecondary Education Data System (IPEDS) Peer Analysis System, Dataset Cutting tool, Executive peer tool, etc.
http://nces.ed.gov/IPEDS/
WHAT TYPE OF COMPARISON DATA IS AVAILABLE?
Leveraging Comparative Analysis for Institutional
Decision Making
Sources of Comparison DataRecruitment & Retention
Noel-Levitz National Enrollment Management Survey
Consortium for Student Retention Data Sharing Data on retention rates & graduation rates
IPEDS Peer Analysis SystemThe College Board Admitted Student
Questionnaire (ASQ and ASQ plus)ACT, Inc
Sources of Comparison DataStudent Engagement
National Survey of Student Engagement (NSSE) SPSS Syntax Files
UCLA’s Higher Education Research Institute Surveys Cooperative Institutional Research Program (CIRP) Your First College Year College Senior Survey
Education Benchmarking Inc. (Resident Student Assessment, First Year Initiative Survey, etc.)
Sources of Comparison DataStudent Learning
Collegiate Learning Assessment (CLA)
ACT Collegiate Assessment of Academic Proficiency (CAAP)
ETS Measurement of Academic Proficiency and Progress
Sources of Comparison Data Financial Operations
NACUBO Endowment Study
NACUBO Tuition Discounting Study
Voluntary Support of Education Fundraising results
IPEDS Finance Survey (Peer Analysis System)
Guidestar (990 data for non-profits)
Sources of Comparison DataSatisfaction
Student Satisfaction Surveys Noel-Levitz Student Satisfaction Inventory ACT Survey of Student Opinions In-house surveys
Employee Satisfaction Surveys HERI Faculty Survey Harvard University (Collaborative On Academic
Careers in Higher Education survey)
Alumni Surveys ACT Alumni Survey and Alumni Outcomes Survey
Analytical Tools (Software/Services)
Proprietary Software MS Excel SPSS/SAS/STATA and other Stat packages Rapid Insight Analytics / Data Integration Tableau- Visual Analysis Software
Proprietary Services (Internet based applications) AGB Benchmarking Service Peer Analysis System (PAS) Dataset Cutting Tool Executive Peer Analysis Tool (create your own data feedback report) CUPA- Data on Demand Services Voluntary Support of Education (CAE)- Data tool AAUP Faculty Compensation data published in Academe JMA Associates
Big PictureInitiatives/Projects Available
Council for Independent Colleges CIC KIT This tool provides information relating to enrollment,
staffing, admissions, and financial aid. Key feature: allows you to conduct comparative analysis
using schools with similar financial resources. Sample of CIC KIT Tool
• http://www.cic.edu/projects_services/infoservices/kit.aspCIC FIT Tool
While the KIT tool provides traditional indicators such as acceptance rate, yield rate, and faculty counts, the FIT tool provides detailed financial comparisons
Ratio analysis for overall institutional health Sample of CIC FIT Tool http://www.cic.edu/projects_services/infoservices/fit/index.as
p
PSEUDO CASE STUDY
Leveraging Comparative Analysis for Institutional
Decision Making
Comparative Data for Internal Analysis: Case Study Example
Using comparative data to answer institutional specific questions
Common Question for IR professionals Who are students choosing over us and who are
students choosing us over?(i.e. the win/loss question)
Using Admissions and FASFA Data
Admissions interview dataExtraction of enrolled and not enrolled
students Analysis of fields to identify what other institutions
students sent their FAFSA data to- they can list up to six
Send batch files to the National Student Clearinghouse using the StudentTracker service Return file from NSC shows enrollment history of your
non-enrolling admitted students Match NSC return file data to other institutional data
Internal Data Combined with Student Tracker
Last NameSAT Combined HS GPA EFC Enrolled
College Name
College State 2yr/4yr
Public/Private Major
Smith L 2.2 12000 0 LCCC PA 2yr Public Business
Smith L 2.4 2300 1 Moravian PA 4yr Public Business
Smith L 2.3 7800 0 Desales PA 4yr Private Business
Smith L 2.5 1200 0 PSU PA 4yr Public Business
Smith L 2.9 18000 0 PSU PA 4yr Public Business
Smith L 2.2 11000 1 Moravian PA 4yr Private Business
Smith L 2.6 9000 1 Moravian PA 4yr Private Business
Smith L 2.4 11000 0 LCCC PA 2yr Public Business
Smith L 3.7 3000 0 Albright PA 4yr Private Business
Smith L 2.3 7000 1 Moravian PA 4yr Private Business
Smith L 2.3 12000 1 Moravian PA 4yr Private Business
Smith L 3.2 19000 0 Scranton PA 4yr Private Business
Smith L 3.4 14000 0 Albright PA 4yr Private Business
Example Output- “Win/Loss” Ratio
YOUR INSTITUTION
Competitor Comparison Using FAFSA and NSC Data
# on # on Mean # Enrolled Estimated %
FAFSA FAFSA 2004-05 2005-06 2006-07 3-Year NSC Data Enrolled
Institution Name 2006-07 2007-08 Rank Rank Rank Rank Public 2007-08 2007-08
Your Institution 2499 2585 805 31.1%
Competitor1 198 205 1 1 1 1 Y 180 87.8%
Competitor2 113 138 8 6 2 5.3 Y 60 43.5%
Competitor3 147 128 3 2 3 2.7 35 27.3%
Competitor4 108 120 5 7 4 5.3 95 79.2%
Competitor5 130 106 2 3 5 3.3 Y 100 94.3%
Competitor6 117 102 5 5 6 5.3 10 9.8%
Competitor7 106 95 7 8 7 7.3 18 18.9%
Competitor8 130 93 4 3 8 5 25 26.9%
Competitor9 80 85 11 10 9 10 6 7.1%
Competitor10 78 85 13 11 9 11 Y 12 14.1%
….. ….. ….. ….. ….. ….. ….. ….. ….. …..
Hypothesis
Are institutional aid policies in line with other institutions? Is there a significant difference in EFC of enrolling
and non enrolling business students? Internal analysis
Is there a significant difference in the institutional grant aid awarded to enrolling and non enrolling business students? Internal analysis
How does grant aid compare between our institution and other institutions? IPEDS PAS
Average Aid by Institution
Percentage receiving any financial aid Average federal grant
Average state/local grant
Average institutional grant Average student loan
Albright College 89 4736 3327 12354 10185
DeSales University 99 3542 3284 9408 2984
Moravian College 92 4508 3528 11124 6603
University of Scranton 91 4000 3776 12102 5326
Case Study College 64 8752 3688 18463 7532
INTEGRATING COMPARATIVE ANALYSIS WITH PLANNING
Leveraging Comparative Analysis for Institutional
Decision Making
Reporting Comparative Data
Standard comparative reports Externally processed
Faculty Compensation Report (Academe) IPEDS Feedback Report University of Delaware Study of Instructional Costs and
Productivity NSSE, HERI, and other survey instruments
Internally Processed Dashboards and/or report of Key Indicators report(s) Competitors report and Tuition/Fee Comparison report Other IR reports
Ad-hoc comparative reports Retention- where are our students going? Graduation Rate Study Internal analysis of survey data (comparison of student
satisfaction)
Six-Year Graduation Rate 2003
70.5% 72.1%
63.1%
68.0%
63.1%
69.2%65.0%
54.4%50.0%
64.8%61.4%
44.9%
57.0%
66.2%67.8%63.4%65.2%
55.5%
69.0%70.4%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Peer and Aspirants
Per
cen
tag
e
Yellow Bars- Represent Aspiration InstitutionsDark Blue Bars- Represent Peer-Like InstitutionsAqua Bars- Represent Peer- Below Institutions
Orange line across represents the target institution
Example Institutional Dashboard Summary Dashboard Fall 2008
Student Body Admission Finance (FY 08) Advancement (FY08)
Part-Time UG’s
FR Applicants
Endowment/Reserves
Total Gifts
10,000
2,000
Full-Time UG’s
Graduate Students
1,000 (fall)
97.6%
% UGs in-State
68%
6-year Graduation Rate
Diversity Enrollment
21%
81.6%
First-year Retention
SR Stdnt Satisfaction
85%
Gifts to Capital & Endwmnt
20,000 $ $
60%
37.0%
74%ile
1250
% FR Acceptances
Yield (% Enrolled)
H.S. Avg. Rank
Avg. SAT- Regular
%
%
%
%
%
%
Return on Endowment /Reserves Portfolio
Spending Rate
Unrestricted Annual Fund Gifts (change)
Positive Variance
Student Revenue Reliance
Debt coverage ratio
%
UG Alumni Participation
Number $1,000 donors
Faculty
Student Aid
UG Student/Faculty Ratio
Full-Time Faculty
UG Class Size >=30
UG Class Size <10
Taught by FT Faculty
FT Faculty W/ Term. Deg.
Discount Rate
60%
Inst. Financial Aid as % of Operating Budget
% of FT Students w/ Financial Need
KEY
Change: Higher
Lower
None
Importance of Change:
Green= better
Red= worse
Yellow= neutral55%
% Resident Stds. (FT)
% FT Faculty w/ Tenure50%
60%
29%
74.7%
85%
1017
51.4% CurrentBenchmark
Gross Cost to raise $1 FY 2006
%
Plant Reinvestment Rate (excludes current construction projects)
1.5-2%65%
Another Indicator
15,000
65%
15,000
40%
Conclusion/Discussion
Comparative Analysis provides context for institutional data with respect to decision making/planning/and assessment.
There is a significant amount of data already available. Much of which is almost ready-made for dissemination.
If you do not do the comparative analysis someone else will. (students, government, parents, etc.)