office of institutional research song yan, kristy maxwell, mark a. byrd associate director senior...
TRANSCRIPT
U.S. News & World Report’s American’s Best
Colleges Ranking Methodology
Office of Institutional ResearchSong Yan, Kristy Maxwell, Mark A. ByrdAssociate Director Senior Research Analyst AVP
Wayne State University
U.S. News Best Colleges ranking background
Methodology: ◦ Details on how the Best Colleges rankings are
calculated◦ How each ranking variable is computed and how
each estimate is calculated New changes to the rankings Criticism of the rankings
Agenda
First ranking published in 1983 It’s purpose was to be a starting point for families
searching for a college Allows for comparison of relative cost, “quality,”
retention and graduation rates In September 2011, U.S. News Education section
had 4.5 million unique visitors (announced by U.S. News)
Rankings published by U.S News in 2012: ◦ Best Colleges, September 2012◦ Best Grad Schools, March, 2012◦ Best High Schools, May 2012◦ Top online Programs, January, 2012 (first time)◦ World’s Best Universities, October 2012
Background
Ranking categories:Based on Carnegie Basic Classification◦ National Universities: (Offer a full range of
undergraduate majors, plus master’s and Ph.D. programs, and faculty emphases on research)
◦ National Liberal Arts Colleges (focus almost exclusively on undergraduate education)
◦ Regional Universities (offer a broad scope of undergraduate degrees and some master’s degree programs, but a few, if any, doctoral programs)
◦ Regional Colleges (focus on undergraduate but grant fewer than 50% of their degrees in liberal arts disciplines)
Best Colleges Ranking Categories
Quantitative measures that education experts have proposed as reliable indicators of academic quality
Gather data from each college on up to 16 indictors of academic excellence
Each factor is assigned a weight. Colleges and universities in each category
are ranked against their peers, based on their composite weighted score
Ranking Methodology
Ranking Category National Universities &National Liberal Arts College
Regional Universities & Regional College
Undergraduate academic reputation
22.5% 25%
Student selectivity for fall entering class
15% 15%
Faculty resources 20% 20%
Graduation & retention rates 20% 25%
Financial resources 10% 10%
Alumni giving 5% 5%
Graduation rate performance 7.5% 0%
Total 100% 100%
Ranking Category and Weight
Undergraduate Academic Reputation
Sub - Factor Data Collection National Universities & National Liberal Arts College-22.5%
Regional Universities & Regional College-25%
Peer assessment survey
Completed by peer institution administrators;On a scale from 1 to 5;
66.7% (15% of total)
100%(25% of total)
High school counselor’s ratings
Counselors at public and private high school; On a scale from 1 to 5;
33.3%(7.5% of total)
0%
Note: For the first time this year, two most recent survey results (2011 & 2012) were averaged to compute the high school counselor reputation score.
Equation*:
Academic Reputation= Z avg. Peer assessment score * (15%) + Z avg. HS Counselor Score * (7.5%)
Estimates for missing: None
* For National Universities and National Liberal Arts Colleges
Undergraduate Academic Reputation
Student Selectivity
Sub - Factor Data Collection National Universities & National Liberal Arts College-15%
Regional Universities & Regional College-15%
Acceptance rate Ratio of # admitted to applicants, fall 2011.
10% (1.5 % of total)
10%(1.5 % of total)
High school class standing in top 10%
Proportion of students enrolled who graduated in the top 10%, fall 2011 entering cohort.
40% (6 % of total)
0% (0 % of total)
High school class standing in top 25%
Proportion of students enrolled who graduated in the top 25%, fall 2011 entering cohort.
0% (0 % of total)
40% (6 % of total)
SAT/ACT scores Critical Reading and Math portions of the SAT and ACT composite , fall 2011 entering cohort.
50%(7.5 % of total)
50%(7.5 % of total)
Equation*:
Z Student selectivity= Z test score * (50%) + Z high school class standing * (40%) + Z acceptance rate * (10%)
Estimates for missing: one standard deviation less category’s mean
Student Selectivity
* For National Universities and National Liberal Arts Colleges
Faculty Resources
Sub - Factor Data Collection National Universities & National Liberal Arts College-20%
Regional Universities & Regional College-20%
Faculty compensation
Average faculty pay & benefits, AY 2010-11 and AY2011-12.
35%(7 % of total)
35%(7 % of total)
% Faculty with top terminal degree
% full-time faculty with doctorate or highest in field, current year data.
15% (3 % of total)
15% (3 % of total)
% Faculty full time The proportion of full-time faculty to full-time-equivalent faculty, current year data.
5% (1 % of total)
5%(1 % of total)
Faculty Resources
Sub - Factor Data Collection National Universities & National Liberal Arts College-20%
Regional Universities & Regional College-20%
Student/faculty ratio
Full-time-equivalent students to full-time-equivalent faculty, current fall 2011.
5% (1 % of total)
5%(1 % of total)
Class size, 1-19 students
Percentage of UG classes with fewer than 20 students enrolled, current fall 2011.
30%(6 % of total)
30%(6 % of total)
Class size, 50+ students
Percentage of UG classes with 50 students or more enrolled, current fall 2011.
10%(2% of total)
10%(2% of total)
Equation:
Z Faculty resources= Z avg. salaries * (35%) + Z avg. w/term degree * (15%) + Z faculty ft * (5%) + Z student faculty ratio * (5%) +Z % class <20 * (30%) + Z % class 50 or more * (10%)
Estimates for missing: one standard deviation less category’s mean
Faculty Resources
Graduation and Retention Rates
Sub - Factor Data Collection National Universities & National Liberal Arts College-20%
Regional Universities & Regional College-25%
Average six-year graduation rate
4 year average, fall 2002 through fall 2005 cohort
80% (16 % of total)
80%(20 % of total)
Average freshman 1 year retention rate
4 year average, fall 2007 through fall 2010 cohort
20%(4 % of total)
20%(5% of total)
Equation:
Z Graduation and retention= Z avg. 6-yr grad rate * (80%) + Z avg. fresh retention rate * (20%)
Estimates for missing: IPEDS or NCAA data; Otherwise, one standard deviation less category’s mean
Graduation and Retention Rates
Financial Resources
Sub - Factor Data Collection National Universities & National Liberal Arts College-10%
Regional Universities & Regional College-10%
Financial resources per student
Average spending per FTE student on instruction, research, public service, academic support, students services and institutional support during the FY 2010 and 2011.
100% (10 % of total)
100%(10 % of total)
FTE calculation: full time plus one third of part time.
Alumni Giving
Sub - Factor Data Collection National Universities & National Liberal Arts College-5%
Regional Universities & Regional College-5%
Average alumni giving rate
Average percentage of undergraduate alumni donated to the school.
100% (5% of total)
10% (5% of total)
Graduation Rate Performance
Sub - Factor Data Collection National Universities & National Liberal Arts College-7.5%
Regional Universities & Regional College-0%
Graduation rate performance
Difference between actual six year graduation rate and the predicted graduation rate, cohort 2005.
100% (7.5% of total)
0% (0% of total)
Predicted graduation rate is based upon characteristics of the entering cohort, as well as characteristics of institution.
Rankings categories have been updated◦ Update to Carnegie basic classification (2010)
For-profit institutions are included (2011)◦ All for-profit colleges and universities grant
bachelor’s degree and regionally accredited. Non-responders are handled differently
◦ For schools refused to fill out U.S. News survey for at least two years and for school news to the ranking, made extensive use of data from NCES.
Recent Changes
Now collecting graduation rates of students who received a federal Pell grant, a subsidized Stafford loan but not a Pell grant, and students who received neither.
May add collection of data for affordability and technology◦ Graduation rates by race◦ Net price of attending by EFC◦ Average loan size◦ # of degrees awarded per student◦ Ways colleges using technology (computer
accessibility, wireless Internet, technical support, etc.)
Recent Changes (continued)
Most data come from the college, U.S News Main Survey, Financial Aid Survey
In 2011, 93% of the 1,378 ranked college and universities returned the survey
Obtain missing data from AAUP, National Collegiate Athletics Association, the Council for Aid to Education, and the NCES
Estimates are used in the ranking calculation when schools fail to report particular data points.
Data Sources
Schools are unranked and listed separately by category if ◦ They do Not use the SAT or ACT test scores in
admissions decisions for FTIAC, or◦ Too few respondents to the peer assessment
survey, or◦ Have fewer than 200 total enrollment, or◦ Large proportion of nontraditional students and no
first-year students◦ Also a few highly specialized schools in arts,
business, and engineering
Schools Excluded from the Ranking
AIR members could (& have) provide feedback and suggestions to U.S. News
Two weeks prior to the release of rankings, email alert sent to survey respondents to Main and Fin Aid survey
Free one-month short-terms passes for College Compass One day prior to the official rankings release date, survey
respondents were sent notification and be able to access to two years of published ranking tables in PDF on a non-public website
AIR members can receive U.S. News published ranking table in Excel format (need to send an email to request the Excel spreadsheets to [email protected] or [email protected]
Unpublished rankings available by request (send email to request)
IR and U.S. News Rankings
#10 Oversimplification: Comes from Stanford University’s FUNC (Forget US News Coalition) & Reed College who in the mid-90s argued “that ranking something as complex and variable as a college education with a single number is an oversimplification. “ FUNC claimed “that the process makes college administrations focus on numerical rankings rather than on educating students.“ Casper, Gerhard (18 April 1997)
Top 10 Criticisms of US News College Ranking
#9 Peer assessments:◦ College Presidents may have outdated or no
knowledge of institutions they are ranking◦ College Participation among Presidents was
43% in 2011, down 5 percentage points overall.◦ May rank competitors low◦ Low participation rates, amongst high school
counselors
Top 10 Criticisms of US News College Ranking (continued)
#8 Data manipulation: As early as 1994 when the Wall Street Journal disclosed that institutions flagrantly manipulating data in order to move up in the rankings in U.S.
Top 10 Criticisms of US News College Ranking (continued)
#7 Not much weight given to cost of attendance
Top 10 Criticisms of US News College Ranking (continued)
#6 Academic quality not captured
Top 10 Criticisms of US News College Ranking (continued)
• #5 Institutional differences not taken into account◦ Diverse student body◦ Institutional mission
Top 10 Criticisms of US News College Ranking (continued)
#4 Lack of data definition standards in the data collection process.
Top 10 Criticisms of US News College Ranking (continued)
#3 Proliferation of data collected
Top 10 Criticisms of US News College Ranking (continued)
#2 Cost / Benefit
Top 10 Criticisms of US News College Ranking (continued)
#1 ____________________
Top 10 Criticisms of US News College Ranking (continued)
Thank you for attending.