Report of the NACAC Ad Hoc Committee on U.S. News & World Report Rankings

September 23, 2011

2

ContentsI. Introduction 3

II. RankingsHistoryandBackground 4

III. TheNACACDiscussion 7

IV. MethodologyMatters 10

V. DistortingEffectsandProfessionalPractice 15

VI. TakingtheNextSteps 20

VII. AdHocCommittee’sRecommendations 23

VIII. Appendix 24

IX. Bibliography 25

3

IntroductionIn2010,theNationalAssociationforCollegeAdmissionCounseling(NACAC)convenedagroupofmemberstoform

anAdHocCommitteeonU.S.News&WorldReportRankings.TheCommitteewasconvenedtoconductdiscussions

withU.S.Newsstaffforthepurposeofofferinganorganizationalconduitthroughwhichtoexchangeideas,convey

concerns,andrespondtoquestionsabouteachorganization’srespectiveconstituency.

TohelpinformtheAdHocCommittee,NACACconductedasurveyofassociationmembersinMay2010togauge

attitudesofcollegeadmissioncounselingprofessionalstowardtheU.S.News&WorldReportrankingspublication,

“America’s Best Colleges.”1 This final report incorporates survey findings from the member survey, as well as

informationgleanedfromcommitteediscussionsandmeetingswithmembersofNACAC’sstateandregionalaffiliates

fromaroundthecountry.

Forpurposesof this report, the term“Committee”refersonly to theNACACmemberswhoserveon theAdHoc

Committee.Representatives ofU.S.News&WorldReportmeetwith theAdHocCommittee to ensure an open

dialoguewiththeassociation,buttheirviewsarenotrepresentedinthisreport.Fortheofficialpublicsummariesof

meetingsbetweenbothNACACandU.S.News&WorldReportrepresentatives,visittheAdHocCommittee’sWeb

page at http://www.nacacnet.org/AboutNACAC/Governance/Comm/Pages/NACACUSNewsAdHocCommittee.aspx.

TheCommitteewouldliketoexpressitsappreciationtoRobertMorse,DirectorofDataResearchatU.S.News&

WorldReport,forhisparticipationindiscussionswiththeAdHocCommittee.

Committee Members

PeterCaruso,Chair

BostonCollege,MA

BruceChamberlin

GeorgetownUniversity,DC

RafaelS.Figueroa

AlbuquerqueAcademy,NM

PamHorne

PurdueUniversity,IN

IreneLogan

VirginiaStateUniversity

LeeMelvin

UniversityofConnecticut

JosephPrieto

HinsdaleCentralHighSchool,IL(Retired)

KrisGettingRoach

UniversityofSt.Thomas,MN

MichaelSexton

SantaClaraUniversity,CA

4

Rankings History and BackgroundThe Evolution of Modern Rankings

Throughoutthelatenineteenthcentury,theU.S.BureauofEducationpublishedannualreportscontainingstatistical

dataoncollegesanduniversitiesacrossthenation.Followingthediscontinuationofthesereportsin1890,various

(andvaried)attemptshavebeenmadetorankinstitutionsofhighereducationbasedonacademicquality.2TheU.S.

News&WorldReport’srankingspublication,thoughunprecedentedinitspopularity,isinmanywaysaproductof

therankingsthatcamebeforeit.

Historically,itwascommoninEuropetocountthenumberofeminentmenassociatedwithauniversityinorderto

measure itsworth.Thismethodwasuse in thefirstAmericancollege rankings in1910andremained influential

throughthemid-1960s.4 At leastninesuchrankingswerepublishedbetween1910and1964byJamesCattell,

thefirsttorankU.S.collegesanduniversities,andbyothers.5ReputationalrankingswereintroducedbyRaymond

Hughesin1925.Theirprominencedevelopedslowlyovertime,eventuallyeclipsingthatoftheprevious“eminent

men”rankings.Regardlessofthemethodology,however,mostearlyrankingswereusedprimarilywithininstitutionsof

highereducationandassociationsofcollegesanduniversities.Theygarneredonlyminimalattentionfromthepublic

upuntiltheintroductionoftheU.S.News&WorldReportrankingspublication.6

2Walleri,R.DanandMarshaK.Moss,EvaluatingandRespondingtoCollegeRankings,1995.3CompiledfromWalleri,1995;Meyers,LukeandJonathanRobe,“CollegeRankings:History,CriticismandReform,”CenterforCollegeAffordabilityandProductivity,2009;Rauhvargers,Anrejs,“GlobalUniversityRankingsandTheirImpact,”EuropeanUniversityAssociation,2011;andWebster,David,“ReputationalRankingsofColleges,Universities,andIndividualDisciplinesandFieldsofStudy,FromTheirBeginningstothePresent,”AgathonPress,1992.4Meyers,2009.5Ibid.6Walleri,1995;Meyers,2009.

Table 1. Timeline of Major Developments in Academic Quality College Rankings3

1910 Psychologist James Cattell publishes the first American college rankings in American Men of Science based on the number of identified "eminent men" who had earned degrees from, or were employed at, the ranked institutions.

1925 Chemistry professor Raymond Hughes publishes the first reputational rankings in A Study of the Graduate Schools of America.

1934 Hughes, now chair of the American Council of Education, publishes a second, more extensive, set of reputational rankings of graduate schools.

1957 Journalist Chesly Manly publishes six college rankings in the Chicago Tribune: ten best universities, co-educational colleges, men’s colleges, women’s colleges, law schools and engineering schools.

1959 Humanities professor Hayward Keniston reintroduces reputational rankings in the appendix of a report prepared for the University of Pennsylvania.

1966 Allan Cartter, with the American Council for Education, publishes An Assessment of Quality in Graduate Education, also known as the Cartter Report. It is based on reputational surveys, features improved methodology and sells approximately 26,000 copies.

1967 Jack Gourman begins publishing a college ranking titled The Gourman Report, which applies an undisclosed methodology and is widely criticized for its statistical anomalies.

1970 Kenneth Roose and Charles Anderson replicate Cartter’s methodology in A Ranking of Graduate Programs and attempt to downplay the “pecking order” of institutions.

1973 Peter Blau and Rebecca Margulies publish a reputational ranking of professional schools. 1977 Allan Cartter and Lewis Solomon publish another reputational ranking of professional schools in

response to dissatisfaction with the Blau and Margulies rankings. 1981 Lewis Solomon and Alexander Astin publish an undergraduate reputational ranking. 1982 The National Academy of Sciences and National Research Council publish the Assessment of

Research-Doctorate Programs in the United States, which includes ratings, rather than rankings, of institutions, and is based on both reputational survey responses and quantifiable data.

1983 U.S. News & World Report first publishes its college and university rankings. 2003 Shanghai Jiao Tong University publishes the first global university rankings.

5

Encouragingly,thehistoryofreputationalcollegerankingsismarkedbyvariousimprovementsinmethodologyand

expansionsinscope.Earlyreputationalsurveyswerecarriedoutonaverysmallscaleandthereforeproducedless

reliableresultsthatwerelessinclusive.Forexample,Hughes’firstranking(1925)wasbasedsolelyontheresponses

ofMiamiUniversityfacultymemberswhowereaskedtolist40-60instructorswhotaughtintheirdisciplineandto

ratethequalityofthedepartmentsofonly36institutions.7AlanCartter’s1966rankings,conversely,werebasedon

surveyssenttoseniorandjuniorscholarsanddepartmentchairpersonsfromacrossthenation.Respondentswere

askedtorateinstitutionsseparatelyonthequalityoftheirgraduatefacultiesandthequalityoftheirdoctoraltraining

programs,addingalayerofspecificitytotherankings.Cartter’sfinalproductincluded106institutions,morethanany

previousranking.8In1982,theNationalResearchCouncil’sratingscombinedquantitativedatawiththeresultsof

reputationalsurveysregarding228institutions,makingitmorecomprehensivethananypreviousefforttomeasure

academicqualityatcollegesanduniversitiesintheU.S.9

TheU.S.News&WorldReportrankingspublicationhas,sinceitsfirstpublicationin1983,gonethroughsignificant

changes and developments. The first three editions were based entirely on reputational surveys sent to college

presidentsandwerereleasedeveryotheryear(1983,1985and1987).In1988,anumberofimportantprecedents

wereset: therankingspublicationbegantobereleasedonanannualbasis, itsreputationalsurveysweresent to

academicdeansandadmissionofficersaswellascollegepresidents,and itbased75%of therankingsonnon-

reputationalinputandoutputvariables.Aroundthesametime,U.S.Newsalsobeganreleasingtheirrankingsina

standaloneguidebook,whichprovidedmoreinformationthantherankingsissueofthemagazine.10

Sincetheybeganrankingcolleges,U.S.Newshasimprovedtheintegrityoftherankingsbyconfirmingself-reported

datawithothersourcesandstandardizingandclarifyingvariabledefinitions.Theyalsobeganincludinginformation

onunrankedinstitutionsandroundingaggregatescoresuptowholenumbers(ratherthanfirstdecimalplaces)to

determineties.Duringthemid-1990s,outputmeasuresreceivedaddedemphasis,astheweightonthegraduation

ratevariablewasincreasedandthe“valueadded”(nowreferredtoas“graduationrateperformance”)variablewas

added.Inthe2004editionof“America’sBestColleges,”U.S.Newsrespondedtogrowingcriticismfromthosein

highereducationandabandoned theuseof theyield variable indeterminingselectivity.11 Thepopularityof the

publicationgrewasthemethodologywasrefined.AccordingtoCollege Rankings: Democratized College Knowledge

for Whom?,“USNWRsold485,000reprintsofthatfirstissueandcurrentlysellsanestimated2.3millionofitsregular

collegerankingsissueandanestimated700,000copiesofitsstand-alonecollegerankingsguide.”12

The Effects of Rankings and Our Desire for Order

Rankings seem to fulfill twodemands inmodern society, oneperhapsmoreunderstandable thananother. First,

consumersviewrankingsasamethodfordeterminingvalue.InformationaboundsintheInternetage,thoughfinding

atrustedsourcetomakesenseoftheinformationcanbeadifficulttask.Indeed,theInstituteforHigherEducation

Policy(IHEP)notesthat“stakeholdersgravitatetowardsystemsthatprovidesomeinterpretationoftheinformation”

available about postsecondary education.13 Second, many people, as evidenced by the commercial success of

U.S.News’“America’sBestColleges,”craveanauthoritativesourcetotellthemwhatis‘best,’whoisnumberone.

Rankorderingtoastersorcarsmaymakesomesenseinthiscontext,astherearemoreorlesstangibleoutcomes

whichdepend lessonhuman tendenciesandmoreonmechanicalprecision.However,collegesanduniversities

arecomplex institutionswithmultiplepurposes that, toa largeextent, relyonhuman interactionsand individual

determinationtofunction.

7Meyers,2009.8Walleri,1995;Meyers,2009.9Walleri,1995;10Sanoff,AlvinP.,“TheU.S.NewsCollegeRankings:AViewfromtheInside,”InstituteforHigherEducationPolicy,2009;Meyers,2009.11Ibid.12McDonough,PatriciaM.,etal,“CollegeRankings:DemocratizedCollegeKnowledgeforWhom?”ResearchinHigherEducation,1998,Vol.39.5.13Sponsler,BrianA.,“IssueBrief:TheRoleandRelevanceofRankingsinHigherEducationPolicymaking,”InstituteforHigherEducationPolicy,2009.

6

Thetroublewithrankingsuchcomplexinstitutionsusingsimplemeasuresiswell-documented.Rankingscanbeself-

fulfillingprophecies,theprocessbywhichreactionstosocialmeasuresconfirmtheexpectationsorpredictionsthat

areembeddedinmeasuresorwhichincreasethevalidityofmeasuresbyencouragingbehaviorthatconformstoit.14

Evidenceoftheself-fulfillingprophecyisfoundintheinfluenceofpriorrankingsonresponsestopeerreviewsurveys,

acorecomponentoftheUSNWRrankings.

Rankingscanalsoimproperlyreduceandsimplifycomplexconcepts,decontextualizinginformationthatcouldbe

essentialtoconsumers.AnexcellentexampleofthedecontextualizingeffectsoftheUSNWRrankingswasexpressed

bynumerous respondents to theNACACmembersurvey. In response to thequestion,“Does the title ‘America’s

BestColleges’accuratelydescribethecontentsofthepublication?,”NACACmembersresponded,“Bestforwhom?”

Rankings, according to Sauder and Espeland, are “the culmination of many commensurative practices,” where

complex concepts are reduced to single data points that are deemed indicative of quality or success.15 Such

commensurationleadstoscrutinyoverminisculedifferences,aphenomenoncommonlyascribedtotherankordering

ofinstitutionsusingtheUSNWRmethodology.

Thecumulativeeffectsoftherankings,combinedwiththedistillationofjudgmentsaboutqualityintodatapointsof

varyingdegreesofdefensibility,createquestionableincentivesforinstitutionsandleadtosignificantconfusionamong

consumersaboutthedifferencesbetweeninstitutions.Suchconfusionamongconsumersriskscreatingmismatches

betweenstudentsand institutions,aswellas reputationalbiases thatmaydiscouragestudents frompursuingan

educationataninstitutionthatmaybethe“best”collegefortheirneedsortastes.

14Sauder,MichaelandWendyNelsonEspeland,“RankingsandReactivity:HowPublicMeasuresRecreateSocialWorlds,”AmericanJournalofSociol-ogy,2007,Vol.113.1.15Ibid.

7

The NACAC DiscussionThe Ongoing Discussion

Thediscussionaboutacademicrankingsandtheireffectsisextensiveandongoing.Throughoutmuchofthelast

decade,membersofNACAC,aswellasotherobservers,haveexpressedamyriadofconcernsaboutrankingsona

consistentbasis.TheNACACBoardofDirectorsviewedtheappointmentofanAdHocCommitteeasanopportune

momenttoconsolidatethemembers’concernsintoasingle,organizedeffort.

Thiscommittee’sobservationsandrecommendationsarebestseenassymptomaticoflong-standingconcernsthat

lingerinourprofessionalcommunity,aswellasothers.Aninitialandexpectedobservationofthecommitteewasa

generaldislikefortherankingsoverall.AspartoftheCommittee’ssurvey,NACACmemberswereaskedtoindicate,

onascalefromoneto100,theirgeneralattitudetowardtheU.S.News&WorldReportrankings.Ascoreofone

representsastrenuousobjectiontotheU.S.Newsrankings,while50representsacompletelyneutralattitudeand

100indicatesstrongsupport.Themeanscores,whicharepresentedinFigure1,revealgenerallynegativeopinions

oftherankingsamongrespondents.

Highschoolcounselors(meanscoreof28.67)expressedlowerregardfortherankingsthancollegeadmissionofficers

(meanscoreof38.54),butbothgroupsheldnegativeviews(meanscoreunderfifty)towardtherankings.16Public

highschoolcounselorsviewedtherankingsslightlymorecharitably(meanscoreof35.11)thanprivatehighschool

counselors(meanscoreof23.92).

How Influential Are the Rankings?

Ifcollegeadmissioncounselingprofessionalsare,onthewhole,negativelyinclinedtowardthem,istheinfluenceof

rankingssuchthatweshouldbemoreconcerned,orlessconcerned,abouttheireffectsoncollegeadmissionand

counseling?NACACmemberswereaskedabouttheiropinionsonthechangeinprominenceoftheU.S.News&

WorldReportrankingsoverthelastfiveyears.AsFigure2suggests,amajorityofNACACmembersbelievethatthe

prominenceoftheU.S.News&WorldReportrankingshasincreasedoverthepastfiveyears.

0 20 40 60 80 100

Figure 1. Average Perceptions of U.S. News Rankings on "Feeling Thermometer"

College

High School

Total

16Toensurethatthemeanwasnotmaskinghighlypolarizedopinions,weexaminedthemedianscoresaswell.Highschoolcounselors’medianfeelingthermometerscorewastwentyfive,whilecollegeadmissionofficers’medianscorewasthirtyfive.

8

Best for Whom?

NACACmembersexpressedsomethingapproachingaconsensusonthequestionofwhetherthetitleofU.S.News

&WorldReport’sannualpublication, “America’sBestColleges,”accurately represents the informationpresented

therein.

Only2.9percentofall respondents(2.4percentofhighschoolcounselorsand3.3percentofcollegeadmission

professionals)believedthatthetitleofthepublicationaccuratelyrepresentsthecontentdeliveredbythepublication.

Themajorityofcollegeadmissionofficers(51.3percent)andhighschoolcounselors(61.9percent)reportedthatthe

titleisnotatallaccurate(Figure3).

AsFigure4indicates,publichighschoolcounselorswereslightlymorelikelytobelievethatthetitleatleast“somewhat

accurately”describesthecontentinthepublication.

0.0%5.0%

10.0%15.0%20.0%25.0%30.0%35.0%40.0%

Muchmore

Somewhatmore

Same Somewhatless

Much less

Figure 2. Change in Prominence of USNWR Rankings Over Past Five Years

College

High School

Total

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%

Accurately Somewhataccurately

Not at allaccurately

Figure 3. How accurately does the title "America's Best Colleges" describe content in

publication?

College

High School

Total

9

Morethan600NACACmembersofferedcommentsonthisquestioninadditiontotheirmultiplechoiceresponses.

The most common themes in the open ended responses add substance to the general notion that the title of

“America’sBestColleges”doesnotaccuratelyconveytheinformationcontainedinthepublicationtoconsumers.

Commonthemesincluded:

• “The Best for Whom?”—Many members stated that the best college for an individual student will be

determinedbythequalityoffitbetweeninstitutionandstudent.

• “What’s inaRank?”—Membersargued that thedifference innumeric rankbetweencolleges isat least

somewhatarbitrarybyvirtueoftheweightingsystemusedinthemethodology,thatmakingsubsequentrank

orderdistinctionsbetweencollegesdoesnotprovethatonecollegeis“better”thananother,andthatthe

weightsofthefactors,whenchanged,haveproducedandwillproducedifferentrankings.

• Inputsvs.Outputs—Membersfinditdifficulttoexplainwhattheterm“best”describes—manybelievethat

therankings’useof“input”variables(includingselectivityandtestscores)andothervariablesnotrelated

todirectlymeasurableoutputs(suchasthepeerassessments)leadconsumerstomakedecisionsbasedon

informationunrelatedtothequalityofeducationprovidedattheinstitution.

Overarching Themes

During the committee’s discussion, review of the member survey, and review of the academic literature, three

overarchingthemesemerged.

1) Methodology matters. Ordinalrankingscreateflawedconclusionsandresultingmisperceptionsabout

differencesincollegequality.

2) Distorting Effects and Professional Practice. Whilethereislittleconclusiveevidenceofwidespread

gamingofrankings,thetendencytoconformtorankingsmethodologycreatesincentivestofocus

disproportionateresourcesondataelementsthatcanchangerankingswithoutnecessarilychanging

thequalityoftheinstitution.Moreover,surveyresultsandCommitteediscussionssuggestedtheneedfor

moreprofessionaleducationandconsumerresourceswithregardtorankingspublications.

3) Taking the Next Step. Asthehistoryofrankingssuggests,changeisagradualandincremental

process—butonethattakesplacenonetheless.Therearemanyrecommendationsforchange,including

recentrecommendationsfromtheAmericanAssociationofCollegiateRegistrarsandAdmissionsOfficers

(AACRAO),thattheCommitteesupports.ThisCommitteefoundparticularinterestintheideaofcorrecting

formethodologicalflawsbyde-emphasizingthesingle,ordinallistthatU.S.News&WorldReportpublishes,

andinsteadencouragingmore‘openaccess’useofthedatatoallowstudentsandfamiliestocreatetheir

owncollegelistsusingcriteriatheyfindimportant.

4) Committee Recommendations. ThefinalsectionofthereportofferstheCommittee’srecommendationsas

aresultofitsdiscussions.

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%

Accurately Somewhataccurately

Not at all accurately

Figure 4. How accurately does the title "America's Best Colleges" describe content in publication?

HS Results Only

Public

Private

10

Methodology MattersAccordingtoaNationalOpinionResearchCenter(NORC)evaluationoftheU.S.News&WorldReportrankingsof

undergraduate institutions,“theprincipalweaknessof thecurrentapproach is that theweightsused tocombine

thevariousmeasuresintoanoverallratinglackanydefensibleempiricalortheoreticalbasis.”17Indeed,thereisno

objectivewaytoselectofcriteriaforinclusioninarankingmethod.Oncethecriteriaareselected,thereisfurtherno

objectivemannerforassigningweightstoeachcriterion.Asaresult,thereareaninfinitearrayofpossiblerankings

for colleges anduniversities.Moreover, because thecriteriaused to rankcolleges oftendonotdifferwidely, the

placementofweightstodifferentiateinstitutionswithsimilarcharacteristicsmakesspecificplacementinaranking

“essentiallyarbitrary.”18

The Institute for Higher Education Policy (IHEP) notes that “ordinal rankings are inherently misleading...[T]rue

differencesamongcloselyrankedinstitutionsmaybeminimalorquitelarge,butallrankingarepresentedashaving

thesamemagnitudeofdifference.”19

IHEPfurthernotesthat“[t]heinputsincommonrankingsystems—andtheindicatorsselectedtomeasurethem—

reflectimplicitlyvalue-ladendecisionsabouthowtoappropriatelydefineeducationalquality.Mostoftheindicators

used in theconstructionofcollegerankingshave little todowithpolicygoals relating toaccessandequity; they

createuniformnotionsofeducationalqualityandoverlookimportantdistinctionsineducationalpreparation,personal

experiences,andhistoricaltreatmentofvariousstudentpopulationsinhighereducation.Policymakersandthepublic

areill-servedbyrankingsthatrelyondataindicatorsthatbytheirnatureareexclusionary.”20

Student Selectivity

Withitsrankingmethodology,USNWRturnsahighlysubjectivecollectionofintangiblesintoonequantifiablevalue

thatcanbearrangedagainstotherlikevalues.Theresultingorderedlistattractsaneagerpopulationofinformation

seekerswithitssimpleapproachtoquantifyingaveryimportant,andexpensive,decision.Themethodologyisthe

backboneofthe“BestColleges”publication,andconsumersoftheinformationshouldunderstanditscomponents

beforeusingitasacollegesearchtool.Tounderstandthemethodologyistounderstandwhicheducationalvariables

USNWRdeems themost importantmeasuresof institutionalquality. Inacompanionpiece to the2010rankings

issue,USNWReditorialstaffcalledtherankingsa“startingpointforthecollegesearch.”Therankingsactuallyreflect

moreofanopinionofonenewssourceabouttheappropriatemeasuresofeducationalquality.

ThecurrentUSNWR“BestColleges”rankingformulaisbrokendownintoseveralprimarycomponents:undergraduate

academicreputation,studentselectivity,facultyresources,graduationandretentionrates,financialresources,alumni

giving,andgraduationrateperformance(thedifferenceoftheactualsix-yeargraduationrateandtheratepredicted

byUSNWR).Eachof thesecategories isbrokendown into “subfactors” that occasionallydiffer according to the

institutiontype.Table2focusesontheselectivityandreputationalcomponents.

17NationalOpinionResearchCenter,“TheNORCReportonU.S.NewsandWorldReport,”WashingtonMonthly,1997.18Huggins,PeterandLiorPachter,“SelectingUniversities:PersonalPreferenceandRankings,”CornellUniversity,2008.19Sponsler,2009.20Ibid.

11

This assemblage of factors and subfactors represents the data chosen by USNWR staff to measure institutional

quality,butthepresenceofthesevariablesalsohighlightstheabsenceofotherintangibles.Dataonteachingquality,

studentinvolvement,andabilitytoobtainajobdoesnotexistinapublication-readyformat,soUSNWRfindsother

measurablefactorstorepresentacademicrigor.USNWRcombinesinputvariables(standardizedtestsandclassrank),

processvariables(facultyresources)andoutputmeasures(graduationrates)togenerateafinalrating.However,the

newspublicationdoesnot includedetailedstatistical analysis thatwould lend thenecessary scientificcredibility.

JeffreyEvansStakemakesthispointinananalysisoftheimpactlawschoolrankingshaveoninstitutionalpolicy:

Likemanyrankings,thosepublishedbyU.S.Newsarebasedonanumberoffactors,mostofwhich

makeuseofreadilyavailabledata.Manyofthesecriteriaareofnoinherentinteresttothereadersofthe

rankings.Aprospectivelawstudentwantingtoworkasalawyerhasnoparticularinterestintheamountof

moneyaschoolspends,thenumberofvolumesinthelibrary,thegradesoftheotherstudentsinthe

class,oreventhereputationoftheschoolamongacademicsbecausesuchreputationsarebuiltprimarily

onfacultypublicationsandnotteachingquality.22

Furthermore,theheavyrelianceoninputvariablessuggestsamisinterpretationofthepurposeofhighereducation.

Wheninterpretingtherankingsasatruemeasureofinstitutionalquality,oneequatestheeducationalinputsthesame

astheeducationaloutputs.23Thus,themissionoftheuniversitybecomesirrelevant.

Table 2. Weights of Selectivity and Reputational Components of USNWR Methodology21 Ranking Category

Category Weight Subfactor Subfactor Weight National Universities and National Liberal Arts Colleges

Regional Universities and Regional Colleges

National Universities and National Liberal Arts Colleges

Regional Universities and Regional Colleges

Undergraduate Academic Reputation

22.5% 25% Peer assessment survey

66.7% 100%

High School Counselors’ rating

33.3% 0%

Student selectivity for fall 2010 entering class

15% 15% Acceptance rate

10% 10%

High school class standing in top 10%

40% 0%

High school class standing in top 25%

0% 40%

Critical Reading and Math portions of the SAT and the composite ACT scores

50% 50%

21U.S.NewsStaff,“BestColleges2012:AbouttheRankings/Methodology,”U.S.News&WorldReport,2011.22Stake,JeffreyEvans,“TheInterplayBetweenLawSchoolRankings,Reputations,andResourceAllocation:WaysRankingsMislead,”IndianaLawJournal,2006,Vol.81.299.23Henderson,WilliamD.andAndrewP.Morriss,“StudentQualityasMeasuredbyLSATScores:MigrationPatternsintheU.S.NewsRankingsEra,”IndianaLawJournal,2006,Vol.81.163.

12

Studentselectivityaccountsfor15%oftheoverallranking.Asidefromtheacceptancerate,twounreliablemetrics

wieldmostoftheinputvariable’semphasisontherestoftheranking:classrankandstandardizedtestscores.Ina

2009surveyofadmissionprofessionals,only16percentofrespondentsattributedconsiderableimportancetoclass

rank,24afactorthatcomprises40percentoftheselectivityrating.Theinclusionofclassrankinthemethodology

promotesacriterionthatpenalizesstudentsforselectingacademicallyrigoroussecondaryschools.25

Whilesignificantlymorerespondentsratedstandardizedtestsasconsiderablyimportant(58percent)intheadmission

process,industryexpertshaveagreedontheirexclusionfromanyevaluativemeasuresofeducationalsystems.The

two principle test developers, the College Board and ACT, discourage the use of their test data as measures of

educationalquality.TheNACACTestingCommission,madeupof19admissionandcounselingprofessionalsfrom

bothpublicandprivateinstitutions,concluded“thatthecontinueduseofadmissiontestscoresasaranking-related

measurecreatespressureonadmissionofficestopursueincreasinglyhightestscores.”

BecauseUSNWRmustrelyonquantifiablemeasureslikestandardizedtestdataandclassranktoofferanumerical

schoolrating,intrinsicethicaldilemmasarisefromthecollaterallyproducedincentives.Astheprevalenceofrankings

increases,sodoestheirpressuresonthepolicydecisionsofpostsecondaryadministrators.Admissiondeansand

directorsarefacedwiththechoiceof leadingtheircollegeoruniversityaccordingto institutionalphilosophyor to

rankingmethodologies.

TheimpactoftheLawSchoolAdmissionTest(LSAT)onlawschooladmissionpolicyprovidesatemplateforpolluting

theholisticadmissionprocess.LawschoolshavebeenclingingprecipitouslyclosetotheirrankingintheUSNWR

guide,andoneoftheeasiestmethodstoimproveaninstitution’sstandingintheannuallististhroughthemedian

LSATscore.“Notonlyisittangibleandinternalinawaythattheothervariablesarenot,itiscertain,andthatcertainty

makesitimportantinwaysdifferentfromtheothervariables.”26Wheninstitutionsneedabumpofseveralpointsin

theirmedianscore,theycanquicklyturntheirattentiontonextyear’sapplicantpool.Lendingunduesignificanceto

testscoresinanevaluativeformulaencouragesthesubjectsoftheevaluationtodothesame.

While test scores provide a solution to the largely immeasurable metrics of the college experience, rankings

publicationswouldbeill-advisedinusingthemfortheirratings.Sincethereisastandarderrorofmeasurementfor

standardizedtests,usingthemcomparativelyinarankingformulaisreckless.Thestandarderrorofmeasurementfor

theSATMathematicsandCriticalReadingsectionsis30pointsandabout1pointfortheACTcompositescore.The

CollegeBoard’sGuidelinesontheUsesofCollegeBoardTestScoresandRelatedDataclearlystatethatthepractice

of“makingdecisionsaboutotherwisequalifiedstudentsbasedonlyonsmalldifferencesintestscores”shouldbe

avoided.Becausetherankingspublicationsmeasurethousandsofpostsecondaryinstitutions,theyfacetheveryreal

possibilityofcharacterizinginstitutionsbasedonstatisticallyinsignificantscoredifferences.

Reputational Survey

ThereputationalsurveycomprisesasizeableproportionoftheoverallUSNWRranking.ForNationalUniversitiesand

NationalLiberalArtsColleges,thereputationalsurveycarriesa22.5weighting,makingitthemostinfluentialcategory.

ForRegionalUniversitiesandRegionalColleges,itrepresentsone-quarteroftheoverallrating,whichisonlyequaled

bygraduationandretentionrates.InthecontextofUSNWR’srankinghistory,theweightingseemsappropriate.The

veryfirstattemptfromUSNWRtorankcollegesin1983reliedentirelyonareputationalsurvey.The“BestColleges”

guide,however,hasgrowninrelevancesincethen,increasingitssalesofitsregularcollegerankingissuebyover

2millioncopies in its first15years.27Theannual issuesoonoutgrewa standalone reputational survey, and the

publicationbeganupdatingitsmethodology.

24NACACAdmissionTrendsSurvey,2009.25Ehrenberg,RonaldG.,“ReachingfortheBrassRight:TheU.S.News&WorldReportRankingsandCompetition,”TheReviewofHigherEducation,2002,Vol.26.2.26Johnnson,AlexM.,Jr.,“TheDestructionoftheHolisticApproachtoAdmissions:ThePerniciousEffectsofRankings,”IndianaLawJournal,2006,Vol.81.309.27McDonough,1998.

13

Secondaryandpostsecondaryprofessionalsareoverwhelminglyskepticalofthereputationalsurveycomponentof

theUSNWRrankingsmethodology.Inthecommittee’ssurveyofNACACmembers,onlyfivepercentofrespondents

called the peer assessments a “good indicator” of college quality. By comparison, nearly 40 percent of NACAC

membersagreedthatgraduationandretentionrates,similarlyweightedfactorsintheUSNWRrankings,weregood

indicatorsofcollegequality.

Thepeerassessmentsareinherentlysubjective,whichmightexplainthelowopinionratinginthemembersurvey

comparedtoaquantitativefactorlikegraduationrates.TheNORCreviewoftheUSNWRundergraduaterankings

methodologyhighlightsthelimitationsofusingacademicreputationasaproxyforinstitutionalquality.Inadditionto

addressingthefactthatacademicexcellencealonedoesnotdefinetheentirespectrumofstudentneeds,thereport

alsopointedoutthat“itisgenerallyassumedthatreputationschangemoreslowlythanrealchangeininstitutions,

thusovervaluinginstitutionsthat,infact,maybedecliningandundervaluinginstitutionsthatareimproving.”Inother

words,thereputationalsurveydoesnotserveasagoodindicatorofinstitutionalqualityinagivenyearbecausethe

respondent’sassumptionsmaybebasedonpastevaluations.

Whilesomeinstitutionshaveexperiencedpositionalmobilityintherankingsfromyeartoyear,whetherfromachange

ininstitutionalpolicyorachangeintheUSNWRmethodology,therelativelystaticnatureofthetoptierisrevealing

of attitudinal trendsofuniversity administrators. In theUSNWR lawschool rankings,whichalsoweight thepeer

assessmentsat25percent28,nineoutof10institutionsfrom1987to2004remainedinthetop10.29Assigningthe

largestoroneofthelargestweightstoameasurelikethereputationalsurveyisdangerousbecausetheresponses

submitted toUSNWRmay in factbepartofaself-perpetuatingsystem.Theschools rankedhighly in thewidely

distributed“BestColleges”publicationbenefitfromthereflexivenatureofthereputationalsurvey,whileschoolsin

thelowertierarepenalizedbasedonprevious,andpotentiallyoutdated,critiques.This“echoeffect”tendstopullthe

rankinginthedirectionofthepreviousyear’sassessment.30

Onemajorflawinincludingapeerassessment inarankingsmethodologyinvolvestheinabilityofrespondentsto

identifymeaningfulchangeatotherinstitutions.AsstatedintheNORCevaluation,“Thegreatvariancebothacross

andwithininstitutionsmakesitverydifficulttogetconsensusonqualitycriteriaoronmeasuresforundergraduate

programsingeneral,orevenforgroupsofcollegesoruniversitiesthatmightappearsimilar.”Withnoclearmethod

of identifyingeducationalquality, reputational survey respondents are encouraged to focus solely onmoreeasily

identifiablemetrics,likestandardizedtests.AsWilliamHendersonandAndrewMorrissaddressintheirresearchon

changesinmedianLSATscores,thefluctuationsinvisiblefactorslikestandardizedtestshavemoreofanimpact

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

Goodindicator

Fair indicator Poorindicator

Not anindicator

Figure 5. Peer Assessments

College

High School

Total

28Inthelawschoolrankings,onlyselectivityequalsthepeerassessmentsintermsofweight.Thelawschoolrankingsalsoincludeanassessmentscorebylawyersandjudgesweightedat15percent,whichbringsthetotal“qualityassessment”to40percentofthetotalranking.29HendersonandMorriss,2006.30Stake,2006.

14

on academic reputation than other intangibles like curriculum improvements. Therefore, the reliance on peer

assessmentsplacesadditionalweightontheselectivityrating.Onemightaskwhetherstandardizedtestsdatastarted

toaffectreputationsorreputationsstartedtoaffectselectivity,andultimatelystandardizedtests.Sincetherankings

publicationshavepervadedtheconsciousnessofthecollegeadmissionculture,itnolongermatterswhichcamefirst.

Thereputationshavebeenfirmlyestablishedthroughannualpublication.

Common Misconceptions

Thecommitteefoundthattherewassomemisunderstandingofrankingmethodology,aswellassomeperceptionsthat

werenotentirelyaccurate,amongsurveyresponsesandinconversationswithmembers.Respondentsoccasionally

citedtheneedformeasuresthatarecurrentlyincludedintheUSNWRmethodologyandcriticizedtheinclusionof

criteriathathavebeendroppedorchanged.Forexample,onerespondentstated,“Ithinkthe4yeargraduationrate

isabsurd,”when,infact,USNWRusesasix-yeargraduationrate.Similarly,somerespondentscitedtheneedfor

ameasurethattakesincomingstudentcharacteristicsintoaccountwhencomparinginstitutions’graduationrates.

Suchameasure iscurrently included inUSNWR’smethodologyunder thename“graduationrateperformance.”

Yieldwasrepeatedlycitedasasourceofnegativepressureoninstitutionsdespitethefactthatitisnolongerincluded

intherankingsmethodology.Respondentsalsooccasionallylistedretentionrates,student-to-facultyratios,andthe

percentagesoffacultywithterminaldegreesascriteriathatshouldbeincludedintherankingsmethodology.Though

thesefactorsarealreadyincludedintheUSNWRrankingsmethodology,themisunderstandingamongmembersmay

beinpartattributabletotheirrelativelyminimalweight.

15

Distorting Effects and Professional PracticeInstitutional Responses to the Rankings

Inareportontheeffectofrankingsinhighereducationpolicymaking,theInstituteofHigherEducationPolicy(IHEP)

noted:

Rankingshavethepotentialtoshiftinstitutionalbehaviorsinwaysthatmaynegativelyaffectpolicygoals.

Rankingscreateincentivesforinstitutionstotakeactionsdesignedtoimprovetheirpositions.Thisreactivity

createsconditionsinwhichinstitutionsrespondtotheconceptofeducationalqualityembeddedinrankings,

whichisnotalwaysalignedwithpublicpolicygoals,suchasequityanddiversity.31

Figure6demonstratesthatanoverwhelmingmajority(95.1percent)ofNACACmembersbelievethattheU.S.News

&WorldReportrankings“putpressureoninstitutionstoinvestinstrategiesandpracticesprimarilyforthepurposeof

maintainingorstrengtheningpositionintherankings,”eitherconsistentlyoroccasionally.

Highschoolmembersaremoresuspiciousofinstitutionalresponsestotherankings.Nearlytwo-thirds(63.6percent)

ofhighschoolrespondentsbelievethattherankings“consistently”putpressureoninstitutions,comparedtoonly

46.5percentofcollegerespondents.

Morethan300NACACmembersofferedcommentsonthisquestioninadditiontotheirmultiplechoiceresponses.

Themostcommonthemesintheopenendedresponsesaddsubstancetothegeneralbeliefinstitutionsandschools

arepressuredtomakeprogrammaticchangesineffortstoimprovetheirrankings.Commonthemesincluded:

• Manipulatingnumbers—Manymembersbelievethatschoolsmanipulatethedatathatisusedtocalculate

the U.S. News & World Report rankings, especially admit and yield32 rates, with wait lists, fast-track

applications,andearlydecisionprograms.

• Outsidepressure—Memberscommonlyreportedbeingpressuredbytheirinstitution’spresidents,trustees,

andfacultytoadoptstrategiesthatwouldincreasetheirrank.

• Benefits—Somemembersarguedthatthepressuretoimproverankingscanbenefitschools,colleges,and

studentsbyencouragingpoliciesthatimprovecertainstudent-centeredfeatures,includingretentionrate

andclasssize.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Yes, consistently Yes, occasionally No

Figure 6. Puts Pressure on Institutions to Invest in Strategies to Maintain/Improve Ranking

College

High School

Total

31Sponsler,2009.32YieldratesarenolongerusedintheU.S.Newsrankingsformula.

16

IncontrasttothedatashowninFigure6,54.1percentofNACACmembersrepresentingcollegesreportedthattheir

particularinstitutionsdonotmakeanyprogrammaticchangesbasedontherankings,asseeninFigure7.Because

theU.S.News&WorldReporthighschoolrankingsarelessprominentandinfluentialthanthecollegerankings,only

responsesfromNACACmembersrepresentingcollegesarediscussedforthisquestion.33

Very few NACAC college members (7.6 percent) report that their institutions consistently “make programmatic

changesat least inpartbecauseof their influenceontherankings.”Overone-thirdofcollegerespondents(38.4

percent)reportthattheirparticularinstitutionsdosooccasionally.ComparingFigures6and7yieldsaninteresting

contrast.Collegerespondents’beliefsthatinstitutionsare“gaming”therankingsgenerallyseemstoapplytoother

colleges,whereastheyarelesslikelytoperceivetheirowninstitutionasmanipulatingtheprocess.TheCommitteewill

explorethisfindingfurtherastheymeetandshareinformationwithmembersoverthecomingmonths.

Useful to College and University Recruiting Efforts?

AstheIHEPreportnotes,“[t]heuseofrankingsbypostsecondaryinstitutionshascontributedtotheirpopularity.”34

Indicativeofthecomplexrelationshipbetweenrankingsandinstitutions,thediversificationofdistinctionsconferred

byUSNWRhashadthedouble-edgedeffectsofaddressing(albeitonlypartly)concernsaboutaone-sized-fitsall

rankingandaffordingmoreinstitutionstheopportunitytopromotetheirrankingtothepublic.

ThemajorityofNACACmembersagreedwiththestatement,“U.S.Newsrankingsareusefultocollegeanduniversity

recruitingefforts.”Collegeswererelativelyevenlydividedonthisquestion,as55.6percenteithersomewhatagreed

oragreedand44.4percenteithersomewhatdisagreedordisagreed.Nearly73percentofhighschoolcounselors,

ontheotherhand,eithersomewhatagreedoragreedthattherankingsareusefultocollegeanduniversityrecruiting

efforts(Figure8).

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Yes, consistently Yes, occasionally No

Figure 7. School or Institution Makes Programmatic Changes Because of Rankings,

College Results Only

33Thehighschoolmemberresponsesforthequestionregardingthepromotionofrankingswereasfollows:6.7percentconsistentlymakechangesbasedontherankings,20.4percentoccasionallymakechangesbasedontherankings,and72.9percentdonotmakeanychangesbasedontherankings.34Sponsler,2009.

17

Rankings Create Confusion for Students and Families?

AlargemajorityofrespondentsfromallgroupsagreedthattheU.S.Newsrankingscreategreatconfusionforstudents

and families interested incollege information.Overall,83.4percentof respondentsagreedor somewhatagreed,

versusonly16.7percentofrespondentwhoexpressedsomelevelofdisagreement.Highschoolcounselors(86.6

percent)weremostlikelytosuggestthattherankingscreateconfusionforstudentsandfamilies(Figure9).

Privatehighschoolcounselorswereslightlymorelikelytobelievethattherankingscreateconfusionforstudentsand

families(Figure10).

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 9. Rankings Create Confusion for Students and Families

College

High School

Total

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 10. Rankings Create Confusion for Students and Families, HS Results Only

Public

Private

0.0%5.0%

10.0%15.0%20.0%25.0%30.0%35.0%40.0%45.0%50.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 8. Rankings Helpful for College and University Recruiting Efforts

College

High School

Total

18

Do Rankings Encourage Counter-Productive Behavior Within Colleges and Universities?

An overwhelming majority of the survey respondents (87 percent) either “somewhat agree” or “agree” that the

U.S.News&WorldReportrankingsencouragecounter-productivebehaviorwithincollegesanduniversities.High

school respondents were most likely to either “agree” or “somewhat agree” (89.4 percent) that rankings cause

counterproductivebehavioratcollegesanduniversities, thoughcollege respondentsweresimilarly inclined (84.7

percenteitheragreedorsomewhatagreed)(Figure11).

Amongcolleges,admissionofficersfrompublicinstitutionswereslightlylesslikelytobelievethattherankingscaused

counter-productivebehaviorthanadmissionofficersatprivateinstitutions(Figure12).

Rankings and Recruiting

AmajorityofNACACmembersbelievetherankingsare,generallyspeaking,usefultocollegeanduniversityrecruiting

efforts.Aspecificexampleoftherankings’utilityforcolleges—thepromotionofaninstitution’srankinitsmarketing

materials—isshowninFigure13.BecauseamajorityofNACACmembersrepresentinghighschoolsarenotpresented

atallintheU.S.News&WorldReportrankings,onlyresponsesfrommembersrepresentingcollegesarediscussed

forthisquestion.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 11. Do USNWR rankings cause counter-productive behavior at colleges and universities?

College

High School

Total

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 12. Do USNWR rankings cause counter-productive behavior at colleges and universities?

College Results Only

Public

Private

19

Ofthe88.6percentofcollegememberswhoareincludedintherankings,only2.8percentreportbeingpresented

unfavorably.Amongtheremaining79.8percentofcollegememberswhoarepresentedfavorablyintheU.S.News

&WorldReportrankings,overtwo-thirds(71.3percent)promotetheirrank,thoughmostdosoinalimitedfashion.

Amongcolleges,publicinstitutionswereslightlymorelikelytopromotetheirrankingthanprivateinstitutions(Figure

14).

Rankings in Counseling and Admission Offices

NACACmemberswereaskedwhethertheyspend“agreatdealoftime,”“sometime,”or“notime”discussingor

answeringquestionsabout theU.S.News&WorldReport rankings.AsFigure15shows,amajorityofmembers

spendatleastsometimediscussingU.S.Newsrankingswithstudentsandfamilies.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Yes,promotewidely

Yes,promote in

a limitedfashion

No, do notpromote

Notpresentedfavorably

Notpresented

at all

Figure 14. Promote Rankings in Marketing Strategy, College Results Only

Public

Private

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Yes, promotewidely

Yes, promotein a limited

fashion

No, do notpromote

Not presentedfavorably

Not presentedat all

Figure 13. Promote Rankings in Marketing Strategy, College Results Only

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Great deal of time Some time No time

Figure 15. Time Spent Discussing Rankings with Students and Families

College

High School

Total

20

Taking the Next StepMisleading Conclusions About Institutional Quality?

Anoverwhelming89.1percentofallrespondentsagreedorsomewhatagreedthattheU.S.News&WorldReport

rankingsoffermisleadingconclusionsaboutinstitutionalquality.AsFigure16shows,opinionsdidnotvarysubstantially

betweencollegeandhighschoolmembers.

Disaggregatedresultsamonghighschoolsagainrevealaslightdifferencebetweenpublicandprivatehighschool

opinionsabouttheconclusionsdrawnbytherankingsaboutinstitutionalquality,thoughthedifferencesinthiscase

areslight(Figure17).

Theseresults,combinedwiththeresponsestothequestionabouttheaccuracyofthetitle,“America’sBestColleges,”

offertheclearestindicationofwhereNACACmemberconcernsaregrounded.35

The Case for Personalized Lists

“Thepublicationofasinglerankingprovidesreaderswithasummarythatmaybemisleadingandfailstoproperly

accountforpersonalpreferenceanduncertaintyinweights.”36Asreferencedearlier,the“realproblem,”according

toanumberofacademicanalysesoftherankingsandamongmanysurveyrespondents, isUSNWR’s“arbitrary”

assignmentsofweightstoeachcategoryandsubcategory.37

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 16. USNWR Rankings Offer Misleading Conclusions about Institutional Quality

College

High School

Total

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

Agree Somewhatagree

Somewhatdisagree

Disagree

Figure 17. USNWR Rankings Offer Misleading Conclusions about Institutional Quality, HS

Results Only

Public

Private

35Therewasastrongandstatisticallysignificantcorrelation(-.521,p<.01)betweenresponsestothefeelingthermometerquestionandthequestionabouttheaccuracyofthetitle“America’sBestColleges.”36HugginsandPachter,2008.37Ehrenberg,RonaldG.,“MethodorMadness?InsidetheU.S.News&WorldReportCollegeRankings,”JournalofCollegeAdmission,2005,Vol.189.

21

In 2004, Marguerite Clarke, at the time an Assistant Professor of Research at Boston College, offered four

recommendationsbasedonherresearchworthrepeatinginthiscommittee’sreport:

1) TheeditorsatUSNWRneedtoreexaminethevalidityoftheindicatorsandweightsusedforeachranking.

2) Theeditorsshouldrefrainfromusinganoverallscoretorankschools.Thesescoresarenotreliableenough

toallowfinedistinctionsamongschools.Amoredefensibleapproachwouldbetouseschools’overallscores

toplacethemin“quality”bands,listingthemalphabeticallywithineachband.

3) USNWRcouldallowconsumerstocreatetheirownrankingformulas.

4) Consumersneedtobecomemorecriticaloftheassumptionsunderpinningrankingsaswellasthevalue

systembehindthechoiceofindicatorsandweights.38

TheCommitteebelieves that the researchanddiscussionabout the imprecisionofordinal rankings in the“Best

Colleges”listhasreachedapointwherenotproactivelyacknowledgingthelimitationsoftherankingformulathrough

vigorousconsumereducationandflexible‘ranking’optionsrisksmisleadingconsumersandhascompromisedthe

journalisticintegrityofthepublication.

TheCommittee’ssurveyresearchclearlyindicatesthatcollegeadmissioncounselingprofessionalsfindthedataand

informationavailable through theUSNWRrankingspublicationuseful.NACACmemberswereasked to rank the

featuresofthe“America’sBestCollege”publicationfromonetoseven,onebeingthemosthelpfulandsevenbeing

theleasthelpful.Perhapsreflectingtheirdisapprovaloftheprocessofplacingthingsinordinalrank,aboutathird

ofmembersrated,ratherthanranked,thefeatures(i.e.usedthesamenumberfortwoormorefeatures).Forthe

purposesofthisreport,theirresponseswereexcluded(Figure18).

Articlesonpreparingforandnarrowingthecollegesearch,aswellasthoseonhowtopayforcollege,receivedthe

highestmeanscores,withaveragemeansscoresof2.51,3.1,and3.19, respectively.Unsurprisingly, theannual

rankingsofcollegesarebelievedtobetheleasthelpfulfeatureofthepublicationwithanoverallmeanscoreof5.56.

NACACmemberswerelukewarmtorecommendingtherankingsifstudentscouldcreatetheirownweight,possibly

duetolingeringsuspicionordislikeforrankingsingeneral.Ifgiventheoptiontoselectindividualweightsforvarious

elementsoftheU.S.News&WorldReportrankings,mostmembers(58.6percent)wouldbeneithermoreorless

inclinedtorecommendtherankingstotheirstudents.Overaquarter(26.6percent)ofmemberswouldbemorelikely

torecommendtherankings,giventheoptiontochoosetheirownweights(Figure19).

1

2

3

4

5

6

7

Articles onPreparingfor College

Search

Articles onNarrowing

CollegeSearch

Articles onHow toPay forCollege

Directoryof Colleges

Articles onAdjustingto College

Articles on"How toGet In"

AnnualRankings

of Colleges

Mos

t Hel

pful

Le

ast H

elpf

ul

Figure 18. Member Rank of Features in Rankings Publicaton

College

High School

Total

38Clarke,Marguerite.“WeighingThingsUp:ACloserLookatU.S.News&WorldReport’sRankingFormulas,”CollegeandUniversity,2004,Vol.79.3.

22

However, the Committee believes that offering consumers the opportunity to create their own lists by weighting

dataelementsaccordingtotheirpreferenceswouldsimultaneouslyinformthepublicthatsuchrankingsarehighly

objective,andallowUSNWR theopportunity to feature listscreatedbystudents,counselors,colleges,andother

consumersaccordingtocriteriatheyhavedevised.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

More likely torecommend

Neutral Less likely torecommend

Figure 19. If Students Could Create Own Weights...

College

High School

Total

23

Ad Hoc Committee’s RecommendationsGiventhelongandpunctuatedevolutionofrankingsintheU.S.,theCommitteebelievesitisimportanttopromote

recommendationsforpositivechangethatfollowthedirectionthatadaptationalpathsseemtolead.Moreover,the

Committeebelievesthatprofessionalpractice isanobligationandaprerequisiteforcollegeadmissioncounseling

professionals,whoareresponsiblefortheinterpretationofrankingstostudentsandfamilies,themaintenanceand

constructionof rankings throughparticipation in thesurveys,and theperpetuationof rankingsbyusing them in

professional settings.As such, theCommittee’s recommendations aredirected at bothNACACand theUSNWR

rankingspublication.Ourintentisalsoforseveraloftheserecommendationstoapplytoundergraduateinstitutional

rankingsofanytype.

Tothatend,theCommitteewishestodrawattentiontoasetofrecommendationsputforthbytheAmericanAssociation

ofCollegiateRegistrarsandAdmissionsOfficers(AACRAO),whichconvenedaspecialcommitteeatalmostthesame

timeasNACAC’sAdHocCommittee.(SeeAppendix)

For NACAC

Develop professional education resources for members about rankings.

• The committee found that therewas somemisunderstanding of rankingmethodology, aswell as some

perceptionsthatwerenotentirelyaccurate,amongsurveyresponsesandinconversationswithmembers.

• Thecommitteerecognizedtheneedfortrusted,accurateinformationtoconveytostudentsandfamilies

aboutrankings.

Work with education publishers and data outlets to encourage development of do-it-yourself lists for consumers.

• Asubstantialamountofacademicresearchsuggeststhatrankingsarehighlysensitivetotheunderlying

methodologies.Accordingly,consumersshouldbeabletoestablishtheirown‘weightsandmeasures’that

canresultinliststhatproducebetter“fit”forstudents.

For Rankings Publications

Remove the “class rank” and “standardized testing” metrics from rankings methodologies in favor of factors that

measure student satisfaction and engagement.

• The selectivity rating counts for 15 percent of the overall ranking, with high school class standing and

standardizedtestscoresmakingup90percentofthatrating.

• Only16percentofAdmissionTrendsSurveyrespondentsattributedconsiderableimportancetoclassrank.

• TheNACACTestingCommissionrecommendedthatU.S.News&WorldReporteliminatetestscoresasa

measureofinstitutionalquality,andhighlightedstatementsfromtheCollegeBoardandACTdiscouraging

theincorporationofstandardizedtestsinrankingmethodologies.

Reduce the weight of the reputational survey.

• Peer assessments make up 22.5 percent of the U.S. News National and Liberal Arts rankings and 25

percentoftheregionalrankings.

• Thepeerassessmentsarehighlysubjectiveandmaybedisproportionallyinfluencedbysocialfactorsthat

donotmeasureinstitutionalquality.

• In a NACAC survey, only five percent of respondents agreed that the peer assessments serve as good

indicatorsofinstitutionalquality.

Encourage emphasis on fit through customized rankings.

• NACACconsistentlyencouragesthefocusonfit incollegecounseling.Auser-basedmethodologywould

allowstudentstoexamineinstitutionsbasedontheirownnotionsofinstitutionalquality.

Continue to evolve rankings methodologies through the association’s communication channels.

• NACACwillprovideeducationalinformationthatsupportstheproperinterpretationsofdatafoundinranking

reports.

24

AppendixOnNovember7,2010,theAmericanAssociationofCollegiateRegistrarsandAdmissionsOfficers(AACRAO)hosted

a national executive forum during its annual Strategic Enrollment Management Conference. Titled “The College

RankingsDebateandFutureImplications:AssessingtheValueofanInstitution’sUndergraduateExperience,”the

executive forumwasdesigned to stimulate thoughtful discussionof andconsensusbuildingaroundhowbest to

providethepublicwithmeaningfuldataaboutinstitutions’dedicationto,andsuccessat,studentlearning.Thus,the

desiredoutcomewasaclearsetofprinciplesforcollegeassessmentsthatwouldleadtomeaningfulreformsofcurrent

rankingprograms.ThediscussionleadersandwhitepaperauthorswereGeorgeKuh,Chancellor’sProfessorofHigher

Education at IndianaUniversity andDirector of theCenter forPostsecondaryResearch and theCollegeStudent

ExperienceQuestionnaireResearchProgram;JohnPryor,DirectoroftheCooperativeInstitutionalResearchProgram

(CIRP)andManagingDirectoroftheHigherEducationResearchInstitution(HERI)atUCLA;WatsonScottSwail,CEO

oftheEducationalPolicyInstitute(EPI);JayGoff,ViceProvostofEnrollmentManagementatMissouriUniversityof

ScienceandTechnology;BobBontrager,SeniorDirectorofAACRAOConsultingandStrategicEnrollmentManagement

Initiatives;andand[sic]JasonLane,AssistantProfessorandSeniorResearch[sic],InstituteforGlobalEducation

PolicyStudies(IGEPS)atSUNY-Albany.Approximately75practitionersparticipatedintheforum.Participantswere

askedtodiscussthechallengesofcollegeanduniversityrankingprograms.Thesummarythatfollowsprovidesan

overviewoftheexecutiveforum’sprimaryareasofdiscussion.Theconcludingprinciplesareintendedtohelpguide

developmentofanassessmentsystemthatwouldbetteraidstudentsinthecollegeselectionprocess.

AACRAO-principlesforthecreationofaratingsystem

1) RatingwithoutRanking

2) RecognizeInstitutionalDifference

3) CreateCommonPost-AdmissionMilestones

4) TransparencythroughAgreementonDefinitions,DataInstruments,andCollectionProcesses

5) AccountfortheValue-AddedFeaturesofanEducationalExperience

6) GovernancebyaNon-ProfitEntity

AACRAO-goalsofthenewapproach

• Createpublicunderstanding thathundredsofqualitycollegesanduniversitiesexistand that theymeet

students’learninganddevelopmentalneedsindifferentways.

• Focusonvalue-addedoutcomesbytypesandnumbersofsuccessfulstudentsandbygraduates’satisfaction

with their overall undergraduate experience. This should mitigate the focus on traditional measures of

prestigeandselectivity.

• Provide students, parents, and employers with comparative learning data and information about the

particular learningobjectives/skillsetsthatareemphasized,therebymotivatingschoolstoembracetheir

missionsratherthanaligntheireffortswithcurrentrankingsystems.

• Provideameaningfulalternativetoexistingrankingsystems.

(Availableonlineathttp://www.aacrao.org/Files/Publications/CUJ8604_WEB_L.pdf)

25

BibliographyCarey,Kevin.“CollegeRankingsReformed:TheCaseforaNewOrderinHigherEducation.”Education Sector.Education

Sector,Sept.2006.Web.<http://www.educationsector.org/sites/default/files/publications/CollegeRankingsReformed.

pdf>.

Caron, Paul L. and Rafael Gely. “Dead Poets and Academic Progenitors: The Next Generation of Law School

Rankings.”Indiana Law Journal81.1(2006):1-12.Web.

Clarke,Marguerite.“WeighingThingsUp:ACloserLookatU.S. News & World Report’sRankingFormula.”College

and University79.3(2004):3-9.Web.

Ehrenberg,RonaldG.“MethodorMadness? InsidetheU.S. News & World ReportCollegeRankings.”Journal of

College Admission189(2005):29-35.Print.

---“ReachingfortheBrassRight:TheU.S. News & World ReportRankingsandCompetition.”The Review of Higher

Education26.2(2002):145-162.Project MUSE.Web.

Gladwell,Malcolm.“TheOrderofThings.”The New Yorker14Feb.2011:68-75.Web.<http://www.newyorker.com/

reporting/2011/02/14/110214fa_fact_gladwell>.

Griffith,AmandaandKevinRask. “The Influenceof theUSNewsandWorldReportCollegiateRankingson the

MatriculationDecisionofHigh-AbilityStudents:1995-2004.”Economics of Education Review26(2007):244-255.

ScienceDirect.Web.

Henderson,WilliamD.andAndrewP.Morriss.“StudentQualityasMeasuredbyLSATScores:MigrationPatternsin

theU.S. NewsRankingsEra.”Indiana Law Journal81.163(2006):163-204.Web.

Huggins, Peter and Lior Pachter. “Selecting Universities: Personal Preference and Rankings.” Cornell University

Library.CornellUniversity,7May2008.Web.<http://arxiv.org/PS_cache/arxiv/pdf/0805/0805.1026v1.pdf>.

Johnson,AlexM.,Jr.,“TheDestructionoftheHolisticApproachtoAdmissions:ThePerniciousEffectsofRankings.”

Indiana Law Journal81.309(2006):309-358.Web.

Lane,JasonE.“Rating(NotRanking)theUndergraduateExperience:PrinciplesfromaNationalDiscussion.”College

and University86.4(2011):2-7.Web.

McDonough,PatriciaM.,etal.“CollegeRankings:DemocratizedCollegeKnowledgeforWhom?”Research in Higher

Education39.5(1998):513-537.JSTOR.Web.13April2010.

Meredith,Marc.“WhyDoUniversitiesCompeteintheRatingsGame?AnEmpiricalAnalysisoftheEffectsoftheU.S.

News and World ReportCollegeRankings.”Research in Higher Education45.5(2004):443-461.Web.

Myers, Luke and Jonathan Robe. “College Rankings: History, Criticism and Reform.” Center for College

Affordability and Productivity. Center for College Affordability and Productivity. Mar. 2009. Web. <http://www.

centerforcollegeaffordability.org/uploads/College_Rankings_History.pdf>.

National Opinion Research Center. “The NORC Report on U.S. News and World Report.” Washington Monthly.

WashingtonMonthly,n.d.Web.<http://www.washingtonmonthly.com/features/2000/norc.html>.

26

Posner,RichardA.“LawSchoolRankings.”Indiana Law Journal81.13(2006):13-24.Web.

Rauhvargers,Andrejs.“GlobalUniversityRankingsandTheirImpact.”EUA.EuropeanUniversityAssociation,2011.

Web.<http://www.eua.be/pubs/Global_University_Rankings_and_Their_Impact.pdf>.

Sanoff,AlvinP.“TheU.S. NewsCollegeRankings:AViewfromtheInside.”College and University Ranking Systems:

Global Perspectives and American Challenges.Ed.InstituteforHigherEducationPolicy.InstituteforHigherEducation

Policy,2007.9-22.Web.<http://www.ihep.org/assets/files/publications/a-f/CollegeRankingSystems.pdf>.

Sauder, Michael and Wendy Nelson Espeland. “Rankings and Reactivity: How Public Measures Recreate Social

Worlds.”American Journal of Sociology113.1(2007):1-40.Web.

---“StrengthinNumbers?TheAdvantagesofMultipleRankings.”Indiana Law Journal81.205(2006):205-227.

Web.

- - -“The Discipline of Rankings: Tight Coupling and Organizational Change.” American Sociological Review 74

(2009):63-82.Web.

Sponsler,BrianA. “IssueBrief: TheRoleandRelevanceofRankings inHigherEducationPolicymaking.” IHEP.

Institute forHigher EducationPolicy, Sept. 2009.Web. <http://www.ihep.org/assets/files/publications/m-r/(Issue_

Brief)_The_Role_and_Relevance_of_Rankings.pdf>.

Stake,JeffreyEvans.“The InterplayBetweenLawSchoolRankings,Reputations,andResourceAllocation:Ways

RankingsMislead.”Indiana Law Journal81.299(2006):229-270.Web.

U.S.NewsStaff.Best Colleges 2012: About the Rankings/Methodology.U.S.News&WorldReport,12Sept.2011.

Web.<http://www.usnews.com/education/best-colleges/articles/2011/09/12/best-colleges-2012-about-the-rankings-

methodology?s_cid=related-links:TOP>.

Walleri,R.DanandMarshaKMoss,eds.Evaluating and Responding to College Guidebooks and Rankings.San

Francisco:Jossey-BassPublishers,1995.Print.

Webster,DavidS.“ReputationalRankingsofColleges,Universities,andIndividualDisciplinesandFieldsofStudy,

FromTheirBeginningstothePresent.”Higher Education: Handbook of Theory and Research Volume VIII.Ed.John

C.Smart.Bronx,NewYork:AgathonPress,1992.234-304.Print.