uc berkeley admissions in 2015 and 2016: an analysis of...

UCBerkeleyAdmissionsin2015and2016:AnAnalysisoftheRoleofLettersofRecommendationandAugmentedReview

July13,2016

JesseRothstein,ProfessorofPublicPolicy&Economics

ExecutiveSummaryUCBerkeleymademanychangestoitsundergraduateadmissionprocessesin2016,includingtheadditionoflettersofrecommendation(LORs)formanyapplicantsandtheeliminationoftheseparate“AugmentedReview”(AR)pool.Thesechangeswereintendedinparttobetteridentifyhiddengemsintheapplicantpool–becausesomanymorestudentscouldbeaskedforLORsthanwereeverconsideredunderAR,ifLORswereanywherenearaseffectiveatidentifyingunderrepresentedstudentswhocouldsucceedatCalthenetimpactwouldbetoincreasediversity.Ontheotherhand,thechangesmighthaveraisedbarrierstoadmissionfordisadvantagedstudents,whomightnothaveaccesstoteachersorcounselorswillingandabletowritestronglettersandwhosehiddenstrengthsmightnothavebeenrecognizedwithoutAR.ViceChancellorKoshlandaskedmeinJunetoconductanindependentanalysisofBerkeleyundergraduateadmissions,focusingontheLORandARchanges.IhavedoneacademicresearchoncollegeadmissionsatUCandelsewhere,butIhaveneverbeeninvolvedwithBerkeleyadmissionsprocesses.Topreparethisreport,IconsultedextensivelywiththeadmissionsofficeandwiththeSenateCommitteeonAdmissions,Enrollment,andPreparatoryEducation(AEPE)inanefforttounderstandtheadmissionsprocess.Thisreportreflectsmyindependentanalysisandconclusions.Ireachtwomainconclusions.First,thenumberofapplicantsfromunderrepresentedgroups(low-income,first-generation,fromlow-APIhighschools,and/orunderrepresentedminorities)whowereadmittedrosein2016.Butbecausethenumberadmittedwhowerenotfromthesegroupsrosebyalargerproportion,theshareofadmittedstudentsfromunderrepresentedgroupsfellsomewhat.Second,neithertheadditionofLORsnortheremovalofARcontributedmeaningfullytothisdecline.Ifanything,askingforLORsraisedtherelativeadmissionsratesofapplicantsfromunderrepresentedgroups.IamunabletopreciselyidentifytheimpactoftheeliminationofAR–whilesomeestimatesindicatethatthisslightlyreducedadmissionsforthosewhowouldhavebeenconsideredviaARin2015,othersindicatezeroorevenapositiveeffect.Alloftheestimatesagreethattheimpactwassmallinanycase.Itisbeyondthescopeofthisreporttoidentifywhatdidcausetheshiftin2016.Mypreliminaryinvestigation,however,suggeststhatthedeclineintheshareofadmitsfromunderrepresentedgroupsisinlargepartastatisticalartifactduetotheexpandeduseofthewaitlistin2016.Therewerealsoreductionsintheadmissionschancesoftheunderrepresentedapplicantswiththestrongestnumericrecordsthatcannotbeattributedtothewaitlist.Futureinvestigationshouldfocusonunderstandingwhatinthescoringprocessharmedtheseapplicants.

2

IntroductionBerkeleymadeanumberofchangestoitsundergraduateadmissionsprocessesin2016:

• Itrequestedlettersofrecommendation(LOR)formanyapplicants.• Iteliminatedtheaugmentedreview(AR)pool.• Itshiftedfromapointsystemforscoringapplicationstoathree-category

(Yes/No/Maybe)ratingsystem.• Readersbeganscoringapplicantsonalistofholistic/non-cognitivefactors.• Everyapplicationwasreadtwice,whereinthepastmanywerereadonlyonce• Athirdread,bymembersofthefaculty,wasaddedformanyapplications.• Thewaitlistwasusedmuchmoreextensivelythaninthepast,andmanyapplicants

whoin2015wouldhavebeenadmittedorrejectedoutrightwereinsteadofferedpositionsonthewaitlistin2016.

Table1presentssimplesummariesofadmissionsoutcomesin2015and2016,intheupperpanelpoolingallin-stateandout-of-stateapplicantsandinthelowerpanelrestrictingattentiontoCaliforniaresidentsnotbeingrecruitedasathletes.OfparticularconcernisthedeclineintheshareofadmittedstudentsfromdisadvantagedbackgroundsorgroupsthatareunderrepresentedatBerkeley.InTable1andthroughoutthisreport,Iconsiderfourgroupsofsuchstudents:low-incomestudents(withfamilyincomesbelow$40,000);first-generationcollegestudents(thosewhoseparentsdidnotgraduatefromcollege);studentsfromdisadvantagedschools(withAPIindexesof5andbelow);andunder-representedminoritystudents(UREMs).Irefertothemcollectivelyas“underrepresented.”Iwasaskedtoconductananalysisofadmissionsdataforthe2015and2016cycles,focusingontheimpactoftheadditionoflettersofrecommendationandtheeliminationofaugmentedreviewontheadmissionofunderrepresentedstudents.Inalloftheanalysesbelow,IrestrictattentiontoCaliforniaresidentapplicantswhowerenotclassifiedintheadmissionsprocessasrecruitedathletes.ThelowerportionofTable1showsstatisticsfortheseapplicants.TheoverallcontextIbeginwithseveralpreliminariesnecessarytounderstandingtheimpactsoflettersofrecommendationandaugmentedreviewonadmissionsoutcomes.ThecompositionoftheapplicantpoolThefirstplacetolookforanexplanationforchangingoutcomesischangesintheapplicantpoolitself.Iftheapplicantsfromunderrepresentedgroupswereweaker,onaverage,in2016thanin2015,thiscouldaccountfortheoverallobservedchanges,evenwithoutachangeinpolicy,andwouldconfoundmyLORandARanalyses.UnderstandingthedistributionofapplicantstrengthisalsohelpfulasawayofgaugingwhichtypesofstudentswereaffectedbytheLORandARprograms.

3

Table1.Summaryofadmissionsoutcomes

Applicants

Admits

2015 2016

2015 2016

Allapplicants

Number 78,924 82,578

13,266 14,423

Admissionrate

16.8% 17.5%

Sharefromunderrepresentedgroups

Lowincome(<$40K) 24% 23%

19% 18%

Firstgeneration 17% 16%

13% 12%

LowAPI 13% 12%

12% 11%

UREM 23% 23%

18% 18%

Anyofthesefour 40% 38%

33% 31%

Californiaresidents,excludingrecruitedathletes

Number 45,570 45,626

8,570 9,610

Admissionrate

18.8% 21.1%

Sharefromunderrepresentedgroups

Lowincome(<$40K) 31% 30%

24% 22%

Firstgeneration 24% 23%

18% 15%

LowAPI 23% 22%

19% 16%

UREM 33% 34%

23% 23%

Anyofthesefour 53% 53%

42% 39%

Iconstructameasureofapplicantstrengthbyaggregatinganumberofavailablestudentcharacteristics.1My“admissionsscore”representsthepredictedprobabilitythatastudentwithagivensetofcharacteristicswouldhavebeenadmittedinthefirstround(i.e.,notoffthewaitlist,andnotthroughAugmentedReview),hadhe/sheappliedasaCaliforniaresidenttotheCollegeofLettersandSciencesin2015.2

1Thevariablesarethoseusedinthemodelusedbytheadmissionsofficetopredictapplicationreadscores,andincludemeasuresoftraditionalacademicstrength,measurescharacterizingthehighschool,(including,forexample,theextenttowhichthestudenttookadvantageoftheschool’sadvancedcourseofferings),andthreeofthefourdisadvantagemeasuresconsideredhere.Applicants’raceandethnicity,whichcannotbeconsideredinadmissions,isnotincluded.2IuseonlyL&SapplicantsbecausetheothercollegesmayweightcharacteristicsdifferentlythandoesLettersandSciences.Nevertheless,myL&S-basedscoreisnearlyperfectlycorrelatedwithascoreconstructedbasedonadmissionsintheCollegeofEngineering.

4

Twoaspectsofthisscoremustbeemphasized:First,itcapturesonlythequantitativecharacteristicsthatarecodedintheadmissionsoffice’sdatabase;readersseemoreinformation,andmayidentifyapplicantsasstrongerorweakerthanisindicatedbymyscore.Second,thecharacteristicsareweightedtobestpredictadmissionin2015.Theweightputondifferentcharacteristics–say,onhighSATscoresvs.takingalloftheAPcoursesofferedatyourhighschool–mightvaryfromyeartoyear,andindeedseemstohavechangedsomewhatin2016(asdiscussedbelow).Butevenwiththesecaveats,theadmissionsscoreneverthelesspresentsausefulsummary.Totakeoneexample,38%ofapplicantsin2016havescoresunder1%.Whileaveryfewofthesestudentsmighthavecharacteristicsnotinthedatabasethatmeritadmission,thisisquiterare;thevastmajorityofstudentsinthisgroupwouldnotbeadmittedundertheregular2015processes.Indeed,only1.8%ofthemwereadmittedin2015,and2.5%in2016.Figure1showsthedistributionofadmissionsscoresacrossCaliforniaresident,non-athleteapplicantstoallcollegesin2015.Thisisheavilyleft-skewed:Mostapplicantshaveverylowchancesofadmission,thoughthereareafewwhoaresostrongonthedimensionscapturedbymyindexthatitisrareforotherfactorstopreventthemfrombeingadmitted.

Figure1.Distributionofadmissionsscoresin2015forCaliforniaresidentapplicants

Becausetheoveralldistributionofadmissionsscoresissodominatedbyapplicantswithextremelylowchancesofadmission,Ifindithelpfultofocusonthosewhoaremorelikelytobeadmitted.Figure2showsthedistributionofadmissionsscoresforthosewhowereactuallyadmittedin2015(includingARadmitsandthoseadmittedoffthewaitlist),whileFigure3repeatsthisforthefourunderrepresentedgroupsandFigure4repeatsitforapplicantsnotfromthesegroups.

0.1

.2.3

.4Fr

actio

n

0 .2 .4 .6 .8 1Admissions score

Distribution of admissions score in 2015

5

Figure2.DistributionofadmissionsscoresforadmittedCaliforniaresidentapplicantsin2015

ItisapparentinFigure3thattheadmissionsscoredistributionisquiteskewedtotheleftforstudentsfromtheunderrepresentedgroups.Thisistrueeventhoughthepredictionmodelusedtogeneratetheadmissionsscoreincludesindicatorsforlowincome,firstgeneration,andlowAPI(butnotUREM)students.Evidently,manyofthestudentswhoareadmittedfromthesegroupsarepickedoutfromlargepoolswithsimilarobservablecredentialswhoarenotadmitted.Thisismuchlesstrueforstudentsnotfromthesegroups,forwhomthedistributionisshowninFigure4:Here,admittedstudentsaremuchmorelikelytohaveadmissionsscoresabove0.6.

Figure3.AdmissionsscoredistributionsforadmittedCaliforniaresidentapplicantsfromfourunderrepresentedgroupsin2015

0.0

1.0

2.0

3.0

4Fr

actio

n


Distribution of admissions score in 2015For those actually admitted

0.0

2.0

4.0

6.0

8Fr

actio

n


Low income

0.0

2.0

4.0

6.0

8Fr

actio

n


1st Generation

0.0

2.0

4.0

6.0

8Fr

actio

n


Low API

0.0

2.0

4.0

6.0

8Fr

actio

n


UREM

Distribution of admissions scores in 2015For those actually admitted

6

Figure4.AdmissionscoredistributionforCaliforniaresidentsnotfromunderrepresentedgroups,2015

Table2showssummariesofthedistributionofadmissionsscoresforCaliforniaresidentapplicantsandadmittedstudentsin2015and2016,bothforallapplicantsandforapplicantsfromtheunderrepresentedgroups.Itindicatesthatthedistributionofadmissionsscoreschangedsomewhatbetweenyears,withmorestudentswithverylowandveryhighadmissionsscoresin2016thanin2015.Theseroughlyoffseteachother,however,andaverageadmissionsscores,bothoverallandforapplicantsfromunderrepresentedgroups,werequitesimilarin2016asin2015.Overall,changesinthedistributionofobservablecharacteristicsamongapplicants,onitsown,wouldnotlikelyhaveproducedsubstantialchangesinapplicationoutcomes.

Table2.Distributionofadmissionsscoresin2015and2016 Californiaresidents,excludingrecruitedathletes

Allstudents

Underrepresentedgroups

Applicants

Admits

Applicants

Admits

2015 2016

2015 2016

2015 2016

2015 2016

Mean 0.17 0.17

0.49 0.45

0.13 0.13

0.42 0.40Fractionbelow1% 33% 38%

1% 3%

39% 45%

2% 5%

Fractionbelow5% 53% 57%

7% 12%

61% 64%

11% 15%

5thpercentile 0.00 0.00

0.04 0.02

0.00 0.00

0.02 0.0110thpercentile 0.00 0.00

0.08 0.04

0.00 0.00

0.05 0.03


0.22 0.14

0.00 0.00

0.15 0.1050thpercentile 0.04 0.03

0.49 0.43

0.02 0.02

0.38 0.33


0.76 0.75

0.14 0.13

0.67 0.6790thpercentile 0.61 0.65

0.89 0.90

0.45 0.49

0.85 0.87


0.93 0.94

0.67 0.72

0.91 0.9299thpercentile 0.94 0.95

0.97 0.98

0.90 0.92

0.96 0.98

0.0

1.0

2.0

3.0

4.0

5Fr

actio

n


Distribution of admissions score in 2015For those actually admitted

7

Figure5showstheshareofstudentsateachadmissionsscorelevelwhowereconsideredunderARin2015(solidline)oraskedforLORsin2016(dashedline).ThefeaturethatjumpsoutthemostisthattheLORprogramwasmassivelylargerthanARhadbeen–evenatthelowestadmissionsscores,whereARstudentsareconcentrated,theshareof2016studentsfromwhomLORswererequestedgreatlyexceedstheshareconsideredunderARin2015.Thesecondthingtonoticeisthat,whileARstudentswereconcentratedaroundthe0.3mark–thatis,thesestudentswere,basedonobservablesalone,muchweakerthantheaverageapplicant–LORswereusedmostamongstudentswithmuchhigheradmissionsscores,50-80%.Inthisrange,nearlyall2016applicantswereaskedforLORs,but10%orlesswereconsideredunderARin2015.

Figure5.ShareofCaliforniaresidentapplicantsconsideredunderARoraskedforletters

ThewaitlistBeyondtheAR/LORshift,anotherimportantchangein2016wasagreatlyexpandeduseofthewaitlist.Theshareofapplicantsofferedpositionsonthewaitlistnearlydoubled(from4.5%to8.6%)in2016.Manystudentsdeclinedtheseoffers–fully4%of2016applicantsdeclinedofferedpositionsonthewaitlist,ascomparedto1.5%in2015.Thisgreatlycomplicatescomparisonsof2015and2016outcomes,assomeofthestudentswhoturneddownpositionsonthewaitlistin2016wouldhavebeenadmittedoutrightin2015.ManyBerkeleyapplicantsarechoosingbetweenBerkeleyandotherexcellentuniversities,andmanywhoareacceptedwindupgoingelsewhere.In2015,lessthanhalfofadmittedstudentscametoBerkeley,andthissharewassmallerforstrongerapplicants.Inmanycases,studentswillhavealreadydecidedtoenrollelsewherebythetimeBerkeley’sinitialadmissionsoffersaremade.Consider,forexample,astudentadmittedelsewhereunderanearlydecisionprogram.Inprinciple,thisstudentmightwithdrawherBerkeleyapplication,butthishasnotbeeneasytodo,andinanyeventsomestudentsmightnotbotherwithnothingatstake.

0.2

.4.6

.81

Prob

abilit

y


AR in 2015 LOR in 2016

All applicants

0.2

.4.6

.81

Prob

abilit

y


AR in 2015 LOR in 2016

From underrepresented groups

Share of applicants considered under AR and LOR

8

Ifthesestudentsarechosenforinitialadmission,theycounttowardourstatisticsonadmittedstudents.Butiftheyareofferedpositionsonthewaitlist,theyarelikelytoturndownthisoption,andthuswillnotcountasadmissionseveniftheywouldeventuallyhavebeenadmittedoffthewaitlist.Thus,ashiftofsomeadmissionsfromthefirsttothesecondround,asoccurredin2016,willreducetheshareoftheseuninterestedstudentsintheadmitpool(and,asasideeffect,raisetheenrollmentrateamongthoseadmitted).Table3showsthedistributionofadmissionsoutcomes,aggregatingnon-athleteCaliforniaresidentsacrosseachoftheseparateapplicantpools(fordifferentcollegesanddivisions)butseparatingthedifferentstages.Thisillustratesthepotentialdistortioncausedbythewaitlist:Notethattheshareofunderrepresentedapplicantswhowereadmittedrosebyonly0.7percentagepoints(andtheshareadmittedinthefirstroundfellby0.2p.p.),whilethesharewhoenrolledroseby1.9p.p.Thisisbecausethe“yield”rateforadmittedstudentsroseby4percentagepoints,from47%to51%overall,andby10p.p.,from59%to69%,foradmittedstudentsfromtheunderrepresentedgroups.Table3.Stagesoftheadmissionsprocess


Shareofallapplicants(%)

Shareofapplicantsfrom4groups(%)

2015 2016 Change

2015 2016 Change

Initialadmit 16.1 18.1 2.0

13.3 13.0 -0.2Initialdeny 79.4 73.3 -6.1

83.8 78.9 -4.8

Offerwaitlist 4.5 8.6 4.1

3.0 8.0 5.1

Declinewaitlistoffer 1.5 4.0 2.5

1.2 4.3 3.1

Acceptwaitlistoffer 3.0 4.6 1.6

1.8 3.8 1.9

AdmittedfromWL 2.7 3.0 0.3

1.6 2.6 1.0

NotadmittedfromWL 0.3 1.6 1.3

0.2 1.2 1.0

Ultimateoutcomes

Admitted 18.8 21.1 2.3

14.9 15.6 0.7

Enrolled(SIR) 8.8 10.7 1.9

8.8 10.7 1.9

Didnotenroll 10.0 10.4 0.3

6.1 4.9 -1.2

Denied 79.7 74.9 -4.8

83.9 80.1 -3.9

WithdrewafterWLoffer 1.5 4.0 2.5

1.2 4.3 3.1

Table3reinforcesmyconcernthatissuesofself-selectionarequantitativelyimportant.Manystudentswhounder2015processeswouldhavebeenadmittedbutgoneelsewherewereinsteadin2016offeredpositionsonthewaitlistonlytodeclinetheoffers–perhapsasmanyas1%of2016applicants.Thesestudentswouldhavebeencountedasadmitsin2015butnotin2016.Importantly,thisaffectsthestatisticsforunderrepresentedstudents,asthesestudents

9

weredisproportionatelylikelytodeclinepositionsonthewaitlist:47%ofallstudentsofferedpositionsonthewaitlistdeclinedthem,butthissharewas53%forstudentsfromtheunderrepresentedgroups.Unfortunately,thereisnoadmissionsmeasurethatisperfectlycomparableacrossyears–inparticular,thecompositionofboththepoolofinitialadmitsandthepoolofeventualadmitsisaffectedbytheincreaseduseofthewaitlist,evenwithnootherchanges.Inthisreport,Ipresentresultsforfourdifferentmeasures,eachimperfect:

- Initialadmissions(includingbothFallandSpringadmits)- Admittedintheinitialroundorofferedapositiononthewaitlist- Everadmitted,eitherintheinitialroundoroffthewaitlist- Admittedandenrolled(asproxiedbyfilinganSIR,eitherforFallorSpringenrollment)

Thelastofthese,ofcourse,reflectsstudentaswellascampusdecisions(asdoesthethird,whichreflectsstudentdecisionstoacceptaspotonthewaitlist).Nevertheless,inmyviewitistheclosesttocomparableacrossyears.Ifstudents’propensitytoacceptBerkeleyadmissionsoffers,ifmade,didnotchangeacrossyears,andifastudentwhowouldacceptaninitialofferisnotputoffbybeingadmittedoffthewaitlist,changesinthepoolofenrolledstudentscanbeattributedtochangesinadmissionscriteria.AugmentedReviewandLettersofRecommendationThisreportfocusesontheAugmentedReviewandLettersofRecommendationcomponentsoftheadmissionsprocess.Table4showsthenumberofstudentsconsideredunderARin2015,thenumberaskedforLORsin2016,andtheoutcomesofeachgroupofapplications.AsalreadyindicatedbyFigure5,thismakesclearthattheLORprogramwasmuch,muchlargerthantheARpool,whichIunderstandwaskeptsmallduetotheenormousstafftimerequiredtoreviewARapplications.Table4alsoshowsthat15%ofstudentswhowereaskedforlettersdidnotrequestany.Thismightreflectwhatmanywereconcernedabout,thatstudentswouldnothaveaccesstoteacherswillingtowriteletters.Buttheaboveself-selectiondiscussionpointstoanotherpotentialexplanation:StudentsadmittedEarlyDecisionelsewhere,andothersnotveryinterestedinBerkeley,mightsimplynothavebotheredtorequestletters.Forthisreason,Idonotemphasizerequestsoforreceiptoflettersasoutcomes,andfocusontheimpactofthelettersrequestonthestudent’slikelihoodofbeingadmittedorofenrolling.

10

Table4.OutcomesforARandLORstudents


Allapplicants

Underrepresentedgroups

ARin2015

LORin2016

ARin2015

LORin2016

Numberaffected 3,046 14,406

2,793 6,337

Shareofallapplicants 7% 32%

12% 26%

LORoutcomes

Anyrequested

88%

85%

Tworequested

77%

72%

Anyreceived

87%

84%

#received=#requested

83%

79%

Tworeceived

73%

67%

Admissionsoutcomes(shares)

Initialadmit 27% 40%

27% 35%

AdmitorWLoffer 33% 59%

33% 54%

Everadmit 30% 46%

30% 40%

Admitandmatriculate 17% 23%

16% 19%

Admissionsoutcomes(numbers)

Initialadmit 816 5,833

759 2,200

AdmitorWLoffer 995 8,438

923 3,414

Everadmit 907 6,672

841 2,558

Admitandmatriculate 503 3,364

459 1,211

AssessingthelettersofrecommendationcomponentofthechangeIbeginmyanalysisbyattemptingtoassesstheimpactoftheletterofrecommendationcomponentofthe2016admissionsprocess.Analysesbytheadmissionsofficehavecontrastedthosewhoprovidedletterstothosewhowereaskedforlettersbutdidnotprovidethem.Theseareusefulinunderstandingwhichstudentsmayhavetroubleobtainingletters(thoughasnotedabove,astudentwhodoesnotobtainlettersmightjusthavedecidedtogoelsewhere).Itakeadifferentapproach:ThefactorthatisunderBerkeley’scontroliswhetherapplicantsareaskedforletters,soIattempttouncovertheimpactofthisonadmissionsoutcomes,withouttryingtodistinguisheffectscomingfromdifficultyinobtaininglettersfromthosecomingfrom(forexample)thesubmissionofweakletters.Asithappens,thewaythattheLORprocesswasimplementedallowsforacompellinganalysisoftheLORrequest,basedonacomparisonofstudentsaskedforLORswithnearlyidentical

11

studentswhojustmissedbeingasked.ThereweretwowaysthatstudentswereselectedforLORs:

• Theadmissionsofficeestimatedastatisticalmodeltopredictreadscoresin2015.3Those2016applicantswhosereadscoreswerepredictedtobe2.5,2.75,or3wereautomaticallyaskedforLORs.

• Applicantswhowerescoredbythefirstreaderas“Possible”(onthe2016Yes/Possible/Noscale),whenthiswasdonebeforethedeadlineforrequestingLORs,wereaskedforLORs.Abouttwo-thirdsofinitialreadsscored“possible”werecompletedbythedeadline.

Approximately80%ofthoseaskedforLORswereidentifiedbythefirstmethod,and40%bythesecondmethod.(20%wereidentifiedbybothmethods.)Although2015readscoresuseddiscretecategories(witheachreaderassigningascoreof1,2,2.5,3,or4,andwithlowernumbersgiventostrongerapplicants),thestatisticalmodelusedforthefirstmethodgeneratedcontinuouspredictions–thatis,anapplicantmighthavebeenpredictedtogetareadscoreof2.47.Studentswithpredictedreadscoresbetween2.38and3.26wereallaskedforletters,whilestudentswithreadscoresjustoutsidethisrangewereaskedonlyiftheywerecapturedbythesecondmethod.Figure6showstheshareofapplicantsateachpredictedreadscorewhowereaskedtosubmitletters.Forapplicantswithpredictedscoresbetween2.38and3.26,thiswas100%.Butonlyabout60%ofstudentswhowerejustabitstrongerthanthisrange(predictedscoresof2.37)wereaskedforlettersduetothefirstreader’sscore,andonlyabout3%ofstudentswhowerejustabitweakerthanthisrange(predictedscoresof3.27)wereasked.Thesesharpbreakspermita“regressiondiscontinuity”analysisoftheeffectoftheLORrequestonadmissionsoutcomes.Studentswithpredictedscoresof2.37areessentiallyidentical,onaverage,tothosewithpredictedscoresof2.38,andwouldalmostcertainlyhavehadverysimilaradmissionsoutcomeshadLORsnotbeenrequestedforsomanymoreofthelatter.Thus,anydifferenceintheiroutcomescanbeattributedtotheLORrequest.43Thepredictedreadscoreissimilarinspiritto,andreliesonthesamevariablesas,myadmissionsscorediscussedabove.Theydifferbecauselowerreadscoresarebetter,andbecausethepredictedreadscoreweightscharacteristicstopredictthe2015readscore,whiletheadmissionsscoreweightsthecharacteristicstopredictfirst-round2015admissions.Thecorrelationbetweenthetwoscoresisaround-0.85.4Anotherstrategymightbetocompareadmissionsoutcomesofthosewithpredictedreadscoresbetween2.38and3.26in2015and2016,relativetothoseoutsidethisrange.Unfortunately,Idonothaveaccesstopredictedreadscoresfor2015applicants,andhavebeenunsuccessfulinre-creatingthem.Iexpecttobeabletoeventually,butasIwritethisthekeystaffperson(GregDubrow,DirectorofResearchandPolicyAnalysisintheOfficeofUndergraduateAdmissions)isonvacation.Ithusdeferthistofuturestudy.

12

Figure6.FractionofCaliforniaresidentapplicantsaskedforlettersofrecommendation,bypredictedreadscore

Figure7shows2016admissionsoutcomesasfunctionsofthepredictedreadscore.Notsurprisingly,studentswithlowerpredictedreadscoresaremorelikelytobeadmitted.Butnoticetheareaaroundtheverticallinesat2.38and3.26,whereIallowfortheaverageadmissionsoutcomestochangediscontinuously.Applicantstotheleftofthefirstline,only60%ofwhomareaskedforLORs,aresomewhatmorelikelythanstudentstotherightoftheline,allofwhomwereaskedforLORs,tobeadmitted,tobeinvitedtothewaitlist,andtoenroll.BecausethereisnoreasontoexpectdifferencesinoutcomesbetweenthesestudentsexceptforthedifferenceintheirLORtreatment,thisisclearevidencethatforstudentswithpredictedreadscoresaround2.38–strongerthan95%ofapplicantsand81%ofadmitsin2016–beingaskedforanLORreducedtheprobabilityofadmission.

Figure7.Admissionsprobabilitiesbypredictedreadscore,2016,allCaliforniaresidentapplicants

Nowturntothesecondline.Theremaybesmallerdiscontinuitieshere,generallypointingtohigheradmissionsprobabilitiesforthe3.25swhoweredefinitelyaskedforlettersthanforthe

0.2

.4.6

.81

Frac

tion

aske

d fo

r LO

R

1.5 2 2.5 3 3.5 4 4.5Predicted read score from adm. office model

Likelihood of LOR request by predicted read score

0.2

.4.6

.81

Prob

abilit

y

2 2.5 3 3.5 4Predicted read score

Initial admitAdmit or WL offerEver admitSIR

All applicants

13

3.27swhohadonlya30%chanceofbeingasked.Thediscontinuitiesaresmallerhereandmaybeentirelyattributabletostatisticalnoise.However,thereweremanymoreapplicantswithpredictedscoresinthisrangethanaround2.38,soevenasmalleffectofLORson3.25studentswouldbequantitativelyimportant.ItisnotpossibletosaywhethertheLORimpactsseeninFigure7reflectbetteradmissionsdecisionsorworse,asitisnotpossibletoidentifythespecificstudentswhowereadmittedifnotaskedforlettersbutnotadmittedotherwise(orviceversa).ButitisworthnotingthatthisisexactlywhatwewouldexpectiftheLORsprovidedusefulinformation–somestudentswhowouldhavegottenthebenefitofthedoubtduetotheirstrongnumericcredentialswithoutLORswererevealedbytheLORstobeweakerthantheyappeared,whileotherswhowouldnothavegottenthebenefitofthedoubtwererevealedbytheirLORstobeworthadmitting.Ofcourse,theresultsarealsoconsistentwiththepossibilitythatsomestrongstudentsweredeniedadmissionbecausetheywereunabletoprovideLORs.Figure8repeatstheanalysisforapplicantsfromtheunderrepresentedgroups.Thesedataarenoisier,duetothesmallernumberofobservationshere.(Dotsaroundthe2.38thresholdrepresentanaverageof60applicantsinFigure7,andonly10-15inFigure8.)ThereisnosignherethatthestrongerstudentswerehurtbytheLORrequest,onaverage:Thoseontheleftofthe2.38lineareadmittedatessentiallythesamerateasthoseontheright.Figure8.Admissionsoutcomesbypredictedreadscore,2016,Californiaresidentapplicantsfromfourunderrepresentedgroups

Table3presentsquantitativeestimatesoftheeffectofanLORrequestonadmissionsoutcomes,separatelyforthosenearthe2.38thresholdandforthosenearthe3.26threshold.55Intechnicalterms,theseareinstrumentalvariablesestimatesfromafuzzyregressiondiscontinuitydesign.TheyreflectthelocalaverageeffectoftheLORrequestonstudentsneartherelevantthreshold.

0.2

.4.6

.81

Prob

abilit

y

2 2.5 3 3.5 4Predicted read score

Initial admitAdmit or WL offerEver admitSIR

Applicants from underrepresented groups

14

Inthefullapplicantpool,strongstudentsfromwhomLORswererequestedwere8-10percentagepointslesslikelytobeadmittedthantheywouldhavebeenhadtheynotbeenselectedforLORs.Forweakerstudents,theeffectwastoincreaseadmissionschancesby2-4percentagepoints.(Notethatthereareaboutfourtimesasmanystudentsnearthe3.26thresholdasnearthe2.38threshold,sotheimpliednumberofstudentsadmittedduetolettersneartheformeriscomparabletothenumberdeniedduetolettersnearthelatter.)Forunderrepresentedapplicants,the2.38thresholdeffectissmaller,suggestingthatLORrequestswerenotharmfultostrongstudentsfromthisgroup.

Table3.RegressiondiscontinuityestimatesoftheeffectofLORrequestsontheprobabilityofadmissionCaliforniaresidents,excludingrecruitedathletes.Standarderrorsinparentheses

Initialadmit

AdmitorWLoffer

Everadmit

SIR

Allapplicants

Strongerapplicants(lowpredictedreadscores)

-10.4% -6.3% -7.6% -13.6%

(5.5%) (4.7%) (5.4%) (5.2%)

Weakerapplicants(highpredictedreadscores)

+2.0% +3.6% +4.2% +3.6%

(2.3%) (2.5%) (2.4%) (2.0%)

Applicantsfromfourunderrepresentedgroups

Strongerapplicants(lowpredictedreadscores)

-6.5% -6.5% -4.1% -2.3%

(9.4%) (8.3%) (9.3%) (7.9%)

Weakerapplicants(highpredictedreadscores)

-1.8% +0.7% +1.6% +3.2%

(3.8%) (4.1%) (3.9%) (3.2%)

Netimpactonnumberofadmittedstudents

Allapplicants

Number -304 54 40 -300

Proportion -4% 0% 0% -6%

Fromfourunderrepresentedgroups

Number -181 -60 19 123

Proportion -5% -1% 1% 5%

ThefinalrowsofthetableattempttoestimatethenetimpactofLORsonadmissionsateachstage–positivenumbersindicateapositiveneteffect,andnegativenumbersanegativenet

15

effect.6Focusingonthelastcolumn,weseethattheLORrequirementraisedthenumberofunderrepresentedstudentswhoenrolledby123,whilereducingthenumberofenrolleesfromothergroupsby423.Thesearerelativelysmallnumbers,butshownosignofnegativeeffectsofLORsondiversityandindeedimplythatLORsraisedtheunderrepresentedshareofenrolledstudentsbyseveralpercentagepoints.Asnotedabove,wecannottellwhethertheLORaspectofthe2016proceduresledtobetterorworsedecisions.Butthereisnoindicationthatitreducedadmissionschancesforunderrepresentedorweakerstudents,whoseemmostlikelytohavefacedchallengesinobtainingsuitableletters.AssessingtheAugmentedReviewcomponentofthechangeThesecondmajorquestionIaddressiswhethertheeliminationofAugmentedReviewmadeitharderforthetypesofstudentsformerlyidentifiedforARtobeadmitted,orwhetherotherchangesmadetoadmissionsprocesseswereabletocompensateforthelackofaseparateARpool.Unfortunately,thereisnoregressiondiscontinuityresearchdesignavailableforassessingtheimpactofAugmentedReview.Moreover,thereisnowaytoidentifyinthe2016dataexactlywhichapplicantswouldhavebeenreferredtoARhaditbeeninplace,sowecannotcompareoutcomesforthispoolovertime.Asanalternative,Iidentifycandidateswho,basedontheircharacteristics,wouldhavebeenlikelytobereferredtoARin2015,andexaminehowtheiroutcomeschangedovertimerelativetootherswho,basedontheircharacteristics,wouldhavebeenunlikelytohavebeenreferredtoAR.Specifically,Icreateyetanotherscorefromthesamevariablesconsideredtodate,thisonerepresentingthelikelihoodthatastudentwiththesecharacteristicswouldhavebeenreferredtoandconfirmedforARin2015.7

6Thesecalculationsrequireratherheroicassumptions.Iassumethattheestimatedeffectsfoundatthetwodiscontinuitiesextendidenticallyoutsidethem,andIlinearlyinterpolateeffectsbetweenthediscontinuities.Imakenoallowanceforsamplingerrorinthisextrapolation.7Specifically,Ifitalogisticregression,using2015data,wheretheoutcomeisanindicatorforhavingbeenreferredtoandconfirmedforARandpredictorsareaquarticinthepredictedreadscore,indicatorsforthethreedisadvantagegroupsandinteractionsamongthem;andindicatorsandseparatequadraticsinthepredictedreadscoreforthosewith1,2,or3disadvantagefactors.(Amoreflexiblemodelthatincludesalloftheunderlyingvariablesinthemodelusedtogeneratethepredictedreadscoredoesnotgeneratemeaningfullybetterpredictions.)

16

50%ofapplicantshaveARprobabilitiesbelow2%,but10%haveprobabilitiesabove20%.Figure9showshowtheARprobabilityscorerelatestothereadscore,separatelyforstudentswithdifferentnumbersofdisadvantagefactors.(Themultiplelinesineachseriesrepresentdifferentcombinationsofwhichdisadvantagefactorsthestudenthas.)Acrossalldemographicgroups,studentswithpredictedreadscoresnear3.25aremorelikelytobeconfirmedforARthanthosewithhigherorlowerpredictedreadscores.Foranygivenpredictedreadscore,ARprobabilityscoresarehigherforthosewithmoreenumerateddisadvantagefactors.Forstudentswiththreedisadvantagefactorsandpredictedreadscoresbetween2.64and3.91,ARprobabilityscoresareabove0.2,andsometimessubstantiallyso.Studentswhohaveonlytwodisadvantagefactorsmusthavepredictedreadscoresinanarrowerrange,betweenabout3.1and3.7,toachieveARprobabilityscoresthishigh,whilestudentswithzerooronedisadvantagefactorsneverhaveARprobabilitiesabove0.12.

Figure9.EstimatedprobabilityofAugmentedReviewbypredictedreadscoreandnumberofunderrepresentationfactors,Californiaresidentsin2015

Unfortunately,whilethispredictionmodelisfairlysuccessful,itdoesnotachieveasharpdistinctionbetweenARandnon-ARstudents–eventhestudentswiththeabsolutehighestARprobabilitieshaveonlya40%chanceofbeingconfirmedforAR.Inlightofthis,IconsidertwodefinitionsofstudentsmostlikelytobeconsideredintheARpool:

- StudentswithARprobabilityscoresabove0.2(10%ofapplicantsand38%ofthoseconfirmedARin2015)

- Studentswhoarelowincome,firstgeneration,andfromlowAPIschools,withARprobabilityscoresabove0.2(7%ofapplicants,and28%ofthoseconfirmedforARin2015).

Table4showstheadmissionsoutcomesforstudentsineachofthesegroupsin2015and2016,aswellasfortheircomplements(studentswithlowerARprobabilityscores).Relativechangesatallmarginsexceptinitialoutcomesarepositiveorclosetozero.(Acrosseachdefinition,therelativechangesaremostinfavorofthehigh-riskgroupwhentheoutcomeisadmissionorthe

0.1

.2.3

.4Pr

obab

ility

refe

rred

and

confi

rmed

for A

R

2.5 3 3.5 4 4.5Predicted read score

3 factors2 factors1 factor0 factors

Probability of AR confirmation, by predicted read score andnumber of disadvantage factors, 2015

17

offerofawaitlistspot–itseemsthatmanystudentswhowouldhavebeenintheARpoolin2015wereofferedwaitlistspotsin2016butnotadmitted,eitherbecausetheywerenot

Table4.Probabilityofadmission,byARprobability,2015and2016Californiaresidents,excludingrecruitedathletes

Initialadmit

AdmitorWLoffer Everadmit SIR

Definition1:PredictedARprobability>20%

HighprobabilityAR

2015 17.5% 21.5% 19.4% 9.9%

2016 18.4% 32.5% 22.5% 12.3%

Change 0.9 11.0 3.1 2.5

ChangeforlowprobabilityARgroup

2.7 6.7 2.9 2.3

Differenceinchanges -1.8 4.3 0.2 0.2

Definition2:3disadvantagefactorsandpredictedARprobability>20%

HighprobabilityAR

2015 16.3% 19.2% 17.5% 8.9%

2016 16.0% 28.5% 19.7% 11.0%

Change -0.3 9.3 2.2 2.0

ChangeforlowprobabilityARgroup

2.7 6.9 3.0 2.3

Differenceinchanges -3.0 2.4 -0.8 -0.3

ImpactofAReliminationonnumberadmitted

Definition1 -248 603 30 25

Definition2 -396 318 -105 -38

selectedfromthewaitlistorbecausetheydeclinedtheoffer.)NeitherofthedefinitionsindicatesmeaningfuleffectsofARonthenumberofstudentswhoenrolled,andingeneral,itishardtodiscernchangesofmeaningfulmagnitudeintheadmissionsoutcomesofAR-typestudentsbetweenyears,suggestingthatchangesinotheraspectsoftheadmissionsprocessenabledthesestudentstogettheextraconsiderationin2016thattheygotthroughARin2015.TowardanunderstandingoftheoverallimpactofadmissionprocesschangesTheresultsthusfarsuggestthatLORs,ifanything,increaseddiversityoftheenteringclassin2016,andthattheeliminationofARhadatrivialeffect.Buttherewereanumberofotherchangesmadein2016,andoveralltheimpactwassomewhatlessthansatisfactory–theshareofstudentsfromunderrepresentedgroupsamongadmittedstudentsfell,thoughtheshareamongstudentswhoenrolledwasstable.Inthisfinalsection,Ipresentsomeanalysesofoveralloutcomesthatpointtopossiblecontributingfactors.

18

Figure10showsestimatesoftheshareofstudentsateachapplicationscorewhoweresuccessfulin2015and2016,foreachofthedefinitionsofsuccessdefinedearlier.Here,Iadjustthe2015applicantpooltomatchthedistributionacrosscollegesseenin2016,toremovetheinfluenceofshiftsacrossadmissionsprocessesthataremoreorlesscompetitive.Weseethatweakerapplicants(asmeasuredbyadmissionsscoresbetween0.1and0.4)weremorelikelytobeadmittedin2016thanin2015,butstrongerapplicants(scoresabove0.7)weresomewhatlesslikelytobeadmitted.Thelatterchangedisappearswhenweincludewaitlistedstudentswithinitialadmits,butitreappearsandisevenlargerwhenweexaminetheshareofapplicantswhowereeveradmitted(countingasfailuresthosewhowererejectedoutrightaswellasthosewhowereofferedwaitlistspotsbuteitherdeclinedthemorwerenotadmittedoffthewaitlist).Bycontrast,thestrongestapplicantsweremorelikelyin2016thanin2015tomatriculate.

Figure10.Admissionsprobabilitiesbyadmissionsscore,Californiaresidentsin2015and2016

Thecontrastbetweenthe3rdand4thpanelsisinformative–itsuggeststhatsomeverystrongstudentswhowereadmittedin2015butmatriculatedelsewherewerereclassifiedasnon-admitsin2016,eitherbecausetheadmissionsdecisiontookaccountinsomewayofthelikelihoodofmatriculationorbecausethesestudentsdroppedoutatthewaitliststage.Inanyevent,weseethatboththeverystrongestandweakerapplicantsweremorelikelytomatriculatein2016thanin2015,whiletherewaslittlechangeforthoseinthemiddlerange(between0.4and0.8).Figure11repeatsthisexercise,thistimeonlyforapplicantsfromthefourunderrepresentedgroups.Asintheoverallpool,weseeincreasedadmissionschancesin2016forapplicantswithadmissionsscoresaround0.2.Buthereweseefairlydramaticdeclinesinadmissionsofapplicantswithscoresabove0.5thattranslateintoreducedmatriculationaswell.

0.2

.4.6

.8Pr

obab

ility


20152016

Initial admit

0.2

.4.6

.81

Prob

abilit

y


Admit or WL offer

0.2

.4.6

.8Pr

obab

ility


Ever admitted

0.1

.2.3

Prob

abilit

y


Matriculated

Admissions probabilities, 2015 and 2016All applicants

19

Figure11.Admissionsprobabilitiesbyadmissionsscore,Californiaresidentapplicantsfromunderrepresentedgroups

Evidently,somethingintheadmissionsprocessesusedin2016reducedtheadmissionschancesofthestudentsfromunderrepresentedgroupswhowere,by2015standards,thestrongestintheirobservedcharacteristics.Onecandidateexplanationistheuseofnon-cognitivescores,whichmighthavebeensubtlybiasedagainststudentsfromunderrepresentedgroups;anotheristhatreadersmighthaveputlessweightonthefactorsmeasuringstudentsrelativetotheirschoolsinevaluating2016applications.Unfortunately,inthelimitedtimeIhadtopreparethisreport,Iwasnotabletogettothebottomofthischange.Itbearsfurtherstudy.Itisworthnoting,however,thatFigures10and11constitutestrongevidenceagainsttheviewthattheeliminationofARplayedamajorrole–recallthatARstudentsareconcentratedaroundadmissionsscoresnear0.2,whereadmissionschanceswentupthemostin2016.ConclusionBerkeleyadmissionsoutcomesforunderrepresentedstudentswere,bysomemeasures,disappointingin2016:Althoughmorewereadmittedoverall,andtheirshareofenrolledstudentswassteady,theymadeupalowershareofadmissionsoffersandparticularlyoffirst-roundadmissionsoffers.ItwasnaturaltowonderwhethertheeliminationofAugmentedReviewandtheadditionofLettersofRecommendationcontributedtothischange.Myanalysisoffersnosupportforthesepossibilities.Lettersofrecommendationseemtohavehurttheadmissionschancesofotherwise-strongapplicantsnotfromunderrepresentedgroups,withsmallerornoeffectsonapplicantsfromthosegroups,andthustohaveraisedtheshareofunderrepresentedstudentsamongadmissions.ItisdifficulttoidentifyanycleareffectofAReitherway,butinanyeventitwassmall.Theexplanationforthechangeinoutcomesin2016mustlieelsewhere,inoneoftheotherchangesmadetoadmissionsprocesses.

0.2

.4.6

.8Pr

obab

ility


20152016

0.2

.4.6

.8Pr

obab

ility


0.2

.4.6

.8Pr

obab

ility


0.0

5.1

.15

.2.2

5Pr

obab

ility


Admissions probabilities, 2015 and 2016Applicants from underrepresented groups

uc berkeley admissions in 2015 and 2016: an analysis of...

Documents