berkeley admissions bias

8/12/2019 Berkeley Admissions Bias

1/8

Sex Bias in Graduate Admissions: Data from BerkeleyAuthor(s): P. J. Bickel, E. A. Hammel, J. W. O'ConnellSource: Science, New Series, Vol. 187, No. 4175 (Feb. 7, 1975), pp. 398-404Published by: American Association for the Advancement of ScienceStable URL: http://www.jstor.org/stable/1739581Accessed: 22/09/2008 20:12

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at

http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unlessyou have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you

may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at

http://www.jstor.org/action/showPublisher?publisherCode=aaas.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed

page of such transmission.

JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with the

scholarly community to preserve their work and the materials they rely upon, and to build a common research platform that

promotes the discovery and use of these resources. For more information about JSTOR, please contact [email protected].

American Association for the Advancement of Scienceis collaborating with JSTOR to digitize, preserve and

extend access to Science.
http://www.jstor.org/stable/1739581?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/action/showPublisher?publisherCode=aaashttp://www.jstor.org/action/showPublisher?publisherCode=aaashttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/1739581?origin=JSTOR-pdf


2/8

Sex Bias in GraduateAdmissions:Data from Berkeley

Measuring bias is harder than is usually assumed,and the evidence is sometimes contrary to expectation.

P. J. Bickel, E. A. Hammel, J. W. O'Connell

Determining whether discriminationbecauseof sex or ethnic identityis be-ing practiced against persons seekingpassagefrom one social statusor locusto another is an importantproblem inour society today. It is legally impor-tant and morally important.It is alsooften quite difficult.This article is anexploration of some of the issues ofmeasurementand assessment involvedin one example of the general prob-lem, by means of which we hope toshed some light on the difficulties.Wewill proceed in a straightforwardandindeed naive way, even though weknow how misleading an unsophisti-cated approachto the problem is. Wedo this becausewe think it quite likelythat other persons interested in ques-tions of bias might proceed in justthe same way, and careful exposureof the mistakes in our discovery pro-cedure may be instructive.

Data and AssumptionsThe particularbody of data chosenfor examination here consists of ap-plications for admission to graduatestudy at the University of California,Berkeley, for the fall 1973 quarter.Inthe admissionscycle for that quarter,the GraduateDivision at Berkeley re-ceived approximately 15,000 applica-tions, some of which were later with-drawn or transferred to a different

proposed entry quarter by the appli-cants. Of the applications finally re-mainingfor the fall 1973 cycle 12,763were sufficiently complete to permit aDr. Bickel is professor of statistics, Dr.Hammel is professor of anthropology and associ-ate dean of the Graduate Division, and Mr.O'Connell is a member of the data processingstaff of the Graduate Division, at the Universityof California, Berkeley 94720.

398

deceisionto admitor to deny admission.The questionwe wishto pursue s wheth-er the decision to admitor to deny wasinfluencedby the sex of the applicant.We cannot know with any certaintythe influenceson the evaluators n theGraduateAdmissionsOffice,or on thefaculty reviewing committees, or onany other administrative ersonnelpar-ticipating in the chain of actions thatled to a decision on an individualap-plication. We can, however, say thatif the admissionsdecision and the sexof the applicant are statisticallyasso-ciated in the results of a series of ap-plications, we may judge that biasexisted, and we may then seek to findwhether discrimination existed. Bybias we mean here a pattern of as-sociation between a particulardecisionand a particularsex of applicant, ofsufficient strength to make us con-fident that it is unlikely to be the re-sult of chance alone. By discrimina-tion we mean the exerciseof decisioninfluencedby the sex of the applicantwhen that is immaterial to the quali-fications for entry.The simplest approach (which weshall call approachA) is to examinethe aggregate data for the campus.This approachwould surely be takenby many persons interested n whetherbias in admissionsexists on any cam-pus. Table 1 gives the data for all12,763 applicationsto the 101 grad-uate departments nd interdepartmentalgraduate majors to which applicationwas made for fall 1973 (we shall referto them all as departments). Therewere 8442 male applicants and 4321female applicants. About 44 percentof the males and about 35 percent ofthe females were admitted. Just thiskind of simple calculation of propor-tions impels us to examine the datafurther. We will pursue the question

by using a familiarstatistic,chi-square.As alreadynoted, we are awareof thepitfalls ahead in this naive approach,but we intend to stumble into everyone of them for didacticreasons.We must first make clear two as-sumptions that underlie considerationof the data in this contingency tableapproach.Assumption1 is that in anygiven discipline male and female ap-plicantsdo not differin respectof theirintelligence,skill, qualifications,prom-ise, or other attribute deemed legiti-mately pertinentto their acceptanceasstudents.It is preciselythis assumptionthat makes the study of sex biasmeaningful,for if we did not hold itany differences in acceptance of ap-plicants by sex could be attributedtodifferences n theirqualifications, rom-ise as scholars,and so on. Theoretical-ly one could test the assumption,forexample,by examiningpresumablyun-biasedestimatorsof academicqualifica-tion such as GraduateRecord Exam-ination scores, undergraduategradepoint averages,and so on. There are,however, enormous practical difficul-ties in this. We thereforepredicateourdiscussion on the validity of assump-tion 1.

Assumption2 is that the sex ratiosof applicants to the various fields ofgraduatestudy are not importantlyas-sociated with any other factors in ad-mission. We shall have reason to chal-lenge this assumptionlater, but it iscrucial in the first step of our explora-tion, which is the investigationof biasin the aggregatedata.

Tests of AggregateDataWe pursuethis investigationby com-putingthe expected frequenciesof maleand female applicants admitted anddenied, from the marginal totals ofTable 1, on the assumptionthat menand women applicants have equalchances of admissionto the university(that is, on the basis of assumptions

1 and 2). This computation,also givenin Table 1, shows that 277 fewer wom-en and 277 more men were admittedthan we would have expected underthe assumptionsnoted. That is a largenumber,and it is unlikelythat so largea bias to the disadvantageof womenwould occur by chance alone. Thechi-squarevalue for this table is 110.8,and the probability of a chi-squarethat large (or larger) under the as-sumptions noted is vanishingly small.We should on this evidence judgeSCIENCE, VOL. 187


3/8

that bias existed in the fall 1973 ad-missions. On that account, we shouldlook for the responsible parties to seewhether they give evidence of dis-crimination. Now, the outcome of anapplication for admission to graduatestudy is determined mainly by thefaculty of the department to which theprospective student applies. Let usthen examine each of the departmentsfor indications of bias. Among the 101departments we find 16 that eitherhad no women applicants or deniedadmission to no applicants of eithersex. Our computations, therefore, ex-cept where otherwise noted, will bebased on the remaining 85. For astart let us identify those of the 85with bias sufficiently large to occur bychance less than five times in a hun-dred. There prove to be four suchdepartments. The deficit in the numberof women admitted to these four (un-der the assumptions for calculatingexpected frequencies as given above)is 26. Looking further, we find sixdepartments biased in the opposite di-rection, at the same probability levels;these account for a deficit of 64 men.

These results are confusing. Afterall, if the campus had a shortfall of277 women in graduate admissions,and we look to see who is responsible,we ought to find somebody. So largea deficit ought not simply to disappear.There is even a suggestion of a sur-plus of women. Our method of ex-amination must be faulty.Some Underlying Dependencies

We have stumbled onto a paradox,sometimes referred to as Simpson's inthis context (1) or spurious correla-tion in others (2). It is rooted in thefalsity of assumption 2 above. We haveassumed that if there is bias in theproportion of women applicants ad-mitted it will be because of a link be-tween sex of applicant and decision toadmit. We have given much less at-tention to a prior linkage, that betweensex of applicant and department towhich admission is sought. The tend-ency of men and women to seekentry to different departments ismarked. For example, in our data al-most two-thirds of the applicants toEnglish but only 2 percent of the ap-plicants to mechanical engineering arewomen. If we cast the application datainto a 2 X 101 contingency table, dis-tinguishing department and sex of ap-plicants, we find this table has a chi-7 FEBRUARY 1975

Table 1. Decisions on applications to Graduate Division for fall 1973, by sex of applicant-naive aggregation. Expected frequencies.are calculated from the marginal totals of the observedfrequencies under the assumptions (1 and 2) given in the text. N =12,763, 2 = 110.8,d.f. = 1, P 0 (18).Outcome DifferenceApplicants Observed Expected

Admit Deny Admit Deny Admit DenyMen 3738 4704 3460.7 4981.3 277.3 - 277.3Women 1494 2827 1771.3 2549.7 - 277.3 277.3

square of 3091 and that the probabilityof obtaining a chi-square value thatlarge or larger by chance is aboutzero. For the 2 X 85 table on the de-partments used in most of the analysis,chi-square is 3027 and the probabilityabout zero. Thus the sex distributionof applicants is anything but ran-dom among the departments. In ex-amining the data in the aggregate aswe did in our initial approach, wepooled data from these very different,independent decision-making units. Ofcourse, such pooling would not nullifyassumption 2 if the different depart-ments were equally difficult to enter.We will address ourselves to that ques-tion in a moment.Let us first examine an alternativeto aggregating the data across the 85departments and then computing astatistic-namely, computing a statisticon each department first and aggregat-ing those. Fisher gives a method foraggregating the results of such in-dependent experiments (3). If we ap-ply his method to the chi-square sta-tistics of the 85 individual contingencytables, we obtain a value that has aprobability of occurrence by chancealone, that is, if sex and admissionare unlinked for any major, of about29 times in 1000 (4). Another com-mon aggregation procedure, proposedto us in this context by E. Scott, yieldsa result having a probability of 6times in 10,000 (5). This is consistentwith the evidence of bias in somedirection purportedly shown by Table1. However, when we examine thedirection of bias, the picture changes.For instance, if we apply Fisher'smethod to the one-sided statistics, test-ing the hypothesis of no bias or ofbias in favor of women, we find thatwe could have obtained a value aslarge as or larger than the one ob-served, by chance alone, about 85times in 100 (6).Our first, naive approach of examin-ing the aggregate data, computing ex-pected frequencies under certain as-sumptions, computing a statistic, and

deciding therefrom that bias existedin favor of men has now been castinto doubt on at least two grounds.First, we could not find many biaseddecision-making units by examiningthem individually. Second, when wetake account of the differences amongdepartments in the proportions of menand women applying to them andavoid this problem by computing astatistic on each department separately,and aggregating those statistics, theevidence for campus-wide bias in favorof men is extremely weak; on thecontrary, there is evidence of bias infavor of women.The missing piece of the puzzle isyet another fact: not all departmentsare equally easy to enter. If we castthe data into a 2 X 101 table, distin-guishing department and decision toadmit or deny, we find that this tablehas a chi-square value of 2195, withan associated probability of occurrenceby chance (under assumptions 1 and2) of about zero, showing that theodds of gaining admission to differentdepartments are widely divergent. (Forthe 2 X 85 table chi-square is 2121and the probability about zero.) Now,these odds of getting into a graduateprogram are in fact strongly associatedwith the tendency of men and womento apply to different departments indifferent degree. The proportion ofwomen applicants tends to be high indepartments that are hard to get intoand low in those that are easy to getinto. Moreover this phenomenon ismore pronounced in departments withlarge numbers of applicants. Figure 1is a scattergram of proportion of ap-plicants that are women plotted againstproportion of applicants that are ad-mitted. The association is obvious oninspection although the relationship iscertainly not linear (7). If we use aweighted correlation (8) as a measureof the relationship for all 85 depart-ments in the plot we obtain A = .56.If we apply the same measure to the17 departments with the largest num-bers of applicants (accounting for two-

399


4/8

thirds of the total population of ap- all of identical size (assumption 1),plicants) we obtain p = .65, while the swim towardthe net and seek to pass.remaining68 departmentshave a cor- The female fish all try to get throughresponding = .39. The significanceof the small mesh, while the male fishA under the hypothesis of no associa- all try to get through the large mesh.tion can be calculated.All three values On the other side of the net all theobtainedare highly significant. fish are male. Assumption 2 said thatThe effect may be clarifiedby means the sex of the fish had no relation toof an analogy.Picturea fishnetwithtwo the size of the mesh they tried to getdifferentmesh sizes. A school of fish, through. It is false. To take another

Table 2. Admissions data by sex of applicant for two hypothetical departments. For total,X2 = 5.71, d.f. = 1, P = 0.19 (one-tailed).

Outcome DifferenceApplicants Observed ExpectedAdmit Deny Admit Deny Admit Deny

Department of machismaticsMen 200 200 200 200 0 0Women 100 100 100 100 0 0Department of social warfareMen 50 100 50 100 0 0Women 150 300 150 300 0 0TotalsMen 250 300 229.2 320.8 20.8 - 20.8Women 250 400 270.8 379.2 - 20.8 20.8

O Number of applicants ` 40

CCZg

e rO DO~~~~~~~~aC

E

m-- L I I E1i ZL

DD

OC

ED D DC

C C C0

0J

LIl riC

example that illustratesthe danger ofincautious pooling of data, considertwo departmentsof a hypotheticaluni-versity-machismatics and social war-fare. To machismatics here apply 400men and 200 women; these are ad-mitted in exactly equal proportions,200 men and 100 women. To socialwarfarethere apply 150 men and 450women; these are admitted in exactlyequal proportions, 50 men and 150women. Machismaticsadmittedhalf theapplicantsof each sex, social warfareadmitted a third of the applicants ofeach sex. But about 73 percent of themen applied to machismatics and 27percent to social warfare,while about69 percent of the women applied tosocial warfare and 31 percent tomachismatics.When these two depart-ments are pooled and expected fre-quencies are computed in the usualway (with assumption2), there is adeficit of about 21 women (Table 2).A discrepancy in that direction thatlarge or larger would be expectableless than 2 percent of the time bychance; yet both departments wereseen to have been absolutely fair indealing with their applicants.The creation of bias in our originalsituation is, of course, much morecomplex, since we are aggregatingmany tables. It results from an inter-action of the three factors, choice ofdepartment, sex, and admission status,whose broad outlines are suggested byour plot but which cannot be describedin any simple way.In any case, aggregation in a simpleand straightforward way (approach A)is misleading. More sophisticated meth-ods of aggregation that do not relyon assumption 2 are legitimate buthave their difficulties. We shall havemore to say on this later.

DisaggregationD 1 .The most radical alternativeto ap--DC10 proach A is to considerthe individual

h o..D .3 graduate departments, one by one.[P Li \ However, this approach (which we.. may call approachB) also poses diffi-[ - o culties. Either we must sample ran-[-' domly from the differentdepartments,l0 or we must take account of the proba-D C_L~ _] ~~bility of obtaining unusual sex ratiosof admittees by chance in a number

o -10 20 30 40 50 60 70 80 of simultaneously conducted indepen-Percent women s dent experiments. That is, in examiningPercentwomenapplicants 85 separate departmentsat the same.Proportion of applicants that are women plotted against proportion of appli- e dea es a e aeadmitted, in 85 departments.Size of box indicates relative number of applicants time for evidence of bias we are con-denartment. ducting 85 simultaneous experiments,

SCIENCE, VOL. 187

100

90

80t1, 1

70

60 F0.4-PEU,Cc.)0.0.CL0r_U0-

50

40

30

20

10

0

Fig. 1cantsto the400

_d-

-r--

J-^


5/8

and in that many experiments theprobability of finding some markeddepartures from expected frequenciesjust by chance is not insubstantial.The departmentwith the strongestbiasagainst admitting women in the fall1973 cycle had a bias of sufficientmagnitudeto be expectableby chancealone only 69 times in 100,000. If wehad selected that departmentfor ex-amination on a random basis, wewould have been convinced that itwas biased. But -we did not so selectit; we looked at 85 departments atonce. The probability of finding adepartment hat biased against women(or more biased) by chance alonein 85 simultaneous trials is about 57times in 1000. Thus that particulardepartment is not quite so certainlybiased as we might have first believed,.057 being a very much larger num-ber than .00069, althoughstill a smallenough probabilityto warranta closerlook. This departmentwas the worstone in respect of bias against womenin admissions;the probabilityof find-ing departments ess biased by chancealone is of course greater than .057.We can also examine events in theother direction. The departmentmostbiased against men had a bias suffi-ciently largeto be expectableby chancealone about 20 times in a million,and the chance of findinga departmentthat biased (or more biased) in thatdirectionby chance alone in 85 simul-taneoustrials(9) is about .002.There is a further difficulty in ap-proach B. Although it makes a greatdeal of sense to examine the individualdepartmentsthat are in fact the in-dependent decision-makingentities inthe graduateadmissionsprocess, someof them are quite small, and even insome that are of ordinary size thenumber of women applying is verysmall. Calculationof the probabilityofobserveddeviations from expected fre-quencies can be carried out for suchunits, but when the numbers involvedare very small the evidence for decid-ing whether there is no bias or grossbias is really worthless (10). This de-fect is evident not only in approachB but also if we use some reasonablemethod of aggregationof test statisticsto avoid the pitfalls of approach Asuch as that of Fisher, or even theapproach we suggest below. That is,large biases in small departmentsorin departmentswith small numbers ofwomen applicantswill not influence areasonableaggregatemeasure appreci-ably.7 FEBRUARY 1975

Pooling admitted if they were being treatedjust like the men. We do this computa-The difficulty we face is not only tion for each department separately,technical and statistical but also ad- since each is likely to have a differentministrative. n some sense the campus probabilityof admissionand a differentis a unit. It operates under general number of women applying, and weregulations concerning eligibility for sum the results to obtain the numberadmission and procedures for admis- of women expected to be admitted forsion. It is a socialcommunity hat shares the campus as a whole (11). This esti-certain values and is subject to certain mate proves to be smaller by 60 thangeneral influences and pressures.It is the numberof women observedto haveidentifiable as a bureaucraticunit by been admitted (Table 3).its own members and also by external The computation of Table 3 is asagencies and groups. It is, as a social follows: For a four-cell contingencyand cultural unit, accountable to its table of the following format:various publics. For all these reasons-it makes sense to ask the question, Admit DenyIs therea campusbias by sex in gradu- Men ai biate admissions?But this questionraises Women c_ diserious conceptual difficulties.Is cam- t p c othe particular cell of interest is c%pus bias to be measured by the net t n ocontaining the number of women ad-bias across all its constituentsubunits?. mitted. The expected frequency underHow does one define such a bias?

How. .. . .suchabias? the hypothesis of no bias is E = wip1For any definition, it is easy to imagine _ ( + ) ( +c,)/Ni, where N= isa situation in which some departments te tal of a icas to departmentare biased in one direction and other he o , 0i., The observed number, O, is thedepartments in another, so that the .number in c,. The differencebetweennet bias of the campus may be zero these to quaniie - s ethese two quantities, O - E, summedeven though very strong biases are ap- over n departments sparent in the subunits.Does one lookinstead at the outliers, those depart- o - E) DIFFments that have divergences o extreme i=as to call their particular practices Then,into question? How ex-treme is extreme in x2 _ _ _ (DIFF)2such a procedure,and n 2what does one do (a + b)(a + c)(c, + di)(b, + df) N,(N,- I)about units so small asto make such assessmentmeaningless? with d.f. = 1. Ninety-six departmentsWe believe that there are no easy were included n the computation,sinceanswersto these questions,but we are 5 of the total 101 each had only 1prepared to offer some suggestions. applicant.If Ni - 1 is replacedby N inWe propose that examinationof cam- the denominator,all 101 departmentspus bias must rest on a method of can be included, yielding X2= 8.61;estimation of expected frequenciesthat O- E remains 60.1 and the expectedtakes into account the falsity of as- and observedfemale admitteesare eachsumption 2 and the apparentpropen- increasedby 1. (This statistic makes itsity of women to apply to departments possible to include contingency tablesthat are more difficult to enter. havingan empty cell, so that no infor-We reanalyzeTable 1, using all the mation is lost; there is thus an advan--data leading to it, by computing the tage over methods that pool the chi-expected frequencies differently than squarevaluesfrom a set of contingencyin approach A, since we now know tables.)the assumptionsunderlying hat earlier The probability that an observedcomputationto be false. We estimate bias this large or larger in favor ofthe number of women expected to be women might occur by chance aloneadmittedto a departmentby multiply- (under these new assumptions) ising the estimated probability of ad- .0016; the probabilityof its occurringmission of any applicant (regardless if there were actual discriminationof sex) to that department by the against women is, of course, evennumberof women applyingto it. Thus, smaller. This is consistent with whatif the chances of getting into a de- we found using Fisher's approachandpartment were one-half for all appli- aggregatingthe test statistics: there iscants to it, and 100 women applied, evidence of bias in favor of women.we would expect 50 women to be [The test used here was proposed in

401


6/8

another context by Cochran (12)and Mantel and Haenszel (13).]We would be remiss if we did notpoint out yet another pitfall of ap-proach A. Whereas the highly signifi-cant values of the Mantel-HaenszelorFisher statistics just mentioned for1973 are evidence that there is biasin favor of women, the low values ob-tained in other years (see below) donot indicate that every department wasoperating more or less without bias.Such low values could equally wellarise as a consequenceof cancellation.We illustrate'with the hypotheticalde-partments of machismatics and socialwarfare.If machismaticsadmitted250men and 50 women, creating a short-fall of 50 women, while social warfareadmitted 200 women and no men,creating an excess of 50 women, theaggregate measure of bias we haveintroduced would be zero. We onlyargue that if an aggregatemeasure ofbias is wanted the one we propose isreasonable. Of course, if we combinetwo-sided statisticsby the Fisher meth-od this phenomenon does not occur.We would conclude from this exam-ination that the campusas a whole didnot engage in discrimination againstwomen applicants. This conclusion isstrengthenedby similarlyexaminingthedatafor the entirecampusfor the years1969 through 1973. In 1969 the num-ber of women admitted exceeded theexpected frequency by 24; the prob-ability of a deviation of this size orlarger in either direction by chancealone is .196. In 1970 there were fourfewer women admitted than expected,the probability of chance occurrencebeing .833. In 1971 therewere 25 morewomen than expected, with a proba-bility of .249. In 1972 there were sevenmore women than expected, the prob-ability being .709. For 1973 as shownabove the deviation was an excess of60 women over the expected number:the probability of a chance deviationthat large or larger in either directionis .003. These data suggest that thereis little evidence of bias of any kinduntil 1973, when it would seem signifi-cant evidence of bias appears, n favorof women.This conclusionis supportedby all the other measures we haveexamined. For instance, pooling thechi-squarestatisticsby Fisher's methodyields a probability of .99 in 1969,1970, and 1971, a probability of .55in 1972, and a probability of .029 in1973 (14).We may also take approach B and

402

Table 3. Sum of expected departmentalut-comesof women'sapplications omparedwithsum of observed utcomes,GraduateDivision,Berkeley, all 1973.x2= 8.55, d.f. = 1, P =.003 (two-tailed).Expectedemaleadmittees 1432.9Observedemaleadmittees 1493.0Difference (O - E) 60.1

look for individualdepartmentoutliers.Because the numbers of women stu-dents applyingto some of them in anyone year are often small,we aggregatedthe data for each departmentover the5-year span, using the method just ex-plained. (This procedure of coursehides the kind of change that the ag-gregating approach reveals when pur-sued through time, but it enables usto focus on possible offenders ineither directionin a campusthat is onthe average behaving itself.) Duringthe 5-year period there were 94 unitsthat had at least one applicantof eachsex and admittedat least one applicantand denied admissionto at least one inat least one year. Two of the 94 units,one in the humanitiesand one in theprofessions, show a divergence fromchance expectationssufficient o arouseinterest. One of these admitted 16fewer women than expected over 5years, a shortfall of 29 percent; theprobabilityof such a result by chancealone in 94 trials is about .004. Theother unit admitted 40 fewer womenthan expected over the 5-year period,a shortfall of 7 percent, with a prob-ability in 94 trials of about .019. Thenext most likely result by chance wasat a level of .094 and the next afterthat at .188. Conversely here were twounitssignificantlybiasedin the oppositedirection, with chance probabilitiesofoccurrence of .033 and .047, account-ing for a combined shortfall of 50 men,13 and 24 percent respectivelyof theexpected frequencies in the individualunits.The kinds of statistics we may wishto use in examination of individualdepartmentsmay differfrom those em-ployed in these general screening pro-cesses. For example, in one of thecases of a shortfall of women citedabove, it seems likely that an intensifieddrive to recruit minority group mem-bers caused a temporary drop in theproportion of women admitted, sincemost of the minority group admitteeswere males. In most of the cases in-volving favored status for women itappearsthat the admissionscommittees

were seeking to overcome long-estab-lished shortages of women in theirfields. Overall, however, it seems thatthe admissions procedure has beenquite evenhanded. Where there are di-vergences from the expected frequen-cies they are usually small in magni-tude (although they may constitute asubstantialproportionof the expectedfrequency), and they more frequentlyfavor women than discriminateagainstthem.

MoreGeneralIssuesWe have already explained why as-sumption 1-the equivalence of aca-demicqualifications f men andwomenapplicants-is necessaryto the statisti-cal examinationof bias in admissions.But the assumptionis clearly false in

its most extensivesense;there are areasof graduatestudythat men and womensimply have not hitherto been equallyprepared o enter. One of the principaldifferentiators s preparation n mathe-matics,which is prerequisiten an elab-orate stepwise fashion to a numberoffields of graduateendeavor (15).This differentiationwould have littleeffect on women's chances to entergraduateschool if it were unrelated todifficulty of entry. But it is not. Al-though it would appear in a logicalsense that the departmentsrequiringmore mathematicswould be more diffi-cult to enter, in fact it appearsto bethose requiring less mathematics thatare the more difficult. (For the 83graduate programswith matching un-dergraduatemajors, the Pearson r be-tween proportion of applicants ad-mitted and number of recommendedor required undergraduate units inmathematicsor statistics s .38.) In partthis may be because departmentsre-quiringless mathematicsreceive appli-cations from persons who might havepreferred o enterothers but cannot forlack of mathematical (or similar)background,as well as from personsintrinsically nclinedtowardnonmathe-matical subjects. In part it is becausein the nonmathematical ubjects (thatis, the humanitiesand social sciences)students take longer to get throughtheir programs;in consequence,thosedepartments have lower throughputand thus less room, annually,to acceptnew students. Just why this is so is amatter of debate and of great complex-ity. Some of the problem may lie in thevery lack of a chain of prerequisites

SCIENCE, VOL. 187


7/8

such as that characterizing graduatework in, let us say, the physical sci-ences. Some may lie in the nature ofthe subject matter and the intractabil-ity of its data and the questionsaskedof the data. Some may lie in the lessfavorablecareer opportunitiesof thesefields and in consequencea lower pullfrom the professionalemploymentmar-ket. Some may lie just in the higherproportionof women enrolled and thepossibility that women are under lesspressure o completetheir studies (hav-ing alternativeoptions of social rolesnot open generally to men) and haveless favorable employment possibilitiesif they do complete,so that the pull ofthe market is less for them. Whateverthe reasons, the lower productivityofthese fields is a fact, and it crowds thedepartments n them and makes themmore difficultto enter.The absence of a demonstrablebiasin the graduateadmissionssystem doesnot give grounds for concluding thatthere must be no bias anywhere elsein the educational process or in itsculmination in professional activity.Our intention has been to investigatethe generalcase for bias againstwomenin a specific matter-admission tograduateschool-not only because wehad the data base to do so but alsobecause allegations of bias in the ad-missions process had been aired. Ourapproach in the beginning was naive,as befits an initial investigation. We

found that even the naive questioncould not be answered adequatelywithout recourse to sophisticatedmethodology and careful examinationof underlying processes. We take thisopportunityto warn all those who areconcerned with problemsof bias aboutthese methodologicalcomplexities(16).We also find, beyond this immediateareaof concernin graduateadmissions,that the questionsof bias and discrimi-nation are more subtle than one mighthave imagined, and we mean this inmore than just the methodologicalsense. If prejudicialtreatment is to beminimized, it must first be locatedaccurately.We have shown that it isnot characteristicof the graduate ad-missions process here examined (al-though this judgment does not elimi-nate the possibility of individualcasesof prejudicial treatment, and it doesnot deal with politically or morallyde-fined null hypotheses). The fairness ofthe faculty in admissions is an impor-tant foundation for further effort.Thateffort can be made directly by univer-7 FEBRUARY 1975

sities in seeking to equalize the prog-ress of men and women toward theirdegrees (17). A universitycan use itspowersof suasionto equalizethe prep-arationof girls and boys in the primaryand secondary schools for entry intoall academic fields. By its own objec-tive research it may be able to deter-mine where and how much bias anddiscriminationexist and what the suit-able corrective measures may be.

SummaryExamination of aggregate data ongraduate admissions to the Universityof California, Berkeley, for fall 1973shows a clear but misleading patternof bias against female applicants.Ex-amination of the disaggregated datareveals few decision-makingunits that

show statisticallysignificantdeparturesfrom expected frequencies of femaleadmissions, and about as many unitsappear to favor women as to favormen. If the data are properlypooled,taking into account the autonomy ofdepartmental decision making, thuscorrectingfor the tendency of womento apply to graduatedepartments hatare more difficult for applicants ofeither sex to enter, there is a smallbut statisticallysignificantbias in favorof women. The graduate departmentsthat are easier to entertend to be thosethat require more mathematics in theundergraduatepreparatorycurriculum.The bias in the aggregateddata stemsnot from any patternof discriminationon the part of admissionscommittees,which seem quite fair on the whole,but apparently from prior screeningat earlier levels of the educationalsys-tem. Women are shunted by their so-cialization and educationtoward fieldsof graduate study that are generallymore crowded,less productiveof com-pleted degrees, and less well funded,and that frequently offer poorer pro-fessional employmentprospects.References and Notes

1. C. R. Blyth, J. Am. Stat. Assoc. 67, 364(1972).2. J. Neyman, Lectures and Conferences onMathematical Statistics and Probability (U.S.Department of Agriculture Graduate School,Washington, D.C., ed. 2, 1952), p. 147.3. R. A. Fisher, Statistical Methods for Re-search Workers (Oliver and Boyd, London,ed. 4, 1932).4. Fisher's statistic isaF --2 ln p(T1)i==l

where p(Ti) is the P value of the test statisticcalculated for the ith experiment (department).

F is referred to the upper tail of a chi-square distribution with 2n degrees offreedom where n = number of experimentalresults to be aggregated, here 85. In ourapplication here, Ti is the usual contingencytable chi-square statistic, with P value ob-tained from a table of the chi-square dis-tribution with 1 degree of freedom.5. This method uses as a statistici

X2xi= 1having a chi-square distribution with d.f. =n (= 85 here). X,2 is the usual X statistic inthe ith 2 X 2 table.6. In this application of Fisher's statistic (4),Ti, is ? the square root of the chi-squarestatistic with sign plus if there is an excessof men admitted and sign minus otherwise;the P value is the probability of a standardnormal deviate exceeding Ti.7. Transformation to linearity by simple changesof variable, for example to log (odds), is alsonot successful.8. If 7ri, pi, and p't represent, respectively, theprobability of applying to department i, theprobability of being admitted given that ap-plication is to department i, and the prob-ability of being a male given that applicationis to department i, then a reasonable mea-sure of the association of the numbers pi, p'iis the correlation (weighted according to theshare of each major in the applicant pool)p = Z7r(pt - p.)(p'I - p'.)l

[Z7t(pt - p.)2Z,r,(p't _ p',)2]where p., p'. are defined by ZTripi, Szrip'i,respectively. As usual, Ipl = 1 indicates lineardependence between the pi, p'i while p = 0suggests no relation. Positive values indicatepositive association and so on.This correlation can be estimated bysubstituting the observed proportions of ap-plicants to department i, admitted applicantsto department i among applicants to de-partment i, and male applicants to departmenti among applicants to department i for7ri, pi and p'i, respectively. This is the sta-tistic we call p.We can use p as a test statistic for thehypothesis that p = 0. To do so we need thedistribution of p under that hypothesis. Itturns out that p/Var p,/2 has approximatelya standard normal distribution. The expressionVar p is complicated because of the sta-tistical dependence between pt and p'i. Edi-torial considerations have prompted its de-letion. It is obtainable from the authors.9. The probability that an observation as ex-treme as (or more extreme than) the mostextreme cne would occur by' chance alone,where n number of simultaneous indepen-dent experiments or observations, and p = prob-ability of occurrence by chance of the mostextreme observation if it had been selectedat random for a single observation, is 1 -(1 -p) , and thus for p close to zero isapproximately np.10. Smallness of numbers of women applicantsalso invalidates the normal approximationused in the significance probabilities of ap-proach B, but this can be remedied.11. This may be expressed as

85E (w)(pi)

where wi is the number of women applyingto the ith major and pi is the probability ofentry of any applicant into the ith major,the latter being estimated from the numberof admittees divided by the number of ap-plicants.12. W. G. Cochran, Biometrics 10, 417 (1954).13. N. Mantel, J. Am. Stat. Assoc. 58, 690(1963).14. Further analysis of these data, in particularexamination of individual units through time,is in progress.15. Research currently being conducted by L.Sells at Berkeley shows how drastic thisscreening process is, particularly with respectto mathematics.16. There is a real danger in naive determina-tion of bias when the action following posi-tive determination is punitive. On the basisof Table 1, which we have now shown to be

403


8/8

misleading, regulatory agencies of the federalgovernment would have felt themselves justi-fied in withholding substantial amounts ofresearch funding from the university. Afurther danger in punitive action of this kindis that, being concentrated in the researcharea, which provides an important sourceof support for graduate students, it punishesnot only male but also female students-women in areas in which women have tra-ditionally been enrolled, such as the social

misleading, regulatory agencies of the federalgovernment would have felt themselves justi-fied in withholding substantial amounts ofresearch funding from the university. Afurther danger in punitive action of this kindis that, being concentrated in the researcharea, which provides an important sourceof support for graduate students, it punishesnot only male but also female students-women in areas in which women have tra-ditionally been enrolled, such as the social

sciences, and also pioneering women in thephysical and biological sciences, where fed-eral support has been more concentrated.17. In fact, data in hand at Berkeley suggest adramatic decrease in the early dropout ratesof women and the disappearance of thedifferential in dropout rates of men andwomen. It will be several years before wewill be able to judge whether this phenome-non is one of decreased or simply of delayedattrition.

sciences, and also pioneering women in thephysical and biological sciences, where fed-eral support has been more concentrated.17. In fact, data in hand at Berkeley suggest adramatic decrease in the early dropout ratesof women and the disappearance of thedifferential in dropout rates of men andwomen. It will be several years before wewill be able to judge whether this phenome-non is one of decreased or simply of delayedattrition.

18. If the same naive aggregation is carried outfor the 85 departments used in most of theanalysis, N = 12,654, X2= 105.6, d.f. = 1,P =0.19. The investigation was initiated by E.A.H.,using data retrievable from a computerizedsystem developed by V. Aldrich. Advice onstatistical procedures in the later stages ofthe investigation was provided by P.J.B., andprogramming and other computation was doneby J.W.O'C.

18. If the same naive aggregation is carried outfor the 85 departments used in most of theanalysis, N = 12,654, X2= 105.6, d.f. = 1,P =0.19. The investigation was initiated by E.A.H.,using data retrievable from a computerizedsystem developed by V. Aldrich. Advice onstatistical procedures in the later stages ofthe investigation was provided by P.J.B., andprogramming and other computation was doneby J.W.O'C.

Crisis Management:SomeOpportunities

International emergency cooperation involvinggovernments,technology, and science is now foreseeable.

Robert H. Kupperman, Richard H. Wilcox, Harvey A. Smith

Crisis Management:SomeOpportunities

International emergency cooperation involvinggovernments,technology, and science is now foreseeable.

Robert H. Kupperman, Richard H. Wilcox, Harvey A. Smith

Many alarmingtrends of our pres-ent cultureshare common roots.World-wide inflation, worldwide resourceshortages, extensive famine, and theinexorablequestfor more deadlyweap-ons may very well reach crisis pro-portionsif these trends continue.Theyserve already as examples of nationaland international failures of efficientresource allocation and communica-tions. It is important that we under-stand the possible future implicationsthat these failures hold and, more im-portant, that we develop means fordealing with them.In discussingthe crisis managementdemanded by such situations it istempting to start by defining what ismeant by a crisis, but this is a difficultmatter. Crises are matters of degree,being emotionally linked to such sub-jective terms as calamity and emer-gency. In fact it is not necessary todefine crises in order to discuss prob-lems generally common to their man-agement, including the paucity of ac-curate information, the communica-tions difficulties that persist, and the

Many alarmingtrends of our pres-ent cultureshare common roots.World-wide inflation, worldwide resourceshortages, extensive famine, and theinexorablequestfor more deadlyweap-ons may very well reach crisis pro-portionsif these trends continue.Theyserve already as examples of nationaland international failures of efficientresource allocation and communica-tions. It is important that we under-stand the possible future implicationsthat these failures hold and, more im-portant, that we develop means fordealing with them.In discussingthe crisis managementdemanded by such situations it istempting to start by defining what ismeant by a crisis, but this is a difficultmatter. Crises are matters of degree,being emotionally linked to such sub-jective terms as calamity and emer-gency. In fact it is not necessary todefine crises in order to discuss prob-lems generally common to their man-agement, including the paucity of ac-curate information, the communica-tions difficulties that persist, and the

Dr. Kupperman is Chief Scientist and Mr. Wil-cox is Chief of Military Affairs of the U.S. ArmsControl and Disarmament Agency, Washington,D.C. 20451. Dr. Smith is Professor of Mathe-matics at Oakland University, Rochester, Michi-gan 48063.404

Dr. Kupperman is Chief Scientist and Mr. Wil-cox is Chief of Military Affairs of the U.S. ArmsControl and Disarmament Agency, Washington,D.C. 20451. Dr. Smith is Professor of Mathe-matics at Oakland University, Rochester, Michi-gan 48063.404

changing character of the players asthe negotiationsfor relief leave one ormore partiesdissatisfied.In a sense, crises are unto the be-holder.What is a crisis to one individ-ual or group may not be to another.However, crises are generally distin-guished from routine situations by asense of urgency and a concern thatproblems will become worse in theabsence of action. Vulnerability o theeffects of crises lies in an inability tomanage available resources in a waythat will alleviate the perceived prob-lems tolerably. Crisis management,then, requires that timely action betaken both to avoid or mitigate un-desirable developments and to bringabout a desirable resolution of theproblems.Crisesmay arise from naturalcausesor may be induced by human adver-saries, and the nature of the manage-ment required in response differs ac-cordingly. Thus the actions requiredto limit physicaldamagefrom a severehurricane and to expedite recoveryfrom it differ substantiallyfrom thetactics needed to minimize the eco-nomic effects of a major transportationstrike and to moderate the conditionswhich caused it. Yet each also exhibitssome characteristicsof the other. Forexample,recoveryfrom the devastation

changing character of the players asthe negotiationsfor relief leave one ormore partiesdissatisfied.In a sense, crises are unto the be-holder.What is a crisis to one individ-ual or group may not be to another.However, crises are generally distin-guished from routine situations by asense of urgency and a concern thatproblems will become worse in theabsence of action. Vulnerability o theeffects of crises lies in an inability tomanage available resources in a waythat will alleviate the perceived prob-lems tolerably. Crisis management,then, requires that timely action betaken both to avoid or mitigate un-desirable developments and to bringabout a desirable resolution of theproblems.Crisesmay arise from naturalcausesor may be induced by human adver-saries, and the nature of the manage-ment required in response differs ac-cordingly. Thus the actions requiredto limit physicaldamagefrom a severehurricane and to expedite recoveryfrom it differ substantiallyfrom thetactics needed to minimize the eco-nomic effects of a major transportationstrike and to moderate the conditionswhich caused it. Yet each also exhibitssome characteristicsof the other. Forexample,recoveryfrom the devastation

wrought by the hurricane'swind andfloodwatersbrings competition amongdifferent managers whose conceptionsof recovery differ: Is the goal to re-establish the status quo, includingslums,or to seize upon the opportunityfor urban renewal?Similarly,a trans-portation strike may cause such eco-nomic chaos that the Congress-535crisismanagers-might threaten o passlaws that are detrimental to a unionleadership's prestige and control overits members.It is useful to note the characteris-tics common to most crisis manage-ment. Perhaps the most frustratingisthe uncertainty concerning what hashappened or is likely to happen,coupled with a strong feeling of thenecessity to take some action anywaybefore it is too late. This leads toan emphasison garnering nformation:military commanders press their in-telligencestaffs,and civil leaderstry toget more out of their field personneland managementinformationsystems.Unfortunately, few conventional in-formationsystemsare equal to the taskof covering unconventionalsituations,so managers n a crisis must frequentlyfall back upon experience, intuition,and bias to make ad hoc decisions (1).The problemsof uncertaintyare ex-acerbated by the dynamic nature ofmany crises. Storms follow unpredict-able courses; famine is affected byvagariesin the weather;terroristsper-form apparently irrational acts; andforeign leaders, responding to differ-ent value systems or simply interpret-ing situations differently, select unex-pected courses of action. Thus, withlimited information and resources themanager may find it difficult just tokeep up with rapid developments,letalone improvethe overallpictureof thesituation.

During a crisis, not only does aninvolved managersuffer from poor in-formation,but he has the problem ofidentifyingthe objectiveshe wishes toaccomplish and ordering them bypriority in accord with his limited re-SCIENCE, VOL. 187

wrought by the hurricane'swind andfloodwatersbrings competition amongdifferent managers whose conceptionsof recovery differ: Is the goal to re-establish the status quo, includingslums,or to seize upon the opportunityfor urban renewal?Similarly,a trans-portation strike may cause such eco-nomic chaos that the Congress-535crisismanagers-might threaten o passlaws that are detrimental to a unionleadership's prestige and control overits members.It is useful to note the characteris-tics common to most crisis manage-ment. Perhaps the most frustratingisthe uncertainty concerning what hashappened or is likely to happen,coupled with a strong feeling of thenecessity to take some action anywaybefore it is too late. This leads toan emphasison garnering nformation:military commanders press their in-telligencestaffs,and civil leaderstry toget more out of their field personneland managementinformationsystems.Unfortunately, few conventional in-formationsystemsare equal to the taskof covering unconventionalsituations,so managers n a crisis must frequentlyfall back upon experience, intuition,and bias to make ad hoc decisions (1).The problemsof uncertaintyare ex-acerbated by the dynamic nature ofmany crises. Storms follow unpredict-able courses; famine is affected byvagariesin the weather;terroristsper-form apparently irrational acts; andforeign leaders, responding to differ-ent value systems or simply interpret-ing situations differently, select unex-pected courses of action. Thus, withlimited information and resources themanager may find it difficult just tokeep up with rapid developments,letalone improvethe overallpictureof thesituation.

During a crisis, not only does aninvolved managersuffer from poor in-formation,but he has the problem ofidentifyingthe objectiveshe wishes toaccomplish and ordering them bypriority in accord with his limited re-SCIENCE, VOL. 187

berkeley admissions bias

Documents