Br. J. clin. Pharmac. (1982), 14, 325-331

STATISTICAL SIGNIFICANCE TESTS

D.R. COXDepartment of Mathematics, Imperial College, London SW7 2BZ

Introduction

Statistical methods for the analysis of data can beclassed as descriptive or as probabilistic. The formermethods include tabulation and graphical display, ofgreat importance both in detailed analysis and also inpresentation of conclusions. Probability enters statis-tical analysis in various ways. The most obvious is thatone recognizes explicitly that conclusions drawn fromlimited data are uncertain and that therefore it is agood thing to measure and control probabilities ofdrawing the wrong conclusions, partly, but by nomeans solely, as a precaution against overinterpreta-tion. The statistical significance test is the most widelyknown device concerned with such probabilities andis the subject of the present paper.The nature, use and misuse of significance tests

have been much discussed in both statistical andnonstatistical circles. The view taken in this paper isthat such tests play an important but neverthelessstrictly limited role in the critical analysis of data.There is an extensive mathematical literature on

the theory of such tests and on the distributionsappropriate for special tests. The mathematics canlargely be separated from critical discussion of theunderlying concepts, that being the concern of thepresent paper.

Nature of statistical significance tests

The first element in a significance test is the formula-tion of a hypothesis to be tested, often called the nullhypothesis. It may help to consider first how to pro-ceed when random variation can be ignored. Wemight, for instance, be interested in a hypothesisabout how some biological mechanism 'really works'.We would do the following:

(i) Find one (or more) test observations (or sets ofobservations) whose values can be predicted from thehypothesis under examination. For instance thehypothesis might predict that a certain set of relation-ships between 'response' and log dose are in factparallel straight lines. The test observations wouldthen be such as to allow assessments of nonlinearityand nonparallelism.

(ii) Make the relevant observations.(iii) If the test observations do not agree with the

predictions, the hypothesis is rejected, or at leastmodified. If the test observations agree with the0306-5251/82/090325-07 $01.00

prediction, we conclude that the data are, in thisrespect, consistent with the hypothesis. Note,though, that we cannot conclude that the hypothesisis true, because there may be other hypotheses thatwould have led to the same test data.While the general idea of significance tests follows

essentially the same steps (i-iii) there are, however,two important differences. First the observations aresubject to non-trivial variability, natural randomvariability and/or measurement 'errors' of variouskinds. Secondly the hypotheses are typically notscientific (biological) hypotheses, but ratherhypotheses about the values of parameters describing'true' values applying to populations of measure-ments. Of course, in some fields at least, theseparameters will be linked to scientific models orinterpretations of the system under study.We now adapt the procedure (i)-(iii) as follows.(i) Find a test statistic whose probability distri-

bution when the null hypothesis is true is known, atleast approximately. Also the test statistic is to besuch that large, or possibly extreme (large or small),values of the statistic point to relevant departuresfrom the null hypothesis.

(ii) Calculate the value of the test statistic for theavailable data.

(iii) If the test statistic is in the extreme of itsprobability distribution under the null hypothesis,there is evidence that the null hypothesis is untrue. Ifthe test statistic is in the centre of its distribution, thedata are consistent with the null hypothesis.More quantitatively, we calculate from the distri-

bution of the test statistic, the probability P that adeviation would arise by chance as or more extremethan that actually observed, the null hypothesis beingtrue. The value of P is called the (achieved) signifi-cance level. If P < 0.05, we say the departure issignificant at the 0.05 level, if P < 0.01 that thedeparture is significant at the 0.01 level, and so on. Inapplications it is nearly always enough to find P fairlyroughly.A natural and fundamental question now arises.

Why consider at all deviations more extreme thanthat observed? Why assess a hypothesis partly on thebasis ofwhat has not been observed.' One answer is asfollows. Suppose that we were to accept the observa-tions as just decisive evidence against the nullhypothesis. Then we would be bound to accept more

© The Macmillan Press Ltd 1982

326 D.R. COX

extreme data as evidence against that hypothesis.Therefore P has a direct, though hypothetical,physical interpretation. It is the probability that thenull hypothesis would be falsely rejected when true,were the data just decisive against the hypothesis.Note particularly that the interpretation is hypo-thetical. In the first place, at least, we use P as ameasure of evidence, not as a basis for immediatetangible action (see also Appendix E).

A simple example

Consider the following much simplified example. Tocompare a drugD with a placebo C, 100 patients eachreceive both drugs, in random order and with appro-priate design precautions. For each patient an assess-ment is made as to which is the more effectivetreatment, ties not being allowed. It is observed thatfor 60 patients D was better and for 40 patients C wasbetter.Now the null hypothesis would typically be that the

long runr proportion of preferences forD is 1/2. Othernull hypotheses could, however, arise; for instance ifa large independent study under comparable con-ditions had shown that 80% preferred D, anotherinteresting null hypothesis would be that the long runproportion in the present study is 0.8.The test statistic is the observed number of prefer-

ences for D. Under the 50% null hypothesis theprobability distribution of the number of preferencesforD can be found; independence of the individuals isassumed. The distribution, which is roughly a normaldistribution of mean 50 and standard deviation 5,is that for the number of heads observed in 100tosses of a 'fair' coin. The distribution could also befound by computer simulation although, exceptpossibly for didactic reasons, this is a poor idea whena mathematical treatment is available. In complicatedproblems however, where a mathematical treatmentis not available, it is a good idea to simulate data froma 'null' system close to that under study. Conformityof the 'real' with the 'simulated' data can be examinedat least informally.Now the probability P of a deviation as or more

extreme than that observed is, at first sight, theprobability of observing 60, or 61, or. . . ,or 100 andis about 0.029.But there is a difficulty here; what if we observed

not 60 preferences forD and 40 for C but 40 forD and60 for C? If under those circumstances we would stillhave examined consistency with the null hypothesis,our significance level P must take account of extremefluctuations in both directions, the test becomes aso-called two-tailed or two-sided test and in thisinstance P becomes 2 x 0.029 - 0.057.That is we have

Pone-sided =prob {60, or 61, or. ,or 100 0.029,Ptwo-sided = prob (60, or 61, or. . . ,or 100; 0, or.

or 39, or 40; 0.057

There has been some controversy as to which isappropriate when a priori we expect any difference tofavour D and this expectation is confirmed in thedata. To an appreciable extent the matter is one ofconvention. In reading the literature, check to seewhich has been used. In the present paper we usetwo-sided tests and leave the matter with thefollowing brief comments.

(a) In some problems the two-sided P is clearlyappropriate, because neither direction of e.fect isbiologically impossible.

(b) It will be rather rare that a substantial observedeffect in an unexpected direction can be dismissedwithout discussion. Further, retrospective explana-tions of unexpected effects can often be put forward.These are the arguments for regarding the two-sidedtest as the one for routine use.

(c) It is largely conventional whether one says thatthe two-sided P is say 0.1 and the effect is in theexpected (or unexpected) direction, or whether onequotes the one-sided P of 0.05.

Section 3 describes the nature of null hypotheses,and we then go on to discuss various points of inter-pretation.

The nature of null hypotheses

Clearly for significance tests to be of much use, sens-ible and interesting null hypotheses should be tested.A full cla .ification of null hypotheses will not beattempted, but it is useful to have the following typesin mind:

(i) null hypotheses of a technical statistical charac-ter to the effect that such and such special assump-tions are satisfied, such as that distributions have aparticular mathematical form or that preliminarytransformation of data is unnecessary. We do notconsider these further here;(ii) null hypotheses closely related to some scien-tific hypothesis about underlying biology, such as,in genetics, that two genes are on differentchromosomes. These are of critical importance incertain kinds of research, but are perhaps of lessdirect importance in clinical trials, although incomplex trials with elaborate data, it may befeasible to formulate such hypotheses;(iii) null hypotheses of homogeneity. Thesetypically are that a certain effect is the same, forexample in several independent studies or in differ-ent subgroups of patients. An emphasis in statis-tical work is frequently on designing and analysingindividual investigations so that these investiga-

STATISTICAL SIGNIFICANCE TESTS 327

tions, in isolation, lead to convincing conclusions.Yet, clearly, judgment about any issue shoulddepend on all available information; the mostconvincing conclusions are those backed by severalindependent studies and preferably by differentkinds of evidence. Tests of homogeneity play somerole here. It is a reasonable comment on somestatistical analyses that not enough emphasis isplaced on the combination of evidence fromseveral sources.(iv) Finally, and perhaps most commonly, thereare hypotheses that assert the equality of effect oftwo or more alternative treatments, for example, adrug and a placebo. One way of looking at suchhypotheses is that they divide the possible truestates of affairs into two (or more) qualitativelydifferent regimes. So long as the data are consistentwith the equality of two treatments, the data do notfirmly establish the sign of any difference betweenthem. From this point of view significance testsaddress the question: how firmly do the data estab-lish the direction of such and such an effect?

Some comments on interpretation

There follow a number of miscellaneous commentson the interpretation of significance tests.

(i) There are clear distinctions between statisticalsignificance and the more important notions of bio-logical significance and practical significance. Statis-tical significance is concerned with whether, forinstance, the direction of such and such an effect isreasonably firmly established by the data underanalysis. Biological significance is concerned withunderlying scientific interpretation in terms of morefundamental mechanisms. Practical significance isconcerned with whether an effect is big enough toaffect practical action, e.g. in the treatment ofpatients. The three kinds of significance are linkedbut in a relatively subtle way.

(ii) Failure to achieve an interesting level of statis-tical significance, i.e. the observation of valuesreasonably consistent with the null hypothesis, doesnot mean that practically important differences areabsent. Thus in the Example of Section 2 the observa-tion of55 preferences forD and 45 forC would lead toPtwo-sided = 0.37 and to the conclusion that the data arejust such as to be expected if the null hypothesis weretrue. On the other hand it can be shown that long runeffects ranging from 45% to 65% preferences for Dare consistent with the data (at the 5% significancelevel) and, depending entirely on the context, thesemight correspond to differences of practical impor-tance. By constrast with 1000 patients a split 550 forDand 450 for C would be a highly significant departurefrom the null hypothesis (two-sided = 0.0018), therange of preference for D consistent with the data is

from 52% to 58% and yet it might well be that thesewould all be of little practical importance. In thisinstance the direction of the effect is firmly estab-lished but its magnitude is such as to make the effectof little practical importance; such instances ofobservational overkill are probably rare in clinicaltrials.

(iii) The strong implication of (ii) is that, forimportant effects at least, it is necessary to calculateso-called confidence limits for the magnitude ofeffects and not just values of P. This is of crucialimportance. It is very bad practice to summarise animportant investigation solely by a value of P.

(iv) Failure to achieve an interesting level ofsignificance in a study does not mean that the topicshould be abandoned. Significance tests are notintended to inhibit the free judgment of investigators.Rather they may on the one hand warn that the dataalone do not establish an effect, and hence guardagainst overinterpretation and unwarranted claims,and on the other hand show that an effect is reason-ably firmly proved.

(v) If an effect is established as clearly correspond-ing to a bigger difference than can be accounted for bychance, two conditions have to be satisfied before thedifference can be given its face-value interpretationas, say, an improvement due to the new drug undertest. One condition is that the calculation of statisticalsignificance is technically sound in that no unreason-able detailed assumptions have been made incomputing the probabilities. The most common faultis, in effect, to overlook an important source ofvariability and thereby to underestimate the un-certainty in the conclusions. It is a key role of carefuldesign to ensure that, so far as is feasible, all majorsources of uncertainty are controlled and that theirmagnitude can be assessed. Specialised statisticaladvice may be needed on this point. This is related tothe second and more serious possibility that the effectis a bias or systematic error and not a genuine'treatment' effect. Biases are relatively much morelikely in observational studies, and comparisonsinvolving historical rather than concurrent controls,than in carefully controlled experiments involvingrandomisation and, so far as feasible, firmly enforced'blindness'. To a very limited extent biases can beinvestigated and corrected by more elaborate statisti-cal analysis (see Appendix D).

(vi) Pooling of information from independentstudies is very important, subject to tests of homo-geneity. If there is clear evidence of heterogeneitybetween studies this needs to be explained in subject-matter terms if at all possible.

(vii) If a number of independent studies are done,possibly in different centres, and only those yieldingan effect significant at say the 0.05 level are reported,it is clear that a distorted picture may emerge. Thecriterion for publication should be the achievement of

328 D.R. COX

reasonable precision and not whether a significanteffect has been found; it may be, however, thatinvestigations yielding interesting new conclusionsmerit publication in greater detail than investigationswhose outcome is predominantly negative or con-firmatory, but that is a separate issue.

The choice of tests and levels: multiple testing: choiceof sample size

A question frequently asked is 'what significancelevel should be used, 0.05, 0.01 or what?' The viewtaken in the previous discussion is that the questiondoes not arise, at least in the sense thatwe may use theobserved value ofP as a measure of the conformity ofthe data with the null hypothesis. What overallinterpretation is made or what practical action istaken depends not only on the significance level P,but also on other information that may be available,including biological plausibility on the magnitude ofeffect present and, in a decision-taking context, onthe consequences of various modes of action.

Nevertheless some rough guide for qualitativeinterpretation is useful and the following seems fairlywidely accepted, sensible, and leads to somestandardization of procedure:

P I- : accord with the null hypothesis is as goodas could reasonably be expected;

P'- : there is, for most purposes, overwhelming100 evidence against the null hypothesis;

intermediate: there is some evidence against the nullhypothesis.

Values ofP should be quoted approximately and rigidborderlines avoided. Thus in terms of the interpreta-tion of data it would be absurd to draw a rigid border-line between P = 0.051, not significant at the 0.05level, and P = 0.049, significant at the 0.05 level, eventhough in decision-making contexts with strictlylimited data such borderlines do have to be adopted.Some special problems arise when several statis-

tical tests are applied to the same data. We can dis-tinguish a number of situations:

(i) Scientifically quite separate questions can beaddressed from the same data. This raises nospecial problems. Note, however, that if one of theanalyses led to doubts about data quality, the otheranalyses would become suspect too.

(ii) The data may be searched by examiningdifferent kinds of effect and ultimately focusingattention on the most significant. For instance onemight classify the patients in various ways (age, sex,etc.) to look for differential effects. Or we may,

entirely legitimately, examine several measures ofresponse: death rate (all causes), death rate(particular causes), 'minor' critical event rate, andso on. Quite clearly if many different effects areexamined it is quite likely that a superficially quitesignificant result will emerge. Individual P valueswill apply to each effect in isolation, but to judgewhether there is overall evidence of an effect, aso-called allowance for selection is essential injudging significance. A rough rule is to multiply thesmallest P value by the number of effects examinedbefore selecting the most significant. For instancesuppose that an overall analysis of a clinical trialcomparing drug D with placebo C shows littleevidence of a difference. Analyses are made formen and women, for 'young' and 'old' patients andfor two different Centres, say taking the 8 com-binations of sex-age-Centre. The older women atCentre 2 show a difference significant at about the0.05 level. But this is the largest out of 8 com-parisons and therefore, after the rough correctionfor selection, is to be assigned a value of P of about8 x 0.05 = 0.4, i.e. it is just such as would beexpected by chance. Note that if the investigationhad been set up specifically to investigate olderwomen in Centre 2, P = 0.05; the magnitude of theeffect is the same in both context. This generalpoint is especially important because major trialsare complex with many measures of response,categories of patient etc.

(iii) The third possibility is that essentially thesame scientific issue is tackled by trying a numberof tests similar in spirit but different in detail. Thusto compare two samples one might consider the ttest, the sign test, the Wilcoxon test, the Fisher andYates score test, and so on. Each is appropriateunder slightly different conditions and usually,except possibly for the sign test, the results will bevery similar; the differences arise primarily fromthe relative weights attached to extreme observa-tions. This blunderbuss procedure of trying manytests is wrong if what is done is concealed. Inprinciple, for each specific scientific question thereshould be a single test in the light of the error struc-ture present. If a search through a number of teststatistics is carried out, an allowance for selection isessential. The simple correction outlined above ismuch too crude, being really appropriate for statis-tically independent tests rather than for closelyrelated tests. If the procedure were used, which isnot recommended, computer simulation to attainthe appropriate correction for selection seems themost practicable procedure.(iv) A rather more specialised issue concerns thenormal use of significance tests in trials in which thepossibility of stopping the trial before its nominalcompletion date is determined at least in part byformal data-dependent rules. We regard this topic,

STATISTICAL SIGNIFICANCE TESTS 329

in its practical and theoretical aspects, as outsidethe present discussion.(v) A simpler issue is the use of hypotheticalsignificance test calculations to determine anappropriate scale of investigation, e.g. the numberof patients or study duration. The argument is torequire that if a difference of an amount judgedimportant is present, then the probability oftendenoted by A3 should be at least say 0.9 that adifference significant at the 0.05 level oftendenoted by a is achieved; the values of, and a canof course be altered. Often an estimate has to bemade of such matters as the overall critical eventrate likely to be encountered. In many ways asimpler approach is to work with the standard errorof the estimated difference likely to be achieved,aiming to control this a suitable level. Somecalculation of this sort is highly desirable to preventthe setting up of investigations either absurdlysmall or on an absurdly extravagant scale. Severalsomewhat arbitrary elements are involved in theabove calculations: in principle the scale of investi-gation could be settled by decision analysis, costingthe effort spent in the study and the consequencesof wrong decisions. This is but part of the broaderissue of the role of formal decision analysis inmedical work, and a briefcomment on this follows.

Relation with decision analysis

An accompanying paper in this series (Spicer, 1982)deals with the interesting topic of decision analysis ina medical context. Superficially there is a connexionbetween the so-called two decision problem, i.e.deciding between the two alternative courses ofaction, and the significance test of an appropriate nullhypothesis. In fact, at least from the viewpoint of thepresent paper, the procedures are best regarded asquite distinct. The significance test is concerned withsome aspects of assessing information provided by aninvestigation. A decision or diagnostic rule indicatesan appropriate action, based on a synthesis of theinformation provided by data, of so-called priorinformation available from other sources and of utili-ties measuring the consequences of various actions ingiven circumstances. The decision rule requires agreater variety of quantitative input than does asignificance test and in a sense reaches a corres-pondingly stronger conclusion.

Reference

SPICER, C.L. (1982). Statistical decision theory andclinical trials. Br. J. clin. Pharmac., 14 (in press).

Appendix

Some further issues, including some idealised mis-conceptions

There follow a few statements involving significancetests, together with brief comments. Probably noneof A-C has ever been made in quite the form givenhere, but the statements may nevertheless pinpointpotential errors of interpretation. D and E outlinesome further issues of importance, marginal to themain theme of the paper. By 'effect' is meant, forexample, a difference in survival rate as between twoalternative regimes. The null hypothesis is that thisdifference is zero.

A. The effect is significant at the1% level: that is, theprobability ofthe null hypotheses ofzero effect is only1%. Not so! This is not what significant at the 1%level means (see Section 2). So-called Bayesiansignificance tests do allow the calculation of proba-bilities of hypotheses being true (or false), but requirea great deal of extra information. Most of the generalpoints in the paper apply also to Bayesian significancetests; see Appendix E.

B. Oftwo independent studies ofthe same topic, onegives no significant effect and the other givessignificance at the 1% level. Thus the two studies areinconsistent. Not necessarily so! A modest effect anda rather larger effect in the same direction will oftensatisfy the conditions stated. The magnitudes of theeffects should be examined, and their mutualconsistency considered. This is usually done bytesting the null hypothesis of homogeneity, that themagnitudes are equal in the two studies. Subject toconsistency, a pooled estimate of the effect can becalculated.

C. An effect is highly significant (say atl% level) butis unexpected and apparently inexplicable on bio-logical grounds. There is thus a conflict between statis-tics and biology; biology should win. Up to a point,this is reasonable enough. The conflict is not, how-ever, between statistics and biology, but between thespecific data and the biology: the statistical analysisserves to make the conflict more explicit and ratherless a matter of personal judgment than mightotherwise be the case. Resolution of the conflict willoften require new data but sometimes it may besought on the initial data via one or more of thefollowing: (i) there is a bias (systematic error) in thedata; (ii) the calculation of P is in error, e.g. byoverlooking some source of variability; (iii) anextreme chance fluctuation has arisen; (iv) the initialbiological understanding is faulty. Possibilities (i) and(ii) can be the subject of further statistical investi-gation. Possibility (iii) is unverifiable. As to (iv), the

330 D.R. COX

traffic between biological understanding and more

empirical studies should be two-way; to disregard allempirical studies in conflict with prior expectationwould be very dangerous.

D. Biases can be more important than randomerrors and biases are not susceptible to statisticalanalysis. Biases are indeed likely to be relativelyimportant in large studies, where the effect ofrandomvariation is controlled by the averaging of responses

over a large number of patients. It is one of theprimary objectives of careful design to eliminatebiases, by balancing or by randomizing (with con-

ceaiment) all important sources of variability. Inobservational studies, for instance when historicalcontrols are used instead of the much more desirablerandomized concurrent controls, biases are appreci-ably more likely and certainly the main sources ofpotential bias should be examined and if necessary

adjustments calculated. In broad terms, if the source

of bias can be identified and is short of a total con-

founding with the treatment effect under study, thenthere is a possibility of largely removing the bias bysuitable statistical analysis. On general grounds it is,of course, very undesirable that interpretation shoulddepend in an essential way on elaborate analysis:direct appreciation of the meaning and limitations ofthe data is thereby inhibited.

Biases are of three broad types: (i) those arisingfrom systematic differences between the two (ormore) arms of the trial, involving patient character-istics on entry; (ii) those concerned with theprocedures for measurement of response and othervariables; (iii) those concerned with the implementa-tion of the treatment and with the care of the patientduring any subsequent period of observation.

Randomization is intended to ensure the absenceof the first kind of bias. Sometimes, however,randomization may be defective; further, in observa-tional studies appreciable differences may arisebetween a new 'treated' group and a historical controlgroup and these will need adjustment by some

process of standardization, for example by analysis ofcovariance. Significance tests are now more complexbut not different in principle. A major concern is thatonce it is clear that two groups are unbalanced withrespect to an observed property (e.g. age or sex) thepossibility of a difference with respect to an un-

measured though important property is enhanced.This would lead to a bias not removable by analysis.

Biases connected with measurement can ariseeither when 'blindness' is impossible or seriouslyineffective or when different observers, laboratories,etc. are involved. It is difficult to suggest an effectivepractical method of dealing with the first kind of bias:independent confirmation of particularly sensitiveobservations may sometimes be feasible, possibly ona sample of patients. Interobserver and inter-

laboratory systematic differences can in principle beestimated and eliminated, provided of course therather disastrous situation is avoided wherein, saybiochemical measurements in the two treatment armsare made in two different laboratories. Externalevidence on the degree of standardization achievedwill always be valuable.

Biases of the third kind, concerned with treatmentimplementation, are in some ways the most awkward.A conventional answer is that the usual comparison ina randomized clinical trial tests the total con-sequences of being assigned to one treatment armrather than to the other. Thus subsidiary effects, suchas differences in general level of care, different levelsof compliance, unacceptability of side effects, etc. arepart of the consequences of the treatments understudy. From this point of view, biases are spiritedaway by definition. It is clear that this approach maysometimes be quite unsatisfactory, both scientificallyand for decision making about the treatment ofindividual patients. If the conditions of thisexperiment are quite different from those obtainingin practical use, what is sometimes called an externalsystematic error arises. Practical ways of dealing withthese matters of 'treatment bias' are not welldeveloped. In principle, if properties associated withtreatment administration and subsequent care can bemeasured, they may be either deliberately varied bydesign or alternatively measured for each patient orfor small groups of patients and their effect on themajor response assessed by suitable analysis. Thedifficulties of measuring accurately such aspects ascompliance are well known.

E. The Bayesian approach to statistical tests allowsthe incorporation of external evidence, that is alwaysavailable, and avoids the conceptual difficulties ofordinary significance tests. It would be out of place inthe present paper to write at length on the founda-tions of statistical inference. These foundations havefor many years been under active discussion anddebate: such discussion should be taken as evidenceof the intellectual vigour of the subject and not at allas implying an unsoundness in application.

Briefly, significance tests as described in the mainpart of the paper aim to summarise what the data tellabout consistency with a hypothesis or about thedirection of an effect. The P-value has, before actionor overall conclusion can be reached, to be combinedwith any external evidence available and, in the caseof decision-making, with assessments of the con-sequences of various actions. Clearly the P-value is alimited aspect: note also point (iii) of some commentson interpretation where the importance of calculatinglimits of error for the magnitude of an effect isstressed.

It appears superficially very attractive tostrengthen interpretation and to bring in the external

STATISTICAL SIGNIFICANCE TESTS 331

information, by use of the so-called Bayesianapproach. In my opinion, there is room for differentapproaches in different contexts, but the argumentsare fairly persuasive against the Bayesian tests in mostapplications to the analysis and interpretation ofdata. First it may often help to keep separate theimpact of data and external information. Secondly, toput the external information in the required quanti-tative form raises formidable conceptual and practicaldifficulties. (In some cases, such as certain problems

of diagnosis, external information is in the form of astatistical frequency, and then there is no dispute thatthe Bayesian approach should be applied.)

Finally in problems in which the direction of aneffect is under study and the external information isrelatively weak, the one-sided P value is approxi-mately the (Bayesian) probability that the time effecthas the opposite sign to the effect in the data and thetwo approaches are largely reconciled.