chalmers method for assessing the quality of a randomized c

A Method for Assessing the Quality of a Randomized Control Trial

Thomas C. Chalmers, Harry Smith, Jr., Bradley Blackburn, Bernard Silverman, Biruta Schroeder, Dinah Reitman, and Alexander Ambroz

From the M o u n t S toa t M e d i c a l C e n t e r a n d M o u n t S toa t S c h o o l o f M e d i c i n e o f the Cz ty

U m v e r s t t y o f N e w Y o r k , N e w Y o r k , N e w Y o r k

ABSTRACT: A system has been constructed to evaluate the design, lmplementat lon, and analysis of randomized control trials (RCT) The degree of quadruple bhndlng (the randomizat ion process, the physic ians and pat ients as to therapy, and the physicians as to ongoing results) is considered to be the most important aspect of any trial The analytic techniques are scored with the same emphasis as is placed on the control of bins in the p lanning and implementa t ion of the studies. Descnpt ,on of the pat ient and treatment materials and the measurement of various controls of quali ty have less weight An index of quali ty of a RCT is proposed with its pros and cons. If pub l i shed papers were to approximate these prIncWles, there would be a marked ,mprovement m the quali ty of r andomized control trials. F~nally, a reasonable s tandard design and conduct of trials wdl facdttate the mterpreta t lon of those with conflicting results and help in making vahd combinat ions of unders tzed trials

KEY WORDS randomtzatzon, control trtal, bzas

I N T R O D U C T I O N

T h e c o m p a r i s o n of o n e t r e a t m e n t w i t h a n o t h e r r e q u i r e s tha t the p a t i e n t s b e i n g c o m p a r e d r e s e m b l e o n e a n o t h e r in all p o s s i b l e prognostmcaUy i m p o r - t an t w a y s excep t for t he t r e a t m e n t s u n d e r c o m p a r i s o n . If c o m p l e t e m a t c h i n g c o u l d b e a c h i e v e d , the t e s t m g of t h e r a p i e s w o u l d b e e a s i e r . H o w e v e r , the m a r k e d v a r i a b i l i t y of h u m a n r e s p o n s e s to b o t h d i s e a s e s a n d t r e a t m e n t s a n d the r e a l i t i e s of o b s e r v e r e r ro r a n d b i a s m a k e p r o p e r m a t c h i n g i m p o s s i b l e to a t t a i n . For th i s r e a s o n r a n d o m i z a t i o n is r e c o g n i z e d as t he b e s t a v a i l a b l e t e c h m q u e of a p p r o x i m a t i n g the e q u a h t y of p a t i e n t g r o u p s b e m g c o m p a r e d . T h e v a l i d i t y of a c l in ica l t r ia l , h o w e v e r , d e p e n d s on m u c h m o r e t h a n the

The authors prefer the term "'random~=ed control tmal'" to the more common term "'randomized controlled trtal "'

Address requests for reprints to Thomas C Chalmers, M D , President and Dean, The Mount Stoat Medzcal Center, One Gustave L Levy Place, New York, NY 10029

Recewed October 18, 1979, accepted September 17, 1980

Controlled Chnlcal Trials 2, 31-49 (1981) 31 H 1981 Elsevaer North Holland, lnc, 52 Vanderbtlt Avenue, New York NY 10017 0197-2456/81/010031019502 50

32 Thomas C Chalmers et al

p roper conduct of the r andomiza t ion process. Interpretat ion and appl icat ion of the results depends on an adequa t e descr ip t ion of the pat ients accepted as well as ot the pat ients not accepted in the trial, the exper imenta l and s u p p l e m e n t a r y t r ea tment r eg imens , wi thdrawals , b l inding where appropr i - ate, t e shng of h o w well the rules have been followed, and the use of p roper statistical analysis This p a p e r ou thnes a sys tem for r e w e w l n g the pub l i shed results of r a n d o m i z e d control trials so that an a s ses smen t of the quah ty of the research can be made .

The re are a n u m b e r ot r easons for a t t e m p t i n g to deve lop a useful t echn ique for a ssess ing the q u a h t y of a r a n d o m i z e d control trial For example , in m a n y ins tances small and Inconclus ive s tudies have been repor ted to show no effect of a therapy that s eemed p romis ing enough to war ran t I n d e p e n d e n t s tudies . It is conceivable that useful clinical effects migh t be bet ter unde r s tood and accepted ~f one could combine the data f rom several w e l l - d e s i g n e d s tud ies . O n the o ther hand , large s tud ies s o m e t i m e s have conflict ing conclusions. Is this the result ot the inclusion of different pat ients , e m p l o y m e n t of different therapies , i m p r o p e r control of bias, or i m prope r analysis of the results? Valid resolut ion of conflicting conclus ions would be facili tated by an a s sessmen t of quahty.

Verification and e v o l u h o n of s t andards for r a n d o m i z e d control trials would i m p r o v e the des ign and i mp lemen ta t i on of future RCTs The degree of adherence to these s tandards would be a gu ide to the reso luhon of the p rob l ems lust cited.

I N S T R U M E N T S FOR THE A S S E S S M E N T OF AN RCT

Four aspects of an RCT are necessary for the a s ses smen t of the quali ty of the research: (1) basic descr ip t ive material , (2) the s tudy protocol, (3) the analysis of the data, and (4) data useful for potent ia l combin ing of several RCT results

In this p a p e r the descr ip t ion of the s tandards for a good RCT is followed by the p r e s e n t a h o n of a scoring sys tem des igned to allow a quant i ta t ive a s ses smen t of repor ted trials Al though it is based on our accumula t ive exper ience in the analysis of trials, the sys tem ~s a rb i t ra ry and requires v a h d a h o n . At tempts at va l ida t ion are unde r way. Some cor re lahons of ev idence of qual i ty with reality have been found in s tudies of the t rea tment of duodena l ulcer [1].

BASIC DESCRIPTIVE MATERIAL: FORM 1 (APPENDIX)

The in format ion on Form 1 (Appendix) includes basic ident i fy ing data that are used in d e t e r m i n i n g correlat ions be tween these l den t l fymg data and the var ious good and bad aspects of the trial indicated by the r ema in ing qual i ty forms. In addi t ion to these essential ident i fy ing data, o ther po tenha l ly impor t an t facts are recorded, such as whe the r or not the iournal articles are k n o w n to be peer rev iewed; sources of financial suppo r t for the trial; whe the r or not a blostat ls t ic lan is an au thor or m e n t i o n e d as a consultant , sources and n u m b e r s of pa t ien ts The type of trial is also recorded.

Finally, the pape r ~s read to d e t e r m i n e the au thor ' s s t a tement of s lgmhcan t

Assessmg a Randormzed Control Trial 33

f indings. This is done before any assessment is made of the val idi ty of the trial and the statistical m e t h o d o l o g y used by the author(s) This represents the au thor ' s op in ions of the facts as he/she unde r s t ands and interprets them ei ther stat ist ically or o therwise . Form 1 can be filled out by a t r amed secre tary or technician.

THE STUDY PROTOCOL: FORM 2 (APPENDIX)

Form 2 (Appendix) IS filled out by two or more professional ly t r amed people with exper ience in clinical trials To m l m m i z e observer bias, the Methods section ot the trial is pho tocop ied by the technician in such a way that the authors , source, results, and discuss ion thereof are omi t ted (differential pho tocopy ing ) . Forms 1, 3, and 4 are not seen or comple ted by these evaluators until after Form 2 has been filled out Differences a m o n g evaluators are resolved in conference. The des ign of the clinical trial is cons idered the most Impor t an t part of an RCT The specific e lements of good protocol des ign and p lann ing are d iscussed in the fo l lowmg sections, where the i tems are p resen ted in the s ame order as in Form 2 (Appendix)

Selection Description

If the conclus ions of an RCT are to be mte rp re ted proper ly and to be appl icable to the practice of medic ine , the sublects mus t be descr ibed in such a way that the reader obta ins a clear pic ture of the pa t ien ts s tudied A detai led descr ip t ion of both the ent ry and rejection criteria is requi red for a wel l -des igned trial. A mere s ta tement that so m a n y pahen t s with such and such d iagnos is were r andomized is unsat isfactory. Someth ing in be- tween these two ext remes is cons idered part ly correct. This descr ip t ion usually reflects the stated goal of the mvest~gators in the protocol Whether or not they ach ieved those goals is mdlca ted by the a s sessmen t of i tem 2 2, pa t ients actually rejected, ~tem 2.10, the n u m b e r of pahen t s required to detect an ~mportant difference, 2. l l , d l s t n b u h o n of p re t r ea tmen t variables, i tem 3.7, Form 3, a re t rospect ive analysis of the pat ients entered

Reject Log

Equally as impor t an t as the documen ta t i on of the popu la t ion s tudied ~s a descr ip t ion of the eligible p o p u l a h o n no t accepted for the trial A log of those pat ients who are not al lowed to enter the trial should be kept. It should mc lude specific reasons for their nonehg lb lh ty This log IS most useful in the mon i to r ing process to be certain that one type of pa t ient ~s not be ing inord ina te ly omit ted. For example , in the coope rahve trials of the t rea tment of C rohn ' s d isease it was mos t useful to the trial moni tors even though this ln format lon was not hsted In the final report [2] It is also impor t an t in trials of surgical therapy to present data on those not r andomized so that some Idea of potent ia l bias in p a h e n t se lechon can be obta ined . Ideally, the ou tcome in these relected pat ients should be compared to the ou tcome in the r a n d o m i z e d pat ients to detect any ~mportant b~ases in the selection of pahen t s , especial ly at different centers in coopera t ive s tudies


Withdrawals After Randomization

In most repor ted trials, a n u m b e r of pat ients drop out or are w i thd rawn after the trml is underway . Because the lmt~at~ve for removal ~s otten absent , w l t h d r a w a l s a n d d ropou t s a r e c o n s M e r e d Mentlcal in this pape r Dropouts shou ld be hs ted by d i agnos i s , t r ea tmen t , and reason tor w i thd rawa l , whe the r wi thdrawal occurs as a result of pat ient or ~nvesttgator ~mt~at~ve D~tferent k inds of wi thdrawal s could bias the t, nal m a k e u p of each t reatment group, thus d l m m l s h , n g the efficacy ot the r andomiza t ion procedure for ob ta in ing s imilar k inds ot pa t ients in each t rea tment group. In this category, RCTs that do not men t ion wi thdrawals , or whose wi thdrawals exceed 10% in the case of pa t ients fol lowed for 3 mon ths or less, or whose wzthdrawals exceed 15°o for longer than 3 -mon th fol low-up should be carefully scrutz- razed. Removal of pat ients by the invest igator atter comple t ion of the s tudy ~s a ser ious defect

Description of the Therapeutic Regimens

The descr ip t ions of both exper imenta l and all ancll larv therapies must be comple te enough to allow p rope r ~nterpretat~on of the results and rephcat~on in other s tudies or practice A s tudy should descr ibe the exact t iming and a m o u n t of daffy therapies ~n the trial and all a l lowable other therapies When relevant , diet and activity al lowed should be men t ioned .

Description of Placebo

If a trial used ,3 placebo, it is insufficient merelv to men t ion that a placebo was g iven Ident i ty of appea rance and taste where apphcab le should be d o c u m e n t e d according to a carefully p lanned protocol However , since this is practically never done , the pape r should ment ion that the placebos are Mentlcal In one s tudy 13], the bl ind was broken by approx imate ly half the pa t ien ts when they b roke open the capsules and tasted them Removal o! these pat ients nega ted a s~gnlflcant difference ~n s y m p t o m a t i c benet~t

BLINDING PROCEDURES The mos t impor tan t de t e rminan t s of the quah ty of a r andomized control trial are the b h n d l n g procedures . Four separa te b l ind ing assessments are made: (1) b h n d m g of the r andomiza t ion process; (2) b l inding ot pat ients , (3) b h n d l n g of phys ic ians , and (4) b l inding of par t ic ipat ing physic ians with regard to the ongo ing results of the trial. The relative tmpor tance ot this g roup of ac twl t les is var iable and d e p e n d s on the type of s tudy

Randomization Blinding When a pa t ien t is accepted for s tudy, the inves t igator mus t not be able to suspect which t rea tment is next in hne. if the in format ion is potential ly

Assessing a Randomized Control Trial 35

available as a result of a c rude me thod of allocation such as birth date, chart number , or an open table or randomiza t ion , there is an oppor tun i ty for bias to inf luence t rea tment ass ignment . This step is much more impor tan t than the b l inding of therapies when the endpo in t is death because there are often obvious prognost ic factors in such studies, whereas the de te rmina t ion of the fact of death is not subject to bias. Randomiza t ion should be carried out by the me thod of consecut ively n u m b e r e d opaque envelopes or te lephone ass ignments or, in the case of a drug trial, having drugs prepackaged and n u m b e r e d for each pat ient before the t ime of randomiza t ion .

It is especially impor tan t to bh n d the randomiza t ion process when the t reatments are not b l inded or the t rends in the s tudy are known to the admi t t ing physic ian (see unde r Blinding of Physicians and Patients). For example, if there is bias for or against a therapy that is thought to be next up, it is easy to allow some reason to exclude the pat ient in w h o m a suspec ted ou tcome migh t favor one t rea tment or the other . The most effective means of accomphsh ing this kind of bias is via the in formed consent procedure . The good physic ian can persuade almost all his pat ients to enter a s tudy and sign the Informed consent p rocedure if he or she really believes that it is in the pat ient 's best interest. If his bias is toward one t rea tment or the other , and he knows the wrong one is next in line, unconsciously it is easy to influence the pat ient not to consent to loin the study. Simon [4] stated this ve ry clearly: "Perhaps , the major accomplish- ment of the in t roduct ion of randomiza t ion ir~ m o d e r n clinical research has been the control of (patient) selection bias ." We add to this comment by insist ing on the b l inding of the randomiza t ion process. The val idi ty of this concept is now u n d e r invest igat ion by compar ing the dis t r ibut ion of p values for p r e r andomiza t i on variables of s tudies whose r andomiza t ion process was not b l inded with those s tudies that were bl inded. Presumably, bias will reveal itself to a greater than expected degree when randomiza t ion is not b l inded.

Occasionally in a clinical tmal there will be purposeful decep t ion by par t ic ipat ing physic ians who feel that their responsibi l i ty to the pat ient comes first [5] and who do not elect to w i thd raw from the s tudy [6]. To avoid such tempta t ions the ass ignment envelopes should be n u m b e r e d successively and be truly opaque and the pat ient 's name on the outs ide of the enve lope should be re tu rned to the biostat lst ician for verification of ass ignment . Te lephone ass ignment is safest, but occasionally the physic ian mis takenly hears the t rea tment he prefers , so verification is necessary.

Blinding of Patients

When the ou tcome (endpoint) 1 is subject ive and when dropouts or noncom- pliance are likely to result from pat ients knowing their therapy and becom- ing d iscouraged, b l inding of pat ients is especially desirable.

'The words " endpomt ' " and " o u t c o m e " have the same m e a m n g in thls pape r


Blinding of Physicians as to Therapy Received For soft as well as hard e n d p o m t s such as death, b l inding of ph.vslclans as to therapy is ~mportant In the latter case, knowing that a pat ient is r ece ,wng what ,s suspec ted of be ing the less-effectzve t rea tment may lead to compen- sa tory efforts that would be bad tor both the s tudy and for the pahen t s For example , the phys ic ian may Inst i tute some addl t ,onal adlunct~ve therapy not p rov ided m the protocol or the nurse may be g~ven dlrect ,ons to check on the par t icular pa t ien t more often than stated ~n the protocol

Blinding of Physicians and Patients as to the Ongoing Results of the Study

For both scmenhhc and ethical reasons, b h n d m g ot phys ic ians and pa tmnts as to the ongo ing results of the s tudy ,s mlpor tan t If the t rea tments and r andomi za t i on process are not adequa te ly bhnded , and ,t is often imposs ib le to accomphsh the former , knowledge of the trial t rends could lead the conscient ious phys ic ian to distort the ,ntake (as explained earher under Randomi za t i on Bhnding) or to mf luence the wi thdrawals dl t ferenhal ly (as expla ined earlier t inder Wi thdrawals After Randomtza tmn) From the ethical s t andpo in t , the phys~cian can no longer ask a p a h e n t to loin a s tudy or keep h im or her m the s t udy ff the phys ic ian knows ot an m~presslve trend Al though the s i tuat ion ~s s t n k m g only when stat~stical s~gmf, cance has a lmost been reached, the p rmc lp l e apphes , albe, t de m , m m u s , wheneve r there is any trend [6].

A d a t a - m o m t o n n g commi t t ee or poh%'y adv i so ry board charged w,th s tudy ing the results and no t f fymg the ~nvestigators when a change In protocol should be cons idered ~s a p roper mclus ,on in the ~nformed consent p rocedure [7].

TESTING PROCEDURES The next g roup of d e t e r m m a n t s of a good s tudy has to do w~th cer tam m e a s u r e m e n t s that the inves t igators should under t ake before or dur ing the

study.

Sizing the Study There should be ev idence that a pr ior es t imate of the n u m b e r s of pat ients requi red has been done. An excellent pape r lists ~n ~ts Methods sechon the expected control e n d p o m t , the i m p r o v e m e n t of clinical interest that should not be missed , the chosen levels of risks (~ and /3), and the n u m b e r s r eqmred . Few s tudles do this [8].

Distribution of Pretreatment Variables O n e me thod of test ing r a n d o m l z a t m n is the m e a s u r e m e n t of the pretreat- men t charac tenshcs of the g roups be ing compared It is m s u f h c m n t to list only d e m o g r a p h i c compar i sons such as the usual age and sex d l s t n b u h o n .


The d l s tnbu t ion of known prognost ic features by t reatment category should be shown in tabular form These data are critical in assessing the efficacy of the bl inding of the randomiza t ion . If the d is t r ibut ion among the t reatment groups of prognost ical ly impor tan t variables is d i spropor t iona te , the cause may be chance or a p rewous ly unsuspec ted bias. In e i ther case, the results of the trial come unde r ques t ion and analysis mus t be carried out to de te rmine whe the r the observed difference, or lack thereof, could be more hkely at t r ibutable to the therapies or to the pre t rea tment d is t r ibut ion of the pat ients

Testing of Blinding

It is not sufficient to assume that a doub le -bhnd procedure is effective In good studies the physic ians and their pat ients are qu izzed at the end of the s tudy to de t e rmine whe the r or not they have guessed the medica t ion involved. The data may be impor tan t in in terpreta t ion of the results [3].

Compliance

The Importance of pat ient compliance in clinical trials has been emphas ized repeatedly. In the case of oral hypoglycemlc drugs, those who took their hypoglycemlc pills had the h igher death rate [9]. The oppos i te relat ionship seemed to exist in the case of an t ihyper tens ive therapy [10], those who took their pills fared better. Some objective methods of ver i fying that pat ients are conforming to the protocol must be described. For example, in a drug trial pill counts would be acceptable. Subjective assessments of compliance are often used. In these cases, a measure of the val idi ty of the subject ive measure should be given. In some trials the assessment of comphance is cons idered not applicable, for example, when a medicat ion is ordered in the hospital the fact that it is actually given is practlcally never directly ver ihed and would not normally be cons idered impor tan t enough to be publ ished.

Biological Equivalent

By biological equiva len t ts meant some a t tempt to measure the therapeut ic agent after absorp t ion or injection, preferably in its active form. Pre- and pos t -vago tomy or post-H2 blocker measurements of gastric acid ou tpu t in therapeut ic trials of pept ic ulcer are good examples of this category [1]. Occasionally a blood or ur ine level of the active agent will also be a measure of compliance.

STATISTICAL ANALYSIS: FORM 3 (APPENDIX) In the analysis of the data ga thered in any chnlcal trial there are certain mmmmal prmnoples that are Indicative of quahty. The analysis has been cons ide red less impor t an t than the des ign and Implementa t ion of the protocol plan. It is recognized that the elements wi th in this category are

38 Thomas C. Chalmers et al

arbitrary. However , those procedures that, in our opin ion , have been considered most proper are discussed next with the items presented in the same order as in Form 3 (Appendix I).

Statistical Significance of Major Endpoints When significance is reported, it should be given in such a way that the reader can make the actual calculations. Both the test statistic and the observed probabi l i ty values should be stated If the observed probabil i ty level is presented but not the test statistic, or the test statistic ~s given wi thout the p robab lh ty level, the reader may have trouble verifying the statistical conclusions. The absence of both is unacceptable

Posterior/3 Estimates of Observed Difference for Negative Tria ls

If the difference be tween the compared treatments turns out not to be statistically significant, then a d iscuss ion of the Type II error and its probabi l i ty should be included in the results or discussion section of the trial Actually est imating a poster ior /3 for a clinically interesting difference is a plus. Ment ioning the problem and admit t ing a necessity for more patients ("problem needs further s tudy to be sure a difference is not being missed") is an indicat ion of a concern for sample sizes, but the report remams incomplete in a statistical sense.

In posit ive trials, a compar ison of the end result with that indicated in the prior d iscuss ion of p r imary endpoin t , sample size, and impor tant clinical differences would be desirable

Statistical Inference

Confidence limits. Confidence intervals should be provided for the propor- tions, rates, or means used as tnal endpoin ts

Life-table or time-series analysis For all discrete endpolnts , such as mortality or morbidi ty, even for trials of short durat ion, life-table or t lme-senes analysis should be carried out. If more than one survival curve is reported, compar ison of the curves should be done by the appropr ia te statistical procedure, e.g., log-rank test. The data should be presented in a form that would allow the reader to reproduce the survival curve or curves.

Regresswn analyses or correlations. Where appropriate , such as when the critical response is a funct ion of drug dosage or predetermined quantifiable clinical factors, regression analyses or correlations should be carried out.

Appropriate Statistical Analyses Most RCT papers report some statistical analyses. This section on Form 3 (3.4) Is used to provide an overall assessment of the statistical analysis. A trial that meets all the preceding criteria and that shows the end results


clearly and precisely is ou ts tanding . Al though this is s o m e w h a t difficult to do, our g roup of evaluators have been consis tent in mak ing this assessment .

Handling of Withdrawals Withdrawals (dropouts) mus t be men t ioned in the results or d iscuss ion sections. D e p e n d i n g on the type of s tudy, wi thdrawals are handled in m a n y different ways, for example:

1. The results are ana lyzed with the d ropou t s handled in all feasible ways. The V.A. co rona ry a r te ry s u r ge ry s t udy is a good example of this approach [111.

2 Patients are cons idered as an end result for the g roup to which they were or iginal ly a s s igned regardless of wha t h a p p e n s to them. In sp i te of cri t icism, this was the best way of hand l ing the UGDP data [9]

3. Patients can be counted as an end result at the t ime of wi thdrawal . 4. Patients can be ignored or e l iminated from the s tudy at the t ime of

w i t h d r a w a l - - t h u s , not counted as an end result Al though this is done mos t often, it is rarely a defens ib le procedure .

5. Patients may change g roups and be cons idered as an end result in the new group. This is never defenslble .

Statistical Discussion of Side Effects Side effects should be repor ted and discussed, and adequa t e s tahshcal analyses should be carried out C o m p a r i s o n of percentages with a statistical test of s ignif icance and the obse rved probabi l i ty should be done if the sample size warran ts it. Both the test and the obse rved p robab i l i ty level mus t be recorded. In the event of "no s lgnihcant d i f ference ," the Type II error should be m e n h o n e d . Listing s ide effects wi thou t analysis or not m e n t i o n i n g them at all is unsat isfactory. Ment ion that no side effects occur makes the ques t ion nonapphcab le .

Retrospective Analyses

In a clinical trial the post hoc ahalysis of s u b g r o u p s that were not specified at the outset of the trial are t e rmed re t rospect ive because , hav ing been selected after the data are in, they may be subject to selection bias like any de novo re t rospect ive study. These re t rospect ively selected s u b g r o u p s should be examined with due caut ion and recogn ihon that no hard t rea tment conclus ions can be d r awn from them. They are useful to sugges t new s tud ies . In add i t i on , a good re t rospec t ive analys is will po in t out any inadequac ies due to the r andomiza t ion , d ropouts , or comphance difficulties and d e t e r m i n e their effect on outcomes .

Blinding of Statistician

In s o m e s tudies it may be impor t an t for the stat ist ician to be work ing with coded material and, in the case of mul t icenter trials, to p resen t the material in a b l inded m a n n e r to data mon i to r ing commi t t ees If the stat is t ician has


been properly involved m the p l a n i n g stages of the protocol and trial, this can easily be done wi thout causmg problems The various statistical reports on the UGDP studies hsted by author according to his/her biases [9,12,13], are good examples of this phenomenon . In one s tudy the treatments were ldenhfied by code numbers that vaned and differed for main and sMe effects [ 14]

The Problem of Mult iple Looks The authors should consMer what mfluence "mult iple looks" might have on the levels of risk and indicate the techniques for statistical d e o s l o n s on when to stop the trial. The use of sequenttal destgns in climc trials recognizes this problem. In the rare case of a fixed sample size trial with no early looks, this section is not applicable.

PRESENTATION OF RESULTS: FORM 4 (APPENDIX)

Although many papers state the specific oblechves of the study, ~t ~s often very difficult to find the results in the paper that apply to the specific objectives The end results are obscured by a myriad of tables. Thus we have placed the clear presenta t ion of results m a separate category Form 4 is also used to record speof lc ~tems of mterest from each type of study, items that may be necessary for sensible combinat ions For instance, In a s tudy of ant icoagulant therapy of acute myocardial infarction, prognost ic details of the patients were placed on this sheet [15]. The important elements of Form 4 are de scnbed below

Dates of Study Studies should hst the starting and s topping dates of accession so that the results can be interpreted in the hght of other changes in therapy that may have been going on at the same time, as in the case of the advent of coronary care umts [16].

Results of Prerandomization Data Analysis

Analyses done to assess the baseline comparabi l i ty of the s tudy groups may give the interpreter of the results of the trial reasons for being very cauhous or even susp loous . Thus we record here if the analysis has been done well and if so what IS the prognost ic direction If one is to use trial results for potential combmat lon with other trial results, this item may be critical m dec ldmg whether or not to do so.

Tabulation of Events Employed as Endpoints The good s tudy will tabulate all events employed as endpomts so that the reader can check the calculations and also use the data more effechvely m c o m b m m g the ,esults of different studies

Assessmg a Randomized Control Trial 41

Timing of Events

Quite often mortali ty rates are presented as crude mortality rates for a specific t ime period, e.g., a 5-year death rate. If the time of each event (e.g., death) and the h i e for each wi thdrawal are given, the reader can construct survival curves. The latter data should be given in some form in the paper, or a justification for the repor tmg of k-t ime death rate should be given.

THE INDEX OF RCT QUALITY

O n e objective of the au thors ' research has been to create a quantmtahve score reflectmg the overall quali ty of each RCT. In creating this overall index, three aspects of each trial were graded: (1) the design of the trial and the trials parotocol (Form 2, weight = 0.600), (2) the statistical analysis of the trial (Form 3, weight = 0.300); and (3) the presentat ion of trial results (Form 4, weight = 0.100).

A score is awarded to each item on each form when the item is applicable. The total score is then divided by the total possible score, 1.e., nonappl icable item scores are not counted in the denomina tor . This then becomes a kind of bat t ing average Finally, an overall quality mdex for each trial is obta ined by a d d m g up the item s c n r o q from all three forms and d iv id ing by the total possible score.

Table 1 Illustrative Grading of a Trial"

No of Total pomts possible earned points

Bhnded evaluation (Form 2) 1. Selection description 3 3 2 Number of patients seen or

relect log 0 3 3 Withdrawals 3 3 4 Therapeutic regimens

dehmhon 0 3 5A Control regimen (placebo)

appearance 1.5 1 5 5B Control regimen (placebo)

taste 0 1.5 6. Randomlzahon bhndlng I0 10 7 Bhndmg of patients 8 8 8 Bhndmg of physicians re

therapy 8 8 9. Bhndmg of physicians re

results 4 4 10 Prior eshmate of numbers 3 3 11 Testing randomlzahon 3 3 12. Testing bhndmg 0 3 13 Testmg compliance 3 3 14 Biological equivalent 3 3

49 5 60 conhnued


Table 1 ( c o n t i n u e d )

Unbhnded analysis data (Form 3) 1 On malor endpomts 2 Posterior/3 est imate of

observed difference for negative trials

3A. Statistical Inference Confidence hmlts

3B. Statistical reference L,te-table or t ime-series anaJys~s

3C Stahshcal reference Regression analysis correlahon

4. Appropr ia te stahstlcal analys~s 5 Handhng of wi thdrawals 6 Side effects, stat,shcal

discussion 7 Proper re t rospechve analysls 8 Bhndmg of s tahshclan or

analyst re results 9 Multiple looks considered

No of points earned

2 4 2

3 2

0 3

24

Total possible points

30 Supplementa ry data p resen tahon

(Form 4) 1 Dates of start ing and ending

a ccess~ on 2 2 2A. Results of prerandomlzat~on

Data analysis 2 2 3 Tabulation of events employed

as endpoln ts for each treatment 2 2

4. Timing of events 4 4

Total points earned Quahty score. 0.835

10 10 83 5 100

"Coronary Drug Project Asp~nnln coronary heart disease ]ChromcDts 29625-642, 1976

T h e s c o r i n g s y s t e m ~s o n e " b e s t g u e s s " at th ts t ime . A m o r e s e n s i t i v e s c o r i n g s y s t e m wil l b e d e v i s e d at s o m e t i m e In the f u t u r e w h e n s t u d i e s tha t a r e t h o u g h t to b e m i s l e a d i n g can b e c o m p a r e d w i t h t h o s e tha t s e e m to be r e f l e c t i ng t h e t ru th . S o m e d a t a p e r t i n e n t to th i s q u e s t i o n a re a v a i l a b l e f rom a r e v i e w of 89 RCTs of t he m e & c a l t h e r a p y of d u o d e n a l u lce r [1]. C u r r e n t l y a s s i g n e d s co re s a r e i l l u s t r a t e d in Tab le 1.

T h e u s e f u l n e s s of a s c o n n g s y s t e m d e p e n d s on its f l ex lbd l ty . A n d l u s t r a - t ton of th i s is t he m e t h o d b y w h i c h p o i n t s a re a l l oca t ed to t he con t ro l of b i a s (Sec t ion 2.6). Tab le 2 i n d i c a t e s o u r p r e s e n t j u d g m e n t on th i s In s t u d i e s w i t h d e a t h as an e n d p o m t in a d i s e a s e h a v i n g a c c u r a t e p r e d l c h v e d a t a , b h n d m g of t he r a n d o m i z a t i o n ~s m o r e ~ m p o r t a n t t han b l i n d i n g of the

Tab

le 2

P

lan

for

Van

able

P

oin

t S

cori

ng

of

Eff

orts

to

Co

ntr

ol

Bxa

s b

y B

hn

dm

g

Rel

ativ

e in

flue

nce

of p

rogn

osti

c si

gns

and

soft

ness

of

end

po

mts

Max

imal

po

ints

for

sat

isfa

ctor

ily

cont

rolh

ng b

las

Ran

dom

l-

Ph

ys,

oan

s re

P

hy

slo

ans

re

zatl

on

Pat

ient

th

erap

y re

sult

s T

otal

po

ints

1 H

ard

end

po

mt

such

as

deat

h,

good

pro

gnos

tic

radi

ces

on

adm

issi

on

, no

t v

ery

su

scep

tibl

e to

oth

er t

her

apy

[5

] ~

2.

Har

d en

dp

om

t bu

t b

hn

dm

g o

f pa

tlen

ts a

nd p

hy

sici

ans

rela

tive

ly ~

mpo

rtan

t, e

speo

ally

re

wit

hd

raw

als

and

ano

llar

y

ther

apie

s [ 1

7].

3.

Sof

t en

dp

om

t, f

ew g

ood

prog

nost

lc r

adic

es o

n ad

mis

sio

n,

rela

tive

ly s

hort

th

erap

y so

tha

t re

sult

s no

t ve

ry i

nflu

enti

al,

but

do

ub

le

bh

nd

mg

Im

port

ant

[1]

4 S

oft

end

po

mt,

few

goo

d pr

ogno

stic

md~

ces,

pro

lon

ged

st

udy

so t

hat

emer

ging

new

tr

eatm

ents

or

d~sc

oura

gem

ent

may

inf

luen

ce m

vest

2gat

ors

[31.

16

4 4

6 30

10

8 8

4 30

4 10

12

4

30

4 10

8

8 30

"Ref

eren

ces

to i

llus

trat

ive

exam

ples

r,..o


phys ic ians or pa t ients In others where there is no way to predict ou tcome and the e n d p o m t is soft, b h n d l n g of the physic ian and pat ient ts most Impor tan t . If a d m i s s i o n s to a s t udy are d ragged out ~n t~me and the e n d p o m t s b h n d e d , b h n d l n g of the results warrants a h igher score These relative values are reflected in the a s s ignmen t s and scores ~n Table 2

I L L U S T R A T I O N OF M E T H O D

As of April 1980, more than 260 pub l i shed climcal trial papers have been evaluated. To illustrate the p roposed me thod , a clinical trlal of a s p m n in coronary heart d isease [171 has been chosen. The actual scormg sys tem, including indiv idual i tem scoring, for this pape r is s h o w n in Table 1. (Note The grad ing is done on what appea r s in the pubhca t lon ; it is poss ib le that more was done on the trial than ended up in the publ ica t ion )

O n the s tudy protocol (Form 2), points were lost for the failure to ma in ta in a relect log, lack of de tad on therapeut ic reg imen, inadequa te repor t ing of the placebo a rm of the trial (taste test not reported) , and failure to check on the efficacy of the b l ind ing The score on this part was 0.495 out of 0.600..

O n the ana lys i s sec t ion (Form 3), po in t s were lost for h a n d h n g of wi thdrawals (data not ana lyzed m more than one way), lack of statistical inference analysis (confidence intervals not presented) , and lack of b l inding of statistical or data m o m t o r m g groups . The score was 0.240 out of 0.300

On the end results section (Form 4), the m a x i m u m score of 0.10 was g iven

Thus the overall score of the trial was 0.835, a good score in our exper ience w~th this scoring mechan i sm. Howeve r , we do have some trials w~th scores in the 0.900s

D I S C U S S I O N

Any m e t hod of evaluat ing quah ty of p u b h s h e d scientific s tudies is b o u n d to s t imulate d iscuss ions of val idi ty The present me thod has evolved over a per iod of 3 years and will p robab ly be changed as exper ience is g a m e d m specific therapeut ic fields. Among the poss ib le defects are var ia t ion due to sublect lve ludgment s , inaccuracies due to incomple te repor t ing , inclusion of ~tems that migh t not be cons idered mer i to r ious by some experts , and exclusion of some i tems cons idered impor tan t by others.

The first defect has been hand led by the me thod of differential photoco- pies of the protocol and by mul t ip le s c o n n g fol lowed by conferences Incomple te repor t ing f requent ly occurs. Al though we have not rout inely wri t ten to au thors abou t the u n k n o w n i tems, we did so on 59 trials. Forty- one of the au thors r e sponded , and 58% of the miss ing ~tems were found actually to have been carried out. Howeve r , in in terpre t ing trial results, the reader has only the p u b h s h e d pape r on which to rely

Some migh t argue that ~t is not impor tan t to descr ibe the relected pat ients or to present data on size es t imates because they are e~ther so rout inely done or imposs ib le to do, but they are both cons idered c ruoa l for p roper in terpre ta t ion of the results Blinding of the p h y s i o a n s and mves t lga to rs as to ongo ing results is impor tan t for ethical reasons as well as for statistical

Assessing a Random,zed Control Trial 45

val id i ty . M a n y w o u l d c o n s i d e r b h n d l n g of the s t a t i s t i c i ans or the data

m o n i t o r i n g c o m m i t t e e (e g . , t r e a t m e n t s cou ld be r e fe r red to as A a n d B) to

be too d i f f icul t to a c c o m p l i s h a n d u n n e c e s s a r y as well .

Final ly , the o m i s s i o n s that m i g h t be c o n s i d e r e d i m p o r t a n t by s o m e are

u n d o u b t e d l y many. O t h e r r e v i e w s [18-23] h a v e h s t ed these a n d s o m e o t h e r

a t t r i b u t e s tha t the r e a d e r m i g h t w i s h to a d d to his or he r s c o n n g sy s t em .

For e x a m p l e , the m o n i t o r i n g of the data co l lec t ion s y s t e m to e n s u r e q u a h t y

is an i m p o r t a n t a spec t of any trial. The s y s t e m of d o i n g this is so rare ly

r e p o r t e d in any trial tha t w e h a v e r e v i e w e d that we h a v e not i n c l u d e d the

i t e m in ou r a s s e s s m e n t p r o c e d u r e H o w e v e r , ~t is an o m i s s i o n that n e e d s to

be a d d e d .

ACKNOWLEDGMENT This work was supported by National Institutes of Health, Nahonal Library of Medicine, grant 5 R01 LM03116-02

APPENDIX

Form 1 Basic Descriptive Material ID # Study #

Title

Reader

Journal or pubhcatlon

Peer reviewed Yes

Year of pubhcatlon

No Unknown

1 1 Blostat~st~c~an 1. Author 2 Credits 3 Neither 4 Unknown

1 2 Country 1 U S 2 U K 3 Scand 4 Other 5 Unknown

1 3 Center status ..... 1 Single center

2 Cooperative study <5 grps __ 3 Cooperatwe study :>5 grps

1 4 Source of hnanctal support (multiple items possible)

A. N.1H. or M R C 1 Yes 2 No

B -V A. 1. Yes

..... 2 No C. Drug Co

1 Yes 2 No

D Other 1 Yes 2. No

E None given 1 5 Source of patients

(multiple 1terns possible) A. Umversl ty

1 Yes _2 No

B Pubhc .... 1 Yes

2 No C Private

_ 1. Yes 2 No

D Chmc (no hosp 1 Yes 2 No

E Industry 1 Yes

. . . . 2 No F None given

1 6 Number in Controls ._. Tr grp 1__ Tr grp 2 ._ Tr grp. 3.

46 T h o m a s C C h a l m e r s et al

1.7 T y p e of trial 1 S i m p l e c o m p a r a t i v e 2 Res t r i c ted (b lock ing) 3 St ra t i f ied 4 C r o s s o v e r 5 Factorial 6 O t h e r 7 U n k n o w n

1 8 S ign i f i cance of f i n d i n g s A Major e n d p o m t s

1 + + Sta t i s t ica l ly s ign i f i can t ( t r e a t m en t )

2 + T rend ( t r e a t m en t ) 3 0 No d i f f e rence 4 - T r e n d (control) 5 - - Statzst~cally s l g m f l c a n t

(control)

6 S~gn,f lcant ~n a u t h o r ' s o p i n i o n bu t no s ta t is t ica l test w~th p r o b - a b l h t y s ta ted

B M i n o r e n d p o m t s 1 + + S4ahst lcal ly s lgnlf~cant

( t r ea tmen t ) 2 + Trend ( t r ea tmen t ) 3 0 No d i f f e rence 4 -- Trend (Control) 5 - - S tahs t l ca l ly s ign i f i can t

(control) 6 N o n e

1 9 S~de effects, s ta t is t ical h n d l n g 1 -t- + Sta t is t ica l ly s ign i f i can t 2 ~- Trend 3 0 No s ide effects 4 N .A

Form 2 The S t u d y Protocol ID # S t u d y #

2.1 Se lec t ion d e s c r i p t i o n 1 A d e q u a t e 2 Fair 3. I n a d e q u a t e

2 2 N u m b e r of p a t i e n t s s een a n d reject log

1. Yes 2 Part ial 3 No 4 U n k n o w n

2 3 W i t h d r a w a l s 1 List g i v e n 2 No w R h d r a w a l s 3 No hs t 4 U n k n o w n 5 > 15% w i t h d r a w a l s

to t l o n g - t e r m s tud , es a n d > 10% for s t u d i e s l a s t ing less t han 3 m o n t h s

2.4 T h e r a p e u h c r e g i m e n s d e f i m t l o n 1 A d e q u a t e 2 Fair 3 I n a d e q u a t e

2 5 A Cont ro l r e g i m e n (placebo)

a p p p e a r a n c e 1 S a m e 2 Di f fe ren t 3 U n s t a t e d 4 N A

B Cont ro l r e g i m e n (p lacebo) tas te

1 S a m e 2 Di f fe ren t 3. U n s t a t e d 4 N A

Reader

2 6 R a n d o m i z a t i o n b h n d m g " 1 Yes 2 Parhal 3 No 4. U n k n o w n

M e t h o d ot r a n d o m b h n d m g 1 Enve lope 2 P h a r m a c y 3 O t h e r 4 U n k n o w n 5 N A .

2 7 B h n d l n g ot p a t i e n t s " 1 Yes 2 No 3 U n k n o w n 4 N A

2 8 B h n d , n g ot p h y s i c i a n s re t he r apy" 1 Yes 2 Parhal 3 No t U n k n o w n 5 N A

2 q B h n d m g of p h y s i c i a n s a n d p a h e n t s re resul ts"

I Yes 2 Part la l 3 N o 4 U n k n o w n

2 I0 Pr lo r es t lma te of n u m b e r s ( e n d p o m t s selected, d l f f of chmc in teres t c~ and /3 es t lma ted)

] Yes 2 N o 3 U n k n o w n

"See Table 1 tor explanation o! multiple possible scoring

A s s e s s i n g a R a n d o m i z e d Cont ro l Trial 47

2.11 Tes tmg r a n d o m i z a t i o n 2.13 Tes tmg compl i ance I. Yes 1 Yes 2. Partial 2. Partial 3. No 3. No 4. U n k n o w n 4. U n k n o w n

2.12 Tes t ing b l i n d i n g . . . . . . 5. N.A. . . . . I Yes 2.14 Biological equ iva l en t

2. Partial ' _ . . . . . . 1. Yes 3. No 2 No 4. U n k n o w n 3 U n k n o w n 5 N A . 4 N.A

Form 3. Statist ical Ana lys i s ID # S t u d y # Reader

3 1 O n malor e n d p o m t s 1. If poss ib le , test s tat is t ic and o b s e r v e d p r o b a b l h t y value are s ta ted 2. If o b s e r v e d p robab i l i t y level g iven , bu t test s ta t is t ic value not s ta ted

. . . 3. If test s ta t is t ic but not o b s e r v e d p robab i l i t y value g iven ____ 4 If n e i t h e r test s ta t is t ic nor o b s e r v e d p robab i l i t y level g iven

3.2 Pos te r io r /3 e s t ima tes of o b s e r v e d d i f fe rence for nega t ive trials 1 Yes

. . . . . . . 2. M e n t i o n e d and necess i ty for m o r e pa t i en t s 3. No 4 N A

3 3 Statistical i n fe rence A. Conf idence h m l t s

1 Yes 2 No 3. N.A.

B. -Life-table or t~me- senes analys~s 1 Yes 2 S h o w n bu t mcor rec t 3. No 4 N.A.

C Reg re s s ion ana lys i s cor re la t ion 1 Yes 2 No 3 N A

3 4 A p p r o p r i a t e stat ist ical ana lys i s 1. Excellent 2. G o o d 3. Poor

_ _ _ 4 I n a d e q u a t e 3.5 H a n d l i n g of w i t h d r a w a l s

..... 1 A n a l y z e d several ways 2 Inc luded in or Igmal r a n d o m i z a t i o n

. . . . . 3 C o u n t e d as end resul t at t~me of w i t h d r a w a l . . . . 4 D i sca rded

5. C h a n g e d g r o u p s 6 U n k n o w n 7 No w i thd rawa l s /N A

3.6 Side eftects , s tat is t ical d i s cus s ion . . . . 1 A d e q u a t e

2. Fair ... 3. Poor

4. N A

48 T h o m a s C C h a l m e r s et al

3.7 P r o p e r r e t r o s p e c t i v e ana lv s l s 1 G o o d 2 P a m a l 3 N o n e

3 8 B h n d m g of s t a t l s t l o a n or ana lys t re resul t s 1 Yes 2 No 3 U n k n o w n 4 N A

3 9 Mul t i p l e looks c o n s i d e r e d 1 Yes "~ Fixed s a m p l e s ize, no look 3 No

F o r m 4 P r e s e n t a t i o n of Resul ts ID # . . . . . . . . . . . . . . . . . . . . . . . Stud}, # - R e a d e r . . . . . . . . . . . . .

4 1 D~tes of s t a r t i ng a n d s t o p p i n g a c c e s s m n 1 Yes 2 No

4 2 ReSults of p r e r a n d o m l z a t l o n A Data Analys~s

1 A d e q u a t e 2 Fair 3 I n a d e q u a t e

B P r o g n o s t l c a l l y f avo r ing 1 T r e a t m e n t 2 Con t ro l 3 Equivocal 4 U n k n o w n

4 3 F a b u l a t t o n of even t s e m p l o y e d a,s e n d p o m t for each t r e a t m e n t 1 P r e s e n t e d 2 Not p r e s e n t e d

4 4 T i m i n g ot e v e n t s 1 C o m p l e t e 2 Avai lable to r eade r 3 Ne i the r

REFERENCES

l S f l v e r m a n B, C h a l m e r s TC, S h a r e k EP et al R a n d o m i z e d cont ro l tr, als m g a s t r o e n t e r o l o g y wRh pa r t i cu la r a t t e n t i o n t o d u o d e n a l ulcer (Paper s u b m i t t e d

2 M e k h l l a n I-tS, Swl tz DM, M e l n y k CS et al C h m c a l fea tures a n d n o r m a l h t s t o r v of C r o h n ' s d i s ea s e Gastroenterology 77 898-906, 1979

3 Kar lowsM TR, C h a l m e r s TC, Frenkel ED et al Asco rb l c a o d for the c o m m o n cold A p r o p h y l a c t i c a n d t h e r a p e u t i c trial lAMA 231 1038-1042, 1975

4 S i m o n R A d a p t w e t r e a t m e n t a s s i g n m e n t m e t h o d s a n d chmca l t na l s Btometrws 33 743-749, 1977

5 Car l e ton RA, S a n d e r s CA a n d Burack WR l l e p a n n a d m l m s t r a t l o n a t te r acute m y o c a r d i a l in fa rc t ion N Engl f Med 263 1002-1005, 1960

6 S h a w LW a n d C h a l m e r s TC Ethics m c o o p e r a t w e chmca l trials Anu NY,4cad .qct 169 487-49.5, 1970


7. Chalmers TC and discussants How to turn off an experiment In Cooper, JD and Ley, HD eds Ethical Safeguards m Research on Humans. Washmgton, D C , ln terd lsc lphnary Communicat ions Associates, pp. 119-143, 1976

8. Frelman JA, Chalmers TC, Smith I--I et al The importance of beta, the type II error and sample size in the design and interpretat ion of the randomized control trial Survey of 71 "negat ive" trials N Engl J Med 299 690-694, 1978

9 Cornfield J The Umversl ty Group Diabetes Program a further statistical analysis of the mortal i ty f indings JAMA 217:1676-1687, 1971

10 Haynes RB, Sackett DL, Gibson ES et al Improvement of med~catlon comphance in uncontrol led hyper tens ion Lancet 1 1265-1268, 1976

11. Murphy ML, Hultgren HN, Detre K et al Treatment of chronic stable angina A pre l iminary report of survival data on the randomized Veterans Adminis t ra t ion cooperat ive s tudy N Engl J Med 297 621-627, 1977

12 Shor S The Universi ty Group Diabetes Program a statistlc~an looks at the mortal i ty results JAMA 217 1671, 1971

13 Kilo C, Miller JP and Wllhamson JR. The Achilles heel of the Umversl ty Group Diabetes Program ]AMA 243-450-457, 1980.

14 Urokmase Pulmonary Embolism Trial Study Group Urokmase-s t reptokmase embohsm trial: phase 2 results. A cooperative study JAMA 229 1606-1613, 1974.

15 Blackburn BA, Chalmers TC, Smith H et al . Evaluation of trials of ant , thrombotic agents m acute myocardial infarction Presented at the Vllth International Congress of Thrombosis and Haemostasis , London, England, July 1979

16 Chalmers TC, Matta RJ, Smith H et al Evidence favoring the use of anticoag- ulants in the hospital phase of acute myocardial infarction N Engl ] Med 297 1091-1096, 1977

17 Corona ry Drug Prolect Asp i r in in coronary heart d~sease ] Chronic Dzs 29:625-642, 1976

18. Peto R, Pike MC, Armitage l' et al Design and analysis of randomized cllmcal trials requir ing prolonged observat ion of each pat ient 1 Introduction and design Br J Cancer 34 585-612, 1976

19. l 'eto R, Pike MC, Armltage P et al Design and analysis of randomized chnlcal trials requir ing prolonged observat ion of each patient II Analysis and examples Br J Cancer 35 1-39, 1977

20 Mosteller F, Gilbert Jl' and Mcl'eek B: Reporting s tandards and research strategies for controlled trials agenda for the editor. Controlled Chmcal Trials 1 37-58, 1980

21 Byar DP, Simon RM, Fnedenwald WT et al. Randomized chnlcal trials per- spectives on some recent ideas N Engl ] Med 295 74-80, 1976

22 Ederer F, ed The randomized controlled chnlcal trial National Eye Institute Workshop for Opthalmolog~sts Am J Opthalmol 79 750-789, 1975

23 Remington RD Design and management of large mult i-center chmcal trials Ann NY Acad Sct 304 244-253, 1978

chalmers method for assessing the quality of a randomized c

Documents