completeness of reporting of studies of diagnostic accuracy in cardiology journals: a stard...
TRANSCRIPT
Completeness of reporting of studies of diagnostic accuracy in Cardiology
journals: a STARD assessment
Key words: STARD checklist, Cardiology, Diagnostic accuracy
Class 6: Ana Borges, Ana Torres, André Tojal, Célia Soares, Francisca Trigo, Francisco Coutinho, Joana
Ribeiro, José Fernandes, Pedro Vasconcelos, Raquel Ferreira, Sílvia Madureira
Supervisors: Altamiro da Costa Pereira, MD, PhD; Filipa Almeida, MD
Introduction to Medicine; Porto Faculty of Medicine 2008
IntroductionIntroduction Objectives Methods AcknowledgmentsResults
• Accurate diagnosis in Cardiology is crucial for clinical intervention and is increasingly important as the number of validated treatment for specific conditions increases.
• The use of systematic literature review to inform evidence based practice in diagnostics is rapidly expanding.
• Studies of the new diagnostic tests, and their development, are reported in scientific articles. (1)
(1) - Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, de Vet HCW. Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative. Ann Intern Med. 2003; 138:40-44
Conclusion
IntroductionIntroduction Objectives Methods AcknowledgmentsResults
•Current cardiology practice relies on diagnostic tests using sophisticated technologies that are constantly evolving. Diagnostic accuracy studies are required to validate new diagnostic tests before they are introduced into clinical practice.
• These studies typically report sensitivity and specificity, likelihood ratios, diagnostic odds ratio, or area under a receiver operating characteristics (ROC) curve as measures of diagnostic performance. This information enables a clinician to make judgments regarding the potential utility of new tests. (2)
(2) Deeks J. Systematic reviews in health care: systematic reviews of diagnostic tests, BMJ 2001; 44:639-650
Conclusion
IntroductionIntroduction Objectives Methods AcknowledgmentsResults
•A rigorous evaluation process of diagnostic tests before introduction into clinical practice could not only reduce the number of unwanted clinical consequences related to misleading estimates of test accuracy, but also limit health care costs by preventing unnecessary testing.
• Several systematic reviews (1995, Reid et al) (3) have emphasized the poor quality of reporting in diagnostic accuracy studies. This poor reporting hampers an adequate judgment of both the internal and the external validity of a study.
(3) Scholten RJ, Deville WL, Opstelten W, Bijl D, van der Plas CG, Bouter LM. The accuracy of physical diagnostic tests for assessing meniscal lesions of the knee: a meta-analysis. J Fam Pract. 2001; 50:938–944.
Conclusion
IntroductionIntroduction Objectives Methods AcknowledgmentsResults
•In 19991999, in Rome, the Cochrane Diagnostic and Screening Test Methods Working group discussed the low methodological quality and sub-standard reporting of diagnostic test evaluations, that resulted in the intent of developing a checklist of items that should be included in those reports.
•The STARD steering committee started with an extensive search to identify publications on the conduct and reporting of diagnostic studies up to July 20002000 .(4)
•In January 20032003, after the discussion based on the search done, we have the first official version of the STARD, which was published simultaneously in eight medical journals.
History of the STARDHistory of the STARD
(4) Bossuyt PM, Reitsma JB, Bruns DE, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 2003
Conclusion
IntroductionIntroduction Objectives Methods AcknowledgmentsResults
STARD CHECKLISTSTARD CHECKLIST
Objective:Objective:•The aim of the Standards for Reporting of Diagnostic Accuracy (STARD) initiative is to improve the quality of reporting of studies on diagnostic accuracy.
How:How:• Using a checklist with 25-item and a flow diagram, which provides information about the method of patient recruitment, the order of test execution and the numbers of patients undergoing the test under evaluation, the reference standard or both. (5)
Advantage:Advantage:•Complete and accurate reporting allows the reader to detect the potential for bias in the study and to evaluate the general ability of the results.
(5) Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA. 1995; 274:645–651.
Conclusion
Objectives Methods AcknowledgmentsResults
• Cardiology is the area with the most published articles;
•This is the first study that analyses how reports on diagnostic accuracy in cardiology
comply with the STARD checklist;
IntroductionIntroduction
Importance of our analysisImportance of our analysis
Conclusion
Introduction ObjectivesObjectives Methods AcknowledgmentsResults
To assess the quality of reporting of studies on diagnostic tests in cardiology journals, based on the STARD checklist.
Main objectiveMain objective
Secondary objectivesSecondary objectives
•To analyse the quality of diagnostic tests studies in the course of the years;
•To compare the differences between the articles published before and after the creation of the STARD checklist;
•To evaluate the influence of the journals’ impact factor on the quality of the articles which report diagnostic tests in cardiology journals.
Conclusion
Articles’ searchArticles’ search
Articles’ selection
Articles’ evaluation
Statistical analyze
Introduction Objectives MethodsMethods AcknowledgmentsResults
Standard query to find articles that report studies of diagnostic accuracy (in PubMed)
(((((((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields)) OR (((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All fields]) OR "roc and"[All fields]) OR "roc area"[All fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields])) OR "likelihood ratio"[All Fields])
Devillé W. Conducting systematic reviews of diagnostic studies; didactic guidelines. Medical research methodology. 2002
Conclusion
Introduction Objectives MethodsMethods AcknowledgmentsResults
Search the journals
Articles’ searchArticles’ search
Articles’ selection
Articles’ evaluation
Statistical analyze
Included all journals classified as “cardiac and cardiovascular systems” in the ISI web of knowledge except those who are not found on pubmed.
Included all 71 journals found on Pubmed
Conclusion
Introduction Objectives MethodsMethods AcknowledgmentsResults
Final query (after addition of terms to specify the search to the journals)
Articles’ searchArticles’ search
Articles’ selection
Articles’ evaluation
Statistical analyze
(((((((("sensitivity and specificity"[All Fields] OR "sensitivity and specificity/standards"[All Fields]) OR "specificity"[All Fields]) OR "screening"[All Fields]) OR "false positive"[All Fields]) OR "false negative"[All Fields]) OR "accuracy"[All Fields]) OR (((("predictive value"[All Fields] OR "predictive value of tests"[All Fields]) OR "predictive value of tests/standards"[All Fields]) OR "predictive values"[All Fields]) OR "predictive values of tests"[All Fields])) OR (("reference value"[All Fields] OR "reference values"[All Fields]) OR "reference values/standards"[All Fields]) OR ((((((((((("roc"[All Fields] OR "roc analyses"[All Fields]) OR "roc analysis"[All Fields]) OR "roc and"[All Fields]) OR "roc area"[All Fields]) OR "roc auc"[All Fields]) OR "roc characteristics"[All Fields]) OR "roc curve"[All Fields]) OR "roc curve method"[All Fields]) OR "roc curves"[All Fields]) OR "roc estimated"[All Fields]) OR "roc evaluation"[All Fields]) OR ("likelihood ratio"[All Fields])) AND (Circulation [Journal] OR Circulation Research [Journal] OR Journal Of The American College Of Cardiology [Journal] OR European Heart Journal [Journal] OR Cardiovascular Research [Journal] OR Journal Of Molecular And Cellular Cardiology [Journal] OR American Journal Of Physiology-Heart And Circulatory Physiology [Journal] OR Journal Of Thoracic And Cardiovascular Surgery [Journal] OR Trends In Cardiovascular Medicine [Journal] OR American Journal Of Cardiology [Journal] OR Journal Of Heart And Lung Transplantation [Journal] OR Chest [Journal] OR American Heart Journal [Journal] OR Journal Of Cardiovascular Pharmacology [Journal] OR Progress In Cardiovascular Diseases [Journal] OR Journal Of Cardiovascular Magnetic Resonance [Journal] OR Heart [Journal] OR Journal Of Nuclear Cardiology [Journal] OR Annals Of Thoracic Surgery [Journal] OR Journal Of The American Society Of Echocardiography [Journal] OR Current Opinion In Cardiology [Journal] OR Current Problems In Cardiology [Journal] OR Basic Research In Cardiology [Journal] OR Catheterization And Cardiovascular Interventions [Journal] OR Respiratory Medicine [Journal] OR Canadian Journal Of Cardiology [Journal] OR European Journal Of Cardio-Thoracic Surgery [Journal] OR European Journal Of Heart Failure [Journal] OR Journal Of Heart Valve Disease [Journal] OR Clinical Cardiology [Journal] OR Cardiovascular And Interventional Radiology [Journal] OR International Journal Of Cardiology [Journal] OR Herz [Journal] OR Cardiovascular Drugs And Therapy [Journal] OR Journal Of Cardiothoracic And Vascular Anesthesia [Journal] OR Cardiovascular Drug Reviews [Journal] OR Zeitschrift Fur Kardiologie [Journal] OR Pediatric Cardiology [Journal] OR Cardiovascular Surgery [Journal] OR Thoracic And Cardiovascular Surgeon [Journal] OR Journal Of Interventional Cardiac Electrophysiology [Journal] OR Revista Espanola De Cardiologia [Journal] OR Cardiology [Journal] OR Heart & Lung [Journal] OR Cardiology In The Young [Journal] OR Heart And Vessels [Journal] OR Journal Of Electrocardiology [Journal] OR Journal Of Cardiovascular Surgery [Journal] OR International Journal Of Cardiac Imaging [Journal] OR Texas Heart Institute Journal [Journal] OR Scandinavian Cardiovascular Journal [Journal] OR Acta Cardiologica [Journal] OR Europace [Journal] OR Archives Des Maladies Du Coeur Et Des Vaisseaux [Journal] OR Cardiovascular Pathology [Journal] OR Japanese Heart Journal [Journal] OR Journal Of Cardiac Surgery [Journal] OR Kardiologiya [Journal])
Conclusion
Introduction Objectives MethodsMethods AcknowledgmentsResults
Articles’ searchArticles’ search
Articles’ selection
Articles’ evaluation
Statistical analyze
• Random sample of 100 articles stratified by years
• Sample of 10 articles per year
• Years included:- Until 1998, 1999, 2000, 2001, 2002, 2003, 2004,
2005, 2006, 2007.
From all articles retrieved by the query
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Reading and analyzing the abstractArticles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
InclusionInclusion
- to report a study of diagnostic accuracy - the diagnostic test(s) is (are) used in cardiology- the full text is available
The abstract is read and analyzed with the inclusion/exclusion criteria
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Reading and analyzing the abstractArticles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
ExclusionExclusion
- Is a clinical trial, a letter, an editorial, a practice guideline, a meta-analysis or a review
-The article could not be found in the Internet, the facilities of the School of Medicine or the local libraries
-The article is written in languages other than English, Portuguese, French or Spanish
- The article found is not complete
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
RevisionArticles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
If excluded:- the reason of exclusion is registered
If doubted:- the abstract is reanalyzed by other selector
When disagreement or doubt occur, the article will be discussed by both selectors. When no decision could be achieved, a third selector will be consulted
When this situation occurs, PMID of each opened article will be registered
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Final samples
Articles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
2007 → 139 articles
2006 → 136 articles
2005 → 125 articles
2004 → 132 articles
2003 → 125 articles
2002 → 147 articles
2001 → 112 articles
2000 → 172 articles
1999 → 112 articles
Until 1998 → 1585 articles
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Final samples
Articles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
TOTAL → 2785 articles
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Articles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
Flowchart
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Articles’ search
Articles’ selectionArticles’ selection
Articles’ evaluation
Statistical analyze
Gantt Chart
Conclusion
Articles’ search
Articles’ selection
Articles’ evaluationArticles’ evaluation
Statistical analyze
Objectives ResultsMethodsMethods
Selection and Topic Item Describe
TITLE/ABSTRACT/KEYWORDS
1The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity and specificity') ImpliedImplied
INTRODUCTION2
The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups The main objectives are presentThe main objectives are present
METHODS
Participants3
The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected It has inclusion and exclusion criteriaIt has inclusion and exclusion criteria
4
Participant recruitment: was this based on presenting symptoms, results from previous tests, or
the fact that the participants had received the index test(s) or the reference standard? If it has two of the threeIf it has two of the three
5
Participant sampling: was this a consecutive series of patients defined by selection criteria in (3)
and (4)? If not specify how patients were further selected. If 3 and 4 have been markedIf 3 and 4 have been marked
6
Data collection: were the participants identified and data collected before the index test(s) and reference standards were performed (prospective study) or after (retrospective study)? If it says before ou after / impliedIf it says before ou after / implied
Reference standard 7 The reference standard and its rationale Nor that it doesn’t have rational analysis Nor that it doesn’t have rational analysis
Test methods8
Technical specification of material and methods involved including how and when measurements were taken, and/or cite references for index test(s) and reference standardNor that it doesn’t specify how and whereNor that it doesn’t specify how and where
9
Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference standard If it has the two topicsIf it has the two topics
10
The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference standard If it has at least two of the threeIf it has at least two of the three
11Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other test(s) and describe any information available to them If it has the first topic or bothIf it has the first topic or both
Statistical methods12
Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals) Only if it has bothOnly if it has both
13 Methods for calculating test reproducibility, if done If yes / if it has been doneIf yes / if it has been done
STARD CHECKLIST
Each item values 1 point Several criteria was defined for the STAR application
Articles’ search
Articles’ selection
Articles’ evaluationArticles’ evaluation
Statistical analyze
Objectives ResultsMethodsMethodsSection and topic Item Describe
RESULTS
Participants14
When study was done, including beginning and ending dates of recruitment If it has everything or the duration timeIf it has everything or the duration time
15
Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s),
comorbidity, current treatment(s), recruitment center) If it presents at least one of eachf it presents at least one of each
16
How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the reference standard; describe why participants failed to receive either test (a flow diagram is strongly recommended) If it has everything, even it doesn’t have the If it has everything, even it doesn’t have the diagramdiagram
Reference standard17
Time interval and any treatment administered between index and reference standard If it has everythingIf it has everything
18
Distribution of severity of disease (define criteria) in those with the target condition; describe other diagnoses in participants without the target condition If it has the first topic or If it has the first topic or bothboth
Test results
19
A cross tabulation of the results of the index test(s) by the results of the reference standard; for
continuous results, the distribution of the test results by the results of the reference standard If it has the first topic or both, in case the second is verifiedit has the first topic or both, in case the second is verified
20
Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result and how they were handled If it has the itemIf it has the item
21 Adverse events of index test(s) and reference standard If it has bothIf it has both
Estimation22
Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals) If it has two of the threeIf it has two of the three
23
Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done If it has everythingIf it has everything
24 Measures of test reproducibility, if done If it has the item and it’s doneIf it has the item and it’s done
DISCUSSION 25 The clinical applicability of the study findings If it has the itemIf it has the item
STARD CHECKLIST
Each item values 1 point Several criteria was defined for the STAR application
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Evaluators – Distribution for year
Article’s components
- Title,
- Abstract,
- Keywords
- Introduction
- Methods
- Results
- Discussion
One evaluator for each year
Every article´s components are seen for the
same evaluator
10 articles for each evaluator
Articles’ search
Articles’ selection
Articles’ evaluationArticles’ evaluation
Statistical analyze
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Articles’ search
Articles’ selection
Articles’ evaluationArticles’ evaluation
Statistical analyze
• Final score of each article
• Average score of the articles for each group (year of publication, impact factor)
• Interobserver variability
• The status of each STARD item in each year
Points to analyse:
Conclusion
Introduction Objectives AcknowledgmentsResultsMethodsMethods
Articles’ search
Articles’ selection
Articles’ evaluation
Statistical analyzeStatistical analyze
Statistical Analyse
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
Final Selection States
From Articles’ SelectionFrom Articles’ Selection
From Articles’ Evaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
Reasons for the Exclusion
From Articles’ From Articles’
SelectionSelection
From Articles’
Evaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
Final Classification of the Items
From Articles’
Selection
From Articles’ From Articles’
EvaluationEvaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
ICC for each Item of the List
From Articles’ Selection
From Articles’ EvaluationFrom Articles’ Evaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
Evolution of the Articles’ Quality
From Articles’ Selection
From Articles’ EvaluationFrom Articles’ Evaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
The Impact Factor of the Journal
From Articles’ Selection
From Articles’ EvaluationFrom Articles’ Evaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods Conclusion
. . SSome items of evaluation are rigorously asked for in a section, but in the articles,
several times, they appear in another one;
. . Because some STARD items are potentially affected by the subjective judgment of
the reviewers due to differences in interpretation of the STARD, disagreement arises;
About the About the STARD CHECKLIST – STARD CHECKLIST – Evaluators’ point of viewEvaluators’ point of view
Introduction Objectives AcknowledgmentsResultsResultsMethods
Relation Between the Articles’ Score and the Impact Factor of the Journal
From Articles’ Selection
From Articles’ EvaluationFrom Articles’ Evaluation
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods Conclusion
. . The data of the participants (age, sex, symptoms,…) and their dates of
recruitment should be included in the methods;
. . The list has 25 items, but it’s possible each one can have more then one aspect
to evaluate, which creates confusion in the classification to attribute.
About the About the STARD CHECKLIST – STARD CHECKLIST – Evaluators’ point of view Evaluators’ point of view
Introduction Objectives AcknowledgmentsResultsResultsMethods
Expected Results vs Obtained Results
Expected Expected •As we can see in other medical areas where this project had been done, such as
obstetrics and gynecology and others, we expect that this checklist had had impact
on the articles written after it as well as that they had improve its qualities due to
STARD Checklist.
Conclusion
Introduction Objectives AcknowledgmentsResultsResultsMethods
Expected Results vs Obtained Results
ObtainedObtained. . The differences found in the articles were nothing to do with variances observed between groups, we mean the year of publish is not related with the improvement of the articles (p<0,005). . . Therefore we assume that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy and so we suppose that the improve in this kind of articles is due to the progress of knowledge along the years and not related with the creation of this checklist, as shown in the graphic.
. . We come to conclusion that there is no correlation between them, since we reach a p value equal to 0,210, that is not significant and by the way that we can declare that the year of publish or the improve verify in the articles does not interfere with the impact factor of the journals.
Conclusion
Introduction Objectives AcknowledgmentsResultsMethods
Conclusion
Our study shows that STARD doesn’t affect the quality of the articles reporting tests of diagnostic accuracy .
Conclusion