AABB practice guidelines for the use of plasma transfusion: overview of methodology Benjamin Djulbegovic, M.D.,Ph.D. Professor of Medicine and Oncology H. Lee Moffitt Cancer Center University of South Florida A case 65 y/o man is undergoing elective abdominal surgery. So far, he lost about 300 mls red blood cells and a surgeon calls you to ask you whether he should give him F F P. What is going to be your advice? You recall the last week Grand Rounds in which a speaker cited a RCT (Hedstard et al) in which prophylactic use of liquid stored plasma was no better than dextran infusion in patients undergoing similar surgeries. However, you also recall that in another (retrospective study)by Borgman et al patients who received F F P had dramatically improved survival. Your colleague remained you that the Borgmans study was done in a combat hospital setting but that Gajic et al in another retrospective study found increased mortality in patients who received plasma transfusions. Confusion? Should you recommend plasma transfusion for this patient? What does the patient and his physician need? Evidence-based principles Evidence: selective citation vs. totality of evidence Need for systematic reviews of the totality of research evidence Simultaneous instead of separate presentations of evidence (benefits and harms) Assessing the quality of knowledge Critical appraisal of the quality of clinical research is central to informed decision in health care quantity, quality (internal validity), consistency Benefits and harms Relative effect measures vs. absolute effect measures (NNT) Minimizing framing effect Probability vs. certainty Patient-oriented evidence vs. disease-oriented evidence Evidence on survival, DFS, QOL is more important than evidence on tumor response, markers etc Help with decisions Effective health-care recommendations vs. preference-sensitive health-care recommendations (decisions) Mortality Rx1=10% Mortality Rx2=5% RRR=(10-5)/10= 50% ARD=5% NNT=100/5%= 20 Mortality Rx1=1% Mortality Rx2=0.5% RRR=(1-0.5%)/1= 50% ARD=0.5% NNT=100/0.5%=200 US systems delivers only 55% of safe and effective recommended care N Engl J Med 2003;348:2635 Improving decisions: practice guidelines Systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances Institute of Medicine, 1990 Importance of grading quality of evidence and strength of recommendations Patients and physicians using clinical practice and other recommendations need to know how much confidence they can place in the recommendations Clinical guidelines are only good as good as the evidence and judgments that are based on Systematic and explicit methods of making judgments can reduce errors and improve communication BMJ 2004;328:1490-4 Key EBM principles: Evidence and Decisions Hierarchy of evidence: not all evidence is created equal We are more confident about decisions based on evidence that offers greater protection against bias and random error Evidence is assessed on the continuum scale of credibility Evidence is necessary but is not sufficient for decision-making decision-making is a categorical exercise we decide or not Many systems to develop guidelines. Who is confused? Evidence Recommendation II-2B C+ 1 StrongStrongly recommended Organization USPSTF ACCP GCPS Number of Systems to Rate the Quality of Individual Studies total =106. A critical appraisal of systems for grading the quality of evidence and strength of recommendations All existing systems have important shortcomings and none is considered useful Therefore, their continuing use will not help reduce errors in making of recommendations, will not help improve communication between guidelines developers and guidelines users, and thus will not help people make well-informed decisions. Grade Group, BioMed Central, 2004 II Guidelines Development Process The principles From Evidence to Decision-making (recommendations) Continuum from Study Quality Through Strength of Evidence to Guideline Development Strength of Evidence Clinical Practice Guidelines Quality of Studies Quality Magnitude Consistency Categorical recommendations (statements) Central role of evidence I II Broad Categories -Good vs.High -Fair vs. Moderate -Poor vs. Low Very low * FOR * AGAINST * Cant recommend The need for research synthesis Health care decision makers need to access research evidence to make informed decisions on diagnosis, treatment and health care management for both individual patients and populations. There are few important questions in health care which can be informed by consulting the result of a single empirical study. Evidence: selective citation vs. totality of evidence Systematic reviews of the totality research evidence represents a scientific foundation for development of clinical practice guidelines and health technology assessments. Guidelines Logic The derivation of pathways and algorithms that conform to the clinical decision-making process Courtesy of Dr. R. Winn Practice guidelines: Linking guidelines/HTA to systematic reviews Bleeding Each of question in guideline can be linked to systematic review What are the effects of FFP? GRADE criteria (BMJ 2004; 328: 1490) Grade of Recommendation, Assessment, Development and Evaluation:Medical patient Surgical patient No bleeding BleedingNo bleeding What are the effects of FFP? What are the effects of FFP? What are the effects of FFP? Guidelines developement proccess After GRADE,2004 Define the question: What intervention is considered for which outcomes Are outcomes critical/important? Stop Assess the quality of evidence for each important outcome (high, intermediate, low, very low) No Yes Assess the quality of evidence for across all important outcome (high, intermediate, low, very low) Assess the balance between benefits and harms (net benefits, trade offs, uncertain net benefit, no net benefit) Formulate recommendations (do it, probably do it, toss up, probably dont do it, dont do it) Factors affecting guidelines recommendations Sponsor Academic/professional societies vs. industry vs. insurance Conflict of interest Group composition Composition of panel influences recommendations Members of a specialty are more likely to advocate techniques that involve their specialty Balanced groups Select the appropriate group leader Necessary technical skills including information retrieval, systematic reviewing, health economics, group facilitation, project management, writing and editing Include or have access to content experts Courtesy of Dr. Schnemann GRADE method: a brief overview Quality assessment Making recommendations Possible Explanations for the Results of a Study Represents the correct result (Truth) Represents Random Variation (Chance) Represents Systematic Error (Bias) The results should be accepted when Effects > bias + random error There is only one way to get the research results right but literally thousands way to get the results wrong: Most Published Research Findings Are False Ioannidis PLoS Med August; 2(8): e124 This applies both to RCTs and observational studies Methodological quality 1.Attrition bias13.Misclassification bias 2.Berkson bias14.Missing clinical data bias 3.Confounding bias 15.Non blind diagnosis bias 4.Centripetal bias16.Non blind test interpretation bias 5.Diagnostic access bias17.Non-respondent bias 6.Diagnostic purity bias18.Partial verification bias 7.Diagnostic suspicion bias19.Performance bias 8.Diagnostic vogue bias20.Prevalence-incidence bias 9.Exposure suspicion bias21.Recall bias 10.Family information bias22.Referral filter bias 11.Interviewer bias23.Selection bias 12.Membership bias 24.Unmasking bias Sackett, JCD 1979 Published Conceived Performed Submitted Cited Sources of bias in the production and dissemination of evidence Courtesy of Dr. Mike Clarke Formulation of guidelines: main principles Separate evidence from decision-making Quality of evidence indicates the extent to which one can be confident that an estimate of effect is correct represented on a continuum scale of credibility Strength of recommendations indicates the extent to which one can be confident that adherence to a recommendation will do more good than harm Represent decision-making about choice and is categorical exercise (we recommend or do not) Important Domains and Elements for Systems to Rate Grade the Strength of Evidence Quality: the aggregate of quality ratings for individual studies, predicated on the extent to which bias was minimized. Quantity: magnitude of effect, numbers of studies, and sample size or power. Consistency: for any given topic, the extent to which similar findings are reported using similar and different study designs. Why Grade Recommendations? Assumed link between high quality evidence and truth strong recommendations strong methods Large effects precise estimates few down sides of therapy weak recommendations weak methods imprecise estimate small effect substantial down sides Adapted from G.Guyatt What are we grading? two components quality of evidence extent to which confidence in estimate of effect adequate to support decision high, moderate, low, very low strength of recommendation strong and weak Courtesy of Dr. Guyatt 1. Assessing the quality GRADE: categories of quality High: Considerable confidence in the estimate of effect. Moderate: Further research likely to have important impact on confidence in estimate, may change estimate. Low: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low: Any estimate of effect is very uncertain. Courtesy of Dr. Guyatt Quality assessment criteria BMJ 2004;328:1490-4 Factors in deciding on quality of evidence: GRADE meth od Factors that might decrease quality of evidence Study limitations (in detail of design and execution) (6 major factors) Inadequacy of allocation concealment; lack of blinding, large drop- outs, failure to perform ITT, failure to report outcomes, stopping early for benefits Inconsistency of results Variability or heterogeneity in results due true differences in treatment effect (due to P-I-C-O) Statistical: large I 2 (e.g.>50%); clinical: (PICO) Indirectness of evidence (2 types) Lack of head-to-head comparisons differences in treatment effect (due to P-I-C-O) Imprecision A few events (< ?), small studies (N 5 (< 0.2) Plausible confounding, which would reduce a demonstrated effect is accounted for without affecting treatment effect Dose-response gradient Factors in deciding on quality of evidence: GRADE method BMJ 2008 Copyright 2008 BMJ Publishing Group Ltd. Guyatt, G. H et al. BMJ 2008;336: Hierarchy of outcomes: patient-oriented outcomes vs. disease-oriented outcomes Ex. according to importance to patients to assess effect of phosphate lowering drugs in patients with renal failure and hyperphosphataemia GRADE recommends that the guideline developers consider the quality of evidence across outcomes as that associated with the critical outcome with the lowest quality evidence. Rating quality of evidence across outcomes BMJ 2008 2. Making recommendations Elements of decisions making in medicine Reliable research evidence: good internal validity Patient circumstances: settings, generalizibility Individual patient preferences, values, rights: patient-oriented outcomes Synthesis of the totality of research evidence: systematic reviews/meta- analysis Optimizing rational decision- making: clinical experience+ Decision-aids Guidelines Decision-modeling No evidence is perfect and no recommendation can be perfect What are acceptable evidentiary standards for the guidelines developers? How far are we willing to generalize from imperfect evidence? Decision making in average patients: Practice guidelines Evidence vs. consensus-based guidelines in individual patients: Applying evidence at bedside E.g. decision-analytic models/nomograms to individualize treatment decisions Populations vs. Individuals Our answers from the clinical trial present a group reaction. They show that one group fared better than another, that given a certain treatment, patients, on the average, get better more frequently or rapidly, or both. We cannot necessarily, perhaps very rarely, pass from that to stating exactly what effect the treatment will have on a particular patient. But there is, surely, no way and no method of deciding that Hill AB. The clinical trial. New Eng J Med 1952;247: , Clamers I, 2006. What is a good (rational) recommendation ( decision-quality)? Improving outcomes rational decision-making aims to maximizes the value of consequences Decision quality Informed decision consistent with decision-makers values consistent/coherent with process of decision- making Rationality of choice is a matter of the process of choosing, NOT of what is chosen a good decision can result in bad outcomes a bad decision can result in good outcomes After Hastie & Dawes, 2001 The importance of context: conclusions vs. decisions Quality of evidence (=conclusions) The extent of confidence that an estimate of effect is correct i.e. representing the truth Important for systematic reviews Making recommendations The extent to which confidence in an estimate of the effect is adequate to support recommendations Important for guidelines panels NB as long as there is judgment that benefits>>>harms, recommendation can be strong even if the quality of evidence is low or very low assumes that the error making a strong recommendation will be regretted less than the error making a weak recommendation Quality of evidence Uncertainty about the balance between desirable and undesirable effects Uncertainty or variability in values and preferences Practice setting/uncertainty about whether the intervention represents a wise use of resources Factors that effect strength of recommendations BMJ 2008 Balance between benefits and harm: GRADE system Net benefits: The intervention does more good than harm. Trade-offs: There are important trade-offs between the benefits and harms. Uncertain net benefits: It is not clear whether the intervention does more good than harm. Not net benefits: The intervention does not do more good than harm. BMJ 2004;328:1490-4 GRADE categories for Recommendations Do it or Dont do it Indicating judgment that most well informed people will make the same choice Probably do it or Probably dont do it Indicating judgment that a majority of well informed people will make the same choice, but a substantial minority will not GRADE Group, Courtesy of Dr. A. Oxman BMJ 2004;328:1490-4 Wording strong and weak discomfort with weak clinicians will ignore alternative wording: discretionary strong we recommend weak we suggest never we recommend (or suggest) you consider Courtesy of Dr. Guyatt Copyright 2008 BMJ Publishing Group Ltd. Guyatt, G. H et al. BMJ 2008;336: Fig 2 Representations of quality of evidence and strength of recommendations Effective health-care recommendations Effective health-care (strong recommendations) when benefits >>>harms: candidate for quality criteria Preference-sensitive recommendations Judgments about benefits/harm ratio uncertain, depend on patient values and preferences May be based on quantitative or qualitative judgments about (explicitly) summarized evidence decision-making process must be transparent and explicit with clarity regarding the critical criteria that informed recommendations; based on shared deliberation & must include appeal process Accountability for reasonableness which may help legitimize specific choices that may favor one set of stakeholders over others GRADE: stressing explicitness and transparency and less reproducibility After OConnors, 2003; Teutsch et al, 2005 ; Daniels at al, 1997 Making decisions/recommendations: some principles Re-cap Elements of decisions making in medicine Reliable research evidence: good internal validity Patient circumstances: settings, generalizibility Individual patient preferences, values, rights: patient-oriented outcomes Synthesis of the totality of research evidence: systematic reviews/meta- analysis Optimizing rational decision- making: clinical experience+ Decision-aids Guidelines Decision-modeling Challenge 2 Evidence is necessary but not sufficient for optimal decision-making Making categorical recommendations (considered judgments) Qualitative exercise Occasionally is supplemented with quantitative (decision-analytic) modeling Evidence & Decision making No evidence is perfect and no recommendation can be perfect What are acceptable evidentiary standards for the guidelines developers? How far are we willing to generalize from imperfect evidence? Guidelines recommendations and rational decision-making Is the one that maximizes the value of consequences Based on the possible consequences (outcomes) and values associated with each consequences of a choice When these consequences are uncertain, their likelihood is evaluated according to the rules of probability theory Rationality of choice is a matter of the process of choosing, NOT of what is chosen a good decision can result in bad outcomes a bad decision can result in good outcomes After Hastie & Dawes, 2001 What is a good (rational) recommendation ( decision-quality)? Decision quality Informed decision consistent with decision-makers values Improving outcomes rational decision-making aims to maximizes the value of consequences consistent/coherent with process of decision- making Rationality of choice is a matter of the process of choosing, NOT of what is chosen a good decision can result in bad outcomes a bad decision can result in good outcomes After Hastie & Dawes, 2001 Has a mistake been made? We can always make a mistake Recommend ineffective treatments, or Fail to recommend effective treatments Sense of loss, or regret difference between the utility of outcomes of the action taken and the utility of outcome of another action we should have taken, in retrospect Qualitative Decision Making (Effective Decision-Making) (Driven by Evidence) Goals/Objectives: cure, increased life expectancy, quality of life Regret associated with wrong decision (Mostly False) Decisions Evidence (Continuum of Credibility) Quantitative Medical Decision Making (Preference-Based Medical Decision Making) (Categorical Exercise) (Threshold) Decision making in average patients: Practice guidelines Evidence vs. consensus-based guidelines in individual patients: Applying evidence at bedside E.g. decision-analytic models/nomograms to individualize treatment decisions Rational decision-making Benefits >>> harms for the most important outcomes that we care about Making decisions for individuals The elicitation of preferences is a cornerstone of medical decision making. The protection of preferences in clinical care and research is prescribed by the courts Preference-sensitive health-care recommendations (decisions) Judgments about benefits/harm ratio uncertain, depend on patient values and preferences May require formal elicitation of preferences and quantitative modeling (explicitly structuring problems) (decision-analysis) Methods for elicitation of preferences in its infancy Mazur D. MDM 2006; O'Connor A,BMJ 2003 Effective health-care recommendations When interventions benefits>>>harms May be based on quantitative or qualitative judgments about (explicitly) summarized evidence decision-making process must be transparent and explicit with clarity regarding the critical criteria that informed recommendations; based on shared deliberation & must include appeal process Accountability for reasonableness which may help legitimize specific choices that may favor one set of stakeholders over others Making decisions for populations After OConnors, 2003; Teutsch et al, 2005 ; Daniels at al, 1997 Populations vs. Individuals Our answers from the clinical trial present a group reaction. They show that one group fared better than another, that given a certain treatment, patients, on the average, get better more frequently or rapidly, or both. We cannot necessarily, perhaps very rarely, pass from that to stating exactly what effect the treatment will have on a particular patient. But there is, surely, no way and no method of deciding that Hill AB. The clinical trial. New Eng J Med 1952;247: , Clamers I, 2006. Evidentiary Standards for Clinical Decision Making Intervention - Benefits - Harms - Costs Goals - Prevention -Treatment - Cure - Prolongation of Survival - Improvement of quality of life Acceptable likelihood of error/bias Very Low (Societal decision) Acceptable Standard of Evidence Very High - Low to moderate (depending on the patients preference and resources available) - High to - Moderate Proposed Evidentiary Standards for Clinical Recommendations as a Function of Treatment Goals and Acceptable Regret (Adapted from Djulbegovic B, JCO, 2005) Goals of Treatment Uncertainty about Rx effectsBenefit-Harm RatioRegretEvidentiary StandardsExamples Prevention (healthy individuals) HighImportant trade-offs between the benefits and harms HighHighest standard of experimental evidence Tamoxifen to prevent breast cancer CureLowBelief that intervention does more good than harm LowMay accept lower level of evidence Surgery of isolated liver metastasis of colorectal CA Increased years of survival LowBelief that intervention does more good than harm LowMay accept lower level of evidence Imatinib in chronic myeloid leukemia; Increased days to months of survival HighImportant trade-offs between the benefits and harms, or it is not clear HighHighest standard of experimental evidence Chemotherapy in metastatic lung cancer Palliation (improveme nt of quality of life) Low/ moderate Belief that intervention does more good than harm LowMay accept lower quality of evidence if costs are low Morphine for pain control Palliation (improveme nt of quality of life) HighUncertain whether the intervention does more good than harm ModerateHigh-quality particularly if costs are high Bisphosphonates; erythropoietin