Gordon J. Chelune, PhD, ABPP(Cn)Professor Emeritus, Department of Neurology

University of Utah School of Medicine

Reducing Diagnostic Uncertainty: What the Evidence-based Researcher

Can Do for Clinical Practitioners

Kessler Foundation, November 2019

This presentation will have 2 parts:

• General background on EBP and what researchers can do facilitate clinicians’ use of their research findings

• Specific methods for using data derived from research based on populations to inform clinical decisions about individuals

Evidence-based PracticeThe purpose of any evaluation is to reduce uncertainty about a patient’s diagnosis, management, or care.

EBP provides a framework for doing this in an empirical and systematic way that incorporates published data from critically appraised research to inform decisions about individual patients.

Every Patient Evaluation

➢Represents a Clinical Outcome➢Every Test Score is part of the Outcome➢Can/Should be interpreted within

context of Evidence-based Research

From “Description” to “Outcomes”

The Outcomes Movement of the 80s and 90s

What is Evidence-based Practice?

What vs. How

What is Evidence-Based Clinical

Neuropsychological Practice?

Evidence-based clinical neuropsychological practice (EBCNP)

is value-driven pattern of clinical practice that attempts to

integrate the “best research” derived from the study of

populations to inform clinical decisions about individuals

within the context of the provider’s expertise and individual

patient values with the goal of maximizing clinical outcomes

and quality of life for the patient in a cost-effective manner

while addressing the concerns and needs of the provider’s

referral sources.(Chelune, 2017, p 160)

Seven Components of Evidence-based Neuropsychological Practice

1. Value-driven pattern of clinical practice

2. Integrates “best research”

3. Derived from the study of populations to inform clinical

decisions about individuals

4. Within the context of the provider’s expertise and

individual patient values

5. Goal of maximizing clinical outcomes and quality of life for

the patient

6. In a cost-effective manner

7. Addressing the concerns and needs of the provider’s

referral sources

Adapted from Chelune, 2017

The “Value” or Clinical Significance of Evaluations

Patients “deserve decisions and recommendations that are founded increasingly upon empirical validation. The instruments chosen to produce data to resolve questions in a valid fashion should be selected for their power to reduce uncertainty with respect to those questions…”

Costa, JCN, 1983, p. 7.

Our ability “to reduce uncertainty” provides value to patient care

That’s the “What”

Now for the “How”

Evidence-based Practice

Evidence-based practice begins with asking appropriate and answerable questions, which define the parameters of our search, what to evaluate, and how to apply research findings.

Asking appropriate and answerable questions

Type of Question Type of Evidence

Etiology: Disease causes and modes of operation Case-control or cohort studies

Diagnostic: Signs, symptoms, or tests for diagnosis a disorder

Diagnostic validations studies

Prognosis: The probable course of disease over time Inception cohort studies

Therapy: Selection of effective treatments which meet patient values

Randomized controlled trials

Cost-effectiveness: Comparison of efficacy/cost ofinterventions

Economic evaluation

Quality of Life: What will QoL of the patient be Qualitative study

The Question dictates what to look for, where to look, and what to expect

Adapted from Heneghan & Badenoch, 2006

What is the Question

• Patient• Intervention• Comparison• Outcome

Asking:Well-Built Clinical Questions (PICO)

• Background: Do patients with AD and FTD have different patterns of semantic and phonemic fluency?

• Foreground: In patients with

Patient: Frontotemporal dementia

Intervention: patterns of phonemic and semantic fluency

Comparison: compared to Alzheimer’s dementia

Outcome: are different (sensitive/specific)?

Asking:Well-Built Clinical Questions (PICO):

Diagnosis

Acquire Relevant Data:Informatics skills in finding answers

Acquire Relevant Data:Informatics skills in finding answers

Database ContentMedline/PubMed General medical database; many journals not

referencedPsychINFO General psychological literature, including book

chaptersCINAHL Nursing/Allied Health

Embase Pharmacologic and biomedical database including international entries

BIOSIS Biological and biomedical sciences; journal articles, conference proceedings, books, and patents

HSTAT Health Sciences Technology and Assessment; clinical guidelines, AHRQ and NIH publications

CCRCT Cochrane Central Register of Controlled Trials

CDSR Cochrane Database of Systematic Reviews

DARE Database of Abstracts and Reviews of Effects; critically appraised systematic reviews

Campbell Collaboration

Systematic reviews in education, criminal justice, and social welfare

Common Web-based electronic research databases

Supports Boolean Operators:AND, OR, NOT, Nesting

https://www.ncbi.nlm.nih.gov/pubmed

PubMed

Knowing what is good, bad, acceptable, etc.

Appraisal Skills:

➢ Meta-analyses, Systematic Reviews, and Critically Appraised Topics

➢ Randomized Cohort Studies➢ Cohort Studies➢ Case Controlled Studies➢ Case Series and Case Reports➢ Book Chapters & Expert Opinions

Common “Types” of Evidence:Some are better than others

The Evidence Pyramid

Case Control StudiesNote reverse directionality, health outcome occurs before study begins.

Note forward directionality; health outcome occurs after study beginsCohort Studies

Identifying “Best Research” is not easy

Incomplete and inadequate reporting of research hampers the assessment of the strengths and weaknesses of the studies reported in the medical and neuropsychological literature. Readers need to know what was planned (and what was not), what was done, what was found, and what the results mean.

Reporting Guidelines:Moving toward greater transparency

❖ STROBE

❖ CONSORT

❖ STARD

Consolidated Standards of Reporting Trials.Website: http://www.consort-statement.org/

STrengthening the Reporting of OBservational studies in Epidemiology.Website: http://www.strobe-statement.org/

STAndards for the Reporting of Diagnostic accuracy studies.Website: http://www.stard-statement.org/

Composite of the two PET groups

Results: The SI ratio was significantly different between PET groups (p< .000), with patients with AD PET patterns showing lower SI scores. A 2x2 Group x Fluency repeated measures ANOVA was calculated and there was a significant non-orthogonal interaction (p < .000) showing a marked difference between fluency measures among the AD PET group. ROC analysis of SI yielded an AUC of .742 (p < .000).

Flowchart of Data Selection: Data collected from January 2006 – June 15, 2011

3092 cases in the patient registry

1245 patients with Neuropsychological evaluations

928 cases meeting inclusion criteria: MMSE ≥18 Age ≥ 55 yrs. Education > 8 yrs. English as primary language

180 patients with both PET imaging and neuropsychological

testing meeting inclusion criteria

351 patients with PET imaging

Patients are rank ordered by SSP hypometabolic (z-score) differences

between AD vs. FTD regions

Upper and lower quartiles labeled prototypic AD and FTD groups

(n=45 in each group)

✓ Asked the Relevant Question

✓ Acquired the Relevant Clinical Research

✓ Appraised the Research

Now …

Recap of the 5 A’s:

✓ To Apply the Information to a given Patient

One of the defining features of evidence-based practice is the use of data derived from research based on populations to inform clinical decisions about individuals…

…how do we move from group data to data that is applicable at the level of the individual?

Evidence-based Practice and Research

Ask Yourself:

If my patient, with his/her specific test score(s), had participated in this study, in which group would they most likely have been ???

COI RP

Do Patients with a Condition of InterestDiffer from Reference Population?

How Much vs. How Many

How Much How Many

Using Clinical Research

Is the difference between groups statistically reliable?

p < .05

Performance

COI RP

Performance

Reference

Population (RP)Healthy Controls

B C

True Positives True Negatives

FalsePositives

FalseNegatives

Condition of

Interest (COI)Alzheimer’s Dementia

A

Sensitivity = % True Positives

Specificity = % True Negatives

AD HC

Test Operating Characteristics

“How Many”

TruePositive

FalsePositive

FalseNegative

TrueNegative

Condition of Interest

Yes (COI) No (RP)

Factor(event)

Yes+

No-

A B

C D

The Basic 2x2 Table

Sensitivity

Specificity

Test Operating Characteristics

% Prevalence Odds

% Overall Correct Hit Rate Odds Ratio

Sensitivity Relative Risk Ratio

Specificity Likelihood Ratio

Positive Predictive Power Pre – Post Test Odds

Negative Predictive Power Pre – Post Test Probabilities

Bayesian approach:

Analyses of Changes in Base Rates

Bayes’ Theorem: What we know after giving a test in equal to what we knew before doing the test times a modifier (based on the test results). Test results are used to adjust a prior distribution to form a new posterior distribution of scores.

Value Driven Pattern of Practice

Nomogram for using Likelihood Ratios

(LR) to determine Post-test

Probabalities of a COI if the Pre-test

Probability and LR are known

E.g. Prevalence of COI = 20%

LR+ = 10

LR- = 0.1

Does Testing Matter

p.24

http://araw.mede.uic.edu/cgi-bin/testcalc.pl

Condition of Interest

FTD AD Totals

SI > .524 12 6 18

SI Cutoff A B

SI < .524 4 26 30

C D

Totals 16 32 48

Test Operating Characteristics for FTD

% Prevalence (Baserate) of COI 33.33

% Positive Test Result 37.50

% Negative Test Result 62.50

% Overall Correct Hit Rate 79.17

Sensitivity (% True Positives) 0.7500

Specificity (% True Negatives) 0.8125

Positive Predictive Power 0.667

Negative Predictive Power 0.867

Odds having COI w. Pos. Test 2.000

Odds having COI w. Neg. Test 0.154

Odds Ratio 13.0000

Likelihood Ratio (LR+) 4.0000

Pre-Test Odds 0.5000

Post-Test Odds 2.0000

Pre-test Probabality 0.3333

Post-Test Probabality 0.6667

Risk Ratio (cohort studies) 5.0000

Test Operating Characteristics for AD

% Prevalence (Baserate) of COI 66.67

% Positive Test Result 62.50

% Negative Test Result 37.50

% Overall Correct Hit Rate 79.17

Sensitivity (% True Positives) 0.8125

Specificity (% True Negatives) 0.7500

Positive Predictive Power 0.867

Negative Predictive Power 0.667

Odds having COI w. Pos. Test 6.500

Odds having COI w. Neg. Test 0.500

Odds Ratio 13.0000

Likelihood Ratio (LR+) 3.2500

Pre-Test Odds 2.0000

Post-Test Odds 6.5000

Pre-test Probabality 0.6667

Post-Test Probabality 0.8667

Risk Ratio (cohort studies) 2.6000

Condition of Interest

AD FTD Totals

SI < .524 26 4 30

SI Cutoff A B

SI > .524 6 12 18

C D

Totals 32 16 48

FTD AD

Likelihood FTD when SI is > .524 Likelihood AD when SI is < .524

My Patient’s SI Score is 0.65

How likely is my patient FTD?

Hey –What about my Patient??

Two Challenges:

• Investigators do not always provide information about base rates and cutoff scores.

• When provided, the cutoff scores are not specific to the patient’s actual observed scores

1. Z-score = (X – M)/ SD and tells us the area under the curve –percentile rank

2. If you know the sample size (N), you can estimate the actual number of cases above and below that z-score.

Given N=500 and a z-score of -1.533a. 500 x .94 = 469 aboveb. 500 x .06 = 31 below

True Positive False Positive

False Negative True Negative

Condition of Interest

Yes (AD) No (HC)

Factor(Test Score)

Yes< 77

No> 77

A B

C D

The Basic 2x2 Table

31

469

Specificity

??

??

Sensitivity

Ask Yourself:

If my patient, with his/her specific test score of 77, had participated in this study, in which group would they most likely have been ???

Likelihood Ratio

True Positive False Positive

False Negative True Negative

Condition of Interest

Yes (COI) No (RP)

Factor(Test Score)

Yes< 77

No> 77

A B

C D

The Basic 2x2 Table

31

469

Specificity

37

11

Sensitivity

If one has the Means, Standard Deviations, and Sample Sizes for the groups in question it is possible to estimate the number of cases in each group that would fall above or below a patient’s observed score and to calculate the TOC characteristics for that score

Caveat

**IMPORTANT** Calculations assume normal distribution of scores.Use only within the scope of this assumption

The Next Challenge:

What to do about “non-normal” distributions

The Problem of Skew

Pearson Coefficient of Skewness for grouped data

If the Mean, SD, and Skew Coefficient are known, one can algebraically solve for the Median (Md) which represents the score at which the 50th percentile occurs.

What If …

An Investigator provides:

Mean, SD, N, and Skew Coefficient …AND…

… a transformation formula used to successfully normalized the data, can you use the Mean and SD of the transformed scores to generate base rates for observed scores?

How to program a Feigan Nomogram in Excel?

How to deal with variables that are skewed?

If variables with skewed distributions can be normalized, can the Mean and SD of the normalized scores then be used to determine the number of cases above and below a patient’s score ?

Open to Suggestions/Comments/Help:

Johannes Moreelse