Birds, Bears, Turtles, Trains, Coupons, Cocks…and Moses

Gary SchwitzerPublisher

Association of Health Care JournalistsBoston – March 14, 2013

We review stories that include claims about…

• Medical treatments• Tests• Products• Procedures

Our criteria: Does the story explain…

• What’s the total cost?• How often do benefits occur?• How often do harms occur?• How strong is the evidence?• Is the condition exaggerated?• Is this really a new approach?• Is it available?• Are there alternative choices?• Who’s promoting this?• Do they have a financial conflict of interest?

After 1,800 stories over 7 years

~70% of stories fail to:

✔ Discuss costs

✔Quantify potential benefits

✔ Quantify potential harms

✔ Evaluate the quality of the evidence

News stories often paint a kid-in-candy-store picture

of U.S. health care

TerrificRisk-freeWithout a price tag

Exaggerate or emphasize benefits

Ignoreor

Minimize Potential Harms

The award for worst news coverage:

THE WINNER IS:

SCREENING TESTS(often involving this gland)

Turtles, birds, bears• Turtles move too slowly to ever be dangerous and don’t need treatment. • Birds are so fast you’ll never catch them. Too late to try to treat.

• Bears are dangerous, but move slowly enough that you can catch them. • Screening tests can only make a difference for the

bears.

Beth gets on in Boston

Nancy boards in NY

Train crashes in Washington. Both die. It may look like Beth was a longer-term survivor of the train ride, but only because she got on earlier than Nancy.

Washington

Lead-time bias

While running for president Rudy Giuliani announced in a 2007 campaign ad:“I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer—and thank God, I was cured of it—in the United States? Eighty-two percent. My chance of surviving prostate cancer in England? Only 44 percent under socialized medicine.”

Yet despite this impressive difference in the five year survival rate, the mortality rate was about the same in the US and the UK.

How can that be? Remember the train? Lead-time bias and overdiagnosis. Far more USmen get PSA test than men in the UK, contributing to US’ higher survival rate.

Susan G Komen for the Cure’s mammography advertisement during breast cancer awareness month, 2011.

Woloshin S , and Schwartz L M BMJ 2012;345:bmj.e5132

©2012 by British Medical Journal Publishing Group

Common flaws: too much stenography – not independent vetting of studies in journals

Glorifying big names/big journals - Publication in a medical journal does not guarantee the findings are true (or even important).

Not ready for prime time – journals meant for conversation among scientists

Never intended to be sources of daily news. So if you’re going to use them that way, you simply must be aware of the landscape:

• retractions, research fraud, fabrication, falsification of data

• unpublished data (BMJ special edition on “the extent, causes and consequences of unpublished evidence”)

• ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)

Failure to evaluate inherently weak science

Idolatry of the surrogate – Not understanding or simply not reporting that surrogate outcomes (like tumor shrinkage) may not translate into clinically meaningful outcomes (longer life).

Reckless extrapolation - Predicting what may happen in humans – and soon - based on very preliminary animal / lab science.

Lack of awareness of conflicts of interest & other ethical issues

Going soft on business stories or on local stories

Common flaws

What CNN didn’t report…

• Resignations, questions about conflicts of interest and fraud probes of the program in question.

• Not likely to be the kinds of angles and issues one pursues when there are “exclusive” reporting arrangements between a news organization and a medical center.

Prostate cancer scan advance helps Mayo Clinic doctors with early detection

Minnesota Public Radio

“The scan allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”

Stories about tests should explain sensitivity and specificity

• Sensitivity = how many true positives• Specificity = how many true negatives• What the Mayo story didn’t report was that studies

have shown false positive tests of 15-47% - meaning biopsies and more risks.

• And one Mayo report noted 11% false negatives• Shouldn’t that be reported alongside “The scan

allows men with cancer to receive treatment that is both faster and potentially more effective than current tests.”

Evidence – not just excitement – about expensive new technologies

Using causal language to describe observational studies

• Observational studies cannot prove cause and effect so it is inaccurate to use terms like “benefits…protects…reduces risk”

• These studies can only show a statistical association, so all you can say is that. We offer a primer, “Does The Language Fit The Evidence? – Association Versus Causation.”

“Relative finger length could be used as a simple test for prostate cancer risk”

Based on what?

The men were shown pictures of different finger length patterns and asked to identify the one most similar to their own.Men whose index and ring fingers were the same length -- about 19 percent of those studied -- had a similar prostate cancer risk, but men whose index fingers were longer than their ring finger were 33 percent less likely to have prostate cancer.

Association ≠ Causation

Exaggerating effect size – Absolute vs. Relative Risk

• Two ways of saying the same thing

• One way – relative risk reduction – makes effect size seem larger

• Other way – absolute risk reduction – makes effect size seem smaller.

• We absolutely think you should use the absolute figures.

Nolvadex (tamoxifen)Reducing breast cancer risk by 50 percent.

For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk.

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

48% of what ???

Women who took Nolvadex had 48% fewer breast cancers….

48

Extremely Fancy Store

%OFF

On selected items!

It's like a coupon…

What if selected items were..

TVs, washing machines?

Things like candy?

save $100s

save pennies

“48% of what” matters!

Know the REGULAR price!

Chance of getting breast cancer (over 6 years) with NOLVADEX was 1.6 % points lower than with placebo

NOLVADEX

3.3%

PLACEBO

1.7%

What is the coupon really worth? Chance of getting breast cancer over 6 years

Savings = Regular price – Sales price

How much do you save?

If 100 women took NOLVADEX instead of placebo for 6 years, there would be about 2 fewer cases of breast

cancer.

Nolvadex: The SALES price!Placebo:The REGULAR price!

= 1.6%

3.3% 1.7% Absolute risk reduction

NOLVADEX

3.3%

PLACEBO

1.7%

What is the effect of Nolvadex?How good is the sale?

Relative Risk =Chance of outcome (intervention)

Chance of outcome (control)

NOLVADEX

3.3%

PLACEBO

1.7%

Relative Risk = = 0.52 ???

???

1.7%

3.3%

Describing the effect of NOLVADEX

"% Lower" format =1 - RR =1 - .52 =.48

So finally....this is how you get to

the 48% off sale! It's the relative risk reduction

At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.

48

Extremely Fancy Store

%

OFF

On selected items!

1.6%

Extremely Fancy Store

SAVINGS

On selected items!How you say it matters!"Framing"

Two ways of saying the same thing: the benefit of NOLVADEX

One feels big One feels small

??

Relative vs. absolute risk reductions

Relative Absolute

30% 10%

Chance of death at 1 year Risk reduction

67%

67%

67%

3% 1%

0.003% 0.001%

??20%

Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)

2%

0.002%

"% off" "savings"

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

”Strokes, cataracts more common with Nolvadex. Most women experience some level of hot flashes and vaginal discharge".

"Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women".

BenefitHarm

No numbers

Present benefits and harms the same way

210% more uterine cancer and potentially life threatening blood clots in the lung and legs.

210% more

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

Over the next 6 years, what happened …

Benefits: Nolvadex lowered chance Getting breast cancer

Harms: Nolvadex increased chance Having a serious blood clot

3.3% 1.7%

0.5% 1.0%

Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer

Getting uterine cancer 0.5% 1.1%

NOLVADEX PLACEBO

• Surrogate markers may not tell the whole story• Does The Language Fit The Evidence? – Association Versus

Causation• 7 Words (and more) You Shouldn’t Use in Medical News• Problems with Reporting on News from Scientific Meetings• Absolute vs. Relative Risk• Number Needed to Treat (more on this in the afternoon)• Commercialism• Single Source Stories• Phases of Drug Trials• Medical Devices• Animal & Lab Studies

Progression free survival = The length of time during and after the treatment of cancer, that a patient lives with the disease but it does not get worse.

Versus

Overall survival = do people live longer?

Read John Ioannidis to learn pitfalls of a steady diet of journal stories

PLoS Med 2005; 2(8): e124

Journals complicit in miscommunication

• Editors of the HEART Group journals recently stated that “inappropriate word choice to describe results can lead to scientific inaccuracy.”

– J AM COLL CARDIOL, Vol. 60, No. 23, 2012

• “Are we making a mountain out of a mole hill? A call to appropriate interpretation of clinical trials and population-based studies”

– Am J Obstet Gynecol, published online 11/29/12

• “Spin and Boasting in Research Articles.”- Commentary in Arch Pediatr Adolesc Med:

[published online October 2012]

Annals of Oncology: “Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer”

(published online January 9, 2013)

Reuters: “Doctors relying on studies published in top journals for guidance about how to treat women with breast cancer may not be getting the most accurate information.”

What the authors point out:

• “Investigators go overboard to make their studies look positive.”

• In 2/3 of studies, that meant not listing serious side effects in the abstract – which is all many may read

• In 1/3 of studies, if the treatment didn’t work as hoped, researchers reported results the study was not designed to test – “secondary endpoints”

September 2012

Positive “spin” was identified in about half of press releases and news stories. The main factor associated with “spin” in press releases was the presence of “spin” in the journal article abstract conclusion.

In other words, a direct link from published study news release news story.

Where is the reader left behind in this food chain?

Public Misunderstanding of Cancer Risk

• Studies have consistently shown people overestimate their own risk of cancer

• Other studies show people overestimate risk factors that have not been proven and underestimate risk factors that are well-established

“Unrealistic Optimism in Early-Phase Oncology Trials”

People tended to overestimate the benefits of the trial they were enrolled in and underestimate its risks.

-- IRB: Ethics & Human Research 2011

We hold a mirror up to journalists – Ask yourselves:

• If you cover studies/clinical news, how many of those stories are just about new treatments, tests, products or procedures?

• Do you think you might be reporting too much of this?

• If so, have you spoken with editors about this? Do you want our help?

• If you don’t think you report too much of this, what kind of grades would you get on our 10 criteria?

• If you don’t accept or agree with our 10 criteria, what criteria do you use?

Drowning from a firehose of infoxication

@garyschwitzer

@ivanoransky

My email: Gary@HealthNewsReview.org

Thank you for your attention

Thanks to Drs. Steve Woloshin & Lisa Schwartz of Dartmouth Medical School for slides on absolute vs. relative risk