What are your criteria in reporting on health care research?

Gary Schwitzer, Publisher, HealthNewsReview.org

Background on HealthNewsReview.org, what we do, how we do it, why, and what we’ve found

Recurring pitfalls we’ve seen in 6 years of daily monitoring

An editor’s perspective on addressing these issues

We’ll review a story – together – using HealthNewsReview.org criteria.

Plenty of time for Q & A – tapping Ivan’s editor expertise

Handouts: Summary of our 6-year, 1,700 story experience Resources for reporting on studies Story to review & scoresheet

Our plan for next 2 hours:

We review stories that include claims about…

Medical treatments

Tests

Products

Procedures

Site stats

28 reviewers: 19 clinicians or researchers, 7 journalists, 2 breast cancer survivors

Site launched April 17, 2006 – 6th anniversary week!

>1,700 stories reviewed

Blog launched in 2004 - >2,500 posts

Sole funding from the Informed Medical Decisions Foundation, founded by Jack Wennberg et al

= HealthNewsReview.org (U.S.), Media Doctor sites in Australia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites in UK and Austria

= talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, Puerto Rico only in two years’ time

Global Reach for such efforts

Our criteria: Does the story explain…

What’s the total cost?

How often do benefits occur?

How often do harms occur?

How strong is the evidence?

Is the condition exaggerated?

Is this really a new approach?

Is it available?

Are there alternative choices?

Who’s promoting this?

Do they have a financial conflict of interest?

After 6 years and 1,700 stories

70% fail to adequately discuss costs.

66% fail to quantify benefit – often exaggerating potential benefit

65% fail to quantify harm - often minimizing potential harm

62% fail to evaluate the quality of the evidence

57% fail to compare new idea with existing options

Kid-in-candy-store picture of U.S. health care

Everything is terrificNothing is riskyNo price tags

Recurring themes seen after 6 years of daily

monitoringFailure to explain limitations of

observational studies. Stories that conflate association and causation.

Framing numbers to exaggerate benefit, minimize harms

Stories about screening tests that emphasize only benefits, minimizing or ignoring potential harms

Be skeptical about observational studies

Remind readers that the findings may be more about the people being studied than the "exposure".

They can point to a strong statistical association, but they cannot prove cause and effect.

White rice increases risk of type 2 diabetes

Thursday, March 15, 2012

The risk of type 2 diabetes is significantly increased if white rice is eaten regularly, claims a study published today on bmj.com.

More than 400 words in the news release but NONE about the limitations of such an observational study.

Does your language fit the evidence of observational studies?

Absolute versus relative risk

Following slides courtesy Drs. Steve Woloshin & Lisa

SchwartzDartmouth Medical School

White River Junction, Vermont VA

Part of their syllabus at: NIH Medicine in the MediaMIT Medical Evidence Boot

Camp

Nolvadex (tamoxifen)Reducing breast cancer risk by 50

percent.

For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk.

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

48% of what ???

Women who took Nolvadex had 48% fewer breast cancers….

48

Extremely Fancy Store

%OFF

On selected items!

It's like a coupon…

What if selected items were..

TV’s, washing machines?

Things like a pack of gum?

save $100s

save pennies

“48% of what” matters!

Know the REGULAR price!

What is the

48% off coupon really worth?

How much do you save?

"Breast cancer risk"

NOLVADEX???

Placebo

???

Chance of getting breast cancer

The chance of getting breast cancer over 6 years

in the PLACEBO group

3.3%

The REGULAR price!

PLACEBO

The "base rate"

Absolute risk in the placebo group

Event rate in the placebo group

NOLVADEX

The chance of getting breast cancer over 6 years

in the INTERVENTION group

3.3%

The SALES price!The REGULAR price!

PLACEBO

1.7%

Chance of getting breast cancer (over 6 years) with NOLVADEX was 1.6 % points lower than with placebo

NOLVADEX

3.3%

PLACEBO

1.7%

What is the effect of Nolvadex?How good is the sale?

Savings = Regular price – Sales price

How much do you save?

If 100 women took NOLVADEX instead of placebo for 6 years, there would be about 2 fewer cases of breast

cancer.

The SALES price!The REGULAR price!

= 1.6%

3.3% 1.7% Absolute risk reduction

NOLVADEX

3.3%

PLACEBO

1.7%

What is the effect of Nolvadex?How good is the sale?

Relative Risk =Chance of outcome (intervention)

Chance of outcome (control)

NOLVADEX

3.3%

PLACEBO

1.7%

Relative Risk = = 0.52 ???

???

1.7%

3.3%

Describing the effect of NOLVADEX

"% Lower" format =1 - RR =1 - .52 =.48

So finally....this is how you get to

the 48% off sale! It's the relative risk reduction

At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.

48

Extremely Fancy Store

%

OFF

On selected items!

1.6%

Extremely Fancy Store

SAVINGS

On selected items!How you say it matters!"Framing"

Two ways of saying the same thing:

the benefit of NOLVADEXOne feels

bigOne feels

small

??

Relative vs. absolute risk reductions

Relative Absolute

30% 10%

Chance of death at 1 year

Risk reduction

67%

67%

67%

3% 1%

0.003% 0.001%

??20%

Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)

2%

0.002%

"% off" "savings"

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

”Strokes, cataracts more common with Nolvadex. Most women experience some level of hot flashes and vaginal discharge".

"Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women".

BenefitHarm

No numbers

Present benefits and harms the same way

210% more uterine cancer and potentially life threatening blood clots in the lung and legs.

210% more

The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.

Harm

6-yr chance of venous thromboembolic event:

Like inflation…doubled the price!

NOLVADEX

0.5%

PLACEBO

1.0%

Over the next 6 years, what happened when women…

Benefits: Nolvadex lowered chance

Getting breast cancer

Harms: Nolvadex increased chance

Having a serious blood clot

3.3% 1.7%

0.5% 1.0%

Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer

Getting uterine cancer 0.5% 1.1%

NOLVADEX PLACEBO

Take home messagesExtrapolate with caution! Don't tell people what to worry about – or do –

based on very preliminary animal / lab science.

Recognize pseudo-evidence Publication in a medical journal - even "the New England Journal of Medicine" - does not guarantee the findings are true (or even important).

Be wary of inherently weak science Without a comparison group, it is impossible to be sure if the drug was responsible for the findings.

Pay attention to the outcome Surrogate outcomes (like tumor shrinkage) do not reliably translate into clinically meaningful outcomes (longer life).

Be skeptical about observational studies Remind readers that the findings may be more about the people being studied than the "exposure".

Avoid exaggerated numbers Use absolute risks for both benefits and harms.

Pitfalls of a steady diet of journal

stories

PLoS Med 2005; 2(8): e124

The problem begins with the public’s rising expectations of science. Being human, scientists are tempted to show that they know more than they do. The number of investigators—and the number of experiments, observations and analyses they produce—has also increased exponentially in many fields, but adequate safeguards against bias are lacking. Research is fragmented, competition is fierce and emphasis is often given to single studies instead of the big picture. Much research is conducted for reasons other than the pursuit of truth. Conflicts of interest abound, and they influence outcomes.

Spinning results of randomized clinical

trialsBoultron et al, JAMA May 26, 2010, 303 (20): 2058

In this representative sample of RCTs published in 2006 with statistically non-significant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.

Reporting bias in medical research - a narrative

review

McGauran et al. Trials 2010, 11:37

We identified reporting bias in 40 indications comprising around 50 different pharmacological, diagnostic, and preventive interventions. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.

• What Ivan’s project has shown about retractions, research fraud, fabrication, falsification of data

• Unpublished data (BMJ recently published 8 articles in one edition on “the extent, causes and consequences of unpublished evidence”)

• Conflicts of interest – guideline setting

BE AWARE OF…..

• Commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies (Carl Elliott)

• Ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)

• The focus on surrogate markers in many studies may be hurting patient care. (primer in HealthNewsReview.org online toolkit)

News coverage & poor public discussion of screening tests is one of the most concerning public policy issues.

Worst, most biased coverage I’ve seen in 37 years

“The decision to start regular, biennial screening mammography before the age of 50 years should be an individual one and take patient context into account, including the patient's values regarding specific benefits and harms.”

What the Task Force actually wrote:

As 37-year ChiTrib & NYT vet John Crewdson wrote in The Atlantic…

“There are multiple reasons women are ill-informed about breast cancer. The fault lies primarily with their physicians, the cancer establishment, and the news media--especially the news media. Until coverage of breast cancer rises above the level of scary warnings mixed with heartwarming stories of cancer survivors, women are likely to go on being perplexed."

Other examples:

Chicago Sun-Times

Wall Street Journal

Washington Post

New York Daily News

Minneapolis Star Tribune

All TV networks

The ink isn’t even

dry on studies before

marketin

g begins

HealthDay wire service didn’t challenge researcher promoting universal pancreatic cancer screening for everyone over 50.

After study of tissue from 7 people!

We could have given the finger to this story

"Relative finger length could be used as a simple test for prostate cancer risk, particularly in men aged under 60," said one of the researchers.

Screening Madness

Crusading

one-sided

advocacy

Why don’t we deliver this

message?

“All screening tests cause harm;

some may do good.”

But much health journalism consistently emphasizes benefits & minimizes harms

A form of disease-mongering

Selling sickness

Selling the search for weapons of mass destruction inside everyone

Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK

“Screening the apparently healthy potentially saves a few lives. But it definitely drags many others into the system needlessly — into needless appointments, needless tests, needless drugs and needless operations. This process doesn’t promote health; it promotes disease. People suffer from more anxiety about their health, from drug side effects, from complications of surgery. A few die. And remember: these people felt fine when they entered the health care system.”

Communication of the evidence is a key health policy issue

“I honestly believe it is better to know nothing than to know what ain’t so.”  

Josh Billings (pen name of humorist Henry Wheeler Shaw, 1818 – 1885)

Gary@HealthNewsReview.org

Thank you