How do CME speakers use research results to support therapeutic

recommendations?A quantitative and qualitative study

Michael Allen, Tanya Hill, Richard Handfield-Jones, Mike Fleming, Doug Sinclair, Tom Elmslie

SACMERancho Mirage, CA

April, 2009

3

Research TeamI’m Mike Allen,

the PI from Dalhousie CME

I’m Richard Handfield-Jones,

the PI from University of Ottawa CME. I’m in the army now and AWOL.

I’m Mike Fleming from Dalhousie CME and I’m

not sure how I got involved in this.

I’m Doug Sinclair from Dalhousie CME. I can

find time to do anything.I’m Tanya Hill the research associate from Dalhousie CME. I don’t care who says they’re the PI, I’m running this

show.

I’m Tom Elmslie. I’m an ideas man and helped with

experimental design.

4

Funding

Royal College of Physicians and Surgeonsof Canada

Nova Scotia Department of Health throughDrug Evaluation Alliance of Nova Scotia

6

Think about this RCT …

Outcomes: non-fatal myocardial infarction and death from coronary heart disease Subjects: No history of coronary heart disease (primary prevention)Duration: 3.3 years N: 5100 patients in the control and 5100 patients in the drug groupThe patient characteristics are:

80% male mean age = 63 yrs. mean blood pressure = 164/95

ASCOT Lancet 2003;361:1149-58

7

1. The drug led to a 36% decrease in the incidence of non-fatal MI and CHD death (relative risk reduction).

2. The drug decreased the rate of non-fatal MI and CHD death from 3.0% to 1.9%, an absolute risk reduction of 1.1%.

3. You would have to treat 94 patients for 3.3 years to prevent one non-fatal MI or a death from CHD (number needed to treat).

4. The 95% confidence intervals around the previous result (ie, treat 94 patients for 3.3 years to avoid one non-fatal MI or CHD death) are 60 and 215.

5. At the end of 3.3 years, 97.0% of patients who don't take the drug will remain free of a cardiac event and 98.1% of patients who take the drug will remain free of a cardiac event (inverse absolute RR).

What is your interpretation of the following results?How likely might you be to prescribe or take the drug based on each one?

8

Patients not having MI or dying: Placebo

Patients having MI or CHD death

Patients NOT having

MI or CHD death

9

Patients not having MI or dying: Treatment

Patients having MI or CHD death

Patients NOT having

MI or CHD death

10

Forrow L, Taylor WC, Arnold RM. Absolutely relative: How research results are summarized can affect treatment decisions. Am J Med 1992; 92:121-4.

Naylor CD, Chen E, Strauss B. Measured enthusiasm: Does the method of reporting trial results alter perceptions of therapeutic effectiveness? Ann Intern Med 1992; 117:916-21.

Nikolajevic-Sarunac J, et al. Effects of information framing on the intentions of family physicians to prescribe long-term hormone replacement therapy. J Gen Intern Med 1999; 14(10):591-598

Nexoe J, Kristiansen IS, Gyrd-Hansen D, Nielsen JB. Influence of number needed to treat, costs and outcome on preferences for a preventive drug. Fam Pract 2005; 22(1):126-131.

11

ObjectivesTo determine:

How much emphasis speakers place on presenting data in absolute vs relative termsHow familiar FPs and speakers are with statistical terms (especially absolute and relative terms)The attitudes and preferences of FPs about having research data presentedHow well speakers link therapeutic recommendations to research data

12

Methods

2 annual 3-day Family Medicine review conferencesHalifax (Dalhousie University CME)- 230 registrants- 42 presentations (lecture and concurrent sessions)

Ottawa (University of Ottawa CME)- 209 registrants- 38 presentations (lecture and concurrent sessions)

3 data collection methods for each conference

13

Methods – Data Collection1. Questionnaire

Registrants and SpeakersEvaluate understanding of statistical terms

2. Focus groups1.5 hour with FPs from each siteDetermine preferences about types of statistics used

3. Analysis of presentationsPowerPoint filesVideo recording of presentations

14

Methods – Data Analysis1. Questionnaire

Descriptive statistics

2. Focus groupsRecorded and transcribedContent analysis

3. Analysis of presentationsPowerPoint files - How research data presented – slide counts

Videorecording- Linking of research data to recommendations – Likert scales

Tworesearchers

15

Results: Questionnaire

Registrants (N=121; 29% response rate)97% physicians; 81% certificant of CFPC51% male; 49% femaleAvg. yrs practice = 18.027% completed EBM workshop in last 10 years

Speakers (N=20; 33% response rate)19 specialists, 1 family physician70% male; 30% femaleAvg. yrs practice = 18.030% completed EBM workshop in last 10 yrs

17

Results: Questionnaire (Registrants N=121)

% understand vs. % correct response

5136 40

4755

38

0

20

40

60

80

100

NNT ARR RRR

% understand andcould explain

% correctlycalculated

18

Results: Questionnaire (Speakers N=20)

% understand vs. % correct response

80

45 50

75 70 65

0

20

40

60

80

100

NNT ARR RRR

% understandand couldexplain

% correctlycalculated

19

Results: Questionnaire

06

3438

22

05

40

3025

0

10

20

30

40

50

60

Very little Some A lot

Registrant(N=121)

Speaker(N=20)

"How much emphasis should CME speakers place on presenting research results when making therapeutic recommendations?"

Per

cent

resp

onse

s

20

Results: Focus group

2 focus groups9 males; 5 females

Most preferred statistics:1. Numbers needed to treat 2. Inverse ARR3. Confidence Intervals

22

Results: Focus group

Trust in speakers and CME providersTrust that specialists can critically evaluate researchSkepticism about specialist ability to interpret dataAttend CME because specialists simplify data and make recommendations supported by research Trust CME offices to ensure programs present data accurately and completelyDisclose conflict of interest

24

Preliminary results: PowerPoints (N=26)

General termsGraphsRisk (event rate)PrevalenceAbsolute or relative not specified

Absolute termsAbsolute risk reduction or increaseNumber needed to treat or harm

Relative termsRelative riskRelative RROdds ratioHazard ratio

95% confidence intervals

25

Preliminary results: PowerPoints (N=26)

PresentationTotal

content slides

Therapeutic recmdation

General terms

Relative terms

Absolute terms 95% CIs

1 29 4 9 0 0 02 49 14 0 0 0 03 45 10 15 0 0 04 72 22 10 1 0 25 37 5 1 0 0 06 16 3 0 0 0 07 111 14 27 4 0 28 131 23 18 10 6 9

Number of slides with . . .

26

Preliminary results: Powerpoints (N=26)

Presentation Total Therapeutic recmdation

General terms

Relative terms

Absolute terms 95% CIs

9 49 6 3 4 2 110 26 7 6 3 0 111 32 7 10 3 1 212 21 7 4 0 0 013 49 16 2 0 0 014 69 22 24 5 1 115 39 2 10 2 4 216 102 8 4 0 0 017 51 9 10 0 0 018 48 12 0 0 0 0

Number of slides with . . .

27

Preliminary results: PowerPoints (N=26)

Presentation Total Therapeutic Recmdation

General terms

Relative terms

Absolute terms 95% CIs

19 31 10 5 0 0 0

20 50 7 2 0 0 0

21 29 6 0 0 0 0

22 68 8 8 5 4 4

23 21 0 1 0 0 0

24 51 3 3 0 0 0

25 59 3 6 5 0 3

26 59 22 12 5 0 2

TOTAL 1344 250 200 47 18 29

Number of slides with . . .

28

Preliminary results: Presentations (N=26)

PresentationTotal

content slides

Therapeutic recmdation

General terms

Relative terms

Absolute terms 95% CIs

1 29 4 9 0 0 02 49 14 0 0 0 03 45 10 15 0 0 04 72 22 10 1 0 25 37 5 1 0 0 06 16 3 0 0 0 07 111 14 27 4 0 28 131 23 18 10 6 9

Number of slides with . . .

29

Preliminary results: Videotapes (n=19)

Scale: 1=very little, 3=some, 5=a lot *n/a = 13 presentations

Mean* (SD)

1. How much time did speaker spend on therapeutic interventions of any kind? 4.0 (.82)

2. How much time did the speaker spend relatively between relative/absolute terms?* 3.2* (.41)

3. How much time/effort did the speaker take to explain the research data? 2.8 (1.4)

4. How often did the speaker mention relative/absolute terms when not on slide? 1.7 (1.2)

5. How thoroughly did the speaker link relative/absolute terms with recommendations? 2.1 (1.2)

6. How much evidence did the presentation warrant? 3.4 (.90)

7. How much emphasis did the CME speaker place on presenting research results when making therapeutic recommendations? 2.9 (1.4)

30

ConclusionsMost FPs have low to moderate level of statistical comprehension

Speakers have higher level of statistical comprehension than FPs

FPs want recommendations to be supported with research

FPs want speakers to critically evaluate the research

Most speakers present little data in relative or absolute terms or with 95% CIs

Focus group recommendations for CME providersProvide short review on statistics for FPsDevelop consistent format for presentation of research data

31

Next Steps

Provide CME for FPs to help them understand research dataDevelop format for presenting data that FPs find helpful and understandableProvide faculty development for speakersAdd question to evaluation form re presentation of research data

32

Questions?