d1 nisha almeida

A framework for translating evidence to recommendations

within a hospital-based HTA unitNisha Almeida, PhDMcGill University Health centre

2 Disclosure

I have no actual or potential conflict of interest in relation to this topic or presentation.

3 Background

Proliferation of HTA units Hospital-based HTA units to support decision-making

Useful tools to adopt cost-effective and clinically pertinent technologies Yet, while there are numerous guidelines to evaluate and rate the

clinical evidence (e.g. GRADE, Cochrane), there are few comparable tools to guide decision-makers in framing recommendations what factors must be considered in this decision-making process?

4 Factors important to decision-making Everyone agrees that efficacy, safety and cost

of the technology are important considerations Reviewed 24 past TAU reports Correlated the overall strength of 3 factors

(efficacy, cost, safety) with the final recommendation Approved: overall assessment supports a

recommendation for routine use with support through the hospital operating budget

Approved for evaluation: overall assessment does not support permanent approval but is promising enough to warrant an evaluation,

Not approved: The technology is not approved for financial support through the hospital operating budget, but may be re-evaluated based on new evidence

But many more factors/dimensions to considerApproved Approved for evaluation Not approved

0

1

2

3

4

5

6

7

8

Weak Moderate Strong

Recommendation type

Freq

uenc

y

5Previous frameworks

GRADE1 OHTAC2

Domain CommentBalance between desirable and undesirable effects

The larger the difference between the desirable and undesirable effects, the higher the likelihood that a strong recommendation is warranted. The narrower the gradient, the higher the likelihood that a weak recommendation is warranted

Quality of evidence

The higher the quality of evidence, the higher the likelihood that a strong recommendation is warranted

Values and preferences

The more values and preferences vary, or the greater the uncertainty in values and preferences, the higher the likelihood that a weak recommendation is warranted

Costs (resource allocation)

The higher the costs of an intervention—that is, the greater the resources consumed—the lower the likelihood that a strong recommendation is warranted

Domain Sub-domainOverall clinical benefit

EffectivenessSafetyBurden of illnessNeed

Feasibility of adoption

Economic feasibility Organizational feasibility

Value for money Economic evaluationsConsistency with expected societal and ethical values

Expected societal valuesExpected ethical values

1 Grading of Recommendations Assessment, Development and Evaluation, McMaster University

2 Ontario Health Technology Appraisal Committee

6 Goal Exhaustively enumerate relevant domains important to the decision-

making process within a hospital setting To create a checklist that is structured and transparent

to ensure that all relevant domains are given equal consideration to adequately document the factors and reasons shaping the final

recommendation That can be applied in a systematic fashion to each technology to be

evaluated

7 Objectives

Identify relevant domains that stakeholders in the decision-making process recognize as important to shape a recommendation.

Propose specific criteria to rate the strength of the evidence, information, or experience, for each domain.

Estimate the extent to which members of the TAU Policy Committee endorse the dimensions and criteria, and identify barriers and facilitators to adoption of the checklist for the decision making process.

8 Methods Used OHTAC framework as a starting point to define global domains:

Clinical benefit; Cost; Feasibility; Ethics

Performed a review of our past reports, to identify domains that played a role in the decision-making process Two members of TAU independently reviewed a random sample of past reports

Considered domains from a survey of hospitals in France, that included 47 determinants (grouped under 8 domains) based on an extensive literature review

9 Methods

Developed a preliminary list of important domains Created a survey in order to solicit the opinions of experts within and

outside the MUHC community To determine the importance of the suggested domains Gather other relevant domains Collect information on MUHC values and preferences

10Survey domainsQuestion: Which of the following domains do you think are most important for consideration by stakeholders when framing recommendations for use of a health technology in a hospital setting?Domain Sub-domainClinical benefit • Magnitude of effectiveness

• Quality of evidence for effectiveness• Safety• Burden of illness to patient• Absence of alternative treatment options

Value for money • Costs related to the technology• Increased hospital efficiency• External financial support

Feasibility • Ease of implementation• Prior hospital experience with the technology• Need for evidence of effectiveness in the local setting

Ethics and values

• Disease is a public health priority• Disease is rare• Benefit of technology to society• Impact on delivery of equitable care• Impact on patient-important outcomes

Strategic considerations

• Impact of technology on attracting new patients and/or health professionals• Impact on creating research opportunities and external collaborations• Ability to offer a cutting-edge technology or new alternative treatment• Availability of the technology in other local centres

11 Comments

12 Results

We sent the survey out to 61 panelists, and received 52 completed responses (85%)

Type N (Total= 52)

MUHC member 38MUHC non-member 14HTA member 16Administrator 12

Magnitude of effectivenessQuality of the evidence for effectiveness

Safety of the technology

Burden of illness to patient

Absence of alternatives

Cost related to the technology

Increased hospital efficiency

External financial support for the technology

Ease of implementation

Prior hospital experience with the technologyNeed for evidence of effectiveness in the local setting

Disease is a public health priority

Disease is rare

Benefit of technology to society

Impact on delivery of equitable care

Impact on patient-important outcomes

Impact of technology on attracting new patients and/or health professionals

Impact on creating research opportunities and external collaborations

Ability to offer a cutting-edge technology or new alternative treatment

Availability of the technology in other local centres

0

50

100

Percent of respondents rating the domain 4 or 5

13

14

Percent of respondents rating the domain 4 or 5

Magnitude of effectivenessQuality of the evidence for effectiveness

Safety of the technology

Burden of illness to patient

Absence of alternatives

Cost related to the technology

Increased hospital efficiency

External financial support for the technology

Ease of implementation

Prior hospital experience with the technologyNeed for evidence of effectiveness in the local setting

Disease is a public health priority

Disease is rare

Benefit of technology to society

Impact on delivery of equitable care

Impact on patient-important outcomes


Impact on creating research opportunities and external collaborations

Ability to offer a cutting-edge technology or new alternative treatment

Availability of the technology in other local centres

0

50

100

MUHC membersMUHC non-members

15

Consensus on domainsDomain Sub-domain Consensus Clinical benefit

• Magnitude of effectiveness 100

• Quality of evidence for effectiveness 94.2

• Safety 98.0

• Burden of illness to patient 76.9

• Absence of alternative treatment options 57.7

Value for money

• Costs related to the technology 92.3

• Increased hospital efficiency 92.3

• External financial support 21.2

Feasibility

• Ease of implementation 78.8

• Prior hospital experience with the technology 34.6

• Need for evidence of effectiveness in the local setting 61.5

Ethics and values

• Disease is a public health priority 69.2

• Disease is rare 46.2

• Benefit of technology to society 73.1

• Impact on delivery of equitable care 78.8

• Impact on patient-important outcomes 90.4


• Impact of technology on attracting new patients and/or health professionals

36.5

• Impact on creating research opportunities and external collaborations 51.9

• Ability to offer a cutting-edge technology or new alternative treatment 69.2

• Availability of the technology in other local centres 63.5

16 Absence of alternativesRating Frequenc

y0 (Don't know) 1.91 (Not at all important) 1.92 (Not very important) 1.93 (Somewhat important) 36.54 (Important) 38.45 (Indispensable) 19.2

Comments Depends on context of the patient; would

be important to consider in rare diseases with serious negative outcomes

Clearly important given financial constraints

The weight to give at this criterion is unclear. Absence of alternative treatment should not be used to justify the decision of introduce a medical technology when there is weak evidence;

alternative treatment options arguably belongs in the ethics domain

There are always alternative options. It should be explicit and understood that "magnitude of effectiveness" means magnitude compared to the next best options

17Prior hospital experience with technologyRating Frequency0 (Don't know) 01 (Not at all important) 7.72 (Not very important) 23.03 (Somewhat important) 34.64 (Important) 26.95 (Indispensable) 7.7

Comments if you are doing a reassessment,

then this item would be important.

Not intrinsically important although it may help to get going

Not unless this prior experience generated data on clinical benefit, cost or other aspects of interest. Again, something to be flagged on a "Do not consider" list.

18 Suggestions for new domains Clarify: comparative effectiveness and safety Level of innovation Impact on patient outcomes No. of patients affected Cost-effectiveness Quantify cost-avoidance Ability to increase cross-institution collaboration Impact on budget of other units Impact on services of other units (disruptiveness) Need for the technology Impact on attractiveness of the centre Clinical pertinence

Reorganization of domainsOld version

Domain Sub-domainClinical benefit

1. Magnitude of effectiveness2. Quality of evidence for effectiveness3. Safety4. Burden of illness to patient5. Absence of alternative treatment options

Value for money

6. Costs related to the technology7. Increased hospital efficiency8. External financial support

Feasibility

9. Ease of implementation10. Prior hospital experience with the technology11. Need for evidence of effectiveness in the local

settingEthics and values

12. Disease is a public health priority13. Disease is rare14. Benefit of technology to society15. Impact on delivery of equitable care16. Impact on patient-important outcomes


17. Impact of technology on attracting new patients

18. Impact on creating research opportunities19. Ability to offer a cutting-edge technology20. Availability in other local centres

Domain Sub-domainClinical benefit

1. Magnitude of effectiveness2. Quality of evidence for effectiveness3. Safety

Impact on Patient

4. Impact on patient convenience5. Personal utility: patient values and

preference6. Patient-reported outcomes (QoL)

Value for money (local setting)

7. Budget impact (Net cost)8. Costs avoided /increased hospital efficiency9. No. of patients affected by technology10. Budget impact on other services11. Cost-effectiveness

Feasibility (impact on local setting)

12. Availability of local expertise13. Disruptiveness14. Need to generate local evidence15. Ability to increase cross-institution

collaboration16. Personnel satisfaction17. Impact of innovativeness of the technology

Impact on healthcare system

18. Benefit to society (reduces health care costs)

19. Burden on other health care centres20. Need: unnecessary duplication

Ethical considerations: Disruption of access to care

New version19

Rating the strength of each criterion 20DOMAINS Sub-domains RATING, COMPARED TO ALTERNATIVES Clinical benefit

Magnitude of effectiveness Strong effect Moderate effect Non-inferiorQuality of the evidence Strong Moderate WeakSafety of the technology Strong effect Moderate effect Non-inferior /Concern

for harmImpact on Patient

Patient perception and preferences/personal utility

Superior Equivalent Inferior

Impact on patient convenience (pain, side effects, time, invasiveness)


Pt-reported outcomes( satisfaction, impact on QoL/reduction in period of disability)


Value for money

Budget impact (net cost) Low Equivalent HighCosts avoided High Equivalent LowNumber of patients affected Low Equivalent HighImpact on budget of other departments Low Equivalent HighCost-effectiveness, if available High Equivalent Low

Feasibility Availability of local expertise Yes NoDisruptiveness of other services Low Equivalent HighNeed to generate local evidence Yes NoCross-institution collaboration Yes NoSatisfaction of personnel High Equivalent LowInnovativeness of the technology High Equivalent

Impact on health system

Benefit of the technology to society Yes NoBurden on other healthcare services Low Equivalent HighNeed for the technology Yes No

Ethical considerations

Disruption of access to care Low High

An example: IORT for early breast cancer 21Decision domain Judgment Reason for judgment

Clinical

benefit

Magnitude of effectiveness Inconclusive Recurrence rate: 3.3% vs. 1.3%. No CIs provided to judge non-inferiority criterion.

Quality of the evidence Low single RCT with serious concerns in the analysis and presentation of resultsSafety profile of intervention Equivalent to alternative Complication rate: 17% in IORT vs 15.5% in EBRT;

Patient

impact

Patient personal utility Superior Impact on patient convenience Superior Would reduce inconvenience associated with weekly external beam radiationPt-reported outcomes Superior

Value for

money

Net cost (Budget impact) Moderate If 15 IORT procedures are performed annually instead of EBRT, it would result in an increase of $11500 in the budget

Costs avoided Balanced Balanced because IORT increases OR time (takes an additional 40-60 minutes after surgery), but may result in decrease in wait times for EBRT

Number of patients affected Moderate 16 women have been treated with Intrabeam® since November 2013Budget impact on other services

Cost-effectiveness Feasib

ilityAvailability of local expertise YesDisruptiveness of other servicesNeed to generate local evidence

Yes single RCT with several concerns and inconclusive results; a field evaluation may help answer concerns about recurrence rates, and risk

Cross-institution collaboration Satisfaction of personnel High MUHC radiation oncologists report good experience with using Intrabeam Innovativeness of the technology

High

Impact on

health syste

m

Benefit of the technology to society

Yes

Burden on other health services

Need for the technology Yes Ethical concerns None Overall assessment Moderate: Approval for

evaluationIntrabeam has the potential of reducing unnecessary exposure to radiation, and the burden on women of repeated EBRT visits.However, the procedure remains experimental, and without longer term results on local breast cancer recurrences and safety, this technology can only be approved within the context of a research study at the MUHC

22 Future directions

Pilot test the checklist and ratings among members of the MUHC TAU policy committee: hospital administrators, patient representatives, and quality and risk management representatives

solicit their input on the practicality and usability of the checklist Based on feedback, the checklist will be updated and re-tested at a

second meeting, with the eventual goal of pilot testing the checklist at other hospital-based technology assessment units.

23 Conclusions

Explicit frameworks directed at hospital-based HTAs do not exist Our framework would

help in creating a systematic and standardized evaluation process by ensuring all relevant domains are given due consideration

document the decision-making process in an explicit fashion enable an understanding of the value judgments and reasoning behind

reaching a recommendation

Final list of domainsDOMAINS

Clinical benefit Magnitude of effectiveness, all clinically important outcomesQuality of the evidence (including consistency of magnitude and direction of effect between studies)Safety of the technology

Impact on Patient

Impact on convenience (invasiveness, pain, time)Patient personal utility: Patient perception and preferencesPatient-reported outcomes (QoL)

Value for money (Impact on the local setting)

Net cost (includes acquisition, maintenance, procedure and training costs, and accounts for external support) Costs avoided, measured as increased hospital efficiencyImpact on budget of other departmentsNumber of patients affected by the technology (budget impact)Cost-effectiveness, if available

Feasibility (Impact on the local setting)

Availability of local expertise (clinical, technical) in the technologyImpact on resources of affected department and/or of services provided by other departments (disruptiveness)Need to generate evidence of effectiveness in the local setting (due to lack of strong evidence from RCTs)Ability to increase cross-institution collaboration (through case-sharing or research activities)Satisfaction of personnel involved with or affected by the technologyImpact of innovativeness of the technology (increase attractiveness of hospital to patients /professionals)

Impact on healthcare system /society

Benefit of the technology to society (e.g. technology enables patient to return to work faster, or reduces use of antibiotics, resulting in cost-savings to healthcare system) Burden on other healthcare centres: transfers to other centres/ increase in home monitoringNeed for the technology: does it support local innovation and economic growth, or is it unnecessary duplication of services that increases healthcare costs?

Ethical considerations

Disruption of access to care by introduction of technology; or by refusal to introduce technology because of cost alone, when no alternatives available

26 Burden of illnessRating Frequency0 (Don't know) 01 (Not at all important) 02 (Not very important) 3.83 (Somewhat important) 19.24 (Important) 67.35 (Indispensable) 9.6% rating 4 or 5 76.9

Comments Mostly important on a national

level Important but can be very

difficult to capture/compare In theory this information should

be captured by a cost-effectiveness metric. A disease with a low burden of illness should result in a smaller gain in quality-adjusted life years and a large ICER.

burden of illness is controversial...not sure it belongs in clinical benefit

important factor but would have like to had the prevalence of the problem for patient

27Survey domains

Question: Which of the following domains do you think are most important for consideration by stakeholders when framing recommendations for use of a health technology in a hospital setting?

Domain Sub-domainClinical benefit •Magnitude of effectiveness

•Quality of evidence for effectiveness• Safety•Burden of illness to patient•Absence of alternative treatment options

Value for money •Costs related to the technology•Increased hospital efficiency•External financial support

Feasibility •Ease of implementation•Prior hospital experience with the technology•Need for evidence of effectiveness in the local setting

Ethics and values•Disease is a public health priority• Disease is rare•Benefit of technology to society•Impact on delivery of equitable care• Impact on patient-important outcomes


•Impact of technology on attracting new patients and/or health professionals•Impact on creating research opportunities and external collaborations•Ability to offer a cutting edge technology or new alternative treatment• Availability of the technology in other local centres

28

DOMAIN SUB-DOMAIN % OF PARTICIPANTS RATING THE DOMAINS AS IMPORTANT OR INDISPENSABLE (A RATING OF 4 OR 5)

MUHC MEMBER (N=38)

MUHC NON-MEMBER (N=14)

HTA (N=16)

ADMINISTRATOR (N=12)

TOTAL (N=52)

CLINICAL BENEFIT

Magnitude of effectiveness 100 100 100 100 100Quality of the evidence for effectiveness 92 100 100 92 94Safety of the technology 97 100 100 92 98Burden of illness to patient 79 71 75 83 77Absence of alternatives 50 79 81 50 58

VALUE FOR MONEY

Cost related to the technology 89 100 100 92 92Increased hospital efficiency 97 79 88 92 92External financial support for the technology 21 21 25 17 21

FEASIBILITY

Ease of implementation 79 79 81 75 79Prior hospital experience with the technology

37 29 25 42 35

Need for evidence of effectiveness in the local setting

61 64 69 75 62

ETHICS AND VALUES

Disease is a public health priority 74 57 56 75 69Disease is rare 42 57 44 33 46Benefit of technology to society 82 50 56 75 73Impact on delivery of equitable care 84 64 69 92 79Impact on patient-important outcomes 92 86 88 100 90

STRATEGIC CONSIDERATION


39 29 31 33 37

Impact on creating research opportunities 53 50 44 50 52Ability to offer a cutting-edge technology 74 57 44 83 69Availability of the technology in other local centres

63 64 69 58 63

d1 nisha almeida

Healthcare