good research practices task force - ispor

Statistical Analysis of

Discrete-Choice Experiments:

Discussion of the Report

ISPOR Conjoint Analysis

Good Research Practices

Task Force

Moderator

2

Maarten J. IJzerman, PhD Professor of Clinical Epidemiology & Health Technology Assessment,

Dean of Health & Biomedical Technology, School of Science & Technology

University of Twente, Enschede, the Netherlands

Chair: A. Brett Hauber, PhD, Senior Economist & Vice President, Health Preference

Assessment, RTI Health Solutions, Research Triangle Park, NC USA

Juan Marcos Gonzalez, PhD, Senior Research Economist, Health Preference

Assessment, RTI Health Solutions, Research Triangle Park, NC, USA

Karin G.M. Groothuis-Oudshoorn, PhD, Assistant Professor, Health Technology and

Services Research, University of Twente, Enschede, Netherlands

Thomas Prior, PhD Candidate, Department of Biostatistics,

Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

Deborah A. Marshall, PhD, MHSA, Canada Research Chair, Health Services and

Systems Research; University of Calgary, Calgary, AB, Canada

Charles Cunningham, PhD, Professor, Department of Psychiatry and Behavioural

Neuroscience, McMaster University, Hamilton, Ontario, Canada

Maarten J. IJzerman, PhD, Professor of Clinical Epidemiology & Health Technology

Assessment (HTA) and Dean, University of Twente, Enschede, the Netherlands

John F. P. Bridges, PhD Associate Professor, Johns Hopkins Bloomberg School of

Public Health, Baltimore, MD, USA

Task Force Members

3

Task Force Background

This is the third ISPOR Conjoint Analysis Task Force Report.

Conjoint Analysis Use in Health Studies - a Checklist: A Report of the ISPOR Conjoint Analysis in Health Good Research Practices Task Force (the Checklist) (Bridges et al., 2011).

Downloads as of 9 November 2016: 7,311

Constructing Experimental Designs for Discrete-Choice Experiments: Report of the ISPOR Conjoint Analysis Experimental Design Task Force (Johnson et al., 2013).

Downloads as of 9 November 2016: 7,477

4

Task Force Background

Increasing number of researchers conducting conjoint-analysis studies

Diverse backgrounds, potential lack of basic training in the theoretical underpinnings of conjoint analysis and the statistical approaches to analyze DCE data

Published studies based on statistical analyses not appropriate for the data generated by the conjoint questions

Software used for analyses without understanding properties of statistical models – therefore, unable to evaluate strengths and limitations of the statistical analyses

5

Overview of the forum

Present a primer on relevant statistical methods

including multinomial/conditional logit, random-

parameters logit, hierarchical Bayes, and latent-class

analysis;

Have a discussion of good research practices for

statistical analysis (including the ESTIMATE

checklist)

Discuss case studies on how these methods can be

applied to discrete-choice experiments and other

stated-preference methods.

6

Speaker

Juan Marcos Gonzalez, PhD Senior Research Economist, Health Preference Assessment,

RTI Health Solutions, Research Triangle Park, NC, USA

7

Role of Analysis

Analysis is to infer the strength of preference for each attribute

and attribute level.

Estimates are referred to as preference weights or part-worth

utilities.

Preference weights are estimated on a common scale, and allow

calculation of ratios representing trade-offs people are willing to

make. For instance:

– Monetary equivalence, i.e. willingness to pay

– Risk equivalence, i.e. maximum acceptable risk (MAR)

– Time equivalence

In contrast to use in marketing predicting choice, CA studies in

healthcare are mostly used to estimate preference for attributes.

8

Choices Made and Structure

The primary objective is to provide an educational

resource for readers from wide range of disciplines.

Task force authors decided to structure the paper with a

simple example to explain the concepts for analyzing

choice data.

An archetypal case is introduced with the intention to

demonstrate different analytic approaches and to contrast

their main differences.

9

Assumptions and Remarks

Assumptions and choices made sometimes oversimplify the challenges of statistical modeling.

For instance, “simple regression analysis” is introduced for educational purposes but not recommended in practice.

The report concludes with an overview of some of the current discussions on the topic.

Readers are referred to the advanced statistical methods for more details about statistical analysis.

10

Attributes Levels

A1 Efficacy L1 10 (best level)

L2 5 (middle level)

L3 3 (worst level)

A2 Side effect L1 Mild

L2 Moderate

L3 Severe

A3 Mode of

administration

L1 1 tablet once a day

L2 Subcutaneous injection once a week

L3 Intravenous infusion once a month

Attributes and Levels in the

Archetypal Case

11

Example of a Choice Task

for the Archetypal Case

Feature Medicine A Medicine B

Efficacy

10 on a scale from 1

to 10

where 10 it the best

5 on a scale from 1 to

10

where 10 is the best

Severity of

side effects Severe Mild

How you

take the

medicine

Subcutaneous

injection

once a week

Intravenous infusion

once a month

Which

medicine

would you

choose?

12

Dummy Versus Effects Coding

Archetypal Case

Attribute level

presented in the

profile

Dummy-variables Effects-coded variables

L1 L2 L1 L2

L1 1 0 1 0

L2 0 1 0 1

L3 0 0 -1 -1

13

Dummy Versus Effects Coding

0

1

2

3

4

5

6

7

8

9

Level1

Level2

Level3

Level1

Level2

Level3

Level1

Level2

Level3

Attribute 1 Attribute 2 Attribute 3

Pre

fere

nce W

eig

hts

Attributes and Levels

Dummy coded

Mean effects

-5

-4

-3

-2

-1

0

1

2

3

4

Level1

Level2

Level3

Level1

Level2

Level3

Level1

Level2

Level3

Attribute 1 Attribute 2 Attribute 3

Pre

fere

nce W

eig

hts

Attributes and Levels

Effects coded

• Estimate coefficients relative to mean

attribute effect

• Omitted category is the negative sum of

the coefficients of the non-omitted levels

• Tests of significance in output typically are

not direct tests on differences between

estimated coefficients

• Coefficients represent a measure of preference for

levels of an attribute relative to the omitted level of

that attribute

• Test of significance for a coefficient reflects that this

level is significantly different from the reference

level

• If aim is to obtain odds ratios then dummy coding is

preferred. In case of effects coding, exponentiation

of the estimated coefficients yields the ratio of the

odds for the particular attribute level to the

geometric mean of the odds. 14

Common Models Resulting from

Different Choice Probabilities

Linear probability

Conditional logit

Random-parameters logit

Hierarchical Bayes

Latent-class finite-mixture logit

15

Linear Probability Model

Conventional regression model (Ordinary Least Squares

[OLS]) linking choice to differences in each attribute

Assumes the chance that something is selected is

linearly determined by the characteristics of the

alternatives considered

Pr(𝑐ℎ𝑜𝑖𝑐𝑒) = 𝛽𝑜 + 𝛽𝑖𝑋𝑖𝑖

16

Advantages & Limitations

Linear Probability Model

Advantages Limitations

• Can be used in small samples

• Can be used to identify non-trading

at the individual level

• Is available in most statistical

software packages and even in

some non-statistical software

• Does not appropriately account for

repeated observations from

respondents

• Can produce negative or greater

than one predictions of probabilities

• No more than 2 alternatives in a

single choice occasion is possible

17

Conditional Logit Model

Regression model that acknowledges the discrete nature of

choice as dependent variable

Based on random utility theory (Mc Fadden, 1974)

Ui = V(Xi β) + i

i independently, identically distributed type 1 extreme

value distribution.

Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖 =𝑒𝑉 𝛽,𝑥𝑖

𝑒𝑉 𝛽,𝑥𝑗𝑗

18

Results Conditional Logit

Attribute Level Coefficient

Std. Error t-value p-value

Efficacy L1 0.26 0.01 24.89 <0.01

L2 0.02 0.01 2.24 0.03

Side Effect L1 0.32 0.01 30.18 <0.01

L2 0.02 0.01 2.02 0.04

Mode of

Administration

L1 0.03 0.01 2.49 0.01

L2 0.18 0.01 17.61 <0.01

Log Likelihood -17388

Log likelihood of

model without

predictors

-18715

AIC 34788

BIC 34841 19


Conditional Logit Model


• Focuses on average preferences

• Parsimonious estimator with unique

solution

• Commonly available in software

packages

• Requires the smallest sample size

from all the models in this table

• Assumes homogeneity in

preferences

• Does not accounting for panel

nature of the data

• It is not guaranteed to converge

without large enough samples or

with lack of variability in the

response variable

20

Effects-Coded Results

21

Dummy-Coded Results

22

Random-Parameters Logit

Expands conditional logit model to account for across-

respondents variations in preferences

When is assumed normally distributed with mean and

standard deviation

, for across n respondents in the sample

23

Mean Estimates

Attribute Level Coefficient Std. Error T-value P-value

Efficacy L1 0.31 0.01 20.92 <0.01

L2 0.03 0.01 2.43 0.02

Side Effect L1 0.38 0.02 24.22 <0.01

L2 0.02 0.01 1.72 0.09

Mode of

Administration

L1 0.03 0.01 2.68 0.01

L2 0.22 0.02 12.13 <0.01

Standard Deviation Estimates


Efficacy L1 0.31 0.02 19.98 <0.01

L2 0.14 0.02 6.48 <0.01

Side Effect L1 0.34 0.02 21.37 <0.01

L2 0.24 0.02 14.69 <0.01

Mode of

Administration

L1 -0.05 0.03 -1.66 0.10

L2 0.48 0.02 27.72 <0.01

Log Likelihood of

model -16672.7

Log likelihood of

model without

predictors

-17387.6

AIC 33369.4

BIC 33476.2 24


Random-Parameters Logit Model


• Models heterogeneity

• Accounts for the panel nature of the

data

• Becoming more available in

software packages

• Can deal with scale heterogeneity

with the use of model options (allow

correlation of preference

heterogeneity across attributes)

• More difficult to use than conditional

logit

• Requires assumptions about the

distribution of parameters across

respondents

• Requires larger sample sizes than

conditional logit models

25

Hierachical Bayes Model

Generates preference estimates for each individual

Underlying model is multinomial logit

Sample density model: 𝛽𝑛~𝑁(𝑏,𝑊) (or other distributions

like lognormal, triangular, uniform)

Likelihood level / lower level/ individual choices:

Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖 =𝑒𝑉 𝛽𝑛,𝑥𝑖

𝑒𝑉 𝛽𝑛,𝑥𝑗

𝑗

Gibbs sampler: iteratively estimates b, W, 𝛽𝑛

26

Mean Standard Deviation

Attribute Level RPL HB RPL HB

Efficacy L1 0.31 0.35 0.31 0.43

L2 0.03 0.03 0.14 0.33

Side Effects L1 0.38 0.43 0.34 0.50

L2 0.02 0.03 0.24 0.42

Mode of

administration

L1 0.03 0.03 0.05 0.36

L2 0.22 0.26 0.48 0.66

Comparison of Results of

Different Models

27


Hierarchical Bayes Model


• Quicker convergence

• Models individual preferences

• Requires fewer respondents to

construct the sample mean

preferences

• Deals with scale heterogeneity

without requiring model options

(individual results allow correlation

of preferences across attributes)

• Not available in many software

packages

• May require more choices per

respondent to obtain individual

preference estimates

• Results may be difficult to explain

• Inferences cannot be made at the

respondent level

28

Latent-Class Finite-Mixture

Logit Model

Assumes that attributes of the alternatives have

heterogeneous effects on choices across a finite number

of groups or classes of respondents

Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖|𝑥𝑖 , 𝑥𝑗 = Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖|𝑥𝑖 , 𝑥𝑗; 𝛽𝑞𝑞

𝜋𝑞

𝜋𝑞 are the class probabilities

Within a class: Pr 𝑐ℎ𝑜𝑖𝑐𝑒 = 𝑖|𝑥𝑖 , 𝑥𝑗; 𝛽𝑞 =𝑒𝑉 𝑥𝑖;𝛽𝑞

𝑒𝑉 𝑥𝑗;𝛽𝑞

𝑗

.

29

Class 1


Efficacy L1 0.29 0.02 11.70 <0.01

L2 0.04 0.02 2.01 0.04

Side Effect L1 0.39 0.03 13.28 <0.01

L2 -0.02 0.02 -1.06 0.29

Mode of

Administration

L1 0.21 0.02 9.38 <0.01

L2 -0.33 0.04 -9.25 <0.01

Class 2


Efficacy L1 0.27 0.02 14.37 <0.01

L2 0.01 0.02 0.80 0.42

Mode of

Administration L1

0.30 0.02 14.20 <0.01

L2 0.05 0.02 3.09 0.00

Mode of

Administration L1

-0.09 0.02 -5.40 <0.01

L2 0.54 0.03 19.60 <0.01

Class probability

function

Constan

t -0.41 0.13 -3.05 <0.01

Log Likelihood of

model -16985

Log likelihood of

model without

predictors

-18714

AIC 33996

BIC 34102 30


Latent-Class Logit Model

PRO CON

• Models latent classes

• Describes heterogeneity by class

• Parsimonious estimator with a

unique solution

• Requires smaller samples than RPL

and HB models

• Requires specialized software

• Judgment required to determine

appropriate number of classes to be

estimated

• Difficult to interpret results from any

given class when the chance of

being in all classes is more or less

the same across respondents

• The required sample size varies

with the number of classes in the

model

31

Speaker

Deborah A. Marshall, PhD, MHSA Canada Research Chair, Health Services and Systems Research;

Professor, Department of Community Health Sciences,

University of Calgary, Calgary, AB, Canada

32

Principles for Good Research

Practice In Analysis

33

The ESTIMATE

Checklist

Estimates

Stochastic

Tradeoffs

Interpretation

Method

Assumptions

Transparent

Evaluation

Estimates

Describe the choice of parameter estimates

resulting from the model appropriately and

completely, including:

Whether each variable corresponds to an effects-

coded level, a dummy-coded level, or a

continuous change in levels

Whether each variable corresponds to a main

effect or interaction effect

Whether continuous variables are linear or have

an alternative functional form

34

Stochastic

Describe the stochastic properties of the analysis,

including:

The statistical distributions of the parameter

estimates

The distribution of parameter estimates across

the sample (preference heterogeneity)

The variance of the estimation function, including

systematic differences invariance across

observations (scale heterogeneity)

35

Tradeoffs

Describe the tradeoffs that can be inferred from the

model, including:

The magnitude and direction of the attribute-level

coefficients

The relative importance of each attribute over the

range of levels included in the experiment

The rate at which respondents are willing to trade

off among the attributes (marginal rate of

substitution)

36

Interpretation

Provide interpretation of the results taking into

account the properties of the statistical model,

including:

Conclusions that can be drawn directly from the

results

Applicability of the sample, including subgroups

or segments, to the population of interest

Limitations of the results

37

Method

Describe the reasons for selecting the statistical

analysis method used in the analysis, including:

Why the method is appropriate for analyzing the

data generated by the experiment

Why the method is appropriate for addressing the

underlying research question

Why the method was selected over alternative

methods

38

Assumptions

Describe the assumptions of the model and the

implications of the assumptions for interpreting the

results, including:

Assumptions about the error distribution

Assumptions about the independence of

observations

Assumptions about the functional form of the

value function

39

Transparent

Describe the study in a sufficiently transparent way

to warrant replication, including descriptions of:

The data setup, including handling missing data

The estimation function, including the value

function and the statistical analysis method

The software used for estimation

40

Evaluation

Provide an evaluation of the appropriateness of the

statistical analysis method to answering the research

question, including:

The goodness of fit of the model

Sensitivity analysis of the model specification

Consistency of results estimated using different

methods

41

Limitations

Focus is on traditional DCE, and there are

modifications (e.g. opt-out) and other choice formats

(e.g. BWS, threshold techniques) that are not

covered.

Emerging and specialized statistical methods are not

described in the report.

No formal evaluation of the extent to which each

method is being used.

No rating of the importance of different properties of

different models.

42

Speaker

John F. P. Bridges, PhD @jfpbridges

Associate Professor,

Johns Hopkins Bloomberg School of Public Health,

Baltimore, MD, USA

43

Case Studies on Research Practices

The work presented here is based on results of a randomized

control trial on preferences in patients with type 2 diabetes

• Posters on this study will be presented at poster session IV

» B27: Treatment preferences of patients with type 2 diabetes in the united states: an application of good research principles for discrete choice experiments

» B30: Developing a stated-preference instrument to assess the barriers and facilitators to the self-management of type 2 diabetes

• Funded by The Patient-Centered Outcomes Research Institute (PCORI) Methods Program Award (ME-1303-5946) and the the Johns Hopkins Center of Excellence in Regulatory Science and Innovation and the Food and Drug Administration (UO1FD004977).

44

Overview

1. Prioritization methods

– Comparing Likert and BWS data

2. Preference methods

– Benefits of Mixed Logit for DCE and BWS case 2

3. Using latent classes

– Determining the number of latent classes

4. Advances in presenting preference results

45

1. Prioritization Methods:

Likert v BWS case 1 (rho>0.9)

46

-0.400

-0.300

-0.200

-0.100

0.000

0.100

0.200

0.300

0.400

0.500

0.600

Sc

ore

LikertScale

BWS

1. Prioritization Methods:

Rescaling Likert Data

47

-3.15

-3.06

-2.86

-2.79

-2.16

-0.53

0.20

0.35

1.12

1.39

4.05

4.62

5.68

-4.00 -2.00 0.00 2.00 4.00 6.00

Capacity to manage

Other health conditions

Faith and religious practices

Places to exercise

Time commitments

Communication

Healthy food

Current health insurance

My family commitments

Resources in local community

My language and culture

Active support group

Personal understanding

-3.15

-3.06

-2.86

-2.79

-2.16

-0.53

0.20

0.35

1.12

1.39

4.05

4.62

5.68

-4.00 -2.00 0.00 2.00 4.00 6.00

Capacity to manage



Places to exercise

Time commitments

Communication

Healthy food







-3.15

-3.06

-2.86

-2.79

-2.16

-0.53

0.20

0.35

1.12

1.39

4.05

4.62

5.68

-4.00 -2.00 0.00 2.00 4.00 6.00

Capacity to manage



Places to exercise

Time commitments

Communication

Healthy food
















10.13

9.53

8.84

4.77

4.63

3.48

3.40

3.03

1.28

0.39

0.26

0.25

0.18

0.00 2.00 4.00 6.00 8.00 10.00 12.00

Regular access to healthy food

My personal understanding of diabetes

My capacity to manage my diabetes

My current health insurance

Other health conditions that I have

Access to convenient places to exercise


My faith and religious practicses

Access to an active support group


Stress about time commitments

Resources in my local community

Theoretical mean

5.68

4.81

4.62

1.44

1.27

0.35

0.19

-0.38

-1.70

-3.39

-2.69

-3.72

-2.96

-4.00 1.00 6.00

Observed mean

2. Preference Heterogeneity:

Mixed Logit Individual Parameters

48


Correlation for Mixlogit and Clogit

49

y = 0.7434x + 3E-18

y = 0.9712x - 1E-17

-1.500

-1.000

-0.500

0.000

0.500

1.000

1.500

-1.500 -1.000 -0.500 0.000 0.500 1.000 1.500

Clo

git

Mixlogit

DCE

BWS


Number of Latent Classes

2 classes 3 classes 4 classes 5 classes 6 classes 7 classes

Obs 17508 17508 17508 17508 17508 17508

ll(model) -4700.229 -4498.228 -4339.453 -4204.632 -4167.422 -4139.929

df 0 0 0 0 0 0

AIC 9400.458 8996.457 8678.906 8409.263 8334.845 8279.859

BIC 9400.458 8996.457 8678.906 8409.263 8334.845 8279.859

50

Model fit of different latent class models


Number of Latent Classes

51

Variable Class

1 Class

2 Class

3 Class

4 Class

5 Class

6

A1c decrease 0.77 0.24 0.47 0.87 3.52 0.28

Stable blood glucose 0.67 0.70 0.45 2.53 0.67 0.23

Low blood glucose 0.75 0.57 0.53 0.72 0.49 0.52

Nausea 1.03 1.39 3.55 0.76 1.54 0.31

Treatment burden 0.79 3.34 0.71 0.49 0.30 0.33

Cost 3.53 1.40 0.16 0.23 0.96 0.28

Relative attribute importance in 6 class latent class model

4. Presenting Preference Results

52

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

1%

0.5

0%

0%

6 d

ays/

wee

k

4 d

ays/

wee

k

2 d

ays/

wee

k

No

ne

Day

Day

an

d/o

r n

igh

t

No

ne

30

min

ute

s

90

min

ute

s

1 p

ill

2 p

ills

1 p

ill a

nd

1 in

ject

ion

$1

0

$3

0

$5

0

A1cdecrease

Stable bloodglucose

Low bloodglucose

Nausea Treatmentburden

Out-of-pocketcost

Mea

n p

refe

ren

ce w

eigh

ts w

ith

95

% C

on

fid

ence

In

terv

al

Traditional preference chart


53

-1.0

-0.5

0.0

0.5

1.0

1%

0.5

0%

0%

6 d

ays/

wee

k

4 d

ays/

wee

k

2 d

ays/

wee

k

No

ne

Day

Day

an

d/o

r n

igh

t

No

ne

30

min

ute

s

90

min

ute

s

1 p

ill

2 p

ills

1 p

ill a

nd

1 in

ject

ion

$1

0

$3

0

$5

0

A1cdecrease

Stable bloodglucose

Low bloodglucose


Out-of-pocketcost

Pre

fere

nce

wei

ghts

Shifted bar graph with line graph


54

-2.2-1.8-1.4-1.0-0.6-0.20.20.61.01.41.82.2

1%

0.5

0%

0%

6 d

ays/

wee

k

4 d

ays/

wee

k

2 d

ays/

wee

k

No

ne

Day

Day

an

d/o

r n

igh

t

No

ne

30

min

ute

s

90

min

ute

s

1 p

ill

2 p

ills

1 p

ill a

nd

1 in

ject

ion

$1

0

$3

0

$5

0

A1cdecrease

Stable bloodglucose

Low bloodglucose


Out-of-pocketcost

Mea

n p

refe

ren

ce w

eigh

ts Box plot to indicate heterogeneity

Moderator

55

Maarten J. IJzerman, PhD Professor of Clinical Epidemiology & Health Technology Assessment,

Dean of Health & Biomedical Technology, School of Science & Technology

University of Twente, Enschede, the Netherlands

FORUM

1. Go to the ISPOR homepage:

www.ispor.org .

2. Click on the GREEN TASK FORCE menu

at the TOP of the homepage

3. Select JOIN on the pull-down menu.

Join the Conjoint Analysis Task Force

Review Group

56

http://www.ispor.org/

Forum Slides

57

FORUM SLIDES are AVAILABLE!

http://www.ispor.org/Event/ReleasedPresentations/2016Washington

Questions?

58

good research practices task force - ispor

Documents