doing more with less - thinkrga.com · ©2015 roger green + associates, inc. doing more with less...

©2015 Roger Green + Associates, Inc.

Doing More With Less Utilizing Small Sample Analytics to Get the

Most Out of Your Research Budget


Top Three Take-Aways

Small sample sizes can produce projectable and actionable insights

In a small sample setting, integrating both quantitative and qualitative methods can enhance your research ROI

Used properly, certain small sample methods can produce insights that rival larger sample studies

Doing More With Less


Forecasting an Orphan Drug

Scenario #1


Scenario

There are approximately 5,000 Hereditary Angioedema (HAE) patients in the U.S.

New drug has been developed that treats “moderate” patients (in terms of acute attack severity and frequency)

4


• Forecast peak annual number of doses

• Support forecast with information on which patients will / will not get product and provide information on why

• Understand how product compares with existing therapies and what drives these perceptions

Research Goals


Researcher’s Dilemma

Need to understand suggests qualitative research

• Unstructured discussion flow allows probing to enhance understanding

• Give “flavor” to why physicians make their decisions

Forecast requirement suggests quantitative research

• Representative sample

• Precision for share estimate

5



Qualitative research with ER physicians • 30 phone-to-web interviews

• Recruited from ER physicians across U.S. who have experience treating HAE

Review a series of charts describing

patients who present with HAE

symptoms

Review and discusses

product profile

Review additional patient profiles and

makes treatment decisions with new product as option

Discusses decisions

Approach That Solves Dilemma

6


Charts include all pertinent information,

e.g. frequency and severity of acute attacks

ER Physician Interview Flow


Sources of Patients for MDs to Treat

7


•MD selects records according to specification

•Difficult to get specific information because of selection inconsistencies across physicians

Bring Your Own

•Researcher creates set of profiles to represent key segments

•Provides usage information by segment Archetypes

•Researcher creates profiles by combining patient characteristics using experimental design

•Experimental design allows assessment of impact by patient characteristic

Experimental Design


Type of Patient Should be Driven by Information Needs

8


Advantages

• Face validity because real patients used

Disadvantages

• Additional cost for physician chart pull

• Limits on number that physician will bring

• Identification of key characteristics comes only from interview

MD Brings Patient Records



9


Advantages

• Better control / representation of segments / market

Disadvantages

• Identification of key characteristics comes only from interview

• Potential validity issues if profiles do not include necessary information

Archetypes Reflecting Key Patient Segments



10


Advantages

• Better control / representation of segments / market

• Key characteristics derived from prescribing; augmented with qualitative

Disadvantages

• Potential validity issues if profiles do not include necessary information

Patients Created Using Experimental Design


Delivering Actionable Outputs

11


Brand Share

Post-Launch Pre-Launch

Mono / Combination Therapy

Why?

Likely / Unlikely


Are these results representative of the population?

12


Sample representativeness is determined by selection

methods

Our sample is selected from ER physicians who

treat the disease

A small sample can still

represent the target

population

The orphan drug study remains representative

with careful selection of a small sample


Are these results precise?

13


RG+A ran a Monte Carlo simulation to produce the standard error that would result if the same number of physicians had completed a Treatment Event survey seeing 4, 6, or 8 sample patients

Each doctor’s allocated share was assumed to be the probability that they would write a prescription for a simulated patient

RG+A simulated each brand for the sample of doctors 1000 times using a Monte Carlo model

RG+A then compared the standard error for allocation versus simulation and estimated how much more sample would be required for the allocation estimate to be as precise as the simulation estimate


In 95% of cases, treatment event methodology improved precision compared to allocation


95%

5%

Overall Proportion of Cases Where Treatment Events Improve Share

Precision

Improve precision Decrease precision

88%

12%

4 Treatment Events

98%

2%

6 Treatment Events

100%

0%

8 Treatment Events

With X… Treatment Events

14


Considerations and “Watch Outs”

Forecasting results will be more accurate if product profile is followed by competitive response

Discussion sequence can bias results • Avoid topics that could influence acceptance

of new product (i.e. too much focus on needs; deficiencies among current products)

• Finish all patient treatments before probing

15



Understanding Benefit of Potential Features on a New Medical Device

Scenario #2


Scenario

Medical device manufacturer wants to evaluate 4 potential features for new product

• Each feature has 3 alternatives

• Each alternative has a unique development cost, timeline and risk

• Engineering wants the decision to be based on projectable sample

17

Medical Device Feature Optimization

• Understand the benefits provided by features and whether they provide a competitive advantage

Research Goals



Marketing needs information that would typically come from qualitative research

• Give “flavor” to the appeal of certain device features

Engineering information needs suggest some form of quantitative tradeoff research

• Multiple combinations of features

• Projectable sample

18




Conduct conjoint study as part of a qualitative interview • 30 telephone-to-web interviews

• Recruited from national target specifications

19


9 Card Conjoint

Competitive Profile

Discussion of Evaluations & Potential Use



20


Feature 1 Feature 2 Feature 3 Feature 4

Conjoint Part-Worths

Potential Segments

? Identifying Missing Alternatives

Detailed Reasons For Preference


Can We Trust The Results?

21


Feature 1 Feature 2 Feature 3 Feature 4

Conjoint Part-Worths

Projectable conjoint part-worths that show relative preference for alternatives

Sample representativeness based on how it is recruited

Recruiting from national targets means results projectable to this group

Ability to view / compare alternatives enhances reliability of conjoint ratings



22


Detailed Reasons For Preference

Detailed understanding of reasons for respondent preference and impact in marketplace

Qualitative interviews explore evaluations and provide depth



23


Identification of missing thresholds / more important feature alternatives

Qualitative interviews provide opportunity to identify

? Identifying Missing Alternatives



24


Identification of potential segments that represent 25% or more of the population

Sample of 30 should provide 7 to 8 respondents from 25% segment

• 90% probability of at least 4

Comparing conjoint and qualitative results can confirm and profile segments

• Cluster analysis on part-worths can identify groups with distinctly different preferences

• Qualitative interviews should be analyzed for segment differences as well as overall consensus Potential Segments


Considerations and “Watch Outs”

Limit the number of variables / levels in the conjoint

• Ideally fewer than 10 cards

Avoid discussion that could bias conjoint results

Do not discuss conjoint ratings until all cards evaluated

Smaller samples can work if you ignore potential segments

25



Quickly Estimate Potential Revenue of an In-license Opportunity

Scenario #3


Scenario

Opportunity to in-license product to treat chronic constipation • Both PCPs and gastroenterologists treat this condition

Licensing needs to make a decision in 4 weeks

As always, budget is tight

27


• Estimate peak annual revenue

• OTC’s are available so there’s a question of whether target patient will pay tier 3 prices

Research Goals



Qualitative insight would be valuable for understanding and promoting confidence in results

Estimating peak annual revenue suggests quantitative research

• Budget and time constraints severely limit quantitative options

28


Either approach produces confidence intervals too wide for the licensing decision makers to use



Change the objective from “estimate of peak annual revenue” to “likelihood that peak annual revenue will exceed $XXX”

• Licensing should know revenue needed to make acquisition worthwhile

• More specific question lets research provide better information

29


Monte Carlo

• Construct set of equations to produce estimate of peak revenue using variable distributions

Qualitative

• 15 interviews with Gastros and PCPs to determine willingness to Rx

• 20 interviews with consumers dissatisfied with OTC options

Run Model

• Program samples and observations into model

• Calculate results based on at least 1,000 observations


Monte Carlo Simulation Gives More Than ‘Hit or Miss’ Results

30


RETURNS PROBIBALISTIC

RESULTS

70% 30%

RETURNS HIT OR MISS RESULTS

Each input is randomly generated from its distributions.

The model is run +1,000 times to generate all possible outcomes

STANDARD MODEL

MONTE CARLO SIMULATION

A single input, or “arrow”, is used in a standard prediction model

whose inputs are constant

TARGETED GOAL



31


70% 30%

Probability of Hitting Target 0 5 10

Category 1

Peak Annual Revenue Confidence Interval

Variable 1 Variable 2 Variable 3 Variable 4

Key Decision Variables Why MDs will Rx & Why Patients will Fill Rx



32


Probability of Hitting Revenue Target(s)

Accuracy of probability depends on accuracy of inputs

Probability distributions accurately reflect level of knowledge

Approach robust enough to produce good estimates even when relatively little information available

70% 30%

Probability of Hitting Target



33


Estimate of peak annual revenue with an appropriate confidence interval

Estimate reasonably accurate but may be of limited value because of wide confidence interval

Accuracy increased by asking anchoring questions, “wisdom of crowds” estimates and effective probing

0 5 10

Category 1

Peak Annual Revenue Confidence Interval



34


Identification of key variables that determine whether target will be met

Variables identified based on correlation between variable value and result

Stronger correlations indicate input has greater impact on outcomes

Information can be used to determine whether further research is necessary and if so, on which variables

Variable 1 Variable 2 Variable 3 Variable 4

Key Decision Variables



35


Understanding of when, where and why MDs will use product

• Qualitative interviews explore issues and provide depth

Understanding of why consumers will or will not fill a prescription and how they will use the product

• Qualitative interviews explore issues and provide depth Why MDs will Rx &

Why Patients will Fill Rx


Considerations and Watch Outs

Model success typically rests on two key factors

• Minimizing model complexity

• Accurately representing what you do not know

• Distribution specificity

• Ensuring that range between minimum and maximum values captures unknown value

36



Top Three Take-Aways

Small sample sizes can produce projectable and actionable insights

In a small sample setting, integrating both quantitative and qualitative methods can enhance your research ROI

Used properly, certain small sample methods can produce insights that rival larger sample studies

Doing More With Less

doing more with less - thinkrga.com · ©2015 roger green + associates, inc. doing more with less...

Documents