The Use and Evaluation of Experiments in Health Care DeliveryAmanda KowalskiAssociate Professor of EconomicsDepartment of Economics, Yale University

September 26, 2015

What are the questions that YOU want to answer?

How can I help you to answer them?

A Call to Action

http://www.sciencemag.org/content/347/6223/720.full

Heard in the Trenches:Barriers to Randomization

“Why randomize?”

Way to mitigate influence of confounding factors If you just compare treated group to a non-treated

group, there could be other factors that produce different outcomes

If you just compare treated group after intervention to treated group before intervention, there could be other factors that changed over time

“Can’t you just analyze data collected after the intervention?” It IS important to examine existing

administrative data

Hard to learn about causality if you only have data on the treatment group after the intervention

“Is randomization fair?”We only randomize if we don’t know if the

intervention will work – “equipoise”

Randomization standard in clinical trials for medical interventions

When resources are scarce, randomization can be a fair way to allocate them

“Is randomization fair?”Example: Oregon Health Insurance experimentSome funds to expand Medicaid coverage, but

not enough to expand coverage to all interested parties

Held a lottery in 2008

“Is randomization fair?”Example: Oregon Health Insurance experiment

“Is randomization fair?”Example: Oregon Health Insurance experimentKey findings:

Increased health care utilization Emergency room utilization increased

Decrease in depression No changes to physical health

Reduced financial strain No discernable impact on labor market outcomes

“But some large-scale questions can’t be studied with randomization!”For example, some initiatives like bundled

payments might have larger impacts if they are implemented more broadly

Could also randomize fraction of population affected across different sites

“I’m too busy implementing this initiative to think about anything else!”Randomization can be a seamless part of

implementation

After implementation is too late

“I’m afraid to find out that what I have done does not work!”Health care industry full of altruistic people

who want to improve quality, increase access, and decrease cost

If your intervention does not work, can try something else next

If your intervention does work, evidence indicating so can be useful to others, broadening the impact of your work

“Will the implementation be costly?” Clinical trials often costly

Subject recruitment Informed consent

Randomized experiments in health care delivery Subjects already in system Consent waived

Costs of designing experiment

Costs of implementing randomization

Costs of collecting data – could focus on existing

Costs of analyzing data

“How will results be disseminated?”Results will be published regardless of outcome

Institutional partner can opt for anonymity before publication

“When is a good time to get started?”Before implementing a new intervention

Baseline data can be collected Enriches comparison of treatment to control Ensures that outcomes can be measured

Program probably not rolled out to everyone at the same time anyway

“Which interventions are best studied with randomization?”Potential for large, detectable outcomes

But outcomes are not known

Large potential number of subjects Increases statistical power

Interventions that would be implemented anyway Increases real-world applicability Successful implementation paves the way for

future randomization of other initiatives

“What are the advantages of partnering with an economist?”Statistical techniques

More subtle than comparing treatment group to control group

Dissemination Potential to reach different audience

Results prepared by an independent entity potentially more impartial

“I’m on board with randomization. What’s next?”What is the problem to be addressed?

What administrative data are available and how can they be accessed?

Who will be the implementing partners? C-level advocate to push project through Administrative contact for day-to-day

Let’s talk further!

Let’s talk further!http://www.econ.yale.edu/~ak669/

research.html

Slides on my website, video will be posted

…Tell your friends to examine slides, watch this presentation, propose an idea, and contact me!

What are the questions that YOU want to answer?