quality of care delivered by international medical …...confidential: for review only 1 quality of...

Confidential: For Review O

nly

Quality of Care Delivered by International Medical

Graduates in U.S. Hospitals: Observational Study

Journal: BMJ

Manuscript ID BMJ.2016.035412

Article Type: Research

BMJ Journal: BMJ

Date Submitted by the Author: 07-Sep-2016

Complete List of Authors: Tsugawa, Yusuke; Harvard T.H. Chan School of Public Health, Department of Health Policy and Management Jena, Anupam; Harvard Medical School, Health Care Policy Orav, E.; Harvard School of Public Health, Biostatistics Jha, Ashish; Harvard School of Public Health, Department of Health Policy

and Management, Harvard School of Public Health, 677 Huntington Avenue

Keywords: health policy, quality of care, outcomes research, medical education

https://mc.manuscriptcentral.com/bmj

BMJ


nly

1

Quality of Care Delivered by International Medical Graduates in U.S. Hospitals:

Observational Study

Yusuke Tsugawa, MD, MPH, PhD

Anupam B. Jena, MD, PhD

E. John Orav, PhD

Ashish K. Jha, MD, MPH

Authors:

Yusuke Tsugawa, MD, MPH, PhD

Research Associate, Department of Health Policy and Management, Harvard T.H. Chan School of

Public Health

42 Church Street 2nd Floor, Cambridge, MA 02138, USA

Anupam B. Jena, MD, PhD

Ruth L. Newhouse Associate Professor, Department of Health Care Policy, Harvard Medical School

Department of Medicine, Massachusetts General Hospital

Page 1 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

2

Faculty Research Fellow, National Bureau of Economic Research

180 Longwood Avenue, Boston, MA 02115, USA

E. John Orav, PhD

Associate Professor

Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School

Department of Biostatistics, Harvard T.H. Chan School of Public Health

1620 Tremont Street 3rd Floor, Boston, MA 02115, USA


K.T. Ki Professor of International Health, Department of Health Policy and Management, Harvard T.H.

Chan School of Public Health

Director, Harvard Global Health Institute

VA Boston Healthcare System

42 Church Street 2nd Floor, Cambridge, MA 02138, USA

Corresponding author:


Page 2 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

3

42 Church St. Cambridge, MA 02138

617-384-5367

[email protected]

Manuscript Word count: 3,076 words

Page 3 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

4

Abstract (299 words; Max 300 words)

Objective: International medical graduates (IMGs) account for one-quarter of physicians in both the

US and UK. While concerns have been raised about the quality of care provided by IMGs across many

high-income countries, there is little data on whether patient outcomes differ between IMGs and

domestic medical graduates.

Design: Observational study.

Setting: Using a 20% national sample of Medicare fee-for-service beneficiaries hospitalized with a

medical condition in 2011-2013, we compared patient outcomes (30-day mortality and readmissions,

and costs of care) between IMGs and US medical graduates (USMGs). We adjusted for patient and

physician characteristics and hospital fixed effects (effectively comparing IMGs and USMGs within

the same hospital). As a sensitivity analysis, we focused on “hospitalists,” whose patients are plausibly

quasi-randomized based on physicians’ work schedules.

Participants: Medicare fee-for-service beneficiaries aged 65 years or older hospitalized with a

medical condition and treated by a general internist in 2011-2013.

Main outcome measures: Patients’ 30-day mortality and readmission rates, and costs of care per

hospitalization.

Results: Our sample included 909,434 hospitalizations treated by 40,592 physicians. Patients treated

by IMGs had, on average, slightly more chronic conditions than patients cared for by USMGs.

Page 4 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

5

However, patients treated by IMGs had lower mortality rates compared to patients of USMGs

(adjusted mortality rate 11.0% vs. 11.7%; adjusted risk difference -0.7%, 95% CI: -0.9% to -0.5%,

p<0.001) and slightly higher costs (adjusted costs $1,134 vs. $1,082; adjusted difference +$52, 95%CI:

+$44 to +$60, p<0.001). There was no difference in readmissions between IMGs and USMGs. Similar

differences in patient outcomes were observed among hospitalist physicians. Differences in patient

mortality were not explained by differences in patient volume, length of stay, spending level, or

discharge location.

Conclusions: Using data on hospitalized patients in US Medicare, we found no evidence that patient

outcomes are worse for IMGs than for USMGs.

Page 5 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

6

What this paper adds

What is already known on this subject

- There are very few studies that have examined patient outcomes between international medical

graduates (IMGs) and physicians trained domestically in high-income countries.

- Small studies from a single U.S. state or a single Canadian province found mixed results as to

whether patient outcomes differ between IMGs and domestically-trained doctors.

- No study has examined differences in patient outcomes between IMGs and U.S. medical graduates

using large-scale U.S. data.

What this study adds

- Among hospitalized Medicare beneficiaries, our study found no evidence that patient outcomes

(30-day mortality and readmissions) were worse for IMGs than for USMGs, despite IMGs caring

for patients with, on average, higher rates of chronic conditions.

- Current standards of selecting IMGs for practice in the US appear sufficiently rigorous to ensure

high quality care.

Page 6 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

7

The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all

authors, an exclusive licence (or non-exclusive for government employees) on a worldwide basis to the

BMJ Publishing Group Ltd to permit this article (if accepted) to be published in BMJ editions and any

other BMJPGL products and sublicences such use and exploit all subsidiary rights, as set out in our

licence.

All authors have completed the ICMJE uniform disclosure form at (available on request from the

corresponding author) and declare: support provided by grants from the Office of the Director,

National Institutes of Health (Dr. Jena, NIH Early Independence Award, Grant 1DP5OD017897-01)

and Social Science Research Council and St. Luke’s International University (Tokyo, Japan; Dr.

Tsugawa); no financial relationships with any organisations that might have an interest in the

submitted work in the previous 3 years. Dr. Jena reports receiving consulting fees from Precision

Health Economics, a health economics consultancy to the life sciences industry.

The research conducted was independent of any involvement from the sponsors of the study. Study

sponsors were not involved in study design, data interpretation, writing, or the decision to submit the

article for publication.

Page 7 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

8

Transparency statement: Dr. Tsugawa affirms that the manuscript is an honest, accurate, and

transparent account of the study being reported; that no important aspects of the study have been

omitted; and that any discrepancies are disclosed.

Authors contributions: All authors contributed to the design and conduct of the study, data collection

and management, analysis interpretation of the data; and preparation, review, or approval of the

manuscript. Dr. Jha supervised the study and is the guarantor.

Data sharing: no additional data available.

Page 8 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

9

INTRODUCTION

More than 1 in 4 practicing physicians in the U.S. as well as in the UK graduated from a medical

school outside the country. 1,2 Although these international medical graduates (IMGs) are required to

pass examinations to practice medicine in these countries, there have been no U.S.- or UK-based

standardized accreditations of foreign medical schools, 3 and consequently, both policymakers and the

public have expressed concerns at various points about the quality of care provided by IMGs. 2,4-6 In

response, in the U.S., the Educational Commission for Foreign Medical Graduates (ECFMG) recently

announced the requirement for all foreign medical schools to be accredited through a formal process

by 2023.1 However, it is unclear whether current licensure processes for IMGs provide adequate

safeguards for ensuring high-quality care, and therefore, it is critically important to understand whether

IMGs are able to provide care that is of comparable quality to that provided by doctors trained

domestically in high-income countries.

While concerns about the quality of care delivered by IMGs are often raised, there is surprisingly

limited data comparing patterns of care and outcomes achieved by IMGs and doctors trained

domestically. The evidence that does exist is inconsistent and has focused on measures such as whether

IMGs have comparable test scores7-10 or are compliant with clinical process measures. 11 We are

unaware of any nationally comprehensive study of patient outcomes – what we arguably care about

Page 9 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

10

most. The studies that have examined patient outcomes have been small in size2-4 or from a single U.S.

state2 3 or a single Canadian province,4 making it difficult to draw conclusions about the quality of care

of IMGs more broadly. Given the substantial public interest in and ongoing concerns about the quality

of care IMGs provide in the U.S., the UK, and other high-income countries5 and policymakers’ efforts

to assure more consistency in foreign medical education, empirical data about how IMGs perform

would be immensely helpful.

In this context, using a national sample of hospitalized elderly patients in the U.S., we sought to

answer three key questions. First, within the same hospital, do patients treated by IMGs have different

mortality rates compared to those cared for by U.S. medical graduates (USMGs)? Second, given

substantial efforts and interest in reducing readmissions and costs of care, do IMGs have higher

readmission rates or provide costlier care than USMGs? Finally, do differences in patient outcomes

and costs of care between IMGs and USMGs, if any, vary by clinical condition?

METHODS

Data

We linked four data sources: the 100% Medicare Inpatient Files (2011-2013), the 20% Medicare

Carrier Files (2011-2013), the American Hospital Association (AHA) annual survey on hospital

Page 10 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

11

characteristics, and a comprehensive physician database collected by Doximity (details of this database,

including its validation6-8, are described in supplementary material 1). We identified Medicare

fee-for-service beneficiaries age 65 years or older who were hospitalized with a medical condition

during January 1, 2011 to December 31, 2013, as defined by the presence of a medical diagnosis

related group (MS-DRG) on hospital admission. We restricted our sample to hospitalizations treated in

acute care hospitals and excluded elective hospitalizations and hospitalizations in which a patient left

against medical advice. To allow sufficient follow-up periods, we excluded patients admitted in

December 2013 from analyses of 30-day mortality and patients discharged in December 2013 from

readmission analyses. For the cost analyses, we restricted our sample to patients whose admission and

discharge dates were both in 2011-2013 to ensure that we observe all costs during hospitalizations.

We assigned each hospitalization to a physician based on the National Provider Identifier (NPI) in the

Carrier File that accounted for the most Medicare Part B spending during that hospitalization.9 In the

U.S., costs of care of hospitalized patients consist of Part A (payments to hospitals) and Part B

spending (professional and other fees determined by the physician). We focused on Part B spending

because it encompasses professional and other services at the discretion of physicians; and for

hospitalized patients, Part A spending is largely invariant to physician decisions because of the fixed

DRG payment. On average, 51.1%, 22.0%, and 11.1% of total Part B spending was accounted for by

Page 11 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

12

the first, second, and third highest-spending physicians, respectively. We focused on general internists

to avoid comparing physicians across different specialties. We also restricted our analysis to graduates

of allopathic medical schools.

Identification of international medical graduates

For each physician, the Doximity database includes information on the medical school from which a

physician graduated. We identified countries where each medical school was located using multiple

sources including the International Medical Education Directory (IMED) organized by the Foundation

for Advancement of International Medical Education and Research (FAIMER),10 the Association of

American Medical Colleges (AAMC) database,11 and web pages of individual medical schools. The

data on graduated medical school were available for 83.6% of physicians.

Patient outcomes

The primary outcome was the 30-day mortality rate of patients, and secondary outcomes were 30-day

readmission rates and costs of care. Costs of care were defined as total Part B spending per

hospitalization.

Adjustment variables

Page 12 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

13

We adjusted for patient characteristics, physician characteristics, and hospital fixed effects (i.e.,

hospital-specific indicator variables included as covariates). Patient characteristics included age in

5-year increments (65-69 years through 90-94 years, and ≥ 95 years), sex, race or ethnic group

(non-Hispanic white, non-Hispanic black, Hispanic, other), primary diagnosis (defined by MS-DRG),

Elixhauser comorbidity index12 (27 coexisting conditions), median household income estimated from

residential zip codes (in deciles), and an indicator for dual Medicare-Medicaid coverage. Physician

characteristics consisted of age in 5-year increments (<35 years, 35-39, years and so on through 65-69

years, and ≥ 70 years) and sex. Hospital fixed effects account for both measured and unmeasured

characteristics of hospitals, allowing us to effectively compare patient outcomes between IMGs and

USMGs within the same hospital.13 14

Statistical analysis

We first examined whether patient mortality rates differed between IMGs and USMGs using three

models. Model 1 compared patient outcomes between IMGs and USMGs, adjusting for patient

characteristics using a multivariable ordinary least squares regression model (multivariable linear

probability model). Model 2 adjusted for all variables in model 1 plus physician characteristics,

thereby comparing adjusted patient outcomes of IMGs versus USMGs across hospitals.14 Model 3

adjusted for all variables in model 2 plus hospital fixed effects, effectively comparing IMGs and

Page 13 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

14

USMGs within the same hospital.13 14 We clustered standard errors at the physician level.14

We then evaluated whether patient readmission rates and costs of care differed between IMGs and

USMGs, using the same series of regression models as for the mortality analysis.

Finally, we assessed whether differences in patient outcomes varied according to the primary condition

for which a patient was admitted. We evaluated six major conditions treated by general internists,

selected on the basis of frequency: sepsis, pneumonia, congestive heart failure (CHF), chronic

obstructive pulmonary disease (COPD), urinary tract infection (UTI), and arrhythmia. A list of

International Classification of Diagnosis, 9th Edition (ICD-9) codes for these conditions is available in

Supplementary Table A.

Sensitivity analyses

We conducted sensitivity analyses. First, to address the possibility that IMGs may treat patients with

lesser or greater unmeasured illness severity, we restricted the study population to patients treated by

physicians who specialize in the care of hospitalized patients (“hospitalists”). Hospitalists typically

work in shifts that stretch several continuous days; therefore, within the same hospital, patients treated

by hospitalists are plausibly quasi-randomized to a given physician based on that physician’s work

Page 14 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

15

schedule.15 Second, to test sensitivity of our findings to how we attributed patients to physicians, we

attributed patients to physicians who had largest number of evaluation and management (E&M)

claims.9 16 17 Third, to account for the influence of IMGs who were U.S. citizens, we excluded IMGs

who graduated from Caribbean medical schools (because more than three quarters of U.S. citizen

IMGs graduated from Caribbean medical schools5 18 19). Fourth, since differences in patient volume,

length of stay (LOS), utilization of care (total Part B spending per hospitalization), or discharge

location may explain differences in patient outcomes between IMGs and USMGs, we further adjusted

our regression models for these variables. Fifth, to address the impact of unobserved care preferences

of patients, we excluded patients with cancer and patients who were discharged to hospice.20 Sixth, as

IMGs may be more or less likely to work in ICUs as intensivists, we excluded hospitals with a medical

ICU. Seventh, we evaluated whether our findings depended on the model specification by repeating

our analyses using multivariable logistic regression with cluster-robust standard errors. Finally, we

investigated whether patient outcomes varied by countries where IMGs were trained, restricting to

eight countries sending the largest number of IMGs to the U.S. (India, Pakistan, Philippines, Syria,

Nigeria, Mexico, Egypt, and China) to avoid unstable estimates. See supplementary material 1 for

more details.

The study was approved by the institutional review board at Harvard Medical School.

Page 15 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

16

Patient involvement

No patients were involved in setting the research question or the outcome measures, nor were they

involved in developing plans for implementation of the study. No patients were asked to advise on

interpretation or writing up of results. There are no plans to disseminate the results of the research to

study participants or the relevant patient community.

RESULTS

Hospital, physician, and patient characteristics of IMGs

Overall, 25.8% of all physicians and 45.0% of general internists were IMGs in the U.S. Among general

internists, IMGs tended to be younger than USMGs (46.6 vs. 48.5 years old, p<0.001) (Table 1) and

were more likely to work in medium-sized, non-teaching, for-profit hospitals, and hospitals without

ICUs.

In general, compared to USMGs, patients treated by IMGs were more likely to be non-White, have

lower median household income, have Medicaid coverage (26.8% vs. 22.3%, p<0.001), and have more

coexisting comorbid conditions including CHF and COPD (Table 2).

Page 16 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

17

Patient mortality

Our final sample included 909,434 hospitalizations treated by 40,592 physicians for mortality analysis.

The overall 30-day mortality rate was 11.3%; unadjusted mortality rate was 10.9% for IMGs and

11.8% for USMGs. Patients cared for by IMGs had lower risk-adjusted mortality compared with those

treated by USMGs (adjusted mortality rate 11.1% vs. 11.6%; risk difference -0.5%, 95%CI: -0.7% to

-0.4%, p<0.001) (Table 3). This relationship remained essentially unchanged after additional

adjustment for physician characteristics (adjusted mortality rate 11.1% vs. 11.6%; adjusted risk

difference -0.5%, 95%CI: -0.7% to -0.4%, p<0.001), or after further adjusting for hospital fixed effects,

which effectively compared IMGs and USMGs within the same hospital (adjusted mortality rate 11.0%

vs. 11.7%; adjusted risk difference -0.7%, 95%CI: -0.9% to -0.5%, p<0.001).

Readmission rates and costs of care

Our final sample was 884,997 hospitalizations treated by 40,553 physicians for readmission analysis.

The overall 30-day readmission rate was 15.6%; unadjusted readmission rate was 16.2% for IMGs and

14.9% for USMGs. We found that patients treated by IMGs had higher risk-adjusted readmission rates

compared to those cared for by USMGs (adjusted readmission rate 16.0% vs. 15.0%; risk difference

+1.0%, 95%CI: +0.8% to +1.2%, p<0.001) (Table 4). This relationship remained unaffected after

additional adjustment for physician characteristics. However, when we further adjusted for hospital

Page 17 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

18

fixed effects, there was no longer any difference in readmission rates between IMGs and USMGs

(adjusted readmission rate 15.6% vs. 15.5%; risk difference +0.1%, 95%CI: -0.07% to +0.3%, p=0.22),

indicating that higher unadjusted readmission rates for IMGs were driven by differences in the

hospitals in which IMGs work (i.e., IMGs tend to practice in hospitals with higher readmission rates).

Our final sample consisted of 957,204 hospitalizations treated by 41,001 physicians for analysis of

costs. Costs of care were slightly higher for IMGs compared with USMGs. After adjusting for the

hospital, as well as for patient and physician characteristics, IMGs had slightly higher costs of care

(adjusted costs of care $1,134 vs. $1,082; adjusted difference +$52, 95%CI: +$44 to +$60, p<0.001).

Patient outcomes for specific clinical conditions

IMGs had lower patient mortality rates compared with USMGs for sepsis, pneumonia, CHF, and

arrhythmia. We observed no differences in patient mortality between IMGs and USMGs for COPD

and UTI (Figure 1 and Supplementary Table B). The lack of differences in mortality for patients with

COPD and UTI may be because of lack of statistical power to detect differences in the relatively low

overall 30-day mortality rates for this population. We observed no significant difference in adjusted

readmission rates between USMGs and IMGs across all conditions we studied. Costs of care were

slightly higher and statistically significant for IMGs across all conditions we studied.

Page 18 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

19

Additional analyses

Hospitalists who were IMGs treated patients that were similar across a broad range of characteristics

compared with hospitalists who were USMGs, supporting our hypothesis that patients treated by

hospitalists are quasi-randomized based on physician work schedules (Supplementary Table C).

Among hospitalists, IMGs had lower patient mortality than USMGs (adjusted mortality rate 10.7% vs.

11.4%; adjusted risk difference -0.7%, 95%CI: -0.9% to -0.5%, p<0.001; Supplementary Table D).

Our overall findings among general internists were not qualitatively affected by attributing physicians

to patients on the basis of E&M claims; exclusion of IMGs who graduated from Caribbean medical

schools; exclusion of hospitals with ICUs; additional adjustment for patient volume, LOS, utilization

of care, discharge location; exclusion of patients with cancer or discharged to hospice; or the use of

multivariable logistic regression models (Supplementary Table E-H). Patient outcomes varied

modestly by country where IMGs were trained, although our statistical power to detect differences by

individual countries was limited. We found that among the eight countries sending the largest number

of IMGs to the U.S., none had mortality rates higher than USMGs, although the difference between

USMGs and IMGs from these countries was significant in four instances (Supplementary Figure

A-C).

Page 19 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

20

DISCUSSION

Using a nationally representative sample of hospitalized elderly patients in the U.S., we found that

patients treated by IMGs had lower 30-day mortality than those cared for by USMGs. These

differences persisted across a broad range of clinical conditions, and even among hospitalists, where

patient selection may be less of a concern. We found no differences in readmission rates between

patients cared for by these two groups of physicians and slightly higher spending for IMGs, although

these differences were small. Taken together, our findings should reassure policymakers and the

general public that our current approach to licensing IMGs is sufficiently rigorous to ensure

high-quality care.

Possible explanations

Although our findings suggest that differences in patient volume, LOS, spending, or discharge location

do not explain the differences in patient outcomes between IMGs and USMGs, there are several

potential explanations for why IMGs might have better outcomes than USMGs. The current approach

for allowing IMGs to practice in the U.S. may select for, on average, better physicians. Indeed, the

match rate for U.S. residency programs is substantially lower for IMGs (49.4% ) compared to that for

USMGs (94.0% for U.S. allopathic medical schools),21 and therefore, it is possible that IMGs who are

successful in the U.S. matching process may represent some of the best physicians in their country of

Page 20 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

21

origin. The fact that IMGs outperform USMGs in test scores lends some credence to this hypothesis.22

In addition, many of the IMGs who are currently practicing in the U.S. likely underwent residency

training twice, once in their home country and once in the U.S., and such intensive and prolonged

training may be another reason why they may perform better than USMGs.

Strengths and limitations of this study

The main strengths of this study were its use of nationally representative data and use of a natural

experiment study design to minimize the impact of confounders. Patients treated by hospitalist

physicians are plausibly quasi-randomized based on when patients seek medical care and on

hospitalists’ work schedules. Although it is still possible that severity of illness of patients may be

different between IMGs and USMGs in unobservable characteristics, our data showed that, if any,

IMGs appear to treat patients with higher rates of chronic disease and lower socioeconomic status than

USMGs. For example, we found that patients treated by IMGs had lower socioeconomic status (more

often identified as racial and ethnic minorities, came from lower income neighborhoods, and had

higher Medicaid coverage), which is generally associated with worse outcomes. Our findings also

suggest that patients of IMGs have higher rates of chronic conditions such as CHF, COPD, and

diabetes, all of which are again associated with worse outcomes. Moreover, we are unaware of any

other studies that suggest that IMGs are treat systematically healthier patients than USMGs do.

Page 21 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

22

There are limitations to our study. First, we could not identify the mechanism by which patients treated

by IMGs have lower mortality than those cared for by USMGs. Knowing how care patterns differ

between IMGs and USMG and which practices are responsible for better outcomes would be helpful

for interpreting our results. Second, patient mortality may appear lower for IMGs if there were more

complete coding of comorbidity for patients of IMGs. While this is theoretically possible, it is unlikely

that coding practices would vary substantially within the same hospital. Further, we observed lower

unadjusted mortality for patients treated by IMGs, which is not influenced by coding practice. Finally,

our results may differ for surgical conditions, in the outpatient setting, or in the non-Medicare

population. It is also possible that the relationships may be different in other high-income countries.

Comparison with other studies

Our findings are consistent with a limited set of smaller studies. A study of physicians practicing in

Pennsylvania, US found that IMGs had lower in-hospital mortality among patients with medical

conditions compared with USMGs.2 However, another analysis from Ontario, Canada reported no

difference in mortality between IMGs and Canadian medical graduates,4 suggesting that the quality of

care provided by IMGs may depend on the rigorousness of licensure process which varies from

country to country.

Page 22 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

23

Unanswered questions and future research

Future research is warranted to understand whether patient outcomes differ between IMGs and

domestically-trained doctors in other high-income countries.

Conclusions and policy implications

Using national data on hospitalized Medicare beneficiaries, we found no evidence that patient

outcomes were worse for IMGs than for USMGs. To the extent that there was a difference, patients

treated by IMGs appeared to have somewhat lower mortality than patients cared for by USMGs,

although these differences were small. Our findings indicate that current standards of selecting IMGs

for practice in the U.S. are functioning well for at least on one important dimension – inpatient

outcomes. As we consider expanding our physician workforce, these results suggest that systems

modeled on the current rigorous approach to incorporate IMGs should allow for both better access to

care and good outcomes.

Page 23 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

24

Table 1. Physician and hospital characteristics of general internists, U.S. versus International medical

graduates

International Medical Graduates

(n=18,407)

U.S. Medical Graduates

(n=22,485)

Physician characteristics

Physician's age, years 46.6 (10.4) 48.5 (11.2)

Female, % 30.0% 30.4%

No. of hospitalizations per year, n 130.1 90.1

Hospital characteristics

Hospital

size, %

Large (400+ beds) 33.8% 42.2%

Medium (100-399 beds) 59.1% 49.8%

Small (1-99 beds) 7.2% 8.0%

Teaching

status, %

Major teaching 18.8% 29.0%

Minor teaching 34.3% 34.3%

Non-teaching 46.9% 36.7%

Ownership,

%

For-profit 15.7% 11.2%

Non-profit 75.7% 76.3%

Public 8.6% 12.4%

Region, %

Northeast 24.8% 21.2%

Midwest 25.0% 22.9%

South 34.1% 35.0%

West 16.2% 20.9%

Rural-urban

status, %

Urban 86.6% 86.7%

Suburban 2.2% 1.8%

Large rural 9.0% 9.2%

Small rural 2.2% 2.3%

ICU, % Hospitals with ICU 84.6% 88.4%

Hospitals without ICU 15.4% 11.6%

The percentages for hospital characteristics represent the proportion of physicians practicing in that

type of hospital. The p-value for the differences in physician characteristics was <0.001 for age and

number of hospitalizations per year and 0.44 for sex. The p-value for the differences in hospital

characteristics between USMGs and IMGs was <0.001 for all variables except rural-urban status

(p=0.02).

Page 24 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

25

Table 2. Characteristics of patients treated by general internists, U.S. versus International medical

graduates

International Medical Graduates

(n=18,407)


(n=22,485)

Number of patients, n 502,986 425,483

Patient's age, years (sd) 80.6 (8.5) 80.9 (8.4)

Female, % 61.0% 60.5%

Race, %

White 79.2% 82.7%

Black 11.6% 10.4%

Hispanic 6.5% 4.0%

Other races 2.8% 3.0%

Median household income, $ (sd)* $55,969

($22,062)

$56,989

($23,044)

Medicaid status 26.8% 22.3%

Comorbid

condition, %

CHF 20.3% 19.8%

COPD 26.4% 25.6%

Diabetes 33.1% 31.3%

Renal failure 21.8% 21.5%

Neurological disorders 16.5% 16.0%

Cancer 6.7% 7.3%

Mental illness 15.7% 15.5%

Discharge

location, %

Home 57.4% 58.2%

Skilled nursing facility 27.2% 25.7%

Rehabilitation facility 2.5% 2.4%

Others 13.0% 13.7%

The p-value for the differences between U.S. Medical Graduates and International Medical Graduates

was <0.001 for all patient characteristics except for renal failure (p=0.09) and mental illness (p=0.18).

*Median household income is based on beneficiary zip code of residence.

Page 25 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only26

Table 3. Patient mortality rates between U.S. and International medical graduates

N of hospitalizations

(N of physicians)

Patient outcomes

(95%CI) Adjusted risk

difference

(95%CI)

IMG −−−− USMG

p-value

IMGs USMGs

Model 1: Risk-adjusted 30-day

mortality*

935,265 (41,903)

11.1% (11.0% to 11.2%)

11.6% (11.5% to 11.7%)

-0.5% (-0.7% to -0.4%)

<0.001

Model 2: Model 1 + physician

characteristics

909,434 (40,592)

11.1% (11.0% to 11.2%)

11.6% (11.5% to 11.7%)

-0.5% (-0.7% to -0.4%)

<0.001

Model 3: Model 2 + hospital fixed

effects

909,434 (40,592)

11.0% (10.9% to 11.1%)

11.7% (11.6% to 11.8%)

-0.7% (-0.9% to -0.5%)

<0.001

USMG denotes U.S. medical graduate, and IMG denotes international medical graduate. *Risk-adjustment using patient age, sex, race, primary diagnosis, coexisting conditions (Elixhauser comorbidity index), median household income, and Medicaid status.

Page 26 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

Confidential: For Review Only27

Table 4. Patient readmission rates and costs of care between U.S. and International medical graduates

N of hospitalizations

(N of physicians)

Patient outcomes

(95%CI)

Adjusted risk

difference

(95%CI)


p-value

IMGs USMGs

30-day

readmission rate

Model 1: Risk-adjusted

30-day readmissions*

910,083 (41,892)

16.0% (15.9% to 16.2%)

15.0% (14.9% to 15.2%)

+1.0% (+0.8% to +1.2%)

<0.001

Model 2: Model 1 +

physician characteristics

884,997 (40,553)

16.0% (15.9% to 16.2%)

15.1% (14.9% to 15.2%)

+1.0% (+0.8% to +1.1%)

<0.001

Model 3: Model 2 +

hospital fixed effects

884,997 (40,553)

15.6% (15.5% to 15.8%)

15.5% (15.4% to 15.7%)

+0.1% (-0.07% to +0.3%)

0.22

Total Part B costs

per

hospitalization

Model 1: Risk-adjusted

costs per hospitalization*

984,354 (42,325)

$1174 ($1166 to $1181)

$1034 ($1027 to $1040)

+$140 (+$130 to +$150)

<0.001

Model 2: Model 1 +

physician characteristics

957,204 (41,001)

$1179 ($1171 to $1186)

$1030 ($1023 to $1037)

+$149 (+$138 to +$159)

<0.001

Model 3: Model 2 +

hospital fixed effects

957,204 (41,001)

$1134 ($1129 to $1139)

$1082 ($1077 to $1088)

+$52 (+$44 to +$60)

<0.001

USMG denotes U.S. medical graduate, and IMG denotes international medical graduate. *Risk-adjustment using patients’ age, sex, race, primary diagnosis, coexisting conditions (Elixhauser comorbidity index), median household income, and Medicaid status.

Page 27 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

28

Fig 1. Patient outcomes between U.S. and International medical graduates, by primary diagnosis

(A) Adjusted 30-day mortality rate

* Statistically significant with p<0.05. ** Statistically significant with p<0.01. Note: Risk-adjusted mortality with additional adjustment for physician characteristic and with hospital

fixed effects (Model 3). Standard errors were clustered at the physician-level. CHF denotes congestive

heart failure, COPD denotes chronic obstructive pulmonary disease, and UTI denote urinary tract

infection.

Page 28 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

29

(B) Adjusted 30-day readmission rate

p-value for difference >0.05 for all conditions.

Note: Risk-adjusted readmission with additional adjustment for physician characteristic and with

hospital fixed effects. Standard errors were clustered at the physician-level. CHF denotes congestive


infection.

Page 29 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

30

(C) Adjusted total Part B costs per hospitalization

*** Statistically significant with p<0.001. Note: Risk-adjusted readmission with additional adjustment for physician characteristic and with

hospital fixed effects. Standard errors were clustered at the physician-level. CHF denotes congestive


infection.

Page 30 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

31

REFERENCES

1. ECFMG. Medical School Accreditation Requirement for ECFMG Certification. Secondary Medical

School Accreditation Requirement for ECFMG Certification 2010.

http://www.ecfmg.org/about/initiatives-accreditation-requirement.html.

2. Norcini JJ, Boulet JR, Dauphinee WD, et al. Evaluating the quality of care provided by graduates of

international medical schools. Health Aff (Millwood) 2010;29(8):1461-8.

3. Norcini JJ, Boulet JR, Opalek A, et al. Outcomes of cardiac surgery: associations with physician

characteristics, institutional characteristics, and transfers of care. Med Care 2013;51(12):1034-9.

4. Ko DT, Austin PC, Chan BT, et al. Quality of care of international and Canadian medical graduates in

acute myocardial infarction. Arch Intern Med 2005;165(4):458-63.

5. Eckhert NL, van Zanten M. U.S.-citizen international medical graduates--a boon for the workforce? N

Engl J Med 2015;372(18):1686-7.

6. Jena AB, Khullar D, Ho O, et al. Sex Differences in Academic Rank in US Medical Schools in 2014.

JAMA 2015;314(11):1149-58.

7. Jena AB, Olenski AR, Blumenthal DM. Sex Differences in Physician Salary in US Public Medical

Schools. JAMA Intern Med 2016.

8. Olmsted MG, Geisen E, Murphy J, et al. Methodology: U.S. News & World Report Best Hospitals

2015-16: RTI International, 2015.

9. McWilliams JM, Landon BE, Chernew ME, et al. Changes in Patients' Experiences in Medicare

Accountable Care Organizations. N Engl J Med 2014;371(18):1715-24.

10. FAIMER (Foundation for Advancement of International Medical Education and Research).

International Medical Education Directory (IMED). Secondary International Medical Education

Directory (IMED). http://www.faimer.org/resources/imed.html.

Page 31 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

32

11. Association of American Medical Colleges. Association of American Medical Colleges web page.

Secondary Association of American Medical Colleges web page. https://www.aamc.org.

12. Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data.

Medical care 1998;36(1):8-27.

13. Gunasekara FI, Richardson K, Carter K, et al. Fixed effects analysis of repeated measures data. Int J

Epidemiol 2014;43(1):264-9.

14. Wooldridge JM. Econometric analysis of cross section and panel data: MIT press, 2010.

15. Hinami K, Whelan CT, Miller JA, et al. Job characteristics, satisfaction, and burnout across

hospitalist practice models. J Hosp Med 2012;7(5):402-10.

16. McWilliams JM, Hatfield LA, Chernew ME, et al. Early Performance of Accountable Care

Organizations in Medicare. N Engl J Med 2016;374(24):2357-66.

17. Mehrotra A, Adams JL, Thomas JW, et al. Is Physician Cost Profiling Ready for Prime Time? 2010.

18. Boulet JR, Swanson DB, Cooper RA, et al. A comparison of the characteristics and examination

performances of U.S. and non-U.S. citizen international medical graduates who sought

Educational Commission for Foreign Medical Graduates certification: 1995-2004. Acad Med

2006;81(10 Suppl):S116-9.

19. Norcini J, Anderson MB, McKinley DW. The medical education of United States citizens who train

abroad. Surgery 2006;140(3):338-46.

20. Covinsky KE, Fuller JD, Yaffe K, et al. Communication and decision-making in seriously ill

patients: findings of the SUPPORT project. The Study to Understand Prognoses and Preferences

for Outcomes and Risks of Treatments. J Am Geriatr Soc 2000;48(5 Suppl):S187-93.

21. National Resident Matching Program. Results and Data: 2015 Main Residency Match®. Washington,

DC, 2015.

Page 32 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

33

22. Garibaldi RA, Subhiyah R, Moore ME, et al. The In-Training Examination in Internal Medicine: an

analysis of resident performance over time. Ann Intern Med 2002;137(6):505-10.

Page 33 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

1

Quality of Care Delivered by International Medical Graduates in the U.S. Hospitals:

Observational Study (Online-Only Material)

Tsugawa Y, Jena AB, Orav EJ, and Jha AK.

Contents:

Supplementary Material 1

Table A. ICD-9 (International Classification of Diseases, 9th Edition) codes

Table B. Patient outcomes between USMGs and IMGs, by primary diagnosis

Table C. Characteristics of patients treated by hospitalist physicians, USMGs versus IMGs

Table D. Patient outcomes between USMGs and IMGs, among hospitalists

Table E. Patient mortality rates between USMGs and IMGs, additional analyses

Table F. Patient readmission rates between USMGs and IMGs, additional analyses

Table G. Patient costs of care between USMGs and IMGs, additional analyses

Table H. Patient outcomes between USMGs and IMGs, using multivariable logistic regression

Figure A. Adjusted patient mortality rates by country

Figure B. Adjusted patient readmission rates by country

Figure C. Adjusted patient costs of care by country

Page 34 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

2

Supplementary Material 1

Licensure process for international medical graduates to practice in the U.S.

In order for physicians who graduated from medical schools outside the U.S. or Canada to practice

medicine in the U.S., they must pass United States Medical Licensing Examination (USMLE) (Step 1,

Step 2 Clinical Knowledge [CK], and Step 2 Clinical Skills [CS]) and complete residency training in the

U.S..1 USMLE examinations cost international medical graduates (IMGs) approximately $3,600

(including international test delivery surcharges).2 Moreover, while Step 1 and Step 2 CK can be taken at

examination centers outside the U.S., IMGs are required to travel to the U.S. to take Step 2 CS, which

put additional financial burden on IMGs. Many IMGs also undergo training courses and practice

examinations provided by private companies with the aim of improving the likelihood of passing

USMLE examinations (especially in order to prepare for step 2 CS). After passing USMLE

examinations and acquiring Educational Commission for Foreign Medical Graduates (ECFMG)

certification, IMGs who wish to practice in the U.S. must complete a residency program accredited by

Accreditation Council for Graduate Medical Education (ACGME) in the U.S. or Canada. In general,

IMGs enter residency programs through the National Resident Matching Program.

Physician database collected by Doximity

Physician specialties were identified by linking the NPI to a database assembled by Doximity, a

company that provides online professional networking services for U.S. physicians.3 4 Doximity has

collected information on physician specialty for all U.S. physicians (including registered members of the

service and non-registered physicians) from multiple sources and data partnerships, including the

National Plan and Provider Enumeration System NPI Registry, the American Board of Medical

Specialties, other specialty societies, state licensing boards, and collaborating hospitals and medical

Page 35 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

3

schools. Details and validation of the Doximity database are described elsewhere.3-5

Assignment of patients to physicians

For each hospitalization, we assigned a physician who accounted for the largest Part B spending. We

focused on Part B spending because it encompasses professional and other services at the discretion of

physicians; and for hospitalized patients, Part A spending is largely invariant to physician decisions on

an individual patient’s treatment because of the fixed DRG payment (only 1.0% of hospitalizations in

our data received an outlier payment). It is also possible that a patient was admitted on a weekend or

during the night but most of the clinical decisions about that patient’s care happened in subsequent days

by a different physician. In this case, the practice style of the second physician, not the admitting

physician, should be responsible for the outcomes of that patient. Moreover, according to Research Data

Assistance Center (ResDAC) of Centers for Medicare & Medicaid Services (the government entity

responsible for providing assistance to researchers with regard to Medicare data), both the admitting

physicians (physicians who filed the first claim for a given hospitalization) and attending physicians

identified in Medicare data (claim attending physician NPI number in Medicare Inpatient files) are not

reliably coded in the Medicare data (mainly because they are not directly linked to the Medicare

payments). Using the 2011 Medicare data, we found that 51.1%, 22.0%, and 11.1% of the total Part B

spending is accounted for by the highest, second highest, and third highest-spending physicians,

respectively, suggesting that a single, rather than multiple, physicians drive care. Similar methods have

been used by other studies.6-8 As a sensitivity analysis, we reanalyzed the data using the plurality of

evaluation-and-management (E&M) claims to assign physicians (assigning physicians who accounted

for the largest number of E&M claims) and found that it produces nearly identical results. For patients

transferred to other acute care hospitals, we attributed patient outcomes to the physician of the initial

Page 36 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

4

hospitalization.9 10

Statistical analysis

We examined whether patient mortality rates differed between IMGs and USMGs. To create a

homogeneous group of physicians who underwent similar medical school training, we restricted to

graduates of allopathic medical schools. We constructed three models. First, we compared patient

outcomes between IMGs and USMGs after adjusting for patient characteristics using a multivariable

ordinary least squares model (multivariable linear probability model). Second, we adjusted for both

patient and physician characteristics, thereby comparing adjusted outcomes of patients treated by IMGs

versus USMGs across hospitals.11 Third, because IMGs may practice in hospitals with otherwise higher

rates of mortality and readmissions (e.g., rural hospitals12 13), we estimated the above multivariable

model with the additional inclusion of hospital fixed effects, effectively comparing IMGs and USMGs

within the same hospital.11 14 To account for potential correlation of outcomes among patients treated by

the same physician, we clustered standard errors at the physician level.11 We presented risk-adjusted

patient mortality rates, calculated by estimating predicted probabilities of death for each hospitalization

averaged over the distribution of covariates in our national sample and fixing the indicator variable for

IMG at either one or zero (known as the marginal standardization form of predictive margins15 16).

Sensitivity analyses

We conducted a series of sensitivity analyses. First, to address the possibility that IMGs may treat

patients with lesser (or greater) unmeasured illness severity, we restricted the study population to

patients treated by hospitalist physicians. Hospitalists are physicians who specialize in the care of

hospitalized patients. The specialty of hospitalist in the U.S. health system, which began in mid-1990s,

Page 37 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

5

is fast emerging; it is estimated that approximately 21,100 to 22,900 hospitalists were practicing in the

U.S. in 2010.17 Hospitalists typically work in shifts,18 and therefore, within the same hospital, patients

treated by hospitalists may plausibly be quasi-randomized to a given physician based on that physician’s

work schedule. We used a previously validated definition of hospitalist as general internists with at least

20 E&M claims in a given year (equivalent to 5 or more E&M claims in a 5% sample), who filed at least

90% of their total E&M claims in an inpatient setting19 as defined by Current Procedural Terminology

[CPT] codes 99221-99223, 99231-99233, and 99251-99255. This approach had been validated with a

high sensitivity (84.2%) specificity (96.5%), and a positive predictive value (88.9%) for identifying

hospitalists in the Medicare sample.19

Second, to test whether our findings are sensitive to how we attributed patient outcomes to physicians,

we reanalyzed the data attributing patient outcomes to physicians who had largest number of E&M

claims, instead of largest spending. Similar methods have been used in previous studies.6-8

Third, while U.S. citizens who go abroad for medical school (U.S. citizen IMGs) constitute about

one-fifth of all IMGs,20 our data precluded us from distinguishing U.S. citizen IMGs from non-U.S.

citizen IMGs. Studies have suggested that U.S. citizen IMGs who attend medical schools in the

Caribbean may not perform well.21 22 In order to remove the influence of these U.S. citizen IMGs, we

reanalyzed the data after excluding IMGs who graduated from the Caribbean medical schools (known as

offshore medical schools).23

Fourth, as differences in patient volume, length of stay (LOS), utilization of care (total Part B spending

per hospitalization), or in discharge location may explain the difference in patient outcomes between

Page 38 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

6

IMGs and USMGs, we further adjusted our regression models for these variables. Patient volume, LOS,

and utilization of care were modeled as a continuous variable with quadratic and cubic terms to allow

for a non-linear relationship. Discharge location was used as a categorical variable: home, skilled

nursing facilities (SNFs), rehabilitation facility, hospice, and others.

Fifth, in order to address the possibility that unobserved care preferences of patients such as

do-not-resuscitate (DNR) directives may confound our findings, we reanalyzed the data after excluding

patients with cancer (as cancer is one of the strongest predictors of DNR directives24) and patients who

were discharged to hospice.

Sixth, it is possible that IMGs may be more or less likely to work in ICUs as intensivists and have

severely ill patients. To address this issue, we reanalyzed the data after excluding hospitals with a

medical ICU.

Seventh, we evaluated whether our findings depended on the model specification by repeating our

analyses using multivariable logistic regression with cluster-robust standard errors for the analyses of

mortality and readmissions. For the logistic models, MS-DRG codes without an outcome event were

combined into clinically similar categories in order to allow model convergence. 25

Finally, we investigated whether patient outcomes varied by countries where IMGs were trained. We

restricted to eight countries that had largest number of hospitalized patients in our sample (India,

Pakistan, Philippines, Syria, Nigeria, Mexico, Egypt, and China) to avoid unstable estimates.

Page 39 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

7

Analyses were performed using SAS, version 9.4 (SAS Institute, Cary, NC) and Stata, version 14

(Stata-Corp, College Station, Texas).

Page 40 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

8

REFERENCES

1. American Medical Association. Practice medicine. Four key steps to begin practicing medicine in the

U.S. Secondary Practice medicine. Four key steps to begin practicing medicine in the U.S.

http://www.ama-assn.org/ama/pub/about-ama/our-people/member-groups-sections/international-

medical-graduates/practicing-medicine.page.

2. Educational Commission for Foreign Medical Graduates (ECFMG). Secondary.

http://www.ecfmg.org.

3. Jena AB, Khullar D, Ho O, et al. Sex Differences in Academic Rank in US Medical Schools in 2014.

JAMA 2015;314(11):1149-58.

4. Olmsted MG, Geisen E, Murphy J, et al. Methodology: U.S. News & World Report Best Hospitals

2015-16: RTI International, 2015.

5. Jena AB, Olenski AR, Blumenthal DM. Sex Differences in Physician Salary in US Public Medical

Schools. JAMA Intern Med 2016.

6. McWilliams JM, Hatfield LA, Chernew ME, et al. Early Performance of Accountable Care

Organizations in Medicare. N Engl J Med 2016;374(24):2357-66.

7. McWilliams JM, Landon BE, Chernew ME, et al. Changes in Patients' Experiences in Medicare

Accountable Care Organizations. N Engl J Med 2014;371(18):1715-24.

8. Mehrotra A, Adams JL, Thomas JW, et al. Is Physician Cost Profiling Ready for Prime Time? 2010.

9. Ross JS, Normand SL, Wang Y, et al. Hospital volume and 30-day mortality for three common

medical conditions. N Engl J Med 2010;362(12):1110-8.

10. Drye EE, Normand SL, Wang Y, et al. Comparison of hospital risk-standardized mortality rates

calculated by using in-hospital and 30-day models: an observational study with implications for

hospital profiling. Ann Intern Med 2012;156(1 Pt 1):19-26.

Page 41 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

9

11. Wooldridge JM. Econometric analysis of cross section and panel data: MIT press, 2010.

12. Thompson MJ, Hagopian A, Fordyce M, et al. Do international medical graduates (IMGs) "fill the

gap" in rural primary care in the United States? A national study. J Rural Health

2009;25(2):124-34.

13. Fordyce MA, Doescher MP, Chen FM, et al. Osteopathic physicians and international medical

graduates in the rural primary care physician workforce. Fam Med 2012;44(6):396-403.

14. Gunasekara FI, Richardson K, Carter K, et al. Fixed effects analysis of repeated measures data. Int J

Epidemiol 2014;43(1):264-9.

15. Williams R. Using the margins command to estimate and interpret adjusted predictions and marginal

effects. Stata Journal 2012;12(2):308.

16. Zhang Y, Baik SH, Fendrick AM, et al. Comparing local and regional variation in health care

spending. N Engl J Med 2012;367(18):1724-31.

17. Colleges AoAM, Harbuck SM, Follmer AD, et al. Estimating the number and characteristics of

hospitalist physicians in the United States and their possible workforce implications, 2012.

18. Hinami K, Whelan CT, Miller JA, et al. Job characteristics, satisfaction, and burnout across

hospitalist practice models. J Hosp Med 2012;7(5):402-10.

19. Kuo YF, Sharma G, Freeman JL, et al. Growth in the care of older patients by hospitalists in the

United States. N Engl J Med 2009;360(11):1102-12.

20. Boulet JR, Norcini JJ, Whelan GP, et al. The international medical graduate pipeline: recent trends in

certification and residency training. Health Aff (Millwood) 2006;25(2):469-77.

21. Boulet JR, Swanson DB, Cooper RA, et al. A comparison of the characteristics and examination

performances of U.S. and non-U.S. citizen international medical graduates who sought

Educational Commission for Foreign Medical Graduates certification: 1995-2004. Acad Med

Page 42 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

10

2006;81(10 Suppl):S116-9.

22. Norcini J, Anderson MB, McKinley DW. The medical education of United States citizens who train

abroad. Surgery 2006;140(3):338-46.

23. Eckhert NL, van Zanten M. U.S.-citizen international medical graduates--a boon for the workforce?

N Engl J Med 2015;372(18):1686-7.

24. Covinsky KE, Fuller JD, Yaffe K, et al. Communication and decision-making in seriously ill

patients: findings of the SUPPORT project. The Study to Understand Prognoses and Preferences

for Outcomes and Risks of Treatments. J Am Geriatr Soc 2000;48(5 Suppl):S187-93.

25. Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Stat Med

2002;21(16):2409-19.

Page 43 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

11

Table A. ICD-9 (International Classification of Diseases, 9th Edition) codes

Condition ICD-9 codes

Sepsis

0031, 0202, 0223, 0362, 0380, 0381, 03810, 03811, 03812, 03819, 0382, 0383, 03840, 03841, 03842, 03843, 03844, 03849, 0388, 0389, 0545, 449, 77181, 7907, 99591, 99592

Pneumonia

00322, 0203, 0204, 0205, 0212, 0221, 0310, 0391, 0521, 0551, 0730, 0830, 1124, 1140, 1144, 1145, 11505, 11515, 11595, 1304, 1363, 4800, 4801, 4802, 4803, 4808, 4809, 481, 4820, 4821, 4822, 4823, 48230, 48231, 48232, 48239, 4824, 48240, 48241, 48242, 48249, 4828, 48281, 48282, 48283, 48284, 48289, 4829, 483, 4830, 4831, 4838, 4841, 4843, 4845, 4846, 4847, 4848, 485, 486, 5130, 5171

Congestive

heart failure

39891, 4280, 4281, 42820, 42821, 42822, 42823, 42830, 42831, 42832, 42833, 42840, 42841, 42842, 42843, 4289

Chronic

obstructive

pulmonary

disease

490, 4910, 4911, 4912, 49120, 49121, 49122, 4918, 4919, 4920, 4928, 494, 4940, 4941, 496

Urinary

tract

infection

03284, 59000, 59001, 59010, 59011, 5902, 5903, 59080, 59081, 5909, 5950, 5951, 5952, 5953, 5954, 59581, 59582, 59589, 5959, 5970, 59780, 59781, 59789, 59800, 59801, 5990

Arrhythmia 4270, 4271, 4272, 42731, 42732, 42760, 42761, 42769, 42781, 42789, 4279, 7850, 7851

Page 44 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

12

Table B. Patient outcomes between USMGs and IMGs, by primary diagnosis

N of

hospitalizations

(N of

physicians)

Patient outcomes

(95%CI)

Adjusted

difference

(95%CI)

IMG –––– USMG

p-value

IMGs USMGs

30-Day

Mortality Rate

Sepsis 79,532

(23,417)

24.6%

(24.1% to 25.0%)

25.6%

(25.1% to 26.2%)

-1.1%

(-1.9% to -0.3%) 0.006

Pneumonia 65,659

(23,183)

9.3%

(9.0% to 9.7%)

9.9%

(9.6% to 10.3%)

-0.6%

(-1.1% to -0.1%) 0.03

CHF 67,286

(23,098)

11.3%

(11.0% to 11.7%)

12.0%

(11.6% to 12.4%)

-0.7%

(-1.3% to -0.1%) 0.02

COPD 57,106

(20,777)

4.6%

(4.3% to 4.8%)

4.7%

(4.5% to 5.0%)

-0.2%

(-0.6% to +0.2%) 0.39

UTI 48,638

(19,799)

6.1%

(5.8% to 6.5%)

6.1%

(5.7% to 6.4%)

+0.07%

(-0.4% to +0.6%) 0.79

Arrhythmia 30,222

(15,859)

5.3%

(5.0% to 5.7%)

6.0%

(5.6% to 6.4%)

-0.7%

(-1.3% to -0.05%) 0.04

30-Day

Readmission

Rate

Sepsis 69.785

(22,203)

16.5%

(16.1% to 16.9%)

16.1%

(15.7% to 16.6%)

+0.4%

(-0.3% to +1.0%) 0.28

Pneumonia 64,298

(22,986)

14.0%

(13.6% to 14.4%)

14.3%

(13.9% to 14.7%)

-0.3%

(-0.9% to +0.4%) 0.41

CHF 65,553

(22,812)

21.2%

(20.8% to 21.7%)

20.7%

(20.2% to 21.2%)

+0.5%

(-0.2%to +1.3%) 0.16

COPD 57,243

(20,834)

19.0%

(18.5% to 19.4%)

19.0%

(18.4% to 19.5%)

+0.02%

(-0.8% to +0.8%) 0.95

UTI 48,554

(19,797)

14.7%

(14.2% to 15.1%)

15.0%

(14.5% to 15.5%)

-0.3%

(-1.1% to +0.4%) 0.40

Arrhythmia 30,168

(15,864)

14.4%

(13.8% to 15.0%)

14.6%

(14.0% to 15.3%)

-0.2%

(-1.2% to +0.8%) 0.71

Total Part B

Costs per

Hospitalization

Sepsis 84,318

(24,070) $1646

($1627 to $1665) $1573

($1553 to $1593) +$73

(+$42 to +$104) <0.001

Pneumonia 68,357

(23,613) $967

($957 to $976) $899

($889 to $909) +$67

(+$52 to +$83) <0.001

CHF 70,373

(23,587) $1172

($1161 to $1183) $1106

($1094 to $1119) +$65

(+$47 to +$84) <0.001

COPD 59,155

(21,160) $870

($861 to $879) $832

($822 to $842) +$38

(+$23 to +$52) <0.001

UTI 50,288

(20,173) $795

($786 to $803) $747

($737 to $756) +$48

(+$34 to +$63) <0.001

Arrhythmia 31,818

(16,341) $1003

($990 to $1016) $938

($925 to $952) +$64

(+$43 to +$85) <0.001

Risk-adjusted mortality with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). USMG denotes U.S. medical graduate, and IMG denotes international medical graduate. Costs of care are defined as an average total Part B spending per hospitalization. CHF denotes congestive heart failure, and COPD denotes chronic obstructive pulmonary disease.

Page 45 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

13

Table C. Characteristics of patients treated by hospitalist physicians, USMGs versus IMGs

International Medical

Graduates

(n=9,467)


(n=9,885) p-value

Number of patients, n (%) 298,051 246,383

Patient's age, yr 80.5 (8.5) 80.7 (8.5) <0.001

Female, % 60.5% 60.4% 0.42

Race, %

White 82.0% 82.4%

<0.001 Black 10.3% 10.1%

Hispanic 5.1% 4.2%

Other races 2.6% 3.3%

Median household income $56,410

($22,139) $57,768

($22,091)

Medicaid status 24.7% 23.1% <0.001

Coexisting

condition,

%

CHF 19.8% 19.6% 0.02

COPD 25.3% 24.8% <0.001

Diabetes 32.1% 31.1% <0.001

Renal failure 22.2% 22.1% 0.91

Neurological disorders 15.7% 15.8% 0.13

Cancer 6.8% 7. .% <0.001

Mental illness 15.2% 15.4% 0.06

Discharge

location, %

Home 58.8% 58.8%

<0.001 Skilled nursing facility 26.3% 25.8%

Rehabilitation facility 2.5% 2.2%

Others 12.5% 13.3%

USMG denotes U.S. medical graduate, and IMG denotes international medical graduate.

Page 46 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

14

Table D. Patient outcomes between USMGs and IMGs, among hospitalists

N of

hospitalizations

(N of

physicians)

Patient outcomes

(95%CI) Adjusted difference

(95%CI)


p-value

IMGs USMGs

30-Day

Mortality Rate

Model 1:

Risk-adjusted

30-day

mortality*

527,459 (17,130)

10.8%

(10.7% to 11.0%)

11.3%

(11.2% to 11.5%)

-0.5%

(-0.7% to -0.3%) <0.001

Model 2:

Model 1 +

physician

characteristics

512,272 (16,598)

10.8%

(10.7% to 11.0%)

11.2%

(11.1% to 11.4%)

-0.4%

(-0.6% to -0.2%) <0.001

Model 3:

Model 2 +

hospital fixed

effects

512,272 (16,598)

10.7%

(10.6% to 10.8%)

11.4%

(11.3% to 11.6%)

-0.7%

(-0.9% to -0.5%) <0.001

30-Day

Readmission

Rate

Model 1:

Risk-adjusted

30-day

readmissions

517,638 (17,129)

15.4%

(15.2% to 15ｖ5%)

14.7%

(14.5% to 14.9%)

+0.7%

(+0.5% to +0.9%) <0.001

Model 2:

Model 1 +

physician

characteristics

502,785 (16,597)

15.4%

(15.2% to 15.5%)

14.8%

(14.6% to 14.9%)

+0.6%

(+0.4% to +0.9%) <0.001

Model 3:

Model 2 +

hospital fixed

effects

502,785 (16,597)

15.1%

(14.9% to 15.2%)

15.1%

(15.0% to 15.3%)

-0.08%

(-0.3% to +0.2%) 0.50

Total Part B

Costs per

Hospitalization

Model 1:

Risk-adjusted

in-hospital

costs*

558,497 (17,196)

$1090 ($1081 to $1100)

$980 ($971 to $988)

+$111 (+$98 to +$124)

<0.001

Model 2:

Model 1 +

physician

characteristics

542,452 (16,662)

$1092 ($1082 to $1102)

$979 ($970 to $987)

+$114 (+$100 to +$127)

<0.001

Model 3:

Model 2 +

hospital fixed

effects

542,452 (16,662)

$1061 ($1055 to $1066)

$1018 ($1012 to $1024)

+$43 (+$34 to +$52)

<0.001


Page 47 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

15

Table E. Patient mortality rates between USMGs and IMGs, additional analyses

N of

hospitalizations

(N of physicians)

Adjusted 30-day mortality rate


difference

(95%CI)


p-value

IMGs USMGs

Alternative method for

attributing patient outcomes

to physicians*

877,658 (40,524)

10.6%

(10.5% to 10.7%)

11.1%

(11.0% to 11.2%)

-0.5%

(-0.7% to -0.3%) <0.001

Excluding IMGs of

Caribbean medical schools

876,539 (39,267)

11.1%

(11.0% to 11.2%)

11.6%

(11.5% to 11.7%)

-0.6%

(-0.7% to -0.4%) <0.001

Additional adjustment for

patient volume†

909,434 (40,592)

11.1%

(11.0% to 11.2%)

11.6%

(11.5% to 11.7%)

-0.5%

(-0.7% to -0.3%) <0.001


LOS

909,205 (40,591)

11.1%

(11.0% to 11.2%)

11.6%

(11.5% to 11.7%)

-0.5%

(-0.6% to -0.3%) <0.001


Part B spending

909,434 (40,592)

11.0%

(10.9% to 11.1%)

11.7%

(11.6% to 11.8%)

-0.7%

(-0.9% to -0.6%) <0.001


discharge location

909,434 (40,592)

11.1%

(11.0% to 11.2%)

11.6%

(11.5% to 11.7%)

-0.5%

(-0.7% to -0.4%) <0.001

Excluding patients with

cancer or discharged to

hospice

812,646 (40,036)

7.7%

(7.6% to 7.8%)

8.4%

(8.3% to 8.5%

-0.6%

(-0.8% to -0.5%) <0.001

Excluding hospitals with ICU 118,774 (8,035)

11.4%

(11.1% to 11.6%)

11.9%

(11.6% to 12.2%)

-0.5%

(-1.0% to -0.06%) 0.03

Risk-adjusted mortality with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). USMG denotes U.S. medical graduate, IMG denotes international medical graduate; LOS denotes length of stay. Patient volume, LOS, and Part B spending was used as a continuous variable with quadratic and cubic terms to allow for non-linear relationship.

*Patient outcomes were attributed to physicians who accounted for largest number of evaluation-and-management claims.

Page 48 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

16

Table F. Patient readmission rates between USMGs and IMGs, additional analyses

N of

hospitalizations

(N of physicians)

Adjusted 30-day readmission rate


difference

(95%CI)


p-value

IMGs USMGs



to physicians*

854,165 (40,464)

15.5%

(15.4% to 15.7%)

15.4%

(15.3% to 15.5%)

+0.1%

(-0.4% to +0.3%) 0.12

Excluding IMGs of


853,115 (39.231)

15.6%

(15.5% to 15.8%)

15.5%

(15.4% to 15.6%)

+0.1%

(-0.07% to +0.3%) 0.20


patient volume

884,997 (40,553)

15.6%

(15.5% to 15.7%)

15.6%

(15.5% to 15ｖ8%)

-0.06%

(-0.2% to +0.1%) 0.53


LOS

884,740 (40,552)

15.6%

(15.5% to 15.8%)

15.6%

(15.4% to 15.7%)

+0.09%

(-0.1% to +0.3%) 0.35


Part B spending

884,997 (40,553)

15.5%

(15.4% to 15.6%)

15.7%

(15.5% to 15.8%)

-0.1%

(-0.3% to +0.05%) 0.15


discharge location

884,997 (40,553)

15.6%

(15.5% to 15.8%)

15.5%

(15.4% to 15.7%)

+0.1%

(-0.06% to +0.3%) 0.19



hospice

792,549 (39,964)

15.9%

(15.8% to 16.0%)

15.8%

(15.6% to 1..9%)

+0.1%

(-0.07% to +0.3%) 0.22


16.0%

(15.7% to 16.3%)

16.1%

(15.7% to 16.5%)

-0.1%

(-0.7% to +0.4%) 0.60

Risk-adjusted mortality with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). USMG denotes U.S. medical graduate, IMG denotes international medical graduate; LOS denotes length of stay. Patient volume, LOS, and Part B spending was used as a continuous variable with quadratic and cubic terms to allow for non-linear relationship. *Patient outcomes were attributed to physicians who accounted for largest number of evaluation-and-management claims.

Page 49 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

17

Table G. Patient costs of care between USMGs and IMGs, additional analyses

N of

hospitalizations

(N of physicians)

Adjusted costs of care per

hospitalization Adjusted difference

(95%CI)

IMG –––– USMG

p-value

IMGs USMGs



to physicians*

925,754 (40,973)

$1190 ($1185 to $1195)

$1134 ($1129 to $1139)

+$56 (+$49 to +$64)

<0.001

Excluding IMGs of


922,764 (39,668)

$1148 ($1142 to $1153)

$1092 ($1086 to $1097)

+$56 (+$48 to +$64)

<0.001


patient volume

956,686 (40,597)

$1131 ($1126 to $1136)

$1087 ($1081 to $1092)

+$44 (+$36 to +$52)

<0.001


LOS

956,946 (41,000)

$1144 ($1140 to $1148)

$1098 ($1094 to $1102)

+$45 (+$39 to +$52)

<0.001


discharge location

957,204 (41,001)

$1134 ($1129 to $1139)

$1083 ($1078 to $1088)

+$51 (+$43 to +$59)

<0.001



hospice

854,752 (40,460)

$1115 ($1110 to $1120)

$1063 ($1058 to $1068)

+$52 (+$44 to +$60)

<0.001


$1143 ($1131 to $1155)

$1103 ($1088 to $1118)

+$39 (+$19 to +$60)

<0.001

Risk-adjusted mortality with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). USMG denotes U.S. medical graduate, IMG denotes international medical graduate; LOS denotes length of stay. Patient volume, LOS, and Part B spending was used as a continuous variable with quadratic and cubic terms to allow for non-linear relationship. *Patient outcomes were attributed to physicians who accounted for largest number of evaluation-and-management claims.

Page 50 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

18

Table H. Patient outcomes between USMGs and IMGs, using multivariable logistic regression

N of

hospitalizations

(N of

physicians)

Patient outcomes

(95%CI) Adjusted odds ratio

(95%CI)

IMG / USMG

p-value

IMGs USMGs

30-Day

Mortality

Rate

Model 1:

Risk-adjusted

30-day

mortality*

935,251 (41,903)

11.1%

(11.0% to 11.2%)

11.7%

(11.6% to 11.8%)

0.93

(0.91 to 0.95) <0.001

Model 2: Model

1 + physician

characteristics

909,421 (40,592)

11.1%

(11.0% to 11.2%)

11.6%

(11.5% to 11.8%)

0.93

(0.91 to 0.95) <0.001

Model 3: Model

2 + hospital fixed

effects

908,276 (40,436)

11.0%

(10.9% to 11.1%)

11.7%

(11.6% to 11.8%)

0.92

(0.90 to 0.94) <0.001

30-Day

Readmission

Rate

Model 1:

Risk-adjusted

30-day

readmissions*

910,083 (41,862)

16.0%

(15.9% to 16.1%)

15.0%

(14.9% to 15.2%)

1.08

(1.07 to 1.09) <0.001

Model 2: Model

1 + physician

characteristics

884,997 (40,553)

16.0%

(15.9% to 16.2%)

15.1%

(15.0% to 15.2%)

1.08

(1.06 to 1.09) <0.001

Model 3: Model

2 + hospital fixed

effects

884,227 (40,369)

15.7%

(15.5% to 15.8%)

15.5%

(15.4% to 15.7%)

1.01

(0.99 to 1.03) 0.23


Page 51 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

19

Figure A. Adjusted patient mortality rates by country

Risk-adjusted mortality rates with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). *Statistically significantly different from the outcomes of patients treated by U.S. medical graduates with p<0.001.

Page 52 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

20

Figure B. Adjusted patient readmission rates by country

Risk-adjusted readmission rates with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). The readmission rates for all countries studied were not statistically significantly different (p>0.05) from patients treated by U.S. medical graduates.

Page 53 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960


nly

21

Figure C. Adjusted patient costs of care by country

Risk-adjusted costs of care with additional adjustment for physician characteristic and with hospital fixed effects (Model 3). *Statistically significantly different from the outcomes of patients treated by U.S. medical graduates with p<0.001.

Page 54 of 54


BMJ

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

quality of care delivered by international medical …...confidential: for review only 1 quality of...

Documents