David C. Chang, PhD, MPH, MBADirector of Outcomes ResearchUCSD Department of Surgery

Introduction to Outcomes Research Methods and Data Resources

Surgery and public health

Problem in surgical clinical research

•Unregulated

•FDA regulation applies only to “devices” (whether a real device, or a molecular device in the form of a drug)

•Procedural medicine are not regulated

• Many reasons: complexity, difficulty in standardizing, difficulty of enforcement (“surgeons know best” attitude)

•Self-regulation

Erroneous literature

RCTs often too late

“Tipping Point”

EVAR-1, DREAM OVER

Social responsibility

•It is our responsibility in academic medicine, to shoulder the responsibility that, in other fields of medicine, has been assumed by the FDA

•To ensure that only good treatment modalities are applied to patients

Biggest barrier to good research?

•Not having a correctly constructed hypothesis

•Incorrect design

•Don’t know how to get data

•Fear of statistics

Typical questions

•Components

• What/why/when/how• Verb• Condition

•“Why is the sky blue?”

•“What is the typical presentation of appendicitis?”

•Open-ended

Open-ended questions

•Descriptive analysis

•Observational study = no comparison = no statistical test

•Only one denominator

• May have more than one numerator, generating more than one ratio

• All ratios are calculated with the same denominator

43%

57%

Descriptive statistics

P value not applicable to compare different parts of the same population

Value and pitfall

•To explore the unknown

• When you know nothing, the first step is to explore and document the numbers

•Risk of over-generalizing

45%

55%

43%

57%

Inferential statistics

P value applicable for comparing parts of two populations

What is a hypothesis?

•Question ≠ hypothesis

•Questions: usually open-ended

•Hypothesis: usually is closed-ended, asking for a yes/no answer

• Statistical testing can only give yes/no answers

The process – study design

Study design phase Data preparation Analysis phase

Question development Select database Univariate

Define population Link database Bivariate

Define subset Select data elements Multivariable

Define outcome Generate new data elements Sensitivity

Define primary comparison Subset analysis

Define covariates

Steps in constructing a hypothesis

•Specify the outcomes (O in PICO)

• Common oversight: Often focus on the P, but vague about O (a typical question, “What is the outcome (?) of xyz patients?”)

•Specify the comparisons (C in PICO)

• Not done in open-ended questions

•Specify covariates (control variables, adjustment)

Hypothesis statement

•y = b1X1 + b2X2 + b3X3

•Death = age + race + gender + insurance…

Inclusion/exclusion criteria

•Just like a clinical trials (“eligibility criteria”)

•Diagnosis and/or procedure codes?

•Common mistake

45%

55%

43%

57%

Comparison

Outcome

•Mortality?

• Rare

•Complications

•Length of stay

•Charges

•Be judicious

Covariates / independent variables

•Patient demographcis

•Patient comorbidity

•Surgeon volume

•Hospital volume

•Hospital type (teaching vs non-teaching)

•Area (rural vs urban)

Hierarchy of influence on surgical outcomes

Technique and Management

Patient

Surgeon

Hospital

Region

Nation

Outcomes research

Clinical trials

The process – data preparation

Study design phase Data preparation Analysis phase

Question development Select database Univariate

Define population Link database Bivariate

Define subset Select data elements Multivariable

Define outcome Generate new data elements Sensitivity

Define primary comparison Subset analysis

Define covariates

Overview of public and semi-public databases

Multi-specialty

•Administrative Databases

• Nationwide Inpatient Sample (NIS)

• Medicare, Medicaid• California OSHPD

•Clinical Databases

• National Surgical Quality Improvement Program (NSQIP)

Specialty-specific

•Trauma

• National Trauma Databank (NTDB) •O

ncology• Surveillance, Epidemiology, and

End Results (SEER)• National Cancer Databank (NCDB)

•Transplant

• United Network for Organ Sharing (UNOS)

Administrative databases

Advantages

•Large patient numbers

•Less selection bias

•Can be linked to other databases containing other non-medical information

Disadvantages

•Limited clinical course information

•Limited surgical procedure information

NSQIP/non-NSQIP in-hospital mortality

Select data elements

Generate new data elements

•Most time consuming step of outcomes analysis

•Not every component of your research question is readily available in the database

• For example, comorbidity• Charlson Index, Elixhauser Index

•Some common concepts actually undefined

• Readmission?

What is a “re-admission”?

•Not all “admissions” are “re-admissions”

•30-day?

•Elective?

•Transfers?

•Diagnosis-specific?

•Preventable?

The process – analysis

Study design phase Data preparation Analysis phase

Question development Select database Univariate

Define population Link database Bivariate

Define subset Select data elements Multivariable

Define outcome Generate new data elements Sensitivity

Define primary comparison Subset analysis

Define covariates

Hypothesis statement

•y = b1X1 + b2X2 + b3X3

•Death = age + race + gender + insurance…

Table 1: Descriptive analysis

Table 2: Bi-variate analysis(unadjusted comparison)

Table 3: Multivariable analysis(adjusted analysis)

Analysis for Table 1

43%

57%

Analysis for Table 1

P value not applicable to compare different parts of the same population

Analysis for Table 1

•% for categorical data

•Mean/median/SD for continuous data

•For exploratory studies, descriptive studies, case series, etc., this would be the end of the process

•Reminder, avoid overgeneralizing

Analysis for Table 2

Analysis for Table 2

•Think about data types…

• Continuous data• Categorical data• (Ordinal data)

Analysis for Table 2

•Two questions to think about when picking a stats test…

• What is my outcome/dependent variable? What is my independent/input variable?

• What type of data do I have for each?• 4 possible combinations:

• 2 variables• 2 data types

X = inputY = outcomeCat.

Cat.

Cont.

Cont.

T-test

Rank sum

ROC2

Correlation

Analysis for Table 2

Analysis for Table 3

X = inputY = outcomeCat.

Cat.

Cont.

Cont.

Logistic regression

Linear regression

T-test

Rank sum

ROC2

Correlation

Analysis for table 3

Subset analysis

•Consistency of findings

•Generalizability

Generalizability

“This is not research anymore”

“That guy”