educational research 102: selecting the best study design for your research question

Educational Research 102:

Selecting the Best Study Design for your Research QuestionFrancis S. Nuthalapaty, MD

2010 APGO Faculty Development Seminar

Disclosures

No relevant financial disclosures to declare

Learning Objectives• Describe types of research and study

designs• Understand the characteristics of a

good research question• Understand how to convert the

research question into a hypothesis• Understand how to select the most

appropriate study design to test the hypothesis

What do we already know?Educational Concept True or

False?1. High-fidelity medical simulations facilitate learning.

2. Restriction in resident duty hours improves the quality of patient care.

1. Issenberg SB, McGaghie WC, Petrusa ER, Gordon DL and Scalese RJ (2005) Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Medical Teacher 27, 2, pp 10-28.

2. Fletcher KE, Davis SQ, Underwood W, Mangrulkar RS, McMahon LF Jr, Saint S. Systematic review: effects of resident work hours on patient safety. Ann Intern Med. 2004 Dec 7;141(11):851-7.

TRUE

FALSE

Write an educational

research question in which you have

interest

Types of Research

Empirical vs. Non-empirical

Basic vs. Applied

Empirical vs. Non-Empirical Research

Empirical

• Involves collection of data first hand

Non-Empirical

• No first hand data collection

Example: Empirical or Non-empirical?

The Effect of House Staff Working Hours on the Quality of Obstetric

and Gynecologic CareBailit, J et al, Obstet Gynecol 2009

• OBJECTIVE: To measure the effect of house staff working hours reforms on the quality of obstetric and gynecologic care.

Example: Empirical or Non-Empirical?

Features and uses of high-fidelity medical simulations that lead to

effective learning: a BEME systematic review.

Issenberg SB et al, Med Teach 2005

• OBJECTIVE: Review and synthesize existing evidence in educational science that addresses the question, 'What are the features and uses of high-fidelity medical simulations that lead to most effective learning?'.

Types of Research

Empirical vs. Non-empirical

Basic vs. Applied

Basic vs. Applied Research

Basic• Results apply to a great

many people and situations

• Results are related to general theory or to a general field of knowledge

• Results need not have immediate or even clear implications

Applied• Results are applicable

only to a specific group of people in a particular situation.

• Results are not necessarily related to a broader field of knowledge

• Results must have immediate and clear implications for practice

Example: Basic or Applied?Assessing Vaginal Surgical Skills

Using Video Motion AnalysisDiwadkar G et al, Obstet Gynecol 2009

• OBJECTIVE: To demonstrate the feasibility of using video motion analysis to quantitate a key step of vaginal hysterectomy and define measurable differences between novice and experienced surgical trainees during vaginal hysterectomy.

Example: Basic or Applied?Effects of a Depression Education Program on Residents’ Knowledge,

Attitudes, and Clinical SkillsLearman L et al, Obstet Gynecol 2003

• OBJECTIVE: To determine whether an interactive educational program would improve obstetrics and gynecology Residents' knowledge, attitudes, confidence, and skills in caring for depressed patients.

Research Methods

What is the difference between Qualitative

Research and Quantitative Research?

Qualitative vs. Quantitative Research

Qualitative

• Descriptive and exploratory focus

• Used to gain insight into attitudes, behaviors, values

• Analysis of unstructured information

• Narrative reporting

Quantitative

• Hypothesis driven

• Used to identify association and/or causation

• Analysis of discrete variables

• Statistical reporting

Quantitative Research• Experimental & Quasi-

experimental• Non-Experimental

–Causal-comparative–Correlational

Quantitative Research• Experimental & Quasi-experimental

– Assess effect of an independent variable on dependent variables

– Comparison of 2 or more groups– Control over ‘treatment’ &

measurement– Randomization– Control group

Quantitative Research Designs

Study Population

Treatment

Control

Randomize

Post-test only Control Group

ExperimentalMeasureme

nt

Measurement


Study Population

Treatment

Control

Measurement

Measurement

Randomize

Pre-test/Post-test Control Group


nt

Measurement


Study Population

Treatment

Control

Measurement

Measurement

Solomon Four Group


ntMeasureme

nt

Treatment

Measurement

Control Measurement

Randomize


Treatment MeasurementStudyPopulation

Quasi-Experimental

One-shotCase Study


TreatmentMeasurement MeasurementStudyPopulation

Quasi-Experimental

One-groupPre-test/Post-test


Treatment 1 MeasurementStudyGroup 1

Quasi-Experimental

Static Group




Quasi-Experimental

Pre-test/Post-test Matched Subjects


Measurement

MeasurementMat

chin

g

Quantitative Research• Non-Experimental

–Causal-comparative–Correlational

Causal-Comparative• Baseline differences are pre-existing

– Non-permutable: Ethnicity, Gender– Permutable: Teaching style

• Determine the cause or consequences of differences

• Associations can be identified• Causality cannot be determined


Exposures Outcome

Causal-Comparative

Exploration of Effects

Control Outcome


Exposures Outcome

Causal-Comparative

Exploration of Causes

Exposures Control


Exposure Outcome

Causal-Comparative

Exploration of Consequences

Outcome

Group 1

Group 2

Correlational Research• Single group of subjects• Describe degree to which 2 or more

quantitative variables are related• Help explain important human

behaviors• Predict likely outcomes• Identifies associations• Causality cannot be determined

Correlational Research

SubjectsObservations

O1 O2

Student A

Student B

Student C

Student D

Student E

Correlational Research• Correlation Coefficient

– Calculated by regression• < .35 = only a slight relationship• .40 - .60 = possible theoretical value• .50 = minimum for crude predictions• >.65 = reasonably accurate predictions• >.85 = close relationship

– Positive correlation = direct association– Negative correlation = inverse

association

Threats to Internal Validity• Subject characteristics• Loss of subjects• Maturation• Repeated measures• Statistical regression• Investigator bias

External Validity• Can the findings from the study be

generalized to larger populations?– Subject characteristics– Environment (lab vs. natural setting)– Did the subjects act differently b/c they

were enrolled in a study (Hawthorne effect)?

What are the steps in the

research process?

The Research ProcessDefine

ResearchQuestion

ConductLiterature

Review

RefineResearchQuestion

DesignStudy

ObtainIRB Approval

CollectAnalyze Data

DisseminateResults

No holein the

literature

Holein the

literature

FINER Research Questions

• F – Feasible• I – Interesting• N – Novel• E – Ethical• R - Relevant

FINER Research Questions• Feasible:

–Can be investigated with available:• Time• Money• Space• Other resources

FINER Research Questions• Interesting:

– Interesting to you– Interesting to others in the field

• Novel:–Will answers to the question

advance knowledge in the field?

FINER Research Questions• Ethical:

– Will any physical or psychological harm come to anyone as a result of the research?• Protect participants from harm• Ensure confidentiality of data• Avoid knowing deception of participants

• Relevant:– How might answers to this question

improve educational practice?

What is a research hypothesis and how is

it different from a research question?

Research Hypothesis

A prediction regardingthe possible outcomes

of the study

Wallen & Fraenkel. Educational Research: A Guide to the Process. 2nd Ed.

Research Hypothesis• A statement which:

–Summarizes the elements of the study• The sample• The design• The predictor and outcome variables

Yoder E, MERC Workshop, 2009 AAMC Annual Meeting

Research Hypothesis• Based upon a FINER research

question• Simple – one difference or

relationship• Specific – clearly stated, defined• Stated in advance – before data

collectionYoder E, MERC Workshop, 2009 AAMC Annual Meeting

Research Hypothesis• May be stated:

– As the alternative hypothesis• Non-directional

– The difference is unknown• Directional hypothesis

– Existing studies/data suggest direction of relationships

– As the null hypothesis

Yoder E, MERC Workshop, 2009 AAMC Annual Meeting

Research Hypothesis• Ha: There is a difference between

groups or variables– Female medical students have a different IQ

than male medical students– μf ≠ μm or μf > μm

• H0: There is no difference between groups or variables– Female medical students have a different IQ

than male medical students– μf = μm Yoder E, MERC Workshop, 2009 AAMC Annual

Meeting

Example: HypothesisImproving Resident Competency in the Management of Shoulder Dystocia With

Simulation TrainingDeering S, et al, Obstet Gynecol 2004

• METHODS: Residents from 2 training programs were randomized by year-group to a training session on shoulder dystocia (SD) management that used an obstetric birthing simulator or to a control group with no specific training. Both groups were subsequently tested on a standardized SD scenario, and a physician grader rated the resident's performance with a standardized evaluation sheet.

Example: Hypothesis• What is the research question?• What is the null hypothesis?• What is the research study design?

– Basic or Applied?– Qualitative or Quantitative?– Experimental or Non-Experimental


Study Population

Treatment

Control

Randomize

Post-test only Control Group


nt

Measurement

Small Group Activity

• Share your research questions

• Select one research question to convert to a hypothesis

• Select the most appropriate study design

Give Your Research Impact• Investigate important questions• Connect your study to prior studies

and help build the body of work• Uses appropriate research design• Consult with a biostatistician in the

design phase• Dissemenate your results

Session content, including narrated MS Powerpoint

slides available at:

http://www.obgynknowledgebank.net

educational research 102: selecting the best study design for your research question

Documents