the basics of reading a research article
DESCRIPTION
The Basics of Reading A Research Article. Amy Gutman MD ~ EMS Medical Director [email protected] / www.TEAEMS.com. EMS Research History Definitions Breaking down research articles E vidence-based medicine Research resources Summary. - PowerPoint PPT PresentationTRANSCRIPT
The Basics of Reading A Research Article
Amy Gutman MD ~ EMS Medical [email protected] / www.TEAEMS.com
OBJECTIVESInformation Presented Meets or Exceeds NREMT Educational Standards
EMS Research History
Definitions
Breaking down research articles
Evidence-based medicine
Research resources
Summary
OVERVIEW
Much “sensationalized” but bad research Cellphones cause brain cancer Vaccines cause autism
How do you tell “good” from “bad”? Is “good” research “important”?
Review basics of reading a research paper so you can keep up with current trends in prehospital care
WHAT IS RESEARCH?
Science is systematized knowledge derived from observation & experimentation
Research is a systematic investigation to discover or establish facts by utilizing the “scientific method”
The scientific method is how researchers construct an accurate, reliable & repeatable answer to a question
SCIENTIFIC METHOD Anyone can perform &/or read research
Observe something Ask a question, or “hypothesis”, about
observation Use hypothesis to make prediction Test prediction by experimentation Modify hypothesis after reviewing
experimentation results Repeat until satisfied hypothesis
answers question Ask more questions!
HYPOTHESIS & PREDICTIONS
A hypothesis is a question or theory based upon an observation
Often common sense, but common sense is neither common, nor true
Examples: “Why do I get long-distance transports at
the end of my shift?” “Does anyone really drink ‘two beers’”?
SCIENTIFIC METHOD: EXAMPLE Observation, Hypothesis & Prediction
“My Medical Director is a an awesome, short, chunky, red-headed female. Therefore I believe that ALL awesome Medical Directors are short, chunky, red-headed females”
Testing the Hypothesis / Methods: “According to providers I asked, there are at least two awesome Cape Cod Medical Directors
who are not red-headed, short, chunky or female”
Results: “Because there are awesome Medical Directors who didn’t meet the hypothesis, I modified
the hypothesis to “One awesome Medical Director is short, chunky & red-headed, but not all awesome Medical Directors must be physically similar in appearance”
Reproducibility & Clinical Applicability: Easily be reproduced in any prehospital environment Medical Director physical characteristics may not indicate awesomeness
EMS RESEARCH
The goal of EMS research is integrating results into clinical practice after an evidence-based initiative to improve patient outcomes in a competitive & cost-conscious healthcare market
• Domains: Clinical: Study of direct patient care
Systems: Effects of EMS system design & operations on resource utilization
Educational: Training methodology for prehospital providers
EMS RESEARCH HISTORY
EMS practices initially modeled after battlefield/hospital care 1st paper published in 1966 (BMJ) showed patients “suffocated” due to poor EMS training
1973 EMS Systems Act (Public Law 93-154) funded development of regional EMS systems, protocols & research Pre-1980 only 3 “Efficacy Studies” published that evaluated if a practice or drug works in
ideal conditions (but not necessarily in the field!) A 1989 editorial stated “Impressively Deficient” data for “efficacy of scope of EMS practice”
1993 NHTSA FEMSC “EMS for Children Report” identified need for pediatric research
2001 NHTSA National EMS Database founded to answer research questions 1st paper evaluated prehospital ETI in pediatric cardiac arrest & found that most providers
lacked training & that patients worse outcomes
Today: better funding, peer-reviewed journals & some clinically relevant research
EMS RESEARCH BARRIERS
Few trained researchers with minimal funding
Failure to translate research into clinical practice
Lack of integrated IT systems linking prehospital data with patient outcomes Multiple locations / personnel / researchers HIPPA, IRBs
Unstable patients often w/o diagnosis at time research conducted Uncontrolled research environment Critical nature limits practical & ethical experimental interventions Difficulty in obtaining informed consent
CURRENT NATIONAL EMS RESEARCH (www.EMS.GOV/Research)
National EMS Assessment (2009-2011) analyzed 20,000 agencies, 82,000 vehicles, 826,000 personnel
EMS Evidence-Based Guidelines
National EMS Information System (NEMSIS)
Longitudinal EMS Demographic Study
National EMS Research Agenda
National EMS Assessment
EMS Workforce for the 21st Century: Feasibility for Safety & Surveillance
EMS Systems Configuration Study
EMS Performance Measures & Outcomes Evaluation
Motor Vehicle Occupant Safety Survey: Crash Injury & EMS
EMS RESEARCH ORGANIZATIONS / WEBSITES
PREHOSPITAL RESEARCH JOURNALS JEMS is most widely-read print journal
Not peer-reviewed Opinions, summaries & commentaries that are
informative but not necessarily evidence-based
Peer-Reviewed Journals Prehospital Emergency Care Prehospital & Disaster Medicine Annals of Emergency Medicine Pediatric Emergency Care Air Medical Journal
EVIDENCE-BASED MEDICINE (EBM)
Scientific method to answer a question & make patient care decisions
Establish “standards of care” after multiple studies replicate similar results, or a multi-center study shows undeniable results Example: Maine Selective Spinal Clearance
Study
In cost-cutting age, insurance often will not reimburse for non-EBM practices Litigation possible if non-EBM procedures
PEER-REVIEWED RESEARCH
Rigorous review by persons of similar credentials to researchers
Maintains standards of quality & provides credibility by critically evaluating research & exposing flaws prior to publication
A review will find the article: Acceptable (rare) Acceptable but requires revisions (common) Not acceptable (common)
HOW AN ARTICLE GETS TO PRINT (TIMELINE)
Days 1-7 Hypothesis Formulated Days 1-60 Methodology formulated; Funding
application begins Days 30-120 IRB proposal written, submitted,
resubmitted Months 4-5 Identify & recruit investigators &
subjects Month 6-9 Investigator training; Research,
data collection & article prep begins (background, methodology)
Months 9-? For RCTs 12-24 months, Chart Reviews usually 3-6 months
Year 2+ Abstract prepared & submitted for conference presentation; article prepared for submission
Year 2+ 8 Mo If accepted for revisions, editing begins. If refused, hunt for another
journal begins Year 2+ 10 Mo Article accepted; published 3-6
mos later unless special circumstances
• Peer-reviewed EBM articles appear in many journals (some prestigious, some not)• Most research happens years before publication
JUDGING A RESEARCH PAPER Credible source? Applicable to EMS? Design appropriate to answer question
being asked? Variables controlled to ensure
reliability of results? Results conclusive / logical? Alternative explanations for results? Do findings impact current evidence or
require changes in clinical practice? How biased is the research?
BASIC VS APPLIED RESEARCHBASIC APPLIED / CLINICAL
Performed in lab or clinical setting as preliminary data collection to refine a hypothesis
Independent variables manipulated to observe & describe effect(s) on a dependent variable
Often examines properties of drugs or used in development of diagnostic tests
Every drug starts as a chemical whose properties are examined through experimentation
Real subjects in real-world situations
Effects of a drug, device, or procedure studied on humans
Test & prove (or disprove) therapies, procedures & protocols
LEVELS OF SCIENTIFIC EVIDENCE
Ia: Randomized Controlled Trial (RCT) meta-analysis
Ib: RCT (single)
IIa: Prospective Non-RCT
IIb: Experimental study (case series, animal studies)
III: Retrospective or Descriptive studies (case-control, comparative,
correlation studies)
IV: Expert opinions, extrapolations
V: Rational conjecture
Strongest
Weakest
RANDOMIZED CONTROLLED TRIAL (RCT)
Subjects randomized into control & experimental groups
Groups evaluating for variables & outcomes of interest Example: Outcomes of STEMI patients
given aspirin vs those not given aspirin
Uses the scientific method & most “valid” research method
RCT EXAMPLE Mattox KL, Bickell B, Pepe PE*, Burch J, Feliciano D.
Prospective MAST study in 911 patients. J. Trauma. 1989;29:1104-12
911 adult trauma pts with SBP <90mmHg randomized by alternate days & transported to Ben Taub Experimental Variable: MAST or no-MAST Dependent Variable: survival from trauma Independent Variables: etiology, age, race, sex, location of injury, trauma scores,
injury severity scores
Results: Mortality 31% MAST group, 25% in non-MAST group
Conclusions: MAST trousers in trauma pts increases mortality
RCT: COHORT STUDIES
Subjects with a condition or receiving a treatment compared with another group not affected by condition / treatment under investigation
Different than a double blind study as this is an observational study only No intervention by the researchers
EXAMPLE: COHORT STUDY
Gottesman BE, Gutman A et al. Radiation exposure in emergency physicians working in an urban ED: a prospective cohort study*.
HYPOTHESIS: EM MDs are exposed to radiation levels above NCRP limits
METHODS: Prospective cohort study conducted at a level I Trauma ED. Dosimeter radiation badges placed on all ED MDs & their phones carried 24/7/365. At the end of the study exposure dose for each subject calculated
RESULTS: 75 MDs enrolled. Compliance 99%. Annual extrapolated exposure for subject with highest radiation level was 50 mrem, below 5000 mrem exposure limit for health care workers
CONCLUSION: EM MDs working in an urban, academic, level I trauma center ED are not at risk of exceeding NCRP dose limits for ionizing radiation exposure
*Am J Emerg Med. 2010 Nov;28(9):1037-40
RCT: DOUBLE BLIND STUDY
Neither subject(s) nor researchers knows whether subject receiving treatment, or control (placebo) treatment
Example: Aufderheide TP, et al. Resuscitation Outcomes Consortium (ROC) PRIMED trial:
rationale & methodology for the impedance threshold device (ITD)* AIM: Compare OOHCA survival to discharge between CPR+ITD vs CPR vs fake ITD METHODS: Prospective, double-blind, RCT in OOHCA pts by EMS systems
participating in the ROC. 1.4% absolute survival difference found in CPR + ITD CONCLUSION: If ITD demonstrates hypothesized survival improvement, an
estimated 2700 deaths from OOHCA could be prevented annually*Resuscitation. 2008 Aug;78(2):179-85
SYSTEMATIC REVIEWS (RCT META-ANALYSIS)
Summary of literature & statistical analysis after detailed search to determine effectiveness of interventions / procedures
Results of multiple studies evaluating a similar question are pooled & statistically analyzed
Highest level of scientific evidence as reduces possibility of false negative results Poor quality studies are often excluded from
the statistical analysis Because patient numbers are so high (hundreds
to thousands), minimal chance of inaccurate conclusion by “chance”
ANIMAL RESEARCH Precursor to evaluating a human intervention
Computer models often initially used vs animal models for ethical reasons
Results often cannot be extrapolated to draw conclusions on what will happen in humans Isoproterenol dosing based on rat studies resulted in 3500 British asthmatics
dying from overdoses Pharma, 1971; vol 18:272
Domperidone withdrawn from market after 25 pts died from lethal arrhythmias not seen in dogs with 70 X toxic human dose Drugs, 1982, vol 24:360-4
Digitalis testing delayed in animals, while used safely for decades in humans because of toxic rodent effects Tox in vitro, 1992, vol 6:47-52
EPIDEMIOLOGICAL / CORRELATION STUDY Evaluate relationship between variables to
determine if there is any correlation “Positive”, “Negative”, “Null”
Often miscategorize “association” with “cause” Positive correlation does not mean “A” causes “B” Example: Obese persons drink soda, therefore
soda causes obesity
Rely on memory recall or data documentation, which is often incomplete or unreliable
Used in surveys or retrospective chart reviews which may lead for a more controlled study
EPIDEMIOLOGICAL / CROSS-SECTIONAL SURVEYS
Collects data at a single point in time to evaluate a policy or public health issue Example: A case of hepatitis reported at
“Restaurant Q”, so all persons who ate at that restaurant are questioned to discover if they have symptoms consistent with hepatitis
Limited by event memory, difficulty in contacting subjects
EPIDEMIOLOGICAL / LONGITUDINAL OR CASE CONTROL STUDIES
Follows subjects over a period of time, serially asking research questions for study duration Example: Framingham Heart Study has
followed residents of Framingham, MA since 1948 looking for heart disease patterns
In case control studies, subjects with a condition are compared with those who do not Observational, not interventional
therefore classified as “epidemiological”
CASE REPORTS Case Study / Case Report
Detailed info about a single subject or a small group of subjects
Examples: 1st time for new surgical technique, “weird” presentations of diseases
Case Series Report on a series of subjects with one
specific unifying factor but no control group Examples: Outcomes of pelvic trauma
patients with MAST trouser application
LITERATURE REVIEW
Exhaustive search of all relevant evidence-based literature related to a topic using multiple databases Cochrane reviews (meta-analysis) PubMed (NIH study repository) CINAHL
Asking a question of Google or Wikipedia is NOT an exhaustive (nor reliable) literature review
Accredited WikipediaResources
RETROSPECTIVE CHART REVIEWS / HISTORICAL STUDIES
Systematized searches for fact(s), then using information to describe, analyze & interpret the past
Example: Bledsoe BE, Smith MG. Medical Helicopter Accidents
in the United States: A 10-Year Review. Methods: Review all US air medical helicopter
accidents from 1997-2002 from NTSB database Results: Majority of air medical helicopter accidents
resulted from pilot error or mechanical failure
*J Trauma. 2004;56:1325-1329
EXPERT OPINIONS
Examples: Systematic reviews Narrative reviews Pure opinion pieces Position papers
Based on a literature review (including RCT meta-analysis) but considered low on the “Scientific Evidence” schematic due to not being an actual “study”
RATIONALE CONJECTURE
Lowest level of scientific validity but often accurate & important
Anecdotal reports often fit into this category, as do “interesting observations”
Often amusing due to “duh” factor
“DUH” EXAMPLE: Back MD. Knowing Your Mate Value: Sex-Specific Personality Effects on Accuracy of Expected Mate Choice*.
Males rated photos of women displaying “exploitability cues” indicating appropriateness for one-night stands vs “stability cues” appropriate for marriage or long-term dating
22 exploitability cues correlated with “sexual receptivity”, including: Lip licking/biting, appearing intoxicated, tight clothing,
obesity, dumb conversations, rubbing breasts / genitals
Conclusions: Males prefer drunk, slutty**, dumb women for 1 night stands
**Authors’ words, not mine!
*Psychol Sci. 2011 Aug;22(8):984-9.
PSEUDOSCIENCE: THINGS WE BELIEVE ARE TRUE THAT AREN’T
• Epinephrine saves lives in OOHCA
• No RCT has shown better outcomes with epinephrine vs placebo in OOHCA
• Amiodarone improves outcomes in pediatric OOHCA• Only 1 RCT suggests risk > benefit of amiodarone (not statistically significant)
• ATLS & the “Golden Hour”• Evidence suggests ATLS has less impact on outcomes than believed • Zero studies have shown anything magical about the 1st 60 minutes managing
trauma patients
ARTICLE COMPONENTS
Author Title Key Words Abstract Introduction / Background Methodology Results Discussion Conclusion References / Citations Acknowledgements
AUTHOR INFORMATION
Primary, secondary & associate authors Primary & secondary do majority of work Associates edit, collect data, produce diagrams If a multi-center, 1 author from each site listed If >10 sites, authors listed in acknowledgements
Lists primary site / contact information
Dirty secret…many “authors” have never seen the research – they are department chairs, “famous” or “friends of famous”, all of which lend credence to research (& help get it published)
TITLE
“Headline”* clearly & briefly describing what paper examines
May be misleading to attract attention
Example: “Cellphones linked to brain cancer” is sexier than … “Environmental risk factors for cancers of the brain and nervous system: the use
of ecological data to generate hypotheses.”
*Keith Wesley
KEY WORDS
Important “search terms”
Different query results from: “Prehospital” “Paramedics” “EMTs” “EMS”
ABSTRACT
Short summary of study
Distills key study elements into a few easily understood sentences
After reviewing, the reader determines if article is “worth my time to read”
INTRODUCTION / BACKGROUND
Familiarize readers with subject & research field
Examines key question & reviews current relevant literature
Provides rationale for why research is important
Presents hypothesis attempting to answer the key question
METHODOLOGY
Evaluate research quality by showing how study conducted Study design & population description Data collection & interpretation methods
Provides details so other researchers can replicate & validate findings
Reader “assumes” articles published in a peer-reviewed journal are statistically valid “Valid” data still may be misinterpreted by
the authors or media
RESULTS Objective description of data indicating
statistical significance / insignificance using raw & calculated data A “p value” <0.05 indicates result not likely to
have happened by chance, and is “probably true”
Data interpretation reserved for discussion section
Results must be interpreted by reader keeping in mind validity of the hypothesis, study methodology & author bias
DISCUSSION Study results interpreted & evaluated against existing body of literature
Attempts to connect results to bigger picture & show how results applied clinically Statistically significant findings may not have much clinical application
Author’s opinion about results is valuable but taken with a grain of salt Any bias or unexpected results reviewed & explained
No one starts a study expecting to “fail” Example: If a author states “trend towards significance”, it means that the study failed to
show a positive outcome, so the author “stretches the truth” to imply that data are really significant, but the study itself was flawed
Even well-designed / conducted studies limited by imperfect nature of EMS research
CONCLUSIONS Summarizes info with recommendations based
upon data analysis
Recommendations range from changing clinical practice to more research
Reader must weigh all parts of the study & determine if methodologically sound , clinically reproducible & valid enough to change or continue current practice
Race, gender, geography, economic status & “sensitive” factors should not be ignored due to political correctness Example: Research links asthma incidence & severity
with race/ socio-economic status; therefore providers should expect patients with more severe respiratory distress in urban, black & Hispanic areas vs rural Caucasian areas
REFERENCES / CITATIONS
Each cited author & paper plus quotes, ideas & content in a standard format
Source of further info on a topic Decisions rarely made after one study References provide further info on that
subject in which you are interested
WHAT DOES THAT CITATION MEAN?
Gottesman BE, Gutman A, et al. Radiation exposure in emergency physicians working in an urban ED: a prospective cohort study. Am J Emerg Med. 2010 Nov;28(9):1037-40. Epub 2010 Mar 9. PMID: 20825934
Authors / Researchers “And Others”; usually part of the research team
Title of study, describes the hypothesis & results
Journal Name; does not indicate if peer-reviewed
Print publication date: year, month; journal year(issue):pages
Electronic publication date: year, month, date
PubMed ID#
ACKNOWLEDGEMENTS & DISCLOSURES
Research expensive & often sponsored by a university or company
Authors must disclose financial or other support (i.e. airfare to a conference) to explain potential conflicts of interest
“Disclosures” indicate bias that may affect study’s outcome If design adheres to scientific method, little chance for bias to affect outcomes Example: If a study determines a drug to be ineffective / harmful, & if that study
were sponsored by a competitor of the drug then readers should know that there is not only bias present, but potentially a data misinterpretation
NEMSRA RECOMMENDATIONS TO IMPROVE FUTURE EMS RESEARCH
Structured training programs i.e. Fellowships
Collaborative Centers of Excellence combining investigators, resources & public policy i.e. The Resuscitation Outcomes Consortium (The “ROC”) i.e. Recognizing that prehospital care is an actual “academic” branch of medicine
Better evidence before implementing procedures, devices, or drugs i.e. More RCTs & evidence-based practice
Standardized data collection using uniform prehospital data elements i.e. Utstein criteria
The FDA & Office for Human Research Protections should re-evaluate exception from informed consent i.e. HIPPA and human subject testing exemptions
Investment in infrastructure to overcome obstacles impeding EMS research i.e. IT, funding, data-sharing
REFERENCES Pierce, L. L. Twelve steps for success in the nursing research journey. The Journal of Continuing Education in Nursing, 2009. 40, 154-162
My Athletic Life (Scientific Research 101 series). 2012. http://www.eecs.harvard.edu/~michaelm/postscripts Bledsoe B. An Introduction to EMS Research (presentatiaon). 2009 www.researchagenda.org (National EMS Research Agenda) www.naemsp.org (National Association of EMS Physicians) www.acep.org (American College of Emergency Physicians) www.rescuehouse.com (Rescue House Fire Resources) www.nhtsa.gov (National Highway Transportation Safety Agency) www.signal51group.com (Shreveport Fire Department Signal 51) www.ci.shreveport.la.us
Edenburn D. Medical research made easy (presentation). 2013
Fox J, Day S, Reynolds L, et al. Post-clinical trial survey of EMS providers: Research experience and attitudes. Air Med J. 2010;29(1):34–36
Schmidt TA, Nelson M, Daya M, et al. EMS providers’ attitudes and experiences regarding enrolling patients in clinical research trials. PEC. 2009;13(2):160–165
US Department of Health & Human Services. Code of Federal Regulations 45 CFR46. 2010
US FDA Regulations Relating to Good Clinical Practice and Clinical Trials. 2011
Ryan KJ, Brady JV, Cooke MD, et al. Belmont Report: Ethical principles and guidelines for the protection of human subjects of research. In Office of Human Subjects Research. 1979
Literature / image searches: Wikipedia, Google, Bing, www.pubmed.com
[email protected] / www.TEAEMS.com
EMS must be driven by science rather than conjecture & pseudoscience
Anyone can undertake a research project – just ask a question & try to scientifically find out the answer
EMS providers should analyze research & drawing appropriate conclusions
By understanding evidence-based medicine, providers play an active role in the advancement of patient care EMS must be driven by science