Resident Research Using a Logic Model to Design, Implement and Evaluate a

Program

Mary Jo Knobloch, MPH, CPHWREN Conference 2009

Association of American Medical Colleges Study

• Primary healthcare professionals need more training in basic research skills and increased opportunities to become involved in research

• Research education in the early years of medical training associated with continued interest in scholarly activity

Objectives

• To demonstrate use of a logic model as a framework for program development, implementation and evaluation

• To present preliminary results of 2008 and 2009 assessment of first year residents

Current Setting

• Marshfield Clinic Residency Program• Each program different in research learning approach

– 5 Dermatology – 11 General Surgery – 19 Internal Medicine – 8 Medicine/Pediatric– 11 Pediatric – 2 Pharmacy

Logic Model Framework

• Basic Framework – A living document

• Inputs

• Outputs – Activities and Participation

• Outcomes-Impact– Short – Medium – Long

UWEX Logic Model Site

• http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Short-term Outcome Measures Using Resident Research Database

THE NUMBERS -

• Number of resident projects

• Completed resident projects

• Abstracts, posters, manuscripts

• Number of active research mentors

• Diversity of project study designs– QI and public health or advocacy projects

Resident Research DatabaseTracking the Short-Term Outcomes

Medium and Long-term Outcomes

Using Assessment Tool

• Increase in knowledge and skill level related to basic research methods and processes

• Increase in knowledge related to identification of study design and statistical methods

• Increase in skill level related to literature searches

• Change in confidence level related to understanding and implementation of research

Medium-Long Term Outcomes

• Continued interest in scholarly activity

• Use of evidence to guide practice

• Ongoing provision of multiple training opportunities

Results to Date

• Projects (the numbers)

• Assessment results (the learning)

Projects

• Incidence of Eosinophilic Esophagitis in MESA• Relationship of Urine Cotinine Levels and

Wound Implications in Patients Undergoing Reduction Mammoplasty

• Effects of HMR Weight Loss Intervention Program on Cardiovascular Disease Risk Factors

• Effect of Hyperglycemia on Outcomes in Acute Exacerbation of COPD

Projects • CNS Blastomycosis in Marshfield Clinic System: A Case Series

• A Pilot Project to Promote Early Literacy

• Cardiovascular Morbidity and Mortality among SLE Patients in a Rural Population-based Cohort – ICTR project

• Severely Suppressed Bone Turnover and Atypical Skeletal Fragility• Maja Visekruna, MD, Medicine/Pediatrics

• Faculty and Resident Perspectives on the Dermatology Integrative Competency Curriculum

• Modeling Clearance of Warfarin

Projects

• Implementation of a Resident Well-Being Survey

• Hamman-Rich Syndrome (Idiopathic Acute Interstitial Pneumonitis)

• Evaluating Clinical Compliance with IDSA Guidelines for Diagnosis and Management of Lyme Disease

• Incidence and Description of Anaplasmosis

• Rapunzel Syndrome in a 5 Year-Old (Case Report)

• Risk Factors Related to Recurrence of Umbilical Hernia

Projects

• Renal Cell Carcinoma – Incidence Rates in Wisconsin

• A Retrospective Review of Patients Diagnosed with Stenosing Tenosynovitis (Trigger Finger and Thumb)

• Comparison of Wounds Before and After Sharp Debridement Using 16S DNA Sequencing

• Identification of Drug Therapy Problems through Medication Reconciliation in Heart Failure Patients

• The Role of Pharmacist Evaluation in Activation of Adult Asthma Patients (ICTR project).

• A Public Awareness Campaign about Insulin Dependent Diabetes

Projects• Natural History of Idiopathic Neonatal Hypertension and the Long

Term End Organ Effects in Children with Neonatal Hypertension

• Do Clinicians Document the AAP Recommended Clinical Information Used to Diagnose and Manage Otitis Media with Effusion?

• Defining Services and Referral Patterns for Infants with Positional Plagiocephaly within the Marshfield Clinic System

• Implementing a Pharmacy Continuing Professional Development (CPD) Project at Marshfield Clinic

• Factors Associated with Successful Matching and Residency Matriculation by Non-traditional Applicants to Dermatology Residency Programs

• Effects of Metabolic Syndrome on Disease State and Long-Term Outcomes in Patients with Colorectal Cancer

Projects

• Are Lytic Bacteriophages against Staph Aureus Found in the Anterior Nares of Humans?

• A Rare Presentation of Painless Aortic Dissection as Seizures

• Analysis of Dermatology Resident Self-Reported Successful Learning Styles and Implications for Core Competency Curriculum Development

• Caustic Ingestion in a 2-year old (Case Report)

• Assessment of Medical Residents’ Knowledge of Drug Interactions after an Educational Intervention

Combined Results for 2008 and 2009 First-Year Residents(the learning)

• Assessment of attitude, confidence level and knowledge– 33 residents

Attitude Questions

• To be an intelligent reader of the literature, it is necessary to know something about statistics– 100% agree or strongly agree

Attitude Questions

• I often use statistical information in forming opinions or making decisions– 51% disagree or strongly disagree

• Given the chance, I would like to learn more about biostatistics– 75% agree or strongly agree

Level of Confidence Questions

• Finding clinical practice guidelines for patient care questions– 57% say they have fair or little amount of

confidence

– Exploring evidence using Cochrane Database• 39% say little or no confidence

Confidence Questions

• Identifying factors that influence a study’s statistical power– 78% say fair, little or no confidence

Confidence Questions

• Interpreting P-value for a given result– 70% say fair, little or no confidence– 30% say a lot or complete confidence

Knowledge Questions

• Which of the following is true regarding p-values?– 27% correct– 73% incorrect

• Which of the following is NOT considered a vulnerable population?– 38% correct– 62% incorrect

Knowledge Questions

• Which of the following is an example of a testable null hypothesis – 40% correct– 60% incorrect

Knowledge Questions

• Which of the following is true with regard to the differences between efficacy and effectiveness– 33% correct– 66% incorrect

• Which one of the following measures of central tendency is least affected by outliers? – 37% correct – 63% incorrect

Knowledge Questions

• Which statement is not true about confidence intervals?– 36% correct– 63% incorrect

• The purpose of a double-blind or double-masked study– 97% correct– 3% incorrect

Knowledge

• These 2008/2009 first year residents had difficulty answering questions related to

– Study design – Identifying discrete versus continuous variables– Differences between efficacy and effectiveness– Definition of bias – Differences between mean, median, mode– Sensitivity and specificity– Definition of prevalence

Conclusions

• A program planning tool usually used for public health or intervention planning or grant planning can be used to set up and evaluate a resident research program

• Tracking systems must be in place

• Preliminary assessment of resident attitude and knowledge of basic epidemiology and biostatistics demonstrates the need for continued work in teaching research methods and skills

Questions?