framework for excellence assessing provider behavior change resulting from aetc and related training...
TRANSCRIPT
Framework For Excellence
Assessing Provider Behavior Change Resulting from AETC and
Related Training Activities
Facilitator: Janet Myers, Director AETC National Evaluation Center
July 27, 2004
Framework for Excellence Measuring Results
Which helps in:– Refining Site Analysis
– Marketing
– Curriculum Design
– Needs Assessment
– Course Delivery and Development
– Further Measurement and Evaluation!
Presenters
Cheryl Hamill, RN, MS, ACRN & Nancy Showers, DSWDelta Region AETCHIVQual Results 2002-2003Sample RW Title III Community Health Center in Mississippi
Mari Millery, PhDNY/NJ AETCLessons from Assessing Knowledge & Practice Outcomes of Level III Trainings
Jennifer Gray, RN, PhD & Richard Vezina, MPHTX/OK AETC, Women & HIV Symposium (JG)Pacific AETC, Asilomar Faculty Development Conference (RV)
Debbie Isenberg, MPH, CHES & Margaret Clawson, MPHSoutheast AETCIntensive On-Site Training Evaluation: A Mixed Methods Approach
Brad Boekeloo, PhD, ScMNMAETC, Delta AETCAnalysis of HIV Patient-Provider Communication
Measurement and Evaluation
Why evaluate?– To determine if the training was successful in
meeting aims (for participants and faculty)– To decide how to change training content– To improve the quality of training
Why measure provider behavior change?– To determine if training has the desired effect
on participants and ultimately, on quality of care
Kirkpatrick’s Model (from Kirkpatrick, Donald L. Evaluating Training Programs (2nd edition) 1998)
Training Level: Key Evaluation Question:
Level 1: Reaction How do participants react to the training?
Level 2: Learning To what extent do participants change attitudes, improve knowledge and/or increase skill as a result of the training?
Level 3: Behavior To what extent do changes in behavior occur because of participation in the training?
Level 4: Results What are the final results (e.g., patient perception of care or outcomes of care) that occur because of participation in the training?
The HIVQUAL Project
Nancy Showers, DSW
Delta Region AETC
The HIVQUAL Project Capacity–building and organizational support for QI Individualized on-site consultation services
– Strengthen HIV-specific QI structure – Foster leadership support for quality – Guide performance measurement – Facilitate implementation of QI projects – Train HIV staff in QI methods
Performance measurement data with comparative reports
Partnership with HRSA to support quality management in Ryan White CARE Act community-based programs
HIVQUAL Participants - 2003
Title III Title IV Total
Active 87 12 99
Independent
27 3 30
Total 114 15 129
Annual PAP Test
43
65
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
Annual Syphilis Screen
45.5
86.7
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
Hepatitis C Status Known
50
90
01020304050
60708090
100
2002 2003
Per
cent
age
of P
atie
nts
Adherence Discussed
38.5
85.7
40
93.3
20
100
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
1/1-4/305/1-8/319/1-12/31
Viral Load Every 4 Months
18.2
56.7
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
MAC Prophylaxis (CD4<50)
67.7
100
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
Annual Dental Exam
30.4
60
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
Annual Mental Health Assessment
18.2
86.7
0102030405060708090
100
2002 2003
Year
Per
cent
age
of P
atie
nts
Delta AIDS Education and Training Center (DRAETC)Mississippi LPS - Training Summary Report
Reporting period: July 1, 2002 - June 30, 2003for Targeted RW Title-Funded Community Health Centers
Cheryl Hamill, MS, RN, ACRNInstructor of MedicineResource Center Directorhttp://hivcenter.library.umc.edu
HIV/AIDS ProgramUniversity of MS Medical Center2500 North State StreetJackson, MS 39216-4505
MS LPS Training Programs Totals by Level & DisciplineFor Targeted RW Title III Funded Clinic
July 2002-03
Lessons from Assessing Knowledge and Practice Outcomes of Level III Trainings
Mari Millery, PhD
Decided to focus more outcome evaluation efforts on Level III because it is the most intensive and a high priority modality; and participants can be asked to devote time to extra paperwork
Pre-test, post-test, and 3-month follow-up surveys
Measures:– Self-rating of comfort in performing clinical
tasks– Case-based knowledge questions
1. Please rate your current level of comfort in performing the following:(Circle only one answer for each question.)
2. Mrs. Z is a 34 year-old female with HIV CDC A2 disease, CD4 300 cells/cmm and viral load 50,000 copies/ml, who presents for treatment. Which of the following is the most appropriate initial regimen?
a) Zidovudine (AZT)/stavudine (D4T)/indinavirb) Didanosine (DDI)/zalcitabine (DDC)/nevirapinec) Zidovudine (AZT)/lamivudine (3TC)/efavirenzd) Stavudine (D4T)/lamivudine (3TC)/nelfinavir/ritonavir
Very low Low Medium High Very high
Choosing an appropriate HAART regimen 1 2 3 4 5
Evaluating ongoing adherence in HIV patients
1 2 3 4 5
Deciding to change HIV medications
1 2 3 4 5
Pilot Project Results (Oct 2002-June 2003)Respondent Averages Across All Topics/Questions:
Wave 1 (n = 26), Wave 2 (n = 21), Wave 3 (n = 7)
2.5
4.2
3.5
5.8
3.5
6.3
0
1
2
3
4
5
6
7
Average Comfort Self-Rating Average Number of Correct Answers
Rati
ng
/Nu
mb
er
Co
rrect
Wave 1
Wave 2
Wave 3
Lessons Learned Can be done but getting follow-up surveys back is a
challenge Preliminary results are encouraging – self-reported
practice comfort and case-based knowledge questions appear to work as measures
Survey needs to be minimum length Dropped knowledge questions in post-test because
they were too soon after baseline – post-test focuses on feedback on training
Nature of Level III varies: intensity/length, profession trained, topics covered, etc.– Developed special versions for nurses and HepC
40 surveys collected with revised instruments this year – still working on getting all follow-up surveys back
Measuring Training Outcomes Through Qualitative Interviewing
TX/OK AETC Women & HIV Symposium (JG) and Asilomar Faculty Development Conference (RV)
Jennifer Gray, RN, PhD (JG) Richard Vezina, MPH (RV)
TX/OK AETC Pacific AETC
TX/OK AETC Women & HIV Symposium (JG)
First time region-wide symposium
Multidisciplinary planning committee Lack of knowledge about
gender-specific care
Increased # of HIV infections among women in the region.
Symposium goal: Improved care of HIV+
women
Annual region-wide training conference
125 Participants, all PAETC faculty and program staff
Conference goals: Improved skills and
knowledge among faculty/trainers
Improved training outcomes throughout region as a result of staff development
Asilomar Faculty Development
Conference (RV)
Evaluation PlansJG Email one month post to all registrants Simple open-ended questions, for all disciplines Identify how content was used with patients and shared
with peers.
RV Post-Post:
Form A: Self-assessment at end of Conference Identify skills and content learned, areas in which
to integrate new skills and content Form B: 6 month Follow-Up
Individualized telephone interviews, reviewing Form A
Focus on how skills/content were applied; barriers
Why these evaluation methods?
Able to assess at multiple levels (Kirkpatrick model): Level 2 (Learning: improved knowledge) (RV) Level 3 (Behavior: change in practices) (JG, RV)
Seeking specific content regarding conference (RV)
Limited resources and time (JG)
No existing tool found that met needs (JG)
Findings
Major Themes: (RV)
Identified high need for continued skills training
Transferred new skills/information to coworkers and employees
Barrier to continued integration: Time constraints
Major Themes: (JG) Impact on patients
13 had taught patients information learned at the symposium 3 had used info for referrals 3 system changes- i. e. assessment forms, clinical strategies
Shared information with others: 8 informally, 1 structured, 4 created materials Most common topics: medication/adherence, HIV in general
Strengths & Challenges of Methods
What went well:
Announced at end of symposium/conf. (JG, RV)
Brief instrument encouraged higher response (JG)
Longer instrument yielded rich responses (RV)
What’s Next:
Provide Incentives (JG, RV)
Change instrument Shorter, easier
instrument for higher response rate (RV)
longer instrument for greater depth (JG)
More effective confirmation of contact information (JG, RV)
Intensive On-site Training Evaluation:
A Mixed Methods Approach
Debbie Isenberg, MPH, CHESMargaret Clawson, MPH
Southeast AETC
Study Overview Main research questions
– Process and Impact (Reaction and Learning)• What was the quality of the training?• How well were learning objectives met?• What are the trainees’ intentions to change their clinical
practice?
– Outcome (Learning and Behavior)• How has the provider’s experience in the clinical training
program impacted his/her ability (if at all) to provide HIV quality care to PLWH?
Study Protocol Phase One
– Post training CQI form completed by participants
Phase Two– Recruitment packets mailed 3 months after last
IOST– Research staff contact potential participants 1
week later for interview
Phase Three– Reminder letter for 2nd interview sent 9 months
after initial interview (total 12 months post IOST)– Research staff contact participants 1 week later for
interview
Content: Phase Two and Three Written Demographic Assessment (PIF+)
Semi-Structured Phone Interview (Tape recorded)- Quantitative: participant asked to rate the effect of
training in each specific training area
- Qualitative: participant asked to give concrete examples of how training has affected their skills in the clinical area
If no effect reported, participants are asked for more explanation
Strengths and Challenges
Strengths Challenges
Quantify and qualify Timely follow-up
Flexible study design Getting forms back
Addresses Reaction, Learning and Behavior
stagesParticipants’ recall
Provides ongoing training and trainer feedback
Staff turnover
Lessons Learned Think about what motivates the training audience to
participate in the study when deciding on study design
Develop the protocol to lower respondent form and time burden
Don’t be afraid to change the protocol midway in the study if not working
Consider the resources that you have to collect and analyze the data in choosing a study design
Analysis of HIV Patient-Provider Communication
Bradley O. Boekeloo, Ph.D., Sc.M.
University of Maryland
Grant #6 H4A HA 00066-02-01 from the National Minority AIDS Education and Training Center, Health Resources and Services Administration
MethodsProviders Randomized (n=8) Brief cultural competency training vs. none
Audiotapes of HIV Visits (n=24) 3 patient visits tape recorded per physician. Tapes transcribed.
Patient Exit Questionnaire (n=24) Interviewer read patient questions and patient
answered on an answer form.
RESULTS:Randomized Trial Audiotape Observations
Study Group
Control Intervention (n=4) (n=4)
Audiotape Variables Mean + S.D. Mean + S.D.
Patient Word Count 991 + 490 1050 + 629
Length of visit (minutes) 20 + 8.3 20 + 7.2
RESULTS:Randomized Trial Exit Interview Observations (1=Very uncomfortable, 4=Very Comfortable)
Study GroupControl Intervention (n=4) (n=4)
Exit Interview Variables Mean + S.D.
Mean + S.D.
Comfort talking to Dr. about sex 3.3 + .7 3.6 + .7
Comfort talking about substance use 3.5 + .5 3.3 + 1.0
Comfort talking about medication 3.6 + .9 3.7 + .9
Hypothesis Based on Exploratory Data and Next Steps
Brief Intervention not enough for change Patients may be more comfortable discussing
medical therapy than personal risk behaviors Try to determine whether different types of
communication on audiotapes account for differences in patient comfort communicating with physician.
Presenter Contact Information NY/NJ AETC:
Mari Millery, PhD
212-305-0409
Delta Region AETC:
- Cheryl Hamill, RN, MS, ACRN
601-984-5552
- Nancy Showers, DSW
732-603-9681
Southeast AETC:
- Margaret Clawson, MPH
404-712-8448 [email protected]
- Debbie Isenberg, MPH, CHES
404-727-2931 [email protected]
Pacific AETC:
Richard Vezina, MPH
415-597-9186 [email protected]
TX/OK AETC:
Jennifer Gray, RN, PhD
817-272-2776
NMAETC, Delta AETC:
Brad Boekeloo, PhD, ScM
301-405-8546 [email protected]
ASSESS materials available at www.socio.com
AETC National Evaluation Center:
Janet Myers, PhD, MPH Director
415-597-8168 [email protected]
Conference Call Evaluation
Call 8: July 27, 2004
http://www.ihi.org/feedback/survey.asp?surveycode=AETCCall072704
Survey Code: AETCCall072704
For assistance contact: Lorna Macdonald at [email protected]