1 client level data collection: the california experience jeffrey bernstein, ms center for aids...
TRANSCRIPT
1
Client Level Data Collection: The California Experience
Jeffrey Bernstein, MSCenter for AIDS Prevention Studies, California Technical Assistance
Program
Shanna Livermore, BS
Steven R. Truax, PhDCalifornia Department of Health Services, Office of AIDS,
HIV Education and Prevention Branch, Sacramento
Presentation for the 4th Annual Center for AIDS Prevention Studies Conference
San Francisco, April 16, 2004
2
Today’s Presentation
• Overview of ELI
• What challenges have been experienced at the state and local levels?
• How does ELI work successfully in California at the state and local levels?
• How has this technology been used to inform public health decisions?
4
What is ELI?
• Evaluating Local Interventions (ELI) is a web-based evaluation system that enables California’s prevention providers to collect and access information critical for the planning and implementation of effective HIV prevention programs.
5
Background
• Evaluation is important for planning and enhancing programs and for securing resources.
• Office of AIDS (OA) has long recognized the need for evaluation and technical resources required to support it.
6
Background
• CDC Guidance on evaluation released in 1999 added additional motivation.
• CDC’s and OA’s shift in new directions
8
Benefits• Systematic data collection
– Gives the client a voice
• Consistent reporting requirements
9
Benefits
• There is a strong relationship between data collection and program planning
• Helps in grantwriting
• Helps structure your services/conversations
11
Prevention Activities Required to be Evaluated
• Individual-level interventions
• Group-level interventions
• Outreach interventions
• Prevention case management
• Health Communication
12
ELI Development Partners
• Coordination & Development– California State Office of AIDS
– Universitywide AIDS Research Program/UCOP
• Program Evaluation Technical Assistance– Center for AIDS Prevention Studies, UCSF
• Guidance & Input– Local Health Departments
– CDC
13
Strategic Planning Process for Development of ELI
• Sought input from stakeholders
• Assessed
– existing evaluation efforts
– state of the art measures, methods
– local feasibility and utility
15
Pilot Process for ELI
• ELI piloted in 5 counties• Feedback collected on
– data collection forms & protocols
– web system
– resource issues
– technical assistance needs
• Trainings and TA developed• ELI revised
16
Pilot Feedback: Key Concerns• Use of individual-level forms in outreach and
group settings• Need for additional fields, data elements, &
reports• Ability to download data• Integration of existing data collection efforts into
ELI system• Need for training and technical assistance on data
collection, system use, and use of data• Resources
17
OA Response to Pilot Feedback• Revised protocol for data collection in outreach and group
settings
• Created additional data fields, optional fields & reports
• Developed downloading capability
• Designed & implemented statewide trainings
• Developed partnership with CAPS to provide evaluation TA to support use of forms & data
• OA is available for – system technical assistance – consultation for integration of ELI forms into existing data
collection efforts
19
Prevention Interventions to be Evaluated Using ELI
• Intervention: A specific activity intended to bring about HIV risk reduction– 1 Individual-Level Interventions (ILI)
– 2 Group-Level Interventions (GLI)
– 3 Outreach
– 1 Health Communication
– 1 Prevention Case Management (PCM)
23
Overview of ELI Elements
• Agency Information
• Intervention Info.–Intervention setting–Intervention type–Referrals–Risk reduction materials
• Demographics–Gender–Race/ethnicity–Sexual orientation–Matching criteria
• Behavioral Info.– Sexual Behavior– Hepatitis & STD History– Substance Use History
• Supplemental Info.
24
What are the Matching Criteria & Why are They Important?
• Date of Birth
• First Letter of Last Name
• Gender
• Race/Ethnicity
• Zip code
25
Selecting Forms
• Discuss form selection
• Forms should be decided on by the time intervention is set-up
• Some degree of form flexibility upon Office of AIDS approval
26
ELI Data Collection Forms
• Not a script!– Provide a framework for a conversation with a
client– Optional word use and question order– Use terms familiar to and appropriate for the
client
27
Explanation of Reports
• Over 50 “canned” ELI reports available– Age by Race by Gender– Missing Fields– Intervention Effectiveness– Referrals by Risk
Raw data available
29
Reports
• Budget Report• Referrals by Risk• Gender by Race by Age Group• Sexual Orientation by Race by Age Group• STD by Race by Gender• STD Type by Race by Gender• Drugs by Race by Age Group• Materials Distributed by Risk• Missing Fields• Intervention List
34
Challenges
• Individual client data requirements in certain settings
• Federal vs. State vs. Local requirements • Information Technology is not always a
super highway• Different levels of knowledge• Requires new scope of work format• Intervention set-up: LHJ vs. CBO
35
Challenges
• Data Collection– New forms/form integration– Matching Criteria
• BIG challenge with some populations (e.g., clients with immigration issues)
37
Successes
• Increased communication
• Increased collaboration
• Increased technical assistance
• Increased training opportunities
38
Successes
• Accurate timely information about HIV prevention interventions
• Efficiency increase scope of work review
• Intervention Working Group (IWG)
40
Characteristics of Successful Interventions
• Science based
• Highly targeted
• Clearly defined goals and objectives
• Realistic goals, objectives and implementation timeline
• Focused on reducing specific risk behaviors
• Adequately trained staff
41
Characteristics of Successful Interventions
• Sensitive to target population
• Sustainable resources and support
• Flexible
• Part of community plan
• Culturally, developmentally competent
• Gender specific
42
Lessons Learned
• Secure funding ahead of time
• Lengthen and expand the pilot phase
• Be prepared for major delays
43
Lessons Learned
• Need for forms in Spanish
• Be prepared for varied knowledge levels– Including data issues
• Promote data utilization
44
Lessons Learned:Agency level data use
• Clearly identify the intended users of the data
• Develop buy-in among data stakeholders
• Identify evaluation questions meaningful to the intended users
45
Lessons Learned: TA and Training Issues
• Regional trainings require out of county travel
• Training design must support new and experienced users of ELI
• SOW vs ELI vs Real Life
• Site visits often perceived as punitive
46
Next Steps
• Outcome evaluation
• Continue on site agency level TA on intervention set up
• Data Use– More training required– Report manual
47
Next Steps
• Suggestions for next steps– What should be the focus of future ELI
trainings?– How can we make ELI more agency friendly?
48
Contact info
• Shanna Livermore, Office of [email protected]
916-449-5823
• Jeffrey Bernstein, CAPS/[email protected]
1-800-983-5550