doug piper brooke blaalid manda walters population health institute school of medicine and public...
TRANSCRIPT
Doug PiperBrooke BlaalidManda Walters
Population Health InstituteSchool of Medicine and Public Health
University of Wisconsin – Madison
Webinar 10 January 2012
Population Health Institute - University of Wisconsin - Madison
SOUTH DAKOTA’S SFP SIG Project Evaluation
CSAP’s Evaluation Requirements: Community Outcomes
10 January 2012 1
Population Health Institute - University of Wisconsin - Madison
CSAP’s Evaluation Requirements: Community Outcomes
10 January 2012 2
Today’s discussion topics:
1.An overview of South Dakota SPF SIG evaluation elements
2.Focus on the Community Outcome data reporting requirements
3.Specifics of what UW PHI needs to know about each Coalition’s Community Outcome data
4.Phone conversations to record that information
5.After the main webinar topic – discussion on collecting additional PLI data.
QUESTIONS?
Population Health Institute - University of Wisconsin - Madison
10 January 2012
3
Indicators reflect a conceptual operationalization of the targeted priority, but an indicator is not the actual measure. In this project may be that there are multiple indicators for a given priority. For example, if a coalition is targeting the prevention of underage drinking as a priority, both 30-day use and age of first use might indicators of this priority. There are 3 types of indicators:Consumption patterns are the way in which people drink.
Consequences are the social, economic and health problems associated with the use of alcohol use and abuse, such as addiction, criminal activity, drinking and driving, etc.
Intervening variables are factors that have been identified as being strongly related to - and influential in - the occurrence and magnitude of substance use problems and consequences. Intervening variables are individual traits or experiences which drive their behaviors, such as risk factors and protective factors.
Environmental prevention strategies attempt to change the surroundings in which people choose to use, or not use, substances. They aim at changing environmental conditions that contribute to substance abuse. These strategies often involve changing laws, policies, and regulations to make it more difficult for people to abuse substances.
Individually-focused prevention programs usually endeavor to change individuals within the community or society by influencing risk and protective factors.
Glossary
Population Health Institute - University of Wisconsin - Madison
10 January 2012
4
Glossary (page 2)
Measure pertains to the survey item used to operationalize the indicator.
An example for coalitions using SURVEY DATA and having a priority of prevention underaged drinking, an indicator might be 30-day use of alcohol. The measure is “During the past 30 days, on how many days did you drink one or more drinks of an alcoholic beverage?”
An example for coalitions using SURVEILLANCE DATA and having a priority of drinking and driving, the measure might be alcohol-related auto crashes occurring between 10 p.m. and 6 a.m.
NOMs – the Congressionally required National Outcome Measures. Reporting on one of the NOMS is required of all coalition receiving SPF SIG funds.
Surveillance system data are statistics, records or archives used to track trends of characteristics in your community’s population. Examples of surveillance systems are enumerating events of interest to units of government – such as county health departments, schools or sheriff’s offices. In our case they are most likely to be used as indicators of environmental efforts and changes.
Survey – information taken from local or state questionnaires, whether by telephone, paper-and-pencil or web sites, which represent a specific population’s characteristics, behaviors or views.
5
The 5 SPF Steps and Coalition Work Flow
COALITION’S COMPREHENSIVE STRATEGIC PLAN
Implement EBPs Build coalition-area capacity
Increase coalition-area support of SPF
Step 4: Community capacity and mobilization
10 January 2012
a. Determine evaluation components (design)
b. EST. A DATA COLLECTION PLAN CLI Pts 1 & 2 PLI for individual EBPs COMMUNITY OUTCOMES Unique components
c. Analyze and interpret data
d. Use the evaluation results
Mobilize community/ coalition area’s support for SPF SIG priorities
Population Health Institute - University of Wisconsin - Madison
Access data on substance use and related consequences
Identify underlying causes Decide on the Coalition’s priorities
Do a gap analysis of prevention services
Conduct an environmental scan of the local substance abuse-related system
Choose which EBPs (interventions) to
implement
Increase community capacity to achieve SPF SIG priorities
Flow Chart of Required Data Elements
10 January 2012
Population Health Institute - University of Wisconsin - Madison
6
Type of EBP
(inter-vention) chosen
Type of EBP
(inter-vention) chosen
OUTCOME OBJECTIVEMEASURES1. Participant-level data (PLI)2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.
OUTCOME OBJECTIVEMEASURES1. Participant-level data (PLI)2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.
OUTCOME OBJECTIVE MEASURES1. EBP-related outcome data: changes in policy, compliance checks, social access.2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.
OUTCOME OBJECTIVE MEASURES1. EBP-related outcome data: changes in policy, compliance checks, social access.2. COMMUNITY OUTCOME DATA – EPIDEMIOLOGY DATA SHOWING CHANGE IN TARGET POPULATION.
Coalition-related objectivesCoalition-related objectives Program-related objectivesProgram-related objectives
Individual-focused
intervention
Individual-focused
intervention
PROCESS OBJECTIVE MEASURES1. CLI pt. 2 – intervention strategies2. South Dakota’s KIT reporting3. Archives – material developed
PROCESS OBJECTIVE MEASURES1. CLI pt. 2 – intervention strategies2. South Dakota’s KIT reporting3. Archives – material developed
PROCESS OBJECTIVE MEASURES1.CLI pt. 1 – Coalition Partner Activities2.UW PHI records and archives
PROCESS OBJECTIVE MEASURES1.CLI pt. 1 – Coalition Partner Activities2.UW PHI records and archives
OUTCOME OBJECTIVE MEASURES1.Coalition Capacity Checklist (completed)2.Successful completion of EBPs with target pop.
OUTCOME OBJECTIVE MEASURES1.Coalition Capacity Checklist (completed)2.Successful completion of EBPs with target pop.
Environ-mental
intervention
Environ-mental
intervention
SPF SIG CoalitionSPF SIG Coalition
QUESTIONS?
Community Outcome Data (p. 1)
10 January 2012
Population Health Institute - University of Wisconsin - Madison
7
Community epidemiological outcome data Data collected representative of the population in the Coalition’s service
area Generally gathered through local sources because existing national
sources may not include the correct measure or sample the correct geographic area
Related to SPF SIG targeted priorities and NOMs
Community outcome data should: Relate to the community’s targeted priorities Include at least one NOM for your service area Reflect the areas where SPF SIG programs are implemented
Community outcome data can: Include multiple priorities, indicators, and measures Include intervening variables
What is purpose: to assess the impact of EBPs (interventions) on priorities for the entire service area: the data is parallel to the coalition’s service area.
How is the data collected: As appropriate to the secondary data source being reported.
Who collects:
• The state evaluator (UW PHI) and the State Epidemiological Outcomes Workgroup (SEOW) will make as much data available to the local evaluators as is pertinent.
• Much of the Community Outcome data will be collected from local sources and forward to UW PHI (e.g. local school district data, sheriff’s department data).
• Local evaluators and UW PHI will cooperate to collect pertinent information.
• The UW PHI then puts the Community Outcome data into the MRT.
How often is data reported: The baseline will be from a period prior to implementing the EBP (intervention) and secondary data will be updated as it becomes available. Primary data will be reported as it becomes available.
10 January 2012
Population Health Institute - University of Wisconsin - Madison
8
Community Outcome Data (p. 2)
Review of the NOMs
10 January 2012Population Health Institute - University of
Wisconsin - Madison9
Domain: Reduced Morbidity - Abstinence from Drug Use/Alcohol Use
30 Day Use Perceived Risk/Harm of Use
Age at First Use Disapproval of Substance Use (youth only)
Domain: Employment/ Education
Perception of Workplace Policy
Daily School Attendance
Domain: Crime and Criminal Justice
Driving While Under Influence of Alcohol
Alcohol-Related Traffic Fatalities
Alcohol and Drug-Related Arrests
Domain: Social Support/ Social Connectedness
Family Communication Around Drug Use
QUESTIONS?
10 January 2012
Population Health Institute - University of Wisconsin - Madison
10
Coalitions gather local community
epi data IF necessary
and/or possible
UW PHI gathers data: From national sources (if available) From SEOW From Coalitions
DSS Prevention Program reviews
data entered by UW PHI and submits to
CSAP via MRT
Preliminary data entered into MRT for SD Prevention Program’s review
SD SEOW sends pertinent
epidemiological data to UW PHI
Finalized data is accepted by DITIC for CSAP
Community Outcome Data Flow
Coalition’s own data is available via
MRT download
QUESTIONS?
Overview of Community Outcome Data Necessary for CSAPPRIORITY AND TARGET POPULATION
Details of surveillance or event data for the MRT:1) Data source; who collected & when2) Population on which event data is based3) Geographic unit of the event4) Census of population 5) N of the population on which the event is based6)Event census population characteristics.7)How is the outcome measure calculated?
Details of survey data for the MRT:1) Data source; who collected & when2)Exact wording of the question3)Response options4)Sampling strategy5)Definition of the sample6)N of the population7)Demographic characteristics of the respondents8)Specifics of value reported as an outcome
Start of EBP – did it start prior to the SPF SIG funding or prior to the data collection?
Surveillance or event dataExisting data orNewly collected data
Survey dataExisting data orNewly collected data
An indicator reflects a conceptualization of the priority.
Type of data - consumption patterns; consequences of use; or intervening variables
Is it a NOMs measure? Is it a NOMs measure?
Individually-focused EBPsEnvironmental EBPs EBP implemented
QUESTIONS?
Discussion Guide for Baseline Data – Both Forms
10 January 2012
Population Health Institute - University of Wisconsin - Madison
12
BASIC INFORMATION
1. Coalition name and location
2. Priority
3. Indicators reflect a conceptual operationalization of the targeted priority, but an indicator is not the actual measure.
4. Is there more than one Community Outcome indicator for this priority? IF YES, what are the others?
[The PHI will need this information for each indicator entered into the federal reporting system.]
5. Indicator type: Consumption patterns. Consequences Intervening variables
6. Are you reporting survey data? IF YES, PLEASE SWITCH TO THE SURVEY FORM.
Are you reporting surveillance system data? IF YES, PLEASE CONTINUE BELOW.
Discussion Guide for Baseline Data - Surveillance
10 January 2012
Population Health Institute - University of Wisconsin - Madison
13
DEFINING THE SURVEILLANCE DATA
7. Data source:
Which surveillance system are you using as an indicator? Who or what agency collected the data? When were these data most recently collected? How often is this data collected? Who or what agency stores the data? Is the data accessible on-line?
8. What SPF SIG-funded EBP(s) or intervention addresses this priority?
9. EBPs implemented prior to data collection?
Were any SPF-funded interventions implemented in this community prior to this data collection?
10. Starting date for EBPs related to this priority
During what quarter do you expect EBPs related to this priority to be first implemented in this community?
11. A measure is the data item used to operationalize the indicator.
As an example, for coalitions with a priority of drinking and driving – and using SURVEILLANCE DATA as outcomes – the measure might be alcohol-related auto crashes occurring between 10 p.m. and 6 a.m. in the target county.
12. NOMs itemDoes this measure qualify as a NOMs item? Do you have, or expect, a waiver from CSAP regarding the
NOMs to be reported?
13. Connection between priority and measure
What is the logical connection between your priority and objectives and this indicator and measure?
Discussion Guide for Baseline Data – Both Forms
10 January 2012
Population Health Institute - University of Wisconsin - Madison
14
CHARACTERISTICS OF THE POPULATION IN THE SERVICE AREAON WHICH THE SURVEILLANCE DATA IS BASED [This usually encompasses a larger population than is
included in the surveillance data]
14. Population on which surveillance data is based
Who constitutes the population for the surveillance data being reported for this measure?
15. Geographic unit of surveillance or event
What is the geographic unit of the data being reported?
16. Surveillance census population age range: maximum and minimum
What are the maximum and minimum ages of the population on which the surveillance data is based?
17. N of population for surveillance data
How many persons comprise the population on which the surveillance data is based for this measure?
Demographic data of the population from which surveillance data comes (if known)
18. Gender % Female % Male
19 Hispanic Ethnicity % Hispanic/ Latino % Non-Hispanic/Latino
20. Race % Asian% Native Hawaiian or other Pacific Islander
% Multiracial
% American Indian or Alaska Native
% Black or African American
% White % Other
Discussion Guide for Baseline Data - Surveillance
10 January 2012
Population Health Institute - University of Wisconsin - Madison
15
OUTCOME DATA
21. Reported outcome for surveillance or event data
What is the result you are reporting for this measure?
22. Number of eventsHow many times did this event occur in the community?
23. What format will the data be in when reporting to the UW PHI?
QUESTIONS?
Discussion Guide for Baseline Data – Survey Data
10 January 2012
Population Health Institute - University of Wisconsin - Madison
16
DEFINING THE SURVEY
7. Data source:
Which survey are you using to collect data for this indicator? Who or what agency collected the data? When were these data most recently collected? How often is this data collected? Who or what agency stores the data? Is the data accessible on-line?
8. What SPF SIG-funded EBP(s) or intervention addresses this priority?
9. Were any SPF SIG-funded EBP(s) implemented in this community prior to the dates during which the survey data were collected?
During what quarter were EBPs related to this priority first implemented in this community?
10. A measure is the data item used to operationalize the indicator.
Source item For which survey item are you reporting data? What is the exact question that measures your NOM? (exact wording)
Response options What are the response options for the survey item used for this measure? (exact wording)
11. NOMs Item Does this measure qualify as a NOMs item? Do you have, or expect, a waiver from CSAP regarding the
NOMs to be reported?12. What is the logical connection between your priority and objectives and this indicator and measure?
13. What format will the data be in when reporting to the UW PHI?
BASIC INFORMATION (same as for surveillance data)
Discussion Guide for Baseline Data – Survey Data
10 January 2012
Population Health Institute - University of Wisconsin - Madison
17
POPULATION FROM WHICH THE SURVEY DATA WAS TAKEN
14. Population from which survey sample is drawn
From what population was the survey sample drawn?
15. Population age or gradeWas the survey population defined by age or grade level?
16. Survey population age range minimum and maximum
What were the minimum and maximum ages of members of the survey population?
17. Survey population grade range What grade levels were targeted by the survey?
18. N of populationApproximately how many people were in the survey population?
Survey Sample Information
19. Sampling Strategy What sampling strategy was used for this survey?
20. N of SampleHow many persons were drawn as part of the survey sample?
21. Number of Respondents How many persons participated in the survey?
22 – 24. Demographic characteristics (if known) of the targeted population from which the survey is drawn. (SAME CATEGORIES AS PREVIOUSLY DISCUSSED)
Discussion Guide for Baseline Data – Survey Data
10 January 2012
Population Health Institute - University of Wisconsin - Madison
18
OUTCOME DATA
25. Describe the survey data reported as an outcome.
Provide a brief description of the outcome being reported with this measure and, thus, being used to track changes in the related indicator. Please be specific about the behavior being assessed.
26. Value typeType of number reported (e.g., percentage, mean, rate) in the estimated value.
27. Statistical details Standard error Standard deviation
95% confidence interval
28. Number of valid responses for this specific survey item
How many respondents provided a valid response (i.e., not missing) to the survey item of interest?
29. Possible #of valid responses for specific survey item
How many respondents could have provided a valid response to the survey item of interest?
30. Denominator definitionHow is this measure’s denominator defined for calculation?
31. Denominator valueWhat is the value of the denominator for this measure calculation?
32. Calculated value of the outcome measure.
The reported outcome pertains to the actual result and related statistic you are reporting for this measure. What is the value of this measure for the community?
33. Validity comments or concerns
Please comment on any concerns about how well each measure provides an accurate and unbiased assessment.
34 – 36. Demographics of the survey respondents (SAME CATEGORIES)QUESTIONS?
Some Coalitions will need to send local data to UW PHI depending upon their EBPs (interventions) and their evaluation design. This will be worked out with each Coalition during technical assistance sessions.
Conversations with the national evaluators of the SPF SIG indicate that the most problematic areas for other states and coalitions have been:
Finding or collecting community outcome data corresponding to the Coalition’s service area.
Finding or collecting data corresponding to the target population – especially the 18 to 25 age group.
Ensuring that there is a logical or theoretical link between the community outcome data collected and the EBP (intervention) being implemented.
Ensuring that data is collected and reported consistently from year to year and across all targeted sub-populations.
10 January 2012
Population Health Institute - University of Wisconsin - Madison
19
Community Outcome Data:
Potential Pitfalls
QUESTIONS?
Help Using the Reporting Community Outcomes Data via the MRT
10 January 2012
Population Health Institute - University of Wisconsin - Madison
20
Technical difficulties or questions on the use of the web-based data entry system:DITIC Help Desk
1-888-DITIC-4-U (1-888-348-4248)[email protected]
Questions specific to the content for South Dakota:
Doug PiperSD SPF SIG Evaluation Director(608) [email protected]
Sarah Linnan SD SPF SIG Evaluation Data-base Manager(608) [email protected]
Brooke Blaalid, MSWSPF-SIG Evaluator/Trainer for Eastern SDHome office (605) 689-0478 Cell (605) [email protected]
Manda Walters, MSSPF-SIG Evaluator/Trainer for Western SDHome office (605) 791-0259Cell (605) [email protected]