miechv benchmark data collection and evaluation mary anne wilson, m.s.w. center for prevention...
TRANSCRIPT
![Page 1: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/1.jpg)
MIECHV Benchmark Data Collection and Evaluation
Mary Anne Wilson, M.S.W.Center for Prevention Research and DevelopmentInstitute of Government and Public AffairsUniversity of Illinois
![Page 2: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/2.jpg)
Overview
• What are the benchmarks and why are they necessary?
• Federal requirements• Evaluation and data collection• Performance measures• Continuous quality improvement (CQI)
![Page 3: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/3.jpg)
Purpose of Presentation Increase understanding for role of Benchmarks,
Benchmark Reporting Requirements Understand the reporting and data collection
requirements for the MIECHV programs Identify the roles that each organization involved
in Benchmark reporting will play, and what additional roles that CPRD can play for MIECHV and home visiting services
Identify key performance measures for MIECHV programs, both for Benchmarks and continuous quality improvement efforts
Understand the role of the field data collectors
![Page 4: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/4.jpg)
And what is not the purpose of this presentation?
![Page 5: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/5.jpg)
What is MIECHV?Maternal, Infant, Early Childhood Home Visiting
Over time, families and home visitors build partnerships and work together to:• Improve health and development.• Prevent child injuries, child abuse, neglect, or maltreatment,
and reduce emergency department visits.• Improve school readiness and achievement.• Reduce crime, including domestic violence.• Improve family economic self-sufficiency.• Improve the coordination and referrals for other community
resources and supports.
![Page 6: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/6.jpg)
How do we define success?
• Providing high-quality home visiting services that positively impact the lives of children and families enrolled in the MIECHV programs
• Meeting and exceeding Benchmarks• Findings ways to pool our knowledge collectively to improve
statewide delivery of home visiting services.
![Page 7: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/7.jpg)
How does MIECHV fit in?• MIECHV is designed to expand HV services into priority
populations:• Reside in communities in need of services & have low income• Include pregnant women who have not attained the age of 21• Have a history of child abuse• Have a history of substance abuse• Have users of tobacco products• Have a history of, or have children with, low student achievement• Have children with developmental delays or disabilities• Include members of the military
![Page 8: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/8.jpg)
MIECHV Priority Communities
1. Rockford- Winnebago County
• 65.1% White, 20.5% African American, 15.8% Hispanic2. Vermilion County
• 83.6% White, 13.3% African American, 4.4% Hispanic3. Chicago- Cook County
• 45% White, 32.9% African American, 28.9% Hispanic4. Elgin City ( border of Cook and Kane Counties)
• 69.5% White, 7.4% African American, 43.6% Hispanic5. Cicero Town- Chicago
• 51.9% White, 3.8% African American, 86.6% Hispanic6. Macon County
• 79.8% White, 16.6% African American, 2.0% Hispanic
Doula Sites: N. Lawndale, E. Garfield Park, Vermilion County, Rock Island County and Waukegan
![Page 9: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/9.jpg)
The program goals, per the HRSA website:
1. Strengthen and improve the programs and activities
2. Improve coordination of services for at-risk communities
3. Identify and provide comprehensive services to improve outcomes for families who reside in at-risk communities
![Page 10: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/10.jpg)
MIECHV Benchmarks
• There are a total of 6 Benchmark areas
• These reflect overall domains in which we expect home visiting services to have an impact.
• Each benchmark has a set of constructs, and these total 37 constructs across the 6 benchmarks
• Benchmarks were developed in the Legislation, and the constructs were developed by HRSA to address the benchmarks.
![Page 11: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/11.jpg)
Why do we have Benchmarks?
The legislation funding MIECHV required quantifiable, measurable improvements for the populations receiving services. Programs must demonstrate improvement in the following Benchmark areas:
1. Improved maternal and newborn health2. Prevention of child injuries, child abuse, neglect, or
maltreatment, and reduction of emergency department visits
3. Improvement in school readiness and achievement4. Reduction in crime or domestic violence5. Improvements in family economic self-sufficiency6. Improvements in the coordination and referrals for other
community resources and supports
![Page 12: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/12.jpg)
Where do the Benchmarks come from?
![Page 13: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/13.jpg)
Now, let’s look at the Benchmarks individually
• The Benchmarks contain constructs, specific measures to address each concept
• Reflect outcomes of home visiting services
![Page 14: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/14.jpg)
Constructs: • Prenatal Care• Prenatal use of alcohol, tobacco, or illicit
substances• Postpartum use of contraception and
Interpartum Interval• Screen for Maternal Depressive Symptoms• Duration of breastfeeding, well-child visits,
and maternal health insurance coverage
1. Improved Maternal and Newborn Health
![Page 15: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/15.jpg)
2. Prevention of Child Injuries, Child Abuse or Maltreatment and Reduction of Emergency
Department Visits
Constructs:• Visits for child and mother to the emergency
department from all causes• Information provided or training of participants on
prevention of child injuries• Incidence of child injuries requiring medical treatment• Reports of suspected maltreatment for children in the
program, reported substantiated maltreatment, and first time victims of maltreatment
![Page 16: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/16.jpg)
3. Improvements in School Readiness and Achievement
Constructs:• Parent support and knowledge for children’s learning and
development• Parenting behaviors and parent-child relationship• Parent emotional well-being or parenting stress• Child’s communication, language and emergent literacy,
cognitive skills• Child’s positive approaches to learning and attention• Child’s social behavior, emotional regulation and emotional
well-being• Child’s physical health and development
![Page 17: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/17.jpg)
4. Domestic Violence
Constructs:• Screening for domestic violence• Of families identified for the presence of domestic
violence, number of referrals made to relevant domestic violence services
![Page 18: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/18.jpg)
5. Family Economic Self-Sufficiency
Constructs:1. Household income and benefits2. Employment or education of adult (mother and
father) members of the household3. Health Insurance Status
![Page 19: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/19.jpg)
6. Coordination and Referrals for Other Community Resources and Supports
Constructs:1. Number of families identified and referred to available
community resources2. The number of families who complete referrals to available
community resources3. Number of agencies with which the home visiting provider has
a clear point of contact in the collaborating community agency 4. Number of agencies with which the home visiting provider has
established a formal memorandum of understanding
![Page 20: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/20.jpg)
What the Evaluation is and is not• The Evaluation is:
• A way to measure and address the challenges in scale-up implementation of HV services
• A way to show Congress that tax dollars spent on additional HV services is important and useful
• A way to introduce statewide continuous quality improvement efforts
• A goal setting mechanism• The Evaluation is not:
• A performance assessment to be used for funding decisions• A method to weed out the poor performing HV sites• A way to collapse the variety of model HV programs into a single
government-managed HV organization
![Page 21: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/21.jpg)
Illinois MIECHV: A Partnership Approach
CPRD is involved in the following areas:• In partnership with the state, developing measurement
procedures and systems which adequately address the Benchmark reports
• Coordinating Benchmark data collection with state and sites • Work together with home visitors to gather benchmark data
• Query data from MIECHV data system and prepare reporting to the feds annually
• Utilize data to track individuals through time in services• Collect home visiting services satisfaction information
• Help clarify possible barriers to engagement• Lead Home Visiting Continuous Quality Improvement efforts in
MIECHV and throughout the Home Visiting State System
![Page 22: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/22.jpg)
What if we don’t make improvements on the benchmarks after 3 years?
• Would need to develop a plan of action
• CPRD, the Governor’s Office, and individual community
programs would work together to develop a plan and
next steps
• The CQI process is crucial to next steps
![Page 23: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/23.jpg)
What is CQI and how will CPRD aid in the CQI efforts?
I can tell you with 95% confidence that there is less than a 65.6%
possibility that the CQI will simply generate 34.8% more meaningless
statistics.
![Page 24: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/24.jpg)
What are the cohorts mentioned in the Benchmarks?
• The term “cohort” refers to the idea of a group of families
who enter services together.
• But families are entering and exiting HV services all the time
• Cohorts in MIECHV follow one group of people and compare them
to another group enrolled a year later
• Cohorts help us measure program change and effectiveness over
time
• How will we know if individuals are changing?
• Individual change is reviewed in CQI process
![Page 25: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/25.jpg)
Who are Field Data Collectors?
• What does it mean to be a field data collector?
• How will we all stay in contact?
• How will we ensure regular communication and updates?
• Working with the home visiting sites and home visitor
• Will use a shared calendar through Google Calendar (E.g. '[email protected]‘), e-mail, call and text.
![Page 26: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/26.jpg)
The Role of Field Data Collectors in the Benchmarks
• Some Measures require rigorous approach for reliability
• The Benchmarks created a large and taxing load on home visitors
• Some benchmarks require that the parent not be hindered by watching their child, such as needing to fill out a longer assessment
![Page 27: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/27.jpg)
On-site Data Collection Field data collectors will work with home visitors to collect
measures from families Assess the families at three time periods
2-4 weeks after enrollment1 year and 2 year follow-ups
Informed Consent – depending on age or emancipation status Parental consent – if mother is less than 18 years old, written
parental/guardian consent is required and self-assent Self-consent – emancipated or 18 years or older, written assent
is required Gift cards are provided to participants
![Page 28: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/28.jpg)
What is the PICCOLO?Parenting Interactions with Children: Checklist
of Observations Linked to Outcomes
• PICCOLO items describe positive parenting
• PICCOLO helps practitioners identify parenting strengths to encourage parents to do more
![Page 29: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/29.jpg)
What does PICCOLO measure?
From the research literature, there are identified four important kinds of “developmental parenting” that support child development:
• Affection• Responsiveness• Encouragement• Teaching
![Page 30: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/30.jpg)
What does PICCOLO look like?
![Page 31: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/31.jpg)
29 THINGS PARENTS DO THAT PREDICT SCHOOL READINESS
• 1. Speak warmly 2. Smile at child 3. Praise child 4. Stay physically close to child 5. Say positive things to child 6. Interact in positive ways with child 7. Show emotional warmth 8. Pay attention to what child is doing 9. Change activities to meet child’s interests or
needs 10. Be flexible when child changes interests 11. Follow what child is trying to do 12. Respond to child’s emotions 13. Look at child when child talks or makes sounds 14. Reply to child’s words or sounds 15. Wait for child’s response after making a
suggestion
16. Encourage child to do things with toys 17. Support child’s choices 18. Help child do things on his or her own 19. Verbally encourage child’s efforts 20. Offer suggestions to help child 21. Show enthusiasm about what child does 22. Explain reasons for something to child 23. Suggest activities to build on what child is
doing 24. Repeat or expand child’s words or sounds 25. Label objects or actions for child 26. Engage in pretend play with child 27. Do activities in a sequence of steps 28. Talk about characteristics of objects 29. Ask child for information
![Page 32: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/32.jpg)
What is the KIDI?
Knowledge of Infant Development Inventory
• The theoretical base for the KIDI was the theory that parental knowledge of infant development can influence parental behavior and child rearing practices.
• The KIDI is a 58-item instrument that was designed to obtain comprehensive information on parents’ factual knowledge of parental practices, child developmental processes, and infant norms of behavior.
• The KIDI is designed to be easily accessible to persons with limited education and to be culturally neutral.
![Page 33: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/33.jpg)
What is the PSI?
The Parenting Stress Index - Short Form• The PSI-SF consists of 36 items derived from the PSI which comprise
three scales: Parental Distress, Difficult Child Characteristics, and Dysfunctional Parent-Child Interaction.
• Designed to evaluate the magnitude of stress in the parent-child system
• The PSI-SF is commonly used as a screening and triage measure for evaluating the parenting system and identifying issues that may lead to problems in the child's or parent's behavior.
• Asking parents to complete the PSI—SF can be uncomfortable. Field Data Collectors explain that it will provide information that will help identify the sources and different types of stress that every parent can experience.
![Page 34: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/34.jpg)
What is the HOME?• Home Observation for Measurement of the
Environment (HOME) is a descriptive profile which yields a systematic assessment of the caring environment in which the child is reared.
• The primary goal of the instrument is to measure, within a naturalistic context, the quality and quantity of stimulation and support available to a child in the home environment.
• Its focus is on the experience of the child in the home environment.
![Page 35: MIECHV Benchmark Data Collection and Evaluation Mary Anne Wilson, M.S.W. Center for Prevention Research and Development Institute of Government and Public](https://reader035.vdocuments.site/reader035/viewer/2022062722/56649f395503460f94c56d61/html5/thumbnails/35.jpg)
Other Resources• Visit Tracker has various recorded webinars available. New staff should watch Illinois MIECHV Training for
New Staff. https://visittrackerweb.com/ • The Governor’s Office webpage has numerous up-to-date documents posted including a MIECHV Data
Collection Manual. http://www2.illinois.gov/gov/OECD/Pages/MIECHVP.aspx • Field Data Collection brochures for each MIECHV community (contact Mary Anne Wilson for copies)• Contacts
• Teresa Kelly, Project Director, Strong Foundations Partnership- Governor's Office of Early Childhood Development 312-814-0905 [email protected]
• Lesley Schwartz, Manager of Program Evaluation- Governor's Office of Early Childhood Development 312-814-4841 [email protected]
• Joanna Su, Community Systems & Capacity Building Manager-Governor's Office of Early Childhood Development 312-814-6741 [email protected]
• Audrey Moy, Manager of Compliance- Governor's Office of Early Childhood Development 312-814-3141 [email protected]
• Peter Mulhall, Director , CPRD-University of Illinois 217-244-3231 [email protected]
• Mary Anne Wilson, MIECHV Research Project Specialist at CPRD, University of Illinois 217-333-3231 [email protected]
• Deborah Kemmerer, MIECHV CQI Specialist at CPRD, University of Illinois 217-300-4158 [email protected]