united way for southeastern michigan 424 › sites › default › files › grants › 16si1… ·...
TRANSCRIPT
-
Narratives
Executive Summary
OVERVIEW: United Way for Southeastern Michigan (UWSEM), an existing grant making
institution, is the intermediary of the Bib to Backpack (BtoB) SIF initiative. BtoB is geographically
based in the Metro Detroit region of Michigan (Wayne, Oakland and Macomb counties) and is
focused on Youth Development (Technology, Early Childhood Education) This tri-county area faces
many economic, social and educational challenges, with particularly acute need in Detroit and
Pontiac. Subrecipients will include non-profit agencies that have deep roots in tri-county communities
of need and understand the cultural, environmental and social context our families are facing. Wayne
State University, Social Solutions, and Help Me Grow-Michigan will each play a key role in
implementation of strategy. UWSEM requests $1.5 million for each of 5 years. Sources of
commitment of match include Community Impact funds directed to BtoB and UWSEM Campaign
funds.
PROJECT SUMMARY OVERVIEW: The proposed SIF, Bib to Backpack, impacts society on three
levels. From a systems-building perspective, we will work with subrecipient organizations to expand
programming, strengthen connections between programs, and increase outcomes. From a
community and family perspective, we will support skill-building for parents and other adults who
care for young children to empower those adults to expand their social and resource networks and
increase their self-sufficiency and use of resources. All this work converges on the third level, that of
young children, improving their developmental readiness and setting them up for lifelong success.
THEORY OF CHANGE: We will build a parenting continuum that recruits, engages, and
empowers parents to improve their family's self-sufficiency and parenting skills and, in turn, their
children's developmental readiness with the following framework: (1) Recruit eligible families through
a network of new and existing partners and systems into The Incredible Years, REDI-P adapted Play
& Learns and Baby Box; (2) Co-create a plan with the family to address concerns identified by
engaging the families with the Family Check-up model, the Arizona Self-Sufficiency Matrix and the
Ages and Stages Questionnaire; (3) Use our existing 211 community-resource call center and 211 Care
Advocates to create a 'light-touch' care coordination system to help families stay on track and
navigate the early childhood ecosystem; (4) Continue to monitor participant families annually to
discuss progress, new priorities, and potential supports; (5) Evaluate the impact of the integrated early
childhood (EC) system on families' self-sufficiency and developmental readiness scores.
SUBRECIPIENT STRATEGY: A clear and effective Request for Qualifications (RFQ) process will
include inviting agencies in Macomb, Oakland, and Wayne counties to bid on work based on our
For Official Use Only
Page 2
-
Narratives
Theory of Change. Selection will depend on a scoring rubric and an external review panel. Clear goals
and targets that support the end-goals and the evaluation of same will be included in contracts and
reviewed. Continuing funding will be based on performance. UWSEM has a proven track record of
effective subrecipient growth by taking an asset-driven perspective and applying technical assistance
and capacity-building supports in order to expand upon an agency's strengths.
UWSEM'S ASSETS: UWSEM has in place a highly-effective, robust early childhood department,
including a network of Early Learning Communities that achieve early childhood outcomes with early
learning resources, technical assistance, training, evaluation support and funding. UWSEM's
experience, accomplishments and outcomes operating and overseeing programs in Technology
include leading 211, Michigan's largest community resource Call Center. We also have two Social
Solutions Efforts to Outcomes databases supporting our EC work and have several staff capable of
adapting the software to new data and reporting needs. UWSEM's internal Data and Evaluation Team
manages central data collection and evaluation activities for our partners and for 211. Staff and
consultants are richly experienced and well-educated in a wide range of disciplines.
Program Design
(b) PROGRAM DESIGN
(i) RATIONALE AND APPROACH
Intro: The United Way for Southeastern Michigan (UWSEM) believes that every family has the
right to the supportive networks and resources that middle- and upper-income families take for
granted. If we can increase access to supportive networks and resources for families in at-risk
communities, then the families and children in those communities will be more resilient. Parents and
other adult family members will be self-sufficient. Children will be healthy, valued and thriving.
We take a multigenerational approach to the work of early childhood (EC) well-being and
developmental readiness. By ensuring that the adults who care for children have access to the
supports they need, and that they are empowered to make the best use of those supports and to be the
voice for their young children, those adults can take better care of their children and themselves.
The proposed SIF builds on our existing early child work, "Bib to Backpack", and impacts society
on three levels. From a systems-building perspective, we will work with subrecipient organizations to
expand programming, strengthen connections between programs, and increase outcomes. From a
community and family perspective, we will support skill-building for parents and other adults who
care for young children to empower those adults to expand their networks and increase their self-
sufficiency and use of resources. This work all converges on the third level, that of young children,
For Official Use Only
Page 3
-
Narratives
improving their developmental readiness and setting them on a trajectory for lifelong success.
This body of work is conceived of as a continuum -- a series of discrete elements that lead to a
transformational change (Overton, 2013) for families and children. We will build out a continuum
that starts with pregnant mothers and ends with a connection to school entry. Continuum approaches
are ways to track, analyze, and build connectivity among families and organizations to influence
systems for successful results (Bailey & Terry, 2006). The welcoming, intentional pathways of BtoB
will empower our target populations, pregnant mothers and the adults who care for young children,
to easily access a series of evidence-based resources that will lead to positive outcomes for the children,
the adults, and our communities.
STRATEGY B: The proposed BtoB SIF will be a geographically based program focused on Youth
Development throughout Michigan's Wayne, Oakland, and Macomb counties. Within this region, it is
our mission to ensure that children 0-5 are healthy, valued, and thriving and we believe this happens
as a result of knowledgeable, responsive, and integrated parenting. Our goal, specific to that mission,
is to build a parenting continuum that will recruit, engage, and empower parents to improve their
family's self-sufficiency and, in turn, their children's developmental readiness. By building this system
on a foundation of evidence based parenting strategies and technologically driven care-coordination
supports we believe we can both educate and empower parents to improve the developmental
readiness and well-being of their children.
Theory of change: Build a parenting continuum that will recruit, engage, and empower parents to
improve their family's self-sufficiency and parenting skills and, in turn, their children's developmental
readiness. This will be accomplished by using the following framework: (1) Recruit target-population
participants through a network of new and existing partners and systems into The Incredible Years,
the Research Based Developmentally Informed Parenting Program (REDI-P) adapted Play & Learns
(REDI-PPL) and Baby Box (BB); (2) Once they are enrolled in programming, engage interested,
eligible families with the Family Check-up model (FCM) and perform full family self-sufficiency and
developmental screenings using the Arizona Self-Sufficiency Matrix (ASSM) and Ages and Stages
Questionnaire (ASQ) and co-create a plan with the family to address immediate concerns; (3) Use a
'light-touch' care coordination system to help BtoB SIF families stay on track and navigate the early
childhood ecosystem. We will use our existing 211 system and 211 Care Advocates to help monitor
participant progress and make additional referrals based on the families' plans; (4) Continue to
monitor participant families annually to discuss progress, new priorities, and potential supports; (5)
Evaluate the impact of the integrated EC system on families' self-sufficiency and developmental
For Official Use Only
Page 4
-
Narratives
readiness scores.
CRITICAL PROBLEM (RATIONALE): The tri-county area in Southeastern Michigan faces many
economic, social and educational challenges: almost half (48%) of the estimated 236,511 children 0-4
years of age live below 200% of the federal poverty rate. Detroit's unemployment rate at 11% is twice
the national average. County unemployment rates are more in line, as the Bureau for Labor Statistics
reports them in March 2016 at 6.4, 4.3 and 5.7 percent in Wayne, Oakland and Macomb, respectively.
While economic challenges are prevalent throughout the region, the need is particularly acute in
some communities, including the cities of Detroit and Pontiac. Detroit has a poverty rate of 40%, with
two-thirds of children under 18 living at less than 125% of poverty (U.S. Census Bureau, Table S1703,
2010-2014). Of adults over the age of 25, 22% do not have a high school diploma (compared to 13.6%
nationally; U.S. Census Bureau, Table S1501, 2010-2014). In Pontiac, conditions are similar with a
38% poverty rate and 61% of children under 18 living at less than 125% of the federal poverty rate
(U.S. Census Bureau, Table S1703, 2010-2014). Of adults in Pontiac, 23% do not have a high school
diploma (U.S. Census Bureau, Table S1501, 2010-2014).
Language will also be an important consideration for implementing the proposed SIF initiative. In
Macomb County, 13.5% of the population speaks a language other than English, with 42.8% speaking
English "less than very well". In Oakland and Wayne Counties, 14% and 37.2% of the population
speaks a language other than English respectively, with 32.6% and 37.2% speaking English "less than
very well". (U.S. Census Bureau, Table S1601, 2010-2014).
The social and economic challenges faced by our region have many implications for the health and
developmental readiness of our young children. Infant mortality remains a great community concern
as Detroit is over twice the national average with 13.2 infant deaths per 1,000 reported, compared to
5.9 nationally. There are 10.3 infant deaths per 1,000 reported in Pontiac and the infant mortality
rates in the three counties, Wayne, Oakland, and Macomb are 9.6, 6.3 and 5.8 respectively (Michigan
Department of Health & Human Services, 2010-2014).
The developmental readiness of our region also remains an area of deep concern. UWSEM heavily
invests in and promotes the use of the Ages and Stages Questionnaire (ASQ), a screening tool that
monitors young children's (ages 0-5) development and focuses on children's general and social-
emotional development. Investment in these screenings is important because children living in high-
risk communities suffer the most frequent and the most severe developmental delays and take longer
to recover from delays than children living in other communities. High-risk communities are
generally characterized by concomitantly high rates of poverty, unemployment and violence.
For Official Use Only
Page 5
-
Narratives
71,749 Ages and Stages Questionnaire screens have been completed on children 0-5 as of
September 2015 in the three counties and confirm this finding. In each domain, over a third of
children screened were at risk. The following is a breakdown of percentage of children at strong risk or
potential risk, combined, of each developmental domain: Communication- 36%, Gross Motor- 38%,
Fine Motor- 54%, Problem Solving- 38% and Personal/Social- 35%. According to James Heckman,
Nobel Laureate in economics, addressing these deficiencies early has vast returns both socially and
financially for both the children and society (www.heckmanequation.org).
CURRENT SOLUTIONS NOT WORKING: In these at-risk communities, there is no system that
readily connects people to all of the resources they need to support them as parents and to meet their
objectives as families. While many resources are available, they are sometimes underutilized and there
is a lack of continuity and connectedness to ensure that families can easily identify and access the
specific supports they need. Parent support programs exist in Greater Detroit, yet they are not
coordinated and connected in a way that can be monitored to better understand how to best serve our
community and achieve outcomes at scale, while efficiently stewarding our resources.
Although we have deep regional penetration around ASQ screenings, following-up and connecting
parents and caregivers with referrals and ongoing support remains a challenge and an area that
requires further investment. While each county has active pilots to better understand how to track and
support connections to referred services, few have the staff capacity to make sure that the referrals
lead to connections. Creating a connection to our 211 system under this pilot would fund our Care
Advocates to track our participants using Help Me Grow (HMG; an ASQ implementation
organization) services and perform the follow-ups necessary to ensure connection to services, while
allowing HMG staff to focus on building their network of referral agencies and analyzing screens for
referrals.
We see a similar issue in our own 211 call center. Our call center is a 24/7 referral service that
connects people with information and resources to build healthy, safe communities. The center serves
eight (8) counties in southeastern Michigan, including Wayne, Oakland and Macomb counties. Every
year the center receives approximately 300,000 calls from individuals needing assistance. 211
maintains a validated database of approximately 12,000 programs and 2,000 organizations and
agencies.
There were a total of 10,347 calls from individuals seeking individual, family, and community
assistance, not including. However, not all of these calls result in a referral for the individual for a
variety reasons. 7.7%, 18.8%, and 9.5% of calls seeking individual, family, and community assistance
For Official Use Only
Page 6
www.heckmanequation.org).�
-
Narratives
calls ended with no referral being made for Wayne, Macomb and Oakland respectively.
Additionally, not all individuals referred by 211 receive services from the agencies they were
referred to. On a 2015 211 follow-up survey comprised of 1,125 respondents, 69.4% reported not
receiving services from the referred agencies. The most common reasons respondents reported not
receiving services were 1) no funds available (17.8%), 2) agency determination pending (13.8%) and 3)
Full/At Capacity/ Waitlist (11.4%). By maintaining consistent contact with B3 families we hope to
keep them engaged with our system to work through these barriers in partnership.
Each of these programs are largely successful at what they were designed to do and we know that
with additional coordination supports we can bolster their effectiveness with our most at-risk families
through the approach outlined below.
APPROACH - PROPOSED EVIDENCE-BASED INTERVENTION COMPONENTS: These are
descriptions of the core, evidenced-based components of our BtoB continuum. Please see the
evaluation section for the justification regarding their evidence status; each core component is
currently either at a preliminary or moderate level of evidence. The section following this describes
how the components are integrated and how a family will navigate through the system under our
Theory of Change.
Family Check-Up Model (FCM): the FCM is a care management model that involves a lighter
version of care coordination than many existing models. It differs from traditional clinical models and
practice in three important ways: (1) it utilizes a health maintenance model, (2) it derives much of its
power from a comprehensive assessment, and (3) it emphasizes motivating change. In contrast to the
standard clinical model, the health maintenance approach of the FCM explicitly promotes periodic
contact with families (yearly at a minimum) over the course of key developmental transitions (Gill, A.
et al, 2008), potentially reducing coordination costs.
The FCM facilitates change through two main components: motivational interviewing and family
management practices. In working with families of young children, the FCM motivational
interviewing session is designed to elicit motivation for the parent(s) to change problematic behavior
in their child, which is often achieved by modifying parenting behavior (Forgatch et al., 2005) or
aspects of the caregiving context that compromise parenting quality. The motivational interviewing
focuses on sharing the assessment data results and working with the family to identify potential
problem areas they wish to address (Gille, A., et al, 2009). Our model includes use of the Ages and
Stages Questionnaire (ASQ), when appropriate, to empower parents with additional developmental
readiness information and referrals aimed at improving or maintaining their children's developmental
For Official Use Only
Page 7
-
Narratives
readiness. By using these techniques and focusing on priority areas for the family we hope to increase
and maintain engagement in the SIF parent continuum.
After understanding a family's motivation for change, the FCM provides options for intervention.
Options may include referrals for help with problems outside of parenting, but the focus of most
interventions address family management issues. Family management includes a collective set of
parenting skills, commonly referred to as Parent Management Training (PMT), based on social
learning principles of reinforcement and modeling (Forgatch et al., 2005; Patterson, 1982; Webster-
Stratton & Hammond, 1997). PMT has been consistently associated with improvement in parenting
and reductions in child conduct problems (Bullock & Forgatch, 2005; Patterson, Reid, & Dishion,
1992).
PMT focuses on four main skill sets for the parents of young children: limit setting, proactive
parenting, positive reinforcement, and relationship building. Using PMT typically involves providing
parents with a rationale to stimulate interest, careful explanation of new skills, and in-session practice
using role plays and practice with the child. In the FCM, PMT is applied to specific behavior problems
highlighted in the assessment. Our PMT interventions will consist of the Incredible Years Training
(IYT) and Play and Learns adapted from the REDI-P home visiting model (REDI-PPL), as well as
coordinated referrals into other UWSEM parenting classes and/or community supports to deal with
specific issues that may not be covered in the IYT or REDI-PPLs.
Incredible Years Training: This training is based on principles of video modeling, observational and
experiential learning, rehearsal and practice, individual goal setting, self-management, self-reflection
and cognitive self-control. It looks to reduce risk factors and increase protective factors through
increasing parent, teacher, and child competencies. Most of our recruited families will have self-
selected into the parenting classes and our focus will remain on the parenting components of the IYT.
Numerous randomized control group trials have been conducted over three decades by the developer
and independent investigators with parents in high risk populations (e.g., Head Start and Sure Start)
showing positive outcomes for parents.
REDI-P - REDI-P is a parent-child interaction intervention that is adapted from a preschool based
intervention. It involves a 16 "lesson" home visiting curricula which will be adapted for this proposal
to be used in play-and-learn groups (REDI-PPL). This intervention is considered to have preliminary
to moderate evidence as it has been evaluated with at least one randomized control trial comprised of
200 ethnically diverse children and families recruited from three counties in Pennsylvania. More
importantly, the components of the intervention are based upon an extensive review of the literature
For Official Use Only
Page 8
-
Narratives
on positive parent-child interactions and their link to positive child outcomes (Bierman et al, 2015).
Baby Box: Finland's universally-distributed Baby Box is literally a box outfitted with a mattress
that can be used safely as a bassinet. It is credited with helping to reduce that country's infant
mortality rate from 65 to 3 deaths per 1,000 children from 1938 to 2013. When distributed the Box is
filled with infant products and information linking new parents to critical resources.
Baby Box University: A vital component of the Baby Box program is Baby Box University. Expectant
and new parents have access to thousands of short, online videos, in many languages, from
professionals talking about safe sleep, breastfeeding, positive parenting and other important aspects of
caring for newborns. These educational opportunities are available on mobile devices 24/7 and
engagement can be tracked. Research has shown that one of the most universally impactful tools for
reducing child mortality is parent education.
THEORY OF CHANGE (TOC) and PROGRAM DESCRIPTION: Building a parenting continuum
that will recruit, engage, and empower parents to improve their family's self-sufficiency and parenting
skills will improve their children's developmental readiness. By adopting Strategy B and the above
evidenced-based interventions we can exercise greater oversight over the evaluation, increase the
chance of successful scaling by using well-known, evidenced-based national models, and family
resiliency.
Initially, families will be recruited into the SIF intervention by our partner agencies and
subrecipients and placed into one of two integrated pathways: expectant mothers and existing families
(families with children 0-5 already in the home). Specifically, we will recruit pregnant women,
parents, and families of children age 5 and under living at or below 200% or less of the FPL who reside
in Wayne, Oakland, or Macomb County. Initial recruitment will be based on a need recognized by our
partner organizations and the mother or family's interest in the Baby Box and/or our two parenting
supports: Incredible Years Training and REDI-P Play and Learns. All families participating in these
interventions will be prescreened for BtoB eligibility to ensure they are within our target population.
Subrecipients are ideally either parent-centric or multifaceted family services agencies with deep
roots in tri-county communities of need that understand the cultural, environmental, and social
context that our target families are facing. As an existing SIF intermediary, we have extensive
experience in identifying prospective partners whose work aligns with our TOC, and providing value-
added activities and supports to support their success. We will draw on this experience to ensure that
selected subrecipients are provided with capacity building technical assistance including but not
limited to: finance and accounting systems, procurement policies, National Service Criminal History
For Official Use Only
Page 9
-
Narratives
Check procedures, program goal setting and implementation, data collection and reporting, and rapid
cycle evaluation support.
Existing families meeting the BtoB SIF eligibility requirements will be recruited to fully participate
in the BtoB SIF program. Each BtoB SIF family will be engaged with the Family Check-up Model
(FCM), including a full family assessment with portions of the Arizona Self-sufficiency Matrix
(ASSM), a nationally vetted family self-sufficiency assessment, looking at relative poverty and a
supplemental child developmental screening using the ASQ. These assessments and screens will
inform the Family Plan that will be created in conjunction with a trained FCM Specialist (FCMS)
using motivational interviewing to help parents identify their family's priority areas as they discuss the
assessment results together.
At the end of the co-planning session, the FCMS discusses a menu of family-based interventions
with the caregivers. These options may include (a) regular follow-up support, either in-person or by
phone; (b) assistance with specific child behavior problems or parent issues; (c) Parent Management
Training; (d) preschool/day care consultations; and (e) community referrals with light care
coordination provided by specially trained 211 Care Advocates. The FCMS encourages the parents to
choose the level and type of services that best meet the family's needs.
Having completed the initial stages of the FCM, a family will have identified focus areas that they
wish to work on during the next 6-12 months and UWSEM will support their continued connection to
both evidenced-based SIF PMT models as well as other community referrals. The families FCM/ASSM
and ASQ results and family plan will be recorded and their activities will be tracked using the 211
system REFER and 211 care advocates. 211 care advocates, in partnership with the FCMS, will
monitor a family's progress through a centralized system put in place to send reminders, track PMT
activities, community referrals and connections, and other messages to keep families engaged with the
system. A variety of strategies will be tested to understand and improve the effectiveness of this
approach.
Text messaging was used successfully by nurse discharge planners to remain in contact with new
mothers who were discharged early. (Boe et al., 2015). The study suggested nurses were able to offer
support and information to mothers and their families in a way that enhanced the families' sense of
security and self-efficacy. The intervention included text chats with nurses as needed and automated
messages about their babies' first 4 days, delivered every 12 hours. The nurses regarded the automated
messages as a tool to stimulate both the families' curiosity and capacity to take control of their
situation. The nurses believed that the interactive links in the automated messages encouraged parents
For Official Use Only
Page 10
-
Narratives
to read additional information material in the knowledgebase or question the nurses on the postnatal
ward directly through their phones. They also felt it helped manage the vast amounts of information
provided at discharge (Boe et al., 2015). Both our 211 and ETO databases can be configured for text-
based messaging and information delivery and we will work with our IT department to design content
and messages around our BtoB focus areas, e.g. have you read to your child today?, follow this link
for a free interactive story.
Additionally, research performed by Dr. Vishal Mehta relying on randomized, controlled trials and
systematic reviews of the literature concludes that "there is now convincing data that text messaging
can influence health behavior and decisions." UWSEM's Healthy Kids staff have developed a similar
text-messaging system to help people find locations and set reminders for meal times at their food sites
around MI.
Engaging Expectant Mothers: Expectant mothers will largely be recruited using the UWSEM Baby
Box that will be promoted through partner organizations, including hospitals, family service
organizations, WIC offices, Nurse Family Partnerships, Maternal Infant Health programs, Early Head
Start programs and others who will inform expectant or new mothers of the box and provide them
with a link to three mandatory Baby Box University classes (approx. 5 minutes each). We will select
three pilot cities within our tri-county regions for universal deployment, meaning there will be no
restrictions on who may receive a Baby Box. Universal deployment is important because it removes
any implied stigma around receiving the box if every mother in the three initial test cities receives one.
Upon completion of these classes, families will receive an electronic certificate of completion which
they will present to a Baby Box distribution partner (all BtoB partners will be Baby Box distribution
partners) and receive their box. BtoB sites embedded in our most at-risk neighborhoods will then use
this contact point to recruit the mothers into BtoB SIF. Those interested mothers screened as eligible
will be enrolled into the BtoB SIF program and the FCM for additional assessment. It is expected that
UWSEM will distribute approximately 12,000 Baby Boxes each year, casting a wide net for
recruitment.
In addition to the initial entry tracks of expectant mothers and existing families, we have identified
three focus pathways that will be priorities as part of our pilot continuum. These are connections to
health care services, including insurance and primary care; connections to nutrition programming;
and connections to Help Me Grow's referral network. By addressing each of these early we can allow
the family to begin to focus on other social determinants and our parenting supports.
Connections to Primary Care: UWSEM recognizes that connection to consistent and quality
For Official Use Only
Page 11
-
Narratives
primary care is important for both maternal and child health and all BtoB SIF participants will be
screened on insurance and primary care status to determine if a referral needs to be made. We are
actively engaged in a pilot test using 211 Care Advocates to screen 211 callers for potential connection
to a primary care physician, health insurance, and other health services through our Children's
Healthcare Access Programs (CHAP) partners. CHAPs provide families with resources, programs,
services and specific health supports based on the callers' family needs and interests. Under this model,
a 211 Care Advocate addresses the caller's immediate reason for calling by making the needed referral;
however, the 211 CA will also ask probing questions to ascertain if a CHAP connection would be
worthwhile. The 211 Care Advocate then gives the caller general information about CHAP. Consent is
provided by the caller to get connected to a Virtual CHAP Specialist or local CHAP program, based
upon the county they reside in. The CHAP partner then helps connect the family to needed health
services. Under the BtoB model the connections and activities supported by CHAP will be recorded and
monitored to ensure that referral connections are being made and any barriers to services addressed.
Connections to Help Me Grow: We have a longstanding partnership with Help Me Grow -- a
national affiliate organization supporting the use of the ASQ -- to provide screening and referral
services for families using the ASQ. They currently have a network of approximately 400
organizations to help provide ASQ screenings and developmental services for families. Aside from a
few pilots to track referrals to pre-school and Head Start, their ability to monitor connections to
referred services is limited. Under this BtoB model if a family chooses to connect with HMG as a result
of a ASQ screen, our 211 Care Advocate would be able to record that referral and follow up to see if the
connection was made and work to remove any barriers if it was not, sharing back this information
with HMG regularly to help maintain an active network of agencies.
Connections to Nutrition Programming: Lastly, we intend to build on the work of our very
successful Meet Up and Eat Up summer meals program. Research indicates that proper parental
modeling during the toddler years plays a significant role in establishing longer-term eating behaviors
(Birch et al, 2007). We intend to use Cooking Matters to help our families learn how prepare healthy
meals on a budget. The first-ever long term study of Cooking Matters showed the six-week cooking,
shopping and nutrition course has a powerful, sustained impact that is significantly greater than
changes that would have occurred without an intervention. After the course, families felt more
confident that they will be able to afford enough food, were eating healthier, putting them at lower
risk for diet-related diseases like obesity, diabetes, heart disease and cancer, and cooking meals more
often, and making meals healthier and more budget-friendly. Additionally, there was a 10% increase
For Official Use Only
Page 12
-
Narratives
in families that are more confident in their cooking abilities and an 11% decrease in barriers seen by
families to making healthy, affordable meals. Cooking Matters is the nutrition education program of
Share Our Strength's (a current SIF Intermediary) No Kid Hungry campaign, an initiative to end
childhood hunger in America.
ALIGNMENT TO CURRENT SIF: We are not applying for renewed funding of our current 2011
SIF project, but UWSEM is a current recipient of other federal funding, and this proposal is an
opportunity to supplement and integrate those existing bodies of work into our continuum.
Specifically: (1) As a SIF intermediary (Youth Development), we fund five subrecipients to test
innovative approaches to kindergarten readiness. Approaches include family/child-centric
interventions, interventions focused on immigrant communities, and a healthy eating intervention.
This 2016 SIF proposal is an opportunity to create a true continuum that has the potential to connect
with the 2011 interventions. (2) As a SIF subrecipient of the intermediary Share Our Strength
(Healthy Futures), UWSEM is establishing regional collaborative tables to engage key local leaders in
each of the project's six geographic program target areas. They are working to provide comprehensive
outreach for their "Meet Up and Eat Up" summer meals program, contracting with breakfast coaches
to encourage alternative breakfast models, and recruiting year-round sites to serve both summer and
after school meals. Eventual integration with this network is a mid-term goal for our scaling efforts.
(3) As the state of Michigan's implementation partner for the Race to the Top - Early Learning
Challenge in Macomb, Oakland, and Wayne Counties, we work closely with licensed and unlicensed
childcare providers to ensure that children receive quality care. This SIF proposal further supplements
that work by ensuring that parents are provided with the supports they need to parent effectively and
understand the need for quality childcare.
Creating a coordinated system that can eventually incorporate all of our UWSEM programming
will help support and expand each of these programs, by helping ensure families that need the
supports offered by these programs connect with these programs.
In creating a continuum of adult-focused supports for children's earliest years, BtoB SIF will
ultimately incorporate and/or connect to all of the UWSEM's interventions that make up our existing
body of work in early childhood, Bib to Backpack. With appropriate data collection, care coordination,
and referrals, we will ensure that families empowered by BtoB SIF are invited to other interventions.
Each intervention will become part of a fully-integrated system supporting young children and their
families. Parents and other adult family members will be self-sufficient. Children will be healthy,
valued and thriving.
For Official Use Only
Page 13
-
Narratives
ii. Proposal for Subrecipient Selection
SUBRECIPIENT PROFILE: Subrecipients best suited to implement our approach would share a
profile that demonstrates the ability to implement parent-focused interventions with fidelity.
Investments will be made in tri-county nonprofit agencies that have the capacity to recruit and retain
participants, to undertake data collection, to help users navigate through the social services system,
and to integrate referrals across the continuum for the best possible user experience. Consistent with
our theory of change agencies will meet parents at whichever door they enter, and engage and
empower them to best prepare their children to succeed.
PLAN FOR SUBRECIPIENT SELECTION: Our plan to carry out a competitive selection process is
clear and comprehensive and is formulated based on extensive experience in doing so successfully.
Subrecipient agencies are ideally either parent-focused or multifaceted family services agencies with
deep roots in tri-county communities of need. We expect to fund four to six agencies with grants
ranging from $150,000 to $300,000. This will allow for a total programmatic investment of
$300,000 to $600,000 per subrecipient. Subrecipients will meet the criteria outlined in Appendix 1.
Based on our eight years' experience implementing Early Learning Communities in greater Detroit,
we are confident that budgets in this range will allow for appropriate programming, data collection,
and referral support.
As an existing SIF intermediary, we have extensive experience in identifying prospective
subrecipients whose work aligns with our Theory of Change, and providing value-added activities and
supports to support their success.
Our plan for carrying out a clear and effective bidding process will mirror our recent experience in
re-bidding our Early Learning Communities (ELCs) in 2015, using a process similar to STRATEGY B.
We provided a Theory of Change and a complete Request for Qualifications (RFQ) document to
agencies in Macomb, Oakland, and Wayne counties, inviting them to bid to undertake work as an
evolved ELC. We modeled the entire process on SIF, including a scoring rubric and an external
review panel.
A total of 13 agencies submitted bids. The review process was thorough and included scoring each
bid against a rubric, writing a narrative summary of the bid and the agency's relevant body of work,
and the convening of a volunteer Review Panel that reviewed all these materials and provided
recommendations. A UWSEM staff member with the appropriate background and expertise to enable
him/her to evaluate the bid content scored each rubric section. The scoring was consistent in that the
For Official Use Only
Page 14
-
Narratives
same person scored the same section(s) of each bid. The Review Panel included a member of
UWSEM's board, a program officer from a major funder of our early childhood work, an external
expert on our region's early childhood system, and a UWSEM Team member with expertise in
financial stability.
The process from release of the RFQ to final determination of agency partners took just over three
months. Based on this experience, we are confident in proposing the following timeline, so that the
selection process for this 2016 SIF will be complete six months following the start of our SIF
intermediary award.
Timeline: September 2016: Intermediary award start date. Draft Request for Qualifications
(RFQ), including Theory of Change, detail on specific interventions to be implemented, target zip
codes, expected capabilities of subrecipient agencies, and scoring rubric. October 2016: Solicit
volunteers for review panel, include review of potential conflicts of interest; panelists are likely to
include representatives from area universities, hospitals, foundations and parents. Release RFQ to
agencies operating in greater Detroit, including not just current and past UWSEM-funded agencies
but all those that may be eligible to apply. November 2016: Letter of Interest deadline with 25-30
LOIs expected. Technical assistance (TA) workshops (in-person) conducted. December 2016:
Subrecipient bids due; 15-20 bids expected. Initial intake and review of bids to ensure completeness
and eligibility. January 2017: Site visits by staff and review panel. Convene review panel for final
funding recommendations. February 2017: Staff review of funding recommendations, including
consideration of potential conflicts of interest. Notify bidders and begin contracting process.
IMPLEMENT PROCESS: In addition to our current SIF initiative, we have demonstrated a
selection process with a high likelihood of identifying high performing subrecipients that meet the
criteria outlined in Appendix 1 through the process used to select grantees for the Evolved ELC project.
The process included details on number of awards, funding levels and associated criteria, range of
implementation, and evaluation stipulations. Bidders, in turn, provided data-informed proposals
demonstrating a capacity to implement the RFQ with fidelity. TA, written and disseminated, on-line
and in-person regarding the process was provided.
UWSEM has extensive experience selecting and awarding competitive grants to non-profits. In the
last year, we awarded $12.2 million Community Impact investment dollars in the tri-county area.
HIGH PERFORMING RECIPIENTS UNDER STRATEGY B: Under Strategy B, we will select
subrecipients who are well suited to implement the predefined intervention. Successful applicants will
demonstrate that the proposed interventions will be an integral component of their agency offerings,
For Official Use Only
Page 15
-
Narratives
including demonstrating how the goals of the BtoB SIF are but one part of a larger system in which
we all operate. Agencies will also be selected based on their expected ability to implement the evidence-
based interventions with fidelity.
Commitment to families with young children will be demonstrated by an agency's near perfect
alignment with UWSEM's three Early Childhood Outcomes. These outcomes are children are
developmentally ready to learn; parents and caregivers nurture children's development and
communities promote children's well-being. Agencies will demonstrate that the Early Childhood
Outcomes are integral to their body of work as a whole and that they raise awareness and inspire
action on interventions to ensure children age 0-5 are developmentally ready to learn.
To demonstrate a commitment to BtoB SIF, an agency will demonstrate a superior understanding
and alignment of BtoB with the agency's own mission. It will demonstrate an extended and successful
history, knowledge, and experience serving pregnant women and parents of young children. It will
show how this initiative directly supports or is related to the organization's mission or vision. And it
will give clear example(s) of foreseen obstacle(s) and solution(s) and how unforeseen obstacles have
been dealt with in the past.
A successful candidate will demonstrate a community presence by showing a strong track record of
success in recruiting and retaining program participants. Moreover, it will demonstrate a superior
understanding and alignment of recruitment and retention strategy with other agency efforts.
The agency must be known for, and demonstrate, making newcomers feel welcome and
encouraging parents and caregivers to return for additional enrichment. It is identifiable and easily
recognizable as a resource to the community. The agency knows the target population and provides
resources and content to meet their needs and creatively works to ensure services are accessible.
Agency's facilities are known for being easily accessible and it has measures in place to assess
participant safety and comfort while on site.
A subrecipient demonstrates the use of partnerships to deliver deeper results or to reach more
participants. Providing a list of key partners will be required. The agency helps to foster a cultural
change and movement around the importance of early childhood interventions and the importance of
parent engagement and empowerment in the community.
Implementation fidelity and data competence are critical attributes of the successful bidder. The
agency demonstrates a successful history of delivering program components in ways known to
enhance effectiveness. It demonstrates a strong capability for implementation fidelity for the model
component(s) the agency proposes to undertake. There is a process in place for monitoring model
For Official Use Only
Page 16
-
Narratives
implementation and continuously checking program quality (e.g. staff training on the model,
constant monitoring of staff and programs, etc.). The agency demonstrates strong experience
monitoring program performance. Its strategy includes measures for quality and impact and captures
participant feedback. Agency has accountability measures in place for data quality and timeliness and
has strong experience with different methods of collecting data. The subrecipient demonstrates
significant capacity for data collection and performance management including necessary staff to
manage data collection, technology to manage and store data, and policies/procedures to collect data.
Finally, the agency successfully disseminates knowledge and data to other organizations and in a
manner relevant to the EC Network.
READINESS ASSESSMENT: Specific criteria for assessing applicant readiness will include
considerations relevant to staffing, use of data, community integration and more. As an existing SIF
intermediary, we have extensive experience in providing value-added activities and supports to
subrecipient agencies. We would draw on this experience to ensure that selected subrecipients are
provided with capacity building technical assistance including but not limited to: Finance and
accounting systems, procurement policies, National Service Criminal History Check procedures,
program implementation, data collection, and reporting.
iii. Proposal for Evaluation
CAPACITY TO ENSURE EVALUATION: Our capacity to successfully evaluate our portfolio has
been demonstrated repeatedly. Specifically, we have extensive experience in managing and supporting
evaluations of past funded program models, including our current SIF program. This SIF experience
also demonstrates how UWSEM has supported recipients to improve program performance, to apply
evaluation results to decision making and to successfully complete evaluations of their respective
program models.
We are currently in year five of a current SIF grant in which subrecipients are conducting rigorous
evaluations of programs designed to increase skills and attributes related to school readiness among
children aged 0-5; increase the capacity for parents of at-risk children aged 0-5 to support school
readiness in the home; and increase community level awareness of the need to support early
childhood learning as well as increase formal and informal processes through which communities
support families and children in achieving improved school readiness.
The initial portfolio was comprised of 11 different sub-grantee partners implementing a wide array
of intervention strategies. The sub-grantee selection process was designed to ensure that interventions
For Official Use Only
Page 17
-
Narratives
ranged from community based primary prevention efforts, to secondary prevention interventions
focused on early childhood initiatives implemented through preschool, head-start, and parenting
groups, to secondary and tertiary prevention efforts focusing on intervening directly with families in
the home.
UWSEM developed a technical support system and review process, which led to the successful
development of evaluation protocols that were able to meet at least moderate levels of evidence by
year 4 of the SIF programming for all subrecipients. These evaluations involve rigorous evaluation
designs including Randomized Comparison Groups at both individual and group levels, Waitlist
Randomized Controls, and Propensity Score Matched Control Designs. In addition, all but two of the
evaluations incorporate multiple post-test repeated measures designs in conjunction with
experimental or quasi-experimental control groups.
UWSEM currently has a portfolio of 5 SIF sub-grantees which have successfully implemented
sophisticated and rigorous evaluations across a 4 year period. In addition, an overall portfolio
evaluation has successfully integrated the individual SIF evaluation data into a comprehensive
overview of SIF impact across the entire portfolio.
UWSEM has implemented a number of strategies for managing and supporting evaluation efforts
at both the portfolio and sub-grantee level. These include: Monthly Evaluator Network meetings,
bimonthly Learning Community meetings integrating SIF program discussions with evaluation
discussions, an open system of continuously available technical support for all aspects of evaluation,
annual SIF summits through which, best practices, lessons learned, and resources are shared across
SIF programs, the development of a unique universal identifier for all SIF participants to facilitate
data management at the local and portfolio level of evaluation and the creation of a data
management system and protocol for all SIF programs at the portfolio level.
We have used the model developed in the SIF program to manage and support evaluation efforts in
other initiatives. The basis for the RFQ in our Evolved ELC initiative rested in part on an evaluation
conducted by JFM Consulting. Those evaluation results, coupled with what we had learned in the
execution of the SIF model, led to a tightening of the logic model/theory of change, a reallocation of
resources, a reduction in number of partner agencies and a reinvigorated collaborative decision-
making structure. The on-going evaluation of the Help Me Grow Initiative, by Oakland University,
has scaled efforts toward proven successful strategies.
In addition to incentivizing agency partners through the RFP/RFA/RFQ process, UWSEM has
continually moved forward in building our capacity and policy of providing technical support and
For Official Use Only
Page 18
-
Narratives
resources to facilitate program partners to increase their own capacity for and organizational culture
toward rigorous evaluation. For example, UWSEM has invested in an improved data system for its
Early Childhood work. Internal and partner staff have been trained in Social Solutions' Efforts to
Outcomes database to collect and analyze data in order to make efficient resource allocation decisions,
improving program performance.
Ten years ago UWSEM began an organizational cultural and practice shift toward evidence based
decision making and data informed organizational strategy. This process has culminated in a recent
reframing of organizational strategy throughout the entire organization highlighting the importance
of applying evidence and evaluation results to decision making.
UWSEM also has developed an entire unit focused on evaluation and research relating to UWSEM
programs. The organization has committed to continuing to build capacity to use performance and
results data to assess strategies, measure results and engage in ongoing improvement efforts for
UWSEM initiatives and partnerships.
UWSEM has worked extensively with contractors to facilitate capacity building both internal to the
organization and with all of our program partners for the successful implementation and utilization of
rigorous evaluation designs and findings. Contractors include Child Trends, Data Driven Detroit,
Here2There, Ty Partridge Ph.D, and JFM Consulting.
Under our Strategy B plan, we will lead the evaluation effort of the BtoB SIF model and evaluation
design detailed below. A critical part of this plan will be assessing sub-recipient applicants for readiness
and capacity to fully engage and support this effort. Our scoring rubric for evaluating potential
program partners will include their ability to implement evidence-based interventions with fidelity and
to collect, store and use data at a level consistent with an evaluation plan that can achieve moderate
or strong evidence over a three-to-four year time period.
Since we will be leading the single evaluation plan, we envision the technical assistance needed will
be focused on ensuring that partners understand the plan and are able to support the data collection
needed. Our selection process is rigorous enough to make this determination going in. We will draw
on the SIF model of regular Learning Community meetings and the ELC model of providing data
collection TA. Our RFQ will include descriptions of the role of staff. Our experience in the Evolved
ELC RFQ supports this approach.
ASSESSING EVALUATION CAPACITY and TECHNICAL ASSISTANCE: We intend to use the
first six months of the grant period to conduct a comprehensive evaluability assessment to identify,
develop and appraise evaluation documentation and assessment procedures, implementation logistics,
For Official Use Only
Page 19
-
Narratives
data management protocols, and organizational capacity in terms of staffing, training, technological
infrastructure, and other evaluation related resources.
This is an essential part of the evaluation process as it identifies on the front end, prior to
participant enrollment and data collection, that the evaluation process can be effectively implemented
and has the capacity to yield high quality information that captures program implementation as well
as program outcomes and impact. The evaluability assessment will be comprised of at least 4 core
elements: (1) Assessment of partner agencies, (2) development of a data model, (3) development of a
data collection and management procedure, and (4) specification of specific evaluation questions at all
phases of evaluation (i.e., implementation, outcome, and impact) and analysis strategies which are
isomorphic to each evaluation question.
UWSEM will develop a scoring rubric through which partner agencies will be assessed for their
readiness to participate in a rigorous evaluation process. The rubric will include, but not be limited to:
(1) Agency experience implementing and utilizing high quality evaluations in their work; (2) a
demonstrable organizational culture which embraces evaluation and uses a data driven strategy; (3)
the technical/technological capacity for the collection and management of evaluation data; (4) the
agency's staffing and budgetary capacity to be able to participate in the BtoB SIF evaluation and meet
the expected standards of rigor.
UWSEM will base partnering and sub-grantee decisions in part on the agency's evaluation
readiness score using this rubric. UWSEM will also provide technical assistance to agencies that play a
critical role in program implementation and have adequate evaluation capacity, but would benefit
from increasing their evaluation capabilities.
UWSEM will develop and document a comprehensive data map which will identify each source of
information required to accurately assess program implementation, potential barriers to
implementation, program outcomes, and program impacts. Each information source will be
operationally defined and a quantitative indicator(s) will be identified. All data elements will be
explicitly linked to the logic model and theory of change. Further each data element will include a
description of the data source, responsibility for collection, and the procedures for obtaining,
managing, and accessing the data. In the case of evaluation outcomes and impacts which employ a
psychometric assessment tool, a full description of the measure, reliability and validity information,
and a verification that the measure meets or exceeds acceptable scientific criteria for use (e.g.,
Chronbach's alpha >.7 and demonstrated construct validity). Drawing on the data model,
implementation, outcome and impact evaluation questions will be operationally specified.
For Official Use Only
Page 20
-
Narratives
As a final step in this evaluability assessment, UWSEM will develop a training protocol to be
delivered to all evaluation project staff at UWSEM and partner agencies. This will be a comprehensive
training focusing on evaluation design, data collection and data management protocols. UWSEM will
also implement "booster" trainings on a regular basis (at least quarterly). The purpose of these booster
trainings will be to obtain feedback regarding effective evaluation related activities, barriers and/or
challenges to the evaluation process, and to ensure that any drift in the fidelity of evaluation
procedures is minimized.
Following the six-month evaluability assessment phase of the project, UWSEM will initiate an
evaluation implementation pilot phase. Evaluation procedures will be implemented on a randomly
selected group of participants with a target sample size of n=40. This will allow for a thorough
assessment of the evaluation process including, data collection, data management, and
methodological protocols prior to implementing the evaluation on a large scale basis. The piloting of
the evaluation process is intended to identify any barriers or challenges to the integrity of the
evaluation and address those issues with minimal impact on the quality of the evaluation findings.
The assessment of the pilot evaluation will involve a complete review of all aspects of the evaluation
and the use of key informant interviews (frontline staff, program leads, program participants, data
collection and management staff, and all other staff involved in the evaluation process). The
procedures developed to conduct this review will be maintained on a smaller scale throughout the
course of the entire evaluation to ensure timely feedback between all key individuals involved in the
evaluation process for continuous quality control.
In addition, special focus will be given to assessing the efficacy of program recruitment and
retention procedures and the adequacy of any propensity and/or case-control matching procedures.
Moreover, an initial data entry auditing process will be implemented in which no less than 30% of the
pilot cases will be double entered and checked for consistency. Any data entry discrepancies will be
compared to the original raw data source and corrected. If there appear to be systemic factors which
made data entry errors more likely the UWSEM evaluation team will adjust the data collection
protocol in order to resolve these issues. If the data entry error rate exceeds 10% a regular auditing of
20% of all evaluation cases will be implemented throughout the evaluation process in order to
continue to ensure the highest quality data accuracy possible without incurring the cost of complete
double entry methods.
Beginning with year 2 of the BtoB SIF project UWSEM will begin to implement the full scale
evaluation. The detailed evaluation plan will be further developed and refined during the first year
For Official Use Only
Page 21
-
Narratives
evaluability and piloting phase of the project. However, we have outlined a core set of evaluation
requirements which represent the minimum standards for the evaluation plan.
ACHIEVING MODERATE EVIDENCE with STRATEGY B: Because of the nature of the proposed
intervention program the evaluation design will inherently involve a longitudinal design. A
longitudinal design has a number of key strengths for elucidating evaluation outcomes. Specifically,
the use of longitudinal designs will allow for the assessment of intra-individual variation which is a
necessity when the research aim is to uncover the impact of intervention effects on developmental
trajectories. The proposed intervention model is heavily informed by contemporary developmental
science and the key unit of analysis is the process of change within the individual over time. Indeed,
what we are most focused on in this evaluation is the prediction and ability to change the course of
development within individuals and families as opposed to mean scores on a single measure at a given
point in time.
In addition, within the context of the longitudinal design we will employ a baseline assessment
with multiple post-intervention assessments. This design feature controls for a number of threats to
internal validity when comparing baseline and post-intervention differences. It also increases the
statistical power of the evaluation.
The key evaluation research question concerns the degree to which the intervention alters the
developmental trajectory of "at-risk" children and families in comparison to the absence of the
intervention. The longitudinal/repeated measures design will allow for the assessment of trajectories
of change, however, in order to fully address this key research question a comparison group which is
also assessed longitudinally is necessary. Complete random assignment is the gold standard design and
has the greatest degree of efficiency for drawing causal inferences. However, because we are using a
"no wrong door" approach to program enrollment a true random assignment design is impractical
from a logistical standpoint. Moreover, the comprehensive ecological nature of the proposed
intervention further restricts the feasibility of a employing a randomized control group. As such we
propose to use a propensity scoring method from which we can draw a comparable control group
from a sample of children and families served by UWSEM program partners who are not directly
involved in BtoB SIF.
The use of propensity scoring methods has been expanding in program evaluation as it provides an
alternative to randomized control trials, which are often untenable to implement (Guo & Fraser,
2013). Propensity scores are conditional probability estimates of receiving treatment, given a set of
measured covariates. This can be expressed as a binary logistic regression, where P is the probability of
For Official Use Only
Page 22
-
Narratives
person i receiving treatment T, given that person i has a specific set of values x on the vector of
conditioning or covariate variables X. These covariate variables X are potential confounds related to
self-selecting into treatment, which might be an alternative explanation for mean differences in the
outcome for those receiving treatment and those in a control group.
Using this approach, an individual propensity score can be calculated in the form of a logit, or log
odds, estimating the likelihood of receiving treatment--given the values of the covariates for that
person. Once these logits are calculated, individuals in the treatment group and the control group are
matched based on their propensity scores, or their individual logit values. Since these values are
usually not exactly the same, a method is needed to determine how close a control propensity score
needs to be to a treatment propensity score in order to be considered a match.
There are a number of methods for estimating matches, two of the most common being nearest
neighbor-matching and calculating Mahalanobis d (i.e. distance). Mahalanobis d is commonly used in
multivariate statistics to estimate the average difference in scores on a set of variables across
participants. The method has been adapted to propensity score-matching as well.
We will use the Mahalanobis d method because of the potentially large number of variables across
which we will want to match intervention and control participants. Propensity scores will be
calculated based on the following covariates: family income, age of parents, age of participating child,
number of other support programs in which the family participates, parent education, primary
language used in the home, ethnicity, and other variables identified during the evaluability and pilot
phases.
In addition we will take advantage of two features of the proposed intervention strategy which will
likely create natural control groups. The intervention model targets families during the prenatal and
neonatal periods and also incorporates a routine check-up intervention in which families are regularly
contacted and can be assessed. We anticipate that not all potential program participants will utilize the
program during the initial contact or even follow-up contacts. However, we intend to assess those
families to the extent possible thus providing a quasi-waitlist control group.
Baseline equivalence: As a result of the no wrong door policy for program entry a randomized
control group cannot be feasibly implemented. Further, there will be multiple "baseline" conditions
depending upon the manner and timing in which a family enters the program. As such, we will
rigorously assess families in the intervention and in the selected control groups to determine
equivalence at baseline entry points. All key measures will be compared between the two groups using
an independent sample t-test using an a priori alpha significance criterion of alpha=.10 in order to
For Official Use Only
Page 23
http:alpha=.10
-
Narratives
obtain a conservative estimate in the context of a large sample size. Any variables on which the
intervention and control groups are found to differ at the p=.10 level will be included as baseline
controls in all subsequent evaluation analyses comparing the two groups.
External validity: One of the central aims of the proposed BtoB SIF initiative is to bring the
intervention to scale to at least a regional scale. It is important therefore to include an explicit
assessment of the external validity of the findings. UWSEM will develop a regional demographic
model using a variety of data sets obtained through 2010 census data, Data Driven Detroit, and Kurt
Metzger and Associates. Using this demographic model an assessment tool will be developed to use
with all program participants. This measurement tool will allow for the direct comparison of the
intervention sample to the larger regional population.
Organizational Capability
Sample Size, Statistical Power, and Effect Size Considerations: Through the current SIF project
UWSEM has served, through our SIF sub-grantees, 19,000 children and families. Additionally,
through the Help Me Grow project, UWSEM has gathered data on over 40,000 children aged 0-3 in
the Metro-area and approximately 95,000 early childhood assessments using the Ages and Stages
Questionnaire. Since inception, UWSEM has provided services to approximately 29,000 adults,
through our ELC networks. As such, UWSEM has developed the capacity to recruit and assess
relatively large samples drawn from the same "at-risk" population which we are aiming to serve in the
current proposal.
Based on our working history with the currently operating components of the proposed
intervention we anticipate enrolling approximately 1,200 youth and families drawn from
approximately 12,000 screened families as well as recruiting a control group of a minimum of 800
families across the four year intervention period. In addition we intend to get a minimum of three
assessments of all outcome measures for each family. Assuming an 80% retention rate across the
three assessment periods we will have a base observation sample of 2,880 observations. This will yield
more than adequate statistical power for all proposed statistical models. However, for each proposed
analysis a detailed power analysis will be conducted. For traditional general linear model analyses
(e.g., ANOVA, MANOVA, regression analyses) statistical power will be assessed using G*power v. 3.1.
For more advanced statistical modeling (e.g., Latent Growth Models, Structural Equation Models,
Latent Class Models, etc.) statistical power will be estimated using the Monte Carlo feature of Mplus v.
7.1. This latter method allows for the specification of the statistical model to be tested with minimally
acceptable parameter estimates. These fully specified models are then simulated using Monte Carlo
For Official Use Only
Page 24
-
Narratives
methods across 1,000 simulated samples. The proportion of samples with which the path coefficients
are determined to be statistically significant at the p=.05 level are generated and if the proportion is
greater than 80% with a specified sample size, then it is determined the model has adequate statistical
power.
In addition, for all structural equation based models we will calculate power at the model level
using the root mean square error of approximation (RMSEA) fit index as the effect estimate
epsilonfor; epsilon=0 (exact fit), epsilon less than or equal to .05 (close fit), and epsilon > .05 (poor
fit). This approach estimates the likelihood of the value of epsilon; under the null and alternative
hypotheses for a given sample size N, significance level alpha; and model degrees of freedom (ratio of
observed variables to estimated parameters).
All statistical power estimates will assume alpha=.05, power =.80, and a moderate effect size of d =
.35. Generally a Cohen's d of d = .2 is considered a small effect size and d = .5 a medium effect size.
Effect is of particular importance when testing statistical models in large samples because significance
levels are exponentially proportional to sample size such that as sample sizes increase, smaller effects
are determined to be statistically significant. By explicitly reporting the effect size, the results can be
interpreted in terms of meaningfulness as opposed to statistical significance. This is particularly
relevant for applied evaluation research which has the aim of informing policy and practice. In this
context reporting very small effects to be statistically significant resulting from a large sample size will
likely result in the inefficient use of money and resources to produce a very small impact. As such, we
will not interpret findings with an effect size smaller than d=.35 which is considered to be in between a
small and medium effect.
Missing Data: All evaluation analyses will use an intent-to-treat analysis approach and incorporate
all participants in both intervention and control groups irrespective of program adherence and /or
missing assessments. Intent-to-treat analyses correct for the potential overestimation of treatment or
intervention effects which result from removing non-adhering participants from the sample.
In addition, prior to analysis all data will be assessed for patterns of missingness in order to
determine the likelihood that missing data are; (1) Missing Completely At Random (MCAR), (2)
Missing At Random (MAR); or (3) Missing Not At Random (MNAR). If at least MAR a combination
of multiple imputation techniques and the use of Full-Information Maximum Likelihood estimates
will be employed depending on the type of analysis being conducted. Under conditions of MNAR,
patterns of missingness will be included in all statistical models as a control.
Analysis strategies - Implementation and Program Fidelity Analyses: The complete set of
For Official Use Only
Page 25
http:alpha=.05
-
Narratives
evaluation research questions will be specified and refined during the evaluability and pilot phases of
the project. However, there is a core set of general evaluation questions which will be required as part
of the final evaluation.
All key program components will be tracked and evaluated to assess the degree to which the fidelity
of the intervention model was upheld and the degree to which the intervention was fully implemented
across participants. Fidelity and implementation questions will include: adequacy of training for
intervention staff, degree of adherence to intervention protocols, number of children and families
served, types of services provided, and number of referrals to partner agencies.
In addition, program participation variables will be assessed and incorporated as moderators in all
outcome and impact analyses. Because UWSEM is adopting a "no wrong door" program engagement
policy it will be important to identify participants characterized as early adopters (i.e., prenatal,
neonatal, early infancy), middle adopters (late infancy thru toddlerhood), and late adopters (preschool
-- K). Evaluation analyses will utilize discriminant function analysis to identify clusters of baseline and
demographic variables which predict adopter status (i.e., early, middle, late). Adopter status will also
be included as a moderating variable for analyses of program outcomes and impacts.
Similarly, Cox regression hazard models will be conducted to predict the likelihood of intervention
duration or the length of time a participant stays engaged with the intervention. Baseline and
demographic variables will also be included as predictors of intervention duration. Hazard models are
required for these types of analyses because incorporating a unit of time as an outcome variable
violates several core assumptions of inferential statistics (David 1972, Dickman, Sloggett et al. 2004).
Time is a monotonically increasing variable and thus cannot be treated as a random variable with a
normal distribution and independence of observation. Additionally, there will be individuals for whom
intervention participation does not end before the end of evaluation data collection. This creates an
inestimable probability distribution referred to as right data censoring which severely biases traditional
statistical tests. Survival or hazard models are explicitly designed to address these two features of
"time-to-event" data such as intervention duration.
The level of intervention engagement will also be assessed. Intervention engagement will be
operationally defined as: (1) the ratio of number of intervention services for which a participant is
eligible to the number of intervention services received; (2) the number of intervention components a
participant utilizes; and (3) a self-report index of the degree to which intervention participation is felt
to be an important part of the participant's life. A composite engagement variable will be created from
these three indice (Whether this is a latent construct or a weighted linear sum will be determined
For Official Use Only
Page 26
-
Narratives
during the pilot phase of the evaluation) and used in all outcome and impact analyses as either a
predictor or as a moderator.
Full Intervention Model v. Intervention Components: The proposed intervention model (see
LM/TOC) is a multifaceted intervention with multiple intervention components. It is anticipated that
some participants will engaged in some intervention components but not others. Indeed, the adoption
of a "no wrong door" intervention access policy ensures differential intervention utilization patterns to
some degree. To the extent possible, the evaluation strategy will attempt to capitalize on this aspect of
the intervention and evaluate intervention outcomes for both the entire intervention model as a whole
as well as individual components of the intervention as quasi-standalone interventions. This will allow
for the comparative efficacy of the integrated intervention model relative to individual intervention
components.
Outcome and Impact Analyses: As discussed above, the specification of specific evaluation research
questions, especially with respect to impact and outcome analyses is premature at this point and will
be fully developed during the evaluability and pilot phases of the evaluation. However, as with the
implementation and fidelity evaluation there are several analytic strategies which will be necessary
requirements for the final evaluation plan.
Multilevel Models: The data collected as a part of this evaluation plan will inherently be nested or
non-independent on a number of levels of analysis. Because of the longitudinal design the repeated
observations over time within each participant will by definition be non-independent. Additionally,
because components of the intervention will be delivered in a group context it is assumed that
participants within a given group will have non-independent data. There are also potential non-
independence effects due to factors such as neighborhood, classroom, hospital setting, etc. In cases
where data could potentially have a nested or non-independent structure intra-class correlations
(ICC) will be estimated for all salient variables. The ICC is an index of non-independence of
observation ranging from 0 (independent) to 1 (complete dependence). In cases where the ICC is
statistically significant, indicating non-independence all analyses will be conducted using a multilevel
modeling framework. Where level 1 represents the relationship between the independent and
dependent variable and level 2 disaggregates the independent relationship of the independent and
dependent variable and the variance due to non-independence of observation such as time nested
within person or person nested within group. These models yield unbiased estimates of relationships
between independent and dependent variables, as well as allowing for the explicit estimation of group
level predictors. This multilevel framework has also been extended beyond the regression based
For Official Use Only
Page 27
-
Narratives
framework to latent variable model frameworks as well.
Latent variable structural equation modeling (LVSEM): One of the key evaluation questions is to
test an ecological model linking social determinants of family risk to child outcomes through a
number of mediating child and family process variables. Social determinants will include; relative
poverty as measured by the Arizona Self-Sufficiency Matrix (ASSM), barriers to health care access,
and cumulative family risks. Child outcomes will include assessments of the five core areas of child
health and well-being: (1) language and communication; (2) gross and fine motor skills; (3) general
cognitive skills; (4) socioemotional well-being; and (5) physical health and nutrition. A
comprehensive set of mediating variables which are directly linked to the intervention targets in the
proposed intervention model will be assessed as well. The selection of these variables will be drawn
from an extensive review of the infant and early-childhood risk and resilience literature and will
include factors such as: Caregiver health and emotional well-being, caregiver social support,
knowledge and use of positive parenting practices, parent-child relationship quality, access to and
utilization of high quality childcare, home supports for child development. Further, important
moderating factors including; caregiver race/ethnicity, community risk factors, and the economic,
health, and family policy contexts will also be considered.
In order to fully evaluate these process models latent variable structural equation models will be
employed (MacCallum and Austin 2000, Tomarken and Waller 2005, Weston and Gore 2006, Little,
Card et al. 2007). These models have a number of advantages over regression based models. First,
they allow for more precise measurement models of key constructs by estimating latent variables
across multiple observed variables which are considered to measure the same underlying construct.
These latent variables can be represented in matrix form as a series of weighted linear sums generally
referred to as a measurement model. The second key advantage of LVSEM is that it allows for the
specification of complex relationships between latent constructs through simultaneous estimation of
structural (i.e., regression) equations. This structural part of the model can also be represented in
matrix form. Further, the path coefficients in the model can be decomposed into direct and indirect
effects to provide formal tests of mediation processes.
In order to test categorical moderating factors at the statistical model level, multi-group LVSEM
will be used in which a particular SEM will be specified for each level of the moderating variable (e.g.,
ethnicity) and all path coefficients will be constrained to be equal across moderator levels. Overall
model fit in this constrained model will be assessed and a poor fitting model will suggest moderation
effects. Subsequently, the path coefficient constraints will be removed across levels of the moderator
For Official Use Only
Page 28
-
Narratives
allowing each level to produce different path coefficient estimates which can then be compared to
better interpret the distinct statistical processes for each level of the moderator.
Mixture model latent growth modeling: Another key evaluation question will be to determine the
process of change in a number of key index variables including relative poverty and child outcomes.
Of particular interest is the assessment of trajectories of change in these variables over time as
opposed to a single point-in-time assessment. By using latent growth modeling (LGM), which is a
special case of LVSEM, intra-individual variance or growth trajectories can be estimated for each
participant and predictors of variability (e.g., treatment v. control) can be incorporated (Muthén and
Muthén 2000, Li, Duncan et al. 2001, Duncan and Duncan 2004). These models are an integration of
the multilevel models and LVSEM approaches described above in which the latent variables are
comprised of two weighted linear sums of individual scores on a given measure over time referred to
as the latent intercept and the latent slope. The means of the intercept and slope are the sample
average for a measure at a given point in time(Intercept) and the average rate of change in that
measure over time (slope). The variance of these two latent variables provides an index of inter-
individual variance in intra-individual variability over time (Nesselroade & Ram, 2004; Willett,
2004). As such, change in a variable over time is explicitly incorporated and can be both predicted by
and predictive of other salient variables in the model.
Mixture models or latent class models are a special case of latent growth models in which latent
classes C are estimated from the variance of the latent intercept and slope and represent distinct
statistical populations based on the overall level and rate of change in the outcome variables over time
(Muthén and Muthén 2000). So for example, a class may be comprised of individuals that have
relatively high initial levels of the outcome variable and show rapid reductions over time whereas
another class may be comprised of relatively high initial levels but show little to no change in outcome
over time. These latent classes can be used to identify participants that could be categorized as "high
responders", "moderate responders" and "low responders" to the intervention. These latent classes of
"responder" can then be predicted by baseline, demographic or intervention engagement/utilization
variables thus informing optimal target populations or the need to adjust the intervention for some
groups of participants based on the response likelihoods.
STRATEGY B -- CURRENT AND POTENTIAL LEVELS OF EVIDENCE:
Family Check-Up Model: The family checkup model is a motivational interviewing base