ccdo feb25 evaluating cc is
DESCRIPTION
Presented at national Weed & Seed CCDO Conference in Baltimore. Feb 25, 2010TRANSCRIPT
Evaluating community Evaluating collaborationEvaluating capacityTom Kelly Annie E. Casey Foundation
Tanja Kubas-Meyer Making Connections Providence
www.aecf.org public.me.com/[email protected] slideshare.net/tomkaecf
(410) 223-2932
Established in 1948 by Jim Casey, founder of UPS
Casey “cousins” with Casey Family Program, Marguerite Casey Foundation, Jim Casey Youth Opportunity Initiative
Mission: To improve the lives of America’s most vulnerable children
Operates foster care as Casey Family Services
Annie E. Casey Foundation
Annie E. Casey Foundation
Began grantmaking in 1980’s– State child welfare reform– Juvenile detention reform– New Futures – community/schools
Grantmaking in the 1990’s– Urban workforce – Jobs Initiative– Urban children’s mental health– Teen pregnancy prevention – Plain Talk– Rebuilding Communities Initiative
Grantmaking in the 2000’s– Community change initiative – Making
Connections– PRI’s and responsible redevelopment –
East Baltimore
Overview of Comprehensive Community Initiatives (CCIs) Goals focus on multiple levels,
including individual/family, neighborhood, organizations, community, region
Focus on building capacity of neighborhood stakeholders and local institutions
Attend to interrelated influences Multiple players, layers, systems
Challenges in Evaluating Comprehensive Community Initiatives: Outcomes are either too distal
(reduced poverty rates) or too abstract (improved community capacity)
Difficulty developing a counterfactual (e.g., Community wide intervention, other programs tackling same issue, lack of control group or comparable community)
Outcomes and theories of change shift, as community needs fluctuate and learning occurs
Unexpected outside influences
Range of Tools for a Continuum of Data Needs/Questions
Random Control
ComparisonT1 vs T2
SurveysOutcome Evaluation
Sampling
Anecdote
Focus Groups
Observations
Measuring Success
Change in PLACE Change in PEOPLE Change in COMMUNITY
CAPACITIES Change in SYSTEMS OF SUPPORT
& SERVICES Change in ENVIRONMENT &
CONTEXT Contribution of Weed & Seed -
Explain WHAT, HOW and WHERE the initiative is making a difference
RBA Model (in one slide)1. Who are we concerned about? Population
2. What conditions or change do we want for this population? Result
3. How can we measure these results? Indicators
4. How would we know if we are making progress? Performance measures
5. Who has a role to play? Partners6. What works? Best practices7. What do we propose to do? Our theory
and strategies
How Much We Did OUR
EFFORTS
How Well We Did
Quantity# of Customers # of Activities
Quality% Timely activities% Attending% Customers Satisfied $ Costs per unit% Standards met
What Difference We Made for Customers
By the #s
Changes in …# Behavior# Attitude# Circumstance# Knowledge# Skill
OUR
EFFECT
By % of AllChanges in Changes in …% Behavior% Attitude% Circumstance% Knowledge% Skill
Results…what difference did we make in terms of: IMPACT on children and
families INFLUENCE on policies,
practice, systems, and services
LEVERAGE of resources
Impact, Influence, LeverageWe seek to make a positive difference in
the lives of children and families through three types of RESULTS:
Impact: Changes in a condition of well being for the children, adults, families or communities directly served by grants, programs, agencies, or service systems. Influence: Changes in capacity (including cultural competence), policies, regulations, systems, practice or public opinion. Leverage: Changes in investments by other public or private funders in our strategies to improve outcomes for children and families.
Place-based community accountability Program-level AND population-
level (whole neighborhood) accountability– Beyond traditional contractual
obligations to a funder– Strengthening shared and
collaborative accountability to the whole population across multiple partners and organizations
– Increased partnership of all partners, players, and funders working in a place
Accountability requires…
Funders and decisionmakers (i.e., those with power and resources) are held accountable for the decisions they make and they must publicly affirm and report their targets and results
All stakeholders must have publicly available data with which to hold all accountable
Community residents must be prepared to be organized and participate and act in decisionmaking and accountability processes
Transparent and open decisionmaking processes and structures need to exist to hold all stakeholders accountable
Incentives for positive progress and sanctions for negative change
Results Accountability
www.resultsaccountability.com
http://raguide.org/
www.aecf.org “Casey ResultsNet”
Weed & Seed
CollaborationCoordinationCommunity participationLeverage
Weed & Seed
CollaborationCoordinationCommunity participation
AUTHENTIC DEMANDLeverage
CORE CAPACITYASSESSMENT
Measuring Leverage
A Practical Guide to Documenting Influence and Leverage In Making Connections Communities
Reisman, Jane; Langley, Kasey; Stachowiak, Sarah; Gienapp, Anne
www.organizationalresearch.comwww.aecf.org
Making Connections
Poor outcomes for children are geographically clustered, requiring a specific neighborhood-based strategy
10+ year investment in 10 communities (2000 to 2010)
Hartford IndianapolisLouisville MilwaukeeOakland ProvidenceDenver Des MoinesSan Antonio Seattle/White Center
Results/Outcomes
Adults have work and earnings Families save, build and protect
assets Families access quality services Families are connected to social
supports Children are healthy and prepared
to succeed in school (grade-level reading)
Local Learning Partnerships (LLPs) Established to support local Making
Connections sites (for 10-years and beyond) with data and information to help strategy development, and self-evaluation
Collaborative group of data holders, researchers, evaluators, and data users (including nonprofits and residents)
Multiple roles:– Building/consolidating local data warehouses– Performance and results measurement and feedback to
work teams– Quantitative and qualitative data collection on families and
neighborhoods, including evaluation of process and outcomes
– Promoting local capacity (by residents and systems) to use data strategically
Community capacities
Program implementation and management
Data, self-evaluation, and accountability
Communication, public policy, and will-building
Resident leadership and engagement (Authentic Demand)
Financial sustainability and infrastructure
Community capacities
SCOPE – coordinated and integrated services and service systems
SCALE – policy and system changes that reach large numbers of families
SUSTAINABILITY – financing and infrastructure
Authentic demand/resident engagement
Data, learning, and accountability
Measuring Community Capacities Began with stages of change
framework (INSITES, Bev Parson)
Maintaining Business as
Usual
Building Awareness of And Demand for Change
Exploring New
Approaches and Small
Scale Changes
Refining, Coordinating,
and Increasing
the Impact of Strategies
Effective Approaches are Taking Hold and
Transforming Business as
Usual
Maintaining Business as
Usual
Building Awareness
of And
Demand for Change
Exploring New
Approaches and Small
Scale Changes
Refining, Coordinating, Increasing the Impact
of Strategies
Effective Approaches are Taking
Hold
SCOPE
SCALE
SUSTAIN
RESIDENTS
DATA
Building a scale
Developed common (general) indicators across sites
Worked with sites to revise and better define concepts and stages
Tested in individual sites Added a “score”
Maintaining Business as
Usual
Building Awareness of And Demand for Change
Exploring New
Approaches and Small
Scale Changes
Refining, Coordinating,
and Increasing
the Impact of Strategies
Effective Approaches are Taking Hold and
Transforming Business as
Usual
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Using the scale
Implemented collaborative and participatory process of measuring capacities in each site
Evaluators facilitated a review and discussion of specific areas of work (eg, workforce, assets, school readiness)
What is the evidence? Site specific score
Contributing to evaluation Annual assessments produce a
site specific T1T2T3 Use additional data to answer
questions like:– Did technical assistance help sites
advance? – What patterns exist across sites?– Does higher capacity lead to
more/bigger/better results for families?
Lessons
Collaborative and participatory process was powerful way to reinforce shared understanding of all partners of concepts and theory of change
(decision) More important to focus on change within sites than comparing sites given different environments/context
Lessons/Challenges Important to anchor discussion of
capacities in terms of results/work– Capacity for WHAT?
Iterative process Still difficult to summarize across
sites Need additional verification of
evidence and perspectives Tool/scale less important than the
process
Measuring…
Resident engagement Community engagement Participation Social networks/social capital Civic engagement Community organizing
Authentic Demand “Honeycomb”
Residents advocate for results in strategy work groups and play key decision making roles
Membership networks continue to grow in size and effectiveness
Resident leadership training institutionalized in community colleges and other local partners
Organizing leads to policy changes at local and state level
Resident leaders provide impetus for forming service provider networks
What Does Authentic Demand Look Like?(The Forms)
Leadership strategies that offer learning opportunities and training to residents and other partners focused on various skills and capacities including strategic planning, data analysis, policy advocacy and others;
Strong social networks that build
relationships and reciprocal exchange among and between members of the community, and link residents to new resources and opportunities;
What Does Authentic Demand Look Like?(The Forms)
Community organizing efforts that mobilize community members in the collective and in strategic alliances with other stakeholders seeking better outcomes for children, families and neighborhoods
The kind of civic participation that enables
residents to establish community priorities and then hold elected officials and service providers accountable through voting, local forums, community research, and consumer and policy advocacy.
How do we know that it’s meaningful?(Interim Outcomes)
1. Voice. There are ample opportunities for community members to not only make their needs, desires and opinions known, but also to participate with dignity in environments where decisions affecting the community are being made.
2. Accountability. Those who represent community interests at decision-making tables are representative of and accountable to well-defined and legitimate constituencies, including voters, members of a community or faith-based organization, or agency leadership. As important, there is evidence of a “demand environment” services, resources and opportunities are determined more by the needs and desires of those in the community rather than what organizations have available or what service providers “decide” families and the neighborhood need.
3. Learning, skills and capacity. There are consistent, reliable and trustable opportunities for leadership training, mentoring, or other skill building opportunities that enhance the competence and confidence of community members at decision-making tables and in civic life.
4. Identity. People believe they hold a stake in the larger community. In addition, community identity is inclusive and represents its diversity -- different cultural groups within the community are genuinely represented and embraced as members of the community.
5. Reciprocity. Community members are actively encouraged to contribute their time and talent, because there is a clear understanding that everyone has something to offer the change agenda. Service to the community is rewarded through public recognition, ceremonies, and events. A social “norm” of participation and of action is nurtured.
6. Choice. Not only are there many options to engage in and contribute to change efforts, but those options connect to the needs and desires of the community – what has value for them – not the agendas of someone else.
SKILLS AND COMMUNITY CAPACITYWhat/How Much Was Done How Well Was It Done
# of residents trained and prepared to take up engagement or leadership (completing training in community engagement, leadership, budget analysis, race matters, RBA, etc.)
# of partners completing training in authentically engaging the community (social networks, race matters/Undoing Racism, etc.)
% of training graduates getting involved in issue organizing or taking on new leadership roles
Deepened knowledge and ability in various types of skills and capacities (identified site-by-site)
Improved rates of program participation across Making Connections strategies
# of residents running and securing public office
CONSTITUENCY, SEATS AT THE TABLE AND COMMUNITY ACCOUNTABILITY
What/How Much Was Done How Well Was It Done # of residents attending town hall
or community meetings
# of parents engaged
# of public forums held
# of new participants attending forums to express perspectives and needs with elected officials
# of residents (i.e. constituency) that leaders can mobilize for an event, action or campaign
Increase in the number of tables where residents are represented at the community and system levels
Increased diversity of participants in town hall/community meetings
% of the new residents getting involved that remain involved after one year
Increase in the number of examples of collaboration across individuals and organizations to solve complex problem
Increase in the stability and capacity of resident-led organizations or associations (increase in budget; increase in staff; increase in membership; increase in collaborations)
What Difference Did It MakeImpact Influence Leverage
Increase in access to jobs via resident/network referrals
Increased access by residents and families to community resources and job supports
Increase in SCHIP or health care coverage for vulnerable children and families
# of positive community solutions achieved by residents and partners working together (key wins are identified site-by-site based on existing strategies). For example:
1) Increase in bi-lingual resources, staff and literacy options in the schools
#/% of targeted partners adopting or embedding new practices that incorporate authentic demand principles
#/% of targeted partners engaging residents in boards, committees and as advisors
Evidence of a positive change in practice due to partners responding to feedback from community members
Funders begin to require their grantees to demonstrate evidence of community capacities on their board, staff and practice
New funds are invested in fostering authentic demand skill and capacity building for residents and public/private agencies
New funds invested in engaging and strengthening community constituencies
New funds invested in mentoring opportunities
New funds invested in and towards engaging and strengthening resident and partner capacity and effectiveness in diversifying seats at the table
Multi-year funding secured for authentic demand
Tools & ways of measuring Software for tracking program,
process, and contextual data in a central place (Backpack- Backpackit.com)
Tracking resident engagement (Constant Contact and YS.net)
Mapping participation (GIS)
Tools & ways of measuring pre/post...or often better...post-
then assessments of learning charting timelines learning meetings core capacity assessment individual and group interviews social network analysis
Core
Link Node
Cluster Periphery
Hub
Social Network Analysis: A Few Helpful Definitions
Source: Monitor Institute
Lessons learned
Other resources Race Matters Toolkit
http://www.aecf.org/racematters.aspx
Guide to Evaluating Advocacy & Policy Change http://www.organizationalresearch.com/publications/a_guide_to_measuring_advocacy_and_policy.pdf
When and How to Use External Evaluators
http://www.aecf.org/publications/data/using_external_evaluators.pdf
Tools and Resources for Assessing Social Impact (TRASI) (beta)
foundationcenter.org/trasi