CFBSA PowerPoint Presentation FINAL VERSION

Download CFBSA PowerPoint Presentation FINAL VERSION

Post on 15-Jan-2017

240 views

Category:

Documents

6 download

TRANSCRIPT

Social Representation Theory Approach

Key terms

PROGRAM EVALUATION

Fred KassChristopher TremelMiriam Saleh

The idea behind the projectTo assist the Community Food Bank of Southern Arizona in developing an overall evaluation framework that will enable them to track the outcomes of their feed the line programsThis enables the Community Food Bank of Southern Arizona to identify successful programs and target resources more effectively

2

Project ScopeCreate logic models for each feed the line program.Tools for completion Define Food Delivery to Food Justice in The Community Food Bank contextDefine the steps required for short term changePresent Evaluation and Process models for long term change.

The institutional evaluation framework will assess the feed-the-line hunger-relief line-of-business, as well as its shorten-the-line self-sufficiency services/program. Special emphasis is placed on the feed-the-line, as this historically has been an output-measured endeavor rather than an outcome-based one.

3

Implementation StepsFill in the gaps in the logic models. Answer the key Logic model questions for each programDefine a new mission statement for the entire agencyCreate an environment of changeEnsure Employee willingness to changeImplement short term changes in the feed the line programsChoose one of the Process and Evaluation Models to implement for long term change

4

Immediate Action

Logic Model Construction

Completed within outline of programsTo be completed by individual programsFood Justice: articulate an overarching goal for the agency

The inputs, outputs and strategies section is filled out for each of the core feed-the-line programs. The right side of the model (outcomes and impact measures) were vague, ambiguous or non existent. 6

Table of contents

Logic Model Questions:Going forward, in order to establish linkages and outcomes, the following questions must be answered:

Each program has to define long-term outcomes

2. Each program has to define how they interact with other programs

3. How does each programs outcome contribute to the goal of food justice for the entire agency?

The first step: complete the logic models for each program by answering the three key logic model questions. Doing this, will give your agency a complete picture of everything you do and how the programs interact. 7

Agency Market Logic Model

Rationales/Assumptions: Non-profit agencies such as the Community Food Bank benefit from food contributors of agency markets to provide their facilities with extra produce for their clients.

Situation: No Applicable Measures

The green arrow indicates a correlation between outcome measures and impact measures.

8

The Emergency Food Assistance Program (TEFAP) Logic Model

Rationales/Assumptions: Many families, individuals and elderly persons still require extra food assistance. TEFAP provides additional food options to reduce hunger.

Situation: No Applicable Measures

TEFAP https://www.azdes.gov/main.aspx?menu=359&id=5292

Food Plus (CSFP) Logic ModelRationales/Assumptions: No Applicable MeasuresSituation: Low income elderly persons requiring additional food assistance

Grocery Rescue Logic ModelRationales/Assumptions: Food nearing the expiration date can be salvaged and distributed to food agencies, providing nutritious foods to those in need.Situation: Food nearing the expiration date or unappealing to customers gets thrown out

11

Supplemental Nutrition Assistance Program (SNAP) Logic ModelRationales/Assumptions: Providing extra nutritious foods and benefits to those in need can increase quality of life for those in need. Situation: Low income individuals and families require additional food assistance and health benefits to contend against hunger. 2014 Arizona Food Bank Conference

http://www.fns.usda.gov/snap/supplemental-nutrition-assistance-program-snap

12

DisclaimerThese program logic models were created based on published materials and external sources.To prevent ambiguities or incorrect data, please have each program meet to discuss their logic models. This will lead to a more accurate representation of each program.

Disclaimer: Data may exist already but not able to find in the public documents (which we were working off from). Hence, there were many inconsistencies and obscurities in the data, especially with the outcome/impact measures.

13

Preparing the Organization for Changeand Pilot Programs

14

Change Theory

Change theory essentially states that organizations are created with a limited purpose in mind, but as they grow they realize that this limited focus will not solve the issue so they have to take on additional task. In your case simply handing out food does not end hunger in Tucson we have to tackle the sources of food shortages like education or access to social services. Thus you created the shorten the line programs to deal with these issues. And your morphed yourselves from a simple distribution network to a food justice organization.

15

Change Theory for the Feed the Line ProgramsThe Feed the Line Programs were created with the culture of a food distribution center in mind not Food Justice.Currently the Feed the Line Programs only measure OutputsWhile these output measures can lead to correlations they can not prove causality.

The Current feed the line Programs were developed when the community food bank was only concerned with being a food distribution center, and that is reflected in the metrics that are used to measure its success. The tonnage of food that is handed out does this really well.

However you had indicated that you wished there way a way from the feed the line programs to have a measurable outcome. The problem is the measurements that you are using (lbs of food distributed) are not sufficient. You can look at long term trends between the food distributed and proxies for self sufficiency and food justice like SNAP enrollment in the area but the trends are really shaky and can only prove correlation not causality.

To remedy this situation and bring the Feed the Line programs into line with the community food bank becoming a food justice organization there needs to be discrete data collected on all clients who use the programs so that their progress can be tracked over time.

And that is where change theory for the Feed the line programs come into play You have the good fortune that clients come to you for help and you have a good that gets them in the door to begin with, that doesn't exist in a lot of nonprofit work. So you can leverage this advantage by adding a the mission of data collection to the feed the line programs. This way you can track the outcomes of the clients who receive the benefits of these services by comparing it to long term trends and you can better serve your notion of a food justice organization by helping every client (or at least pointing them in the direction of) services that address the underlying causes of their food insecurities.

16

Organizational Change Theory and overcoming Resistance to ChangeOrganizational Change Theory: Implications for Health Promotion Practice

The Idea of change theory may be awesome but there will be organizational resistance to any new tasks being implemented. The three major ways that it can be combated are though:

Creating appropriate conditions for change Changing the cultureAnd having explicit follow though. 17

Appropriate Conditions for Change

Appropriate Conditions For Change:

If people think that they are ok in their current setting even talking about a change initiate will require immense amounts of political capitol from a director. The employees have to take it on faith that their boss is right, and that is big ask if things are ok as they are. At the community Food bank this would involve changing Solution: A new mission statement has to be created for the feed the line programs in which the change from a delivery service to a food justice service which is client centered is explicitly stated. Doing this sets the a new bar to be met, and staff/volunteers can see that the current system is no longer adequate.

18

Cultural Willingness to Change

Appropriate Conditions For Change:

But simply creating a new mission statement for staff/volunteers to read once will not do the trick. It needs to be reinforced over and over and over again.

This has to be a change in the culture of the organization its almost a full internal rebranding. Sings in the break room or department slogans may seems silly but given enough time they really do get in your head. With enough emphasis eventually people will be willing to accept the change. This is not a costly process it is just long term.During this time period there has to be a culture of learning that is established. Every employee has to understand why new procedural changes are required to meet this new goal in the feed the line programs. And more importantly they have to understand their role in this new mission, how they contribute.

19

Follow Though

Appropriate Conditions For Change:

The enthusiasm for new projects and missions can quickly die out if the new tasks are pile on too quickly. To make sure cultural change is permanent and not a fleeting thing there needs to be explicit follow though and evaluation of even the most minute details to make sure they work and that they are being implemented with the lowest entry cost possible

The way that I would envision that a food justice mission would be implemented though a client centered model in the beginning is though a simple survey that clients fill out when they receive goods or services. It would track how often they are coming, relevant data like do they have SNAP? and if they accessed any services other than the food programs. This is already done sometimes on an individual basis but there is no programmatic wide way of tracking long term outcomes for the feed the line programs.

This simple survey is easy to implement and allows the impacts of the feed the line programs as a data entry point to be easily seen.

20

Long Term ChangeFive Years Out

Industry TrendsRecipient Basedor Client Centered

Results-Based AccountabilityRE-AIM

My portion of this project involved contacting food banks across the nation to see what tools they use for program evaluation. It is important to note that Chris touched on a major industry trend, the shift from food delivery to food justice. New evaluation methods play a significant role in this shift. Food justice is much more difficult to define and measure than the # of meals served to a set # of people. The movement toward social change begins with asking whether or not services are client centered, but food banks who are further along in the process are using evaluation frameworks such as Results-Based Accountability or RE-AIM to add structure and definition to the concept of food security.22

VermontSanta CruzSanta BarbaraOregonOhioNorthern AlabamaNYFoodlinkWhich Evaluation Method?

Vermont Food Bank is using the Results Based Accountability evaluation framework.Santa Cruz and Santa Barbara are using the RE-AIM evaluation frameworks.Oregon Food Bank and Ohios Second Harvest Food Bank are both building their own recipient based frameworks. This seems to be similar to the current undertaking at the Community Food Bank of Southern Arizona. Client centered logic models have been created. Needs assessments are performed. Client impact-based outcomes are reimagined.Northern Alabama and NYFoodlink appear to be using Feeding Americas Performance Framework Dashboard. This framework fits the food delivery model more than other options.23

(TEFAP) Logic Model with Applied Evaluative Frameworks RBAClient CenteredRe-AimArticulation of community/agency goals. % of food distributed by agency. % of food received by clients. Continuous data reporting Reach: community populationEffectiveness: Impact of providing emergency food assistanceAdoption: representativeness of an interventions and obtaining supportImplementation:use of resources and strategiesMaintenance: appropriate frameworks and programs become standard.

Allows the client to choose the services necessary for their needs in terms of service delivery. Removal of a one size fits all approach.

This is an example of the three evaluative frameworks (Results Based Accountability, Client Centered or Recipient Based, RE-AIM) applied to the TEFAP program. The following slides provide more depth into each framework.

Results Based Accountability: data-driven decision making to solve problems. Programs and staff are held accountable for each action and process that occurs.

Client Centered: Clients have flexibility in choosing a food assistance service that fits their needs. Uses logic models and Needs Assessments

Re-Aim: Moves beyond client centered models and asks a series of research questions to analyze social change.24

Client CenteredRecipient based impact measuresLogic modelsNeeds assessment

We have spoken a little bit about client centered models. What do these evaluation methods look like? What is different?Impact measures are recipient-based instead of commodity-based. This means measuring the coordination of family food sources such as WIC, SNAP, School lunch programs and markets), community inclusion and participation, behavioral changes that increase health (Meal planning, buying in bulk, cooking at home, using left-overs, reducing fat and sodium, increasing fresh fruit and vegetables). It means stopping the focus on the pounds of food, meals or people served. Logic Models which are client centered (next slide)Needs assessments, also client centered These are steps that the Community Food Bank of Southern Arizona has already taken.25

Client Centered Logic Model

This is a sample of Oregons client centered logic model for the Child Hunger Programs. It is included in our annotated bibliography. It is important to note the distinction between intermediate and long term impacts, and the separate description of a program theory. 26

Needs Assessment, which was also provided by Oregon. The Needs Assessment seems to be an adaptation of Mazlows Hierarchy of Needs. It doesnt specifically connect to food security.27

RE-AIM: AN OVERVIEWRE-AIM is an acronym for Reach, Effectiveness, Adoption, Implementation, and Maintenance.

RE-AIM begins with a series of research questions designed to move beyond the current reductionist approach to assess interventions in order to isolate effective programs or activities. RE-AIM hypothesizes that the overall social-change impact of an intervention is a function of all five RE-AIM dimensions not simply client-based outcomes. That is all five dimensions are important and equally in need of evaluation.Importantly, Foodbank RE-AIM evaluation allows summary indices (Success Score) to be developed for the use in determining overall impact of individual programs as well as initiative areas.

This is a general description of RE-AIM, which was the evaluation framework adopted by Santa Cruz and Santa Barbara. RE-AIM was originally developed by the Department of Human Nutrition at Virginia Tech. Like RBA, this framework begins with research, and from there, hopes to shift evaluative measurements toward social change. 28

RE-AIM

This is a visual representation of the RE-AIM framework. In this vision, all 5 elements should work in tandem. 29

MEASURING SUCCESS OF A RE-AIM INTERVENTION

This is an example of an evaluation matrix for RE-AIM. Each RE-AIM element has a specific set of evaluative questions attached. After asking these questions, you receive a success rate and an impact rate. These rates are calculated based on RE-AIM calculation tools, which are available in our annotated bibliography. RE-AIM also offers a free online module which trains you on the evaluation techniques. 30

Pros & ConsEmphasizes cohesionWell-researchedNuanceClear implementationIncludes a timeline

Labor intensive processNo software No specific measurement of social change, although all factors measured facilitate change

In 5 years? 10 years?

Well researched: RE-AIM had an academic inception. Since the original paper in 1999, there have been approximately 100 publications on RE-AIM by a variety of authors in diverse public-health related fields

Aside from balanced and cohesive services, a clear vision of the next 5 or 10 years isnt immediately apparent. A dedicated staff member would have that answer after an in-depth research project. RE-AIM research guidelines are very clear, but the research process doesnt come in the form of a simple survey. There are no multiple choice answers. In this regard, RE-AIM is quite true to its academic roots. An ideal situation would be to have a graduate student take this on as a dissertation. The process would be quite clear, but the research will certainly take more than one semester.31

Overview: Results Based AccountabilityStart with ends, work backward to means. What do we want? How will we recognize it? What will it take to get there?Keep accountability for populations separate from accountability for programs and agencies.Customer or client results are the responsibility of program managers.Use data (indicators and performance measures) to gauge success or failure against a baseline.Use data to drive a disciplined business-like decision making process.Involve a broad set of partners.Get from planning to action as quickly as possible.

Results Based Accountability is an open source evaluation framework. The implementation guide is available online for downloading. A link to this guide book will be included in our annotated bibliography. Evaluation software and other tools are available for purchase on the same website. This slide describes the general premise of RBA.32

Like RE-AIM, RBA begins with specific research questions. RE-AIM begins with a deep and nuanced analysis. RBAs research questions are much more pointed. They measure the accountability of programs to recipients. This is a sample of questions which are asked.

Before selecting an evaluation framework, we would recommend a thorough reading of both RBA and RE-AIMs research questions. Which questions are most important to the core client population? Which questions fit the goals, mission and culture of the food bank? 33

5 STEPS TO CREATE SOCIAL CHANGE

RBA specifically measures social change. After determining a baseline trend, RBA performance indicators illustrate whether or not change is being achieved.34

RBA: Performance Measurement Tools

Several performance measurement tools are included in on the RBA website.35

This is Oregons example of an RBA performance measure in action. This sample is specifically connected measuring effectiveness of their Backpack Program. 36

Pros&ConsData driven measurement of social changeEvaluation is distributed among many partnersPrograms become clearly accountable for food justice resultsClear process with an implementation timelineThe tremendous amount of data collection might be tedious Cost of software: $50 per user, monthlyThis is a large change in process, it will be challenging for the organizational culture to adapt In 5 years? 10 Years?

Which framework to use?Academic inception

Deliberate decision making

Institutional knowledge confined to one user

Industry oriented

Rapid deployment

Accountability spread

38

Moving ForwardEvaluate the proposed logic models for each feed the line programCreate a change environment to food justice in the food bankAdopt and customize an evaluation framework such as RBA or RE-AIM Link data between feed the line and shorten the line programs

39

Annotated BibliographyCohen, Barbara. USDA Community Food Security Assessment Toolkit. IQ Solutions, Inc., (July, 2002). Food Assistance & Nutrition Research Program. Efan. http://www.ers.usda.gov/media/327699/efan02013_1_.pdf The USDA Community Food Security Assessment Toolkit report analyzes the issue of food security and provides guidelines and assessment tools, which food banks can adopt to deal with the growing concerns of food access, food security, and other topics. Data collecting tools, surveys and other helpful materials are included for food banks to utilize. Supplemental Nutrition Assistance Program (SNAP). United States Department of Agriculture. http://www.fns.usda.gov/snap/supplemental-nutrition-assistance-program-snap SNAP (formerly known as food stamps) provides low-income individuals and families with a debit card to purchase nutritious foods. The website provides more information on eligibility and purpose.

The Emergency Food Assistance Program (TEFAP). United States Department of Agriculture. http://www.fns.usda.gov/tefap/emergency-food-assistance-program-tefap TEFAP provides an emergency supply of nutritious food to low-income families and individuals. More information is provided on the website.

Annotated Bibliography continuedVancouver Coastal Health Community Food Action Initiative http://www.smartfund.ca/current_cfai.htm Evaluation report: http://www.smartfund.ca/docs/eval_vch_cfai_2011_full.pdf The initiative website provides useful information including evaluation reports, management plans, and other useful materials. The frameworks and models give valuable insight in dealing with the issue of food security.

Le Groupeconseil baastel lte. Baastel: Creating a Macro Results Framework for the Middle East and North Africa Transition Fund Feasibility Assessment Final Report. June, 2014. http://www.menatransitionfund.org/sites/mena_trans_fund/files/documents/Baastel_Macro_Results_Framework_Assessment_Appendix_Final.pdf This feasibility assessment pertains to world monetary banks and focuses on creating a macro framework for the Middle East and North Africa (MENA) Transition Fund. While the report does not discuss food banks, it does provide logic models and theories of change frameworks and evaluation tools.

Annotated Bibliography continuedPerson Centered Planning Preparation and Procedure Guide. 2nd ed. Augusta, Me.: Maine Dept. of Behavioral and Developmental Services, 2003. Print.

"Organizational Change Theory: Implications for Health Promotion Practice."Health Promotion International(2014). Print

RE-AIM: Virginia Tech College of Agriculture and Life Sciences. http://www.re-aim.hnfe.vt.edu/index.html This site provides an explanation of and resources for those wanting to apply the RE-AIM framework. The RE-AIM framework is designed to enhance the quality, speed, and public health impact of efforts to translate research into practice. The Key Features of RE-AIM website are: Tools and resources to facilitate implementation, a comprehensive list of RE-AIM publications and presentations organized alphabetically by year

Results Based Accountability (RBA). http://raguide.org/Website, guidebook, evaluation tools and software for Results Based Accountability.Results-Based Accountability (RBA), also known as Outcomes-Based Accountability (OBA), is a disciplined way of thinking and taking action that communities can use to improve the lives of children, youth, families, adults and the community as a whole. RBA is also used by organizations to improve the performance of their programs or services.

42