the state & community intimate partner violence (ipv) & perinatal depression (pd) resource...

IPV/PD: A Comprehensive Approach

The State & Community Intimate Partner Violence (IPV) & Perinatal Depression (PD) Resource Development Project:

Evaluating Your Efforts

Health Resources and Services Administration

&Social Solutions International, Inc.

January 2013


Webinar Speakers and Guests

• Moderator: Keisher Highsmith, DrPHDirector of Special Initiatives and Program Planning and EvaluationHRSA/Maternal and Child Health Bureau

• Speaker: Kristen Stier, MAResearch AssociateSocial Solutions International, Inc.

• Speaker: Jenny Namur Karp, MPHPresidentSocial Solutions International, Inc.


Objectives

• Explain the purpose of evaluation research

• Introduce types of evaluations, specifically process evaluations and outcome evaluations

• Provide information on the challenges and benefits of program evaluation


Toolkit Goals – Evaluating Your Efforts

• Introduce simple types of evaluations, how to collect information and feedback, and what to do once information is collected

• Provide two tools and multiple resources to assist you in evaluating your efforts


Evaluation Basics

Jenny Namur KarpSocial Solutions

International, Inc.January 2013


Why Evaluate?

• Understand, confirm or increase the impact of products of services on clients

• Improve service delivery mechanisms to be more efficient and less costly

• Verify that you are doing what you think you are doing

• Produce data or verify results that can be used for public relations and promoting services in the community

• Produce valid comparisons between programs to decide which should be retained

• Fully examine and describe effective programs for duplication elsewhere


Role of Evaluation Research

• Accountability• Assess program quality and

continue quality improvement in the program

• Assess attainment of program goals• Determine program effectiveness• Success of new efforts• Enable program duplication


Evaluation Research: True/False #1

# 1: Evaluation research is about producing boring, scientifically generated data with useless conclusions.

– False: More recently evaluation has focused on utility, relevance and practicality at least as much as scientific validity.



# 2: Evaluation research is about proving the success or failure of a program.

– False: This implies that implementing the perfect program means never having to assess or critique it’s impacts. Continual feedback and program adjustment is beneficial to all.



# 3: Evaluation research is very difficult and time consuming and cannot be done without outside experts.

– False: Many of you probably partake in evaluation research regularly even if you do not know it. Part of the goal today is for you to realize the benefits of evaluation research and how to use it to make the most impact with your program.


Specific Evaluation Questions

• What are your program goals?• What are your target outcomes?• What are the steps to achieve the outcome?• What are the anticipated results?

• During and after program you will learn real results

• What is the analysis and action plan?• Meaning what will you analyze and how

• Who are the person or people responsible?• Who is responsible for what

goal/outcome?• What is the time frame?

• Will you complete the evaluation weekly, monthly, annually?


Process Evaluation


Process Evaluation

• Monitor the implementation process of an intervention

• Determine whether or not the intervention is delivered as planned (fidelity; manual-based interventions)

• Document challenges to the project, barriers encountered, and strategies used to address those barriers (case studies)

• Determine whether or not the project is delivering the intervention to the intended target


Elements of Process Evaluation

• A thorough process evaluation should include the following elements:• Description of the program environment

and supplying data. • Description of the process used to design

and implement the program. • Description of program operations,

including any changes in the program. • Identification and description of

intervening events that may have affected implementation and outcomes.

• Documentation such as meeting minutes, reports, memorandums, newsletters, etc.


Sample Process Evaluation Methods and Measures

• Conduct observations at project meetings, site visits

• Conduct focus groups with participants and staff re: their perceptions of the project

• Track the number of clients served; their background characteristics

• Track services received, time in program, type of discharge


Questions Answered with Process Evaluations

• Are we implementing the program as planned?

• What aspects of the program are strong? Which ones are weak?

• What can we do to continue to strengthen our program?

• Can we improve program efficiency? • Are there unexpected effects?• Did problems arise?


Benefits of Process Evaluations

• Accountability• To better understand the day-to-day

function of your program• Program development and

improvement• To help others set up similar services

or networks


Outcome Evaluation


Outcome Evaluations

• Determine the effects of participation in the project on the participants

• Determine the factors (individual and programmatic) related to favorable/unfavorable outcomes

• Determine the relationship between “dose” (amount of the time in the program, number of services received) and “response” (outcomes observed)


Elements of Outcome-Based Evaluation?

• Outcome = Impact on End User

• Impact = Changes in:– Behavior– Attitude– Skills– Knowledge– Condition/ Status


Continuum of Program Outcomes

• Condition/ Status• Behavior• Skills• Knowledge• Community connections• Social networks• Attitudes/ Values• Perceptions/ Feelings• Satisfaction with

service• Participation/ Use of

service• Awareness of service

• Note that changes in individuals (outcomes) usually occur in this order, from bottom to top.

• The most difficult changes are those at the top of the list.


Sample Outcome Evaluation Methods

• Self-Assessment of participants and/or trainers• Participant demonstration of knowledge and skills• Questionnaires and Surveys of participants and trainers• Interviews with participants and trainers• Focus groups with participants and trainers• Assess products developed


Questions Answered with Evaluation Questions

• What did you do?• Who did you do it to?• How many people did you serve?• What did they get (or not get)?• What effect did it have on their attitudes

and behavior?• Can other programs use this

information?• How does this information influence

other programs?• Were there unexpected difficulties?


Benefits of Outcome-Based Evaluations

• Useful as a planning tool; requires needs assessment

• Clarifies the purpose of the program/service• Keeps staff and stakeholders focused on

ultimate goals • Stimulates discussion of issues• Helps keep implementation on track;

milestones are identified• Indicates when changes to program are

needed• Energizes staff by demonstrating the real,

human impact their work produces and by stressing common purposes and goals

• Enhances program staff’s knowledge and understanding of how to interpret data


Additional Benefits of Outcome-Based Evaluations

• Gives insight into why and how program services are used

• Assists in fundraising and grant writing by providing statistics on results

• Provides empirical evidence that what you are doing is what you intended to do

• Quantifies the anecdotes and success stories• Identifies effective programs/services• Enables programs to focus on what is most

successful as well as to improve what is not• Demonstrates the ways your program

contributes to solving community problems


Designing an Evaluation


Key Considerations in Designing the Evaluation

• For what purposes is the evaluation being done?•Who are the audiences for the information from the evaluation?•What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences?• From what sources should the information be collected?•How can that information be collected in a reasonable fashion?•When is the information needed?•What resources are available to collect the information?


Sample Data Collection

• Remember, this will be project-specific• Background info (demographics,

education, employment, income, residential status)

• Family & social support• Health status & use of health services• Criminal behavior history• History of trauma and violence• What else can you think of that is

relevant to your program?


Typical Challenges Encountered in Conducting Evaluations

• Seems too difficult• Not thought out ahead of time• Lack of integration of project and evaluation-

related aims/activities• Lack of communication or misunderstandings

between project and evaluation, often from differences in training, orientation, goals

• Evaluation requirements viewed as intrusive or burdensome by project staff

• Interviews are key, but don’t just interview the successes.

• Fear of being graded or judged as performing poorly by the evaluators


Concluding Thoughts

•Clearly, evaluation takes planning•The effort put into evaluating your programs and planning the evaluation as early as possible serves the ultimate goal of effective programming and helping your clients as best as you can, always•Evaluation is a continuing process•The time added is repaid in improved programs and improved communities


Creating a Culture of Evaluation

• An ideal culture of evaluation would encompass several values, including:• Action-oriented – a culture that actively

seeks to problem solve.• Teaching-oriented – evaluations that can

be used and understood by non-technicians• Inclusive and non-hierarchical – solutions

to problems involve a diverse range of participants

Source: Trochim, William M.K. (2006, October 20). An Evaluation Culture. Retrieved from http://www.socialresearchmethods.net/kb/evalcult.php

http://www.socialresearchmethods.net/kb/evalcult.php



Creating a Culture of Evaluation, cont.

• Self-critical – a culture that recognizes its limitations• Interdisciplinary – multi-disciplinary

research groups are beneficial• Forward looking – a perspective of

seeking where evaluation can be helpful, not just when problems are apparent• Open and democratic – evaluation data

should be accessible to all

Source: Trochim, William M.K. (2006, October 20). An Evaluation Culture. Retrieved from http://www.socialresearchmethods.net/kb/evalcult.php




Resources

Resources for Methods in Evaluation and Social Research http://gsociology.icaap.org/methods/

• This website contains a catalogue of free evaluation tools on the Internet

US Department of State, Bureau of Educational and Cultural Affairs, Office of

Policy and Evaluation

http://exchanges.state.gov/education/evaluations/index.htm

http://gsociology.icaap.org/methods/




A Comprehensive Approach for Community-Based Programs to Address the Intersection

of Intimate Partner Violence & Perinatal Depression:

Evaluation of Your Efforts

Kristen Stier, MA

Social Solutions International, Inc.


Evaluating Your Efforts

• Evaluation Can Take Many Forms:

• Assess how well your IPV/PD initiative is working

• Gather feedback, data, and information about your initiative and its activities that you can share with funders and members of your community.

• Define success for your community.


Why is Evaluation Important?

• Assists in securing money and in-kind resources for the initiative

• Garners community support for and involvement in the initiative

• Helps to overcome resistance to the initiative

• Generates ideas about how the initiative can be more effective


How Can Success Be Measured?

• Telephone survey - an opportunity to measure community opinion regarding various issues related to your initiative.

• Survey of goals -- a paper-pencil survey distributed to community members by mail that asks them to rate the importance and feasibility of your community-based initiative’s goals.

• Survey of satisfaction -- a paper-pencil survey distributed to members of the initiative that asks them to rate their satisfaction with issues such as the leadership, community involvement, and planning of your community-based initiative.


How Can Success Be Measured, cont.

• Behavior survey(s) -- a paper-pencil survey distributed annually to your target population to assess their knowledge, attitudes and behaviors in relation to your goal. (For example, you may want to survey women who utilize your IPV/PD services to gauge their knowledge, attitude and behaviors).

• Interviews with key participants -- semi- structured focus group interviews that clarify important events in the life of the initiative and illustrate the value that the initiative has added to the community.

• Survey of outcomes -- a paper-pencil survey that asks initiative members, funders, and outside experts to comment on and rate the significance of changes made in the community.


Measuring Success, cont.

• Goal attainment reports -- reports that identify progress toward proposed changes identified in the action plan.

• On-line documentation system -- an Internet- based system that gathers information regarding the daily activities of the initiative, including community actions, community changes, and services provided.

• Community-level indicators -- information relevant to your initiative that is collected from local, state- or regional-level sources. (For example, you may want to gather statistics about rates of IPV or PD in your state or region that might already be available and track any changes that occur throughout the years).


Post Assessment Tool

• Have there been improvements in awareness towards addressing IPV/PD among staff at the organization?

• Is IPV/PD a part of new staff training at the organization?

• Are there guidelines and procedures in place for referring clients who are experiencing IPV/ PD?

• Do staff routinely refer participants to other programs or organizations for IPV/PD services?

• Has the organization been able to build a referral and/or partnership network?

• Does the physical environment (clinic/office) have culturally competent posters, brochures or awareness materials related to IPV/PD?

• Does the organization utilize National Standards of Care Guidelines for IPV and PD?


Contact Us

Jenny Namur Karp, [email protected] Solutions International, Inc.8070 Georgia AvenueSuite 201Silver Spring, MD 20910Phone: 1-866-901-6583Fax: 1-866-369-6809

Kristen Stier, [email protected] Solutions International, Inc.8070 Georgia AvenueSuite 201Silver Spring, MD 20910Phone: 1-866-901-6583Fax: 1-866-369-6809


Additional Contact Information

For more information, questions, or comments about The State & Community Intimate Partner Violence

(IPV) & Perinatal Depression (PD) Resource Development Project, please contact:

Keisher Highsmith, Dr.P.H.Project Director

[email protected] of Health and Human Services

Health Resources and Services AdministrationMaternal and Child Health Bureau

Division of Healthy Start and Perinatal Serviceswww.mchb.hrsa.gov

mailto:[email protected]

the state & community intimate partner violence (ipv) & perinatal depression (pd) resource...

Documents

comprehensive approach

purpose of evaluation

evaluation hrsamaternal

program duplication

efforts ipvpd

program quality

program planning

program adjustment