measuring success in patient advocacy initiatives
Post on 21-Aug-2014
97 Views
Preview:
DESCRIPTION
TRANSCRIPT
Measuring Success in Patient Advocacy InitiativesJuly 9, 2014
Welcome!
Lori Melançon, Senior Director,
Corporate Affairs, Onyx Pharmaceuticals
Ken Berger, President & CEO, Charity Navigator
Carmen Perez, Manager, Measurement
and Standards, CECP
Progress in Cancer: Measuring Success in Patient Advocacy InitiativesA Metrics Study
Lori MelançonSenior Director, Corporate AffairsOnyx Pharmaceuticals
Measuring Impact in the Evolving World of Nonprofit Organizations (NPOs)
Increased pressure to measure impact
Fierce competition for funding
Measuring success a daunting challenge
Progress in Cancer Study Overview
Objectives• Evaluate how diverse NPOs in cancer measure impact against goals to
assess trends and best practices
Rx4good MethodologyPrimary Research: Group Self-Assessment
▪ 24 NPOs participated in self-assessment of metrics best practices via 8-item Survey Monkey questionnaire
Secondary Research: Assessment of Websites and Annual Reports▪ 45-question survey evaluated how 35 NPOs report on diverse measures of
progress against goals
1
2
Study Sample
Websites and annual reports of 35 NPOs assessed:
Self-assessments completed by 24 (69%) of the 35 organizations
Group Demographics
The majority of the 35 assessed organizations have been operating for 20+ years with annual revenue of $1M–$5M
Age of Organization Annual Revenue
<$1M 20% (n=7)
$1M - $5M 31% (n=11)$5M – $10M 23% (n=8)
$10M - $30M 11% (n=4)
>$30M 11% (n=4)
N/A 3% (n=1)
20+ Years 49% (n=17)
10-20 Years 43% (n=15)
5-10 Years 6% (n=2)
1-5 Years 2% (n=1)
Primary Research FindingsNPO Self-Assessment of Metrics Best Practices
Survey Revealed NPO’s Perspective on Metrics Reporting…
75% of NPOs have changed their view about metrics during last three years• increased
understanding of value of metrics
• funder expectations
96%Metrics are essential for evaluating success against goals
33%The work we do is difficult to measure
25%Metrics are important but we don’t have the resources to measure our programs8%
We would rather invest in programs than in measuring results
Key Obstacles to Measuring Impact
Lack of Resources Determining Outcomes ChangeWe have limited time, staff and funding to do all the things we want to do…And it's hard to increase staff size without an increase in funding.
Measuring impact takes so much staff and time that the programs do not get the attention they deserve. It is the wrong way around.
Not all of our programs are easy to track, especially grassroots and advocacy initiatives.
It can be can be a challenge to determine the best way to measure impact. It can also be challenging to determine if behavior has really changed due to a program or an event.
“ ““ “
“ ““ “
NPOs Measure Impact in Many Ways, Led by Progress Achieved Against Goals
96%
92%
92%
92%
79%
75%
75%
67%
67%
54%
Progress Achieved Against Goals
Numbers Reached, Attended Events
Actions Taken by Stakeholders as a Result of Programs
Social Media or Web Engagement
Media Coverage
Funding Received
Perceptions Changed or Reinforced
Sustainability of Programs or Initiatives
Message Delivery
Funding Given
Q: What are the ways in which you measure your impact now? (Check all that apply)
Metrics Ranked Most and Least Valued by NPOs Mirror Those Attributed to Funders
Q: Please rank the following metrics options in order of importance to your ORGANIZATION/FUNDERS, with one being the most important, 10 least important.
NPO Both Funders
Numbers Reached, Attended Events
Actions Taken By Stakeholders as a Result of Programs
Progress Achieved Against Goals
Perceptions Changed or Reinforced
NPO Both Funders
Funding Received
Funding Given
Message Delivery
Media Coverage
Sustainability of Programs
MOST IMPORTANT*
*Ranked 1 or 2 as “most important” by the majority of survey respondents
*Ranked 9 or 10 as “least important” by the majority of survey respondents
LEAST IMPORTANT*
Organizations Revealed Best Practices for Measurement
Ongoing review of progress against goals
Broad program evaluation & follow-up
Digital reporting
Tracking research advances
Secondary Research FindingsHow NPOs Publicly Report Progress Against Goals via Websites and Annual Reports
Cancer NPOs Have a Strong Commitment to Reporting Results…
91%Publish an annual
report
89%Report annual revenue &
operating expenses
60%Clearly state
organizational goals
46%Report progress against
each goal
…and a Strong Commitment to Transparency
Approximately 80% identify corporate and individual sources of grants
Majority report funding given and funding received (65% and 70%, respectively)
Dollar size, number and impact of grants both given and received are often disclosed
Strong Social Media Presence, Yet Less Often Measured or Reported
97%
97%
91%91%
60%49%
46%Measure impact on progress based on social media or web engagement
29%Quantify growth of social media engagement
Website Research Shed Light on NPO Priority Metrics
Audience reach/event attendance (91%)
Legislative/policy work (74%)
Scientific progress (63%)
Survey patients about impact of programs
(23%)
Media coverage (43%) Message delivery (9%)
Website hits (29%) Unique visitors (31%)
MORE COMMON LESS COMMON
Key Takeaways and Study Implications
Metrics increasingly valued by NPOs, driven largely by funder expectations
▪ Tasked with finding more immediate measures of success▪ Limited resources and knowledge on how to best convey impact
Variety of reporting techniques; more “outputs” than “outcomes”▪ Funders and NPOs largely aligned in metrics they deem most/least important▪ More focused on reporting results than websites/annual reports indicate
NPO demographics have marginal bearing on metrics activities▪ Effective metrics reporting not necessarily linked to size or revenues
Diverse stakeholders and funding obstacles will continue to challenge metrics reporting, but best practices and cost-effective strategies exist to showcase impact
Your Guide to Intelligent Giving
Where the Heart
Meets the Mind
Critical Friend to Charities
The Core Challenge As I See It*“… there is virtually no
credible evidence that most nonprofit organizations actually produce any social value.”
*“The End of Charity” by David Hunter – Philadelphia Social Innovations Journal
The Nonprofit Marketplace Hewlett Fdn
THE ROADMAP TO A SOLUTION TO THE CHALLENGESINSTITUTIONALFUNDERSsupply $ & technical assistance formeaningful information
BENEFICIARIESare engaged& empoweredto providemeaningfulinformation
InstitutionalFunders usea set of standardizedNP reports of meaningful information
Beneficiaries receive feedback on how their meaningful information was utilized
WHAT IS MEANINGFUL INFORMATION FOR MEASURING A CHARITY’S SUCCESS?
RESULTSGOVERNANCE
FINANCIAL HEALTH
How Does This Fit into the Evolution of our Rating System?
• CN 1.0 – Financial HealthLaunched in 2002 with 1,100 charities
• CN 2.0 – GovernanceLaunched Sept 20, 2011 with 5,500 charities
• CN 3.0 – Results ReportingMethodology released and data collection begun Jan 2013, with a goal of 10,000 charities rated by end of 2016
Results Reporting Is…• An assessment of how charities use their results
internally and share them with stakeholders, including donors
• We are looking to see that you are reporting on results measures, and showing how your organization learns and improves based on those measures (i.e. learning and improving over time is more important than a ‘snapshot’ of results)
How Charity Navigator Evaluates Results Reporting
FIVE ELEMENTS:1. Alignment of Mission, Solicitations
and Resources
2. Results Logic and Measures
3. Validators
4. Constituent Voice
5. Published Evaluation Reports
One Element Will Impact Ratings Next Year
Element One: Alignment of Mission, Solicitations and Resources
Element Two: Results Logic and Measures
• Is the organization’s causal logic, theory of change, plausible?
• If applicable, is there an indication of how much of the action is required to produce the pre-defined outputs and outcomes?
• Is this logic based on reasonable evidence?• Are there specified measures (indicators) to be collected
and a plan to do so?
Element Three: Validators
• Have your charity’s results been vetted by another organization?
• Not every charity and cause area will have a validator. That will not diminish your rating.
Element Four: Constituent Voice
• How well does your charity collect and publish feedback from your primary constituents (the people who are meant to be the direct recipients of benefits created by the organization’s actions).
• May not apply to every cause area, but will apply to most.
Element Five: Published Evaluation Reports
• Does your charity publish evaluation reports that cover the results of its programs at least every five years?
• Are those reports based on recognized techniques to understand your results?
• Does you charity explain what, if anything, it is changing as a result of the findings in the evaluation report?
WHAT WE HAVE LEARNED SO FAR
Out of Approximately 1,000 Human Service Charities Evaluated So Far:
• ~ 70% of Charities have a clearly defined program
• ~ 15% of Charities have a clear definition of how much of an action is required to produce the pre-defined outputs and outcomes
• ~2% of Charities have at least 1 of the other 3 elements of Results Reporting (Validators, Constituent Voice or Published Evaluations)
A DREAM THAT YOU CAN PLAY A ROLE IN MAKING A REALITY
RESULTS STATEMENTS,
TODAY TOMORROW?????
To Learn More About How We Measure Success:
To Learn More In General About Results Measurement:Books /Articles/ Studies
1. The Battle for the Soul of the Nonprofit Sector, Berger, Penna and Goldberg, Philadelphia Social
Innovation Journal
2. Money Well Spent by Paul Brest, et. Al
3. Working Hard and Working Well, by David E. K. Hunter
4. Billions of Drops in Millions of Buckets by Goldberg
5. Leap of Reason by Mario Marino
6. The Nonprofit Outcome Toolbox by Dr. Robert Penna
7. Charity Navigator’s webinar on how to use our site
8. Saving Philanthropy Video
9. Independent Sector’s Charting Impact
10. PerformWell Web Site
Email: kberger@charitynavigator.org
Blogs:
www.kenscommentary.org
blog.charitynavigator.org
Twitter: kenscommentary
LinkedIn: http://lnkd.in/jKeK5U
Register for Newsletter, Etc.: https://www.charitynavigator.org/index.cfm?bay=my.login
KEEPING IN TOUCH
Evaluation:Corporate PerspectiveJuly 2014
Carmen Perez cperez@cecp.co 212.825.1581
THE CEO FORCE FOR GOOD
Utilities (n=18)
Technology (n=25)
Materials (n=16)
Industrials (n=25)
Health Care (n=24)
Financials (n=48)
Energy (n=11)
Consumer Staples (n=20)
Consumer Discretionary (n=26)
Communications (n=10)
1.23%
1.06%
0.98%
0.76%
1.58%
0.96%
0.76%
1.12%
1.25%
1.10%
2013 Industry Giving Comparisons – Median Giving as a % of Pre-Tax Profit
For Pharmas: 16.1%
Pharmaceuticals Giving at a Glance, 2013
79%of all companies run a corporate foundation
86%of all companies match
employee gifts
62% of all companies give
internationally
64%of all companies make
non-cash gifts
78% of Pharma
companies
89% of Pharma
companies
100% of Pharma
companies
78% of Pharma
companies
N=261 for full sample. Pharma n=9.
A Majority of Companies are Measuring the Societal Value of their Contributions
Very Experienced (5 Years or More)
Moderately Experienced (3-4 Years)
Slightly Experienced (2 Years or Less)
18%
35%
47%
76% Measure Outcomes and/or Im-
pacts
N=160. N=119.
24% Do Not Measure
Outcomes and/or Impacts
Nearly a Third of Companies Focused On Outcome/Impact Measurement in a Strategic Philanthropic Program
42
• For companies only evaluating grants larger than a specific threshold, the average threshold was approximately $105,000
• “Other” descriptions include companies that: • Only evaluate foundation grants• Evaluate for a combination of strategic programs, grant size, and cause area(s)• Evaluate for as many grants as possible without any specific direction
Description of Impact Measurement Scope % of Companies
All grants, regardless of grant size 23%
Only grants made for a strategic philanthropic program(s) 31%Only grants larger than a specific threshold 16%Only grants larger than a specific threshold AND made to a specific cause area(s) 12%Only grants made to a specific cause area (or cause areas) 8%
Other 10%
How much do companies report spending on evaluation?
43
48%of companies spent
money in 2013 to evaluate grants
5%of 2013 contributions were
dedicated to measuring the outcomes and/or
impacts associated of their grants
Note: Spending money includes contributions to grantees (earmarked for evaluation) and payments to consultants, contractors, and evaluation specialists. N=77.
Note: Median. Includes companies spending money on evaluation. N=37.
Companies Use a Variety of Resources for Measurement Activities in Strategic Giving Programs
Internal R
esource
(Entire
ly In-
House)
Consu
lting
Firm(s)
Internal R
esourc
e (Infor
med by E
xterna
l Reso
urces)
Universi
ties
Publicly
Ava
ilable
Metrics
Resear
ch In
stitutio
n(s)
Other
26% 26%
20%15%
11% 9% 7%
N=105.
66% of Companies Partner with Grantees to Measure the Outcomes and/or Impacts of their Grants
45
% of Strategic Programs Focused on:
Common impact metrics for Education included:
• Literacy Rates• Graduation Rates• College Readiness Scores• Bachelor’s Degree Attainment
Rates• Racial Disparity Changes• Attendance Rates• Teacher Retention Rates• # of girls/women supported• # of people obtaining access to
technology
Common impact metrics for Health & Social Services
included:
• Emergency Room Visits• Hospital Recidivism• Immunizations Administered• Patient Engagement• Number of children receiving
immunizations• Increased farmer yields• Other donors leveraged for
marginalized community• Improved health resiliency
Common impact metrics for Community & Economic Development included:
• Number of Women Impacted• Credit Score Changes• Net Worth Changes• Number of Loans Administered• Number of Jobs Created• Business Growth Metrics• Business Profit Changes
Education: 41% Health: 26% Econ Dev:16%
There’s A Lot Out There To Help!
46
PSI – Impact Magazine Urban Institute: PerformWellHealth and Safety Section
Center for Disease Control and Prevention
Steps CommunitiesCHANGE Communities
RAND Getting to Outcomes 2004ISBN 0-8330-35285-2
Nurse-Family Partnership Theory of Change Logic Model
Foundation CenterTools and Resources for
Assessing Impact
Evaluation:Corporate PerspectiveJuly 2014
Carmen Perez cperez@cecp.co 212.825.1581
Questions & Answers
Grants Announcement
• In partnership with a major university specializing in social innovation and metrics
• Call for entries open late summer• NPOs with a healthcare focus and a demonstrated commitment to
metrics reporting eligible • Submit your email address via post-webinar survey for more
information
The Onyx Advancing Innovation in Measuring Advocacy Outcomes Award
Thank you!
top related