understanding policy, systems, and environmental strategies … · 2019-04-29 · importance of pse...

Understanding Policy, Systems, and Environmental

Strategies &

Evaluation

*Presentation adapted from http://www.health.state.mn.us/healthreform/ship/docs/UnderstandingPSE.pdf

Outline of Presentation •  What is policy, systems, and environmental

(PSE) change? •  How do we evaluate PSE strategies?

Moving Upstream - Story

• While walking along the banks of a river, a passerby notices that someone in the water is drowning. After pulling the person ashore, the rescuer notices another person in the river in need of help. Before long…

Importance of PSE Change •  Where you live affects how you live •  Major health problems will not be solved solely by

individual actions and choices •  By moving upstream to address causes and

improving environments where we live, work, learn, play, and receive health care, we can prevent many people from becoming chronically ill

Events/Programs vs. PSE Change

Characteristics of Events/Programs

•  One time •  Additive: often results in

only short-term behavior •  Individual level •  Not part of ongoing plan •  Short term •  Non-sustaining

Characteristics of PSE Change

•  Ongoing •  Foundational: often

produces behavior change over time

•  Policy level •  Part of an ongoing plan •  Long term •  Sustaining

Spectrum of Prevention

Socio-Ecological Model

McLeroy 1988; Livewell Colorado 2007

© Randy Glasbergcn www.glasbargen.com

"They tr ied adding healthy snacks t o the office vending machine s, but a l l tha t ro t t ing

fruit made the candy bars t as te bad . "

Individual Behavior Change

•  Personal choices are made in context of larger environment

•  Many health and social problems are related to conditions outside the individual’s control

•  Recently there is a growing sense of importance of broader societal trends and policies that affect behaviors

•  While programs can lead to behavior changes in individuals and/or communities during course of programs, unless scaled up and instituted through formal and informal PSE changes, programs may not be sustainable

What is policy change? •  Policy strategies may be a

law, ordinance, resolution, mandate, regulation, or rule (both formal and informal).

•  Some of the most respected policies in tribal communities are not written, specifically policies regarding ceremony.

Tribal Tobacco Policies

•  Non-smoking policy in or around tribal facilities •  Signs to reinforce the policy •  Enforcement

Nutrition Policies

What is environmental change?

•  Environmental strategies involve physical or material changes to the economic, social, or physical environment.

•  Example: Incorporating sidewalks, paths, and recreation areas into community design.

Exercise Facilities

•  Ensuring the community has access to spaces where they can be physically active. •  GranteeA-hasafacilitywithequipment,howeverneedsstaffto

managethesiteestablishopenhoursandmakethespaceavailable.

•  GranteeB-isworkingtoclearbrushfromanoutdoorspacewhereequipmentcanbeinstalledandiscentrallylocatedinthecommunity

•  GranteeC-hascreatedapolicyforemployeestobeabletouseonsitegym

What is systems change? •  Systems strategies

are changes that impact all elements of an organization, institution, or system. •  Systems changes

are often directly related to a policy change

•  Types of systems include: school, transportation, etc.

Tribal Examples

•  Grantee A- referrals from providers to CHRs to link patients to community services such as hospitals, specialized care for chronic disease diagnoses and management

•  Grantee B- integration of behavioral health with medical department

Examples from Schools Programs/Events

Celebrate National Nutrition Month

Host a Family Fitness Night

Participate in Walk to School Day

Provide healthy snacks or breakfast

Participate in Kick Butts Day

Provide health screenings for staff

Policy, Systems, and Environmental Change

Add fruits & vegetables to a la carte

Make school athletic facilities regularly available to families

Establish a Safe Routes to School Program

Adopt a Healthy Food & Beverage Policy

Establish a tobacco-free school taskforce Establish a building-sponsored wellness team

Importance of Partnership

Can this work be accomplished without partnerships? No!

Research shows the importance of: •  Public-private partnerships for policy change

success. •  Embedding targeted policy and environmental

changes in broader community initiatives whenever possible.

•  Continually engaging and interacting with broader community.

Potential Partners

•  Tribes •  Tribal Health Programs •  Elected officials •  School districts and YOUTH •  Other State and National agencies •  Other local public health agencies and health care

providers •  Other local government agencies •  Coalitions, community- and faith-based

organizations, non-profits, etc. •  Private interests and/or companies (media!) •  Remember PSE strategies are supposed to touch

where you live, work, learn, play, and receive health care!

More about evaluation….

Program, Evaluate/Evaluation

Program: a set of resources and activities directed toward one or more common goals Evaluate: to ascertain the worth of or to fix a value on some object Evaluation: the making of a judgment about the amount, number, or value of something Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2004). Meeting the Need for Practical Evaluation Approaches: An Introduction. In Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.), Handbook of Practical Program Evaluation (2nd ed., pp. xxxiii-xliv). San Francisco, CA: Jossey-Bass. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). An Overview of Program Evaluation. In Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (Eds.), Evaluation: A Systematic Approach (7th ed., pp. 1-30). Thousand Oaks, CA: Sage Publications. http://www.oxforddictionaries.com/us/definition/american_english/evaluation

Program Evaluation [Textbook 1] •  systematically assesses the:

o  programresultsandextenttowhichtheprogramcausedthoseresults;and

o  idenCfieswaystoimprovetheprogram [Textbook 2] •  systematically investigates the effectiveness of social

programs in ways that: o  areadaptedtotheirpoliCcalandorganizaConalenvironments;

ando  aredesignedtoinformsocialacContoimprovesocialcondiCons

Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2004). Meeting the Need for Practical Evaluation Approaches: An Introduction. In Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.), Handbook of Practical Program Evaluation (2nd ed., pp. xxxiii-xliv). San Francisco, CA: Jossey-Bass. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). An Overview of Program Evaluation. In Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (Eds.), Evaluation: A Systematic Approach (7th ed., pp. 1-30). Thousand Oaks, CA: Sage Publications.

Program Evaluation (cont.)

Definitions with common focus on: •  systematic approach;

•  program results/effectiveness; and

•  improvement of o program;and/oro socialcondiCons

Why Evaluate? •  assess program impact

•  determine if the program is:

o  meeCnggoalso  havingtheexpectedeffect

•  help identify areas for program improvement

•  share knowledge

•  provide support for future funding http://www.ucdenver.edu/academics/colleges/PublicHealth/research/centers/CAIANH/REACH/Documents/Evaluating%20PSE%20Strategies%205-8-2013.pdf

Evaluation is Systematic •  standards

•  framework/steps

•  adapted to: o  context;o  condiCons;ando  culture

http://www.cdc.gov/eval/framework/index.htm

Evaluation Standards Utility •  useful to

o  stakeholders;ando  helpimproveeffecCvenessandimpact

Feasibility •  practical, realistic

o  basedonresourcesandcapacity;ando  withrespecttocontext,condiCons,andculture

Propriety •  legal, ethical

o  concernforthewell-beingofthoseinvolvedin,servedby,andaffectedbytheprogram

Accuracy •  convey accurate information http://www.cdc.gov/eval/standards/index.htm

Evaluation Framework Engage Stakeholders

•  engage those involved in, served by, and affected by the program and evaluation

Describe the Program •  describe need for and context of the program, as well as expected activities

and effects Focus the Evaluation Design •  assess the greatest priorities and issues the program intends to address Gather Credible Evidence •  collect data, tracking activities and progress, leading to findings about the

program Justify Conclusions •  link conclusions to evidence, applying standards Ensure Use and Share Lessons Learned •  share evaluation data, analysis, results, and knowledge http://www.cdc.gov/eval/steps/index.htm

Evaluation Stages, Methods & Tools Planning & Design

•  Needs Assessment

•  Evaluability Assessment

•  Logic Model

Implementation & Process

•  Implementation Evaluation / Process Evaluation

Outputs & Outcomes

•  Cost-Effectiveness and Cost-Benefit Analysis

•  Outcome Evaluation

•  Impact Evaluation

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (Eds.). (2004). Evaluation: A Systematic Approach (7th ed.). Thousand Oaks, CA: Sage Publications. Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (2004). Handbook of Practical Program Evaluation (2nd ed.). San Francisco, CA: Jossey-Bass.

Evaluation: Planning & Design • Needs Assessment

o  assessestheneedfortheprogramandhowtoaddressneedsthroughtheprogram(e.g.,adaptedCHANGEToolsurveys)

•  Evaluability Assessment o  helpsidenCfyanddesignevaluaConsthatmaybeusefuland

pracCcalo  considersprogramgoals,prioriCes,performance

informaConanduse,costs

•  Logic Model o  adiagrampresenCnghowtheprogramwillwork,tellingthe

storyofexpectedprogramperformanceo  focusesoninputs,acCviCes,outputs,andoutcomes

Logic Model (linear) à à à

resourcesrequiredtosupporttheprogram

alloftheacConstepsnecessarytoproduceprogramoutputs

whatisproducedbyprogramacCviCes

changesinclientsorcommuniCesresulCngfromprogramacCviCesandoutputs(oMendividedintoshort-term,intermediate,andlong-termoutcomes)

Inputs Ac5vi5es Outputs Outcomes

Short-TermOutcomes:changesorbenefitsresulCngfromtheprogram’soutputs

IntermediateOutcomes:changesorbenefitsresulCngfromtheshort-termoutcomes

Long-TermOutcomes:programimpactsaccruedthroughtheintermediateoutcomes

McLaughlin,J.A.,&Jordan,G.B.(2004).UsingLogicModels.InWholey,J.S.,Hatry,H.P.,&Newcomer,K.E.(Eds.),HandbookofPrac-calProgramEvalua-on(2nded.,pp.7-32).SanFrancisco,CA:Jossey-Bass.

Logic Model (linear) PSE example: Walking Path

à à à Inputs Ac5vi5es Outputs

OutcomesShort-TermIntermediateLong-Term

StaffFundingCommunityHealthAssessmentData,CAPTCTTribalLeadershipCommunityCRIHBExisCngpartnerships

TCTmeeCngsPolicyprogressPartnershipcreaConCommunicaCngwithTribalLeadership,community,CRIHBDatacollecConReporCngEvaluaCon

WalkingpathPhysicalacCvitypolicies#oftribalcommunitymembersusingwalkingpath

IncreasedknowledgeofopportuniCesforphysicalacCvityamongtribalcommunityIncreaseduseofWalkingPathIncreasedopportuniCesforintegraConofphysicalacCvitypoliciesacrosstribalcommunity,aswellashealthcare,school,andworksitesectors

SustainedphysicalacCvityamongtribalcommunitymembersIncreasedintegraConofphysicalacCvitypoliciesacrosstribalcommunity,aswellashealthcare,school,andworksitesectors

Decreasedheartdisease,diabetes,andobesityratesamongtribalcommunityReducConinmedicalcostsamongtribalcommunitymembersImprovedqualityoflifeamongtribalcommunitymembers

Logic Model (circular)

Evaluation: Implementation & Process, Outputs & Outcomes

Implementation & Process •  Implementation Evaluation / Process Evaluation

o  assesseshowwelltheprogramacCviCesmatchthosethatwereplannedo  assesseshowprogramoutcomesarebeing/wereachieved

Outputs & Outcomes •  Cost-Effectiveness and Cost-Benefit Analysis

o  assessesprogrameffecCvenessandefficiencyo  considersprogramcostsandoutputs/outcomes

•  Outcome Evaluation o  assessestheeffecCvenessofaprograminproducingintendedoutcomes

•  Impact Evaluation o  assessesoverallprogrameffects,bothintendedandunintended

Evaluation Opportunities: Tribal Commercial Tobacco Strategy

•  # of policy meetings, # of meeting participants

•  # of sectors represented in policy meetings

•  # of partnerships; partnership narrative

•  # of coalitions; coalition narrative

•  # MOUs; MOU narrative

•  narrative of policy progress (e.g., item on agenda, policy formulation, policy adoption, implementation)

•  narrative of policy implementation, leading to system and environment change

o  Signage(#ofno-smokingsigns;narraCveofplacementstrategy)

o  Enforcementofpolicy/policies

•  integration with health care sector (e.g., tribal health office), narrative description (including policy, system, and environment progress/implementation)

o  #referralstoCaliforniaSmokersHelpline;

o  #ofmaterialsdisbursed(e.g.,brochures,mailings,publicaCons)toeducatecommunityoncommercialtobaccoandsecondhandsmoke

•  integraConofcommercialtobaccoeducaConwithotherchronicdiseaseprevenConandmanagementstrategies(e.g.,healthyeaCng,localfoods,physicalacCvity)

o  #ofcommunityeventsinvolvingcommercialtobaccoeducaCon;#oftribalcommunitymembersreached

o  #oftrainingsoncommercialtobaccoeducaConprovided;#oftribalcommunitymembersreached

•  integration with school district, work site, and/or health care sector; narrative description (including policy, system, and environment progress/implementation)

•  surveys, interviews, focus groups

Evaluation Opportunities: Workplace Nutrition Strategy


•  # of departments represented in policy meetings

•  narrative of committee creation, engagement and integration with other organizational operations





o  on-sitefoodavailable(e.g.vendingmachines,cafeteria)

o  healthyfoodopConsatmeeCngsandevents

o  creaConofworkplacesupportgroups,commidees,newsleders,incenCveprograms

o  registereddieCcianrenderingtrainings(#oftrainingsprovided,#employeesadendingtrainings)

o  employeeevaluaConofpolicy,feedbackandcomments

•  # of employees reached

•  # of educational materials disbursed (e.g., brochures, recipe guides) and/or signage (e.g., in break room, encouraging healthy food and beverage choices)

o  integraConofeducaConwithotherchronicdiseaseandmanagementprevenConstrategies(e.g.,physicalacCvity,commercialtobacco),narraCvedescripCon

•  integration with local food provision (e.g., community garden, farmers’ markets), including policy, system, and environment progress/implementation

•  surveys

•  Interviews

•  focus groups

Evaluation Opportunities: Tribal Community Walking Path



•  # of partnerships (e.g., school district, tribal government office, workplace, health department/office); partnership narrative


•  # joint use agreements; narrative



•  # people using walking path, tracking sheets

•  signage, lighting, access to health educational materials and maps; narrative

o  #ofhealtheducaConmaterialsdispersed(e.g.,exerciseguides,factsheets)

o  integraConofeducaConwithotherchronicdiseaseprevenConandmanagementstrategies(e.g.,healthyeaCng,localfoods,commercialtobacco),narraCvedescripCon

•  integration of walking path with other exercise policies (e.g., health care sector, school district, workplace), encouraging use

•  creation of walking groups, use for community events; narrative

•  surveys

•  interviews

•  focus groups

Evaluation Opportunities: Community-Clinical Linkages




o  partnershipsbetweenclinicproviderandpartners(e.g.,fitnessfaciliCes/classes,nutriConists/nutriConclasses,schools,parksandrecreaConoffice/department,localhealthagenciesandspecialists[suchastradiConalherbalists,CommunityHealthRepresentaCve,pharmacists,publichealthnurses,casemanagers,communityhealthworkers,etc.)




•  narrative of system integration

•  integraConwithotherchronicdiseaseprevenConandmanagementstrategies(e.g.,physicalacCvity,healthyeaCng,localfoods,commercialtobacco);narraCve

•  # of clients referred

•  surveys

•  Interviews

•  focus groups

}clientsandproviders

QUESTIONS?

ACORNSWebsite:www.crihbacorns.orgVirginiaHedrick,MPH,(Yurok/Karuk)[email protected],[email protected]