the why behind the what: a process evaluation of healthy@work
DESCRIPTION
The why behind the what: A process evaluation of Healthy@Work. Dr Theresa Doherty Dr Fiona Cocker. Seminar Aims. Introduce a methodology for process evaluation Apply this to the H@W program Identify key evaluation questions Demonstrate what existing program data can answer - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/1.jpg)
The why behind the what:
A process evaluation of Healthy@Work
Dr Theresa DohertyDr Fiona Cocker
![Page 2: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/2.jpg)
Seminar Aims
• Introduce a methodology for process evaluation
• Apply this to the H@W program• Identify key evaluation questions• Demonstrate what existing program data can
answer• Identify what other data we need • Explain how we are going to collect it
![Page 3: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/3.jpg)
Defining Evaluation
• The systematic acquisition and assessment of information to provide useful feedback about some object
• ‘The process of determining the worth, merit, or value of things …or the result of that process.’ (Scriven 1991)
• ‘The systematic collection of information about the activities, characteristics, and/or outcomes of programs to make judgements about the program, improve program effectiveness, and/or inform decisions about future programming.’ (Patton 1997)
![Page 4: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/4.jpg)
Purpose of Evaluation
‘Evaluation determines the merit, worth, or value of things. The evaluation process identifies relevant values or standards that apply to what is being evaluated, performs empirical investigation (often) using techniques from the social sciences, and then integrates conclusions with the standards into an overall evaluation or set of evaluations.’ (Scriven, 1991)
![Page 5: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/5.jpg)
Evaluation vs. Research
• The main difference between research and evaluation is that research is conducted with the intent to generalize the findings from a sample to a larger population. (Priest 2001)
• Evaluation focuses on an internal situation with no intent to generalize the results to other settings and situations.
• Research generalizes, evaluation particularizes.
![Page 6: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/6.jpg)
Same or Different ?
‘Evaluation and research are closely related and should be synergistic but they serve different
purposes.’ (Fain)
Evaluation Research Particularises GeneralisesSeeks to improve Seeks to proveInforms decision making
Draws conclusions
Asks ‘so what?’ Asks ‘what’s so?’How well it works How it worksWhat is valuable What ‘is’ (Matheison)
![Page 7: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/7.jpg)
Meaningful Evaluation
The more interesting endeavour for evaluation:– how to do rigorous and robust
evaluation
We need a plan so we can be confident:– Conclusions are sound; – Recommendations will improve program
delivery and outcomes
![Page 8: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/8.jpg)
Evaluation Plan Checklist
1. Program clarification2. Purpose of evaluation3. Stakeholder assessment4. Program logic underpinning this program5. Scope of evaluation6. Data collection7. Data analysis8. Timeline for evaluation9. Report and dissemination 10. Budget requirements
![Page 9: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/9.jpg)
Program Logic Approach
• Clarify the purpose of a project– what change do you want to bring
about?• Plan how to bring about this change• Allow yourself to explore and test the
assumptions underlying your approach• Test how well your approach is working
and what, if anything, needs changing• Find ways of demonstrating your approach
is working
![Page 10: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/10.jpg)
The Logic Model
What are our results?
INPUTS OUTPUTS OUTCOMESProgram
investments
Short-term
What we
invest
Activities, outputs
What we do
or produc
e
Target
Who we
want to
reach
Medium-term
Long-term
![Page 11: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/11.jpg)
Program Logic Approach
• Provides a program description that guides our evaluation process
• Helps match the evaluation to the program
• Helps identify what and when to measure– outputs and outcomes
• Helps identify key data needs• Makes evaluation an integral part of
program planning and implementation
![Page 12: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/12.jpg)
University of Wisconsin-Extension, Program Development and Evaluation
Logic Model & Evaluation
Types of Evaluation
![Page 13: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/13.jpg)
Process evaluation
Good ideas do not always yield good results
• Process evaluation looks at how a program was implemented
• It assesses the extent to which planned activities were carried out
• Ideally this happens AS a a program is being implemented NOT retrospectively
• It alerts us to changing conditions that may impact on the program
• It allows us to make adjustments to implementation to keep on track OR
• Rethink whether the program design is the right one
![Page 14: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/14.jpg)
pH@W Process Evaluation Project
• Process evaluation was not in the original pH@W project plan
• No specific funding allocated BUT
• Investigators decided it was an important part of understanding Healthy@Work
![Page 15: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/15.jpg)
Purpose of the Evaluation
• What do we want to know?• How was Healthy@Work
implemented across the Tasmanian State Service?– What happened?– What happened, in what
circumstances which led to particular outcomes?
– In light of these findings, would we implement a workplace H&WB program differently?
![Page 16: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/16.jpg)
The Issue of Scope
What is achievable given time and resource limitations?• How was the H@W program conceptualised
and developed?• To what extent did TSS Agencies engaged with
and implement H@W as designed?• What are the implications for sustainable
health promoting workplaces?
![Page 17: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/17.jpg)
H@W Program Logic
![Page 18: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/18.jpg)
H@W Process Evaluation Program Logic - Mark 1
INPUTS OUTPUTS TARGET ST OUTCOMES H@W Funding H@W Project advisory group H@W program Manager H@W Project officer Partnerships with agencies Partnerships with service providers Governance structures Partnering H@W (DPAC, DHHs, UTAS, MRI)
DPAC Ministerial directive Healthy@work website Healthy@work online
survey H&WB Grant program ? marketing and
promotion to agencies AGENCIES X12 Response to MD Develop H&WB governance structures
Develop H&WB implementation plans
Promote H@W to employees
Identify potential H&WB strategies/interventions
H@W coordinators employed
H@W grant
submissions
Heads of Agencies TSS Employees Managers/employers Employees Service providers
MD implemented across TSS H of A engage with H@W Employees engage with H@W Agency H&WB profiles Awareness and knowledge of H&WB Access H&WB programs /information Employees engage with H&WB programs Contracts/agreements with H&WB service providers H@W grants
![Page 19: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/19.jpg)
H@W Process Evaluation Program Logic -Mark 3
INPUTS OUTPUTS TARGET ST OUTCOMES MT OUTCOMES LT OUTCOMES
Ministerial direction H@W Funding H@W Comms and marketing Strategy H@W Project advisory group H@W program staff H@W website H@W online survey H@W Grant program Partnerships with agencies Partnerships with service providers Governance structures Partnering H@W(DPAC, DHHs, UTAS, MRI)
Agency response to MD • Use the H@W
guidelines
• Develop H&WB governance structures
• Develop H&WB implementation plans
• Allocate resources for H@W coordinators
• Promote H@W to employees
Used t he H@W website
Participate din the online survey
Identify H&WB interventions
Apply for H@W grants
STATE SERVICE AGENCIES HEAD of
AGENCY MANAGERS SS EMPLOYEES
SERVICE PROVIDERS
H@W is implemented across TSS H of A engage with H@W Employees engage with H@W Awareness and knowledge of agency H&WB profiles Awareness and knowledge of H&WB risks Access to H&WB programs /information Employee engagement with H&WB programs TSS Employees are motivated to change health risk behaviours Contracts/agreements with H&WB service providers in place H@W grants secured
TSS workplaces have H@WB policies TSS Workplace have health promoting infrastructure /programs in place TSS Employees have health promoting behaviours
Sustainable H&WB programs Health promoting TSS workplaces Healthy TSS workforce
FOCUS of PROCESS EVALUATION
![Page 20: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/20.jpg)
Process Evaluation Activities
1. Review and analyse H@W files – the ‘formal’ story
2. Interview key informants – ‘informal’ story
3. Synthesise these data – deeper understanding of H@W
4. Report the findings – what are the implications of this ‘new’ knowledge?
5. Disseminate the findings – who are the stakeholders?
![Page 21: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/21.jpg)
What do we want to know – Example outputs
Outputs cont.(from your logic plan)
What you produce/what you do
What do we want to know?
Performance indicatorWhat can you count/how will
you know?
How will we
find out?How will we collect the
dataMD implemented To what extents did
agencies follow the guidelines?
# who conducted a needs assessment# who set up a committee# who engaged management support# who developed a H&WB plan
Audit data
H&WB activities in place
What sort of H&WB programs did agencies implement?
#r /type of programs implemented
Audit data
Agency H&WB coordinators (HWBC) allocated
• Did agency create a new position?
• Were H&WB responsibilities added to an existing position?
• Did agency allocate specific resources to this position?
• Where in the org structure does this position fit?
# H&WBC positions created # H&WBC positions filled # agencies with a H&WBC
Agency recordsKey informants
![Page 22: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/22.jpg)
What do we want to know - Example outcomes
ST Outcomes (logic plan)
What you achieve
What do we want to know?
Performance indicatorWhat can you count/how will you
know?
How will we find out?
How will we collect the
dataMD is implemented across TSS
How much progress had agencies made toward implementing a H&WB program in place as prescribed by the MD Did agencies experience barriers to implementation?
# agencies with an approved H@WB plan # of agencies who identified barriers
Agency recordsKey informants
H of A engage with H@W
How did H of A support the H@W initiative?
Funding allocated to H&WB activitiesChanges to organisational structure to accommodate H&WBH of A membership of H&WB governance structures
Key informantsMinutes of meetings
Employees engage with H&WB programs
Did H&WB program meet employees’ needs?Did any employee groups not participate?Did employees experience financial barriers to participation
Participation rates in programsEmployee satisfaction with H&WB activitiesEmployee participation on H&WB committees/ working groupsEmployee identified barriers to participation
Key informantsAgency records
![Page 23: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/23.jpg)
Existing Data Sources
• Annual audit data• Healthy@Work agency reports • Grants program – applications, progress/final
reports• H@W project review and closure report
What they reveal• Baseline H&WB capacity • The sort of activities implemented• Where the grant money was allocated
![Page 24: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/24.jpg)
How will we find out?
Data sources available• Annual audit data• Healthy@Work agency reports • Grants program – applications, progress/final
reports• H@W project review and closure report
What they reveal• Baseline H&WB capacity • The sort of activities implemented• Where the grant money was allocated
![Page 25: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/25.jpg)
Performance Indicator:Needs Assessment
Audit Year
Top Three Priorities Bottom Three Priorities
2010 NutritionSedentary behaviourStress/Psych distress
SmokingHydrationAlcohol
2011 NutritionStress/Psych distressPhysical activity
SmokingHydrationAlcohol
2012 NutritionPhysical activityStress/Psych distress
AlcoholOtherSmoking
• All agencies completed a needs assessment (2010-2012)• Were the organisation’s H&WB issues/needs identified?
![Page 26: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/26.jpg)
Performance Indicator:Needs Assessment
Audit Year
Top Three Motivators
2008/09 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Recruitment / retention strategy
2010 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Ministerial direction
2011 1. Increase health and wellbeing of staff2. Increase engagement / morale of staff3. Increase productivity
2012 1. Increase health and wellbeing of staff2. Injury prevention3. Staff demand
![Page 27: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/27.jpg)
Performance Indicator: Needs Assessment
Audit Year
Top Three Barriers
2008/09 1. Lack of human resources 2. Difficulty agreeing on the program
content/structure 3. Never thought about it
2010 1. Unable to justify costs 2. Insufficient financial resources3. Employee time constraints
2011 1. Employee time constraints 2. Insufficient financial resources3. Lack of human resources
2012 1. Employee time constraints 2. Lack of employee interest 3. Unable to justify costs against other priorities
![Page 28: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/28.jpg)
Performance Indicator:Committee/staffing support?
Resource Allocation 2008/09(14
Agencies, Authorities)
2010(14
Agencies, Authorities)
2011(14
Agencies, Authorities)
2012(19
Agencies, Authorities
)Position/s - program part of normal role 5 9 12 13No allocated position - Volunteer(s) given time to work on project
1 1 1 1
No allocated position. Volunteer/s works on project and normal duties
2 0 1 1
A steering committee/working group given time to work on project
1 5 11 14
A steering committee/working group works on project and normal duties
6 0 1 0
Examples of responsible positions:2008 – Senior HR officer (1/2 hr/week); Senior HR consultant (2 hr/week)2010 – H@W Policy & Program Officer (23 hr/week)2011 – H&WB co-ordinator (up to 20hr/week)2012 – OH&S Manager (7hr/week); HR co-ordinator (4hr/week)
![Page 29: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/29.jpg)
Performance Indicator:Engaged Management?
Examples of Senior Management Engagement
2010(14 Agencies, Authorities)
• Executive Group reviews and approves proposals for H&WB initiatives
• Approves allocation of resources for Agency • Approved & participate in the program• Are very supportive of health & wellness in the workplace
2011(14 Agencies, Authorities)
• Fully endorse & participate in program• Allow the program to be conducted in work time• Encourage all staff to get involved• Committed to having a budget for H&WB initiatives
2012(19 Agencies, Authorities)
• Provision of information, resource allocation and reporting on progress
• Developed the framework• Oversee program content• Have a key role in supporting staff to actively engage in
H&WB activities• Funding support for various programs
![Page 30: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/30.jpg)
Performance indicator:H&WB plan?
H&WB Plan
2010(14
Agencies, Authorities
)
2011(14
Agencies, Authorities
)
2012(19
Agencies, Authorities
)
No 8 3 4Yes 6 11 15
![Page 31: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/31.jpg)
Agency Evaluation of H&WB Program
Audit Year
Most commonly cited method of evaluation
2010 1. Number of participants taking part2. Employee satisfaction with program3. Number of initiatives/activities undertaken
2011 1. Number of participants taking part2. Employee satisfaction with program3. Employee satisfaction with organisation
2012 1. Number of participants taking part2. Employee satisfaction with program3. Number of initiatives/activities undertaken
![Page 32: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/32.jpg)
Still to come
Further examination of:• Annual audit data – Analysis by
agency
Examination of:• Healthy@Work agency reports • Grants program – applications,
reports• H@W project review and closure
report
![Page 33: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/33.jpg)
H@W project review and closure report
• Project performance: – project vision, measured outcomes, outputs, budget
• Lessons learnt: – what worked well, what could be improved, challenges – recommendations to guide future TSS H&WB activities
• Highlights and innovations:– Organisation change and development– Agencies with a H&WB program (3 in 2010, 14 in
2012)– All Agencies report H&WB activities (14 in 2009, 21 in
2010)– Working in partnerships: DHHS, Workcover, UTAS,
Menzies Research Institute Tasmania (pH@W), Healthy Workers (NPAPH)
![Page 34: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/34.jpg)
H@W grants program
• Number of grant rounds
• Who applied for money, in which round?
• What type of project?
• Who was successful?
• How much money did they get?
• Progress/final reports provided
• Any sustainable outcomes?
![Page 35: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/35.jpg)
Purposive Data Collection
• What happened in the implementation of H@W is only part of the story…
• The context in which a program operates can be important data for evaluation
• Semi structured interviews with key informants is one method for collecting this data
![Page 36: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/36.jpg)
What we would like to knowexamples
1. How did you (your agency) gain management support?2. What sort of a coordination mechanism was used to
manage H@W?3. How much did employees engage with H@W? 4. How useful/relevant was the information on the H@W
website?5. How did your organisation review and update the
H&WB Program?6. How did you promote , market, communicate the
H&WB Program to employees?7. Has your agency continued to resource workplace
H&WB?8. Base on the H@W experience what factors do you
believe influence employee participation in workplace H&WB programs?
![Page 37: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/37.jpg)
Data Analysis
Analysis• Qualitative and quantitative methods and descriptive statistics
where appropriate
Interpretation• Interpretavist methods to analyse content of interviews and
documents• What understandings or conclusions can be gained?• What does it mean for the program/project?• How does this inform the evaluation purpose?• Reveal the complexities of implementing a large scale, multi
agency workplace based health promotion initiative • Identify enablers and barriers to successful implementation and
implications for transferability to non state sector workplaces• Inform program design and implementation of future WHP
strategies
![Page 38: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/38.jpg)
Who is this evaluation for ?
Target Audience Dissemination Format
Responsible Person/Author
pH@W partners Written report Evaluators/Process Evaluation Working Group (PEWG)
Key stakeholders Summary reportPresentation
Evaluators/PEWG
Stakeholders Presentation Evaluators/PEWG
Academic peers (UTAS, MRIT)
Peer-reviewed journal articles
Individual authors
Primary audience: pH@W partnersOther interested parties: SSMO, Tasmanian State Service Agencies, TSS employees, key informants
![Page 39: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/39.jpg)
Still to come
• Ethics application prepared for key informant interviews
• Interviews planned for June/July 2013
![Page 40: The why behind the what: A process evaluation of Healthy@Work](https://reader035.vdocuments.site/reader035/viewer/2022062520/56816556550346895dd7d4c6/html5/thumbnails/40.jpg)
Importance of Process and Implementation Evaluation
• Uses scientific rigor to understand the etiology of gaps between expected results and observed outcomes– Information can be gathered during the implementation of a
program or trial • Explores the relationship between process and
outcomes• Generates evidence to guide future implementation
and improve program outcomes JOEM, Volume 55, Number 5, May 2013