Disability Managementand the Effects of Workplace

Interventions:A Meta-Analysis

IFDM GLADNET-IDMRN Symposium

John LuiNorm Hursh

David Rosenthal

meta-analysis 2

Our underlying interests…

• DM is acknowledged as a proven approach that effectively decreases the economic costs and human impact of disability

• Earliest DM practices targeted RTW outcomes as a cornerstone of DM

• BUT, no meta-analyses examining an aggregation of empirical studies has been conducted

meta-analysis 3

Our underlying interests…

• DM is now a global movement, with developing standards, and accepted practices

• Early RTW practices have expanded, DM has evolved, yet benefit systems across countries vary widely

• We wanted to look at the state-of-the-art outcome studies conducted internationally

meta-analysis 4

Our underlying interests…

• Our thinking was…lets look at the most likely DM practice that most countries adopt

• How strong is the research around what we may find to be most consistently defined practice

• What are the drivers of evidence-based practice…of developing best-practice approaches to our work

meta-analysis 5

Meta-analysis

• Gold standard of best practice in medicine

– In medicine, with its positivist scientific methods tradition, the “gold standard” for scientific evidence is still randomized clinical trials and the method of choice for determining the cumulative evidence of the effectiveness of a treatment is meta-analysis.

• Randomized clinical trials• Meta-analysis

meta-analysis 6

Study Authors

• David A. Rosenthal• University of Wisconsin- Madison• John Lui • University of Wisconsin- Stout• Norm Hursh• Boston University• Li Jialiang • National University -Singapore• James A. Curcio• Hewitt Associates

meta-analysis 7

• This study applied meta-analytic processes to investigate the aggregate effect size of the studies that met criteria for inclusion in the research. Initial aggregation processes are within studies (when necessary) and subsequently, across studies, both providing ES estimates.

meta-analysis 8

• Criteria for inclusion of studies consisted of six areas:

• 1) population of interest • 2) nature of intervention or strategy • 3) provider of intervention • 4) receiver of intervention • 5) outcomes and • 6) study design – quantitative

meta-analysis 9

• Populations of interest include a) workers who are off work due to MSK conditions; pain‑related conditions that were episodic or non-episodic or associated with a degenerative or nondegenerative condition; chronic pain; or b) the workers’ compensation claimant population.

meta-analysis 10

• The category of nature of intervention or strategy is specifically aimed at improving RTW outcomes, including disability management; case management; education to workplace staff, insurance case managers or workers; changes in general organizational factors, but specifically aimed at improving RTW outcomes and decreasing durations of disability and associated direct labor costs..

meta-analysis 11

• Providers of intervention come from the workplace, or from an insurance company (private or governmental) or third-party claims administrator (TPA) which could be provided from the workplace. They could also come from healthcare providers in very close collaboration with the workplace.

meta-analysis 12

• Receivers of intervention include workers, workplace staff, and insurance company/TPA case managers.

meta-analysis 13

• Outcomes include work disability durations, described as self-reported time to return to work, time on wage replacement benefits (Workers’ Comp or Short/Long-Term Disability), total duration of lost time (in calendar or business days), total labor cost of direct lost time, recurrences; and point-prevalence of status (e.g., back at work versus not back at work).

meta-analysis 14

• Other outcomes include the associated costs of healthcare, indirect costs associated with wage replacement and intervention (training, administrative, etc.); as well as the quality of life, including mental health, functional status, general physical health during and/or after work interruption, quality of work life, medication taking during and/or after work interruption.

meta-analysis 15

Search

• EBSCOhost was used as a primary searching reference system. Four databases were searched for relevant studies including CINAHL Plus with Full Text, MEDLINE, PsycARTICLES, and PsycINFO between January 1996 and October 2009. In addition, relevant research institutes, including Institute for Work and Health (IWH), and National Institute for Occupational Safety and Health (NIOSH) were also explored.

meta-analysis 16

• Terms used in searching included workers’ compensation, workplace intervention, return to work, return to work and intervention, disability management, short-term disability, long-term disability, .standardized metrics, benefit program performance, and benefit program evaluation.. In total, 2765 papers were identified by the search.

meta-analysis 17

• Study designs must be quantitative and include randomized controlled trial (RCT), nonrandomized trial, cross-sectional, pre-postdesign, time series, case control, and cohorts (retrospective and prospective).

meta-analysis 18

Our Criteria for exclusion of studies consisted of:

• 1) population of interest • Populations of interest include

mental health conditions as a primary condition, phantom limb pain, short duration of self-limiting pain, and pain associated with a malignant condition.

meta-analysis 19

Exclusion

• 2) nature of intervention or strategy– Natures of intervention or strategy

include policies, primary prevention, workplace ergonomic interventions, and clinical interventions provided outside the workplace.

meta-analysis 20

Exclusion

• 3) provider of intervention– Providers of intervention come from

healthcare providers with no or minimal integration with the workplace.

meta-analysis 21

Exclusion

• 4) outcomes – Outcomes include absenteeism

unrelated to MSK or other pain-related conditions.

meta-analysis 22

Exclusion

• 5) study design – not quantitative or quantitative consisting of non-comparative studies, such as case series and case studies.

meta-analysis 23

• Studies included (n=15): US – 4Norway - 1Netherlands – 3Sweden – 2Finland – 1North America – 1Canada – 3

meta-analysis 24

Results

• we obtained an aggregate effect size d+ = 0.372

What does this mean?

meta-analysis 25

• More specifically, computation of an aggregate effect size for each study was accomplished on a Microsoft Excel worksheet. For the 15 studies retrieved for the Disability Management interventions in business and industry that met the criteria for workplace interventions to address managing disability in the workplace and measurable outcome criteria such as cost savings or reduced lost days at work.

meta-analysis 26

• we obtained an aggregate effect size d+ = 0.372, indicating that at the conclusion of treatment, workers in the intervention groups scored, on average, nearly half of a standard deviation higher than workers not receiving such interventions.

meta-analysis 27

Meta-analysis Process Issues

• Comprehensive review• Inclusion criteria

– Finding studies – data bases– Publication bias– Significance bias

meta-analysis 28

The Tyranny of P Values

• Cohen (1994) argued that the results of statistical significance tests (i.e., the test statistic and associated p value) are not valid indicators of study findings, primarily because they are largely determined by the sample size of the study.

• Meta-analysis goes beyond this “illusion of conflicting findings” (Hunter & Schmidt, 1996, p. 326) by ignoring p values in favor of an index of the strength of association between variables, referred to as the observed effect size for each study.

meta-analysis 29

Sample size

• Effect sizes derived from small N studies tend to have large sampling variances, and correspondingly large standard errors of estimate (reflecting relatively low precision).

• Effect sizes derived from large N studies have relatively small sampling variances, correspondingly small standard errors, reflecting their relatively greater precision.

• For this reason, the inverse of the variance for each effect size serves as a weight in meta-analysis. This procedure gives greater weight to more precise effect estimates and lesser weight to less precise effect estimates in computing the aggregate d+. (See VCV matrix in Excel example)

meta-analysis 30

Effect sizes (Cohen)

• Standardized Mean Difference Effect Size– large effect: d = .80– medium effect: d = .50– small effect: d = .20

meta-analysis 31

Fixed Versus Random Effects

• Used Random effects versus fixed (two sources of error - model for generalizability).

• In a random-effects meta-analysis, the variance of study effect sizes has two components: – variance due to sampling error (estimated in the

effect size variances computed earlier) and– – random effects variance component reflecting

differences in effect sizes attributable to systematic differences between studies (Raudenbush, 1994).

meta-analysis 32

Q Statistic – Test of Homogeneity

• Q Statistic – Test of Homogeneity

• Examines whether the variability among study effect sizes is greater than would be expected due to chance (i.e., to sampling error) alone.

• The Q statistic represents a ratio of between-studies variance to within-study variance, analogous to the F ratio in ANOVA.

• The Q statistic has (k – 1) degrees of freedom (where k is the number of studies included in the meta-analysis) to determine whether the null hypothesis (i.e., variability among observed effect sizes is attributable to sampling error alone) can be rejected at a desired probability level (usually a = .05) (Shadish & Haddock, 1994).

meta-analysis 33

- Q Statistic – Test of Homogeneity

• For this study data, Q(k-1=14) = 16.82, p > .05, so the null hypothesis could not be rejected.

• That is, the variance among study effect sizes was not greater than expected based on sampling error, so that there is no warrant for suspecting that systematic differences among studies contributed to differences in their obtained effect sizes.

meta-analysis 34

Additional analyses

• Also will conduct moderator Analyses - (a) type of observation - behavioral performance, or observer judgment) and (b) hours/amount of intervwention provided.

• Moderator analyses using both continuous (hours of treatment) and categorical (type of observation) variables.

meta-analysis 35

Implications for ongoing international research

• Lets discuss:– What are outcome research of

stakeholders from different countries, different systems

– How valued is research by different stakeholders

– How can we develop a common language related to DM outcomes to promote research

meta-analysis 36

Implications for ongoing international research

• Defining DM – what does it mean to different countries when disability benefit systems and healthcare systems are so different

• Is DM passe? How does it Line up for employers who promote wellnes

• With these differences, how do we align collaborative international research to talk the same language…what are common metrics and metrics of WHAT

meta-analysis 37

Other questions?