challenges in evaluating social interventions: would an rct design have been the answer to all our...
TRANSCRIPT
Challenges in evaluating social interventions: would an RCT design have been the answer to all our problems?
Lyndal Bond, Kathryn Skivington, Gerry McCartney, Hilary Thomson
York September 2009
Overview
• Importance of evaluating effects on health of social interventions
• Welfare to work – health benefits – what is known• Pathways Advisory Service• Evaluation challenges
• Design• Comparison group• Data sharing• Ethics & recruitment
• Conclusions & where to from here?
Evaluating the health effects of social interventions programme
Programme’s aim:• To ensure policy decisions are based on the most
reliable research evidence available
By undertaking: • systematic reviews, bringing together existing
research on the effectiveness of social/policy interventions
• new studies evaluating the health impacts of social/policy interventions
What do we study?
The effects on health and health inequalities of:
Housing & community
regeneration
Transport policies
Tobacco control legislation
Work organisation & welfare to work policies
Welfare related policies & programmes
Social/policy interventions affecting:
• employment & income• welfare policies• means of improving
the health of the worst off & reducing health inequalities.
Welfare-related policy interventions: evidence for effectiveness
• Lack of evidence for effects on health
• House of Commons Health Committee (2009) inquiry into health inequalities concluded that the lack of evidence was:
“in large part due to inadequate evaluation
of the policies adopted”
Welfare to Work programmes
• Aim to reduce number of people claiming benefits by moving them into employment
• May be regarded as important ‘healthy public policy’ interventions • potential to impact on health and health
inequalities
• A positive relationship between employment & health (Waddell, 2006)
• ‘work is good for health’
But…
• Potential for health benefits for welfare recipients moving into employment is likely to be dependent on• type & suitability of the job • individual-level factors – age, health etc
• Evidence for health benefits of returning to work for groups such as those in receipt of illness-related benefits is limited
• There is a need to evaluate such interventions for their effect (positive or negative) on health
The Pathways Advisory Service
• PAS places employment & benefit advisors into GP surgeries to engage with IB recipients
• Provides advice about benefits & return to work including • guarantee of benefits in 12 months if return to
work not successful• GPs refer patients to the service• Uptake is voluntary• Patients can self-refer
DWP evaluation of pilot PAS
• Department for Work and Pensions (DWP) had commissioned an evaluation of the PAS pilot programme• process, & employment & benefit
outcomes • no health outcomes (Sainsbury, 2008)
• PAS rolled out before pilot evaluation completed & rolled out service different to that which was piloted
Our planned evaluation of PAS
• Robust evaluation of the effects on health of PAS• Presumed through a return to work?
• Experimental or quasi-experimental design• Self-reported health• Health services use
Challenges
• Evaluability of the intervention
• Who is eligible for and receives the intervention?
• Identifying and recruiting: issues of transfer of contact details to obtain informed consent to the study
• Defining a comparison group
Evaluability of the intervention
Cluster RCT - design of choice• appropriate & justified given uncertainty about harms & benefits
with respect employment, welfare, income & health
But• before an evaluation of health impacts could be planned PAS was
being rolled out across the UK• so not possible to make the case for this design
• use of well conducted non-randomised study designs has been advocated as an appropriate alternative • we proposed a quasi-experimental study; a non-randomised
controlled trial.
Who is eligible & who receives the intervention?
• Service is available to all IB recipients• GPs use their discretion to refer patients to PAS• Uptake is voluntary and patients can self-refer
• About 20% of IB recipients in participating practices are estimated to engage with the PAS intervention
• It is likely that those who are referred have:• relatively good health • are closer to returning to the labour market
Identifying & recruiting
1. IB recipients from DWP records • not possible due to implementation of a data transfer
ban following high profile loss of sensitive personal information by a number of Government departments
2. Target population of IB recipients through GP electronic records• indicating a GP has completed a medical reference for
an IB claim• Limited by GP practices who use particular software• Estimated about 70% of IB patients would be so
recorded
Defining a comparison group
• Need to be similar with respect to eligibility for intervention • in receipt of IB• age, health status, length of time on benefits, employment
history etc • Could identify IB recipients through GP records, registered with GP
practices not offering PAS. BUT • Referral and access to PAS was up to GPs’ discretion & patient
self-selection• proportion of IB recipients engaging with the service is small • those engaging with PAS likely to have relatively good health &
be closer to a return to the labour market than those not engaging with PAS.
• Making it difficult to identify an appropriate & available comparison group
IB Recipients in PAS GP Surgeries
Group 2:
80%
Group1:
20%
Exposed to PAS and engage withPASExposed to PAS and do notengage with PAS
IB Recipients in non PAS GP Surgeries
Group 3:
100%
Not exposed to PAS, and sodo not engage with PAS
Defining a comparison group
IB Recipients in PAS GP Surgeries
Group 2:
80%
Group1:
20%
Exposed to PAS and engage withPASExposed to PAS and do notengage with PAS
Defining a comparison group
Need to rely on capacity to collect appropriate data to adjust for baseline differences between Gp 1 & Gp3
Proposed recruitment
Local Research Ethics Committee required:• written opt-in consent for contact details to be given to researchers• GPs could send out only 1 reminder letter to potential participants.
Estimated <10% response
GP writes to potential participants identified through medical records
`
Potential participants given chance to stop contact details being given to
researchers
GPs pass contact details to the researchers
Researchers write to potential participants with more details about
the study
Participants who give informed written consent are recruited to study
Interpretation of the Data Protection Act
Data Protection Act – not clear • ongoing discussion regarding what the Act means for data sharing
for health research • it does not necessitate individual consent for all medical research • but its wording does not provide clear guidance on when consent
by individuals is necessary for medical researchMedical Research Council Guidance states:• there must be an acceptable balance of risk & benefit for
participants. • justification for disclosure of a limited amount of personal
information given in confidence without consent, • if patients should be given the opportunity to raise objections,
then limited disclosure is unlikely to give rise to any serious objections.
Summary of barriers
• No control over method of implementationcompromising study design
• Ban on DWP data sharing• limited identification• transfer of contact details of potential participants• the identification of a comparison group
• Data Protection Act – interpretation• limited capacity to recruit
RESULT:• unethical to undertake study due to bias and lack of power
Ways forward
• Closer & timely working between policymakers & researchers• Allowing input at early stage for evaluation
design• & develop policy-relevant evaluations
“All too often Governments rush in with insufficient
thought, do not collect adequate data at the beginning about the health of the population which will be affected by the policies, do not have clear objectives, make numerous changes to the policies and its objectives and do not maintain the policy long enough to know whether it has worked. As a result, in the words of one witness, ‘we have wasted huge opportunities to learn’. Simple changes to the design of policies and how they are introduced could make all the difference.” (2009)
House of Commons
Health Committee
Ways forward (2)
• Response to loss of data by Government departments by DWP has been to develop data sharing protocols:
Data sharing protocols• no data sharing protocols between the DWP
and research organisations unless a ‘commissioned’ evaluator for DWP
• DWP not researcher driven research/evaluation
• Way forward???
Ways forward (3)
Need clarity of legal and ethical interpretation of the Data Protection Act
• Is it guidance? Is it legislation? • Who decides the balance between individual & collective
good?• Are the evaluations of ‘non-medical interventions’
perceived to be less important & therefore not worth even ‘low risk’ research?
• Is that acceptable given:• Expensive• Unexpected consequences• Particularly unfair for disadvantaged groups
Back to control group
• This remains a fundamental issue
• Resolution? • Establish a cohort study to enable nested
evaluations of policies • Expensive, large numbers, but could be worth
it• Other suggestions are welcome
Conclusion
• Without resolution robust experimental or quasi-experimental evaluations of these types of interventions are not possible
“… it is nearly impossible to know what to do given the scarcity of good evidence and good evaluation of current policy. Policy cannot be evidence-based if there is no evidence and evidence cannot be obtained without proper evaluation. The most damning criticisms of Government policies we have heard in this inquiry have not been of the policies themselves, but rather of the Government’s approach to designing and introducing new policies which make meaningful evaluation impossible..”
House of Commons
Health Committee