Developing evidence based practice: what does it mean and can it be done?

Miranda Wolpert

Director CAMHS Evidence Based Practice UnitChair CAMHS Outcome Research Consortium

Evidence based practiceQuestions of the individual clinician

• What is the best treatment for this particular child

• What are the pros and cons of different treatments

• What does the research show and what other factors do I need to take into account

• Are there particular reasons for adopting a different approach in a particular case?

Evidence based practiceQuestions of the service developer

• What does the evidence show should be provided by services and in what proportions

• What skill mix is needed to provide child mental health services

• What should be the ratio of investment in different options eg prevention/promotion programmes as opposed to direct interventions

Evidence based practicequestions of the academic

• What does this research really show• Are there other interpretations• How can research be devised to answer the

remaining questions

Evidence based practicequestions of the child and family

• What does this research really show• Are there other interpretations• How can research be devised to answer the

remaining questions

Answering these questions

- Few straight or clear answers- Lots of complexity - Lots of gaps

- Need a realistic way forward…

Evidence

Values Audit and Evaluation

A realistic evidence based practice

When does information become evidence?

Hierarchy of Evidence

• Ia Evidence from meta-analysis of randomised controlled trials• Ib Evidence from at least one randomised controlled trial• IIa Evidence from at least one controlled study without

randomisation• IIb Evidence from at least one other type of quasi-experimental

study• III Evidence from descriptive studies such as comparative studies,

correlation studies and case-control studies• IV Evidence from expert committee reports or opinions, or from

clinical experience of a respected authority, or both.

Alternative Hierarchies?

A non-evidence based approach?

• Reliance on assumptions

• More influenced by anecdote than statistics

• Not testing theories

• Unwillingness to change in light of new evidence

• Most persuasive promoter wins out

Limitations of the evidence

• Paucity of research • Skew in researched areas• Skew in researched populations• Generalisability to range of groups and settings

questionable• Design flaws in studies• Lack of consensus on appropriate outcomes and

perspectives• Lack of model for economic costings• Lack of focus on possible harm• Publication bias

Publication bias (from David Cottrell)

Drug Published Conclusions

Fluoxetine 2 Trials Favourable risk benefit profile

Whittington, CJ, Kendall, T,Fonagy, P, Cottrell, D, Cotgrove, A & Boddington E.(2004) Selective serotoninreuptake inhibitors in childhooddepression: systematic reviewof published versus unpublished

data. Lancet. 363, 1341-1345.

Paroxetine 1 Trial Weak positive risk benefit profile

Sertraline 2 Trials Weak positive risk benefit profile

Citalopram

Venlafaxine

Publication bias (from David Cottrell)

Drug Published Conclusions Not Published

Conclusions

Fluoxetine 2 Trials Favourable risk benefit profile

Safety Data

No change

Paroxetine 1 Trial Weak positive risk benefit profile

2 Trials Risks now outweigh benefits

Sertraline 2 Trials Weak positive risk benefit profile

Additional Data

Risks now outweigh benefits

Citalopram 2 Trials Unfavourable risk benefit profile

Venlafaxine 1 Trial Unfavourable risk benefit profile

Drawing on the Evidence

Wolpert, Fuggle, Cottrell, Fonagy, Phillips, Target and Stein 2002

Based on systematic review: Peter Fonagy, David Cottrell, Mary Target, Zarrina Kurtz, Jeanette Phillips– DoH Mother & Child R&D Fund

Revised edition 2006-Updated in light NICE guidance and major

randomised control trials

Possible summary of what we know works currently

Evidence based interventions

• Cognitive behavioural therapies (CBT)• Behaviour therapy• Parent Training• Medications• Family Therapy• Interpersonal therapy (IPT)• Social skills training• Multi-systemic therapy (MST)• Treatment Foster Care• Individual psychodynamic therapies

BUT….complicating/mediating factors

Demographic factors ?“attachment disturbance”“therapeutic alliance”Non-specific therapeutic factorsFidelity to modelAbility to flexibly adapt model

Proposed checklist for evidence based interventions (adapted from Kazdin 2004)

1. What are the costs, risks and benefits of this intervention relative to no intervention?

2. What are the costs, risks and benefits of this intervention relative to other interventions?

3. What are the key components that appear to contribute to positive outcomes?

4. What parameters can be varied to improve outcomes (e.g. including addition of other interventions, non specific clinical skills etc)?

5. To what extent are effects of interventions generalizable across a) problem areas, b) settings, c) populations of children and d) other relevant domains

Which of these can we answer now?How do we get answers?

Evidence base for service structures

Lots of values much less clear evidence- Fort Bragg Studies- Pooled budgets impact-Suggestive work about impact of service user

involvement

Promising work on economic evaluation of early intervention in psychosis projects

Worcestershire EIS (2006 report Jo Smith)

Duration of untreated psychosis

National

12-18m

EIS (3y) 2003-6 n=78

5-6m

% admitted in FEP 80% 41%

% FEP using MHA 50% 27%

Readmission 50% 27.6%

% engaged @ 12m 50% 100% (79% well engaged)

Family involved satisfied

49%56%

91%71%

Employed 20% 55%

Suicide attempted completed

48%6.6% @ 5yrs

21%0%

Evidence base for skill mix (based on evidence based interventions)

• 3 units of people able to provide behavioural, cognitive and interpersonal therapies

• :1.5 units of people able to provide parent management training

• : 1 unit of people able to provide systemic/multimodal therapy:

• : 1 unit of a person/people able to provide physical treatments, prescription and monitoring.

BUT doesn’t taken account of-under-researched interventionsNon- specific therapeutic and assessment skillsPossible needs of particular populations

Evidence base for children and families

Choosing What’s Best for You

What We Know (And What We Don’t) About the best ways of Helping Children and Teenagers With:

Eg ADHD Information for: Children, Teenagers, Families

Choosing what's best for you

booklet aims to help children young people and their families make informed choices about treatment options

It gives information about what research up till now has shown to help.

It is not designed to give you any general information.

In this booklet we list the most evidence based treatments a the moment

Each treatment option is rated using the following scale:.

* * * = Very likely to help* * - = Quite likely to help* - - = Not that it will help

Choosing what's best for you

• Points to remember

• There are many treatments that we simply don’t know if they work or not yet because research has not been done or is inconclusive- they are not included here

• Even when a treatment has been shown in research to work well for most people, as we are all different it may help some people more than others

• You will have to weigh up the positives and negatives of any approach, including any possible side effects

• • Our knowledge is growing all the time so check if there

have been further developments since this was published

Choosing what's best for you

Points to remember

• There are many treatments that we simply don’t know if they work or not yet because research has not been done or is inconclusive- they are not included here

• Even when a treatment has been shown in research to work well for most people, as we are all different it may help some people more than others

• You will have to weigh up the positives and negatives of any approach, including any possible side effects

• • Our knowledge is growing all the time so check if there

have been further developments since this was published

Choosing what's best for youTypes of Treatment/What Might Help Will it help?

Medication This involves taking one or more tablets a day. There is more than one type of medication. You need to talk to your doctor about which one is best for you. Links to more info about ADHD/medications

* * *

Other things to think about

Any medication is likely to have side effects- you need to talk this over with your specialist.

Behaviour Therapy Behaviour therapy is advice and help on how to learn behaviour that will make life easier.

* * -

Other things to think about Can be used with medication and can mean that you don’t need to take as much medication Taking Omega 3 and Omega 6 Oils This involves taking food supplements rich in these oils * - - Other things to think about This is quite a new area of research

Reflection and evaluation

Routine outcome monitoring- “Mission Impossible” ?? (Einar Heiervang)

• Case evaluation: To provide information about individual children and their

families.

• Clinician evaluation: To provide information about outcomes for the range of children

and families seen by an individual clinician

• Service evaluation: To provide information about the outcomes of particular projects

or services

• Strategy evaluationTo provide information about the impact of a CAMHS strategy

Underpinning values

• All services should routinely audit and evaluate their work

• Data collected made available to clinicians, users and commissioners

• Results used to inform service development• Collaboration essential

Evaluating outcomes

Whose view?

ChildParentsClinician

Where get info from?Conversations Questionnaires Written communicationsInformation held in a data setPopulation statistics For Whom?

What should be be evaluated

• Change in difficulties• General adaptation• Feelings of burden and stress• Satisfaction• Population changes e.g. attendance rates;

exclusions; youth crime; reported self harm; reported substance misuse, rates entering care, attainment rates

CAMHS Outcome Research Consortium (CORC)

Members agreeing on a common approach• Creating reports for reflection on individual

children/practitioners• Collating and centrally analysing data • Promoting use of data to inform service providers,

commissioners, users and others• Facilitating sharing of ideas between members• Supporting dissemination and refinement of

National CAMHS dataset

CORC aims

• Develop and disseminate model of routine outcome evaluation that can be used across a range of services

• Ensure data used to inform service providers, commissioners and users and other relevant stakeholders

• Collate and centrally analyse data from all member sites

• Collaborate in using outcome information to inform and develop good practice

• www.corc.uk.net

CORC approachChild/parent perspective:Symptoms and burden:-Strengths and Difficulties Questionnaire (SDQ) for child (11-16)

and parents of children aged 3-16

Experience of service:- Commission for Health Improvement (CHI) for child (9+) and

parent

Practitioner perspectiveChildren’s Global Assessment Scale (CGAS) – measures overall

functioningHoNOSCA where appropriate

Consultation measureBeing piloted

CORC protocol

• Pre therapy measures for child and parent• First meeting measures for clinician• MDS• 6 month follow up (or case closure if before

this)• Option for repeated follow ups for longer term

contacts

Outcome measurement - SDQ

Interpreting the “evidence”

• Credibility Self Evaluation: right measures, no.s of returns, quality of

returnsResearch studies: right measures, right people, quality of

controls• ContextSelf Evaluation: specific factors to be taken into account eg

demographics, specialist focusResearch Studies: generalisability• ComparisonSelf Evaluation: with baselines, with community with

appropriate other servicesResearch Studies: with other findings

Towards and evidence based practice approach

Need to both acknowledge complexity and to promote clarity- is this possible?

• More research • Explicit recognition of values base• Reflective practice