National Obesity ObservatoryNational Obesity Observatory

The Standard Evaluation FrameworkThe Standard Evaluation Framework

Kath Robertskatharine.roberts@empho.nhs.uk

What is evaluation:bring me the evidence!

One question…..

If evaluation was an animal…what would it be?

Scary?

Tricky to get hold of…

Prickly?

OTHERS…

Does it work?

Has it achieved its objectives?

What happened?

Has it worked?

Could it work?

What questionnaire should I use?

Do I need a double blind randomised control trial?

Was it effective?

Project

Standard Evaluation FrameworkStandard Evaluation Framework (SEF) for weight management (SEF) for weight management interventionsinterventions

www.noo.org.uk/SEFwww.noo.org.uk/SEF

Aim of the SEF:Aim of the SEF:

To support high quality, consistent evaluation of weight management interventions in order to

increase the evidence base

Objectives of the SEF:Objectives of the SEF:

• Increase the proportion of weight management interventions that are evaluated

• Increase the quality of evaluations• Develop a core dataset to increase the

consistency and comparability of evaluations• Provide wider evaluation support

Target audience for the SEF:Target audience for the SEF:• PCT commissioners (to support service

specifications)• PCT & DH obesity leads• Organisations running interventions• External evaluators

The NE region requires the SEF is incorporated as an essential component of all new weight management programmes?

Why should we use the SEF?Why should we use the SEF?

Num

ber o

f int

erve

ntion

s

Speculativeinterventions

Evidence-basedinterventions

Time

What interventions should it apply to?What interventions should it apply to?• Any intervention that explicitly sets out to manage or

reduce body weight as an intermediate or long-term health outcome (including the primary prevention of weight gain)

• Projects focusing on diet, physical activity, or both in combination

• Intended to be applicable to a range of approaches including interventions conducted with individuals on a one-to-one basis, or with people in groups; and in clinical or community settings.

The SEF is not intended for:The SEF is not intended for:

• Medical treatment e.g. bariatric surgery & prescribing

• Wider environmental interventions e.g. changes to the built environment (although the SEF could support the evaluation of such interventions alongside wider evaluation criteria)

Consultation Processes:Consultation Processes: • Developed in consultation with academic

experts, representatives from Public Health Observatories, Primary Care Trusts, Government Offices, Local Authorities and other relevant organisations and bodies

• Initial list of criteria consulted on using an online questionnaire - followed up by individual interviews

• Two consultation meetings to finalise criteria and guidance.

NB the SEF itself will also be evaluated.

What is in the SEF document?What is in the SEF document?

• An introduction to the concepts of evaluation• A list of data collection criteria divided into

‘essential’ and ‘desirable’• A guide to using the criteria and collecting the

data

Introduction to Evaluation:Introduction to Evaluation:

• What is evaluation and why do it?• What are the different types of evaluation?• What are the differences between evaluation,

audit and monitoring?• What sorts of information need to be

collected to carry out evaluations?• What sorts of issues need to be considered

as part of an evaluation?

SEF Criteria:SEF Criteria: 1. Programme details e.g. name of

intervention, aims/objectives2. Demographics of individual participants e.g.

age, sex, ethnicity3. Baseline data e.g. height/weight, dietary

and physical activity behaviours4. Follow-up data (impact and process

evaluation)5. Analysis and interpretation

Supporting SEF guidance:Supporting SEF guidance:• Details about why criteria have been

categorised as ‘essential’ or ‘desirable’• What the data can and cannot tell you• Standardised methods for collecting some of

the data• Suggested standardised definitions for data• Suggested data collection tools• Signposting to further guidance and useful

information

Recommended Physical Activity Recommended Physical Activity Measurement Tools:Measurement Tools:• 4 instruments met the following criteria

– Relatively easy and practical to administer

– Mean difference between energy estimated by self-report and doubly labelled water <40%

– Instrument should have been used in a trial of exercise promotion and detected a significant difference between intervention and control participants

– Population in the validity study generalisable to wider population

Appendix to SEF by Dr Melvyn Hillsdon

Recommended Physical Activity Recommended Physical Activity Measurement Tools:Measurement Tools:

• International Physical Activity Questionnaire (IPAQ) long version– 7day recall– Valid in 15-69 year olds only

• Stanford 7-day recall– Self report for sleep, moderate, hard and

very hard activities– Valid in adults only

Recommended Physical Activity Recommended Physical Activity Measurement Tools:Measurement Tools:

• 7-day physical activity diary– Self-completion diary ‘ticking’ 15min blocks

of activity– Valid in adults only

• New Zealand Physical Activity Questionnaire– 7day Self-completion recall, using prompt

cards– Valid in middle aged adults only

Limitations:Limitations:

• No existing physical activity measure perfectly meets all criteria

• Lack of suitable measures for children• No gold standard for measuring sedentary

behaviours• Most measures have quite complex scoring

systems

Future SEF products:Future SEF products:• Stand alone evaluation guide• Stand alone criteria tables• Example case studies• Excel template for audit data• Excel template for individual study data• Online audit (currently taking place in the NE),

national audit under development.• Dietary tool guide• Commissioners guide to weight management

interventions• Others…

Questions?Questions?

Queries to:

info@noo.org.uk

PHINE SEF support group:

http://www.phine.org.uk/group.php?gid=99

Interactive WorkshopInteractive Workshop• Aims:

– Share how you plan to use the SEF in your locality.

– Discuss using the SEF for an intervention(s) running or due to run in your locality and identify:

Possible barriers/difficulties to using the SEF.

Possible solutions to these barriers and how NOO can best support these.

The advantages the SEF would bring to this intervention.

Workshop Feedback:Workshop Feedback:• Each group to report back:

1. How is the SEF going to be used in your locality?

2. What possible barriers are there to using the SEF?

3. How could these barriers be overcome?

4. What could NOO provide to support use of the SEF further?