SVMPharma Ltd, Landmark House

Station Road,Hook,Hampshire, UK,

RG27 9HA

CONTACTUSenquiry@svmpharma.com

+44(0)1256962220www.svmpharma.com

Afraid Of Non-Randomised

Data?

Afraid of non-

randomised data?Does the thought of dealing with non-

randomised data fill you with dread?

Do you feel non-randomised data

collection brings more than its fair

share of problems and challenges?

Wouldn’t you rather invest your time

and resource in the gold standard - a

randomised controlled trial?

If you answered yes to these questions, let’s

look at something that could change your

mind:

Non-randomised data collection has been

revitalised with a landmark publication from

the National Institute of Clinical Excellence

Decision Support Unit (NICE DSU).1

The significance of this report is that it allays

a number of fears consistently raised by

stakeholders2, the report: a) demonstrates

that non-randomised data can be used to

make robust estimates of treatment effect b)

provides the guidance that was lacking and

sought after c) shows that NICE are actively

encouraging and accepting non-randomised

data within HTA submissions.

Non-randomised data falls within real-world

evidence (RWE) – which can be defined as

any clinical data collected outside of a

conventional randomised controlled trial

(RCT).

In RCTs, random allocation of patientsbetween intervention and comparatorensures that factors which could influencethe outcome of interest are evenly balancedbetween treatment groups. The change inoutcome attributable to the treatment (thetreatment effect) is simply the difference inoutcome between the treated and thecontrol groups.

In non-randomised studies, estimates oftreatment effect face an increased risk ofbias. There are a number of methods whichseek to minimise the risk of selection biaswhen estimating treatment effect. Most ofthese attempt to make treatment and controlgroups comparable (via matching or inverseprobability weighting), or control howprognostic factors affect the outcome (e.g.regression adjustment, via propensity scoresor with instrumental variables). Othermethods mimic randomisation with naturalexperiments (e.g. difference-in-difference andregression discontinuity). Finally, there aremore advance methods which modelselection and the effect of treatment on theoutcome jointly (e.g. with structural modelsor the correction approach).1

As you can imagine, with such a vast array of

methodologies, deciding how best to

estimate treatment effect isn’t easy, and it is

especially difficult to judge the quality of the

trials and the data in order to make a

decision. The NICE DSU document details

each of the methodologies above, and

provides an algorithm to inform the best

choice.

2 ©2016 SVMPharma Ltd. All rights reserved

www.svmpharma.com

©2016 SVMPharma Ltd. All rights reserved

NICE recognise the need for harmonisationand standardisation of non-randomisedstudies. The Consolidated Standards ofReporting Trials (CONSORT) statement wasfirst published in 1996, and provides a widelyadopted checklist and guidelines for theconduct of RCTs which have been updatedregularly over the last 20 years.3

But where is the equivalent for non-randomised data?

There have been a number of attempts tostandardise and advise the reporting of non-randomised data. Examples of this include theGood Research for Comparative Effectiveness(GRACE) checklist4 which provides astreamlined and easy-to-use set of questionsthat should be asked of the data. A similarapproach is taken by the Strengthening TheReporting of Observational Studies inEpidemiology (STROBE) statement.5 An in-depth report has also been produced by theInternational Society For Pharmacoeconomicsand Outcomes Research (ISPOR) Task Force.6

NICE DSU have compared and contrastedthese existing publications, identifying thecommon themes (how selection andconfounding biases were minimised,specification of the outcomes equation,heterogeneity in treatment effect,comparability of treated and control groupsand the assessment of uncertainty). NICE DSUhave proposed their own version of thechecklist titled Quality of EffectivenessEstimates from Non-randomised Studies(QuEENS). This is a set of 25 questions whichcover the essential aspects of conducting anon-randomised study- and offers astructured way of assessing non-randomiseddata. 1

1. Data suitability

•You need to consider whether the data available is appropriate to answer the decision problem and to inform the estimation of the parameter of interest for the cost-effectiveness model.

•You need to think about the patient population, the interventions, the comparators, the setting, and any data available on potential confounders.

2. Choice of methodology and implementation

•You need to explain why you have decided to use your selected method and describe how you have gone about using it.

•The NICE DSU algorithm gives you a tool to make the most appropriate choice from the multitude of options available, and further questions are specific to the modelling approach selected.

3. Interpretation in light of existing knowledge

•You should interpret your data against a background of existing data, knowledge, and understanding, being aware of counter-intuitive associations between covariates and the outcome of interest (this may suggest unobserved confounding).

•You would expect differences in the results but consistency between them (or inconsistencies that are easily explained) will give credibility to the results.

4. Analysis using different methods

•You should conduct pre-planned sensitivity analysis with methods that use contrasting approaches (e.g. matching vs multivariate regression) and different implementations of the same method (e.g. for matching, using a different set of variables/ inclusion of interactions/ polynomial terms).

5. Assumptions and transparency

•You should state upfront the assumed mechanism of causality and explain the assumptions of the methods and implications of the results.

•You should ensure transparent and comprehensive reporting of the all analyses conducted in such a way that clinical experts can understand the analysis and validate the plausibility of the results.

The 5 key areas that you need to cover

3www.svmpharma.com

©2016 SVMPharma Ltd. All rights reserved

1. Faria et al. NICE DSU Technical Support Document 17: The use of observational data to inform estimates of treatment effectiveness for Technology Appraisal: Methods for comparative individual patient data. NICE Decision Support Unit 2015. Available from www.nicedsu.org.uk/TSD17%20-%20DSU%20Observational%20data%20FINAL.pdf Accessed Jan 2016.

2. GetREAL. Review of Policies and Perspectives on Real-World Data (RWD). 2015. Available from www.imi-getreal.eu/Publications Accessed Jan 2016

3. Schulz et al. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010. Available from www.consort-statement.org/Media/Default/Downloads/CONSORT%202010%20Statement/CONSORT%202010%20Statement%20-%20BMJ.pdf Accessed Jan 2016

4. Dreyer N. A validated checklist for evaluating the quality of observational cohort studies for decision-making support. graceprinciples.org 2014. Available from www.graceprinciples.org/doc/GRACE-Checklist-031114-v5.pdf Accessed Jan 2016

5. von Elm et al. The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 2007. Available from www.ncbi.nlm.nih.gov/pmc/articles/PMC2034723/Accessed Jan 2016

6. Berger et al. A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NCP good practice task force report. Value in Health 2014. Available from www.ispor.org/observational-health-study-use-guideline.pdf Accessed Jan 2016

There has long been need for standardisation of the methodology and assessment of non-

randomised data as there is with the CONSORT Guidelines for RCTs.

NICE DSU have provided comprehensive guidance that incorporates existing efforts whilst

putting forward their own recommendations. By using the checklist and recommendations in

this document you are working to a defined methodology for non-randomised data.

Do you need to be afraid of using non-randomised data within your HTA?

4www.svmpharma.com

SVMPharma is an innovative strategic consultancy, specialising in Real World Evidence (RWE) forthe pharmaceutical industry. SVMPharma generates RWE within UK and Europe through bespokeonline Real World Treatment Evaluators, leading to successful health technology appraisal (HTA)submissions. Clinical trial programmes do not reflect real-world clinical practice and outcomes,RWE supplements and enhances clinical datasets. SVMPharma’s specialist teams focus ondelivering the outcomes that matter to your brand. SVMPharma also provides Global RWE, PatientReal World Outcomes, and Big Data Analytics.

To find out more call +44 (0) 1256 962 220

CONTACT SVMPHARMA

Vaneet NayarDirector SVMPharma

enquiry@svmpharma.com

Tel: +44 (0) 1256 962 220

SVMPharma LtdLandmark HouseStation RoadHook, HampshireRG27 9HA

Visit svmpharma.com

Follow @svmpharma

Follow on Google+

Connect via LinkedIn

Get In Touch