critical appraisal of (systematic review) meta-analysis 羅政勤 彰化秀傳紀念醫院
TRANSCRIPT
Critical appraisal of Critical appraisal of (Systematic review)(Systematic review)
Meta-analysisMeta-analysis
Critical appraisal of Critical appraisal of (Systematic review)(Systematic review)
Meta-analysisMeta-analysis
羅政勤彰化秀傳紀念醫院
ObjectivesObjectives
1 To understand the different terminology of Meta-analysis systematic review
2 To understand the key criteria for critical appraisal
3 To select an appropriate checklist or other instrument to use for critical appraisal Validity Impact Practicability (CASP)
TerminologyTerminology
Review ≧2 publication synthesise results + conclusionsOverview(systematic literature review) a review strives to comprehensively identify and track down all literature on a given topicMeta-analysis Specific statistical strategy assembling results of several studies into a single estimate
IntroductionIntroduction
Systematic reviews form a potential method for overcoming the barriers faced by clinicians when trying to access and interpret evidence to inform their practice
Systematic reviewsSystematic reviews
Concise summaries of best available evidence that addresses defined questionsscientific tool used to appraise summarise and communicate results and implications of otherwise unmanageable quantities of research
Systematic reviewsSystematic reviews
Defining a questionA good question will have four componentsndashType of person involvedndashType of exposurendashType of controlndashOutcomes
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
ObjectivesObjectives
1 To understand the different terminology of Meta-analysis systematic review
2 To understand the key criteria for critical appraisal
3 To select an appropriate checklist or other instrument to use for critical appraisal Validity Impact Practicability (CASP)
TerminologyTerminology
Review ≧2 publication synthesise results + conclusionsOverview(systematic literature review) a review strives to comprehensively identify and track down all literature on a given topicMeta-analysis Specific statistical strategy assembling results of several studies into a single estimate
IntroductionIntroduction
Systematic reviews form a potential method for overcoming the barriers faced by clinicians when trying to access and interpret evidence to inform their practice
Systematic reviewsSystematic reviews
Concise summaries of best available evidence that addresses defined questionsscientific tool used to appraise summarise and communicate results and implications of otherwise unmanageable quantities of research
Systematic reviewsSystematic reviews
Defining a questionA good question will have four componentsndashType of person involvedndashType of exposurendashType of controlndashOutcomes
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
TerminologyTerminology
Review ≧2 publication synthesise results + conclusionsOverview(systematic literature review) a review strives to comprehensively identify and track down all literature on a given topicMeta-analysis Specific statistical strategy assembling results of several studies into a single estimate
IntroductionIntroduction
Systematic reviews form a potential method for overcoming the barriers faced by clinicians when trying to access and interpret evidence to inform their practice
Systematic reviewsSystematic reviews
Concise summaries of best available evidence that addresses defined questionsscientific tool used to appraise summarise and communicate results and implications of otherwise unmanageable quantities of research
Systematic reviewsSystematic reviews
Defining a questionA good question will have four componentsndashType of person involvedndashType of exposurendashType of controlndashOutcomes
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
IntroductionIntroduction
Systematic reviews form a potential method for overcoming the barriers faced by clinicians when trying to access and interpret evidence to inform their practice
Systematic reviewsSystematic reviews
Concise summaries of best available evidence that addresses defined questionsscientific tool used to appraise summarise and communicate results and implications of otherwise unmanageable quantities of research
Systematic reviewsSystematic reviews
Defining a questionA good question will have four componentsndashType of person involvedndashType of exposurendashType of controlndashOutcomes
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Systematic reviewsSystematic reviews
Concise summaries of best available evidence that addresses defined questionsscientific tool used to appraise summarise and communicate results and implications of otherwise unmanageable quantities of research
Systematic reviewsSystematic reviews
Defining a questionA good question will have four componentsndashType of person involvedndashType of exposurendashType of controlndashOutcomes
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Systematic reviewsSystematic reviews
Defining a questionA good question will have four componentsndashType of person involvedndashType of exposurendashType of controlndashOutcomes
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
SR and Meta-analysisSR and Meta-analysis
Systematic reviews may or may not include a statistical synthesis called meta-analysiswhether the studies are similar enough so that combining their results is meaningful
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Meta-analysisMeta-analysis
Statistical method for combining the results of trialsMost appropriate for randomized trialsMay also be appropriate for observational studies
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Results of a meta-Results of a meta-analysisanalysis
Forest plots of a meta-analysis of four randomized trials comparing no adjuvant chemotherapy with adjuvant chemotherapy in early-stage ovarian cancer for overall survival (A) and recurrence free survival (B) JNCI Cancer Spectrum 95(2)105-112
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Advantages of meta-Advantages of meta-analysisanalysis
Allows pooling of several studies = increase sample sizeGathers literature in one placeProvides a quantitative summary (possibly less bias than a narrative)Generate hypothesesProvide information for future trials
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Disadvantages of Disadvantages of meta-analysismeta-analysis
Even randomized studies often differ significantly in their design outcome exposure measuresPublication biasStudies differ in qualityTime trendsHealth studies tend to be (comparatively) few
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Interpreting the results Interpreting the results of a meta-analysisof a meta-analysis
Was process valid (question search strategy reproducible)Are studies comparableAre results similarWhat is the estimate and precision of the estimate
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
ConclusionConclusion
Systematic reviews top of hierarchy of evidenceCaution before accepting findings of any systematic review without first appraising it
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
CautiousCautious
Attention paid to patient selection group inter-vention or search strategy SR combined studies in meta-analysis pooled in different intervention or participants included
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
3 reasons validity 3 reasons validity findingfinding
1) Chance2) Bias3) Confounding
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
ChanceChance
Random variation Chance statistical analysis (hypothesis testing and estimation)Avoid random variation adequate sample size
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
BiasBias
Systematic (non-random) error in estimation of population characteristic eg effect of treatment compared to control in a populationSystematic means hellip
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Classification of sources Classification of sources of bias in analytical of bias in analytical
studiesstudiesAllocationPerformancePlacebo-effectAttritionDetectionAnalyticalReporting
SelectionMeasurement
Analysis
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
1 Allocation bias1 Allocation bias
Any treatment allocation method that causes a systematic difference in participant characteristics at the start of a trial (baseline)ndash independent prognostic
characteristics (confounders)ndash failure to plan eg confounding by
indicationndash failure to execute
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
2 Performance bias2 Performance bias
Systematic differences in the care of the two groups other than the intervention being investigatedndash nursing amp supportive care
ndash monitoring for adverse effects
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
3 Placebo-effect 3 Placebo-effect biasbias
Placebo-effect - a beneficial effect gained because the participant believes he is receiving effective therapy (includes satisfying pat-doc relationship as well as medicinal intervention)
In trials with a ldquono-treatmentrdquo arm confounding due to a differential placebo-effect may occur if the subjects are aware they are not receiving active therapy
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Reasons for bias - Reasons for bias - ConfoundingConfounding
When a non-causal association due to a common cause of both T and H prevents us from quantifying any causal association
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescausesRandom variation (chance) impreciseSystematic variation (bias) inaccurateConfounder factor prognostically linked to outcome and unevenly distributed btw study groupsKnown confounders stratify results-Unknown confounders randomisation
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Confounding Confounding ndashndash measured measured amp unmeasured common amp unmeasured common
causescauses
cancerdrug
SmokingSupportive carePlacebo-effect
Non-causal assoc
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
4 Attrition bias4 Attrition biasAll clinical trials have a period of follow-up attrition occurs when subjects do not complete the follow-up process (loss to follow-up)
This is harmful because attrition causes loss of information and hence less precise estimates of the treatment effect if too many subjects cannot be analyzed
Systematic differences in the loss of participants to follow up between groups may cause bias if the analysis is improper eg analyzing only participants who had complete follow-up or who were fully compliant (per protocol analysis)
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
5 Detection bias5 Detection bias
Systematic differences in outcome assessment btw groups ndashmeasurement methodndashfollow-up frequency for outcomes
MY DOCUMENTSlnk
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
6 Analytical bias6 Analytical bias
Bias arising because of the method of analysisndashchoice of subjects to analyze
the analysis dataset
ndashchoice of statistical estimators
biased amp unbiased estimators
ndashchoice of multivariate models
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
7 Reporting bias7 Reporting bias
Selective reporting ofndashclinical outcomes eg surrogate subgroups
ndashtime-points eg earlyUse of composite endpointsndashcomponent events not equally significant
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
What is ApprasialWhat is Apprasial
A technique to increase effectiveness of reading by exclude research studies too poorly designed to inform practice
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Why appraisalWhy appraisal
To free time of concentrate on a more systematic evaluation of studies cross quality threshold and extract salient points
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
How to AppraiseHow to Appraise
Appraising a Secondary studies(Review)
1Validity2Impact(Results)3Practicability(Application)4Instruments tools such as CASP
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Critical Appraisal Critical Appraisal Skills Programme Skills Programme
(CASP)(CASP)httpwwwphrunhsukpagesPHDCASPhtm
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Appraisal tools for Appraisal tools for Systematic reviewSystematic review
10 questions to help you make sense of reviews Is the study valid What are the results Will the results help locally10 questions adapted from Oxman AD Cook DJ Guyatt GH Usersrsquo guides to medical literature VI How to use an overview JAMA 1994 272 (17) 1367-1371
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Screening questionScreening question
First 2 questions Screening questions can be answered quickly Worth proceeding If answer to both is ldquoyesrdquo
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
Screening questionScreening question1 Did the review ask a clearly-focused question 1048713 Yes 1048713 Canrsquot tell 1048713 No
Focused ndash the population studied ndash the intervention given or exposure ndash the outcomes considered
2 Did the review include the right type of study 1048713 Yes 1048713 Canrsquot tell 1048713 No
included studies ndash address the reviewrsquos question
ndash have an appropriate study designIs it worth continuing
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
3 Did the reviewers try to identify all relevant studies 1048713 Yes 1048713 Canrsquot tell 1048713 No
Consider ndash which bibliographic databases were usedndash if there was follow-up from reference listsndash if there was personal contact with expertsndashsearched for unpublished studiesndashsearched for non-English-language studies
4 Did the reviewers assess the quality of the 1048713 Yes 1048713 Canrsquot tell 1048713 No
indash if a clear pre-determined strategy was used todetermine which studies were included Look forndash a scoring system ndash more than one assessor
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
5 If the results of the studies have beencombined was it reasonable to do so Consider ndash the
results of each study are clearly displayedndash the results were similar from study to study(look for tests of heterogeneity) ndash the reasons for any
variations in results are discussed6 How are the results presented and what is the main result Consider ndash how the results are expressed (eg odds ratiorelative risk etc) ndash how large this size of result is and howmeaningful it is ndash how you would sum up the bottom-line result ofthe review in one sentence
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
7 How precise are these resultsConsiderndash if a confidence interval were reported Wouldyour decision about whether or not to use thisintervention be the same at the upperconfidence limit as at the lower confidencelimitndash if a p-value is reported where confidenceintervals are unavailable
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
8 Can the results be applied to the local 1048713 Yes 1048713 Canrsquot tell 1048713 NopopulationConsider whetherndash the population sample covered by the reviewcould be different from your population in waysthat would produce different resultsndash your local setting differs much from that of thereviewndash you can provide the same intervention in yoursetting9 Were all important outcomes considered 1048713 Yes 1048713 Canrsquot tell 1048713 NoConsider outcomes from the point of view of thendash individualndash policy makers and professionalsndash familycarersndash wider communityreported can it be filled in from elsewhere
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not
10 Should policy or practice change as a result of 1048713 Yes 1048713 Canrsquot tell 1048713 Nothe evidence contained in this reviewConsiderndash whether any benefit reported outweighs anyharm andor cost If this information is not