why do we read research articles?

Why do we read research Why do we read research articles?articles?

To increase our knowledge in a particular area To increase our knowledge in a particular area of interestof interest

To search for evidence for our practiceTo search for evidence for our practiceTo review literature in preparation for a To review literature in preparation for a

researchresearch

Should we believe everything Should we believe everything we read?we read?

CRITICAL CRITICAL APPRAISALAPPRAISAL

Kristofferson G. Mendoza, Kristofferson G. Mendoza, PTRPPTRP

Department of Physical TherapyDepartment of Physical TherapyCollege of Allied Medical ProfessionsCollege of Allied Medical ProfessionsUniversity of the Philippines ManilaUniversity of the Philippines Manila

27 May 200927 May 2009

OPST 199 OPST 199 Methods of Methods of ResearchResearch

Learning Objectives

At the end of the session, the learner should be able to:

Identify the level of evidence presented in a research study on the basis of the research design used

Discuss theoretical and methodological aspects of a research study in relation to internal and external validity

Rate the usefulness of evidence presented in a research study using key criteria

Critical Appraisal

the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.

LEVELS OF LEVELS OF EVIDENCEEVIDENCE

OPST 199 OPST 199 Methods of ResearchMethods of Research

Critical AppraisalCritical Appraisal

Levels of Evidence

Various study designs have different levels of rigour

Classic Levels of Evidence

By Sackett et al. For Studies on intervention, prevention,

etiology and harm RCT the most rigorous study design Modification of the classic level of

evidence later on included systematic reviews bec of the proliferation of RCTs

Level 1Level 1

Level 2Level 2

Level 3Level 3

Level 4Level 4

Level 5Level 5

LOW

HIGH

Levels of Evidence

For studies on Prognosis optimal individual study – individual

inception cohort study with greater than 80% ff up

Where the clinical decision rule has been validated in a single population

Clinical decision rule are algorithms or scoring systems that lead to prognosis estimation or diagnostic category

Levels of Evidence

For studies on Diagnosis optimal individual study – cohort study with

good reference standards or clinical decision rule tested within one clinical center

Levels of Evidence

For studies on Differential diagnosis / Symptom prevalence Cohort study with good follow up

Levels of Evidence

Despite differences in the optimal study design, certain consistencies are evident across different types of questions

A systematic review of high-quality studies always

provides the highest level of rigor An individual study using the optimal design for

that type of clinical question is considered level 1 Prospective data collection indicates higher study

quality than retrospective data collection Expert opinion, bench research, conceptual

framework/theories/first principles are always considered the lowest (level 5) evidence

Other Levels of evidence Classification System

Greenhalgh (1997)

Primary Research

Experimental

Observational

For qualitative studies? No consensus about the relative rigour of different methods

Levels of evidence

Systematic reviews &

Meta-analyses

Randomised controlled trials

Cohort studies

Case control studies

Case series and case reports

For quantitative studies

RESEARCH RESEARCH DESIGN ISSUESDESIGN ISSUES



Research Design Issues in Quantitative Research Sampling of Subjects

Objective: A sample that represents the target population

Optimal Design: A random sample selected from the population

Threats to Validity: Differential sampling between groups will affect internal validity; convenience sampling may affect external validity

Research Design Issues in Quantitative Research Sample Size

Objective: The optimal number of subjects is large enough to detect important treatment effects but small enough to conduct the study in a timely, efficient manner

Optimal Design: Correct number of subjects is determined by an accurate sample size calculation

Threats to Validity: Small sample size

Research Design Issues in Quantitative Research Allocation of treatment

Objective: Unbiased allocation

Optimal Design: Random allocation

Threats to Validity: Nonrandom allocation (observational)

Research Design Issues in Quantitative Research Blinding (to treatment allocation and/or outcomes

Objective: To minimize sources of bias introduced by study personnel or participants

Optimal Design: Ideally everyone involved in the study would be blinded to the full extent possible

Threats to Validity: Blinding in rehab is difficult; may result in bias from differential diagnosis, outcome assessments, attention and follow-up procedures

Research Design Issues in Quantitative Research Outcome ascertainment

Objective: To accurately determine pretreatment status; To reflect all important changes in outcome post-treatment

Optimal Design: Outcomes measured at all likely relevant time points; Outcomes measured using reliable, valid, and responsive measures

Threats to Validity: important effects will be missed if relevant time points are not assessed or the outcome measures are not sensitive; outcome measures that reflect only certain domains may be biased towards specific treatments; poor reliability/validity may invalidate the conclusion

Research Design Issues in Quantitative Research Follow-up

Objective: To accurately portray the treatment effects obtained by all the participants

Optimal Design: 100% follow-up

Threats to Validity: differential loss to follow-up can introduce bias in estimate of effects

Research Design Issues in Quantitative Research Statistical analysis

Objective: To provide accurate estimates of the size and significance of the observed effects

Optimal Design: Accurate and appropriate analysis of all data

Threats to Validity: Inappropriate analysis may lead to faulty conclusion

CRITICAL CRITICAL REVIEW FORMREVIEW FORM



Law, M., Stewart, D., Pollock, N., Letts, L., Bosch, J., & Westmorland, M., 1998. McMaster University

www.fhs.mcmaster.ca/canchild

Comments

STUDY PURPOSE:Was the purpose stated clearly?YesNo

Outline the purpose of the study. How does the study apply to OT/PT/SP and/or your research question?

LITERATURE:Was relevant background literature reviewed?YesNo

Describe the justification of the need for the study.

DESIGN:RCTCohortSingle case designBefore and afterCase-controlCross-sectionalCase study

Describe the study design. Was the design appropriate for the study question (e.g., for knowledge level about this issue, outcomes, ethical issues, etc.)?

Specify any biases that may have been operating and the direction of their influence on the results.

Appropriateness of Study Design Knowledge of topic/issue

If little is known, more exploratory (case study, cross-sectional design)

as level of knowledge increases, study design must be more rigorous (RCT most rigorous)

Outcomes If outcomes are easily quantifiable and well-

developed standardized assessment tools are available, design should be rigorous

If outcomes are not fully understood yet (e.g., quality of life), use design that explore different factors (case control)

Appropriateness of Study Design Ethical Issues:

If there is no ethical issue re: withholding treatment, design should have control groups

Study purpose/questions Effectiveness of treatment: RCTs, before-

after design, single-case studies Learn more about an issue, pilot study to

determine treatment: case control an cross sectional

Biases

Sample/Selection Bias Volunteer bias – usually favors treatment

group Seasonal bias – could work either way Attention bias – favors treatment group

(extra attention)

Biases

Measurement/Detection Bias Number of outcome measures used

Only one outcome measure – could favor either groups

Too many outcome measures for sample size – favors control group

Lack of masked or independent evaluation – treatment group is usually favored

Recall or memory bias – favors treatment group

Biases

Intervention/Performance Bias Contamination – favors control group Co-intervention – can influence either

groups Timing of intervention Site of treatment Different therapists

Comments

SAMPLE:N=

Was the sample described in detail?YesNo

Was the sample size justified?YesNo

Sampling (who; characteristics; how many; how was sampling done?) If more than one group, was there similarity between groups?

Describe ethics procedure. Was informed consent obtained?

OUTCOMES:Were the outcome measures reliable?YesNoNot addressed

Were the outcome measures valid?YesNoNot addressed

Specify the frequency of outcome measurement (i.e. pre, post, follow-up).

Outcome areas (e.g., self care, productivity, leisure)

List measures used

Comments

INTERVENTION:Intervention was described in detail?YesNoNot addressed

Contamination was avoided?YesNoNot addressedN/A

Cointervention was avoided?YesNoNot addressedN/A

Provide a short description of the intervention (focus, who delivered it, how often, setting). Could the intervention be replicated in OT/PT/SP practice?

Comments

RESULTS:Results were reported in terms of statistical significance?YesNoN/ANot addressed

Were the analysis method(s) appropriate?YesNoNot addressed

Were the analysis method(s) appropriate?YesNoNot addressed

What were the results? Were they statistically significant (i.e., p<0.05)? If not statistically significant, was study big enough to show an important difference if it should occur? If there were multiple outcomes, was that taken into account for the statistical analysis?

Drop-outs were reported?YesNo

Did any participant drop out from the study? Why? (Were reasons give and were drop outs handled appropriately?)

Comments

CONCLUSIONS AND CLINICAL IMPLICATIONS:Conclusions were appropriate given study methods and results?YesNo

What did the study conclude? What are the implications of these results for OT/PT/SP practice? What were the main limitations or biases in the study?

THANK YOUTHANK YOU

