Jo Hunter

Clinical Outreach Librarian

University of Oxford

Health Care Libraries

Introduction to

Critical Appraisal

Aims of this session

Define “critical appraisal”

Discuss why it is important

Discover how to incorporate it in your own research/practice

Is this evidence good enough?

What is “good” evidence?

• We need to use evidence from research studies to support our arguments or decisions

• How do we recognise a good quality research study?

How can you tell whether a research study is “good quality”?

Factors that tell you about the quality of a study include…

Extrinsic factors:• Journal• Author• Institution• Quoted by other

authors• Quoted in

newspapers

Intrinsic factors:• Study design• Number of

subjects• Confounders• Bias (author,

experimental)

Critical appraisal looks at the intrinsic factors

“Critical appraisal is the process of systematically examining research evidence to assess its validity, reliability and relevance before using it to inform a decision.”

(Adapted from Hill H & Spittlehouse C, EBM 2001; 3:2)

Validity, reliability, relevance

1) Validity: To what extent is the study a close representation of the “truth”?

2) Reliability: Are the results credible and repeatable?

3) Relevance: Will the results help me in my own research or practice?

Pros of critical appraisal

• Systematic way of assessing a paper’s validity, reliability, relevance

• Helps to close the gap between research and practice

• Encourages objective assessment

• Skills are not difficult – common sense & user-friendly checklists

Cons of critical appraisal

• Can be time-consuming initially

• Does not always provide the “easy” answer – may provide more questions

• Can be dispiriting if it highlights poor quality studies

Examples of appraisal checklists

• CASP for experimental studies, systematic reviews

• DISCERN for consumer health information (papers about treatment choices

• CriSTAL for librarianship studies (information needs analysis, user survey)

Getting started

1) Set up a journal club

2) Remember/devise a suitable scenario

3) Find a relevant paper

4) Choose an appropriate checklist

1) Why set up a journal club?

• Regular protected time for honing your “quality-detector” skills

• Facilitates discussing and evaluating new research and its application to your research or practice

• Helps to unravel complex problems

1) Five steps to journal club success

1. Identify a leader

2. Identify the goals of the club

3. Set up a regular meeting time

4. Send the scenario and paper to everyone before the meeting

5. Encourage active participation by using a checklist

2) Why bother with a scenario?

• Provides a practical focus

• Helps you to answer the applicability question

• Helps you to find a paper

2) Example of a scenario

A 32 year old man is found in garage with engine running. He is brought to the ED by paramedics. He is awake and alert. He complains of a headache. His exam is unremarkable. Skin colour is normal. Neurologic exam is normal.

What is the appropriate therapy?

2) Another example of a scenario

You have been invited to join a local implementation group to look at how to improve information services to those studying ethnomusicology at the University.

The group agrees that they need to find out what ethnomusicologists need, before making any improvements.

2) Another example of a scenario

3) Find a relevant paper

• Use key words from the scenario for database searching

• Browse your favourite journals

• Ask your librarian for help

4) Choose an appropriate checklist

• Identify the study type of the paper

• Google! www.google.co.uk

• Adapt a checklist that is nearly appropriate

• Remember: validity, reliability, applicability

Finally..

• Write up a Critically Appraised Topic (CAT) for future reference www.ouls.ox.ac.uk/hcl

• Evaluate: Did you find the process helpful? If not, why not? How could it be improved?