systematic reviewing for nursing, therapy & allied health- qualitative evidence synthesis...

Systematic reviewing for Nursing, Therapy & Allied health- Qualitative Evidence Synthesis (SyNTAQuES)Andrew Booth, Chris Carroll & Janet Harris, School of Health and Related Research (ScHARR), University of Sheffield.

By the end of this workshop participants will be able to:

• Understand role of qualitative evidence syntheses (QES) /systematic reviews and types of questions they can address.

• Be able to formulate an appropriate question to be addressed by a qualitative systematic review/evidence synthesis.

• Be able to construct an efficient search strategy for retrieval of primary qualitative research and to select appropriate methodology filters for qualitative studies.

• Apply explicit criteria regarding collection/analysis of qualitative data to appraise the quality of a qualitative research report.

• Understand main features and techniques required to conduct QES, specifically thematic synthesis and framework synthesis.

• Understand primary considerations when selecting a synthesis method and how these impact upon the conduct and reporting of a qualitative evidence synthesis/systematic review.

Programme

MORNING

What is Qualitative Systematic Review/ Evidence Synthesis

What are your choices?Formulating a Question for Qualitative ReviewSearching & Sifting for Relevant Studies

BreakQuality Assessment [Plenary]Quality Assessment Workshop

Programme

AFTERNOON

Introducing Thematic Synthesis/Framework SynthesisThematic Synthesis/Framework Synthesis WorkshopFeedback

BreakProducing Your Review for PublicationWhere to Next?

Close

What is Qualitative Systematic Review/Evidence Synthesis?

Andrew BoothSchool of Health and Related Research,

University of Sheffield, UK

Umbrella terms:

• Qualitative Systematic Review (but also non-quantitative, e.g. not meta-analysis)

• Qualitative Metasynthesis (Sandelowski et al, 2001)

• Qualitative Research Synthesis (Howell Major & Savin-Baden, 2010)

• Qualitative Evidence Synthesis (Cochrane Collaboration: Hannes & Lockwood, 2012)

Topic=("qualitative systematic review" OR "qualitative evidence synthesis" OR "qualitative research synthesis") OR Topic=(metastudy OR metasynthesis OR "meta synthesis" OR "meta ethnography" OR "meta ethnographic" OR "metaethnography" OR "metaethnographic") OR Topic=("systematic review of qualitative") = 645 Currently about 10 per month

61% from 2008-2012; 37.5% from 2010-2012

So what is a QES?

• qualitative evidence synthesis – an umbrella term increasingly used to describe a group of review types that attempt to synthesise and analyse findings from primary qualitative research studies (Booth et al, 2011).

Qualitative Evidence Synthesis

• Quantitative meta-synthesis, or meta-analysis, aims to pool numerical results of individual quantitative studies, qualitative meta-synthesis looks for “themes” or “constructs” that lie in or across individual qualitative studies.

• Within broader category of “qualitative meta-synthesis,” (Cochrane preferred term = qualitative evidence synthesis) narrow term “meta-ethnography” refers to specific method of data synthesis most widely adopted to date.

Booth et al, 2006

Key Difference

• Goal not aggregative in sense of “adding studies together,” as with meta-analysis.

• Interpretative in broadening understanding of a particular phenomenon.

An Example• Paterson and colleagues

identified 38 studies examining first hand experience of living with diabetes.

• Prevailing metaphor = concept of “balance”. Specific sub-themes identified across multiple studies included “knowing one's body,” “learning how to manage diabetes,” and “fostering supportive, collaborative relationships with others.”

Stages of a Qualitative Evidence Synthesis?

• Formulating the review question

• Conducting a systematic literature search

• Screening and selecting appropriate research articles

• Analyzing and synthesizing qualitative findings

• Maintaining quality control

• Presenting findings(Sandelowski & Barroso, 2007)

Framework for systematic reviews of qualitative research (Garside, 2010)

Stage Typical activities

Developing research question Assemble team; Consult; Agree approach

Scoping exercise Identify relevant research; Refine methods

Identifying relevant literature Develop Inclusion/Exclusion criteria; Focused searches; Citation searches

Initial assessment of study reports Preliminary reading; Identify theories; Assess utility/relevance

Analysis and synthesis Reading & rereading study reports; Constant comparison; Assess validity

Preliminary synthesis Categorising; tabulating; mindmaps; Explore relationships

Full synthesis Thematic analysis; translation of findings; Theory development; rival explanations

Dissemination Target audiences; Limitations of review

Throughout Multiple viewpoints; Reflexivity; Audit trail; Ongoing consultation; revisit review purpose

Why do it?

• Perspectives of Patients, Relatives or Carers• Of Condition• Of Intervention

• Perspectives of Staff• Implementation• Barriers and Facilitators

• Patient Centred Outcomes

• Development and Testing of Theory

• Answering “What Works for Whom Under Which Circumstances?”

• Acceptability of Treatments

• Explaining Differential Effectiveness

• Designing and Evaluating Complex Interventions

Some Examples - 1• Perceptions of patients

and physiotherapists on patient participation

• Systematic search in six databases using a set of key words, extracted relevant data, performed quality assessment and synthesized findings from selected studies.

• Narrative synthesis of qualitative studies. Retained 11/160 studies.

• Two main themes: conceptualization of patient participation and patients' role preferences. Patient participation included goal setting, information exchange, decision-making and exercise training; often influenced power relation between patient and physiotherapist. Patients' willingness to participate varied; they often did not play their desired role.

SCHOEB & BÜRGE (2012)

Some Examples - 2• Incentives and barriers to

lifestyle interventions for people with severe mental illness.

• Eight electronic databases [1985–March 2009] plus Cochrane Library and Google Scholar. Electronic ‘hand’ searches of key journals and explosion of references also undertaken.

• Narrative synthesis of qualitative, quantitative and mixed methods studies

Reports possible incentives and barriers including:

•Illness symptoms, treatment effects, lack of support and negative staff attitudes as possible barriers;

•Symptom reduction, peer and staff support, knowledge, personal attributes and participation of staff as possible incentives.

ROBERTS & BAILEY (2011)

Centres and Networks: EPPI-CentreJoanna Briggs InstituteCochrane CollaborationCampbell Collaboration

SoftwareEppiReviewerQARI

BooksNoblit & Hare (1988) - Meta-EthnographyPatterson et al (2001) - Meta-Study of Qualitative Health ResearchSandelowski et al (2001) - Handbook for Synthesizing Qualitative Research Petticrew & Roberts (2003) - Systematic Reviews in Social SciencesMays, Pope, Popay (2007) - Synthesizing Qualitative and Quantitative Health ResearchHowell Major & Savin-Baden (2010) - Introduction to Qualitative Research SynthesisHannes & Lockwood (2011) - Synthesizing Qualitative Research: Choosing the Right ApproachSaini & Schlonsky (2012) - Systematic Synthesis of Qualitative ResearchBooth et al (2011) – Systematic Approaches to a Successful Literature ReviewGough, Oliver & Thomas (2012) - An Introduction to Systematic Reviews

What are your Choices?

Andrew BoothSchool of Health and Related Research, University of

Sheffield, UK

Specific Methods

Realist Synthesis

Meta-Interpretation

Critical Interpretive Synthesis

Thematic Synthesis

Framework Synthesis Narrative Synthesis

Meta-Ethnography

Best Fit Synthesis

Meta Study

Meta Narrative

EPPI-Centre Method

Matrix Method

Bayesian Meta-Synthesis

Meta- Aggregation

Decision to conduct a qualitative evidence synthesis

To interpret synthesised qualitative evidence and develop

explanatory theory or models

Purpose of the additional qualitative synthesis

or

Choice of Synthesis (Adapted from Noyes & Lewin, 2011)

To aggregate/ summarise/ integrate qualitative data to address questions in relation to a specific intervention review

Primarily to integrate and interpret qualitative and quantitative evidence within a single approach or integrated model. Can be used to develop explanatory theory.

Thematic analysis without theory generationMeta-aggregationMeta-summary

Product: Aggregated findings from source papers

Meta-ethnographyGrounded theoryThematic analysis with theory generation

Framework synthesis Realist Review

EPPI ApproachMatrix MethodNarrative SynthesisBayesian SynthesisCritical Interpretive synthesis

Product: Explanatory theory, analytical or conceptual framework or interpretative framework/mechanism

Best fit synthesis

Bad Reasons for Choosing Method

• Frequency of Use of Method (e.g. Meta-Ethnography)

• Popularity/”Sexiness” of Method (e.g. Realist Synthesis)

• What a friend/ colleague/ mentor has used (once!)

• Bad experiences of others (may have been inappropriate!)

Key Considerations

1. Role of Theory

2. Available Expertise

3. Extent of Description versus Interpretation

4. Intended ProductOther Considerations: Richness of Data and Available Time/ Resources

1. Will You Generate, Explore, Test Theory (Gough et al, 2012)?

• Generate – may require “suspension of disbelief” – quality assessment/ value judgement may come later (cp. Brainstorming)- Grounded Theory, Meta-ethnography

• Explore – looking for patterns - Narrative Synthesis, Thematic Synthesis

• Test – quality assessment differentiates well-supported and unsupported data - Framework Synthesis (incl. Best Fit Synthesis)

• NB. We (Carroll & Booth) are currently conducting empirical work on systematic identification of Theories

2. What Expertise Can You Access?

Epistemological versus Technical:•Expertise in Qualitative Research Methods (e.g. Grounded Theory; Framework Analysis, Thematic Analysis)•Expertise in Synthesis Methods (incl. Searching, Data Extraction, Quality Assessment, Interpretation) •Knowledge of Topic Area

http://www.jiscmail.ac.uk/lists/asqus.html







3. Will You Describe or Interpret?

All Reviews figure on a continuum between Description and Interpretation

•Description – What does the data say? – factual reporting of studies, themes etc…

•Reader does work of interpretation

• Interpretation – What does the data mean? –subjective interpretation of data and themes etc…

• Reviewer does work of interpretation – may be contested

4. What is Your Intended Output/Product?

• “the output of some methods of synthesis (Thematic Synthesis, textual Narrative Synthesis, Framework Synthesis, and ecological triangulation) is more directly relevant to policymakers and designers of interventions than the outputs of methods with a more constructivist orientation (Meta-Study, Meta-Narrative, Meta-Ethnography, Grounded Theory, CIS) which are generally more complex and conceptual” (Barnett-Page & Thomas, 2009)

• Thematic Synthesis (including Meta-Aggregation) and Framework Synthesis produce findings to directly inform practitioners (Thomas & Harden, 2009)

• Interpretive approaches (e.g. CIS, Meta-Ethnography) produce a model that requires practitioners to interpret relevance and applicability to their own context

• Narrative Synthesis or EPPI-Centre (matrix) methods may help to integrate and present quantitative/qualitative work

How Rich (“Thick”) is Your Data?

• Qualitative data from “thin” studies (or textual responses to surveys) will not sustain interpretive approaches

• Limited to Meta-Aggregation, Thematic Synthesis, Framework Synthesis, Narrative Synthesis –type approaches

• Rich/“Thick” reports will sustain Meta-Ethnography/Grounded Theory – may allow selective sampling/ theoretical saturation

• NB. Is “Unit of Analysis” Individual Study (Meta-Aggregation, Thematic Synthesis) or “Body of Evidence” (e.g. Meta-Narrative or Critical Interpretive Synthesis approaches) or even Theory (Framework Synthesis/Best Fit Synthesis)?

How Long?/How Much Have You Got?

• “Richer” approaches make fuller use of data – require fewer studies

• Meta-Aggregation, Thematic Synthesis can handle large numbers of studies

• Framework/best fit approach for speed

32 papers (775 patients and carers) reporting help-seeking experiences for at least 20 different types of cancer.

Synthesis Quiz

For each of the following scenarios identify the review characteristics and try to match to an appropriate type of synthesis

Scenario A: You are working in a general health technology assessment team conducting a synthesis of Group Therapy for Postnatal Depression. There are few studies and most reports are Nursing Standard/Times case reports. There is no prevalent theory. You will produce recommendations on what works for whom.

1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product

Scenario B: You are working as a group of topic experts and experienced qualitative researchers to examine the phenomenon of “Willingness to Hasten Death”. There are less than eight rich qualitative studies. No-one has yet conducted a synthesis to look at what is meant by the concept. Your review will help those who work in terminal care to gain a better understanding of the phenomenon. 1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product

Scenario C: You are a member of a government-funded Institute producing best practice reviews for nurses and nursing managers. You are asked to examine all evidence, quantitative and qualitative, for physical restraint in residential homes. You are expected to turn around a report in a very short time frame. Your final review is expected to include Recommendations for Practice. 1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product

Scenario D: Your review team, with access to clinical expertise, is commissioned to review the evidence on taking vitamins and supplements to prevent colorectal cancer. Because of the focus on long term outcomes there are few directly relevant qualitative studies. You decide to look for “related” studies and hypothesise attitudes to these specific agents from a wider evidence base. Attitudes to medicine are well-theorised but not within such a long-term prevention perspective. Your qualitative review will be a short chapter in a much longer effectiveness/cost effectiveness report.

1. Role of Theory 2. Expertise3. Describe/Interpret 4. Output/Product

Safest Options!

If…•There is a Pre-existing Theory or Framework….

Then•….Framework Synthesis (including Best Fit Synthesis)

If…•There is No Theory or Framework…

Then•…Thematic Synthesis (Can also act as first stage of Meta-Ethnography)

If…•There is a Proximate (Close-ish!) Theory or Framework….

Then•….Best Fit Synthesis

Formulating a Question for Qualitative Review

Janet HarrisSchool of Health and Related Research, University of

Sheffield, UK

A 15 minute guide

• What is the focus?• Brainstorm questions• Refine• Prioritize• Determine next steps• Reflect

(Adapted from Rothstein and Santana (2011) Teaching Students to Ask Their Own Questions: One small change can yield big results. Harvard Educational Letter)

Focus and brainstorm

Our topic of interest today is: •Taking a client-centred approach to rehabilitation after stroke

Our FOCUS – the problem that has initiated a review of qualitative research : •Health professionals have different ideas about client-centred approaches to rehabilitation

Our AIM:•To produce evidence from a synthesis of the literature that will guide effective client centred-approaches

BRAINSTORM•Ask as many questions as you can about this topic•Do not stop to discuss, judge, or answer the questions.•Write down every question exactly as it is stated.•Change any statement into a question

What did you come up with?

• Broad or narrow question?

• Open or closed?

• Advantages/disadvantages of the questions?

Prioritize• Which questions do you want to pursue

and why?• Consider

• Likelihood of finding qualitative studies• Relevance of studies to problem focus• How the research synthesis will be used

What people have

researched

What I think I am

looking for

Next steps would be…

• Conduct a ‘scoping search’ to see whether the published research seems to be a match for your problem focus

• Use the brief scoping of abstracts to refine the question further

What I think I am

looking for

What I can realistically find!!

The final question for the qualitative review was:

• What are stroke survivors’ experiences with rehabilitation? (Peoples et al, 2011)

Searching and Sifting for Relevant Studies


Sheffield, UK

Relevant Studies…

1. Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors), AND

2. Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]

Other Considerations…

• How much literature is there? How is it characterised? e.g. Was it conceived as Qualitative Research?

• How far back will I go? Has the Intervention/Experience of Living with The Condition Changed?

• What Settings, Countries, Languages will I include? How important is Context?

Post acute stroke rehabilitation

http://www.nlm.nih.gov/nichsr/hedges/search.html

Relevant Studies…

1. Will match the focused question for the review (e.g. The Experience of Rehabilitation for Post-Acute Stroke Survivors),

Matching the Focused Question

• Combination of natural language (e.g. stroke) and index terms (e.g. CEREBROVASCULAR DISORDERS and BRAIN INFARCTION)

• May include subheadings e.g. /rehabilitation as well as index terms

• Need to optimise Sensitivity (Getting everything you want) and Specificity (Getting only what you want)

http://www.ncbi.nlm.nih.gov/mesh/68002561

http://www.ncbi.nlm.nih.gov/mesh/68020520

Sample Search Strategy (SIGN Guideline)

Relevant Studies…

, AND

2. Will employ a qualitative research method (for data collection and analysis) [OR, if the review protocol permits, will include qualitative data e.g. from open-ended survey questions]

Challenges

• Qualitative research is small part of the research literature

• Indexing in databases is inconsistent and variable

• Qualitative studies sometimes have creative titles or inadequate abstracts

• MEDLINE might not be your preferred resource.

• If searching in limited databases try MEDLINE and CINAHL

How would you find qualitative studies?

Sources of Evidence

10/04/23 © The University of Sheffield

Identifying Qualitative Research - Terminology

Generic terms: e.g. “qualitative” plus “ESCAPADE”Exploratory Methods: Focus group, Grounded theory, Action Research, Content analysis, Thematic analysis Software: Nudist or NVivoCitations: Glaser & StraussApplication: Ethnology, PsychologyPhenomenon: Perceptions, Attitudes, User Views, Standpoint, ViewpointApproaches: EthnographicData: Stories, Narratives, Descriptions, Themes, FindingsExperiences: Encounters, Experiences

What is a methodological filter?

• “A hedge or filter is a standardised search strategy that is designed to be used in conjunction with a subject search to retrieve valid studies from the (primary) medical literature”.

• Filters work in one of two ways:

• by identifying particular publication types or study designs most likely to answer a question

• by isolating subject or free-text terms most likely to be associated with high-quality studies

How do they work?

• Filters come from 3 different sources:-

• Subject heading

• Keyword

• Publication type

How do I use a methodological filter?Step One: Carry out a subject search as usual

using subject headings and/or free text

Step Two: Apply methodological filter appropriate to question you are asking

One-line filter

Maximum sensitivity filter

Maximum specificity filter

Mid-range filter

One-line filters

• MeSH Headings• e.g. Qualitative Research [Medline 2003-]

• Keyword • “Findings”

• Publication Type• Research [CINAHL only]

Methodological filters - 1

1. qualitative$

2. findings

3. interview$

4. interviews.DE.

5. 1 OR 2 OR 3 OR 4

Grant MJ. “How does your searching grow? A survey of search preferences and the use of optimal search

strategies in the identification of qualitative research.” Health Info Libr J. 2004 Mar; 21(1):21-32.


• Best sensitivity: exp interviews

• Best specificity: audiorecording.sh.

• Best optimization of sensitivity and specificity: exp study design

Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res. 2007 May;17(5):705-10.


• Don’t forget MeSH heading Qualitative Research

• Introduced in 2003, so coverage is limited

• But you can combine it with one of the previous filters by using the “OR” operator

Sources of filters

• ISSG search filters resource http://www.york.ac.uk/inst/crd/intertasc/

• SIGNhttp://www.sign.ac.uk/methodology/filters.html

• McMaster Hedges Project http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx

• PubMed Clinical Querieshttp://www.ncbi.nlm.nih.gov/corehtml/query/static/clinical.shtml

http://www.york.ac.uk/inst/crd/intertasc/

http://www.york.ac.uk/inst/crd/intertasc/

http://www.sign.ac.uk/methodology/filters.html

http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx

http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx

http://www.ncbi.nlm.nih.gov/corehtml/query/static/clinical.shtml

Each Question requires a different solution...

• Findings showed that a simple search strategy (broad-based terms - 3 search terms) was as effective as a complex one (free text - 48 search terms) in locating qualitative research on patients’ experiences of living with a leg ulcer.

• May be feasible to restrict searches with a clear nursing focus to CINAHL bibliographic database.

• Replication with other nursing topics is required.Flemming K, Briggs M. Electronic searching to locate qualitative

research: evaluation of three strategies. J Adv Nurs. 2007 Jan;57(1):95-100.

“Supplementary” Methods

• Citation searches

• Reference lists

• Hand searching

10/04/23 © The University of Sheffield

Citation searching- Google Scholar

In Conclusion• Identifying Qualitative Research is Challenging

• Simple strategy with qualitative terms (e.g. Interview*; qualitative; findings etc) may be more effective than exhaustive lists of terms

• May not be necessary to search as many databases as quantitative topics

• But supplementary methods (e.g for books and theses) may be very important!

• Cp. Cluster searching!

Reading the Reports

Janet HarrisSchool of Health and Related Research,

University of Sheffield, UK

Aims and objectives

• What do we mean by quality assessment?

• What does it involve?

• How do you do it?

• What are the strengths and weaknesses of different tools or checklists?

Quality assessment: Basic criteria• Carrying out ethical research• Importance of the research• Clarity and coherence of the report• Use of appropriate and rigorous methods• Importance of reflexivity or attending to

researcher bias• Importance of establishing validity or

credibility• Importance of verification or reliability

(Cohen et al, 2008)

Quality assessment: 3 stages

1. Filtering against minimum criteria, involving adequacy of reporting detail

2. Assessing technical quality of the study’s elements - methodological soundness, credibility, validity

3. Judging theoretical consistency

Filtering: Relevance and adequacy of reporting

• Relevance: generally judged against title and abstract in the first instance

• Reporting: Is the study process reported well enough to judge technical rigour? This depends on your area of interest…• Intervention

Core elements of the intervention as plannedNon-core elements e.g. components that are added to suit a

particular context or/or population

• Theory development: Relationship between Context, Mechanisms and Outcomes

• Implementation: barriers and facilitators to implementing the intervention

Quality appraisal: Basic criteria

Technical quality – Qualitative concepts TechniquesCredibility: the representation of data fits the views of the participants studied, the findings hold true

•outside auditors or participants validate findings (member checks)•peer debriefing, •attention to negative cases, •independent analysis of data by more than one researcher •verbatim quotes•persistent observation (stay in the field long enough)

Confirmability: findings are qualitatively confirmable through the analysis being grounded in the data, through examination of the audit trail

assessing the potential effects/impact of the researcher during all steps of the research processReflexivity toward personal influences, biasproviding background information on the researcher’s background, education, perspective, school of thought

Dependability: process of research is logical, traceable and clearly documented, particularly on the methods chosen and the decisions made by the researchers

peer review, debriefing, audit trailstriangulation, the use of different methodological approaches to look at the topic of researchreflexivity to keep a self-critical account of the research processcalculation of inter-rater agreements

Transferability: research findings are transferable to other specific settings

providing details of the study participants to enable readers to evaluate for which target groups the findings potentially hold trueproviding contextual background information, demographicsproviding thick description about both the sending and the receiving context

Validity: linking data collection

and analysis to conclusions• When evaluating

methodological soundness, we need to know • whether the set of

arguments or the conclusion derived from a study necessarily follows from the premises.

• whether it is well grounded in logic or truth.

• whether it accurately reflects the concepts, the ideas that it is intended to measure.

Theoretical quality

• Does your review question aim to develop an explanation for something – a theory?

• If it does, then you need to consider theoretical quality

• How was the theory developed?• Is there congruence between the data and the

interpretation of findings? Can you see a logical connection or translation of findings into theory?

Revisiting relevance?

• Relevance can be a multistage process• The first stage involves excluding obviously irrelevant

papers (filtering)

- The second stage involves weighing relevance to the review question with the quality of reporting and technical competence at establishing validity Data may be relevant and insightful but not rigorously analyzed...do you include it?

Criteria1) There is congruity between the stated philosophical perspective and the research methodology.2) There is congruity between the research methodology and the research question or objectives.3) There is congruity between the research methodology and the methods used to collect data.4) There is congruity between the research methodology and the representation and analysis of data.5) There is congruity between the research methodology and the interpretation of results.

6) There is a statement locating the researcher culturally or theoretically.

7) The influence of the researcher on the research, and vice-versa, is addressed.

8) Participants, and their voices, are adequately represented.9) The research is ethical according to current criteria or, for recent studies, there is evidence of ethical approval by an appropriate body.

10) Conclusions drawn in the research report do appear to flow from the analysis, or interpretation, of the data.TOTAL

QARI Critical Appraisal Instrument

Criteria categorization and definition

Criteria categorization and definition

Study design:Yes - if e.g., “a case study approach was used . . .”, “phenomenology was used . . .”No - If paper does not specify study design Unclear if unsure

Method of data collection:Yes - If details of the data collection method are given e.g., piloting; interviews, topic guides for interviews; number of items in a survey; use of open or closed items; validation, and so forth.No - If only states “focus group”, “interviews were used” or “questionnaire was used”Unclear if unsure

Selection of participants:Yes - If the sampling and recruitment of participants is described in full or explicitly as e.g., purposive, convenience, theoretical and so forth.No - If only details of participants are given, e.g. age, gender, numberUnclear if unsure

Methods of data analysis:Yes - If full details of analysis method are given, e.g., transcription and form of analysis (with reference or full description of method), validation tests, and so forth.No - If only states “content analysis” or that “data were analyzed”Unclear if unsure

Quality of Reporting Tool (Carroll et al, QHR 2012)

Appraising a paper for a qualitative review

• Group 1: Use the QARI appraisal tool

• Group 2: Use the Quality of Reporting Tool

• Make your decision about the quality of the paper

• Technical quality

• Relevance

• Theoretical quality

Integrating papers

• Would you use this paper in the qualitative review?• Poor methodological quality but

relevant data and valuable insights

• Good quality, but poor interpretation and limited insight

• Which gets privileged – the quality score or useful insight?

Learning outcomes

• An understanding of • What we mean by quality assessment

• What critical appraisal involves

• How you conduct an appraisal

• The strengths and weaknesses of different tools or checklists

• Which tool might be best for you

Introducing framework synthesis and thematic

synthesis

Chris CarrollSchool of Health and Related Research, University of

Sheffield, UK

Aims and objectives

• What do we mean by framework synthesis and thematic synthesis?

• How do you extract data for synthesis?

• How do you perform both types of synthesis?

• How do you “complete” the synthesis?

Time Content

13.15-13.35 Introduction to two types of synthesis

13.35-14.45 Extracting and coding data: IntroductionPractical (10 mins)Feedback (5 mins)How to do thematic analysisPractical (15 mins)Feedback (10 mins)Analysis and synthesis

14.45-15.00 Feedback

15.20-15.45 Producing your review for publication (AB)

A Framework

How do you create a framework?

Develop own framework or themes or concepts for coding Oliver S et al: A multidimensional conceptual framework for analysing

public involvement in health services research. Health Expectations 2008, 11:72-84.

Brunton G, Oliver S, Oliver K, Lorenc T. A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport London. London, EPPI-Centre, Social Science Research Unit, Institute of Education, University of London; 2006.

Identify or create a thematic framework or conceptual model Carroll C, Booth A, Cooper K. A worked example of “best-fit”

framework synthesis: A systematic review of views concerning the taking of potential chemopreventive agents, BMC Medical Research Methodology 2011; 11: 29

An existing conceptual framework

Conrad K, et al (1996). The worksite environment as a cue to smoking reduction. Research in Nursing & Health, 19 21-31.

A Framework

Framework synthesis

Thematic synthesis

The case study

The question:

What are the barriers and facilitators of patient autonomy post stroke?

The studies:

1. Proot I et al, Patient autonomy during rehabilitation: the experiences of stroke patients in nursing homes, International Journal of Nursing Studies, 2000; 37: 267-76.

2. Mangset M et al, ‘We’re just sick people, nothing else’: ... Factors contributing to elderly stroke patients satisfaction with rehabilitation, Clinical Rehabilitation, 2008; 22: 825-35.

Data extraction

What is it?

An attempt to reduce a mass of material (your included papers) to a much smaller body of text and numbers, amenable to analysis and the interpretation of findings

Data extraction form

Location Setting Sample (n) Age, Gender, Ethnicity and other relevant variables (e.g. Marital

status, socio-economic status) Intervention (if any) Outcomes and results Study design Data collection method Quality assessment criteria Further citations

Format?????

Example form: Proot 2000

What results do you extract?

What is your question?

Question:What are the barriers and facilitators of patient autonomy post-stroke?

Keep the question in mind as you read:Are the data relevant to this question?Is the question answered by the data?

Extraction of results

Results sections of included studies: Authors’ statements clearly-supported by data

and/or Verbatim quotations

“... It was often difficult to distinguish [Schutz’s] first- from second-order constructs”

Atkins et al. Conducting a meta-ethnography of qualitative literature: Lessons learnt, BMC Med Res Methodol 2008; 8: 21

Schutz A. Collected Papers 1971; vol.1: 361.

From: Mangset p.829

Extracting data for thematic analysis

Extracting or “coding” data using framework analysis

Framework developed from ...

Kirkevold M et al. Promoting psychosocial well-being following a stroke: developing a theoretically and empirically sound complex intervention, International Journal of Nursing Studies, 2012; 49: 386-97.


Code number Theme Definition

1 Dignity A self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself

2 Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness

3 Coping Ability to adapt and cope within home and hospital settings

4 Family Support Relations with immediate family

5 Support for patient’s coping efforts

Patients take responsibility and are supported by others in this

6 Bodily changes and impairments

Difficulties generated by residual or post-stroke conditions

Handout 5

Extraction and coding: Practical

Consider the full framework in Handout 5 Consider extracts A, B and C on Handout 6 Code the extracted data according to the framework by

entering the number of the framework code in the right-hand column on Handout 6

Note: Data may be assigned more than 1 theme No code in the framework, in your opinion, might capture, “fit”

or reflect the data. If this is the case, enter “New theme” into the right-hand column

10 minutes: Individual and compare with neighbour; 5 minutes: Discussion of results and experiences

What now?

Secondary thematic analysis Practical: Thematic analysis Discussion of practical Synthesising your data

What now?

You have extracted your data

AND You have coded your data against your framework

(framework analysis) Or you may have coded some of your data but not

others (“best fit” framework synthesis)

OR You are conducting thematic synthesis anyway ...

You now need to assign themes to those data without themes, i.e. Using secondary thematic analysis (data reduction, display, conclusion)

How do you do it?

Generating new themes

“Evidence does not speak for itself. It requires interpretation .... on the basis of our personal experience and a range of ideas that we use to make sense of our observations”

Kelly M et al. Evidence-based public health: A review of the experience of NICE developing public health guidance in England, Social Science and Medicine, 2010; 71: 1056-62.

Developing new themes

C. “…“I will take that paper when P. is finished with it. I want to study it. Then I'll take my shoes off, and practise as I see it in the picture” (p04b, p. 11), Proot, p.271

Other / New theme? Interpret Create Revisit and revise Discuss Agree ...

= Learning self-help skills / guided self-determination

Thematic analysis: Practical

Consider extracts D, E and F on Handout 6 Develop and assign new themes to these extracts and

enter them in the right-hand column on Handout 6

Note: Data may be assigned more than 1 theme There may also be code in the framework which, in your

opinion, might capture, “fit” or reflect the data. If this is the case, also enter this code into the right-hand column

15 minutes: Individual and compare with neighbour; 10 minutes: Discussion of results and experiences

Generating new themes: Practical

Feedback Interpretation Bias?

Reviewers might have agendas too

What is the result of the secondary thematic analysis?

Synthesis

Report the results of the new thematic framework Narrative structured by theme

Report each theme with reference to studies and data in order to: Specify the evidence base for the theme Illustrate the theme with reference to actual data

McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):130-153

Synthesis

Synthesis

Reduce the findings into a smaller number of categories and then a single finding (metasynthesis, according to the Joanna Briggs Institute approach):

McInerney P & Brysiewicz P. A systematic review of the experiences of caregivers in providing home-based care to persons with HIV/AIDS in Africa, JBI Library of Systematic Reviews, 2009 ;7(4):130-153

Synthesis

A new framework

Code number Theme Definition

1 Dignity A self-concept characterised by self-valuing, self-acceptance, usefulness and belief in oneself

2 Well-being A basic mood of joy, pleasure and wellbeing; the absence of feelings of emptiness or sadness

3 Coping Ability to adapt and cope within home and hospital settings

4 Family Support Relations with immediate family

5 Support for patient’s coping efforts

Patients take responsibility and are supported by others in this

6 Bodily changes and impairments

Difficulties generated by residual or post-stroke conditions

7 Self-determination Define theme as a concept, based on data

8 Need for information Define theme as a concept, based on data

9 Lack of shared decision-making

Define theme as a concept, based on data

10 Acceptance of therapy Define theme as a concept, based on data

A new framework and synthesis

Theme Categories

Dignity

Psychological factors

Well-being

Self-determination

Coping

Bodily changes and impairments Physical factors

Need for information

Interaction with professionals and family

Lack of shared decision-making

Family Support

Support for patient’s coping efforts

Acceptance of therapy

Synthesis

Alternatively, go beyond the framework to create a new conceptual model or theory ...

Revisit data to explore the relationships between the themes or findings of your framework

The “richer” or “thicker” the data, the deeper the resulting model or theory

Narrative and diagrammatic representations

Synthesis

Do you stop there?

Test your synthesis? Disconfirming cases

Booth A, Carroll C, Illott I. Desperately Seeking Dissonance: Identifying the “Disconfirming Case”, Qualitative Evidence Synthesis, Qualitative Health Research 2012 (in press).

Morse, J. M. The significance of saturation. Qualitative Health Research 1995; 5, 147-149.

Sensitivity analysis By quality, population, location, setting etc. By frequency and “thickness”

Carroll C, Booth A, Lloyd-Jones M. Should we Exclude Inadequately-reported Studies from Qualitative Systematic Reviews? An Evaluation of Sensitivity Analyses in Two Case Study Reviews, Qualitative Health Research 2012; 22(10)

Boeije et al (2011) Making a difference: towards a method for weighing the evidence in a qualitative synthesis, Journal of Evaluation in Clinical Practice 2011 17(4):657-63

What is the purpose of your review and synthesis?

To answer questions and to solve problems that cannot be addressed by quantitative research

To illuminate patient and client experience and to develop theory

To be useful

Learning outcomes

• An understanding of • What we mean by framework synthesis

• What we mean by thematic analysis

• How you conduct framework and thematic synthesis, or “best-fit” framework synthesis

• The strengths and weaknesses of the different methods

• How you “complete” your synthesis

Summary

Extract and interpret data

Synthesis

Test your synthesis

Integrating and writing-up ...

Producing Your Review for Publication


Sheffield, UK

Between Two Traditions

• Systematic Review Publishing Guidelines (e.g. PRISMA – formerly QUOROM)

• Primary Qualitative Research Reporting Traditions

Qualitative Research

Systematic Reviews

QES

Requirements for a Published QES

• Systematic Review requires:• Explicit methods

• Transparency

• Audit Trail

• Review Question

• Search strategies & sources

• Quality Assessment

• Method of Synthesis

• Strategies to reduce bias

• Primary Qualitative Research requires:

• “Believability”

• Findings Grounded in the Data

• Themes/Constructs

• Selective Findings

• Reflexivity

Purpose of Study

Databases and Keywords

Search Strategy

Methods of Appraisal/ Extraction

Methods of Synthesis

PRISMA Flow Diagram

NB. This is non-standard but informed by QUOROM/ PRISMA

Themes

Findings

Studies

Narrative synthesis

Verbatim Extract

Unsubstantiated Comment

Relationship between Themes

Structuring Recommendations

Where to Next?


Sheffield, UK

Further Reading

Howell Major & Savin-Baden (2010)

Hannes & Lockwood (2011)

Further Resources

http://www.mendeley.com/groups/518691/cochrane-qes-register/



Further Support

http://www.jiscmail.ac.uk/asqus http://cqrmg.cochrane.org/

https://www.jiscmail.ac.uk/asqus

http://cqrmg.cochrane.org/

Further Training

• Look out for SyNTAQuES – II (the Intermediate Version) from the Higher Education Academy

http://www.heacademy.ac.uk/events

• Advanced Training is available from: ESQUIRE13, September 2nd-4th 2013, ScHARR, University of Sheffield

http://www.sheffield.ac.uk/scharr/shortcourseunit

Previous materials at:

http://esquiresheffield.pbworks.com







http://esquiresheffield.pbworks.com/

http://esquiresheffield.pbworks.com/

Some Issues to Think About!

Or it’s not all that simple really!

Qualitative Systematic Review

Systematic Review Qualitative Data

Explicit? (cp. Iterative)

Comprehensive? Qualitative Research?

Reproducible? (cp. Reflexivity/ Subjectivity/ Interpretation)

(cp. Theoretical Saturation; Purposive Sampling)

Particular Type of Study (e.g. all Grounded Theory Studies)?

Follows “System”: from Systematic Review or from Primary Qualitative Research?

With/Without Quantitative Research?

Qualitative Data?

systematic reviewing for nursing, therapy & allied health- qualitative evidence synthesis...

Documents

qualitative review

qualitative metasynthesis

qualitative findings

qualitative research

individual qualitative

meta synthesis

synthesis method

close slide