Information Retrieval for High-Quality Systematic

Reviews:The Basics

6.0

The Eight Steps of a High Quality Systematic Review

1. Formulate Review Questions

2. Define inclusion and exclusion criteria

3. Locate Studies

4. Select Studies

5. Assess Study Quality

6. Extract Data

7. Analyze and present results

8. Select Studies

5.0

Finding Evidence:A Conceptual Framework

Hand Searches Electronic Searches Invisible College

Grey Literature Published Literature

C2-SPECTR

C2-PROT

RCTs CRTs

Electronic

Registers

Systematic Reviews

C2-RIPE What Works Clearinghouse

Co

chra

ne

Co

llab

ora

tio

n

Cam

pb

ell C

olla

bo

rati

on

EPPI Center

Information Retrieval: A Continuous Process

– Preliminary Searches• Supports beginning steps: Definition of key concepts &

research question• Use of standard reference tools and broad searches for

review articles and key primary studies

– Main Searches• Identification of primary studies through searches of

online databases, printed indices, Internet, branching, hand-searches

• Most difficult given a number of challenges

– Final Searches• Occurs towards the end of the Review Process• Refine search terms and update original searches

Main Searches: Decisions

• Scope of search: Which fields should be searched (e.g. beyond Education)?

• Availability of indexing tools: Which tools do we have access to at our institution?

• Format of indexing tools: What format are they in (e.g. online, print, web-based)?

• Date: How far back does the indexing go for each tool?

• Language: What is the language of the material that is indexed? How can we locate non-English material?

• Unpublished work: How can we access dissertations, reports, & other grey literature?

Example: Parent Involvement– Fields searched: Education, Psychology, Sociology, Social

Sciences, Economics, Population, Family

– Geographic scope of indexing tools: Mostly North American, some UK

– Format of indexing tools: Online/web-based databases

– Coverage Date of Indices: Not noted

– Language: Foreign language included (when possible)

– Unpublished work: Search of dissertations, web sites, policy documents

Main Searches: Decisions

– Preparation of Search Strategies• What are the key concepts to be searched?• How are these represented in each field?• What are their related terms?• How are these key concepts represented in the controlled

vocabulary within each indexing tool?

– Construction of the Search Statements• What terms should be searched as descriptors or as “free text”? • What boolean operators should be used?• What limiting features are available to target primary studies

only (e.g. use of Document Type codes)?• What time period should be searched?

– Note. These decisions need to be made for each indexing tool used.

Example: Parent Involvement

– Descriptors: Parent Involvement See: Parent participation

– Related descriptors: Family involvement, Parent-school relationship, Parent-student relationship, Parent role, Parents as teachers

– Related keywords: parent* involvement, parent* effectiveness, parent* support, famil* support, famil* participation

– Descriptors: Academic Performance See: Academic achievement

– Related descriptors: Science achievement, Reading achievement, Literacy Achievement, Writing achievement, Achievement gains

– Descriptors: Elementary School Children See: Elementary school students

– Related descriptors: Elementary education, Primary education, Kindergarten

Example: Parent Involvement

• Combining Keywords/Descriptors using Boolean & Proximity operators:

1. (Parent participation OR Family involvement OR Parent-school relationship OR Parent-student relationship OR Parent role OR Parents as teachers) in DE

2. Parent* NEAR2 involvement or Parent* NEAR2 effectiveness OR Parent* NEAR2 support OR Famil* NEAR2 support or Family NEAR2 participation

3. #1 OR #2

4. (Academic achievement OR Science achievement OR Reading achievement OR Literacy Achievement OR Achievement gains) in DE

5. (Elementary school students OR elementary education OR primary education OR kindergarten) in DE

6. #3 AND #4 AND #5

Limiting commands: Research reports (DTC=143)

Main Searches: ChallengesPrecision vs. Recall (White, 1994)

– Precision: ratio of docs retrieved and judged relevant to all those actually retrieved

– Recall: ratio of relevant docs retrieved to all those in a collection that should be retrieved

– Regular searches have an inverse relationship - High precision vs low recall (a small set of highly relevant records)

– Reviewers want high precision and should want high recall (a large, comprehensive set of relevant records with minimal bias)

– Achieving this balance is the challenge for reviews!

• Use of incorrect terminology within a field may miss relevant records (low recall)

• Use of of incorrect or redundant terminology may yield unwanted records (low precision)

Another Example: Distance Education

Compare the retrieval of 1 record vs 510!

1. elearn (1 record)2. e near1 learn* (459 records)

AU: Holstrom,-LisaTI: Eliminating Barriers for All E-Learners.SO: Educational Technology. v43 n6 p61-62 Nov-Dec 2003DE: *Access-to-Education; *Distance-EducationDE: Feedback-; Higher-Education; Literacy-; Low-Income-

Groups; Skill-DevelopmentIDM: *Barriers-to-Participation; *Electronic-LearningIDR: Educational-Media-Use; University-of-Cincinnati-OH

3. (electronic learning) in DEM,DER,IDM,IDR (104 records)4. ((electronic learning) in DEM,DER,IDM,IDR) or (e near1

learn*) or (elearn) (510 records)

Main Searches: Decisions

• Other Retrieval Strategies– Branching:

• Scanning the reference section of review articles

– Hand searches:• Scanning the Table of Contents of key journals and

conference proceedings

– Personal contacts:• Contacting key researchers in the field

– Consulting C2 resources

Summary “Shoestring-budget information retrieval is likely to introduce bias,

and should be avoided.” (IR Policy Brief, 2004)

• Importance of information retrieval process– Not a “one-shot”deal– Requires expertise in the planning and implementation

of searches– Library personnel are important members of the team

• Use of bibliographic management software– Reference Manager, EndNotes, & Biblioscape

• Ability to replicate review– Documentation of entire process, including search

strategies used for each database, decisions taken, etc.