getting and making sense of “the best available” scientific evidence
DESCRIPTION
Getting and making sense of “the best available” scientific evidence. EBMgt Helping Managers Make Better Decisions. What is evidence?. Evidence is not the same as ‘proof’ or ‘hard facts’ Evidence can be - so strong that no one doubts its correctness , or - PowerPoint PPT PresentationTRANSCRIPT
Postgraduate Course
Getting and making sense of “the best available” scientific evidence
EBMgtHelping Managers Make Better
Decisions
Postgraduate Course
Evidence is not the same as ‘proof’ or ‘hard facts’
Evidence can be
- so strong that no one doubts its correctness, or
- so weak that it is hardly convincing at all
What is evidence?
Postgraduate Course
Evidence of effect (do!)
Evidence of no effect (don’t!)
No evidence of effect (research!)
Don’t confuse
Postgraduate Course
Assignment
CAT
Critical Appraised Topic
Postgraduate Course
CAT: Critical Appraised Topic
A critically appraised topic (or CAT) is a structured, short (2
pages max) summary of evidence on a topic of interest,
usually focused around a practical problem or question. A CAT
is like a “quick and dirty” version of a systematic review,
summarizing the best available research evidence on a topic.
Usually more than one study is included in a CAT.
Postgraduate Course
CAT: structure
1) Question2) PICOC3) Background4) Search strategy5) Results6) Conclusion7) Comments8) Recommendation9) References
Postgraduate Course
Asking the right questions
Part 2
Postgraduate Course
5-step approach
Gathering Best Scientific Evidence isa 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence3. Critical appraise the quality of the found
evidence4. Integrate the evidence with managerial
expertise and organizational concerns and apply
5. Monitor and evaluate the results
Postgraduate Course
Asking the right question?
Does team-building work?
Does leadership development training work?
Does management development improve the
performance of managers?
Does employee participation prevent
resistance to change?
Is 360 degree feedback effective?
Postgraduate Course
P = Problem or population
I = Intervention or successfactor
C = Comparison
O = Outcome
C = Context
Answerable question: PICOC
Postgraduate Course
Scenario: You are a consultant, your client is an insurance company, there are plans for a merger, you have heard that the other company
has a different culture, you want to know if this will effect the outcome
P = Organizations with a different corporate cultureI = MergerC = Organizations with a similar corporate cultureO = Long term profitability(C)) = Profit organizations, competitive market)
Answerable question: PICO(C)
Postgraduate Course
Searching evidence
The problem with finding evidence:
the abundance of literature
Postgraduate Course
Searching evidence
Article 1
Article 2
Article 3
Article 4
Article 5
Article 6
Article 7
Article 8
There are about 1350 articles published on HRM every year. For an HR
professional to keep up this means reading 3 to 4 articles every day!
(most of these publications are not valid or irrelevant)
Postgraduate Course
Searching evidence
Evidence-based searching
In a systematic and transparant way searching for the “best” evidence Part of EBMgt where decision maker is not a ‘subject matter expert’
Postgraduate Course
Searching for scientific evidence
What kind of evidence are we looking for?
1. Studies with a design that best suits the research question
2. Studies with the highest level of evidence
Postgraduate Course
Explanation
Which design for which question?
Postgraduate Course
Levels of internal validity
Postgraduate Course
Searching evidence
Where do we search?
Postgraduate Course
Peer reviewed journals
Postgraduate Course
Which journals are peer reviewed?
Postgraduate Course
Type of Information Source
Current Information Wall street Journal, Financial Times, Business week, Financieel Dagblad
Overview of a subject
General background
Academic Information
Statistical Information
Textbooks and popular books
Encyclopedias, yearbooks & book reviews
ABI/INFORM, Business Source Premier, Emerald, PsychInfo, Science Direct
CBS Statline, Eurostat
Theories about a subject Textbooks and encyclopedias
Information sources
Company information Company Annual Reports, Datastream, Factiva.com, Amadeus
Postgraduate Course
Searching evidence
How do we search?
Search Strategy
Postgraduate Course
Why do we need a search strategy?
Promotes deeper learning about your question
Leads to better yield of quality research.
Saves time in the long run.
Source: Inky Bob, www.flickr.com, Creative Commons, April 2006.
Search strategy
Postgraduate Course
Two types of search strategies
Search strategy
Building blocks methodSnowball method
Postgraduate Course
Snowball method
Starting from one book or article, you search for other literature on the same topic.
Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article).
Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).
Postgraduate Course
Synonyms or related terms
…. …. …. ….
Synonyms or related terms
…. …. …. ….
Synonyms or related terms
…. …. …. ….
Building blocks method
Synonyms or related terms
…. …. …. ….
Keyword 1 Keyword 2 Keyword 3 Keyword 4
AND AND AND
OR OR OR
Postgraduate Course
P = back office employees
I = merger, integration, back office
C = status quo
O = economy of scale
C = healthcare, different organizational culture, unequal
Answerable question: PICOC
1. Underline the keywords
2. Number the order of importance from 1-4
Postgraduate Course
P = back office employees
I = 1. merger, 3. integration, back office
C = status quo
O = 4. economy of scale
C = 5. healthcare, different 2. organizational culture, unequal
Answerable question: PICOC
1. Underline the keywords
2. Number the order of importance
Postgraduate Course
Search terms
Operationalise your Pico elements!
O = long term profitability?
Postgraduate Course
corporate culture: organizational behavior/character, corporate identity
merger: acquisition, take-over, fusion, combination, unification
profitability: profit, advantage, return on investment, shareholder value
The keywords of your PICOC may be enough.If not, select more words by using:
Select keywords
synonyms
alternate spelling, translations
related terms / words / subjects
narrower or broader terms
Postgraduate Course
Search Query
1. Search with #1 PICOC term (incl. alternative terms,
synonyms, alternate spellings, truncations, etc.) in the
thesaurus, title or abstract
2. Combine the results with OR (use the history function!)
3. Search with #2 PICOC term (incl. synonyms, etc.)
4. Combine the results with OR
5. Combine the results of step 2 and 4 with AND
Postgraduate Course
• Merger
• Fusion
• Combination
• Take over
• Acquisition
• Unification
• …
1. Merger 3. Integration
• Healthcare organization
• Non profit• Not for profit
4. Health care organization
AND
Search Query: an example
I I O
OR OR
• Integration • Corporate culture
• Organizational behavior
• Organizational character
• Corporate identity
• Core beliefs
• Shared values
2. Corporate culture
C
AND
OR
Postgraduate Course
Use the history function to combine results
Search Query
Postgraduate Course
Search Query
Postgraduate Course
Boolean operators
AND = both terms (apples AND oranges)
OR = either one of these terms (apples OR oranges)
NOT = without this term (fruit NOT oranges)
NEAR = near this term (apples NEAR oranges)
* = replaces 0,1 or more characters (apple*= apple, apples, applejack, applejuice, applepie, etc.)
?= replaces 1 character (organi?ation= organisation, organization)
Postgraduate Course
orangesapples
apples AND oranges apples OR oranges
orangesapples
Boolean operators
Postgraduate Course
Justify your search strategy
Why?To help the reader of your paper:
Follow the steps of your search process Understand the end results
How? Including keywords used for the search actions Justify information sources used (literature list)
Search Strategy
Postgraduate Course
Include literature references
Why? To give other authors the credit they are due. To show that you have made use of reliable sources To show the relationship between your work and that of others. To show that you have studied the subject in depth To make it possible to check your work. To avoid committing plagiarism !!!
How? Cite & include references to acknowledge all your sources carefully. Include sufficient own / new ideas in your work. You can make use of Reference Manager or Endnote
Search Strategy
Postgraduate Course
Start up
select ‘advanced’ select ‘peer-reviewed’ select ‘ABI/INFORM Global’
Postgraduate Course
ABI / Inform
Postgraduate Course
Learning through play !Try all buttons
Make lots of mistakes
Have fun !
Go do it & report back next week.
Postgraduate Course
Levels of evidence = A hierarchical order for research designs based on their internal validity
Internal validity = Degree the results may be unbiased. Higher when conditions demonstrating causality are present (1. control over “cause”, 2. temporal order, and 3. control over or no plausible alternative explanation for findings). Careful design of primary studies promotes these three conditions but seldom eliminates them. Threats to internal validity are overcome when
accumulated studies with different designs yield comparable findings.
Levels of Scientific Evidence
Postgraduate Course
Levels of internal validity
Postgraduate Course
Levels of internal validity
It is shown that …
It is likely that …
Experts are of the opinion that …
There are signs that …
Postgraduate Course
But … sometimes observational studies are as good as RCT’s
Internal validity
When the size of effect is very large (swamps the bias)
Postgraduate CourseGeneralizability
Degree findings hold across populations, settings, procedures etc. (external validity).
Reasons for rejecting generalizability must be logical and evidence-based (not mere dislike of findings)
Logical threats to generalizability include: Person/Treatment interactions: e.g., incentives based on
dice throw that work for gamblers and not Baptists File draw problem: Studies only published if show
significant effects (why unpublished sources matter)
Postgraduate Course
These treatments have not been tested in RCTs: are they supported by poor evidence?
Internal validity
Heimlich manoeuvre Dehydration: drinking water
Cardiac arrest: AED
Postgraduate Course
Better than a single study: a replication study
Better than a replication study:a systematic review / meta analysis
If there were 100 studies, 99 of which gave a ‘negative’ result (where, say, the new intervention appeared to be not effective), while one had a ‘positive’ result (were the intervention appeared effective), it would obviously be a mistake to consider only the single positive study.
But ….
Postgraduate CourseResearch designs
Systematic review or meta-analysis
Randomized controlled study (experiment)
Non-randomized controlled study (quasi-experiment)
Observational research: cohort-, panel-, case-control and cross-sectional study
Before-after study (pretest – posttest design)
Qualitative research
Postgraduate Course
Systematic review
The intention behind a systematic review is to identify as fully as possible all the scientific studies of relevance to a particular subject and to assess the validity and authority of the evidence of each study separately. As the name indicates, a systematic review takes a systematic approach to identifying studies and has the methodological quality critically appraised by multiple researchers independently of each other, as a consequence of which the review is transparent and reproducible and can be monitored. The use of statistical analysis techniques in a systematic review to pool the results of the individual studies numerically in order to achieve a more accurate estimate of the effect is termed a “meta-analysis”.
Postgraduate Course
Postgraduate Course
Founded in 1993 and named after the British epidemiologist Archy Cochrane
International non-profit and independen organization
Mission: to enable people to make well-informed decisions abouth healthcare
Dedicated to making up-to-date, accurate information about the effects of healthcare readily available worldwide.
Main product: The Cochrane Database of Systematic Reviews 1995: 36 reviews 1999: 500 reviews 2001: 1000 reviews 2004: 2000 reviews + 1400 published protocols
Reviews prepared by healthcare professionals who volunteer (10.000 people worldwide)
Application of rigorous quality standards
Cochrane Collaboration
Postgraduate Course
Controlled study
In a controlled study two or more groups are compared with each other, usually comprising one group in which an intervention is carried out (experimental group) and one group where no or an alternative intervention is conducted (control group).
Postgraduate Course
In the case of randomization, the groups compared with each other are selected entirely randomly, for example by drawing lots. This means that each participant (or other unit such as a team, department or company) has an equal chance of being in the intervention or control group. In this way, the influence of any distorting factors is spread over both groups so that these groups are as comparable as possible with each other with the exception of the intervention.
Randomization
Controlled study
Postgraduate Course
Observational research
Cohort/panel study—entities followed over time
(classic longitudinal study of AT&T managers)
Case-control study—comparisons between entities
with different outcomes (Collins’ “Good to Great”)
Cross-sectional study—one-time assess’t (turnover
rates of high performing employees in 2012
Observational research refers to studies where the researcher merely observes but does not intervene, with the intention of finding correlations among the observed data (synonym: naturalistic study, non-intervention trial)
Postgraduate Course
Case-control study
Longitudinal study in which one group of people or companies with a particular outcome (for example, above-average performance) is compared subsequently (= retrospective) with a group that does not have this outcome.
Postgraduate Course
Cross-sectional study
Study in which data of a statistically significant sample of a population (managers, CEO’s, employees) is gathered at one point in time. It provides a snapshot of the current condition but does not explain cause and effect.
Cross-sectional studies include surveys
Postgraduate Course
Assessing the quality of a study
Postgraduate Course
Intermezzo
How to read a research article?
Postgraduate Course
1. Title
2. Abstract
3. Introduction
4. Background / review of literature
5. Organizational context
6. Methodology
7. Results
8. Discussion
Structure of an article
Postgraduate Course
In general
Don’t let yourself be taken in by scientific jargon and complex use of language!! Good articles are written in plain English.
Even authorative journals with a high impact factor contain bad articles and vice versa.
Focus on research question, study design and outcome.
Don’t worry about statistics!
Be critical!! Always ask yourself: does this make sense?
Postgraduate CourseMethodological pitfalls
Randomization errors Bias Confounding Reverse Causation
Postgraduate Course
Bias: distortion of the outcome due to systematic errors caused by the way the study is designed or conducted.
NB: If bias is not taken into account then any conclusions drawn may be wrong!
Bias
Postgraduate Course
1. Selection bias
2. Information (detection) bias
3. Performance bias
4. Exclusion (attrition) bias
5. Publication bias
…
…
30. …..
Forms of bias
Postgraduate CourseSelection bias
Error in the way participants in a study were selected. Means comparison groups differ in measured or unmeasured baseline characteristics.
Types of selection bias:
Sampling bias (selecting only successful departments or individuals who have committed crimes)
Participation bias (self-selection, non-response, etc.)
Postgraduate Course
Distortion of the outcome due to misinterpretation of information or systematic errors in the the measurement of research variables which leads to misclassification.
Information bias can be prevented by the use of standardized measurement instruments, hard outcome measures, validated questionnaires and objective, independent and blinded assessors.
Types of information bias:
Reporting bias (recall bias)
Observer bias (interviewer bias, halo-effect)
Information bias
Postgraduate CourseConfounding
Confounding is the idea that a 3rd variable can distort or confuse (or confound..) a relationship between two other variables.
Let’s say that a college education is strongly positively correlated to successful completion of firefighter training. Is it true that people with less than four years of college don’t make good fire fighters? Or cannot fulfill the requirements of well-trained firefighter?
Confounding could exist if the reading materials use in firefighter training are written at a much higher level than the job actually requires.
Postgraduate CourseConfounding
Postgraduate Course
Reverse causation
Postgraduate Course
Critical appraisal of studies
Appendix: Appraisal Questions
Making sense of evidence
Postgraduate CourseEffect Size
Strength or meaningfulness of relationship between two variables (cause/effect)…several metrics exist: Practical value (average $ saved, weight lost, gain in test
scores) Effect strength (standardized indicate of d difference
between treatments or r strength of relationship across multiple studies)
Judgment required: Small effects with low cost can be of practical value (e.g. can be relatively easy to create identification or ingroup/outgroup effects)
Postgraduate Course
Standard appraisal questions
1. Did the study address a clearly focused issue?
2. Is the sample size justified?
3. Is the design appropriate to the stated aims?
4. Are the measurements likely to be valid and reliable?
5. Are the statistical methods described?
6. Did untoward events occur during the study?
7. Were the basic data adequately described?
8. Do the numbers add up?
9. Was the statistical significance assessed?
10. What do the findings mean?
11. Are important effects overlooked?
12. What implications does the study have for your practice?
Postgraduate Course
Appraisal of a controlled study
1. Did the study address a clearly focused issue?
2. Were subjects randomly allocated to the experimental and control group? If not, could this have introduced bias?
3. Are objective inclusion / exclusion criteria used?
4. Were groups comparable at the start of the study?
5. Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias)
6. Were outcomes assessed blind? If not, could this have introduced bias?
7. Is the size of effect practically relevant?
8. Are the conclusions applicable?
Postgraduate Course
Appraisal of a cohort / panel study
1. Did the study address a clearly focused issue?
2. Was the cohort / panel recruited in an acceptable way? (selection bias)
3. Was the cohort/ panel representative of a defined population?
4. Was a control group used? Should one have been used?
5. Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias)
6. Was the follow up of cases/subjects long enough?
7. Could there be confounding?
8. Is the size of effect practically relevant?
9. Are the conclusions applicable?
Postgraduate Course
1. Did the study address a clearly focused issue?
2. Were the cases and controls defined precisely?
3. Was the selection of cases and controls based on external, objective and validated criteria? (selection bias)
4. Are objective and validated measurement methods used and were they similar in cases and controls? (misclassification bias)
5. Did the study incorporate blinding where feasible? (halo-effect)
6. Was there data-dredging?
7. Could there be confounding?
8. Is the size of effect practically relevant?
9. Are the conclusions applicable?
Appraisal of a case-control study
Postgraduate Course
Assessment of a survey
1. Did the study address a clearly focused issue?
2. Was the sample size justified?
3. Could the way the sample was obtained introduce (selection)bias?
4. Is the sample representative and reliable?
5. Are the measurements (questionnaires) likely to be valid and reliable?
6. Was the statistical significance assessed?
7. Are important effects overlooked?
8. Can the results be generalized?
Postgraduate Course
Don’t confuse: representativeness and reliability The number of respondents has no direct relationship with
representativeness; even a large random sample can be insufficiently
representative. However, the number of respondents does have an
impact on the reliability of the results.
Assessment of a survey
Postgraduate Course
1. Is your organization / division / population so different from those in the study that its results cannot apply?
2. How relevant is the study to what you are seeking to understand or decide?
3. What are your organization’s potential benefits and harms from the intervention?
4. Is the intervention feasible in your setting?5. What are your executive’s (or client’s) concerns,
preferences and expectations for both the outcome you are trying to prevent and the intervention you are offering?
Organization concerns
Organization Concerns
Ask yourself to what extent the evidence is applicable in your situation:
Postgraduate Course
Turning evidence into practice
Evidence based management:closing the gap between research and practice