Postgraduate Course

6. Evidence based management:

What is the best available evidence?

Postgraduate Course

5-step approach

EBMgt is a 5-step approach

1. Formulate an answerable question (PICOC)

2. Search for the best available evidence

3. Critically appraise the quality of the found

evidence

4. Integrate the evidence with managerial

expertise and organizational concerns and apply

5. Monitor and evaluate the results

Postgraduate Course

Why are disciplines as such as

psychoanalysis, astrology and

parapsychology widely regarded as

pseudo-science?

Intermezzo

Postgraduate Course

“It is easy to obtain evidence in favor of virtually any theory,

but such ‘corroboration’ should count scientifically only if it

is the positive result of a genuinely ‘risky’ prediction, which

might conceivably have been false.

… A theory is scientific only if it is refutable

by a conceivable event. Every genuine test

of a scientific theory, then, is logically an

attempt to refute or to falsify it.”

Falsifiability

Carl Popper

Postgraduate Course

“Inspect every piece of pseudoscience and you will find a

security blanket, a thumb to suck, a skirt to hold.

What have we to offer in exchange? Uncertainty!

Insecurity!”

Falsifiability

Isaac Asimov

Postgraduate Course

Research designs

What is the BEST car?

Postgraduate Course

Which design for which question?

Research designs

Postgraduate Course

Explanation

Which design for which question?

Postgraduate Course

Best research design?

Postgraduate Course

Best available?

Postgraduate Course

The best available evidence =

Studies with the highest internal validity

Studies with the highest external validity

Postgraduate Course

1. Best available evidence:

internal validity

Postgraduate Course

internal validity = indicates to what extent the

results of the research may be biased and is thus

a comment on the degree to which alternative

explanations for the outcome found are possible

(confounding).

Internal validity

Postgraduate Course

Postgraduate Course

Three criteria:

1. the "cause" and the "effect" are related

2. the "cause" precedes the "effect" in time

3. there are no plausible alternative explanations for the

observed effect

When do we know there is causal relation?

Causality

Postgraduate Course

Considerations for research:

Causality

1. Are the "cause" and the "effect” related: effect size

2. Does the "cause" precedes the "effect" in time:

before and after measurement

3. Are there no plausible alternative explanations for the

observed effect: randomization, control group

INTERNAL VALIDITY

Postgraduate Course

internal validity = indicates to what extent the

results of the research may be biased and is

thus a comment on the degree to which

alternative explanations for the outcome found

are possible (confounding).

Internal validity

Postgraduate Course

Bias

Confounding

Reverse Causation

Methodological pitfalls

Postgraduate Course

Bias: distortion of the outcome due to

systematic errors caused by the way the

study is designed or conducted.

NB: If bias is not taken into account then any

conclusions drawn may be wrong!

Bias

Postgraduate Course

1. Selection bias

2. Information (detection) bias

3. Performance bias

4. Exclusion (attrition) bias

5. Publication bias

…

…

30. …..

Forms of bias

Postgraduate Course

Error in the way participants in a study were selected.

Because of this comparison groups differ in measured or

unmeasured baseline characteristics.

Selection bias

Postgraduate Course

Distortion of the outcome due to misinterpretation of information

or systematic errors in the the measurement of research

variables which leads to misclassification.

Information bias can be prevented by the use of standardized

measurement instruments, hard outcome measures, validated

questionnaires and objective, independent and blinded

assessors.

Types of information bias:

Reporting bias (recall bias)

Observer bias (interviewer bias, halo-effect)

Information bias

Postgraduate Course

Confounding is the idea that a 3rd variable can distort or

confuse (or confound..) a relationship between two other

variables. For instance, when factor X causes disease Y,

that relationship could be confounded by factor C that is

associated with both factor X and disease Y. C would be

an alternative explanation for the relationship observed

between X and Y.

Confounding

Postgraduate Course

What are the confounders?

1. Shoe size & quality of handwriting

2. Body length & body weight

3. Number of storks & birth rate

4. Smoking youngsters & better lung function

Postgraduate Course

Confounding

Postgraduate Course

Correlation does not equal causation!

Postgraduate Course

http://kill-or-cure.heroku.com/

Postgraduate Course

Reverse causation

Postgraduate Course

?

Successful companies

Charismatic leaders

Reverse causation

Postgraduate Course

Cause and effect can be established

only through the proper research

design: no amount of statistical hand

waving can turn correlations into

conclusions about causation !!!

Internal validity

Postgraduate Course

Levels of internal validity

Postgraduate Course

Levels of internal validity

It is shown that …

It is likely that …

Experts are of the

opinion that …

There are signs

that …

Postgraduate Course

The levels of internal validity can only be used to determine which type of research is the best method to assess the validity of the cause-and-effect relationship that might exist between an intervention (or moderator) and its outcomes. In this respect, cross-sectional studies and case-studies have the ‘weakest’ design. This of course doesn’t mean that cross-sectional studies and case-studies have a weak design overall. After all, different types of research questions require different types of research designs. A case study for instance is clearly a strong design for assessing why or in which way an effect has occurred, but obviously not the most suitable design for assessing the strength of a possible cause-and-effect relationship.

Keep in mind!

Postgraduate Course

But … sometimes observational

studies are as good as RCT’s

Internal validity

When the size of effect is very large (swamps

the bias)

Postgraduate Course

These treatments have not been tested in RCTs: are they supported by poor evidence?

Internal validity

Heimlich manoeuvre Dehydration: drinking water

Cardiac arrest: AED

Postgraduate Course

2. Best available evidence:

external validity

Postgraduate Course

Ecological validity: Is your organization so different from those in the study that its results may be difficult to apply?

Population validity: Is your population so different from those in the study that its results may be difficult to apply?

External validity: generalizability

Always ask yourself to what extent the evidence is generalizable to your situation:

Postgraduate Course

Generalizability

Same Population?

Same Intervention?

Same Comparison?

Same Outcome?

Same Context?

Postgraduate Course

Generalizability

Keep in mind:

What works in one narrowly defined setting

might not work in another,

but some psychological principles

are generalizable to all human beings.

Postgraduate Course

Internal vs external validity

All research designs are flawed – though each is flawed

differently. For instance, research designs with a high internal

validity, such as controlled studies, may be less suited to

generalization, which restricts their practical usability. Sample

surveys and field research, on the other hand, have lower

internal validity, but can sometimes be more useful for

management practice. So there is always a trade off between

internal validity (precision in control and measurements of

variables) and external validity (generalizability with respect

to populations, setting and context).

Postgraduate Course

Best available evidence?

internal validity

often high

internal validity

often low

external validity

often low

external validity

often …?

external validity

sometimes high