27:03:2013

Set-theoretic, diagnostic

and Bayesian approaches

to impact evidence

Barbara Befani

Impact, Innovation and Learning: Towards a Research and Practice Agenda for the Future

Brighton, 26-27 March 2013

Outline

Set-theoretic Methods (e.g. QCA) and the new

challenges

– Uncertainty (equifinality)

– Causal contribution (multiple-conjunctural

causality)

– Causal asymmetry (necessity and sufficiency)

Diagnostic and Bayesian approaches

– Uncertainty (can be quantified with probabilities)

– The strength of qualitative evidence can be

measured

Defining & explaining events with Set Theory

In uncertain and emergent contexts, we cannot

define “impact” (or success) precisely

An Impact “space” of possible events, all desirable

– All compatible with given values and goals

Success is likely to look like ANY of a number of

events = a LOGICAL UNION

Success looks more like “being on the right track”

than achieving a specific goal

Being “on the right track” means avoiding a number

of pitfalls / dead ends

Sets can be defined as NEGATION of other sets

The three main operators in set theory are

– Negation, union, intersection

Causal Asymmetry and Contribution Analysis

What is a sufficient causal package (a branch of

blue nodes)

Principal Contributory Cause = INUS, a necessary

part of the (sufficient) combination (each blue node

of a given branch)

In Set Theory terminology, a causal package is an

INTERSECTION of contributory causes

A combination of necessary causes (necessary

within that causal package)

Set Theory provides the mathematical basis for

1. analyzing causal contribution

2. dealing with uncertainty (particularly Fuzzy Sets)

UNION of A and B

A U B

A OR B

A + B

INTERSECTION of A and B

A ∩ B

A AND B

A * B

A A

B B

NEGATION of A

=

~A, NOT A

Conf ID

Cold Chain Integrity (necessary)

Vaccine (V) (intervention)

Health System Quality (HSQ)

Success (S) (reduction in specific morbidity)

No. Cases

A 1 1 1 1 3 B 1 1 0 1 1 C 1 0 1 1 2 D 1 0 0 0 2 E 0 1 1 1/0 0 F 0 1 0 1/0 0 G 0 0 1 1/0 0 H 0 0 0 1/0 0

Why are Causal Combinations important?

Impact is contingent on the context

Finding ONE counterfactual is not enough

QCA helps finding many counterfactuals through systematic cross-

case comparison

Probability and Diagnosis: what the evidence

says

Realm of “unknown knowns”

General problem of the strength / quality of

evidence: how to assess it?

In clinical practice, physicians use tests

– Specificity

• Probability that absence of the disease will

return negative evidence on that test

– Sensitivity

• Probability that presence of the disease will

return positive evidence on that test

– (Positive) Predictive Power

• Probability that positive evidence signals

presence of the disease

When is evidence strong?

When it is sensitive and specific

– Sensitive: P ( Evidence | Impact ) high

– Specific: <= P ( Evidence ) low • false positives are low

– Predictive: P ( Impact | Evidence ) high

– The latter can be calculated with the Bayes

formula P ( I | E ) = P ( I ) * P ( E | I ) / P ( E )

Two important principles of high-quality evidence – of all kinds, quali, quanti, etc.

Evidence is strong when:

– The prior probability of observing positive

evidence P ( E ) is LOW (~specificity)

– The probability of observing positive evidence IF

the intervention was successful / had an impact

P ( E | I ) is HIGH (sensitivity)

Impact ‘I’

Predictive

Evidence

Weak

Evidence

Strong

Evidence

Sensitive

Evidence

Specific

Evidence

Evidence ‘E’ (of ‘I’)

REALITY about success of intervention

Intervention Successful (I)

Intervention Unsuccessful (~I)

EVIDENCE about success of

intervention

Evidence Positive (E) True Positive False Positive

Evidence Negative (~E) False Negative True Negative

Seeking evidence of impact: diagnostic tests

REALITY about success of intervention

Intervention Successful (I)

Intervention UNsuccessful

(~I)

EVIDENCE about success

of intervention

Evidence Positive (E) True Positive False Positive

Positive predictive

value = Σ True Positive Σ Evidence Posi

tive

Evidence Negative (~E) False Negative True Negative

Negative predictive

value = Σ True

Negative Σ Evidence Neg

ative

Sensitivity = Σ True Positive Σ Intervention

Successful

Specificity = Σ True

Negative Σ Intervention UNsuccessful

When articulated ToCs explaining impact are

supported by evidence IT IS STRONG

EVIDENCE OF IMPACT

The prior probability of observing a sophisticated ToC

with several components is LOW, because

The probability of a combination is the product of the

probability of components (a very SMALL #)

P (a, b, c, ... N ) = P (a) * P (b) * P (c) * ... * P (n)

When ToCs with many components are confirmed, it

is strong evidence of impact, because:

– the chances of all components being observed

simultaneously were LOW

• P ( E ) low, specificity high

– If the ToC explaining impact holds true, the

probability of observing evidence of all

components is HIGH

• P ( E | I ) = sensitivity high

Conclusion: let’s use other branches of

mathematics

In the form of SET THEORY or PROBABILITY

THEORY (used differently than in frequentist statistics)

They provide new ways of dealing with uncertainty

SET THEORY helps with:

– Defining success in a more flexible, open and

inclusive way (being “on the right track”)

– Explaining success by defining and identifying

contributory causes rigorously through data

analysis (eg. with QCA)

PROBABILITY THEORY helps with:

– Assessing the strength of evidence in terms of

sensitivity, specificity and predictive value

– Qualitative evidence CAN be strong if a number

of conditions are met – Carefully weigh each piece of evidence as in a court of law,

using conditional and subjective probabilities

References

Befani, B. (2013) “Between Complexity and Rigour:

addressing evaluation challenges with QCA” in

Evaluation (forthcoming)

Befani, B. (2013) “What were the chances?

Diagnostic Tests and Bayesian Tools to Assess the

Strength of Evidence in Impact Evaluation”, CDI

Practice Paper (forthcoming)

John Mayne (2013) “Making Causal Claims”

(presentation to this event)

Bruno Marchal (2013) “Conceptual distinctions:

Complexity and Systems – Making sense of

evaluation of complex programmes” (presentation to

this event)