assessing the effect of visualizations on bayesian reasoning through crowdsourcing

Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing

Jean-Daniel Fekete

Pierre Dragicevic

Luana Micallef

0% - 30% 30% - 60% 60% - 100%

The probability that a woman at age 40 has breast cancer is 1%.

The probability that the disease is detected by a mammography is 80%.

The probability that the test misdetects the disease although the patient does not have it is 9.6%.

If a woman at age 40 is tested as positive, what is the probability that she indeed has breast cancer?

ATTENTION

The probability that a woman at age 40 has breast cancer is 1%.

The probability that the disease is detected by a mammography is 80%.

The probability that the test misdetects the disease although the patient does not have it is 9.6%.

If a woman at age 40 is tested as positive, what is the probability that she indeed has breast cancer?

0% - 30% 30% - 60% 60% - 100%

The probability that a woman at age 40 has breast cancer is 1%.

The probability that the disease is detected by a mammography is 80%.

The probability that the test misdetects the disease although the patient does not have it is 9.6%.

If a woman at age 40 is tested as positive, what is the probability that she indeed has breast cancer?

0% - 30% 30% - 60% 60% - 100%

7.8%P ( Cancer | Positive Mammography ) =

95 doctors out of 100

said the answer is between 70% to 80%

Why the correct answer is so low

P ( cancer | +ve mammography )

=

P ( +ve mammography | cancer)

P (+ve mammography | cancer) + P (+ve mammography | cancer)

Bayes’ Theorem

women without cancer

women with cancer

The probability that a woman at age 40 has breast cancer is 1%.

women without cancer

women with cancer

The probability that the disease is detected by a mammography is 80%.

The probability that the test misdetects the disease although the patient does not have it is 9.6%.

If a woman at age 40 is tested as positive, what is the probability that she indeed has breast cancer?

7.8%

Can such visualizations facilitate Bayesian reasoning

Proposed Visualizations

contingency table

bar-grain boxes Bayesian boxes trees

signal detection curves

Euler diagram frequency grid

+

Euler diagram + glyphs

Previous Studies

Mainly in Psychology

Claim that

Bayesian problem representation impacts comprehension

but …

Inconsistent findings

Most effective Bayesian problem representation? UNCLEAR

Inconsistent and sometimes inappropriate diagram designs

Diagrams do not match textual information

(Sloman et al., 2003)

Area-Proportional Not Area-Proportional

and the subjects …

Specific background usually highly-focused university students

Specific age group

Sometimes,

specific department

carried out as part of their course

so … cannot generalize their findings to

a more diverse population of laypeople

Our Work

Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing

to identify…

- the most effective visualization for the crowd

- whether hybrid visualizations are helpful

- the link between the visualizations and different spatial and numeracy abilities

but…

how appropriate is

Amazon MTurk

Used and evaluated for research and InfoVis

Demographics of workers are well-understood

Captures aspects of real-world problem solving better

- a large diverse population with different backgrounds, education, occupations, age, gender

- workers carry out tasks rapidly but accurately to improve their rating

- reduces experimental biases, as demand characteristics

http://www.eulerdiagrams.org/eulerGlyphs

Experiment

168 workers with MTurk approval rate ≥ 95%

Demographics

25 min

$1

3 Bayesian problemsclassics in Psychology

in natural frequencies format

followed by

objective and subjective numeracy tests

paper folding spatial abilities test

brief questionnaire

Results

We failed to replicate previous findings

subjects’ accuracy was remarkably lower

visualizations exhibited no measurable benefit

even though …

reasonably confident with their answer

overall

12% exact answers

6%

no visualization

14% 11% 11%

21% 7% 14% 21%

no vis V0

V1

V2

V3

V4

V5

V6Answer errors for all three Bayesian problems combined

per visualization type (N = 24 each)

21% exact

6% exact

12% 40% - 80%our study

exact answers

previous studies

Thus

we failed to demonstrate measurable

benefits from visualizations to

facilitate Bayesian reasoning.

Qualitative Feedback

53 out of the 168 subjects

participated

89% ‘somehow’ used the diagram

Most found the diagram very useful

BUT

Various did not understand the diagram

Some doubted the diagram’s credibility

However

must understand and trust the diagram

the answer is in the visualization

women without cancer

women with cancer

The probability that the disease is detected by a mammography is 80%.The probability that the test misdetects the disease although the patient does not have it is 9.6%.If a woman at age 40 is tested as positive, what is the probability that she indeed has breast cancer?

7.8%

How

either

help them understand and relate the diagram to the text

or

force them to get the answer from the diagram

change the text

Another Experiment

480 workers with MTurk approval rate ≥ 95%

did not participate in experiment 1

1 Bayesian problemthe Mammography problem

10 out of every women at age forty who participate in routine screening have breast cancer.

8 of every 10 women with breast cancer will get a positive mammography.

95 out of every 990 women without breast cancer will also get a positive mammography.

classic

10 out of every women at age forty who participate in routine screening have breast cancer (compare the red dots in the diagram below with the total number of dots).

8 of every 10 women with breast cancer will get a positive mammography (compare the red dots that have a black border with the total number of red dots).

95 out of every 990 women without breast cancer will also get a positive mammography (compare the blue dots that have a black border with the total number of blue dots).

with instructions

10 out of every women at age forty who participate in routine screening have breast cancer.

8 of every 10 women with breast cancer will get a positive mammography.

95 out of every 990 women without breast cancer will also get a positive mammography.

without numbers

A small minority of women at age forty who participate in routine screening have breast cancer.

A large proportion of women with breast cancer will get a positive mammography.

A small proportion of women without breast cancer will also get a positive mammography.

without numbers

10 out of every women at age forty who participate in routine screening have breast cancer.

8 of every 10 women with breast cancer will get a positive mammography.

95 out of every 990 women without breast cancer will also get a positive mammography.

classic

Results

The Most Effective Textual Representation

A small minority of women at age forty who participate in routine screening have breast cancer.

A large proportion of women with breast cancer will get a positive mammography.

A small proportion of women without breast cancer will also get a positive mammography.

without numbers

exact answers

+no visualization

3.3% exact answers

classic text

+5% exact answers

classic text

5% exact answers

+

text with instructions

1 exact answer (N=120)

+

text without numbers

Answer errors for the Mammography Bayesian problemper presentation type (N = 120 each)

classic + no vis

classic + vis

with instructions + vis

without numbers + vis

Conclusion

Using crowdsourcing, we assessed

6 visualizations and text alone for

3 classic Bayesian problems

We failed to replicate previous findings

subjects’ accuracy was remarkably lower

visualizations exhibited no measurable benefit

A follow-up experiment confirmed …

simply adding a visualization to a textual Bayesian

problem does not help

diagrams can help but numerical values have to be removed and the text should be used to merely set the scene

We need …

novel visualization that holistically combine

text and visualization and promote the use of estimation rather than calculation

more studies in settings that better capture real-life rapid decision making

To …

facilitate reasoning of statistical information

for both layman and professionals

ThanksJean-Daniel

FeketePierre

DragicevicLuana Micallef

error = log10answergiven

answerexpected