determinants of numerical versus verbal probabilities

Acta Psychologica 83 (1993) 33-51

North-Holland

33

Determinants of numerical versus verbal probabilities *

Michel Gonzalez and Cheryl Frenck-Mestre University of Provence and C.N.R.S., Air-en-Prouence, France

Accepted May 1992

The effects of various types of information upon numerical and verbal probability judgements were evaluated in six problems. Subjects judged the occurrence of a target event either by

assessing the adequacy of the expressions probable and improbable or by assessing the numerical

probability of the event. Different types of information were manipulated for a given problem:

the weight of the target in relation to the total weight of all alternatives (global weight) or in

relation to that of each alternative independently (local weights), the change over time of the target’s weight (trend), and the stated base rate.

The results revealed that particular types of information - for example local weight - had

different effects upon the two response modes. A tentative interpretation of this verbal-numeri-

cal discordance is proposed, suggesting different strengths of use of the same information

according to the response modes.

The evaluation of the probability of occurrence of an event can be expressed either numerically or verbally. Suppose that on the eve of a tennis match between two finalists, J. and K., a fan comments: There is a 5% chance that J. will win the match. A second fan states: It is improbable that J. will win the match. Do these two forecasts express the same evaluation of the outcome of the match?

Two experts wish to alert the public to the risks of a nuclear accident due to the use of a new technology. The first claims that there is a 5% risk over a two-year period, whereas the second claims

Correspondence to: Gonzalez, Centre de Recherche en Psychologie Cognitive, Universite de

Provence, 29 avenue Robert Schuman, 13621 Aix-en-Provence CCdex 1, France. Fax: (33) 42 20 59 05, E-mail: [email protected] * This research was supported by the C.N.R.S, the University of Provence, and by the DCN

contract C 87 48 813 517. We wish to thank Helmut Jungermann and anonymous referees for helpful comments on an earlier draft of this paper.

OOOl-6918/93/$06.00 0 1993 - Elsevier Science Publishers B.V. All rights reserved

the risk to be probable. Could the two experts nonetheless agree with each other upon the estimation of risk? Could probable and 5% express a common evaluation of the situation?

If, as suggested here, two events having the probability of occurrence of 5% can be judged as probable or improbable depending upon the context, then one can question whether numbers encompass all of the information used in probability judgements.

In the present article we compare the effects of various characteristics of probabilistic problems on the numerical and verbal evaluations of probability. The characteristics emplo;ed are intended to bring out either the concordance or the discordance of these two means of evaluation. We speak of r~erbal-numerical concordance when the results allow us to conclude that underlying the two response modes is the same evaluation of a probability, and of cerbal-numerical discordance when we are led to conclude that the two response modes result in different evaluations of probability.

It is noteworthy that almost all research on probability judgements has focussed on numerical as opposed to verbal evaluations. This can be explained by the common opinion that numbers provide a privi- leged means of evaluating probabilities, whereas verbal terms are considered imprecise and ambiguous. A case in point is the recom- mendation to express probabilities numerically in the field of medicine (Katz 1984; Nakao and Axelrod 1983; Reiss 1984; Robertson 1983).

There are nonetheless studies which have attempted to attribute numerical meaning to a set of verbal probability expressions, such as probable, likely, possible, improbable, and doubtful. Lichtenstein and Newman (19671, Beyth-Marom (19821, Budescu and Wallsten (19851, and Brun and Teigen (19881 asked their subjects for the number which best transcribed the probability of an event expressed by a verbal term. These authors noted that the reliability and consistency of these numbers were quite weak. Verbal terms have also been numerically evaluated in the context of political statements (Beyth- Marom 1982) and medical diagnoses (Bryant and Norman 1980; Brun and Teigen 1988). Inter-subject consistency of estimations is even less in these cases than for isolated expressions.

The above mentioned results give some indication that verbal terms are not as precise as numbers. However, the claim that they are ambiguous is not necessarily supported. Some studies suggest that verbal probability terms can be translated precisely into a system of

M. Gonzalez, C. Frenck-Mestre / Numerical IX. [verbal probabilities 35

numerical probabilities. In light of Zadeh’s (1975a, 1975b) theoretical position that a verbal term could be represented by a fuzzy subset of the probability interval, Wallsten et al. (1986a), Rapoport et al. (1987), and Zwick et al. (1988) formulated and successfully applied methods of measuring membership functions of verbal terms. The measurement of the verbal meaning of numerical probability terms by fuzzy sets has also been advocated by Reagan et al. (1989) and by Zimmer (1984).

This approach to defining the meaning of verbal terms has the advantage of allowing to delimit the imprecision of these terms. It also furnishes a tool for the identification of verbal-numerical concordance. Indeed, one can compare the meanings of probability evaluations made using verbal and numerical response modes, given that the two modes refer to the same support (i.e. probability interval) through their membership function. Hence, one can speak of concordance when the membership functions of two responses are compatible, independently of the situation.

Verbal-numerical concordance can be demonstrated by comparing the verbal and numerical responses given in the same situation. However, few studies present this type of data (Budescu et al. 1988: judgement section; Rapoport et al. 1990; Teigen 1988: study 4; Zwick 1988; Zwick and Wallsten 1989). Budescu et al. (1988) asked their subjects to make a verbal or numerical judgement of the probability that a dart, pointed at the center of a circle which was radially divided into a shaded and an unshaded section, would land on the shaded section of the circle. The results led the authors to conclude that the two response modes were monotonic and reliable. Zwick (1988) and Zwick and Wallsten (1989) gave subjects the percentages of different age groups represented in a population, and required them to give a numerical or verbal response to the question ‘What are the chances that a randomly selected person will be very old?‘. However, the results presented were centered on the comparison of two alternative models for each response mode, and do not enable one to directly evaluate the concordance between the two response modes themselves.

In conclusion, although the means to analyse verbal-numerical discordance exist, there is little existing data that indicate the situations in which it occurs.

Zimmer (1983) defended the idea that the two modes of expressing probabilities - verbal and numerical - are based upon distinct use of

36 M. Gonzalez, C. Frenck-Mestre / Numericul vs. ~vrbal probabilities

probabilistic information, going so far as to argue that verbal evaluations are in fact more efficient. However, at present there are only a few results which suggest the presence of discordance, and only in very specific situations.

Wallsten et al. (1986b) presented a statement which described an event such as ‘A couple will have at least one child after being married for [version A: one year; version B: five years]‘. Results showed that the probability of the event was higher in version B than in version A. Following this, the word probably was inserted in each of the two versions, and subjects were asked to estimate the value another person had in mind when saying probably in one or the other version of the modified sentence. Version B led subjects to give a mean value 0.14 higher than version A. Hence, it can be said that the two versions evoked distinct numerical evaluations, but identical verbal evaluations, indicating the presence of verbal-numerical discordance.

A series of studies conducted by Teigen (1988b) also suggests the presence of verbal-numerical discordance. In his study 1, the expression probable was applied to events whose numerical probability was judged to be below 50%. Various other studies have revealed, however, that the best numerical translation of the term probable lies between 70 and 80%. Hence, two events may be classified as probable, while the probability of the one is judged to be below 50% and that of the other above 70%.

Yet another example of discordance is provided by a problem taken from Teigen’s study 2:

Tom P. has just handed in his application for a vacant position. [Version A]. . . there are four other applicants and.. . all 5 candi-

dates can be regarded as having equal qualifications for the job. [Version B] . . . there is one other applicant, who is clearly more

qualified. . . Tom’s only possibility is that this rival will not accept the offer. We know that such withdrawals occur in about one out of five cases. (pp. 161-162)

As concerns Tom’s chances to obtain the position, Teigen observed that doubtful was judged to be more appropriate in version B than in version A. If one admits that the numerical evaluation of Tom’s chances in both versions is 20%. then it can be said that two events

M. Gonzalez, C. Frenck-Mestre / Numerical vs. verbal probabilities 37

sharing the same numerical probability differ in regards to how doubtful they appear.

In the present study we seek to determine possible sources of verbal-numerical discordance. To this aim, verbal and numerical evaluations of the probability of an event are compared in various problems in which different types of information concerning the event are presented.

Consider the following problems in which one is required to evaluate the chances that a person in a certain risk category for disease D would actually contract the disease:

Problem 1: 15 cases of D. have been diagnosed out of 300 subjects presenting the risk factors. Problem 2: D. affects 1% of the population on the average. This rate is 5 times higher in subjects presenting the risk factors.

In both problems, the target event has a rate of occurrence of 5%. If indeed the rate is the sole information taken into account, the estimated probability of its occurrence should be the same for both problems. However, problem 2 gives more information. When a risk factor is present, the rate of D, 5%, is 5 times higher than that found in the general population. This extra information indicates that the presence of a risk factor greatly increases the probability of contract- ing D. Insofar as verbal probability expressions are more likely to be affected by this type of information than are numerical expressions, verbal-numerical discordance can be expected to occur. Hence, whereas subjects might give a numerical response of 5% for both problems, they might choose the verbal response probable more often for problem 2 than for problem 1.

Method

Subjects

A total of 304 undergraduate psychology students participated in the experiment in groups of approximately 30.

Materials and design

Six probability problems were developed. There were four versions of each problem, as defined below, and all problems were evaluated via two response modes

38 M. Gonzalez, C. Frenck-Mesire / Numerical vs. rwbal probabilities

(verbal vs. numerical). A given subject saw only one version of each of the six problems and responded to all problems using the same response mode. Each problem was thus presented in a 4 X 2 between-subject design with 38 subjects per condition.

Procedure

Subjects were given an experimental booklet containing the set of six problems, each presented separately on an individual sheet (see Appendix). The set of problems was preceded by a page of instructions and a practice problem.

For all problems, subjects were required to evaluate the probability of an event by using one of the two response modes. In the lserbal response mode, subjects were asked to indicate the adequacy of the terms probable and improbable by rating both terms independently on a 7-point scale ranging from inadequate (1) to adequate (7). In the numerical response mode, subjects did not in fact provide numerical estimations but rather selected one of seven equilarge segments dividing the response scale. The endpoints of the scale were labelled as 0% and lOO%, with 50% as the midpoint, however the segments themselves were not numerically labelled (see Appendix).

Scoring and analysis of results

The data obtained in the two response modes (verbal vs. numerical) were converted into a common score, described below, ranging from 0 to 1. This rendered possible a global analysis of data from the two response modes. As such, it was possible to determine whether response mode interacted with the factors manipulated in a given problem, i.e. whether there was evidence of verbal-numerical discordance.

In the verbal response mode, subjects rated the adequacy (from 1 to 7) of the term probable on the one hand, and improbable on the other. However, preliminary analyses of variance revealed that the independent variables had symmetrical effects on the ratings of the two terms. In fact, the correlation between the mean adequacy of the two terms in each version of a problem was r = -0.96, all problems confounded. Given this, a single score was derived, which corresponded to the difference, d,

between the adequacy of probable, a(p), and that of improbable, a(i): d = a(p) - u(i).

Let us call d the strength of probable. The strength of probable (ranging from ~ 6 to + 6) was converted into a verbal score equal to (d + 6)/12. This rerbul score ranges from 0 to 1. A verbal score of 0 means that probable had a minimum adequacy (1) and improbable a maximum adequacy (7). Conversely, a verbal score of 1 means that probable had a maximum adequacy (7) and improbable a minimum adequacy (1).

In the numerical response mode, the responses were ranked from r = 1 to 7 in accordance with the segment selected, and then converted into a numerical score equal to (r - 1)/6. This score, ranging from 0 to 1, is not exactly a probability estimation. For example, a score of 0.17 does not correspond to a response of 17% but rather to the selection of the second segment on the scale, which corresponded to a lower boundary of 14% and an upper boundary of 28%. The numerical score is thus an approximate evaluation of probability.

M. Gonzalez, C. Frenck-Mestre / Numerical LX oerbal probabilities 39

For each of the two response modes, we thus determined a scale value ranging from 0 to 1. This does not suppose that the scale values for the two response modes can be directly compared. Indeed they should not be since the two scales are obviously of different nature. One can, however, legitimately compare the size of the effect that a given factor had on each of these scales respectively since both scales are bounded and have the same range (from 0 to 1 in the present case). The analyses reported throughout the present paper sought to determine whether a given factor had differential effects on subjects’ verbal and numerical probability estimations respectively by comparing the magnitude of the effects on the respective scales. We thus examined the simple effect of a factor on the two levels of response mode - verbal and numerical - on the one hand, and whether the manipulated factor interacted with response mode on the other.

1. Weight and trend

Herein we examined the effect of modifying trend, and that of modifying the end state on subjects’ verbal and numerical evaluations of the probability of occurrence of an event. We expected subjects’ numerical evaluations to be determined more by the end state than by the trend, since the most obvious solution is to base one’s response on the end state, independently of how it came about. For verbal evaluations, we expected the trend to determine subjects’ evaluations as well as the end state. When deciding the probability of an outcome, while one would most likely not ignore the end state, the evaluation of the outcome may well be determined by other parame- ters, such, for example, as the way in which the end state occurred. This notion is supported by results obtained by Teigen (1988b).

Materials

Subjects were presented with the following common initial scenario and final question:

A company, seeking to fill a vacant position, published an aduertisement in a local newspaper. Jane applied for the position. What do you think the chances are that Jane will obtain the position?

Four versions of the problem were defined by the combination of the factors Trend (positive versus negative) and Weight (1: 3 versus 1: 7). The trend was determined by an event which modified the total number of initial candidates (doubled in the negative trend and halved in the positive trend). The weight was determined by the final number of rival candidates CR), which was equal to either 7 or 3. These factors were manipulated in the paragraph that completed the scenario:

Negative Trend: She is in competition with [(R - 1)/2] candidates. Before making a

decision, the company published a second job announcement. [(R + 1)/2] new

40 M. Gonzalez, C. Frenck-Mestre / Numerical US. L?erbal probabilities

Table 1

Verbal and numerical scores as a function of the trend and the weight conditions of the

candidate problem.

Weight Verbal score Numerical score

Negative Positive Negative Positive

trend trend trend trend

M LSD) M LSD) M (SD) M LSD)

1:7 0.39 (0.19) 0.51 (0.23) 0.18 (0.17) 0.21 (0.26) 1:3 0.46 (0.20) 0.53 (0.18) 0.25 (0.16) 0.35 (0.22)

Note: 38 subjects per condition. Minimum score = 0. Maximum score = 1.

candidates applied for the position. Jane is still a candidate. She is in competition

with [RI other candidates.

Positive Trend: She is in competition with [2R + l] candidates. Following an initial selection, [R + l] candidates were eliminated. Jane is still a candidate. She is in competition with [R] other candidates.

Results

A 2 X 2 X 2 (Trend X Weight X Response mode) analysis of variance was per- formed. The analysis of simple effects revealed that Trend did not differentially affect verbal and numerical scores, whereas Weight did (see table 1). Concerning Trend, both verbal and numerical scores were higher when the trend was positive than when it was negative (F(1, 296) = 8.12, p < 0.01; and F(1, 296) = 4.22, p < 0.05, in the verbal and numerical response mode respectively). Concerning Weight, numerical scores were higher when Jane was faced in the end with 3 competitors than when she was faced with 7 competitors (F(1, 296) = 8.89, p < 0.01) whereas verbal scores were not significantly different (F(1, 296) = 1.94). The interaction between Response mode and Weight was not reliable (F(1, 296) = 1.26).

Discussion

The results do not provide any clear evidence of verbal-numerical discordance. The trend had the same effect on both verbal and numerical evaluations. A target’s probability score was higher when the target was situated in a positive than when it was situated in a negative trend. While the weight had a significant effect only on numerical evaluations, the absence of an interaction precludes the conclusion that this information was used differently in the two response modes.

M. Gonzalez, C. Frenck-Mestre / Numerical US. verbal probabilities 41

2. Local and global weights

Two isomorphic problems examined the influence of varying the weight of alternatives, either considered independently (Local weight) or as a whole (Global weight), on subjects’ verbal and numerical evaluations of a target.

Materials

The two problems made use of natural categories, known to subjects. The Diplo- mat problem used geographic areas, with major cities as exemplars. Subjects were presented with the following information:

Henri is a diplomat. He has recently learned that he will be assigned to a new embassy. The location of the assignment will be decided as a function of [6 or 91 possible embassies that haue vacant positions.

Vacancies in different cities were clustered by geographical areas and subjects were asked about the chances that Henri be sent to the United States. In all versions, the United States had vacancies in 3 different cities (Washington DC, Los Angeles, New York City). Global weight was varied by adding either the same number, or twice as many other embassy positions outside of the United States, i.e. either 3 (1: 1 Global weight) or 6 (1: 2 Global weight).

Local weight was varied by listing possible embassy sites outside of the United States either grouped by three in one area (1: 1 Local weight) or all located in different areas (3: 1 Local weight). For example, in the 1 : 1 condition 3 vacancies in Africa (Brazzaville, Abidjan, Dakarl were listed in addition to the 3 vacancies in the United States, and in the 3 : 1 condition 1 vacancy in Africa (Brazzaville), 1 vacancy in Western Europe (Rome), and 1 vacancy in Australia (Sydney) were additionally listed (see Appendix).

In the Award problem the following scenario was presented:

The H.B. Foundation attributes one award yearly to a meritorious scientific endeavor. A committee is designated to evaluate the various projects submitted.

This was followed by a list of projects grouped by science. Subjects were asked about the chances that the award be attributed to Language Sciences this year. Language Sciences had two levels of Global weight: 1: 4 (two projects in Language Sciences against eight projects in other sciences) and 1: 8 (one project in Language Sciences against eight projects in other sciences).

Local weight was varied by listing the rival sciences either as having the same number of projects as Language Sciences (1: 1) or as having four more projects than Language Sciences (1: 4). In the 1: 4 Global weight condition for example, it was said that the committee examined ten scientific projects, two in Language Sciences and either eight projects in Biology (1 :4 Local weight) or two projects in each of four sciences, Biology, Physics, Economics, and Mathematics (1: 1 Local weight).

42 M. Gonzalez, C. Frenck-Mestre / Numrricul IX llerbal probubilitim

Results

Global weight Significant simple effects of Global weight were found in the Diplomat problem

for the numerical and verbal response modes (F(1, 296) = 48.34, p < 0.001; F(1, 296) = 15.17, p < 0.001, respectively) as well as in the Award problem (F(1, 296) = 5.39,

p < 0.05; F(1, 296) = 8.63, p < 0.01, in the numerical and verbal response mode, respectively). Stated otherwise, subjects gave higher evaluations of Henri’s chances of being sent to the United States when the global weight was the same for the target and rivals (1 : 1) than when weighted in favor of the rivals (1 : 2). Similarly, the chances that the award be given to Language Sciences were evaluated as higher when the global weight of the target was 1: 4 as compared to 1: 8 (see table 2).

In the Diplomat problem there was an interaction between the Response mode and Global weight (F(1, 296) = 4.67, p < 0.05). The effect of varying the global weight was greater on subjects’ numerical than verbal evaluations. This interaction was not apparent in the results from the Award problem (F < 1).

Local weight The simple effect of Local weight on numerical scores was not significant (F < 1,

and F(1, 296) = 2.11 for the Diplomat and Award problem, respectively). For verbal scores, however, Local weight had a significant effect in both the Diplomat and the Award problem (F(1, 296) = 19.34, p < 0.001; and F(1, 296) = 7.01, p < 0.01). In the Diplomat problem, subjects gave higher verbal scores when the target’s local weight was greater. In the Award problem, subjects gave lower verbal scores when the target’s local weight was less.

Table 2

Verbal and numerical scores as a function of the global weight and the local weight conditions.

Global Award problem

weight Verbal score Numerical score

1:4 Local 1: 1 Local 1 : 4 Local I : 1 Local

weight weight weight weight

M (SD) M (SD) M (SD) M (SD)

1:X 0.22 (0.20) 0.3 1 (0.21) 0.14 (0.17) 0.10 (0.15) I:4 0.32 (0.21) 0.39 (0.20) 0.21 (0.22) 0.16 (0.11)

Diplomat problem

1: 1 Local 3: 1 Local 1: 1 Local 3: 1 Local

weight weight weight weight

1:2 0.43 (0.18) 03 1 (0.20) 0.28 (0.12) 0.31 (0.19)

1:l 0.50 (0.14) 0.65 (0.16) 0.49 (0.12) 0.46 (0.16)

Note: 38 subjects per condition. Minimum score = 0. Maximum score = I.

M. Gonzalez, C. Frenck-Mestre / Numerical cs. cerbal probabilities 43

The differential effect of local weight according to the response mode was shown by a Local weight x Response mode interaction (F(1, 296) = 9.31, p < 0.01; and F(1, 296) = 8.40, p < 0.01, for the Diplomat and the Award problem, respectively).

Discussion

Only the global weight of the target had a noticeable effect on subjects’ numerical evaluations, whereas both the global weight and the local weight of the target had effects on subjects’ verbal evaluations. The significant Local weight X Response mode interaction, observed in both problems, provides evidence for verbal-numerical discordance. This verbal-numerical discordance can be explained by the subjects’ weighting local weight more heavily when giving verbal than when giving numerical responses.

3. Base rate

Three problems provided subjects with statistical information and required answer- ing a question about the occurrence of a particular state in a member of a target community. The variable of interest was the effect of a given base rate upon verbal and numerical responses.

In previous studies, a systematic deviation of responses from the probability prescribed by Bayes’ rule has been demonstrated (see notably Hammerton 1973; Kahneman and Tversky 1973; Bar-Hillel 1980, 1983). In particular, it seems that base rates are not sufficiently taken into account.

In the present problems, the use of base rates was facilitated by formulating the problems such that it was apparent to subjects that the answer could be derived by weighting the base rate. The question of interest was whether the probability evaluation integrated this base rate, according to whether the evaluation was verbal or numerical.

Epidemic problem

The Epidemic problem was introduced by indicating that Melyatitis is an infectious disease and posed the question about the risks that an inhabitant of Bankina be infected by Melyatitis. Below are given two versions of the problem.

Version A (without base rate): Epidemiological statistics reuealed that the rate of Melyatitis was IO times higher in Bankina than in the world population.

Version B (with base rate): Epidemiological statistics recealed that the rate of Melyatitis in the world population was 5 in 100,000, and 10 times higher than this in Bankina.

In version A, the base rate of Melyatitis is unspecified and the rate of Melyatitis in Bankina cannot be calculated. In version B, the rate of Melyatitis in the target population is 10 times higher than 5 in 100,000, that is 5 in 10,000.

44 M. Gonzalez, C. Frenck-Mestre / Numerical LX l,erbal probabilities

Table 3

Verbal and numerical scores as a function of the base rate (BR) condition and the ratio level of

the epidemic problem.

Ratio Verbal score ” Numerical score

Without BR With BR Without BR With BR

M (SD) M (SD) M (SD) M (SD)

10 times

higher 10 times

lower

0.71 (0.18) 0.52 (0.27) 0.47 (0.36) ’ 0.22 (0.32) h

0.32 (0.23) 0.29 (0.26) 0.13 (0.19) h 0.05 (0.12) c

Note; Minimum score = 0. Maximum score = 1

” 38 subjects per condition. ’ n = 36. ’ n = 37.

In two other versions of the problem, the information 10 times lower was substituted for 10 times higher. The relevance of the base rate is readily apparent, since the rate of occurrence of the target event can only be determined in relation to the base rate.

Results

The analysis of simple effects revealed that both verbal and numerical scores were lower in the version where the base rate was given than in the version where it was absent (F(1, 290) = 9.83, p < 0.01; and F(1, 290) = 17.93, p < 0.001 respectively). The interaction between base rate and response mode was not significant (F < 1). In the versions where the information 10 times lower was substituted for 10 times higher,

giving the base rate did not have a significant effect on either verbal or numerical scores (see table 3).

It should be noted that in the conditions where base rate information was absent, six subjects (three per condition) actually refused to give a numerical response, arguing that the information provided was not sufficient. Despite this, no subject refused to give a verbal evaluation in the conditions where base rate information was absent. One could speculate that, whereas subjects were aware of the impossibility of calculating the rate of disease correctly in both conditions, they were more willing to give a verbal evaluation than a numerical evaluation since the information provided by the ratio could be translated directly into a verbal expression without having to represent its value on the probability interval.

Mortality problem

This problem examined the effect of the presence versus absence of a base rate in another context.

The Mortality problem was introduced by indicating that Hecastosis is an acute

illness which generally results in death. It required subjects to evaluate the risks that an Oukoustanian man, aged 35 to 45, die of Hecastosis. Below are given two versions of the problem.

M. Gonzalez, C. Frenck-Mestre / Numerical us. verbal probabilities 45

Table 4 Verbal and numerical scores as a function of the base rate (BR) condition of the mortality problem.

Verbal score Numerical score

M LSD) M (SD)

Without BR 0.59 (0.17) 0.49 (0.12) With BR 0.55 (0.29) 0.31 (0.23)

Note: 38 subjects per condition. Minimum score = 0. Maximum score = 1.

Version A (without base rate): In Oukoustan, Hecastosis is the cause of 50% of deaths of men aged 35 to 45.

Version B (with base rate): In Oukoustan, Hecastosis is the cause of 50% of deaths of men aged 35 to 45. In Oukoustan. the death rate of men aged 35 to 45 is 2%.

In version B, one can deduce that death rate due to Hecastosis is l%, i.e. 50% of the 2% overall death rate.

Two other versions of the problem duplicated versions A and B, adding that the rate of death due to Hecastosis was 10 times higher in Oukoustanians than in the general population.

Results Analyses of simple effects revealed that base rate significantly affected only the

numerical evaluations (F(1, 296) = 12.37, p < 0.001; and F < 1, for the numerical and verbal scores respectively). When the base (death) rate was given, the mean numerical estimation of the probability of mortality due to Hecastosis was significantly lower than when this information was absent (see table 4). The different effect of base rate on the two responses modes can be seen in the significant Base rate x Response mode interaction (F(1, 296) = 3.77, p = 0.05). The same pattern of results was observed for the two versions where additional information was provided.

The fact that verbal evaluations did not significantly differ from each other in the versions of the problem suggests that the 50% death rate due to Hecastosis (present in all versions) was in fact the information given the greatest weight in verbal evaluations. On the other hand, results indicate that base rate information of 2% was used in numerical evaluations. We thus have here a source of verbal-numerical discordance.

Health-risk problem

The last problem concerned the predictioe accuracy P(A IB) of a risk factor B concerning a disease A. The two versions of the problem differed according to the base rate value P(A). The following information was given:

46 M. Gonzalez, C. Frenck-Mestre / Numerical OS. t,erbal probabilities

Table 5 Health risk problem. Verbal and numerical scores as a function of the base rate level.

Base rate Verbal score Numerical score level M (SD) M LSD)

9%0 0.54 (0.28) 0.25 (0.34) 45%r 0.63 (0.21) 0.42 (0.27)

Note: 38 subjects per condition. Minimum score = 0. Maximum score = I.

A medical surrley was conducted to determine the risk factors linked to the disease Amenoristosis (a chronic affliction). The following information is known concerning Bulimia (compulsir~e eating). In the general population, the risk of hating Amenocistosis is [ 9 / 1000 or 45 / 1000 according to the version]. Cases of Bulimia are observed 10 times more often in people hauing Amenotaistosis than in the general population. What do you think the risks are that a subject presenting Bulimia hat,e Amenotisto-

sis?

The base rate P(A) was either 0.009 or 0.045. The ratio P(B ]A)/P(B) was always equal to 10 : 1. According to Baycs’ formula, P(A I B) = P(A) X (P(B ]A)/P(B)), otherwise stated to 0.09 or 0.45 according to the version.

Results Modifying the base rate had a significant effect only on numerical scores, as

revealed by the analysis of simple effects (F(1, 296) = 1.80; and Ftl, 296) = 6.19, p < 0.01 for verbal and numerical scores respectively). When the base rate went from 9%~ to 45%0, the numerical scores increased by an average of 0.17 (see table 5). However, no significant Base rate x Response mode interaction was observed (F < 1).

Discussion

In the Health-risk problem, the value of a base rate had a significant effect on numerical evaluations. This result differs from those observed for typical Bayesian problems where judgements of posterior probabilities were sensitive to the likelihood ratio, but not to the base rate (cf. Bar-Hillel 1980).

Moreover, the presence of a base rate had a significant effect on numerical evaluations of probability in all problems, and a significant Response mode X Base rate interaction was obtained in the Mortality problem. The origin of this verbal- numerical discordance is not clear, and cannot be explained by the subjects’ systemat- ically ignoring the base rate when making verbal evaluations. In fact, verbal evaluations were affected by the presence of a base rate in the Epidemic problem.

M. Gonzalez, C. Frenck-Mestre / Numerical LX Lterbal probabilities 47

General discussion

Verbal-numerical discordance was shown for probability responses in three of the six problems presented. We note first that the local weight of the target was used in verbal evaluations of two problems, whereas there is no evidence that it was used in numerical evaluations (part 2). Second, giving a base rate (part 3) had different effects on numerical and verbal evaluations.

The results did not, however, reveal a massive presence of verbal- numerical discordance. Globally, the mean numerical scores for the four versions of the six problems showed a high positive correlation with the mean verbal scores (r = 0.77). The results also revealed sensitivity of numerical probability evaluations to information which could be assumed to be irrelevant for this response mode. A case in point is the rather simple Candidate problem, where the weight of the target could be calculated quite easily from the information provided in the text. Trend could thus be considered irrelevant information in the numerical response mode. Nonetheless, numerical and verbal scores showed sensitivity to the trend in a similar manner.

The question remains open as concerns the differences in information processing that lead to verbal-numerical discordance. Close inspection of the response distributions suggests nonetheless that the selective use of information may be at the source of verbal-numerical discordance.

Fig. 1 shows the distribution of verbal scores obtained in response to versions A and B of the Epidemic problem (see part 3). In version A, it was only stated that ‘the rate of Melyatitis was 10 times higher in

20 EPIDEMIC PROBLEM

0 A (10 times higher) 0

t 0 15

z

y 10 ar

: y 5

0 33 1

VERBAL SCORE

Fig. 1. Epidemic problem. Distributions of verbal scores for versions A and B.

4x M. Gonzulez, C. Frenck-Mcstre / Numerid cs. Lwbal probabilities

Bankina than in the world population’. The obtained verbal scores, which on the whole showed a higher scoring of probable as compared to improbable, thus provide an estimation of the verbal translation of ‘10 times higher’ which was the sole information present in this version. Version B of the problem indicated that ‘the rate of Melyati- tis in the world population was 5 in 100,000, and 10 times higher than this in Bankina’. Fig. 1 shows that the distribution of verbal scores in this version was bimodal. Our interpretation is that it juxtaposes two highly dissimilar unimodal distributions. The first distribution, which shows a higher scoring of improbable over probable, would correspond to the verbal translation of ‘10 times higher than 5 in lOO,OOO’, that is 50 in 100,000. The second distribution, which shows a higher scoring of probable over improbable, would correspond to the verbal translation of ‘10 times higher’ (i.e. to the ratio 10: 1). According to this interpretation, the second distribution should show the same profile as that observed for version A of this problem, given that it corresponds to the translation of the same information. Inspection of fig. 1 reveals results that are indeed in line with the interpretation.

The juxtaposition of two distributions in version B of the Epidemic problem suggests that subjects selectively incorporated different pieces of information in their responses. Certain subjects used the rate of disease by combining the information ‘10 times higher’ with ‘5 in 100,000’. Others used only the information ‘10 times higher’. Stated otherwise, subjects’ verbal responses translated either the rate of disease or its ratio.

Probability evaluations can thus be made on the basis of various types of information. Teigen (1988a) has argued that the use of different terms (e.g. ‘risk’, ‘luck’, ‘probable’) is linked to the attributed origins of probability (for example chance factors and dispositions). An analogous argument can be made in the present study. Let us speak in terms of factors relevant to probability rather than the origins of probability. In our problems, we presented information relevant to various factors such as trend, global weight, local weight, base rate, likelihood ratio, and so on. In line with Teigen, we can hypothesize that each of these factors is preferentially translated by a particular mode of expression. For example, information about global weight would be preferentially translated by a percentage, and information about local weight would be preferentially translated verbally, for example in terms of how ‘probable’ an event is. We can further

M. Gonzalez, C. Frenck-Mestre / Numerical LX verbal probabilities 49

assume that in the case where information was provided about two factors, subjects made use only of the information about the factor preferentially translated in the imposed response mode. Hence, verbal-numerical discordance would occur in problems providing for example information about factors A and B such that A is preferentially translated in the verbal response mode and B in the numerical response mode. This notion of strength of use of information about a given factor, linked to a given response mode, seems a plausible hypothesis to account for the phenomena of verbal-numerical discordance found in the present study.

Appendix

Henri the Diplomat Henri is a diplomat. He has recently learned that he will be assigned to a new

embassy. The location of the new assignment will be decided as a function of 6 possible

embassies that have vacant positions. They are:

Global weight: 3 : 3; Local weight: 3 : 3 3 vacancies in the United States:

in Washington DC in Los Angeles in New York City

3 vacancies in Africa: in Brazzaville in Abidjan in Dakar

Global weight: 3: 3; Local weight: 3: 1

3 vacancies in the United States: in Washington DC in Los Angeles in New York City

1 vacancy in Africa: in Brazzaville

1 vacancy in western Europe: in Rome

1 vacancy in Australia: in Sydney

50 M. Gonzalez, C. Frenck-Mestre / Numerical vs. verbal probabilities

What do you think the chances are that Henri be sent to the United States?

Verbal response mode

It is that Henri be sent to the United States

probable 1 2 3 4 5 6 7 improbable 1 2 3 4 5 6 7

Inadequate Adequate

Circle the degree of adequacy (1 to 7) of each expression.

Numerical response mode

There is a _ probability that Henri be sent to the United States.

0% 50% 100%

Give your response by marking one of the 7 divisions on the scale O-100%.

References

Bar-Hillel, M., 1980. The base-rate fallacy in probability judgments. Acta Psychologica 44,

21 l-233. Bar-Hillel. M., 1983. ‘The base rate fallacy controversy’. In: R.W. Scholz (ed.), Decision making

under uncertainty (pp. 39-61). Amsterdam: North-Holland. Beyth-Marom, R., 1982. How probable is probable? A numerical translation of verbal probability

expressions. Journal of Forecasting 1, 257-269. Brun, W. and K.H. Teigen, 1988. Verbal probabilities: Ambiguous. context-dependent, or both?

Organizational Behavior and Human Decision Processes 41, 390-404. Bryant, G.D. and G.R. Norman, 1980. Expressions of probability: Words and numbers. New

England Journal of Medicine 302, 411.

Budescu, B.V. and T.S. Wallsten, 1985. Consistency in interpretation of probabilistic phrases. Organizational Behavior and Human Decision Processes 36, 391-405.

Budescu, D.V., S. Weinberg and T.S. Wallsten, 1988. Decisions based on numerically and verbally expressed uncertainties. Journal of Experimental Psychology: Human Perception and

Performance 14, 281-294. Hammerton, M., 1973. A case of radical probability estimation. Journal of Experimental

Psychology 101, 252-254. Kahneman, D. and A. Tversky, 1973. On the psychology of prediction. Psychological Review 80,

237-25 I. Katz, J., 1984. Why doctors don’t disclose uncertainty. The Hasting Center Report 14, 35-44.

M. Gonzalez, C. Frenck-Mestre / Numerical us. verbal probabilities 51

Lichtenstein, S. and J.R. Newman, 1967. Empirical scaling of common verbal phrases associated

with numerical probabilities. Psychonomic Science 9, 563-564.

Nakao, M.A. and S. Axelrod, 1983. Numbers are better than words. American Journal of

Medicine 74, 1061-1063. Rapoport, A., T.S. Wallsten and J.A. Cox, 1987. Direct and indirect scaling of membership

functions of probability phrases. Mathematical Modelling 9, 397-411.

Rapoport, A., T.S. Wallsten, 1. Erev and B.L. Cohen, 1990. Revision of opinion with verbally and numerically expressed uncertainties. Acta Psychologica 74, 61-79.

Reagan, R.T., F. Mosteller and C. Youtz, 1989. Quantitative meanings of verbal probability

expressions. Journal of Applied Psychology 74, 433-442.

Reiss, E., 1984. In quest of certainty. American Journal of Medicine 77, 969-971.

Robertson, W.O., 1983. Quantifying the meaning of words. Journal of the American Medical

Association 249, 2631-2632. Teigen, K.H., 1988a. The language of uncertainty. Acta Psychologica 68, 27-38.

Teigen, K.H., 1988b. When are low-probability events judged to be ‘probable’? Effects of

outcome-set characteristics on verbal prbbability estimates. Acta Psychologica 68, 157-174. Wallsten, T.S., D.V. Budescu, A. Rapoport, R. Zwick and B. Forsyth, 1986a. Measuring the

vague meanings of probability terms. Journal of Experimental Psychology: General 115,

348-365.

Wallsten, T.S., S. Fillenbaum and J.A. Cox, 1986b. Base rate effects on the interpretations of

probability and frequency expressions. Journal of Memory and Language 25, 571-587.

Zadeh, L.A., 1975a. The concept of a linguistic variable and its application to approximate

reasoning. I. Information Sciences 8, 199-249.

Zadeh, L.A., 1975b. The concept of a linguistic variable and its application to approximate

reasoning. II. Information Sciences 8, 301-357.

Zimmer, A.C., 1983. ‘Verbal vs. numerical processing of subjective probabilities’. In: R.W.

Scholz (ed.), Decision making under uncertainty (pp. 159-182). Amsterdam: North-Holland.

Zimmer, A.C., 1984. A model for the interpretation of verbal predictions. International Journal

of Man-Machine Studies 20, 121-134.

Zwick, R., 1988. The evaluation of verbal models. International Journal of Man-Machine

Studies 29, 149-157.

Zwick, R., D.V. Budescu and T.S. Wallsten, 1988. ‘An empirical study of the integration of

linguistic probabilities’. In: T. ZCtknyi (ed.), Fuzzy sets in psychology (pp. 91-125). Amster-

dam: North-Holland.

Zwick, R. and T.S. Wallsten, 1989. Combining stochastic uncertainty and linguistic inexactness:

Theory and experimental evaluation of four fuzzy probability models. International Journal

of Man-Machine Studies 30, 69-111.

determinants of numerical versus verbal probabilities

Documents