ambiguous solicitation: ambiguous prescription

AMBIGUOUS SOLICITATION: AMBIGUOUS PRESCRIPTION

ROBERT GAZZALE, JULIAN JAMISON, ALEXANDER KARLAN and DEAN KARLAN∗

We conduct a two-phase laboratory experiment, separated by several weeks. Inthe first phase, we conduct urn games intended to measure ambiguity aversion on arepresentative population of undergraduate students. In the second phase, we invitethe students back with four different solicitation treatments, varying in the ambiguityof information regarding the task and the payout of the laboratory experiment. We findthat those who return do not differ from the overall pool with respect to their ambiguityaversion. However, no solicitation treatment generates a representative sample. Theambiguous task treatment drives away the ambiguity averse disproportionally and thedetailed task treatment draws in the ambiguity averse disproportionally. Finally, thestandard laboratory recruitment e-mail disproportionately draws in those who are notambiguity averse. (JEL A12, C81, C90)

I. INTRODUCTION

Sample selection issues are relevant for anyempirical exercise with human subjects. Westudy this problem directly, in the context of lab-oratory experiments in economics. However, theissue is at least as salient for “field experiments”(see Harrison and List 2004 for an overview).Among other issues, they still need to recruittheir subjects, and thus there is the possibilityfor selection bias. Indeed, no sample is likelyto be fully representative; Gronau (1974) is anearly paper that worries about just such an effecton wage selectivity in labor markets. We discussfurther in the conclusion the relevance for bothfield experiments and for fully naturally occur-ring observed data.

To directly assess such effects, we hypoth-esize in advance that a particular recruitmentprocedure will affect the composition of thesubjects who show up to participate. We use

*The authors thank Williams College for funding.Gazzale: Assistant Professor of Economics, Williams Col-

lege, 24 Hopkins Hall Drive, Williamstown, MA 01267.Phone 413-597 4375, Fax 413-597-4045, [email protected]

Jamison: Senior Economist, Research Center for Behav-ioral Economics, Federal Reserve Bank of Boston, 600Atlantic Avenue, Boston, MA 02210. Phone 617-973-3017, Fax 617-973-3957, E-mail: [email protected]

Karlan: Williams College, 24 Hopkins Hall Drive,Williamstown, MA 01267. E-mail [email protected]

Karlan: Professor of Economics, Yale University, P.O. Box208269, New Haven, CT 06520. Phone 203-432-4479,Fax 203-432-5591, E-mail [email protected]

standard laboratory protocols for comparability,but the implications are similar for either the labor the field. In particular, we vary the amountof information (about the task to be performedand/or the expected payment) revealed at thetime of recruitment. This is a dimension thatvaries in any case, but that is not often explicitlyconsidered or controlled for. It also has a naturaltheoretical link to ambiguity aversion: an aver-sion to uncertainty over states of the worldabout which the probabilities are unknown.1 Wehypothesize that potential subjects who are moreambiguity averse will be less likely to choose toparticipate if they have less information abouttheir possible outcomes. Although we focus onthis single aspect, we stress that our concern isbroader. Selection biases are likely to be presentin almost all situations, along a variety of dimen-sions, and by definition they are unusually dif-ficult to test for and to control for.

To test for this effect here, we begin byinducing a representative sample of undergrad-uates, namely almost all students in severalpre-occurring groups, to voluntarily participatein the first phase of our experiment in whichwe measure ambiguity preferences (specific pro-cedures are described in Section II). This is

1. Ellsberg (1961) provides the canonical thought exper-iment, suggesting that individuals, if forced to choosebetween two lotteries with different amounts of informa-tion available, will prefer to bet on one with a known butunfavorable probability of winning rather than on one withan unknown probability. Camerer and Weber (1992) reviewthe early experimental evidence generally confirming thisintuition. In a more recent study, Ahn et al. (2007) comparevarious empirical measures.

1

Economic Inquiry(ISSN 0095-2583)

doi:10.1111/j.1465-7295.2011.00383.x© 2011 Western Economic Association International

2 ECONOMIC INQUIRY

by no means representative of the populationat large, but if anything it is more homo-geneous—making it more difficult for us toobserve selection effects within that group. Infact, even in this case, we do find a significantselection effect when those same students areinvited to participate in a follow-up experimentvia a randomly varied recruitment e-mail. Inparticular, none of the e-mails that we used suc-cessfully led to the same underlying distributionof types as existed in the base population (sam-ple frame).

The general issue of potential bias both insubject pools and in subject behaviors has beenconsidered by experimental psychologists formany years (Orne 1962; Rosenthal and Rosnow1973). Note that there are two distinct consid-erations: Who volunteers to participate in anexperiment to begin with? And does their behav-ior change relative to other settings? The lattereffect is sometimes referred to as a demand char-acteristic, with many studies finding that sub-jects appear to conform their behavior to thatwhich is “demanded” by the researcher. But ofcourse, without good evidence as to what thebaseline population looks like, it is difficult orimpossible to separate these effects. Experimen-tal economics may suffer slightly less from botheffects: the first because there is always pay-ment for participating and the second becauseoften we have been interested not in individualdifferences but rather in comparing institutionsor testing theories that are supposed to applyto everyone equally. Even within economics,this potential problem was discussed quite early(e.g., Kagel, Battalio, and Walker 1979), but ithas received little attention.

As it matures, however, experimental eco-nomics has become increasingly interested inbehavior differences among groups, and here theselection effects are more acute. For instance,there are (or have appeared to be) robust gen-der differences in a variety of behaviors. Oneof these is risk aversion, and in particular bid-ding behavior in first-price auctions. Womenbid higher than men do, which is less risky,and therefore often earn less money. Chen,Katuscak, and Ozdenoren (2009) replicate thisfinding, but then show that, if one controlsfor the stage of the women’s menstrual cycle,the differences disappear. There is an obviousselection story to back this up: nonmenstruatingwomen have higher estrogen levels, leadingthem to be both more likely to participate inan experiment in the first place (indicative of

pro-social or affiliative behavior) and to be less“aggressive” once they do.

Of course this is not proof of a selectioneffect, but by definition it is difficult to test thatportion of the population that tends not to par-ticipate in experiments. One indirect approach isto look at sorting behavior among subjects whohave already agreed to participate in general butwho can endogenously choose what task to per-form (e.g., what game to play, or indeed whetheror not to play a game at all). A number of recentpapers have explored this issue and found sig-nificant differences in behavior between thosewho were assigned to a treatment versus thosewho chose into it.2 Some of these papers arealso able to relate individual differences (e.g.,overconfidence or risk aversion) to the choiceof treatment, confirming the idea that underlyingpreferences affect not only people’s behavior,holding the environment fixed, but also whatenvironments people end up in. In particular,two papers have looked at endogenous entryinto auctions: Reiley (2005) manipulates reserveprices in a field setting; Palfrey and Pevnitskaya(2008) study the link between risk tolerance,auction participation, and bidding behavior. Allof these clearly have implications for the inter-pretation of any experimental results that wishto test for differences involving exactly thoseunderlying preferences.

Three recent papers directly address the issueof determining which subjects actually physi-cally show up at an experiment, although ofcourse neither of them can fully compare withthe (unknowable) general population—and nei-ther do we. The first to do this is Harrison,Lau, and Rutstrom (2009), which uses a fieldexperiment setting to look at selection effectsdepending on risk aversion. They find smalleffects overall, but a noticeable difference fromthe use of a guaranteed show-up fee: not sur-prisingly, such guarantees are more attractiveto those who are more risk averse. The sec-ond is Jamison, Karlan, and Schechter (2008),which studies the effect of deception in lab-oratory experiments. They find that deceivedwomen and deceived low earners are less likelyto show up for a second (nominally unrelated)experiment. However, they do not focus onthe specifics of the recruitment procedure itself.Finally, Malani (2008) looks at self-selection

2. Some examples include Camerer and Lovallo (1999),Lazear, Malmendier, and Weber (2006), Eriksson, Teyssier,and Villeval (2009), Falk and Dohmen (2006), andGaudecker, Van Soest, and Wengstrom (2008).

GAZZALE ET AL.: AMBIGUOUS SOLICITATION 3

into randomized medical trials, finding a linkbetween optimism concerning treatment efficacy(which is correlated with the treatment’s effectbecause of unobserved individual heterogeneity)and enrollment into the trial.

II. EXPERIMENT DESIGN AND RESULTS

A. Phase 1 Design: Measuring BaselineAmbiguity Aversion

Our goal in Phase 1 was to determine the ambi-guity preferences of a sample of subjects intowhich there would be little or no self-selectionspecific to our experiment. Our starting sampleframe was the population from which economicsexperiments typically solicit: undergraduate stu-dents. We recruited subjects using two methods,both of which are particularly common in eco-nomics experiments. First, we asked students ina number of introductory economics courses tocomplete an ambiguity-aversion survey in thefinal 10 minutes of a class. Second, a researcherapproached every student he/she encountered atcampus libraries and asked each student to com-plete the same ambiguity-aversion survey. Therewas almost no Phase 1 self-selection (beyond tak-ing undergraduate economics and frequenting thelibrary): all 94 students in the selected economicsclasses completed the survey, and 109 out of 111of the approached library students completed thesurvey.3

Both in the classroom and in the library,the researcher first asked students whether theywere willing to complete a brief survey in whichone in three students earns money based on herresponses. All students agreed to participate andsigned an informed consent form. The researcherhanded each student brief instructions, readthe instructions aloud, invited subjects to askquestions, and gave them a survey to complete.Subjects were not informed that they wouldreceive a future solicitation to participate in asubsequent experiment.

The survey (Appendix A) contains 10 of Ells-berg’s hypothetical urn gambles and collectsbasic demographic data. In each of five scenar-ios, there is a hypothetical urn containing 100balls whose distribution of black and red ballsis clearly stated (the known urn), and a second

3. We subsequently dropped six subjects from our sam-ple frame. Four were not on campus when we conductedPhase 2, and two learned of the study’s research objectivesand revealed this through informal communication with oneof the researchers.

hypothetical urn also containing 100 red andblack balls in total, but whose distribution ofred and black balls is clearly stated as unknown(the ambiguous urn).4 The five known urns areoffered in order: 50 red and 50 black balls; 40red and 60 black balls; 30 red and 70 black balls;20 red and 80 black balls; and 10 red and 90black balls. For each scenario, we presented thesubject with two gambles:

(1) If we paid you $10 for pulling a redball on your first try, would you pick from the“known” or the “ambiguous” urn?

(2) If we paid you $10 for pulling a blackball on your first try, would you pick from the“known” or the “ambiguous” urn?

Thus we presented 10 gambles to each sub-ject, and for each gamble asked from which urnthe subject would draw. Asking each gambletwice, once for red and once for black, elimi-nated the chance the subject had a preferencefor a color or mistrusted the administrator.

B. Phase 1 Ambiguity-Aversion Classification

On the basis of Phase 1 survey responses, weplace all Phase 1 participants into one of threecategories: more ambiguity averse (Table 1;27%), not more ambiguity averse (69%), andunable to classify (4%). To classify subjects,we look at the first “known” urn distributionat which a subject chooses the ambiguous urnwhen betting on whether a red ball will be thefirst ball drawn.5 We classify as “more ambi-guity averse” those students who first choosethe ambiguous urn for the red-ball bet when theknown urn contains 30 or fewer red balls, andwho continue to choose the ambiguous urn forthe red-ball bet when the known urn containseven fewer red balls. These subjects chose theknown urn for both bets when the known urncontains either 50 or 40 red.

Seven subjects switched from the ambiguousurn to the known urn even though the knownurn in each round became worse. We presumethese individuals were not paying attention, orwere not understanding the questions well, andthus categorize them as “unable to classify” anddrop them from the analysis.

4. We inform subjects that the distribution of red andblack balls in the ambiguous urn is constant for all decisions.We do not refer to urns as ambiguous or known.

5. Recall that the first-known urn contains 50 red and50 black balls, and each subsequent known urn contains 10fewer red balls and 10 more black balls.

4 ECONOMIC INQUIRY

TABLE 1Full Distribution of Ambiguity Urn Decisions

from Phase 1

Frequency

Chose ambiguous urn forboth red and black at40/60

52.5%

Coded as “0” inbinary variable“More AmbiguityAverse”

Chose ambiguous urn forboth red and black at50/50

63.0%

Chose ambiguous urn foreither red or black at50/50

31.5%

Switched to ambiguousurn at 40/60

12261.9%


4623.4%

Coded as “1” inbinary variable“More AmbiguityAverse”


31.5%


00.0%

Never chose ambiguousurn

52.5%

Unable to classify (i.e.,switched back andforth)

73.6%

Total 197

We classify all other subjects as “not moreambiguity averse.” Note that we do not have acategory “ambiguity neutral” or “ambiguity seek-ing” because only nine and five subjects, respec-tively, would have been categorized as such. Wethus categorize as “not more ambiguity averse”the combination of ambiguity seeking (five sub-jects), ambiguity neutral (nine subjects), and thosethat switched at the first round, when the oddswere 40% for the known urn (122 individuals).

In Table 2 we show, with both ordinary leastsquares and probit specifications, the (lack of) cor-relation between this classification and location ofexperiment, gender, year in school, and major.

C. Phase 2 Solicitation: Observing theDecision to Participate

The real “experiment” in Phase 2 is simplythe decision to respond to our e-mail solicitation.Our goal is to determine whether ambiguity pref-erences affect the decision to participate in lab-oratory experiments, and then more specificallywhether different e-mail solicitations affect thisselection decision differentially.

TABLE 2Determinants of Ambiguity Classification

Model ProbitOrdinary Least

Squares

Dependent Variable

More AmbiguityAverse in First

Experiment

More AmbiguityAverse in First

Experiment(1) (2)

Female 0.038 (0.068) 0.039 (0.068)First experiment

conducted in library0.086 (0.075) 0.087 (0.077)

First experimentconducted ineconomics class

Omitted Omitted

Freshman 0.193 (0.122) 0.174 (0.111)Sophomore 0.192 (0.125) 0.172 (0.111)Junior 0.117 (0.133) 0.100 (0.116)Senior + graduate Omitted OmittedEconomics major 0.093 (0.115) 0.089 (0.109)Psychology major −0.073 (0.208) –0.073 (0.238)R2 0.022 0.025Observations 190 190

Note: Marginal effects are reported for probit specifica-tion. Robust standard errors are in parentheses.

Had anything been significant statistically, then * wouldhave indicated significance at 10%, ** at 5%, and *** at1%. Information on major is unavailable for freshmen andsophomores.

We randomly assigned subjects to one of fourrecruitment treatments (i.e., ambiguity classifi-cation is orthogonal to treatment assignment).Treatments differ only in the amount of detailprovided in the invitation e-mail. We employa 2 × 2 design, with each respondent receiv-ing either an ambiguous or detailed descrip-tion of their task and either an ambiguous ordetailed description of their payout. The “stan-dard” e-mail sent at Williams College, and else-where (Davis and Holt 1992), is closest to theambiguous task/ambiguous pay e-mail.

Details on each e-mail treatment are as fol-lows (the full text is in Appendix B):

(1) Ambiguous task/ambiguous pay: “I amwriting to inform you of an opportunity to par-ticipate in an Economics Department experimenton Tuesday, February 20th from 7:00 p.m. until10:00 p.m. in Hopkins 108. You will earn either$10 or $20 by participating in this experimentand the session will last about 30 minutes.”6

(2) Ambiguous task/detailed pay: Here, thepayout section from the ambiguous e-mail isreplaced by the following: “You will earn either

6. Although this is more information than is usuallyprovided about expected earnings, unlike the detail paye-mails, the exact distribution of payouts is unknown.


TABLE 3Analysis of Who Participated in Second Experiment Mean and Standard Error

Solicitation Treatment in Second Experiment

Full Samplein First

Experiment

AmbiguousTask

Solicitation

DetailedTask

Solicitation

Chi-Square,p-Value(2) = (3)

AmbiguousPayment

Solicitation

DetailedPayment

Solicitation

Chi-Square,p-Value(5) = (6)

(1) (2) (3) (4) (5) (6) (7)

Number ofobservations in firstexperiment

190 95 95 — 94 96 —

Number participatedin secondexperiment

34 22 12 — 21 13 —

Percent participated insecond experiment

18%(0.028)

23%(0.044)

13%(0.034)

0.058 22%(0.043)

14%(0.035)

0.114

(A) Proportion moreambiguity averse infirst experiment

0.284(0.033)

0.284(0.047)

0.284(0.047)

1.000 0.287(0.047)

0.281(0.046)

0.927

(B) More ambiguityaverse in firstexperiment ANDparticipated insecond experiment

0.294(0.079)

0.136(0.075)

0.583(0.149)

— 0.333(0.105)

0.231(0.122)

—

(C) More ambiguityaverse in firstexperiment ANDdid NOT participatein secondexperiment

0.282(0.036)

0.329(0.055)

0.241(0.047)

— 0.274(0.053)

0.289(0.050)

—

Chi-square test,p-value (B)=(C)

0.888 0.079 0.014 — 0.596 0.663 —

Note: Observations include only those subjects whose Phase I ambiguity preferences we were able to classify.

$10 or $20 by participating in this experiment.The experiment is designed so that you have a50% chance of earning $10 and a 50% chanceof earning $20.”

(3) Detailed task/ambiguous pay: Here, eachparticipant is also informed that they will playa game in which they will decide how much oftheir participation fee they want to contribute tocharity, and then will play games of uncertainty,choosing between known and ambiguous urns.

(4) Detailed task/detailed pay: The detailedtask and detailed pay, as noted from the above.

Appendix C provides the detail on theactivities in Phase 2. We do not use any of thesedata in this article, because the sample size is toosmall for meaningful distributional analysis (toobserve whether the differential selection drewin different people, which then would lead todifferent analytical results for the Phase 2 gamesthemselves).

Table 3 presents the key selection results ascomparison of means, and Table 4 presents themin a probit specification. First note that Table 3,Row A demonstrates orthogonality of ambiguity

aversion to assignment to each e-mail treatment,and Table A1 demonstrates the same for otherknown demographic variables.

We have two key hypotheses:

Hypothesis 1: Those who participate in laboratoryexperiments do not differ with respect to ambiguityaversion from those who do not participate.

Table 3 column 1 tests this hypothesis withmean comparison, and Table 4 column 1 teststhis hypothesis with a probit specification. Wecannot reject the null hypothesis. Those whoparticipate in Phase 2 are no more or less ambi-guity averse than those who do not. Note thatthis is a pooled analysis, across all four treatmentsolicitation e-mails. Thus, although we cannotreject the null hypothesis, this null is under thesetting of a blend of solicitation approaches. Wenow turn to examine heterogeneity generated bythe different solicitations.

Hypothesis 2: The level of ambiguity in each solic-itation does not generate differential selection onambiguity personality characteristics with respect towho participates.

6 ECONOMIC INQUIRY

TABLE 4Determinants of Whether a Subject Participated in Second Experiment Probit

Participated in Second Experiment

Binary Dependent Variable (1) (2) (3) (4)

More ambiguity averse in firstexperiment

0.012(0.060)

0.011(0.060)

0.208(0.140)

−0.166(–0.083)∗∗∗

E-mail with ambiguous task — 0.069(0.054)

0.164(0.063)∗∗∗

—

E-mail with ambiguous payout — 0.094(0.054)∗

0.080(0.062)

—

E-mail with ambiguous taskAND recipient “moreambiguity averse” in firstexperiment

— — −0.184(0.039)∗∗∗

—

E-mail with ambiguous payoutAND recipient “moreambiguity averse” in firstexperiment

— — 0.012(0.121)

—

Controls for year gender, year inschool, major, and location offirst experiment

Yes Yes Yes Yes

Pseudo-R2 0.098 0.125 0.174 0.393

Number of observations 190 190 190 45

Note: We exclude those subjects whose Phase 1 ambiguity we were unable to classify. Model (4) observations are thosesubjects in the ambiguous task/ambiguous pay treatment. Marginal effects are reported for coefficients. Robust standard errorsare in parentheses.

Significance ∗at 10%, ∗∗at 5%, and ∗∗∗at 1%.

In Table 3 columns 2, 3, 5, and 6, we testthis hypothesis with respect to ambiguity onboth task and payout. Row A in columns 2and 3 shows the average ambiguity aversionfor those who received either the ambiguoustask or the detailed task is 28.4% in both cases(the randomization was stratified on ambiguity,hence the perfect orthogonality).

However, comparing Rows B and C showsthat the ambiguous task generates differentialselection toward those less ambiguity averse (p-value 0.079), and that the detailed task similarlygenerates reverse selection toward the moreambiguity averse (p-value 0.014). Similar testsfor ambiguity on the payment, however, do notyield statistically significant differences (nor arethey signed as predicted). We discuss this inthe conclusion with conjectures as to why theambiguous payout treatment did not generatedifferential selection, whereas the ambiguoustask treatment did.

Table 4 shows similar results, in a probitspecification. Column 2 shows that the ambigu-ous task generates a 6.9% point higher partici-pation rate (not statistically significant) and theambiguous payout generates a 9.4% point higherparticipation rate (significant at 10%). Column3 presents the key results on heterogeneity

induced by the solicitation. Here, we interactthe e-mail treatment with whether the individ-ual is ambiguity averse or not. We find thesame pattern shown in Table 3 that the ambigu-ous task treatment deters the ambiguity aversefrom participating (significant at 1%), but thatthe ambiguous payout treatment does not gener-ate differential selection patterns.

Of particular interest to the laboratory exper-imentalist is the extent to which the standardrecruitment e-mail (ambiguous task/ambiguouspay) draws in an unbiased sample of the subjectpool. Column 4 shows estimation results froma probit model where observations are limitedto the ambiguous task/ambiguous pay treatment.The participation rate is almost 17% pointslower for the more ambiguity averse group.Therefore the standard solicitation method doesnot elicit a representative sample.

III. CONCLUSION

We examine a new facet of selection into lab-oratory experiments: ambiguity aversion. First,we find that our laboratory instrument measuringambiguity aversion does help predict real-worldbehavior (the decision of whether to participate


in an economics experiment). Furthermore, thechoices made on this instrument cannot be pre-dicted using subject characteristics normally col-lected in experiments. Second, we find thatthe method of solicitation generates potentiallyimportant heterogeneity with respect to ambi-guity aversion in the sample frame that partici-pates in experiments. Thus if ambiguity aversioncould influence the choices participants make,experimentalists should note that the solicitationused could generate higher or lower participa-tion rates, depending on how much informa-tion is given in the solicitation. Further researchcould shed insight on two areas. First, with alarger sample size, further analysis could be per-formed to examine whether analytical resultschange depending on the solicitation method.Second, it would be useful to know whetherother solicitation methods could generate a morerepresentative population. For instance, payingmore money could generate higher participa-tion rates; or alternative wording, somewhere inbetween our ambiguous and detailed treatments,may yield the optimal results. In our setting, it iskey to note that we found no treatment, includ-ing the standard ambiguous text/ambiguous pay,which drew in a representative population.

Although we find differential selection fromthe ambiguous task treatment, we do not finddifferential selection from the ambiguous payouttreatment. We have two conjectures for why thismay be. First, perhaps the information was sim-ply not ambiguous enough. We stated that theywould win between $10 and $20, and althoughthis adheres to the canonical urn question fairlyaccurately, if read quickly it could be perceivedas more informative than intended. Second, it ispossible that recipients of the e-mail did not trustthe researcher (whereas in the urn questions,mistrust of the researcher should not confoundthe analysis by design), and thus simply assumedthat this really means $10, except for very fewwho would win the $20.

Nonrandom selection into experiments posessignificant challenges in extrapolating findingsto the real world (i.e., external validity). Aresearcher quantifying a preference parameter ofinterest (e.g., a risk-aversion coefficient) faces abaseline hurdle in that this parameter may differbetween the population from which he/she issampling and their population of interest. Biasedselection into the experiment only decreasestheir ability to make inferences about the pop-ulation of interest. Although these selectioneffects are likely less important to the researcher

documenting a treatment effect, they cannot beignored. Even if the researcher documents aneffect in some sample, it may not be clear whoexactly is in that sample and whether the treat-ment effect owes its existence to (uncontrolledfor) sample composition. For instance, an inter-vention to improve social cohesion may appearnot to work only because the self-selected par-ticipants are more pro-social than typical (whichis unlikely to be correlated with standard demo-graphic characteristics) and therefore have lessneed of such help.

These results are important not just for labo-ratory experiments; similar issues apply to fieldexperiments. We think about two types of fieldexperiments here, “artifactual” or “framed” fieldexperiments, using the Harrison and List (2004)taxonomy, and “natural” field experiments orfor that matter any observational data collection.First, with respect to those cognizant of being“researched” (artifactual or framed), the resultshere are potentially just as applicable in the fieldas in the laboratory: no method of solicitation inour experiment generated a representative sam-ple frame. Typical methods in the field couldhave similar issues, for example, those who aremore social, those who are more likely to thinkthe games could lead to NGO handouts, thosewho are more curious, and so forth are all morelikely to participate, and the method of solic-itation could exacerbate any of these issues.7

Regarding “natural” field experiments or anysurveying process, the issues raised here are alsorelevant, particularly when subjects are asked toselect into a novel product or institution. Thefundamental idea is not new: external validity.This paper sheds insight into how the solici-tation method can generate more (or, ideally,less) selection which then influences the externalvalidity of this study.

APPENDIX A. PHASE 1 EXPERIMENT INSTRUCTIONS

In this experiment you will be asked to make a seriesof choices. In each scenario, there are two urns. Both willalways contain 100 balls, each ball being either red or black.In each scenario, you will know the exact number of blackand red balls in one urn, but you will not know the numberof each color in the second urn, only that there are 100 ballsin the second urn and every ball is either red or black. Theballs are well mixed so that each individual ball is as likelyto be drawn as any other.

After all questionnaires have been completed, the exper-imenter will select at random one-third of all questionnaires.

7. Recall Malani (2008), which finds exactly such aproblem in the context of medical trials.

8 ECONOMIC INQUIRY

For each questionnaire selected, the experimenter will ran-domly select one of the five scenarios, with each scenario aslikely to be drawn as any other. The experimenter will thenrandomly select one of the two questions within the selectedscenario, with each question as likely to be selected as theother. Finally, if your questionnaire is selected, a ball will bedrawn on your behalf, with each ball as likely to be drawnas any other.

Finally, please note that there are no tricks in thisexperiment. Although in each scenario there is an urn forwhich you do not know the number of black and red balls,the number of unknown balls of each type already has beenselected at random and is on file with the Williams CollegeDepartment of Economics. Likewise, the pull of a ball froma chosen urn will truly be performed at random via a processoverseen by the Department of Economics.

[We present five of the following questions, with M =1,2,3,4,5.]

Decision # {M}Urn A: 100 balls: {50−(M − 1)∗10} red,

{50+(M − 1)∗10} blackUrn C: 100 balls: ? red, ? black

If I were to give you $10 if you pulled a red ball on yourfirst try, from which urn would you choose to draw?

� Urn A � Urn C

If I were to give you $10 if you pulled a black ball onyour first try, from which urn would you choose to draw?

� Urn A � Urn C

APPENDIX B. TEXT OF INVITATION E-MAILS

Ambiguous E-mail

I am writing to inform you of an opportunity toparticipate in an Economics Department experiment onTuesday, February 20th from 7:00 p.m. until 10:00 p.m.

in Hopkins 108. You will earn either $10 or $20 byparticipating in this experiment and the session will lastabout 30 minutes.

Detailed Task, Ambiguous Payment E-mail

I am writing to inform you of an opportunity to partic-ipate in an Economics Department experiment on Tuesday,February 20th from 7:00 p.m. until 10:00 p.m. in Hopkins108. You will earn either $10 or $20 by participating in thisexperiment, and the session will last about 30 minutes. Theexperiment will consist of two sections.

In the first section, you will have the opportunity, inprivate, to donate part of your show-up fee to a charity.The amount you give to the charity will be matched by theexperimenter.

In the second section, you will be asked to make a seriesof decisions. For each decision, you will be asked to chooseone of two options where the outcome of each option isuncertain. In some decisions, you will know the probabilityof each outcome within each option. In other decisions, youwill not know the probability of each outcome for one ofthe options. After you have made your decisions, we willrandomly select some of your decisions and you will be paidaccording to your choices.

Ambiguous Task, Detailed Payment E-mail

I am writing to inform you of an opportunity to partic-ipate in an Economics Department experiment on Tuesday,February 20th from 7:00 p.m. until 10:00 p.m. in Hopkins108. You will earn either $10 or $20 by participating in thisexperiment. This experiment is designed so that you have a50% chance of earning $10 and a 50% chance of earning$20. The session will last about 30 minutes.

Detailed Task, Detailed Payment E-mail

I am writing to inform you of an opportunity toparticipate in an Economics Department experiment on

TABLE A1Verification of Orthogonality of Assignments to Treatments to Data from First Experiment Means

FullSample

ReceivedAmbiguous

PayoutE-mail

Solicitation

ReceivedDetailedPayoutE-mail

Solicitation

Chi-Square,p-Value(2)�=(3)

ReceivedAmbiguousTask E-mailSolicitation

ReceivedDetailed

Task E-mailSolicitation

Chi-Square,p-Value(5)�=(6)

(1) (2) (3) (4) (5) (6) (7)

More ambiguityaverse in firstexperiment

0.29 0.29 0.29 0.00 (0.96) 0.29 0.29 0.00 (0.96)

Female 0.48 0.44 0.50 0.61 (0.44) 0.40 0.54 3.70 (0.06)First experiment

conducted inlibrary

0.52 0.57 0.48 1.46 (0.23) 0.46 0.58 2.70 (0.10)

Freshman 0.38 0.36 0.39 0.12 (0.73) 0.40 0.35 0.68 (0.41)Sophomore 0.28 0.27 0.29 0.04 (0.84) 0.28 0.28 0.01 (0.91)Junior 0.19 0.20 0.17 0.26 (0.61) 0.20 0.17 0.26 (0.61)Economics major 0.12 0.12 0.11 0.04 (0.85) 0.16 0.07 3.89 (0.05)Psychology major 0.02 0.02 0.02 0.00 (0.99) 0.03 0.01 1.00 (0.32)Number of

observations197 99 98 99 98

Note: The definition of “more ambiguity averse” is given in detail in Table 1.


Tuesday, February 20th from 7:00 p.m. until 10:00 p.m.in Hopkins 108. You will earn either $10 or $20 byparticipating in this experiment. This experiment is designedso that you have a 50% chance of earning $10 and a50% chance of earning $20. The session will last about30 minutes. The experiment will consist of two sections.

In the first section, you will have the opportunity, inprivate, to donate part of your show-up fee to a charity.The amount you give to the charity will be matched by theexperimenter.

In the second section, you will be asked to make a seriesof decisions. For each decision, you will be asked to chooseone of two options where the outcome of each option isuncertain. In some decisions, you will know the probabilityof each outcome within each option. In other decisions, youwill not know the probability of each outcome for one ofthe options. After you have made your decisions, we willrandomly select some of your decisions and you will be paidaccording to your choices.

All Four E-mails

To sign up to participate in this experiment, please clickon the link below.

APPENDIX C. PHASE 2 EXPERIMENTINSTRUCTIONS

[Italicized text in brackets details how subject instruc-tions vary. Text in braces identifies the alternative text.]

Welcome to this experiment on decision making andthank you for being here. You will be compensated foryour participation in this experiment, although the exactamount you will receive will depend on the choices youmake, and on random chance. Even though you will make20 decisions, only one of these will end up being usedto determine your payment. Please pay careful attention tothese instructions, as a significant amount of money is atstake.

Information about the choices that you make during theexperiment will be kept strictly confidential. To maintainprivacy and confidentiality, please do not speak to anyoneduring the experiment and please do not discuss your choiceswith anyone even after the conclusion of the experiment.

This experiment has four parts. First, you will be askedto make a series of decisions regarding charitable donations.In the second and third sections, you will be asked to makea series of choices between options, where the outcome ofeach option is not known with certainty. Finally, you willbe asked a series of questions which you will either agree ordisagree with along a scale. More detailed instructions willfollow in each section.

Part I

Today you received four envelopes: a “Start” envelope,a “Me” envelope, a “1” envelope, and a “2” envelope. In thestart envelope you will find 10 $1 bills and 10 dollar-sizepieces of blank paper. You will now have the opportunityto share part or all of the $10 with one or both of twocharities. Any money that you donate to either charity willbe matched, meaning every dollar you donate will result inthe charity receiving two dollars. You may donate as muchor as little of the $10 to each of these charities as you wishby placing dollar bills in the corresponding envelopes. At

the end of the experiment, you will keep the “Me” envelopeand any dollar bills you place in that envelope.

[A subject receives one of four versions. In half, Oxfamis the known charity while Habitat for Humanity is ambigu-ously described, whereas in the other half, Habitat forHumanity is the known charity while Oxfam is ambiguouslydescribed. We controlled for order effects. The ambiguousdescription is in braces.]

(1) Envelope 1: Habitat for Humanity, a nonprofitorganization that builds homes for those in need which hasbeen instrumental in Hurricane Katrina relief efforts in theUnited States. {A nonprofit organization that works to helpvictims of natural disasters in the United States.}

(2) Envelope 2: Oxfam, a nonprofit organization thatworks to minimize poverty through relief and developmentwork in Africa, committed to creating lasting solutions toglobal poverty, hunger, and social injustice. {A nonprofitorganization that works to alleviate poverty in Africa.}

Part II

You will be making 10 choices between two lotteries,such as those represented as “Option A” and “Option B”below. The money prizes are determined by the computerequivalent of rolling a 10-sided die. Each outcome, 1–10, isequally likely. A computer generated “roll” for that decisionwill be made and you will be paid based on your decision.

Finally, please note that there are no tricks in thisexperiment. The roll will truly be performed at random viaa process overseen by the Department of Economics.

[We present 10 of the following questions, with N =1,2, . . . ,9,10.]

Decision {N}If you choose Option A in the row shown below, you willhave a {N} in 10 chance of earning $5.50 and a {10–N} in

10 chance of earning $4.40. Similarly, Option B offers a{N} in 10 chance of earning $10.60 and a {10 − N} in 10

chance of earning $0.28.Option A Option B

$5.50 if the die is 1 $10.60 if the die is 1$4.40 if the die is 2–10 $0.28 if the die is 2–10

� Option A � Option B

Part III

In this section you will be asked to make a series ofchoices. In each scenario, there are two urns. Both willalways contain 100 balls, each ball being either red or black.In each scenario, you will know the exact number of blackand red balls in one urn, but you will not know the numberof each color in the second urn, only that there are 100 ballsin the second urn and every ball is either red or black. Theballs are well mixed so that each individual ball is as likelyto be drawn as any other.

For each questionnaire selected, the experimenter willrandomly select one of the five scenarios, with each sce-nario as likely to be drawn as any other. The experimenterwill then randomly select one of the two questions within theselected scenario, with each question as likely to be selectedas the other. Finally, if your questionnaire is selected, a ballwill be drawn on your behalf, with each ball as likely to bedrawn as any other.

10 ECONOMIC INQUIRY

Finally, please note that there are no tricks in thisexperiment . Although in each scenario there is an urn forwhich you do not know the number of black and red balls,the number of unknown balls of each type already has beenselected at random and is on file with the Williams CollegeDepartment of Economics. Likewise, the pull of a ball froma chosen urn will truly be performed at random via a processoverseen by the Department of Economics.

[We present five of the following questions, with M =1,2,3,4,5.]

Decision # {M}Urn A: 100 balls: {50−(M –1)*10} red, {50+(M − 1)*10}

blackUrn C: 100 balls: ? red, ? black

If I were to give you $10 if you pulled a red ball on yourfirst try, from which urn would you choose to draw?

� Urn A � Urn C

If I were to give you $10 if you pulled a black ball onyour first try, from which urn would you choose to draw?

� Urn A � Urn C

Part IV

Please answer the below questions according to yourown feelings, rather than how you think “most people”would answer. Please be as honest and accurate as youcan throughout, there is no right or wrong answer. Inaddition, please try not to let your response to one statementinfluence your responses to other statements. Think abouteach statement on its own. Please circle your response inthe table.

A = I agree a lotB = I agree a littleC = I neither agree nor disagreeD = I disagree a littleE = I disagree a lotIn uncertain times, I usually

expect the best.A B C D E

It’s easy for me to relax. A B C D EIf something can go wrong, it

will.A B C D E

I’m always optimistic about thefuture.

A B C D E

I enjoy my friends a lot. A B C D EIt’s important for me to keep

busy.A B C D E

I hardly ever expect things to gomy way.

A B C D E

I don’t get upset too easily. A B C D EI rarely count on good things

happening to me.A B C D E

Overall, I expect more goodthings to happen to me thanbad.

A B C D E

REFERENCES

Ahn, D., S. Choi, D. Gale, and S. Kariv. “Estimating Ambi-guity Aversion in a Portfolio Choice Experiment.”ELSE Working Papers 294, ESRC Centre for

Economic Learning and Social Evolution, UniversityCollege London, 2007.

Camerer, C., and D. Lovallo. “Overconfidence and ExcessEntry: An Experimental Approach.” American Eco-nomic Review, 89(1), 1999, 306–18.

Camerer, C., and M. Weber. “Recent Developments in Mod-eling Preferences: Uncertainty and Ambiguity.” Jour-nal of Risk and Uncertainty, 5(4), 1992, 325–70.

Chen, Y., P. Katuscak, and E. Ozdenoren. “Why Can’t aWoman Bid More Like a Man?” Working Paper,University of Michigan, 2009.

Davis, D. D., and C. A. Holt. Experimental Economics.Princeton, NJ: Princeton University Press, 1992.

Ellsberg, D. “Risk, Ambiguity, and the Savage Axioms.”Quarterly Journal of Economics, 75(4), 1961, 643–69.

Eriksson, T., S. Teyssier, and M.-C. Villeval. “Self-Selection and the Efficiency of Tournaments.” Eco-nomic Inquiry, 47(3), 2009, 530–48.

Falk, A., and T. J. Dohmen. “Performance Pay and Multi-Dimensional Sorting: Productivity, Preferences andGender.” IZA Discussion Paper No. 2001, Institute forthe Study of Labor, 2006.

Gaudecker, H.-M., A. Van Soest, and E. Wengstrom. “Selec-tion and Mode Effects in Risk Preference ElicitationExperiments.” CentER Discussion Paper No. 2008-11,CentER for Economic Research, Tilburg University,2008.

Gronau, R. “Wage Comparisons—A Selectivity Bias.”Journal of Political Economy, 82(6), 1974, 1119–43.

Harrison, G. W., M. I. Lau, and E. E. Rutstrom. “Risk Atti-tudes, Randomization to Treatment, and Self-Selectioninto Experiments.” Journal of Economic Behavior andOrganization, 70(3), 2009, 498–507.

Harrison, G. W., and J. A. List. “Field Experiments.” Jour-nal of Economic Literature, 42(4), 2004, 1009–55.

Jamison, J., D. Karlan, and L. Schechter. “To Deceive orNot to Deceive: The Effect of Deception on FutureBehavior in Laboratory Experiments.” Journal of Eco-nomic Behavior and Organization, 68(3–4), 2008,477–88.

Kagel, J. H., R. C. Battalio, and J. M. Walker. “VolunteerArtifacts in Experiments in Economics; Specificationof the Problem and Some Initial Data from a Small-Scale Field Experiment,” in Research in ExperimentalEconomics, edited by V. L. Smith. Greenwich, CT:JAI Press, 1979, 169–97.

Lazear, E. P., U. Malmendier, and R. A. Weber. “Sorting inExperiments with Application to Social Preferences.”National Bureau of Economic Research Working PaperNo. W12041, 2006.

Malani, A. “Patient Enrollment in Medical Trials: SelectionBias in a Randomized Experiment.” Journal of Econo-metrics, 144(2), 2008, 341–51.

Orne, M. T. “On the Social Psychological Experiment: WithParticular Reference to Demand Characteristics andTheir Implications.” American Psychologist, 17(11),1962, 776–83.

Palfrey, T., and S. Pevnitskaya. “Endogenous Entry andSelf-Selection in Private Value Auctions: An Exper-imental Study.” Journal of Economic Behavior andOrganization, 66(3–4), 2008, 731–47.

Reiley, D. “Experimental Evidence on the EndogenousEntry of Bidders in Internet Auctions,” in ExperimentalBusiness Research, Vol. II, edited by A. Rapoport andR. Zwick. Dordrecht, The Netherlands: Springer, 2005,103–21.

Rosenthal, R., and R. L. Rosnow. The Volunteer Subject.New York: John Wiley and Sons, 1973.

ambiguous solicitation: ambiguous prescription

Documents